Nothing Changed and It Broke
Your automation worked for three months, then it didn't, and nobody touched anything. That's not a comfort. That's the whole problem.
The worst call I get isn't "it's down." Down is easy. Down trips an alarm, pages somebody, and gets fixed before lunch. The call that eats a whole Tuesday is the other one: "It used to do this, and now it doesn't, and I swear nobody touched anything." And here's the part that makes it maddening. They're right. Nobody touched anything. That is exactly why it broke.
When you buy normal software, the thing you paid for stays the thing you paid for until a human deploys a change on purpose. There's a version number. There's a changelog. If a new release breaks your workflow, you can point at the release, complain about the release, and in a pinch roll back to the one from last week. The behavior is frozen unless somebody chooses to unfreeze it. That predictability is most of what you're actually buying, even though nobody puts it on the invoice.
A language model is not software in that sense, and this is the thing the person selling it to you will not slow down to explain. It's a product the vendor is constantly, quietly editing. The same input can give you a slightly different output twice in a row. You cannot pin it the way you'd pin a normal piece of software and walk away. You are renting behavior, and the landlord remodels while you're at work.
Now add the move that actually burns small businesses: the silent swap. Every few months the vendor retires the model your automation was built on and "upgrades" you to the newer one. The newer one scores higher on their benchmarks, so from their seat this is a gift. But your automation was tuned against how the old one behaved on your weird, specific edge cases. The new model is better in general and different on the one thing you needed it to do reliably. The tool that sorted your incoming invoices into the right bucket now gets eight percent of them wrong. No error. No red light. Green the whole way down.
You can't roll back a model you were never allowed to pin.
Nobody warns you because nobody's incentive points that way. Moving every customer onto the latest model is cheaper for the vendor to run and better for their marketing. The deprecation notice, if it exists at all, lives in a developer changelog you have never once opened. The first human on earth to notice your automation drifted is not the vendor and not you. It's your customer, reading a quote that's suddenly wrong, or an email that went to the wrong account. By the time it reaches you it's already a reputation problem, not a software one.
Here's what building this for a living taught me, and it's unglamorous. I pin the model version anywhere the vendor lets me, and I hold on the old one as long as they'll allow it, even when they're waving a shinier one in my face. Where they won't let me pin, I build a canary: a small set of known inputs with known correct answers that runs on a schedule and screams the moment the answers start moving. Nobody asks for that. It doesn't demo well. It's a seatbelt, and like a seatbelt it looks like wasted money right up until the one morning it isn't.
You can't build the canary yourself, probably. But you can make it a condition of the sale. So here are the questions, and I'd ask them before a dollar changes hands. Can I pin the model version, and for how long? Will you tell me before you change the model running behind my automation? What's your deprecation window when you retire one? If the answer is a warm, confident "we always keep you on the latest and greatest," translate it out loud so everyone in the room hears it: my automation can change under me, without warning, and the first person to catch it will be a customer.
That's not a reason to skip automating. Plenty of it is worth it. It's a reason to understand what you actually bought. You did not buy a tool that sits still. You rented a moving target, and the rent includes somebody watching it move. If the vendor won't watch, and won't let you freeze it, and won't tell you when they change it, you're not buying automation. You're buying a thing that works perfectly right up until the quarter it quietly doesn't, and then hands you the bill for finding out.