Since the advent of widely available AI tools, they’ve come a long way. You don’t need to be a developer to build many AI workflows — the “no code” platforms will let you wire one together yourself, AI features are being embedded inside all the software you use, and plenty of firms (including mine) will build you something for a fixed fee. That is incredible progress in just a few short years!
However, just because it’s easy to deploy a solution doesn’t mean that it’s easy to do it right. There are still many decisions that need to be made to ensure that the system starts and continues to work the way it was intended. Here are four big decisions that I’ve observed from my own work.
The provider is a business decision, not a technical one
Most people understand that different models have different capabilities and different prices. However, you may not understand just how vast that cost difference is. Recently, I was integrating vision-capable AI models into Docora (my knowledge expert for document repositories) so that it could answer questions about images in a business’s internal knowledgebase (advertisements, drawings, and so on).
The primary provider I use had 341 different vision-capable models available, with the most expensive model being 879 times more expensive than the cheapest. I had to spend a significant amount of time benchmarking the different models to ultimately select the right one based on balancing cost against performance. The wrong choice would have made this functionality cost prohibitive.
Another decision to be considered when selecting a provider is reliability. AI is a hot field, and many companies are jumping into the marketplace. But they are not all created equal, and even the biggest vendors are struggling to maintain reliability. Just last week one of the major providers we use intermittently returned “capacity exceeded” errors over the course of a few days. That leads to the second thing: if a process depends on one provider and that provider has a bad afternoon, your process has a bad afternoon.
A fallback path must be designed in
A fallback means the system knows about a second option and moves to it when the first stops answering. It costs a little more to build, but every business-critical workflow needs this. It isn’t something you can handle manually — your AI-enabled workflows must expect providers to fail occasionally and be built to survive those failures.
Models get retired
In July I evaluated three models for one specific workflow in Docora. In August, the primary provider sent an email saying the model we had chosen was being decommissioned. They gave more than a month of notice, named two replacements, and sent reminders as the date approached. They handled it properly, but if we weren’t watching, it could have been an issue.
Those kinds of notices are sent by email to whichever address signed up for the account. It can land in spam, or go to someone who has left, or go to someone whose primary responsibility is billing and archives anything technical.
If that email is ignored and your system wasn’t designed to failover to a backup provider and send out alerts, this could be catastrophic. One day requests just stop being served, and if that model was serving a background task, it might take quite a while for people to notice.
We were fine, because we have automatic failover, alerts, and someone who monitors our providers. But it was still work that we had to actively do — we had to evaluate and test new models, then switch before the old model was decommissioned.
The question isn’t just whether your provider is reliable, it’s a question of whether your system is resilient and your human review processes are reliable.
Choosing a model is not a spec sheet exercise
The best example I have of this is the vision work that I did in Docora. I needed a model that could look at a scanned drawing or a photograph and describe it well enough to be found later. To do this, I used my experience to select a group of candidates, then measured the candidates at pilot scale and projected to estimate production volume (roughly a hundred times larger).
When evaluating models, the thing I always look at is functionality — if the model can’t do the job, no sense evaluating it further. And my testing revealed that some of the vision-capable models could not provide useable results. Others could, but not consistently enough to trust without a person checking every result, which defeats the purpose.
The next decision point is all about scalability. I don’t hear a lot of people talk about this, but it’s a real consideration for enterprise-grade solutions. Some models are capped by their providers at a fixed number of output tokens per minute. Knowing this early on is incredibly important, because a model with a fixed cap will work perfectly well until usage scales in production and it suddenly becomes an issue.
Now the candidate pool has been narrowed, with mostly low-end models being eliminated. The next decision point is looking at cost. This is more involved than simply looking at the price/million tokens. That’s a good starting point, but some models are much more verbose than others, and as a result consume far more tokens. Some of the seemingly inexpensive models consumed considerably more tokens than more expensive models — on average, about thirty nine percent more tokens per call. This means that price per token is not equivalent to price per task, and there’s no way to tell that without testing.
Finally, we move on to evaluating performance, that is, how fast the model provides a response to our queries or tool calls. This decision point is optional, because not all model usage requires speed. Docora’s vision processing tasks run in the background, so a slow model is perfectly acceptable. However, if this model was being used to serve up information in real time, this is a key consideration.
The two ways to get this wrong
If you take the easy route and pick a frontier model (or any model more capable than the task needs), at low volumes it doesn’t make much of a difference. However, as your workloads scale, work that would cost a few hundred dollars on a well-matched model runs into the thousands on the most expensive one, and further up the curve that gap reaches tens of thousands a year.
Of course, if you simply pick the cheapest model that appears to work, you’ve introduced a more subtle set of problems. You don’t get outrageously high bills, instead, you get results that are slightly wrong or inconsistent on the days the system is busiest. Nothing alerts, and the system ends up providing answers that are merely mediocre. You end up with a system people stop trusting and no clear reason why.
Who should be raising this with you
None of this depends on how you build. A no code platform still asks you to pick a model from a dropdown, your software vendor may allow you to pick from a group of models, or a development firm embeds them in your new system. The decision is being made — the only variable is whether anyone thought about them.
I’ll be very blunt: Whoever is proposing to put AI into your business should raise all of this before you sign: Which provider, and why…what happens when it goes down…how model retirement is handled, and what it costs to run the system at scale. If none of that has come up, either it was considered and nobody thought you needed to hear it, or it was not considered at all.
When this does not matter
If your volumes are genuinely small, most of this is theatre. The difference between a careful choice and a careless one might be a few dollars a year, and your attention is worth more. And if the process you are planning to AI-enable (or its data) is not settled, don’t start here: Get the workflow right, confirm the data is solid and reachable, and decide who owns the output. Model selection matters once there is something worth running.