AI pilots stall for five recurring reasons: nobody defined correctness, the pilot ran on clean data, there was no plan for wrong answers, running cost was never modelled, and no one owned the system after launch. Each is cheap to address before starting and expensive to address afterwards.
We see the same pattern often enough to write it down.
1. Nobody defined what correct means
The pilot produced outputs. People looked at them and thought they seemed reasonable. Nobody built a set of examples with agreed correct answers, so there is no way to state accuracy — and therefore no way to decide whether the system is good enough to depend on.
This is the single most common cause, and it is entirely avoidable. The fix is to build an evaluation set before building the system.
2. The pilot ran on clean data
Pilots get run on a tidy sample. Production gets the malformed documents, the ones scanned at an angle, the ones where a field is missing entirely, and the ones somebody filled in wrong three years ago.
Accuracy measured on clean data does not transfer. If the evaluation set does not include the ugly cases, the number it produces is describing a situation that will not occur.
3. There was no plan for wrong answers
Every AI system is wrong sometimes. A pilot can ignore that. Production cannot.
Somebody has to decide what happens when confidence is low: does the system proceed, stop, or route to a person? Who reviews it? What does the review interface look like? These questions are not model questions, which is why they get deferred — and deferring them is what blocks the move to production.
4. Running cost was never modelled
Inference is a per-use cost. At pilot volume it is negligible. At production volume, on a customer-facing feature, it can consume the margin on the product.
Cost per request needs instrumenting from the start, along with the levers that control it: caching, routing simpler requests to smaller models, and limits so a single heavy user cannot make an account unprofitable.
5. Nobody owned it after launch
Model providers change behavior. A prompt that worked in March behaves differently in September. Data drifts. Volume grows.
Without someone accountable for accuracy in production, a system that launched at 95% quietly becomes a system at 80%, and nobody notices until a customer does.
What to change
Before starting an AI project, get written answers to five questions:
- What does a correct answer look like, with twenty examples?
- What do the worst real inputs look like?
- What happens when the system is not confident?
- What is the cost per request at expected volume?
- Who is accountable for accuracy six months after launch?
If those cannot be answered, the project is not ready to start — and starting anyway is how a pilot becomes a sunk cost rather than a system.
- evaluation
- production
- llm
- strategy