We get asked to add AI to software roughly weekly. About half the requests describe a feature that will pay for itself within a quarter. The other half describe a demo — something that will impress a board once and then sit unused, costing tokens.
Telling them apart does not require technical knowledge. It requires asking four questions in the right order, and being honest about the answers. This is the test we run internally before we agree to build anything with a model in it.
One: does the work involve judgment at volume?
AI earns money in exactly one situation: a judgment a person can make in seconds arrives hundreds of times a week, and each one currently consumes human attention. Reading an invoice and typing its fields. Deciding which inbox a message belongs in. Matching a delivery note to an order. Drafting the routine reply.
If the task is rare, a person should do it. If the task involves no judgment — the data is already structured and the rule is fixed — ordinary code should do it, at a fraction of the cost and with none of the failure modes. AI sits in the middle band: frequent, and requiring the kind of reading or judging that rules cannot express.
The gimmick version fails this question immediately. "An assistant users can ask anything" has no volume of any specific judgment; it is a demo of generality, and generality is what makes it unevaluable.
Two: what happens when it is wrong?
Every model is sometimes wrong. The design question is where the wrongness lands.
A document parser that flags low-confidence extractions for a person to review is wrong safely — the error costs seconds. A system that emails customers unreviewed, posts to a ledger unreviewed, or quotes prices unreviewed is wrong expensively, and one incident can cost more than the feature ever saved.
The rule we apply: the model proposes, and something — a person, a validation layer, a reconciliation check — disposes. If a proposed feature has no place for that check to live, we redesign it until it does or we decline it.
Three: can you write down what correct means?
Before we build any AI capability, we write a sentence of the form: "correct means X, and we will measure it on Y examples before shipping." For invoice extraction: correct means the amount, date and supplier match what a person reads, measured on two hundred real invoices. For triage: correct means the same routing a senior operator would choose, measured against a month of history.
If nobody can write that sentence, the feature is a gimmick by definition — not because the technology is weak, but because nobody will ever be able to say whether it works. Unevaluable features do not get better with iteration; they just accumulate opinions.
This is also the fastest vendor test available. Ask any firm proposing AI work: what will you measure before this ships? A firm with an answer has done this before. A firm that talks about the model instead of the measure is selling you the demo.
Four: does the data exist?
The quiet killer. The feature that reads your contracts needs your contracts — as files, accessible, in usable shape, with permission to process them. The system that predicts from history needs the history to have been recorded. We have seen more AI projects stall on "the data turned out to live in seventeen inboxes" than on any modelling problem.
An afternoon of checking what actually exists, before committing, is the cheapest insurance in this whole field.
The pattern in the failures
Every AI feature we have been asked to rescue failed one of these four questions, and usually the second or third: it shipped without a check on wrongness, or without a definition of correct. None failed because the model was not smart enough. The models are, by now, rarely the constraint — the design around them is.
The four questions cost nothing to ask. If a proposed feature clears all four, build it; those are the systems that quietly return hours every week. If it clears none, you were about to pay production prices for a demonstration.
- ai
- evaluation
- scoping
- llm
- due-diligence