The scarce skill in AI engineering is not prompting or model selection. It is defining what a correct answer looks like for a specific task, building an evaluation set that reflects it, and keeping accuracy from drifting once the system is live. Hire for that and the rest follows; hire for framework familiarity and you get demos.
Why demos are easy and production is not
Getting a plausible answer out of a modern model takes an afternoon. That is genuinely useful, and it is also why AI hiring goes wrong — the demo is no longer evidence of anything.
The work that separates a system you can depend on from one you cannot is unglamorous: assembling real examples with agreed correct answers, scoring every change against them, deciding where a human stays in the loop, instrumenting cost per request, and noticing when a provider changes model behaviour underneath you.
What to test for
Can they define correctness? Give a real task from your business and ask how they would know the system was right often enough. A strong answer starts with examples and a measurement. A weak one starts with a model name.
Do they reach for AI last? Ask about a problem better solved with a rule or a database query. Engineers who have shipped will say so. The instinct to reject AI where it does not belong is a positive signal, not a lack of enthusiasm.
Do they think about being wrong? Ask what happens at low confidence. Look for thresholds, escalation to a person, and an opinion about who reviews and how.
Do they know what it costs? Ask how they would keep inference cost under control at ten times the volume. Caching, routing simpler requests to smaller models, per-account limits. Someone who has never thought about it has never run one.
Have they maintained one? Ask what broke after launch. Provider behaviour changes, data drifts, an upstream format changes. The answer tells you whether they have operated a system or only delivered one.
Interview format
Skip puzzles. They test something unrelated.
Use a real, redacted problem from your business. Ask them to talk through how they would establish whether a solution works, what they would build first, and what they would refuse to automate. Forty-five minutes of that tells you more than any take-home.
If you use a take-home, pay for it and keep it short. The best candidates have options and will decline unpaid multi-day exercises, so those filter for availability rather than quality.
Signals that mislead
Model or framework names on a CV. These change every few months. The evaluation discipline does not.
Kaggle placings. Genuine skill, different job. Competition work optimises a metric on a fixed dataset; production work is mostly about defining the metric and handling everything the dataset excluded.
Research background, unqualified. Valuable for some roles, unrelated to others. Ask what they have put in front of users.
If you are hiring one rather than a team
A single AI engineer with no backend or infrastructure support will spend most of their time on neither. Production AI is mostly a service, a data path, monitoring and a review interface, with a model call inside it. Hire someone who can build the service, or place them alongside someone who can.
That is the most common structural mistake we see: an excellent AI hire, isolated, unable to ship because everything around the model is missing.
- hiring
- evaluation
- llm
- team