Every AI agency demos well. The models are good enough now that a polished prototype proves almost nothing — which means your evaluation has to probe the parts a demo can't fake: deployment discipline, guardrails, economics, and what happens in month three. These are the ten questions we'd ask before hiring anyone in this market, including us. The pattern in the answers matters more than any single response.
1. "What happens in the first two weeks?"
Good answer: discovery against your real workflows, data access scoped, a supervised pilot with humans approving outputs. Bad answer: a contract and a kickoff deck. Agencies that can't describe week one concretely haven't done this often.
2. "When does a human see what the AI does?"
Every serious deployment starts supervised — approval steps, transcripts reviewed daily, escalation rules written down. If the vendor's plan is "it's autonomous from day one," they're handing your customers to an unproven system and you the reputational bill.
3. "What will it refuse to do?"
The revealing inversion. Mature builders talk guardrails unprompted: topics the agent won't touch, actions gated behind approvals, what happens when it doesn't know. A vendor with no refusal story has never watched an agent fail in production.
4. "Who owns the workflows, prompts, and accounts?"
You should own the automations, the integrations, and every account they run on — full stop. Agencies that keep the n8n instance, the phone numbers, or the prompt library in their name are building a hostage situation, not a system.
5. "What does month three look like — and cost?"
Agents need tuning, monitoring, and refresh as your business changes. Ask for the ongoing scope and price now. "It just runs" is the second-biggest lie in this industry, right after the demo.
6. "Show me a deployment that struggled."
Everyone has case studies. Ask for the one that went sideways and what they changed. Builders with real volume answer instantly and specifically; resellers change the subject. (Ours: an early voice agent that transferred too eagerly — the fix reshaped how we tune escalation to this day.)
7. "Which tools, and why those?"
You're listening for reasoning, not brand names. Why n8n over Zapier for this workflow, why GPT-5 here and Claude there, what gets swapped when something better ships. One-platform-for- everything answers usually mean a white-labeled product with your logo on it.
8. "How will we measure this in dollars?"
Before building, there should be a number: hours returned, calls resolved, days of DSO, reorders recovered — with a baseline measured first. Vendors who sell "efficiency" without a baseline are selling a feeling. Our own rule: the ROI math comes before the statement of work, or the project doesn't start.
9. "What did you talk the last client out of?"
Agencies that only ever say yes are order-takers. The right partner has recently told someone their data wasn't ready, their volume didn't justify the build, or that paid ads would pay back faster than SEO. If they've never turned down revenue, you're not buying judgment.
10. "Can we start smaller than your proposal?"
The honest answer is almost always yes. One workflow, one agent, one measurable result in four to six weeks — then expand on evidence. Vendors who need the big annual commitment up front are pricing their churn into your contract.
The pattern to hire for
Read the ten answers together and you're measuring one thing: whether the agency treats AI as a system that must survive contact with your real customers, or as a demo that must survive contact with your signature. Systems people talk about guardrails, baselines, ownership, and month three. Demo people talk about what the model can do. The first kind is who you want — and if you'd like to hear our answers to all ten, that's a thirty-minute call.