Is your AI pilot ready for production? (5 gaps to check)
If your AI pilot works, you’ve already done the hard part everyone talks about. You picked a real problem, got something built, and watched it do the job. That’s more than a lot of companies manage.
There’s a second hard part, and it gets far less attention. It’s the distance between “it works when we show it” and “it holds up when real customers, real data and real mistakes arrive.” That distance is where most AI pilots stop.
This piece is about what’s actually in that gap, and how to check your own pilot against it.
The short version
A pilot proves an idea works once. Production needs 5 things most pilots skip:
- Observability: You can see what it did.
- Evaluation: You test it against known cases and on live traffic.
- Optimization: You use what those tests find to make it better.
- Reliability: it hands off safely when it’s out of its depth instead of guessing.
- Governance: it’s security-tested and documented against a real standard.
Check all five before you fund the rollout. Five questions to test your own pilot are at the end.
Starting isn’t the hard part. Holding up is.
Almost everyone has started. Far fewer have something live that holds up.
11% of organisations are actively using agentic AI in production. Another 38% are still piloting. That’s from Deloitte 1, across all the organisations it surveyed; a further 30% are exploring, and 14% have something ready to deploy.
Other research lands in the same place:
- S&P Global Market Intelligence 2 (May 2025, a survey of 1,006 IT and business professionals in North America and Europe, focused on generative AI): 42% of organisations abandoned most of their AI initiatives before production, up from 17% a year earlier. On average, organisations report 46% of projects scrapped between proof of concept and broad adoption.
- MIT NANDA 3 (July 2025): 95% of enterprise generative AI pilots show no measurable P&L impact. Treat that as directional: the authors call it preliminary, it rests largely on interviews with 52 organisations, it isn’t peer-reviewed, and it defines success narrowly.
- LangChain 4 (1,340 respondents, late 2025): 57% have agents in production, and quality is the top barrier (32%). It surveys teams already building agents, mostly smaller tech companies, not all organisations as Deloitte does, so its production number runs higher than Deloitte’s 11%.
Read together, the pattern is clear. The gap isn’t starting. It’s holding up. Futurice said it well in a job description: “Far fewer have got anything into production that holds up under real load, real data and real scrutiny.”
Gap 1: Observability. Can you see what it did?
Observability means you can reconstruct exactly what the AI did and why: every model call, every tool it used, every hand-off between steps. Not just the answer, but how it got there.
It comes first for a simple reason, and IBM’s AgentOps framework 5 puts it well: you can’t improve what you can’t measure, and you can’t measure what you can’t see.
When it’s missing, it looks like this. Someone asks “why did it decide that?” and the honest answer is a guess. Or the monthly bill comes in higher than expected and nobody can say which step caused it. Deloitte 1 warns that badly configured agents can set off chains of actions and costs that balloon, which is why it argues agents need their own cost tracking.
Three numbers are worth asking for on any AI system:
- How long each case takes, end to end
- What each case costs
- Where hand-offs between steps fail
This is the gap most teams have started to close. In LangChain’s survey 4 of teams building agents, 89% have some observability. Among teams with agents in production it’s 94%, and 71.5% of those have full tracing. So treat it as table stakes. The real question is whether you have it, or only think you do.
Check it yourself
Pick one real case from last week. Ask your team to walk you through what the AI read, decided and did, using the logs alone, not anyone’s memory. If they can’t, you don’t have observability yet.
In my test system
My coverage-approval test system runs as a synthetic benchmark of 20 cases per round, and every case is traced in LangSmith, a tool that records every step. When a case failed, the trace showed the exact step that broke: routing, evidence handling, or the hand-off to a person.
Gap 2: Evaluation. Do you know if it’s any good?
Observability tells you what happened. Evaluation tells you whether it was right. This is where the biggest drop-off sits.
In the same LangChain survey 4, 89% of teams can trace their agents, but only 52.4% test them offline against a set of known cases, and 37.3% score live traffic. Of the teams that evaluate at all, only about a quarter do both. And quality is the top barrier to production, named by 32%. Those numbers are connected: you can’t fix quality you aren’t measuring.
There are two kinds of evaluation, and you need both.
Offline: known cases, before every change
A set of real examples with known right answers, including the awkward edge cases. Every change to a prompt, model or tool runs against it before it goes live. Without this, a small tweak can quietly break something that used to work, and nobody notices until a customer does.
Online: live traffic, after release
Real users bring cases your test set never imagined. Online evaluation scores live work where there’s no answer key, for example by flagging conversations where a user says “that’s wrong” or pastes an error back.
Offline catches what you broke. Online catches what you never thought to test.
Two more habits separate real evaluation from a vibe check:
- Test on cases the system has never seen. A system scored on the same cases it was tuned on will always look better than it is.
- Let your experts set the standard. The people who do the work today decide what “right” means. Automated checks then apply their judgement at scale.
If you buy AI from a vendor, this is also your best contract lever. Covista put it bluntly in a job description: test cases written into every AI build contract, “no eval, no production.” It’s a clause worth borrowing.
Check it yourself
Ask four questions. How many test cases do we have? Where did they come from? When did they last run? Which way is the pass rate going? A vague answer to any of them means this gap is open.
In my test system
The first round scored 70% correct handling on 20 cases: the right outcome for each case, not an automation rate. The second round used 20 fresh cases the system had never seen, and scored 85%. Fresh cases make it a harder test, not a like-for-like one, and 20 cases is a small sample.
Gap 3: Optimization. Are you using what you learned?
Evaluation without optimization is a report card nobody acts on. Optimization means using what evaluation found to make the system better, then proving it got better. This is where the real wins come from.
The pilot is version one. Nobody gets an AI system right the first time, because nobody can predict every case real work will throw at it. The value builds with each tested fix: a routing rule tightened, a retrieval step tuned, a prompt made clearer. Each one is small. Together they’re the difference between a demo and something you’d trust with customers.
Three rules make it work:
- Aim at business outcomes, not just technical accuracy. “Did it choose the right action?” and “did it escalate when it should?” matter more than a model score. So does cost per case.
- Fix the system, not the test. When a check fails, the tempting fix is to loosen the check. The right fix is to change the system, then prove it again on fresh cases.
- Make trade-offs in the open. Better quality can cost more time or money. That’s fine, as long as someone chose it knowingly.
Two numbers tell you whether it’s happening. Improvement velocity: how many tested improvements ship each week. And cost per case, tracked over time, so “better” never quietly turns into “more expensive than it’s worth”.
Finance belongs in this loop too. Dell’s CTO, John Roese, told Deloitte 1 that nothing goes to production at Dell until a finance partner and the head of the business unit have signed off a solid ROI. Everything else stays an experiment.
Check it yourself
Ask: what’s the last change we made because of a test result, and which number moved? If nobody can name one, evaluation is producing reports, not improvements.
In my test system
Between rounds, I fixed three routing and evidence-handling bugs the first evaluation found. Cost per case, including the automated grading of each test, rose from $0.027 to $0.029; the system’s own model cost is under a cent. It excludes the time people spend reviewing.
Gap 4: Reliability. What happens when it doesn’t know?
Every AI system will meet cases it can’t handle. Reliability means every place it can fail has a defined, safe fallback: escalate to a named person, hold the case, or ask for more information. What it never does is guess.
Quality is the top barrier to production in LangChain’s survey 4, and no system reaches perfect quality. A safe fallback is what makes imperfect quality safe to run.
Deloitte 1 describes the same idea from the people side: “agent supervisors”, people who step in at hand-off points that were designed on purpose, not whenever someone happens to notice a problem. The hand-off is part of the design, not a failure of it.
VESTIGAS summed up the goal in a job description: a system that knows “when to flag a document for human review instead of silently guessing.”
Three things make a fallback real:
- Written down. For every step: what happens if it fails, and who picks it up.
- Set from data. The point where a step can run without human review comes from test results, not hope.
- Capped. An agent stuck in a retry loop keeps spending money. There should be a hard limit or an alert on spend, and someone should have tested that it works.
Check it yourself
Ask to see three cases where the AI escalated, and one where it should have but didn’t. If nobody can find them, you don’t yet know how it fails.
In my test system
Across 40 synthetic cases in two rounds, the system took zero unsafe actions, and every case ended with a person’s decision, by design: nothing leaves the sandbox without human sign-off, so none completed straight through. Correct handling includes handing a case to a person when it should.
Gap 5: Governance. Can you prove it’s safe?
Governance decides whether you can defend the system if something goes wrong. It has three parts.
Security: can it be attacked or misused?
The reference point is OWASP, the open security community. Its Top 10 for LLM applications 6 applies to any AI system: prompt injection is the best-known entry, where someone hides instructions inside a document or email the AI reads, and the AI follows them. Once a system can use tools, remember things or work with other agents, a second list applies too: OWASP’s Top 10 for Agentic Applications (December 2025) 7. Three more worth knowing by name: tool misuse, an agent given a broad database tool that ends up running a command nobody intended; identity abuse, an agent that inherits a master API key instead of its own scoped credential, and can then reach data no single user should see; and memory poisoning, a corrupted document in a knowledge base that quietly changes what the agent believes is true for every future case. A simple chatbot needs the first list. An agent needs both, and knowing these by name is what tells you whether your team is testing for the real risks or just the famous one.
In practice, “done” means a documented prompt-injection test suite with a tracked pass rate, and a written list of every tool the agent can use, why, and with the least access it needs.
In IBM’s Cost of a Data Breach 2025 8, a Ponemon study of 600 breached organisations, 13% reported breaches of AI models or applications, and 97% of those lacked proper AI access controls. 63% of the breached organisations studied had no AI governance policy or were still developing one.
Compliance: is it documented against a real standard?
For an EU enterprise, three layers apply, and they do different jobs. AIUC-1 9 tests how the agent actually behaves, across six areas: data and privacy, security, safety, reliability, accountability, and society. ISO 42001 10 audits the organisation’s AI management policies; one checks the system, the other checks the company running it. Underneath both sit the laws themselves: GDPR governs any personal data the system touches, regardless of which AI-specific standard you follow; the EU AI Act adds its own high-risk obligations on a delayed timeline (below); sector rules layer on top where they apply, healthcare and finance in particular. AIUC-1 and ISO 42001 are built to map onto these directly, which is the point of using them instead of inventing your own checklist.
Control: could you investigate any incident?
Every agent action is logged in enough detail to investigate an incident end to end from the logs alone. Deloitte 1 makes the same point: organisations will need tamper-proof logs showing what an agent did, why, and on whose authority. Add control over which tools it can use, and limits on what it can spend.
Security and compliance each deserve a full post of their own; what’s above is the checklist version.
Why the EU date matters
The revised Product Liability Directive 11 applies to products placed on the market or put into service after 9 December 2026, and software, including AI, counts as a product. Liability doesn’t depend on fault. Only individuals can claim, and only for three types of damage: death or personal injury, including medically recognised damage to psychological health; damage to property, except the defective product itself and property used exclusively for professional purposes; and destruction or corruption of data not used for professional purposes. Courts can presume a product is defective, for example when the defendant fails to disclose relevant evidence. The way to rebut the presumptions is evidence, kept at the time, that the system was specified, tested, monitored and controlled. The AI Act’s high-risk rules come later: the AI omnibus 12 moved them to December 2027 for standalone systems and August 2028 for AI inside regulated products. This is general orientation, not legal advice.
Check it yourself
Ask: if a customer’s lawyer asked tomorrow how this system was tested, what would we hand them? If the answer is a policy document rather than test results and logs, this gap is open.
In my test system
The benchmark recorded zero guardrail violations (attempts to break a safety rule) in both rounds, and every case has its own trace, so any single decision can be reviewed step by step.
Five questions to ask your team on Monday
One question per gap. A vague answer tells you where to start.
- Observability: Can you walk me through one case from last week using only the logs?
- Evaluation: How many test cases do we have, where did they come from, when did they last run, and which way is the pass rate going?
- Optimization: What’s the last change we made because of a test result, and which number moved?
- Reliability: Can you show me three cases where it escalated, and one where it should have but didn’t?
- Governance: If a customer’s lawyer asked tomorrow how this was tested, what would we hand them?
Where to start
A pilot that works has proved the idea; production also needs you to see what it did, know it’s right, keep improving it, hand off safely when it’s unsure, and prove all of that. Check it yourself with these five questions, or take the 2-minute AI check to see where you stand.
Whether you have a pilot that isn’t ready or nothing built yet, the AI Delivery Sprint closes these five gaps either way – it starts from wherever you are, and you can see the method in my working example. Either check it yourself first, or ask me and I’ll do it with you.
Sources
- Deloitte Insights, Tech Trends 2026: The agentic reality check (Dec 2025)
- S&P Global Market Intelligence, VotE: AI & Machine Learning, Use Cases 2025
- MIT NANDA, The GenAI Divide: State of AI in Business 2025
- LangChain, State of Agent Engineering
- IBM Technology, Are Your AI Agents Flying Blind? The Truth About AgentOps
- OWASP Top 10 for LLM Applications 2025
- OWASP Top 10 for Agentic Applications (Dec 2025)
- IBM, Cost of a Data Breach Report 2025
- AIUC-1
- ISO/IEC 42001:2023
- Directive (EU) 2024/2853
- Regulation (EU) 2026/1744
Job-description quotes: SmartProduct study of 1,069 AI leader job descriptions, Sep 2026
