Most AI pilots end in a report. Very few end in a decision.
The report says the technology “shows promise”. It lists things that worked and things that need more work. It recommends a further phase. Six weeks and a budget line later, the organisation knows roughly what it knew before: that the technology can do something, under conditions nobody has written down.
This is rarely a failure of execution. It is almost always a failure of definition. A pilot that was never designed to produce a decision will not produce one, however well it is run.
“A pilot that was never designed to produce a decision will not produce one, however well it is run.”
A decision-grade pilot is a different instrument. It is built backwards from a question that has to be answered, and it is designed so that all three possible answers — continue, change, stop — are acceptable outcomes before the work begins.
Why most pilots cannot produce a decision
Four design faults account for the majority of inconclusive pilots.
No baseline was captured. The current workflow was never measured. When the pilot produces a number — eight minutes per listing, ninety-one per cent first-pass approval — there is nothing to compare it against. Teams then reconstruct the baseline from memory, and memory is generous about how the old process performed. Once the comparison is contested, the decision stalls.
Success criteria were written after the results were seen. If the threshold is set at the end, it will be set wherever the results landed. This is not dishonesty; it is the ordinary gravitational pull of sunk cost and organisational momentum. The only defence is writing the thresholds down, with a date, before the system is built.
“If the threshold is set at the end, it will be set wherever the results landed.”
The pilot scope was not the thing that would scale. Pilots gravitate towards the easiest slice of work: the cleanest data, the most cooperative team, the product category with the fewest edge cases. That slice produces good numbers and no information. The question a pilot exists to answer is whether the system holds under the conditions of real operations, which means the scope must include the awkward cases — not all of them, but a representative share.
Nobody was authorised to stop. If no individual has the standing to end the programme, “stop” is not really on the table, and a decision framework with one live option is not a decision framework. This is the most common fault and the least discussed.
Six things to fix before a pilot starts
1. Name the decision owner
One person, named in writing, who will make the continue, change or stop call and who has the authority to make “stop” stick. Not a committee. A committee can approve a decision; it cannot own one.
The decision owner should be the person who carries the operational consequence of the workflow — the head of the function, not the head of technology. If the workflow is multilingual listing production, the decision owner runs commerce operations.
2. Measure the baseline before you build anything
Two to four weeks of the current process, measured as it actually runs, including the bad weeks. Capture it before anyone knows a pilot is coming, if that is possible; measurement changes behaviour.
A usable baseline covers five dimensions:
Throughput
Listings published per person per week
The headline number, and the easiest to game
Quality
First-pass approval rate; defects found downstream
Throughput without this is meaningless
Cost
Fully loaded cost per unit of output, including review time
Model cost is usually the smaller half
Exceptions
% of items that leave the standard path
Where automation quietly stops working
Adoption
% of eligible work actually going through the process
A system nobody uses has no measurable benefit
The exception rate is the one most often omitted and the one that most often decides whether a system survives contact with daily operations. A workflow that handles eighty per cent of cases beautifully and dumps twenty per cent onto a senior person’s desk has not reduced workload; it has concentrated it.
“A workflow that handles eighty per cent of cases beautifully and dumps twenty per cent onto a senior person’s desk has not reduced workload; it has concentrated it.”
3. Write the thresholds down in advance
For each of the three outcomes, state what would have to be true. Numbers, not adjectives.
Continue if: cost per published listing falls at least 30% against baseline, first-pass approval holds at or above the baseline rate, and the exception rate stays below 15%.
Change if: cost improves but first-pass approval falls, or the exception rate exceeds 15% while quality holds. The system works; the design of the human control points does not.
Stop if: cost improvement is under 10%, or exceptions exceed 25%, or the team declines to use the system when it is not mandated.
The specific numbers matter less than the fact that they exist before the results do. Sign them. Date them.
4. Choose a scope that is representative, not convenient
A defensible pilot scope has three properties:
- Narrow enough to finish. One workflow, one team, one product category or market. Not “commerce operations”.
- Complete end to end. A pilot covering only the middle of a workflow measures nothing useful, because the handoffs at either end are usually where the time goes.
- Representative of the difficulty. Include the category with the messy source data or the market with the hardest compliance requirements. If the system cannot handle any of the difficult cases, that is a finding, and it is better to have it in week three than in month nine.
5. Instrument cost and exceptions from day one
Quality gets measured because it is visible. Cost and exceptions have to be instrumented deliberately, or they are reconstructed at the end from guesswork.
Cost means fully loaded: model and infrastructure spend, plus the human time spent reviewing, correcting and handling the cases the system passes back. In most content and commerce workflows the human review time is the larger figure, which is why “the model got cheaper” is a weak argument for scaling. This is the same cost discipline we apply inside the AI Commerce System, where review time is tracked alongside model spend.
6. Fix an end date
A pilot without an end date becomes an unfunded permanent programme. Six to twelve weeks is enough for most single-workflow pilots — long enough for the novelty effect to wear off, short enough that the decision still matters.
Treat all three outcomes as legitimate
Continue is the outcome everyone plans for and the least interesting one. If the thresholds were met, the useful work is defining what the next scope looks like and what would have to hold true for it to expand again.
Change is the most common outcome of a well-defined pilot, and the most valuable. It usually means the automation worked and the control design did not: approval points in the wrong place, exceptions routed to the wrong person, a review step that made sense on paper and became the bottleneck in practice. These are workflow problems, not model problems, and they are cheap to fix at pilot scale and expensive to fix after rollout.
Stop is the outcome that justifies the whole exercise. A pilot that costs a defined sum and prevents a larger programme that would not have worked has paid for itself. This only functions if stopping was genuinely available from the start — which is why the decision owner and the written thresholds are not procedural niceties. They are what makes the other two outcomes mean anything.
The demo trap
One pattern is worth naming separately, because it defeats otherwise well-designed pilots.
A demonstration proves that a system can produce a good output. A pilot has to establish that it produces acceptable outputs repeatedly, on ordinary inputs, operated by the people who will actually operate it, at a cost that holds when nobody is watching.
The gap between those two things is where most AI programmes fail. It is not a gap in model capability. It is the accumulated weight of edge cases, data quality, handoffs, unclear ownership and the review burden that arrives when volume goes up.
This is the reason we run Operating Labs — internal environments where workflow design, control points and cost behaviour are tested before a method is put in front of a client. A pilot should be the second time a method meets reality, not the first.
“A pilot should be the second time a method meets reality, not the first.”
A pilot definition checklist
Before work starts, these should exist in writing:
- Named decision owner, with authority to stop
- Baseline measured across throughput, quality, cost, exceptions and adoption
- Written continue / change / stop thresholds, dated and signed
- Scope: one workflow, end to end, including representative difficult cases
- Named human control points, and who staffs them
- Cost instrumentation covering model spend and human review time
- Fixed end date and a scheduled decision meeting
- Agreement on what data is captured and who can see it
If more than two of these are missing, the exercise being planned is a demonstration. That may be the right thing to do — demonstrations have their uses — but it should be named accurately, budgeted accordingly, and not expected to produce a decision.
Start with the workflow that matters
A pilot is a decision-making instrument. Designed properly, it is the cheapest way to find out whether an AI business system is justified for a specific workflow, and the cheapest way to find out that it is not.
If you are considering a pilot and are not yet confident that it would produce a decision either way, that is the conversation worth having first.