What we measure before we let an AI assistant talk to customers
Most AI pilots stall because nobody agreed what "good" means. Here is the evaluation set we build before writing a single prompt.
Every AI support deployment we have shipped started the same way: not with a model, not with a prompt, but with a spreadsheet of real historical tickets and the answer a good human agent gave to each one.
That spreadsheet is the evaluation set. It is the difference between "the demo looked impressive" and "we can put this in front of customers on Monday".
Why a demo is not evidence
A demo is a sample of one, chosen by the person doing the demo. It tells you the system can succeed. It tells you nothing about how often it fails, what it does when it fails, or whether the failure is embarrassing or expensive.
The questions a board actually asks are statistical: what proportion of contacts can this handle end to end, what happens to the rest, and how confident are we in those numbers. You cannot answer any of them from a demo.
Building the set
We pull 200–400 real tickets spanning the full distribution — the boring password resets, the ambiguous billing queries, the angry escalations, the ones where the customer was simply wrong. Then we grade each one on three axes:
- Correctness. Is the answer factually right against the source of truth?
- Completeness. Does it resolve the contact, or does the customer have to come back?
- Tone. Would you be comfortable seeing this in a screenshot on social media?
Grading is done by the client's own support leads, not by us. They own the standard, which means they own the launch decision.
The threshold conversation
Once the set exists, the conversation changes shape. Instead of "is the AI good?" it becomes "at what confidence score do we let it answer without a human?"
That is a business decision with a dial attached. Set the threshold high and you deflect fewer contacts but almost never embarrass yourself. Set it low and you deflect more but accept some rough edges. We show the curve; the client picks the point.
After launch
The same set becomes a regression test. Every prompt change, model upgrade or retrieval tweak runs against it before release. If a change improves average quality but breaks four previously-passing cases, we see that before customers do.
This is unglamorous work. It is also the reason our assistants are still running a year later instead of having been quietly switched off.
