Your customer support AI is probably going to fail in production!

We were looking to implement a customer support AI agent. Our customer experience (CX) team evaluated several products, with varying levels of confidence in their suitability. A support-AI demo can produce a convincing answer to a product question. The harder test is whether it gives the right answer when the customer has omitted a detail, the documentation is incomplete, or the request needs to be escalated to an engineer.

To avoid choosing a product based on its demo alone, we needed to formalise an AI agent evaluation process. We zeroed in on starting with a combination of 50 resolved tickets and a human-scored rubric. That is a manageable first evaluation, not a statistically established threshold for deployment.

I wanted to understand which kinds of tickets AI could help us handle, how much review its answers would require, and where it would need to hand the work back to a human. We experimented and had several back-and-forth discussions between the engineering and customer success teams. I wanted to summarise what we learned to save other teams time.

This is not a comprehensive guide. We are still learning how to operate this way, and success depends on several factors specific to each business. However, it can serve as a starting point for adapting the process to your own support workflow.

Based on that experience, here is how I would structure an initial pilot.

Start with one job for the AI

“Automate support” leaves too much undefined.

For this pilot, I would give the AI one job: draft the next customer-facing response using the ticket history and approved support information. It could propose a clarification or escalation when appropriate. A human would review every draft.

That scope matters. We would be evaluating response drafting, not autonomous ticket resolution. A useful reply might move an investigation forward without resolving the problem.

Take a customer who says:

My export has been processing for an hour. Can you restart it?

A useful response depends on what we know. Is the job still running? Has it failed? Can restarting it create duplicate work? Does the support team have permission to restart it?

A confident “I’ve restarted your export” would be unacceptable if the AI had neither the tools nor the authority to do so.

Before selecting tickets, write down what the AI can inspect, what it can recommend, and when it must escalate.

Build the sample without giving away the answer

Select 50 recent resolved tickets from the support queue, covering the work the proposed assistant would encounter.

Include common how-to questions, troubleshooting, missing information, and cases that required escalation. Sample across customers and product areas and remove duplicates that would give one recurring incident disproportionate influence.

If we deliberately include extra difficult cases, report them separately. Their failure rate should not represent the normal queue.

For each ticket, choose a point where the AI would have been asked to help. Build two separate records:

• Input: the customer message, preceding conversation, and permitted context available at that point.
• Reviewer reference: the eventual resolution, relevant evidence, and acceptable next steps.

Keep subsequent replies and resolution notes out of the AI’s input. Otherwise, we would be testing whether it can restate an answer already present in the context. Giving the AI information that became available later introduces lookahead bias and makes the evaluation unreliable.

Check the retrieval source too. Removing the resolution from the prompt achieves little if the AI can retrieve the complete closed ticket.

Use an approved environment and remove customer identifiers that are unnecessary for the task. Preserve details that materially affect the answer, such as the product version or relevant account configuration.

Define the rubric before reading the outputs

I would score each draft across five dimensions, using a simple 0–2 scale.

Dimension	0: unacceptable	1: needs correction	2: meets expectations
Factual correctness	Gives incorrect or invented information	Contains a material ambiguity or imprecision	Claims agree with the available evidence
Appropriate next step	Recommends an unsuitable action	Direction is useful but incomplete	Answers, clarifies, or escalates appropriately
Use of context	Ignores a relevant fact	Uses some context but misses an important detail	Accounts for the details needed at this stage
Policy and authority	Crosses a boundary or claims an unperformed action	Leaves a permission or policy condition unclear	Stays within the defined boundaries
Customer usability	Confusing or unusable	Requires avoidable rewriting	Clear, specific, and actionable

The rubric needs examples from our own support workflow. “Appropriate escalation” means little until we define which requests require it and what information the handoff must contain.

I would record critical failures separately from the total score. Exposing another customer’s information or falsely claiming that an action was completed should fail the draft regardless of how well it scores elsewhere.

Anthropic’s guidance on evaluating agents distinguishes the agent’s transcript from the actual outcome. That distinction applies here: a response claiming that an export was restarted is not evidence that the restart happened.

The historical support reply should also be treated as a reference, rather than the only acceptable wording. A different response might be equally valid, or better supported by the information available.

Calibrate the reviewers, then freeze the setup

Within the 50 tickets, I would use 30 for development and reserve 20 for evaluation after the setup was fixed. This is similar to how backtests are run in financial simulations.

Start by having two experienced support reviewers independently score the same five development cases. Compare their scores and discuss disagreements.

If one reviewer rewards a direct answer while another expects clarification, the issue may be an undefined support rule. Resolve that before judging the remaining outputs.

OpenAI’s evaluation guidance recommends clear, task-specific criteria and human calibration. For this pilot, calibration gives us a shared interpretation of what a good response looks like.

Use the development tickets to improve instructions, retrieval, and escalation guidance. Then freeze the model, prompt, knowledge sources, and settings before running the reserved tickets.

If we tune against those reserved results, they become development data. We need fresh tickets for the next independent check.

For every output, retain the draft, rubric scores, critical-failure flag, reviewer correction, and time spent reviewing and editing. Where practical, have both reviewers independently score the reserved set and reconcile disagreements.

Measure the work left for the human

A high average rubric score would be insufficient if every answer still required substantial checking.

Alongside the scores, classify each draft as:

• Usable without edits.
• Usable after minor edits.
• Requiring substantial correction.
• Unusable.

Record appropriate escalations separately from unnecessary ones. Otherwise, an assistant that escalates everything could appear dependable while contributing little.

For a time comparison, ask another reviewer to draft responses without AI from the same input context. Avoid having someone write the baseline immediately after seeing the AI’s answer. Compare human preparation time with the time a human spends reviewing and editing the AI’s draft, and record generation latency separately.

The comparison remains a small offline exercise. It cannot establish customer satisfaction, a reduction in reopened tickets, or end-to-end resolution time.

Report counts as well as percentages. “Three of six troubleshooting drafts needed substantial correction” is more useful than a single average across all 50 tickets.

Use the result to choose a narrower next step

Before running the next iteration, agree on the conditions for progressing: acceptable answer quality, no observed critical failures, manageable review effort, and clear escalation behaviour.

Passing those conditions would justify a limited, human-reviewed live trial. It would not establish readiness for automatic sending.

In our case, the results supported using AI to draft routine how-to answers while leaving troubleshooting with the support team. Another team might find that missing documentation is the main constraint. Either finding gives us a concrete next action.

I would finish the pilot by assigning each recurring failure to an owner and a correction: improve a source, change an instruction, clarify a rule, or restrict the supported scope. Preserve those cases as regression checks, then evaluate the revised system on fresh tickets before expanding its role.