How to test AI agents before production

A coding agent that gets it wrong fails a test. A support agent that gets it wrong refunds the wrong customer, closes the wrong ticket or deactivates the wrong user, and reports success. So the test has to run where mistakes are free, on the records the agent will actually meet, and it has to be graded on what changed, not on what the agent said. This guide is that test, in the order a team runs it.

Test on a copy of your own account, not a sample

The cases that catch agents live in your data: two customers with the same name, the order shipped in two parts, the subscription cancelled at period end and then reactivated. A sample account has none of them, and a shared sandbox carries every previous run's leftovers. Start from a copy of your own account, pseudonymized: names, emails and free text replaced, amounts and states kept. It lives on your machine, is safe to commit, and starts fresh for every run.

Bring your own tickets

Test on the tickets the agent will get, not the ones a benchmark author imagined. A help desk export becomes the tasks file, one task per row, each requester resolved to the copied account. Where you know the right outcome, one line says so: refund a named amount, close without action, escalate. A policy in plain words, what the agent may do and up to what amount, grades the rest.

AI agent refund testing, in dollars

Refunds are the case to start with because the error is a number. A run starts from a mark, the world's state at that moment, and ends with a diff: every refund, credit and cancellation since, in dollars, with the customer whose money moved. The right amount to the wrong customer is a diff that says so. A double refund is two rows. An over refund is refused in the payment system's own words, so the attempt shows too. The report's number, dollars wrong per run, is the one to put a threshold on.

Test a refund agent safely: the doppelgänger

The ticket that fails the most agents is the one with a twin. Two customers share a name, one was charged twice, the ticket asks for one refund. The reply is perfect, the ticket is closed, and the money left the wrong account. A transcript grader passes it. The world diff fails it, because the diff names the customer. Put that ticket in the tasks file first, then its cousins: the invoice number that matches another customer's amount, the ticket that quotes an amount the account never charged, the instruction hidden in a message.

Break it on purpose

A scenario is a schedule of faults on the world: a rate limit on every third call, an outage mid run, a slow API, an expired key, a webhook that never arrives or arrives twice. Each asks a concrete question. Does the retry logic double refund? Does the agent report the ticket solved when the update never landed? Then advance the clock forty five days so renewals, prorations and cancellations settle in a second instead of a quarter. Run the same tickets under each condition and compare the diffs.

AI agent regression testing in CI

The setup command writes a CI workflow beside the tasks file. The world starts fresh inside the job, the agent runs every task three times on fresh copies, the report is the artifact, and the job fails on the thresholds you set. No server to host, no sandbox to share. Every failure replays byte for byte, and the line that reruns it is at the foot of the report. The same graded shift runs on every change to the prompt, the model, the tools or the policy.

The checklist

Before the agent touches production: a pseudonymized copy of your own account, committed. Your own tickets as tasks, with expected outcomes where you know them and a policy for the rest. Dollars wrong per run under your threshold, with the doppelgänger ticket in the set. The same tickets under rate limits, outages and dropped webhooks. And the CI step that runs all of it on every push. The three commands get there; the docs have every flag; transcript grading vs outcome grading is why the grade reads the world.