Transcript grading vs outcome grading

Transcript grading reads what an agent said: the tool calls it reported, the reply it wrote, the reasoning it showed. Outcome grading reads what it did: the records that changed, the money that moved, the ticket that closed. The two agree most of the time. The cases where they disagree are the ones where an agent reports success and did something else, and those are the cases that matter.

What transcript grading sees

An evaluation platform takes the agent's traces and grades them against criteria: was the tone right, did it call the refund tool, did it ask before acting, did it follow the policy. Often the grader is a model, an LLM as judge with a rubric. It is cheap, fast and useful: it catches the agent that never tried, the one that was rude, the one that ignored the policy in its reasoning. Its limit is its input. Everything the judge knows about the world, it learned from the agent.

LLM as judge limitations

The limits are structural. A judge reading a transcript cannot verify a claim about the outside world, because the transcript is its only evidence. It cannot see an effect the agent did not report, like the second refund a retry produced after a timeout, or one the agent misreported, like the ticket it said it escalated and actually closed. And a judge that is right most of the time still passes a share of the runs that should have failed, most of all the runs that were written to look right. A judge is not useless. It is the wrong instrument for the question of what happened.

What a world diff records

Outcome grading reads the system. A run starts from a mark, the world's state at that moment. The agent acts through its stock SDK, and the world keeps every object it creates, changes or deletes under the real system's rules. The diff lists every change since the mark: each refund with its amount and customer, each ticket status and comment, each user deactivated, each order fulfilled. Nothing in it came from the agent's account of itself. It reproduces to the byte, because the same seed and script give the same world.

The case that separates them

A customer writes in about a duplicate charge. Two customers in the account share her name. The agent finds the other one first, refunds the duplicate it sees there, writes an accurate reply and closes the ticket. The transcript is flawless. The diff shows a refund on a customer who never wrote in, an untouched duplicate on the one who did, and a closed ticket with a promise the records do not keep. Every variation behaves the same way: the invoice number that matches another customer's amount, the amount the ticket quotes that was never charged, the instruction hidden in a message.

How the two fit together

Use both, in a fixed order. The diff decides what happened: dollars wrong, objects touched, forbidden actions. The judge then rules on what the diff cannot see, whether the reply and the reasoning met the policy, with the world's facts in front of it, quoting only amounts actually in view. A verdict names the outcome and the conduct, each with its reason, and the worst run replays byte for byte. Keep your traces on your evaluation platform. Take the world and the verdict from the environment.

When transcript grading is enough

Some agents do not act. A drafting agent, a classifier or a summarizer changes nothing outside its reply, and for those the transcript is the outcome. The moment an agent has a tool that changes a record, the transcript becomes a claim about the outcome. If a run can produce a diff, grade the diff first. The comparison shows where an evaluation platform stops; testing agents before production is the run end to end.