The workday, graded — 30 tickets, claude-sonnet-5
30-ticket shift · 5 runs per act · 1 model · graded by the world diff, never the transcript · 2026-09-04 07:12 UTC
| model | Act I — clear skies no storm | Act II — Stripe stormstripe-storm-shift | Act III — tickets stormtickets-storm |
|---|---|---|---|
claude-sonnet-5 |
per-ticket p̄ 95.3% 90.7%–97.7%
flawless shifts 0/5 observed 0%
predicted p̄^30 23.8% (5.3%–50.1%)
dollars wrong $960.00 5 wrong · 0 missed · 2 overreach
reply ≠ world 0 0 claimed-not-done · 0 did-not-claimed · 0 amount
30 days later −$39.60 3/5 match the reference · 1 customer differ
measured 16.91M in · 88k out · $0.95/shift · 6 min/shift 642 turns · 683 tool calls
|
per-ticket p̄ 95.3% 90.7%–97.7%
flawless shifts 0/5 observed 0%
predicted p̄^30 23.8% (5.3%–50.1%)
dollars wrong $960.00 6 wrong · 0 missed · 1 overreach
reply ≠ world 0 0 claimed-not-done · 0 did-not-claimed · 0 amount
30 days later −$19.80 4/5 match the reference · 1 customer differ
measured 17.39M in · 90k out · $0.97/shift · 6 min/shift 662 turns · 703 tool calls · 30 errors
|
per-ticket p̄ 96% 91.5%–98.2%
flawless shifts 0/5 observed 0%
predicted p̄^30 29.4% (7.1%–57.2%)
dollars wrong $960.00 5 wrong · 0 missed · 1 overreach
reply ≠ world 0 0 claimed-not-done · 0 did-not-claimed · 0 amount
30 days later −$19.80 4/5 match the reference · 1 customer differ
measured 18.54M in · 102k out · $1.05/shift · 6 min/shift 661 turns · 733 tool calls · 45 errors
|
p̄ is ticket passes over tickets × runs with a 95% Wilson interval; predicted p̄^N raises the point estimate and both bounds to the shift length (independence assumed). "Flawless" means every ticket right and nothing else touched. "30 days later" is the model-free epilogue: the world's clock advanced 30 days and next month's invoices compared against the perfect employee's world. Cost is estimated from measured tokens at list prices.
Where each model breaks — accuracy per trap class, every act pooled
| model | budget | cancel_period_end | collision | duplicate_subscription | eligibility | identity | injection | partial_refund | phantom | plan_change | policy_interaction | policy_window | procedure | question | void_not_refund |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
claude-sonnet-5 | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 0% | 88.9% | 100% | 100% | 100% | 100% | 100% | 100% |
The cliff
At 95.3% per ticket (90.7%–97.7%), a 30-ticket shift is flawless 23.8% of the time; the interval allows 5.3%–50.1%. Observed 0/5.