Docs: AI agent regression testing in CI

Test your agent on a copy of the systems it acts on, on your own machine, then run the same test in CI on every push. The ten minute path first, then every command, then the questions that come up after.

Quickstart

What you need

• Node 24 or newer.
• Your agent, callable as a single command.
• Optionally, a restricted read only key for the account to copy, and a help desk export.

See it work first, no setup, no key

npx -p sparta_worlds@0.6.0 worlds try --server auto -- <your agent command>

This starts a copy of Stripe, hands your agent one ticket (a customer charged twice for a $40 invoice) and prints what changed in the account. Here is the output for a stand in agent that refunded the wrong charge:

what changed in the world:
  created refund re_e2cT4ZBAljNFP9FT: $199.00 on ch_seed0196
  updated charge ch_seed0196: amount_refunded $199.00, refunded

the reply: “Confirmed the duplicate charge and refunded it to your card.”

verdict: missed — missing refund 4000 on ch_seed_dana_1|ch_seed_dana_2; outside the task: refund 19900 on ch_seed0196 (re_e2cT4ZBAljNFP9FT). ✕
status: policy_failed — 1 of 1 run failed — 1 task verdict against the key (dana-duplicate: missed), $239.00 wrong

ch_seed0196 belongs to a different customer. Dana's $40 duplicate was never refunded. The reply sounded right; the records say otherwise. That gap is what Worlds measures.

Step 1: Connect your agent

Worlds hands each ticket to your agent as JSON, on stdin and in WORLDS_TASK_JSON:

{ "id": "...", "label": "...", "subject": "...", "body": "...", "requester": { "name": "...", "email": "..." }, "priority": "...", "tags": [] }

Your agent prints the reply it would send the customer. No library, no dependency.

Node:

const task = JSON.parse(process.env.WORLDS_TASK_JSON);
const reply = await handleTicket(task); // your agent, as it is
process.stdout.write(reply);

Python:

import json, os, sys
task = json.loads(os.environ["WORLDS_TASK_JSON"])
sys.stdout.write(handle_ticket(task))  # your agent, as it is

Whatever runs those lines once is <your agent command> below. For an HTTP endpoint or a queue, see Connecting your agent, all patterns. A stock Stripe SDK needs no change; a bundled one, see Wiring the Stripe SDK.

Step 2: Set up your project

Run once, in your agent's repository:

npx -p sparta_worlds@0.6.0 worlds init --agent "<your agent command>"

It asks for three things:

• A key, to copy an account onto your machine. For Stripe, a restricted read only key (read on Customers, Invoices, Charges, Subscriptions, Products, Prices) or a test mode key. Used once, in memory, never written to disk. A live secret key is refused. Press Enter to start on a bundled sample account instead.
• A help desk export, if you have one. Its customers are copied in and its rows become your first tasks.
• Your refund and credit policy, five questions with defaults (Enter accepts each). Flags skip the prompts: --refunds 60 --credits 5000x3 --cancel period-end --plan-changes yes --escalate "the billing lead".

It writes a worlds/ folder:

init never overwrites a file. Run it again and it fills in what is missing.

Step 3: Bring in your tickets

A help desk export (Zendesk CSV, Intercom JSON, JSON lines, a spreadsheet) becomes one task per row:

npx -p sparta_worlds@0.6.0 worlds tasks import tickets.csv --into worlds/tasks.yaml

Columns are matched by name and the mapping is printed first; --map <field>=<column> fixes a mismatch. Requesters the account knows are added, real emails become their pseudonyms, and unknown requesters are skipped and counted.

Where you know the right outcome, say so:

expected: refund $243.00 on duplicate

Leave it out and the policy decides. tasks.md has the grammar; examples/help-desk-exports/ has three sample exports.

Check the file before running anything:

npx -p sparta_worlds@0.6.0 worlds tasks validate worlds/tasks.yaml

It names every problem by task id.

Step 4: Run the crash test

npx -p sparta_worlds@0.6.0 worlds passk --server auto --tasks worlds/tasks.yaml --runs 3 --report worlds-report.html -- <your agent command>

Your agent runs three times on fresh copies of your account; the world starts inside the command. worlds-report.html shows:

• Dollars wrong per run, from the world's records.
• A verdict and a reason for every task.
• A replay command for the worst run, byte for byte.
• The runtime version and the seed hash.

Every result ends in one of five statuses (passed, policy_failed, configuration_incomplete, setup_failed, inconclusive) with a next action (deploy.md).

This is the line your CI workflow runs on every push. Commit the worlds folder and you are done.

Two options: --scenario rate-limit-storm adds rate limits and 429s mid run. --judge grades every task against policy.md, including those with no expected answer; see Grading with the judge, the one step that involves a third party.

Everything below is reference.

Every command, in full

Every command is npx -p sparta_worlds@0.6.0 worlds …: the runtime from the npm registry, pinned to one version, nothing installed into your project. The prefix is left off the table.

Flags used across commands

Connecting your agent, all patterns

Stdin and an environment variable: the default, Step 1.

An HTTP endpoint: call it directly.

curl -s -X POST "$AGENT_URL" -H 'content-type: application/json' -d "$WORLDS_TASK_JSON"

A queue or a database: a small wrapper inserts the task and waits for the reply. The wrapper is the <agent command>.

An agent that polls a help desk: --mode shift, no wrapper. See Working from a help desk queue.

Wiring the Stripe SDK

try, passk and doctor preload the configuration (NODE_OPTIONS --import or PYTHONPATH) and point a stock Stripe client at the world when it is constructed. They set STRIPE_SECRET_KEY, STRIPE_API_KEY, WORLDS_API_KEY and WORLDS_BASE_URL. A client that sets its own host is left alone.

If your agent is bundled, uses another SDK, or the preload cannot reach it, pass --no-preload (or WORLDS_PRELOAD=0) and wire it by hand:

// stripe-node
const url = new URL(process.env.WORLDS_BASE_URL);
const stripe = new Stripe(process.env.WORLDS_API_KEY, {
  host: url.hostname,
  port: url.port,
  protocol: url.protocol.slice(0, -1),
});

# stripe-python
stripe.api_key = os.environ["WORLDS_API_KEY"]
stripe.api_base = os.environ["WORLDS_BASE_URL"]

Still not reaching the world? Run doctor. See Troubleshooting.

Working from a help desk queue

If your agent polls a help desk itself, add --mode shift to try, passk or doctor. It reads tickets from the world's help desk at /api/v2/tickets.json with the same key and replies through PUT /api/v2/tickets/:id, as it would against a Zendesk style API. scope.md lists the desk routes.

Grading with the judge

Without it, Worlds grades every task against your expected outcomes and invariants, on your machine. --judge adds a language model that reads policy.md and rules on every task, including those with no expected: line: whether this refund, on this timeline, was reasonable under a policy written in prose is a judgment a fixed rule cannot make.

This is the one place anything leaves your machine: the task's facts, your policy text and the agent's reply go to the model provider. It is off unless you add the flag.

To use it, set ANTHROPIC_API_KEY, add -p @anthropic-ai/sdk after npx and --judge before the -- on the crash test line. In CI, an ANTHROPIC_API_KEY secret turns it on. The judge's verdicts are advisory: they sit beside the automated ones in the report and never change pass, fail or the exit code.

Handing setup to a coding agent

Point Claude Code, Cursor or Codex at https://twinlab-site.vercel.app/llms.txt: the same steps, written for an agent. It asks you for the key at the prompt, never commits the identity map, and creates no account. The prompt to paste is on the get started page.

Client libraries

Python, one file, vendor it:

curl -O https://twinlab-site.vercel.app/dl/worlds_client.py

JavaScript, for tests against the world's admin API (admin-api.md):

npm i -D https://twinlab-site.vercel.app/dl/worlds-client-0.6.0.tgz

Troubleshooting

My agent ran but nothing changed in the world. Its Stripe client never reached the world. Ask why:

npx -p sparta_worlds@0.6.0 worlds doctor --server auto -- <your agent command>

doctor shows the environment the agent was given, the world's request log, and how its SDK connected. The fix is usually --no-preload plus wiring by hand (Wiring the Stripe SDK), or a bundle that hides the environment variables.

A requester in my export is not a task. That email is not in the copied account. Widen the sample with --sample on init, or seed import --from-stripe with a larger --limit.

tasks validate complains about a task I did not write. A column mapped to the wrong field on import. Rerun tasks import with --map for that field.

The report says policy_failed but the agent was right. Check whether your five answers in policy.md cover the case. A policy that never mentions plan changes fails every plan change. Edit the file.

Is the grader itself broken? Run its own check:

npx -p sparta_worlds@0.6.0 worlds selftest --server auto

Five known agents, five known verdicts, no model. If it passes, the problem is in your setup.

FAQ

Does my key get stored anywhere? No. It is used once, in memory, and never written to disk. Live secret keys are refused.

What does "pseudonymized" mean here? Names, emails and free text are replaced. Amounts, invoices, subscriptions and refund history are kept as they were.

I don't have a help desk export yet. Can I still try this? Yes. Without a key or an export, init starts on a bundled sample account with up to 15 starter tasks.

Does this only work with Stripe? Stripe is the world available today. For another system, tell us on the catalog page.

Can I run this in CI? Yes. init writes the workflow. The same passk line runs on every push, and the world starts inside the job.

Does anything leave my machine? Only with --judge, which sends the task's facts, your policy text and the agent's reply to the model provider. Everything else runs locally.

What happens if I run init again? Nothing is overwritten. It fills in missing files only.

Licensing

Every command on this page runs, verbatim, in the release check before a version ships (pnpm release:smoke -- --from-dir <dir>, deploy.md). Evaluation use, not for redistribution, no warranty. Full terms are in LICENSE.txt inside the package.

How Worlds differs

Stateful copies of business systems are not unique to us; hosted platforms sell them, metered per run. What that category does not claim and Worlds does, with something published behind every line. Checked against their public sites on 2026-09-08. Tell us what changed and we will update it.

Eval platforms (Braintrust, Langfuse, Arize, Confident AI) grade what the agent said, from traces. The system it acts on is yours to supply. Keep your traces where they are and take the environment and the verdict from Worlds.

Conversation simulators (Coval, Hamming) simulate the customer talking to your agent. Worlds simulates the systems your agent acts on. The other half of the problem.

Environment vendors for labs (Fleet AI, Halluminate) sell environments in which computer use agents are trained. A different buyer and a different product.