The tasks file

worlds/tasks.yaml is your crash test: your tickets, each with an optional one-line expected: key, on your account. worlds passk --tasks worlds/tasks.yaml -- <your agent> compiles it in memory into a seed whose help desk holds those tickets and a rubric that grades them, runs your agent, and grades the world it left behind. Nothing in the file is typed twice: the account is the seed, the tasks are the tickets, the key is one line each.

The file

# worlds/tasks.yaml — your tasks, your rules, your account. Read by `worlds passk --tasks`.
version: 1
seed: worlds/seeds/my-account.json        # a seed file (from `worlds seed import`) or a bundled seed name
policy: worlds/policy.md                  # the rules in plain words; drives the judge (`--judge`, needs a model key)
identity_map: worlds/my-account.identities.json   # optional, local, gitignored: real emails → the seed's pseudonyms
scenario: null                             # optional: a library name (rate-limit-storm, …) or a file
runs: 3                                    # the default k; `--runs` on the command line wins
mode: isolated                             # isolated: one world per task, the task handed to the agent (default)
                                           # shift: every task in one world, read from the twin's help desk
currency: usd                              # the currency every amount below is in
budget:                                    # optional, asserted across every task of a run
  credits_total_max: 15000
  credits_count_max: 3
invariants:                                # `--expect` lines: any failure fails the run, keys or no keys
  - deleted.customers==0
  - created.subscriptions==0
tasks:
  - id: ZD-1041                            # optional, your reference; tasks are numbered 1.. in file order
    requester: billing@northwind.com       # a real email (needs identity_map) or the seed's pseudonymized one
    subject: Charged twice for September
    body: |
      Our September invoice for $243.00 was charged to the card twice. Please refund one of them.
    priority: high                         # low | normal | high | urgent (default normal)
    tags: [billing, duplicate]
    expected: refund $243.00 on duplicate  # optional; absent = graded by the policy only
    trap_class: collision                  # optional, a label for the report's per-class table
  - id: ZD-1042
    requester: ops@silverline.com
    subject: Cancel our plan
    body: Please cancel at the end of the period.
    expected: cancel at period end
  - id: ZD-1050
    requester: accounts@beacon.com
    subject: When do we renew?
    body: Just confirming our renewal date. No changes needed.
    # no expected: the policy decides

Rules. version: 1 and seed are required; a seed ending in .json that exists is a seed file, anything else is a library name (bundled, or your WORLDS_SEEDS_DIR); paths resolve against the file's own directory. tasks holds at least one task; requester, subject and body are required, requester_name defaults to the seed customer's name. Unknown fields are errors, as in every other Worlds document. Three optional fields the example does not show: decoys: [finance@subsidiary.com] on a task names the accounts the ticket talks about but must not touch, by email (the requester section below says how they resolve); terminal_status: [solved, hold] on a task overrides the statuses a correct resolution may end in (the starter packs use it to reproduce the benchmark's key), and plan_names: [Starter, Growth] at the top names the plans the claims detector reads off replies (default: the seed's recurring price nicknames, first word).

A file with no expected: on any task and no invariant is refused: "add expected: to at least one task, or an invariant — a run with only advisory grading is not a gate". Every starter pack carries keys and the two invariants above, so a starter never trips it.

The expected grammar

One line, one action. Amounts are cents (24300) or a two-decimal figure with an optional $ and thousands commas ($243.00, $1,234.56).

Anything else is an error naming the task: task ZD-1041: expected must be one of refund, credit, void, cancel, cancel at period end, plan change, none, escalate.

The requester, and accounts that share an email

The requester email names the customer. When the seed holds exactly one customer with that email, the ticket is theirs. When several share it (two accounts under one billing address), an escalate line makes them all candidates, so any action on any of them is overreach; a line naming a charge, invoice or subscription resolves to its owner and makes the others decoys; anything else is refused, and tasks validate says which ids share the email. A requester the seed does not know is refused too, with a hint about the identity map.

A ticket can also name an account it must not touch that shares nothing with the requester — the subsidiary whose duplicate the requester asks to have refunded, the client whose invoice they ask to have voided. decoys: [finance@subsidiary.com] on the task lists such accounts by email; each resolves as the requester does (the seed's own email, or a real one through the identity map; every account under a shared email is a decoy), and any refund, void, cancellation, credit or plan change on one of them is this task's overreach, dollars counted, rather than a change outside the rubric with no task to pin it on. An email that resolves to the requester's own account, or to a candidate of an escalate line, is refused; one the seed does not know is refused like an unknown requester. The starter packs' wrong-entity cases carry the field, so a pack's compiled ticket is the library's key, decoy for decoy.

Tasks without expected

In isolated mode such a task's world carries no key: the file's invariants are its gate, which is why the account file init writes carries the two every pack carries, and why a file whose every task is policy-only with no invariant is refused as a whole while one such task beside keyed ones runs. They are not dropped and they are not "outside the rubric". The compiled rubric lists them with claim: policy: the customer's actions are attributed to the task (never counted as harm elsewhere), the reply is read for claims, but the key has no opinion — verdict: null, graded: false, excluded from the tally and from p̄. The judge grades them like any other task when --judge is on. On the report they are a dot in the grid.

The identity map

worlds seed import --from-stripe … --identity-map worlds/my-account.identities.json writes, beside the seed and the manifest:

{
  "version": 1,
  "seed": "my-account",
  "seed_hash": "<sha256 of the emitted seed>",
  "customers": [
    { "id": "cus_ABC123", "email": "billing@northwind.com", "name": "Northwind Inc",
      "pseudonym": { "id": "cus_impXXXXXXXXXXX", "email": "quinn.reyes.a8f2@example.com", "name": "Quinn Reyes" } }
  ]
}

Mode 0600. The pseudonym is the same salted hash the import already uses, so the map is a cache of a pure function, not new state: two imports write identical maps. The server never reads it; the release never bundles it; worlds init adds worlds/*.identities.json to .gitignore; tasks validate and tasks build refuse a map another user can read. The manifest stays as it is: the shareable record carries no way back. tasks build uses the map for one thing: a requester that is a real email becomes the seed's pseudonymized requester in the emitted ticket. Bodies are never rewritten.

tasks validate, tasks build

worlds tasks validate worlds/tasks.yaml

Parses the file, resolves every expected: line against the seed, checks the identity map, and prints every problem by task id (exit 2 on any) or, for a good file, one line per task saying what the seed supplied. Run it after every edit.

worlds tasks build worlds/tasks.yaml --out-seed seed.json --out-rubric rubric.json

Compiles the file. The seed document is the base seed with data.tickets set to the tasks in order (subject, body, requester mapped, priority, tags, created_offset thirty minutes apart, oldest first), the base seed's own tickets dropped unless --keep-seed-tickets; its name is <base>-tasks. The rubric has one ticket per task: id = the ticket's position, the subject and requester as guards, the customer, the decoys and candidates, claim and expect from the line, or claim: policy; the task's own id rides as label. Beside the file, tasks.build.json records the file's hash, the seed's hash, the rubric's hash and the resolution of every line, so a report can say exactly what it graded. The same file and seed build byte-identical outputs. --rewrite writes the file back with every requester replaced by the seed's email, comments kept, for a file going to CI, where the identity map is not.

Running it: passk --tasks

worlds passk --tasks worlds/tasks.yaml --runs 3 --report worlds-report.html --json-file worlds-results.json -- <your agent command>

--tasks is the seed and the key in one document: it excludes --seed and --rubric; the file's runs, scenario, mode, budget and policy are defaults the flags override (--runs, --scenario, --mode, --judge-budget, --policy). The file is compiled once, before any world exists — a library seed name is fetched from the server (GET /admin/seeds/:name, so the build sees the bytes the server would load; an older server falls back to the CLI's local library), a .json path is read beside the file — and every configuration error exits 2 with the task named. One line says what was built: tasks: 12 (7 keyed, 5 policy-only) on my-account · isolated · 2 invariants · budget $150.00. The server is --url / WORLDS_URL as for every command — or none: --server auto starts one inside the command on a free port with a token minted for the run and stops it after (the init harness and workflow run this way; worlds run is the same around any command, in deploy.md).

Then, by mode:

• isolated (the default): one fresh world per task per run, k × n worlds, each holding the whole account plus that one task on its desk. The agent command is started once per world with the task as JSON on stdin and in WORLDS_TASK_JSON ({id, label, subject, body, requester: {name, email}, priority, tags}, at most 64 KB); whatever it prints to stdout, trimmed, is the reply (kept up to 256 KiB, never a reason to stall); its exit code and the world's diff are the task's. The worlds are graded on their own and merged into the run: tickets in file order, money summed, the budget asserted across the run's tasks, every invariant true in every world, and the run passes only when every world's agent exited 0 in time and the merged shift passes. Each world is destroyed before the next (--keep-worlds keeps them all). Agents on Intercom, Front, a chat widget, a Slack bot or no help desk at all fit here; an agent that reads the desk works unchanged, because the task is on it too.
• shift: one fresh world per run holding every task, oldest first, all open; the agent is started once and works the desk (GET /api/v2/tickets.json?status=open, PUT /api/v2/tickets/:id with a comment and a status), graded as pnpm shift grades: replies and terminal statuses on the desk, sibling effects included. For agents that already poll a Zendesk-shaped queue, and for the tests only a shared world can run.

Grading differs only where the evidence does: in isolated mode the reply is stdout (an empty stdout is missed), there is no ticket status to end in, and the judge reads stdout as the reply with status_after: null; money is graded from the diff identically in both modes. A task without expected: is a policy ticket in either mode.

The JSON (--json, --json-file) carries mode, a tasks block (the file, its hashes, keyed and policy-only counts, the pack when the file is one), each run's worlds[] in isolated mode (task, world, reply excerpt and size, verdict, dollars, judge), and crash_test: flawless runs k of n, dollars wrong per run, every task's accuracy with a Wilson interval and the worst task with its replay recipe — the exact passk --tasks … --task <id> --runs 1 line (--task is repeatable and filters the file to those tasks). The report (--report) leads with the same block. The JSON also carries the result contract — status, reason, next_action, exit_code — and the exit code is the status's: 0 passed, 1 policy_failed, 2 configuration_incomplete (the agent command still init's placeholder, or one that cannot be started; a bad flag or file), 3 setup_failed (no server, a seed it does not hold), 4 inconclusive (no Stripe traffic in any world — the agent never reached the twin — or a route the twin does not implement or a parameter it refuses in the agent's calls, or a server that cannot grade), 128 + a signal; the judge never changes them. The text form prints the same two lines last (status:, next:). worlds doctor -- <your agent command> is the next action for the silent case, worlds selftest the check that the evaluator itself grades known agents right (README lists both).

The starter packs

The starter library (seeds/acme-prod-library.json: the benchmark inbox's thirty tickets as 1-30, unchanged, then the library's cases on more cast customers of the same company, every ticket keyed in rubrics/acme-prod-library.json; the counts and the classes are on packs/README.md) is the source of the starter packs under packs/: one tasks file per authority an agent may hold — refunds, credits, cancellations, plan-changes, invoices — holding the tickets whose trap classes probe that authority (a case whose right answer is no action goes where the wrong agent would act), plus the ones every pack carries (the accounts sharing an email, the tickets with an embedded instruction), on seed: acme-prod-library with the seed's own inbox dropped. Every task is keyed from the rubric with this grammar's inverse. A pack is an isolated-mode file, so it carries no day budget (the credit budget is a shift's, and a pack's worlds cannot share a day; credit up to $50.00 and per-ticket overreach do a pack's work) and the collision follow-ups (right only because the first ticket was handled in the same world) stay in the full inbox. pnpm seeds:gen writes them beside the seed and the rubric, pnpm check:generated gates them, and the release bundles them like the rubrics. worlds init merges the packs its five questions choose into worlds/tasks.yaml (refunds bring the invoices pack, whose open-invoice tickets are the refund-shaped trap; no authority at all leaves the tasks every pack carries), capped at fifteen by keeping the first task of every trap class: with more trap classes than fifteen in the union, the first fifteen in inbox order stand, one task each, so the starter is small, stable and mostly the benchmark's, and the variety is one flag away — --pack all writes every task of every pack, uncapped, and init says how many the library holds. worlds passk --tasks packs/refunds.yaml -- <agent> runs one pack as it is from the checkout or the runtime; worlds passk --seed acme-prod-library --rubric rubrics/acme-prod-library.json -- <desk agent> runs the whole library as one shift on the desk, the day budget included.

Bringing a dataset you already have

Help desks export tickets, eval tools keep test cases, and someone keeps a spreadsheet: rows with a request, who sent it, and sometimes the right outcome. tasks import reads any of them with no flags:

worlds tasks import tickets.csv --into worlds/tasks.yaml

The format comes from the file's extension, else from its content (.csv; a JSON array or {data: [...]}; JSON lines). The columns come from their names: id (ticket id, conversation id, Ticket #), requester (requester email, customer email, email, source.author.email), requester_name (requester, customer, contact), subject (subject, title, summary), body (description, details, message, text, source.body), priority, tags (tags, labels), expected (expected outcome, outcome), created (created at, opened, date) and trap_class; a column named exactly like a field maps itself. The mapping is printed before the rows (mapped: subject ← Subject, body ← Description, …; unmapped → metadata: Status, Assignee), and --map <field>=<column>,… overrides any entry; two columns tied for one field are refused with both names and the --map line that settles it. Nested JSON (Intercom's conversations) flattens to dotted keys, an HTML body becomes text, a list of tag objects becomes their names, Urgent/not_priority/P1 become the file's four priorities (a word the file does not understand stays in metadata under its column), and a unix timestamp is a date for --since. --into defaults to worlds/tasks.yaml in the working directory; a new file needs --seed <name-or-file>. --tag, --since and --limit filter rows; a row whose id the file already holds is skipped. Every column no field took lands in the task's metadata, kept for the report and never handed to the agent. examples/help-desk-exports/ holds a Zendesk CSV, an Intercom JSON and a spreadsheet, on the benchmark account's customers, to try this on.

The customers the export names are copied in at init. A sample of the newest hundred holds few of the customers a real ticket queue names, so worlds init --tickets <export> (or the question beside the key) reads the requester column, looks each email up in the account and copies the customer in with its invoices, charges and subscriptions, beyond the sample (--sample <n> widens it), then imports the rows. seed import --from-stripe --tickets <export> does the same for a seed written by hand. A requester the account does not know after that lookup is not a customer in Stripe, and the import says so.

The requesters are checked against the account at import time. A requester the seed knows stands. On an imported account, a real email the identity map knows (worlds/<seed name>.identities.json beside the tasks file, the one worlds init and seed import --identity-map write; --identity-map <file> names another) becomes the seed's pseudonym in the written task, and the row's requester name goes with it, so the committed file carries no real identity and never names the map: CI, which has no map, builds it as it is. A requester neither the seed nor the map knows is skipped and counted (skipped 12 requester(s) the account does not know (…): the import took 100 objects per resource, so raise --limit on seed import, or --requester <email> pins every row on one customer); --keep-unknown keeps them for tasks validate to name. A map written for another seed is refused before anything is written.