Shopify's development stores are for development and testing only: they cannot process real transactions and they cannot be converted into production stores (Dev stores). The other route, switching a live store's payment providers into test mode, has the opposite problem: while the providers are in test mode, customers cannot place live orders. So an agent that handles orders, refunds and fulfillment is tested either in a store that can never take a real order or on the store itself with checkout switched off, and in neither case can the store be reset, broken on purpose or read back for what the agent changed. The Shopify world is a stateful Shopify mock built for testing agents: the same URLs, the same errors and the same state machine as the real API, on your machine, verified against the real system.
Shopify test store limits
A development store has no customers of yours in it and no order history, so the cases that matter, the repeat buyer asking about last month's refund or the order that shipped in two parts, have to be built by hand and rebuilt after every run. A live store in test mode has the history but cannot be reset, and every run leaves refunds, notes and fulfillment events behind that the following run has to work around. A world is a pseudonymized copy of your own store, taken when you ask for it: the same products, the same orders in the same states, the same customers under new names. Every run starts from those bytes and ends by throwing the world away.
A stateful Shopify mock: products, orders, fulfillment, refund flows
The world keeps products and variants with their inventory levels, customers, orders with line items and their financial and fulfillment status, fulfillments, refunds and transactions. It keeps Shopify's rules with them. A refund cannot exceed what was paid, and a second refund of the same line item is refused. A cancelled order cannot be fulfilled. Restocking a returned item changes the inventory level, and the diff shows it. An order paid by two transactions refunds against the right one. Everything the agent creates or changes stays in the world, so the second request sees what the first one did, and the wrong sequence gets the real error in the real words.
Test your AI agent against Shopify
Point the agent's Shopify client at the world. Nothing in the agent changes. The path is three commands: the first copies the store and writes the worlds folder, the second turns your own help desk export into tasks, one per row, with the expected outcome on the rows where you know it, and the third runs the agent three times on fresh copies and writes the report. The three steps take about ten minutes; the GET ACCESS form on this page is where the Shopify world starts.
AI agent refund testing on orders
A run starts from a mark and ends with a diff: every refund issued, in dollars, against which order and which line items; every fulfillment created or cancelled; every inventory level that moved; every note and tag. The ticket asks for a refund on one of two identical orders from the same customer; the transcript says done; the diff says which order lost the money. The ticket asks to cancel a shipment; the transcript says cancelled; the diff shows the fulfillment untouched and a note added. The report says dollars wrong per run, gives every task a verdict and a reason, and replays the worst run to the byte.
Break it on purpose
Shopify's API throttles by a leaky bucket, and an agent working a queue will drain it at the worst moment. In the world that moment is on your schedule: a 429 on every third call, an outage mid run, a slow API, an expired token, an order webhook that arrives twice. Does the retry logic refund twice? Does the agent tell the customer the order was cancelled when the call never went through? Then move the clock, so the return window closes and the refund the agent tries is refused the way the store would refuse it.
Shopify test environment for agents, in CI
The same command that runs on a laptop is the CI step the first command wrote. No store to share, no test mode to switch on: the world starts fresh inside the job, the agent runs every task, the report is the artifact, and nothing leaves the machine. Every world in the catalog takes the same path; WooCommerce, Zendesk and Stripe are three more of them, and the docs are the path in full.
