Can AI Agents Run a Shop? What Six Benchmarks Actually Measured

Since early 2025 a small field of benchmarks has handed AI agents a business to run: a vending machine, a supermarket, a set of online stores, a single product's stock. Each gives the agent money, tools and a goal, then lets the days run. Read together, the results say less about which model is cleverest than about which parts of running a shop an agent can be trusted with today.

1. The six benchmarks

2. Agents make money, but far less than they could

In Vending-Bench 2 the best model ends a simulated year with an average of $16,484; the lab estimates that a straightforward good strategy would make roughly $63k, and writes that such a strategy "could easily do 10x better than the current best LLMs"¹. In RetailBench's supermarket, which opens with 30,000 in the bank and pays rent every day, the strongest model finishes with 24,351, below what it started with, against 131,510 for a rule-based policy that is given more information than the agents³.

The gap is not mainly about intelligence. In the first Vending-Bench the authors found that "all models have runs that derail", through misread delivery schedules, forgotten orders or loops of reasoning that go nowhere². In RetailBench, survival ranged from 58 to 180 days, and only two of seven models kept the shop open for the whole run³. A store has to be run every day; a model that is brilliant on day 10 and confused on day 90 runs it badly.

3. The same task, done every time

τ-bench, from the customer-service company Sierra, asks a different question: if an agent solves a task once, will it solve it again? Its retail domain gives the agent an order database and a returns policy, and a simulated customer who wants a refund or an exchange. In the original paper, even strong agents solved fewer than one task in four on all eight of eight attempts⁵. On the current leaderboard the best models do much better on one attempt, and still lose about a third of their successes when the same task has to be solved four times out of four⁴:

For a store this is the number that matters. A refund handled correctly four times out of five means one customer in five gets the wrong answer, and nobody can say in advance which one.

4. Knowing is not doing

STOCKTAKE isolates the step between seeing a problem and acting on it. An agent replenishes one product for 26 weeks while hidden failures hit demand, suppliers and freight; it is scored from 0, a rule that ignores every warning sign, to 1, an oracle that sees exactly what the agent sees. The four frontier models tested detected 84 to 88% of the hidden failures, usually within a week, yet their scores ranged from 0.62 to −0.23: two of the four ended below the rule that ignores the signs⁶. Even when an agent had diagnosed a stress week correctly, 34 to 43% of those weeks still ended with the shelf empty⁶.

Money handling shows the same split. In E-Commerce Bench, 152 of the 576 suppliers are fraudulent, and "every model sends part of its procurement budget to suppliers that defraud it"⁷. The model that grew its stores the most, from ¥100,000 to ¥1,431,425, ranked 16th of 18 on avoiding fraud⁷. The first real shop run by an agent, Anthropic's Project Vend, failed in similar ways and improved mainly through procedures and checks, a story the guide to AI agents tells in full.

5. What works: formulas count, models adjust

The most useful result comes from InventoryBench, which tested the obvious alternatives on the same 1,320 problems: an operations-research policy alone, a language model alone, and the two in sequence. The best arrangement let the formula compute the order and the model adjust it⁸:

That is a 21% improvement over the formula alone, and the same order of the two steps came out best with a second model too⁸. The model adds what the formula cannot see: a demand shift, a promotion, a supplier that has changed. The formula keeps the model from inventing a shift in what is only noise.

Retail-Bench, the open simulation marql built to test replenishment policies, points the same way. There, a contender in which a language model reviews every evening's orders on top of an engine and a rule book scored 70% of the possible improvement, and the same contender with the model removed scored 71%; the leaderboard's own summary is that "a model's judgement adds nothing yet"⁹. The quantities came from the rules; the model's review did not move the result. The reorder point guide covers the formula those rules start from.

6. How to read an agent leaderboard

Several of these results rest on thin evidence. RetailBench reports one selected run per model on one simulated world, and STOCKTAKE's cost figures rest on one run per seed^{3, 6}. Agent results vary from run to run, and how much a ranking can be trusted depends on how it was built. Five questions settle most of it:

The figures in that checklist come from Anthropic's guidance on error bars¹⁰, Princeton's study of agent costs¹¹, the audit of Chatbot Arena known as the Leaderboard Illusion¹² and the Agentic Benchmark Checklist¹³.

7. Limitations

All six are simulations or controlled tasks, most of them preprints, and several model names refer to versions that change every few months. The vending and online-store settings are simpler than a chain of physical stores; none of the benchmarks tests a real chain's data. Retail-Bench is marql's own simulation, cited here for its published finding, not as independent evidence.

How marql uses this

marql's AI employees follow the split these benchmarks support: quantities come from fixed rules and a buffer engine, the language model explains them and flags exceptions, and a person approves each order. How much authority each one gets, and how its effect is measured against stores where it only advises, is set out in the guide to AI store replenishment.