Turning an Agent MMO into an RL Environment: A Day of Measured Iteration

An RL environment is only as useful as the decisions inside it. If every choice ends the same way, an agent can't learn from it and an evaluator can't tell good play from bad. So before asking what our Agent MMO could teach a model, we asked a blunter question: does anything an agent decides in it actually matter?

To answer it, we built a lab around the game and ran one improvement loop after another. This post is about the method as much as the results. Every number below comes from one results file, with its sample size and a 95% interval. It's a restatement of exploratory work, not a preregistered study, and the last section says what it doesn't show.

The loop

The lab plays the real game server with no database and no network. It runs the same world rules, adventure state machine and action menu that production uses, ticked by hand from a seed. It can play a few hundred adventures a second, and the same seed and choices always reproduce the same world, byte for byte. That reproducibility is what makes everything else possible.

Each round of the loop goes:

1. Change something: an adventure, a rule or the menu of actions an agent chooses from.
2. Play it with a ladder of scripted agents, from random choice up to our full-skill script, plus personalities like cautious, bold and greedy. This costs nothing.
3. Play it with real models the way production does. A strategist model writes a plan, and JEV, our fast decision model, picks each move from the menu. Every paid call reserves its worst-case cost against a hard cap first.
4. Label the decisions. Because the world is deterministic, we replay each adventure up to a decision, take every other option instead, and let the same scripted policy finish. That gives the counterfactual value of every alternative. A decision is consequential if some alternative would have moved the outcome by at least 5% of the scale.
5. Look at it in the real 3D client: we replay the adventure into the spectator view and screenshot the decision moments.
6. Judge, write it down, go again.

What the first measurements said

The original adventures barely had decisions in them. The full-skill script made a median of 3 idle decisions per production adventure (n = 60). A uniformly random agent still completed 96.7% of them (95% interval 88.6% to 99.1%, n = 60). When random play almost always succeeds, the data can't separate a good policy from a bad one.

Mistakes were free. In the trace of a random agent, the hero was knocked down, got up at camp at full health, and finished anyway. Being knocked down was effectively a free heal and a free trip home.

What we changed, and what moved

Put the risk next to the objective. We built new adventures from the existing game data, with no engine change: gather bones beside a skeleton camp, defeat three of five skeletons, and a daily survey whose landmarks are drawn from the date. The scripted agent's hunt carried more decisions than the production adventures (a median of 5 against 3). Across every model decision we later labelled, 36.3% were consequential (35.7% to 37.0%, n = 21,533).

Make mistakes cost something, as an option. With "a knockdown ends the adventure", a random agent completed the three-skeleton hunt 20% of the time (8.1% to 41.6%, n = 20), against 75% under the standard rule (53.1% to 88.8%, n = 20). The full-skill script completed it every time under both rules. That's the separation an evaluator wants. It's opt-in for players, because most people would rather their agent come home.

Tell the agent what the world already knows. Real models kept failing in one particular way: walking toward a landmark inside a camp, getting chased, fleeing, and walking back, until the adventure gave up. We changed only the text of the menu. Each "walk to" option now names the enemy camps the walk passes, using public map data. The test played real brains on the same 203 adventures with and without the notes. Those are the paired adventures from the five brains whose runs were clean (see the infrastructure note below):

• the share of adventures with a flee spiral (five or more retreats in a row) dropped by 7.4 points (3.4 to 11.8);
• the adventure score rose by 35.8 on average on a 0–1,000 scale (10.5 to 62.7).

We also gave the scripted fallback, which plays whenever the model is unavailable or unsure, a route planner that charges for walking past a camp. On the same 20 daily surveys its score rose by 14.5 (5.5 to 24.1).

The loop also found bugs in production

Some of the most valuable findings weren't about game design at all:

• Plans were being thrown away. The strategist model in our top tier wraps its JSON plan in a Markdown code block. Our parser rejected it, so 87.8% of its plans were discarded (85.4% to 89.9%, n = 814), and the agent played on without one. After the fix, none of 105 were rejected.
• Tier choices change with measurement. At the plan budget our top tier actually allows, GPT-6.1 Terra completed 97.2% of adventures (90.4% to 99.2%, n = 72), against 76.4% (65.4% to 84.7%, n = 72) for the model we were using. On the same adventures, GPT-6 Luna completed 12.5 points more than our mid-tier model (4.2 to 20.8, n = 72). Both tiers are moving to these models.
• Measurement infrastructure fails too. Our decision-model account ran out of credit partway through one comparison, and the affected adventures quietly fell back to the script. We caught it because every call is recorded, threw out the contaminated runs, and re-ran the analysis on clean pairs only. The lab now stops a batch when that happens.

The data it produces

The result is a dataset of 2,528 adventures and 21,533 real-model decisions. Each row holds:

• what the agent saw;
• every option it had, with the text it read;
• what it chose and which component decided;
• the counterfactual value of every alternative;
• how the adventure ended.

Every adventure in it replays to the same world. All of it comes from our own lab runs, not from players.

What this doesn't show

• Not preregistered. This is exploratory work, restated against our claim gate afterwards. Two things we hoped for didn't clear it: menu v5 raised completion by 2 points, but that interval crosses zero.
• No outside benchmark yet. Nothing here shows that training on these decisions makes a model better anywhere else. That's the next experiment: fine-tune a small open model on the consequential decisions and test it on BALROG, a public benchmark of long-horizon games, against the untrained model and a frontier model. It will be preregistered, with a spend cap, before it runs.
• Small world. One zone, one agent at a time, no other players.
• The counterfactual labels finish each alternative with a script. So they can't credit a choice whose payoff depends on the model's own later moves. We read them together with outcomes, never alone.

Why we work this way

A good RL environment isn't something you design once. It's tuned, like an instrument: you play it, measure whether the decisions inside it matter, and change it until they do. In one day the loop took our MMO from adventures that random play almost always completed to adventures where careless play fails under high stakes, and where about a third of model decisions change the outcome. Along the way it found a production bug that was discarding most of our top tier's plans. The same loop, a deterministic world, a ladder of agents, counterfactual labels and a hard spend cap, is how we build every environment.

If you build or evaluate agents and want the decision data, or an environment tuned the same way, talk to us.