Most planning failures aren't a reasoning problem — they're a missing-context problem.

Overview

Large language models are surprisingly good at sounding like they can plan a multi-step task. Give one a goal — “craft an iron pickaxe” — and it will narrate a plausible sequence of steps. The trouble starts when you actually run it: somewhere around step three or four, it quietly asks for an item it never made, and the whole plan collapses.

I spent six months trying to find out why. The answer wasn’t what I expected, and neither was the second half of it.

The problem: a missing link, not a missing brain

A crafting task is really a dependency chain. You can’t make planks until you’ve chopped a log; you can’t make a crafting table until you have planks; and so on, all the way to iron. The cover diagram above traces one such chain.

We sort tasks by how deep that chain runs. L1 is one hop. L2 is the multi-hop case — a goal that requires a tool that requires another tool first. That’s where models fall off a cliff. Given no retrieved examples at all, all five models of the primary study land at 10–30% on L2 while clearing 45–95% on L1. GPT-4o is the sharpest illustration — 85% of L1 tasks, 15% of L2, same model, same prompt, one extra hop. The wall holds from 8B to 671B parameters and across four architectural families, which is what makes it look structural rather than incidental.

That gap doesn’t look like a reasoning deficit, because the same model reasons fine one hop earlier. It looks like the prerequisite simply isn’t in context at the moment the model has to commit to a next action. The paper calls it a procedural-context bottleneck: planning fails not at the reasoning step, but at the retrieval step that feeds it.

Two open-weight models sit outside this pattern, and I want to be exact about how much that costs the claim. A Gemma4 pair scores 60–65% on L2 with no retrieval — measured under the same cold-start protocol as the five above, so that number is directly comparable, and the wall genuinely is not universal. What is not comparable is their SAER condition, which ran on a later warm-start pipeline; that is why the pair is reported as a replication panel and excluded from the significance family. The honest summary is that the bottleneck holds across every model in the primary study and two open-weight models clear it unaided.

What I built: SAER

SAER — State-Aware Example Retrieval — picks which worked examples go into the prompt by what the task actually requires, instead of returning whatever is most textually similar. It scores each candidate example on four dimensions, and ships two weight configurations:

Dimension	Default	Cold-start (benchmark)	What it captures
Ingredient overlap	0.40	0.50	how much this example’s recipe shares materials with the goal
Keyword overlap	0.30	0.35	plain token overlap between the request and the example
Mining-tier gap	0.20	0.00	whether the pickaxe the agent has can mine what this example needs
Level proximity	0.10	0.15	is this example at a comparable difficulty

Here is the part I have to state plainly, because the method’s name invites the opposite reading: only one of the four dimensions reads the agent’s live state, and the benchmark runs it at zero weight. Every task in the evaluation starts from an empty inventory, so the mining-tier gap has nothing to discriminate on — a weight it earns back only in mid-task replanning, which this study does not evaluate. The reported gains come from the three state-independent dimensions, chiefly ingredient overlap resolved from the game’s recipe graph.

The evaluation harness

“It works better” proves nothing. I built an evaluation that could tell a real effect from noise: 60 held-out tasks × 7 models × 3 seeds, with a task counted as passed only if it passes on the majority of seeds, and every model run under both conditions — with retrieval and without. The seven are the five primary models plus the two-model replication panel above. Three more cross-family sub-frontier models — Llama-3.1-8B, Ministral-8B, Gemma-3-12B — run in the supplement, to check whether the small-model residual is about size or about being a Qwen.

Significance is tested with an exact-binomial McNemar test on paired outcomes (the same model attempting the same task with and without SAER). The p-values are Bonferroni-corrected across a 13-test family that was fixed before any result was read, so no comparison could be added or dropped once I knew which way it went.

Results

Model	No retrieval	With SAER	Change
Qwen3-8B	23%	43%	+20pp (p = 0.0075, nominal only)
GPT-4o	38%	68%	+30pp (p < 0.001)
Claude Sonnet 4.5	42%	85%	+43pp (p < 0.001)
Qwen2.5-Max	32%	82%	+50pp (p < 0.001)
DeepSeek-V3	55%	87%	+32pp (p < 0.001)
Gemma4-26B †	73%	80%	+7pp (n.s.)
Gemma4-31B †	77%	85%	+8pp (n.s.)

† Replication panel. The no-retrieval column uses the same cold-start protocol as the five above; the SAER column ran on a later warm-start pipeline, so the change and its p-value are not comparable and are excluded from the Bonferroni family. Both models already clear the L2 wall unaided, leaving little for retrieval to recover.

The finding I didn’t expect

All five primary models improve significantly, including the 8B one. The surprise is where the improvement comes from — and it is not the part I designed.

I ran a four-condition ablation on Qwen2.5-Max to separate two things I had been treating as one: whether a worked example is present at all (existence), and whether my scorer picks a better example than a plain lexical baseline (relevance). The split is lopsided:

Condition	Tasks passed (/60)	Gain
No retrieval	19	—
Three examples at random	38	+31pp
BM25 (lexical baseline)	42	+38pp
SAER	49	+50pp

The single largest step is the first one — three examples drawn uniformly at random from the right task category, with no scoring of any kind, buys 31 of the eventual 50 points. Ranking them well adds the rest. SAER against BM25 comes out at b = 7, c = 0, p = 0.0156: nominally significant, above the corrected threshold, higher on four of the five models and one task lower on the fifth. The paper reports that as a consistent but individually underpowered increment, not as a win. One more caveat belongs here rather than in a footnote: the random condition shares SAER’s JSON schema, so part of that first jump may be format exposure rather than knowledge.

So the honest version is: the bottleneck is real, and the thing that fixes most of it is having the example at all. My scorer is the instrument that demonstrated the diagnosis and it earns a real increment on top — but it is not where the bulk of the gain lives, and writing it up that way was a decision I had to make against my own interest.

The other thing I did not get to claim: gain size does not track model size. DeepSeek-V3 at 671B gains +32pp, less than Qwen2.5-Max’s +50pp. The smallest model does gain least, and three cross-family sub-frontier models confirm a real ceiling below the frontier — but the ordering in between is not monotone in parameter count, so the paper makes no causal capacity-scaling claim, and neither will I here.

Why this matters beyond Minecraft

Minecraft is a sandbox, but the failure mode is general. Any agent that plans over multiple steps — a coding agent, a research assistant, a robot — can only act on what’s in its context. The practical reading is cheaper than the one I set out to prove: before tuning how you rank retrieved context, check whether the relevant thing is reaching the model at all. That first check was worth 31 of the 50 points here, and it costs nothing to run.

Skills & tools

JavaScript / Node.js · multi-provider LLM clients over the OpenAI SDK · Mineflayer · experimental design · statistics (McNemar, Bonferroni, ablation). The paper is under review; the repository will be linked here once it’s public.