Programs-of-Layers in LLMs through the Lens of Cortical Areas

With their prompt and without a chat template, the standard pass on our reconstruction of their test set matches their published numbers for LLaMA-3.2-3B and, on some levels, for Qwen1.5-MoE-A2.7B and Qwen3-8B, but on no level for Qwen2.5-3B. With the chat template their evaluation code applies, all three Qwen models fall below their numbers in every cell, so we run their split without it.

On every model and DART-Math level, accuracy rises from the standard pass to skip only, repeat only, and skip and repeat combined. This is the same ordering Li et al. report, with repeat the stronger of the two mechanisms on its own.

On 5 of our 6 models, most solved questions have a valid program shorter than the standard forward pass (50.5–91% per difficulty). On Qwen1.5-MoE-A2.7B, fewer than half do (39–45%).

On every model, the share of questions with at least one valid program rises monotonically as more recurrence steps are allowed. The share of solvable questions that need recurrence rises from the easiest to the hardest level on 5 of 6 models, and the share that needs skipping on 4 of them. The exception on both is LLaMA-3.2-3B.

Li et al. report that 57.7% of segments in valid programs touch a single layer. With our search on our split, 5 of our 6 models have fewer (21–38%); with their search on the PoLar split, all 4 models we ran have more (70–81%), likely because their search allows multi-layer moves only at the root. That segments are repeated at most once holds on all 6 models for the shortest valid program, in 74–86% of solved questions.

At K=100, the menu solves on average 72% of the questions, against 28% for identity. Apart from Qwen3-32B, whose standard pass often starts to reason and hits the 50-token limit before writing a box, no program with a single edit solves clearly more questions than identity: each fixes some questions but breaks about as many others. Since each fixes different questions, a menu of them together covers far more.

An LLM-assisted classifier rated how close each wrong answer is to the ground truth, for every DART-Math question identity answers wrong, across all 6 models and 5 levels. A wrong answer that is numerically or structurally close is rescued about 1.5× as often as an unrelated one, in every model.

We revert one edited segment of a rescuing program at a time back to keep and check whether the reward survives. The break rate is roughly flat across levels in five of six models and ranges from 44.5% (Qwen3-32B) to 66.8% (Qwen1.5-MoE-A2.7B). On 5 of 6 models, most edits inside a working program are individually essential.

On the PoLar split, the router gains only where polar_default breaks the output format (Qwen3-8B, Qwen2.5-3B), how much depends on the search seed, and a single fixed program gains more in every case. With oneshot, neither gains on any model. In our setup, the router’s top-1 decode is the literal identity program on every model, level and training-data treatment tested, also with Li et al.’s released router-training code.

The pass@5 gain does replicate in our setup, and there it is not a format repair: it comes from beam-search diversity. The router never places a correct non-identity program at rank 1, only somewhere in its top 5, and these top-5 predictions collapse to a fixed menu of few programs.

On Qwen3-8B, with answers scored by the logits of the option letters, we reshuffle which option sits behind which letter. Rescuing accuracy falls below identity’s own, while identity itself is close to shuffle-invariant. A dedicated MMLU-Pro router collapses to identity at pass@1 too, and its pass@5 gain is position overfitting: three of its five fixed programs answer a near-constant letter. Gains like this should be checked with robustness tests, such as reshuffling the answer options, before they are read as capability gains, especially if programs are deployed.