Brief

Self-improving (SI) agent papers promise systems that get better at getting better. That claim is only as strong as the evaluation that carries it. This note reports a methods audit of forty-one such papers under a five-trap checklist that the field already knows how to apply: held-out splits, correctness-as-gate, length-matched text controls, budget-matched baselines, and multi-seed variance. The primary result is stark. Under a pre-registered aggregation, 39/41 papers (95.1%) fail at least two traps (ge2). Ignoring the length trap still leaves 38/41 (92.7%). We pre-registered that the main claim holds if ge2 exceeds 50%, and falsifies if ge2 is at or below 20%. The observed rate clears the hold bar by a wide margin.

This is Lionel Arce / Seriora Research work: a population audit of SI-agent papers, not a new equal-budget or reflection experiment. Token Economies (arXiv 2406.06461) and Sample More Reflect Less (arXiv 2607.28576) settle different questions about compute and reflection under matched budgets. We ask whether the SI literature, as published, survives a multi-trap checklist. The paper holds the matrix. This post clarifies the traps, states the locked claim, and says what the number does and does not mean for harness-level self-improvement eval discipline.

Hook

If you work on agents, you have seen the slide: a loop that writes, tests, revises prompts or tools or memory, and claims an upward curve. The curve is often real as a trajectory plot. The question is whether the plot survives the same eval hygiene we already demand for ordinary ML papers.

Held-out test sets. Controls for length and verbosity. Baselines that spend the same budget. Variance across seeds. Gates that do not smuggle labeled correctness into the update rule. None of this is exotic. It is the boring checklist that separates a protocol claim from a demo.

We asked a blunt population question: how many SI-agent papers in a fixed corpus fail those traps? Not "do SI methods work in some setting," and not "are reported gains fake." Whether gains survive under stricter protocols is a later question. First you need to know whether the published evaluations already close the traps they should already know.

What this is (and is not)

What it is. A methods audit. We coded forty-one papers against five traps, with dual machine-assisted passes and a human spot-check on 8/41 papers (35/40 cells in that subsample). We disclose that 33/41 were not human-reread cell-by-cell. The unit of analysis is the paper's evaluation protocol as described, not a reimplementation of every system. The primary metric is ge2: failing at least two traps. Pre-registration fixed the hold/falsify thresholds before we locked the headline rate.

What it is not. It is not a claim that self-improvement gains are null. Papers can fail traps and still contain useful systems, genuine engineering insight, or gains that would hold under cleaner protocols. We do not re-run every method under a gold harness. We do not settle equal-budget or reflection debates. Token Economies and Sample More Reflect Less address those axes. Our object is different: SI-agent papers as a population under a multi-trap checklist.

Scope note. Two papers clear ge2 = 0 in our coding: Phantom Guardrails (arXiv 2607.13083) and Auditing Harness Tampering (arXiv 2609.00069). They are the exceptions that show the checklist is passable, not a gotcha invented after the fact.

The five traps

1. Held-out. Does the evaluation keep a true held-out set for the claim being made, or does improvement bleed into the same slice used to judge success? 2. Correctness-as-gate / oracle-free update gate. Is labeled correctness used as a weighted objective or as a model-judge gate that steers the update? The trap is an update gate that is not oracle-free in the sense the paper's claim requires. 3. Length-matched text control. When the intervention produces more text, is there a length-matched control so "more tokens" is not mistaken for "better method"? 4. Budget-matched baseline. Does the baseline get a comparable compute, call, or token budget, or does the SI loop simply buy more attempts? 5. Multi-seed variance. Are results reported with enough seed or run diversity to separate a method effect from a lucky draw?

These traps are short for a reason. The preprint expands operational definitions and edge cases. The post only needs the checklist you would put on a whiteboard before reading a methods section.

Locked result

Under the locked coding, ge2 = 39/41 = 95.1%. The pre-registered rule was: the claim holds if ge2 > 50%, and falsifies if ge2 ≤ 20%. We are deep in hold territory.

Sensitivity that drops the length trap still yields 38/41 = 92.7% on ignore-length ge2. Per-trap fail rates in the audit:

- Held-out, correctness-as-gate, and budget-matched baseline: each 26/41 = 63.4% fail among the full set (same count across those three in the locked table). - Length-matched text control: 28/28 = 100% fail among applicable papers (13 NA where length was not an applicable axis). - Multi-seed variance: 35/41 = 85.4% fail.

Only Phantom Guardrails and Auditing Harness Tampering sit at ge2 = 0. Everyone else fails at least two traps under the checklist.

Coding process, again for transparency: dual machine-assisted passes, human spot-check on 8/41 (35/40 cells), and an explicit disclose that 33/41 were not human-reread cell-by-cell. Disagreement resolution and cell definitions live in the preprint matrix, not here.

What the number means

Read 95.1% as a statement about protocol claims, not as a null result for self-improvement. The literature often asserts that a loop improved a system. That assertion is a protocol claim: under such-and-such evaluation, the method beat a baseline. If the evaluation leaves held-out porous, gates on labeled correctness, skips length and budget controls, or reports a single seed, the protocol claim is under-specified relative to what the field already knows how to demand.

For harness-level self-improvement, the practical bridge is discipline, not despair. If your loop edits prompts, tools, memory, or routing, you still owe held-out judgment, an update gate that matches what you claim (especially when correctness labels would make the story circular), length and budget matched controls when those are confounds, and variance that survives a second seed. The same checklist that ordinary supervised papers are expected to respect applies when the learner is a harness loop rather than a gradient step.

None of that requires claiming SI is empty. It requires treating SI-agent papers as empirics that can fail known traps. Population failure rates this high are a methods signal: the default published evaluation is not yet the evaluation the claim needs. Cleaner protocols may preserve gains, shrink them, or relocate them. That is an open empirical question. Closing the traps is how you make that question answerable.

Close

The locked claim stands: most self-improving agent papers in this corpus fail the eval traps they should already know. Primary ge2 is 39/41 (95.1%), above the pre-registered hold threshold (>50%) and nowhere near falsify (≤20%). Ignore-length remains 38/41 (92.7%). Two papers clear all five traps in our coding. The rest do not.

The full matrix, operational definitions, and coding notes are in the Seriora Research preprint (linked via the PDF attachment on this note). arXiv posting later. Until then, treat this note as the short claim sheet: traps, numbers, and the scope line that SI gains are not hereby declared null.