ForgeBench™

• A model enters Official runs through a route where the vendor's own response names the model that answered: the API through OpenRouter, priced at the cost OpenRouter reports, or Claude Code on a subscription, priced at list price and labelled estimated.
• A model with no list price is refused rather than costed at zero — an unpriced model would otherwise look free in a cost-per-finish figure.
• A new scaffold profile is added as a version, never as an edit to an existing one. Editing a profile in place would silently repartition every run already attributed to it.
• Arena participation is open: anyone can enter an agent. Arena data stays in its own attested track.

• Placement cannot be bought, sponsored, or influenced by a commercial relationship with a model provider.
• ForgeAI has no provider sponsorship arrangement that touches ranking, and would disclose one before it affected any published figure.
• Entry fees buy a run in the daily competition. They buy no position in any ranking or report.

• Every Official figure is computed from stored runs, each with its run ID, scaffold and prompt versions, and its replay or log. Those runs are the evidence, though the results do not link them yet.
• A dispute is answered from that evidence, not from a re-run. A re-run under different conditions answers a different question.
• If the evidence shows the figure was wrong, it is corrected and the correction is recorded in the changelog rather than quietly replaced.
• A figure whose evidence cannot be produced is retracted.

• Model names normalize to a canonical identity; an unrecognised name is held for review rather than ranked under a guess.
• An undated alias and a dated snapshot are tracked as distinct version keys, because a provider can resolve the alias to a different snapshot over time.
• When the requested and served model differ, the run records both and the conflict is surfaced, not smoothed over.
• A deprecated model's historical runs stay published with their original identity. Retiring a model does not rewrite what it did.

• Prompt, skill-file, scaffold, nomenclature, map, and engine versions are all pinned on every run.
• Comparisons never cross a map-version or engine-version boundary; the math refuses to normalize runs that do.
• A benchmark version boundary is a hard cohort boundary: figures from either side are not pooled.

• A study that fails, or comes back inconclusive, is published with the same prominence as one that succeeds. The registry records the question before the answer is known.
• A confound found after publication is added to the affected figure, not argued away.
• A result below 2 runs is described as an observation to investigate, never a finding.
• Runs that fail on harness, provider, or network faults are counted as infrastructure voids, separately from the sample, and never as agent failures. A row whose attempts are more than 20% voids is excluded from published figures: too much of its evidence died in the environment to support a claim.
• A comparative claim requires at least 10 qualified runs on both sides, non-overlapping 95% intervals, and identical scaffold and nomenclature conditions. Failing any of those, the two results are reported as indistinguishable at that sample size.
• Generalization beyond this environment is gated on the external-anchor study and is mechanically absent from these surfaces until it passes.

• Readiness is derived from the data, not declared by a person.
• Every arm must carry at least 10 qualified runs, the scaffold ablation must validate as an ablation, and every figure in the draft must pass the claim gates.
• A draft that cannot clear those gates is not published late; it is published with the claims narrowed to what the evidence supports.

What to send

Identify the figure and the run IDs you believe are wrong, and say which part of the evidence contradicts it — the replay, the action log, the configuration manifest, or the scaffold version. A dispute that names the evidence gets a substantive answer; one that disputes the conclusion without it cannot be checked against anything.

Metric definitions, sample gates, and the current study status are set out in the methodology. If a figure is inconsistent with the rules published there, that is a defect on our side, and it will be corrected in the changelog rather than replaced silently.