Measurement validity audit
ARC pilot ≠ full ARC accuracy.
For the exact Slayer149-ARC revision, two things changed: answer-scoring rule and evaluation coverage. They must not be explained away as sampling noise.
| Protocol | Correct / items | Accuracy |
|---|---|---|
| Historical pilot: mean answer-token log likelihood, first32 | 25 /32 | 78.125% |
| Same32 items: raw summed answer log likelihood | 22 /32 | 68.75% |
| Full pinned ARC-Easy: raw summed likelihood | 1582 /2376 | 66.58249% |
Nine predictions change when the scoring rule changes. All32 pilot choices are reproduced by dividing full-run answer log-likelihood sums by answer-token counts; maximum difference in the stored mean values is 3.18×10⁻⁷. Item IDs, gold labels and candidate lists align.
The first32 examples are a fixed prefix, not a representative random sample. A confidence interval does not correct selection bias or a changed metric. Artifact hashes and arithmetic consistency do not establish official scorer parity.
Historical pilot evidence · Full compatible ARC-Easy run
Enforced interpretation
- The pilot remains immutable historical evidence with a prominent custom-protocol label.
- The standard aggregate requires complete full-split, current-scope results and excludes this pilot.
- The critical agent reviews metric, sampling and task fit, alongside deterministic admission checks.