Small enough to run.
Useful enough to choose.

Why this benchmark exists

Measure behaviour that survives a change in phrasing.

Public questions about world knowledge are easy to optimise against and only indirectly describe daily use. Our next step is newly generated, checkable tasks: extraction, classification with supplied labels, constrained rewriting, context-only QA, format following and autocomplete. Freshly generated items reduce memorisation of fixed questions; generators can still be gamed.

One task, several ways to fail

Illustrative design example; not a measured test item. A fictional product email lists three prices, two dates and a cancellation clause. Ask for the cancellation deadline and total price as JSON. Generate a paraphrase, reorder fields, add a distractor date, and remove the answer in another variant.

Check valid JSON, exact values and null when the deadline is absent. The robust item score passes only when every variant passes. Report ordinary correctness alongside this stricter all-variants score, so users can see the source of a failure.

Fresh text is the first axis, not the whole answer.

Live BPB measures how efficiently a model predicts a fixed sequence of recent text. We use byte-weighted loss so different tokenizers can be compared on identical raw text. Recent publication dates reduce exposure to old public test sets; they do not prove that a model has never seen a document. This pilot covers one domain and only 20 documents.

The methodological reference is UncheatableEval. Its benchmark targets base models and uses multiple fresh-text sources. This CodeSOTA pilot evaluates instruct and raw-base models on raw text, in separately labelled sections with a separate, narrower protocol. Scores are not official UncheatableEval results or comparable with its leaderboard. Its evaluator currently restricts batching to one; our batch-four implementation is independently versioned. BOS handling and context policy also differ and are recorded in every run.

Measure the experience, then validate it.

The next behaviour diagnostics will track repetition after 200–500 generated tokens, completion stopping and abstention when a context lacks the answer. Staying on topic and following subjective tone requests need stronger validation than exact matching. A small-model, blinded pairwise arena comes last: randomise sides, record ties and control for response length and formatting.

Automatic components earn their place by predicting held-out human preferences beyond simple baselines. Until that validation exists, this page makes no “pleasantness” ranking.

Make over-optimisation visible.

Publish generators and a development set, keep evaluation seeds private, rotate them quarterly, then retire and publish old seeds for audit. Display the dev–private gap with uncertainty. A large gap is a diagnostic signal, not proof of cheating. These controls are planned; the current text corpus is identified publicly by source, date and fingerprint.

Report local model separation, paired uncertainty and evaluation cost for each component. Drop or revise components that fail to separate the relevant cohort. Eleven models and 20 abstracts are a pilot, not enough evidence to fit a reliable IRT model or establish broad ranking stability.