Inspect the measurement.

Qwen/Qwen2.5-0.5B-Instruct · ARC-Challenge (full test, raw likelihood accuracy) · 30.7% accuracy

Full benchmark split · 1172 scored examples. Full pinned split; primary raw summed conditional answer-token log likelihood accuracy; character-normalized accuracy is a separate diagnostic. Explicit pinned prompt schema, no chat/BOS. Left context truncated only if prompt plus answer exceeds 1024 tokens. Generic text diagnostic does not validate specialist declared tasks.

Run ID / completed

1d5a6e1c6de9a839eb75ee0c
17 Oct 2026, 15:02 UTC

Model revision

7ae557604adf67be50417f59c2c2f167def9a775

Comparison scope

e7984b7ea3a9cae0996b0a453de457a6a7d717e9a2c3e9a95d80f697f8bd5afb

Evaluator SHA-256

8ba81403e69d93b8121bd05e0ea4f9c55e0dc5c721449f50adfda136d7d976f1

Observed resources

19s elapsed · 2.45 GiB peak torch allocation
Dollar cost: not measured

Download evidence

result.json samples.jsonl environment.json evaluate.py manifest.json

Inspect individual examples

Read the recorded prediction, reference and outcome for each example.

Scored examples: 1,172
Passed: 360
Failed: 812
Metric: accuracy

Examples are downloaded only when requested. Search covers all loaded records; 25 examples are displayed per page. Download all examples · JSONL

Example evidence has not been loaded.