Inspect the measurement.

mithulaartigala/reasoning-summarizer-qwen3.5-0.8b-preview · HellaSwag (full validation, character-normalized accuracy) · 51.9% accuracy_char_normalized

Full benchmark split · 10042 scored examples. Full pinned split; primary character-normalized accuracy: argmax summed conditional answer-token log likelihood divided by Python len(choice), excluding the leading delimiter; raw accuracy is a separate diagnostic. Explicit pinned prompt schema, no chat/BOS. Left context truncated only if prompt plus answer exceeds 1024 tokens. Generic text diagnostic does not validate specialist declared tasks.

Run ID / completed

1ad1a5a63d273c0c9f35fb2e
17 Oct 2026, 16:25 UTC

Model revision

558fc651bf01bbe7cf306ebf2fb32032d7b4e818

Comparison scope

e83048a65cc6e110ed09bc40c410945ce72105f15f101b049aa100bf096278dd

Evaluator SHA-256

eb01abebdaa64a9d8003d37b2006b198582b99f61377a6fe6d0645db15934307

Observed resources

590s elapsed · 5.18 GiB peak torch allocation
Dollar cost: not measured

Download evidence

result.json samples.jsonl environment.json evaluate.py manifest.json

Inspect individual examples

Read the recorded prediction, reference and outcome for each example.

Scored examples: 10,042
Passed: 5,208
Failed: 4,834
Metric: accuracy_char_normalized

Examples are downloaded only when requested. Search covers all loaded records; 25 examples are displayed per page. Download all examples · JSONL

Example evidence has not been loaded.