Independent measured model card
Qwen/Qwen2.5-1.5B
Hugging Face source ↗ · JSON for agents
1,543.7M unique parameters (8faed761d45a)
Every result links to its execution evidence. Compare scores only within matching protocols; no overall best-model claim. GPU-slot time includes evaluator setup and CPU work.
All current benchmark results
| Benchmark / coverage | Result | Version / protocol | GPU-slot time | Evidence |
|---|---|---|---|---|
| AI2 ARC-Challenge (test) subset · 32 items | 46.88% accuracy · max | 8faed761d45acodesota-choice-pilot-v2-batched | 5.0s | Inspect run · Manifest |
| BoolQ subset · 32 items | 68.75% accuracy · max | 8faed761d45acodesota-pilot-v2-batched | 7.8s | Inspect run · Manifest |
| GSM8K subset · 32 items | 9.38% exact_match · max | 8faed761d45acodesota-pilot-v2-batched | 20.4s | Inspect run · Manifest |
| HellaSwag subset · 32 items | 50.00% accuracy · max | 8faed761d45acodesota-pilot-v2-batched | 4.8s | Inspect run · Manifest |
| HumanEval subset · 32 items | 75.00% pass@1 · max | 8faed761d45acodesota-pilot-v1 | Unknown | Inspect run · Manifest |
| HumanEval (full 164, serial greedy pass@1) full · 164 items | 35.98% pass@1 · max | 8faed761d45acodesota-humaneval-full-v2 | 518.3s | Inspect run · Manifest |
| OpenBookQA main (test) subset · 32 items | 40.62% accuracy · max | 8faed761d45acodesota-choice-pilot-v2-batched | 4.8s | Inspect run · Manifest |
| SNLI subset · 32 items | 37.50% accuracy · max | 8faed761d45acodesota-pilot-v2-batched | 7.4s | Inspect run · Manifest |
| SST-2 subset · 32 items | 62.50% accuracy · max | 8faed761d45acodesota-pilot-v2-batched | 7.2s | Inspect run · Manifest |
Archived results / other protocols
| Benchmark / coverage | Result | Version / protocol | GPU-slot time | Evidence |
|---|---|---|---|---|
| GSM8K subset · 32 items | 9.38% exact_match · max | 8faed761d45acodesota-pilot-v1 | Unknown | Inspect run · Manifest |
| GSM8K subset · 32 items | 9.38% exact_match · max | 8faed761d45acodesota-pilot-v1 | Unknown | Inspect run · Manifest |
| BoolQ subset · 32 items | 68.75% accuracy · max | 8faed761d45acodesota-pilot-v1 | Unknown | Inspect run · Manifest |
| HellaSwag subset · 32 items | 50.00% accuracy · max | 8faed761d45acodesota-pilot-v1 | Unknown | Inspect run · Manifest |
| SST-2 subset · 32 items | 62.50% accuracy · max | 8faed761d45acodesota-pilot-v1 | Unknown | Inspect run · Manifest |
| SNLI subset · 32 items | 37.50% accuracy · max | 8faed761d45acodesota-pilot-v1 | Unknown | Inspect run · Manifest |
| AI2 ARC-Challenge (test) subset · 32 items | 46.88% accuracy · max | 8faed761d45acodesota-choice-pilot-v1 | Unknown | Inspect run · Manifest |
| OpenBookQA main (test) subset · 32 items | 40.62% accuracy · max | 8faed761d45acodesota-choice-pilot-v1 | Unknown | Inspect run · Manifest |