Independent measured model card
Qwen/Qwen2.5-0.5B
Hugging Face source ↗ · JSON for agents
494.0M unique parameters (060db6499f32)
Every result links to its execution evidence. Compare scores only within matching protocols; no overall best-model claim. GPU-slot time includes evaluator setup and CPU work.
All current benchmark results
| Benchmark / coverage | Result | Version / protocol | GPU-slot time | Evidence |
|---|---|---|---|---|
| AI2 ARC-Challenge (test) subset · 32 items | 31.25% accuracy · max | 060db6499f32codesota-choice-pilot-v2-batched | 4.1s | Inspect run · Manifest |
| BoolQ subset · 32 items | 68.75% accuracy · max | 060db6499f32codesota-pilot-v2-batched | 7.4s | Inspect run · Manifest |
| GSM8K subset · 32 items | 3.12% exact_match · max | 060db6499f32codesota-pilot-v2-batched | 18.0s | Inspect run · Manifest |
| HellaSwag subset · 32 items | 37.50% accuracy · max | 060db6499f32codesota-pilot-v2-batched | 4.2s | Inspect run · Manifest |
| HumanEval subset · 32 items | 62.50% pass@1 · max | 060db6499f32codesota-pilot-v1 | Unknown | Inspect run · Manifest |
| HumanEval (full 164, serial greedy pass@1) full · 164 items | 28.66% pass@1 · max | 060db6499f32codesota-humaneval-full-v2 | 343.3s | Inspect run · Manifest |
| OpenBookQA main (test) subset · 32 items | 40.62% accuracy · max | 060db6499f32codesota-choice-pilot-v2-batched | 5.5s | Inspect run · Manifest |
| SNLI subset · 32 items | 37.50% accuracy · max | 060db6499f32codesota-pilot-v2-batched | 7.0s | Inspect run · Manifest |
| SST-2 subset · 32 items | 93.75% accuracy · max | 060db6499f32codesota-pilot-v2-batched | 8.0s | Inspect run · Manifest |
Archived results / other protocols
| Benchmark / coverage | Result | Version / protocol | GPU-slot time | Evidence |
|---|---|---|---|---|
| GSM8K subset · 32 items | 3.12% exact_match · max | 060db6499f32codesota-pilot-v1 | Unknown | Inspect run · Manifest |
| BoolQ subset · 32 items | 68.75% accuracy · max | 060db6499f32codesota-pilot-v1 | Unknown | Inspect run · Manifest |
| HellaSwag subset · 32 items | 37.50% accuracy · max | 060db6499f32codesota-pilot-v1 | Unknown | Inspect run · Manifest |
| SST-2 subset · 32 items | 93.75% accuracy · max | 060db6499f32codesota-pilot-v1 | Unknown | Inspect run · Manifest |
| SNLI subset · 32 items | 37.50% accuracy · max | 060db6499f32codesota-pilot-v1 | Unknown | Inspect run · Manifest |
| AI2 ARC-Challenge (test) subset · 32 items | 31.25% accuracy · max | 060db6499f32codesota-choice-pilot-v1 | Unknown | Inspect run · Manifest |
| OpenBookQA main (test) subset · 32 items | 40.62% accuracy · max | 060db6499f32codesota-choice-pilot-v1 | Unknown | Inspect run · Manifest |