Language / ARC-Easy (full test split, raw accuracy)
ARC-Easy / full sub-1B standard evaluation
Question + candidate answers → selected answer
Model size vs score
Static, watermarked figures. SlayerLab models are red. The close-up expands the crowded small-model cluster without changing any coordinates.
Measured results
Full pinned test split · 2,376 examples · accuracy higher is better. No global SOTA claim.
| System | Score | Samples | Elapsed¹ | Evidence |
|---|---|---|---|---|
| SlayerLab/Slayer149-ARC 0e6793ecfb85 | 66.6% | 2376 | 84s | Inspect run → 17 Oct 2026, 02:57 UTC |
| SlayerLab/Slayer139-1.02 c4298d757ba3 | 65.9% | 2376 | 71s | Inspect run → 17 Oct 2026, 02:59 UTC |
| Qwen/Qwen2.5-0.5B-Instruct 7ae557604adf | 65.6% | 2376 | 30s | Inspect run → 17 Oct 2026, 02:56 UTC |
| SlayerLab/Slayer139-1.01 2e72c696d7c5 | 63.1% | 2376 | 71s | Inspect run → 17 Oct 2026, 02:56 UTC |
| SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 7d11e8fd6376 | 62.2% | 2376 | 83s | Inspect run → 17 Oct 2026, 02:55 UTC |
| SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 e7d4b8dea441 | 59.3% | 2376 | 84s | Inspect run → 17 Oct 2026, 02:59 UTC |
| SlayerLab/Slayer149 7857dd28ec05 | 56.8% | 2376 | 79s | Inspect run → 17 Oct 2026, 02:55 UTC |
| HuggingFaceTB/SmolLM2-360M-Instruct a10cc1512eab | 56.5% | 2376 | 24s | Inspect run → 17 Oct 2026, 03:01 UTC |
| SlayerLab/Slayer-149M-Decision-v1 ccfd7d3d91f3 | 56.4% | 2376 | 82s | Inspect run → 17 Oct 2026, 02:58 UTC |
| SlayerLab/Slayer139-1.01b 283ffb6d620b | 54.7% | 2376 | 72s | Inspect run → 17 Oct 2026, 03:02 UTC |
| HuggingFaceTB/SmolLM2-135M-Instruct 12fd25f77366 | 54.0% | 2376 | 13s | Inspect run → 17 Oct 2026, 03:00 UTC |
¹ Evaluation elapsed time includes model loading after dataset selection. It is not serving latency or a fair speed comparison.
What this measures
Benchmark origin
Dataset / split
allenai/ai2_arc / test
Immutable dataset revision
210d026faf9955653af8916fad021475a3f00453Upstream license metadata
cc-by-sa-4.0 · consult upstream terms
Protocol
codesota-sub1b-arc-easy-standard-v2 / choice
Selection
All 2376 examples in the pinned test split. SNLI rows without a valid label are excluded.
Configuration
Seed 42. Greedy generation for math and code. Summed conditional answer-token log likelihood; raw accuracy. Character-normalized accuracy is recorded separately. No model judge.
Planned runs
0 qualified visits in 30 days · priority 1000.0. Interest changes execution order, within the approved baseline list.
| System | Status |
|---|---|
| SlayerLab/Slayer149-ARC | completed |
| SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 | completed |
| SlayerLab/Slayer149 | completed |
| SlayerLab/Slayer139-1.01 | completed |
| Qwen/Qwen2.5-0.5B-Instruct | completed |
| SlayerLab/Slayer-149M-Decision-v1 | completed |
| SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 | completed |
| SlayerLab/Slayer139-1.02 | completed |
| HuggingFaceTB/SmolLM2-135M-Instruct | completed |
| HuggingFaceTB/SmolLM2-360M-Instruct | completed |
| SlayerLab/Slayer139-1.01b | completed |