Language / ARC-Challenge (full test, raw likelihood accuracy)
ARC-Challenge / full sub-1B common core (native)
Provided context/question + candidate answers → selected answer
Model size vs score
Static, watermarked figures. SlayerLab models are red. The close-up expands the crowded small-model cluster without changing any coordinates.
Measured results
Full pinned test split · 1,172 examples · accuracy higher is better. No global SOTA claim.
| System | Score | Samples | Elapsed¹ | Evidence |
|---|---|---|---|---|
| SlayerLab/Slayer149-ARC 0e6793ecfb85 | 33.9% | 1172 | 43s | Inspect run → 17 Oct 2026, 15:00 UTC |
| HuggingFaceTB/SmolLM2-360M-Instruct a10cc1512eab | 31.7% | 1172 | 15s | Inspect run → 17 Oct 2026, 15:10 UTC |
| Qwen/Qwen2.5-0.5B-Instruct 7ae557604adf | 30.7% | 1172 | 19s | Inspect run → 17 Oct 2026, 15:02 UTC |
| SlayerLab/Slayer139-1.02 c4298d757ba3 | 29.5% | 1172 | 38s | Inspect run → 17 Oct 2026, 15:11 UTC |
| SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 7d11e8fd6376 | 27.5% | 1172 | 43s | Inspect run → 17 Oct 2026, 15:25 UTC |
| SlayerLab/Slayer139-1.01 2e72c696d7c5 | 26.6% | 1172 | 37s | Inspect run → 17 Oct 2026, 15:10 UTC |
| SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 e7d4b8dea441 | 26.5% | 1172 | 44s | Inspect run → 17 Oct 2026, 15:02 UTC |
| SlayerLab/Slayer-149M-Decision-v1 ccfd7d3d91f3 | 26.1% | 1172 | 43s | Inspect run → 17 Oct 2026, 15:17 UTC |
| HuggingFaceTB/SmolLM2-135M-Instruct 12fd25f77366 | 25.9% | 1172 | 9s | Inspect run → 17 Oct 2026, 15:01 UTC |
| SlayerLab/Slayer149 7857dd28ec05 | 24.5% | 1172 | 41s | Inspect run → 17 Oct 2026, 15:02 UTC |
| SlayerLab/Slayer139-1.01b 283ffb6d620b | 22.6% | 1172 | 38s | Inspect run → 17 Oct 2026, 15:03 UTC |
¹ Evaluation elapsed time includes model loading after dataset selection. It is not serving latency or a fair speed comparison.
What this measures
Benchmark origin
Dataset / split
allenai/ai2_arc / test
Immutable dataset revision
210d026faf9955653af8916fad021475a3f00453Upstream license metadata
cc-by-sa-4.0 · consult upstream terms
Protocol
codesota-common-core-native-arc-challenge-v1 / choice
Selection
All 1172 examples in the pinned test split. SNLI rows without a valid label are excluded.
Configuration
Seed 42. Greedy generation for math and code. Summed conditional answer-token log likelihood; raw accuracy. Character-normalized accuracy is recorded separately. No model judge.
Planned runs
0 qualified visits in 30 days · priority 1000.0. Interest changes execution order, within the approved baseline list.
| System | Status |
|---|---|
| SlayerLab/Slayer149-ARC | completed |
| HuggingFaceTB/SmolLM2-135M-Instruct | completed |
| SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 | completed |
| SlayerLab/Slayer149 | completed |
| Qwen/Qwen2.5-0.5B-Instruct | completed |
| SlayerLab/Slayer139-1.01b | completed |
| HuggingFaceTB/SmolLM2-360M-Instruct | completed |
| SlayerLab/Slayer139-1.01 | completed |
| SlayerLab/Slayer139-1.02 | completed |
| SlayerLab/Slayer-149M-Decision-v1 | completed |
| SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 | completed |