Language / ARC-Challenge

ARC-Challenge / sub-1B cohort

Question + candidate answers → selected answer

Measurements Evaluation protocol Planned runs Cost & information

Model size vs score

Static, watermarked figures. SlayerLab models are red. The close-up expands the crowded small-model cluster without changing any coordinates.

ARC-Challenge: all measured models, with a dotted Pareto frontier and peach favourable region — Overview · Download PNG · Download SVG

ARC-Challenge: small-model close-up, with a dotted Pareto frontier and peach favourable region — Small-model close-up · Download PNG · Download SVG

Measured results

Pilot comparison · 32 examples · token-normalized choice accuracy (custom protocol) higher is better. No global SOTA claim.

Custom pilot: choices are ranked by mean token log likelihood. This differs from raw summed likelihood and character normalization used by reference harnesses. Do not compare this column with model-card scores. Full ARC-Easy with reference scoring →

System	Score	Samples	Elapsed¹	Evidence
Qwen/Qwen2.5-0.5B-Instruct 7ae557604adf	40.6%	32	2s	Inspect run → 17 Oct 2026, 02:49 UTC
SlayerLab/Slayer149-ARC 0e6793ecfb85	40.6%	32	2s	Inspect run → 17 Oct 2026, 02:49 UTC
HuggingFaceTB/SmolLM2-135M-Instruct 12fd25f77366	31.3%	32	1s	Inspect run → 17 Oct 2026, 02:48 UTC
SlayerLab/Slayer139-1.02 c4298d757ba3	31.3%	32	2s	Inspect run → 17 Oct 2026, 02:47 UTC
HuggingFaceTB/SmolLM2-360M-Instruct a10cc1512eab	28.1%	32	1s	Inspect run → 17 Oct 2026, 02:47 UTC
SlayerLab/Slayer149 7857dd28ec05	28.1%	32	2s	Inspect run → 17 Oct 2026, 02:48 UTC
SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 e7d4b8dea441	28.1%	32	2s	Inspect run → 17 Oct 2026, 02:49 UTC
SlayerLab/Slayer139-1.01 2e72c696d7c5	28.1%	32	2s	Inspect run → 17 Oct 2026, 02:48 UTC
SlayerLab/Slayer139-1.01b 283ffb6d620b	28.1%	32	2s	Inspect run → 17 Oct 2026, 02:47 UTC
SlayerLab/Slayer-149M-Decision-v1 ccfd7d3d91f3	28.1%	32	2s	Inspect run → 17 Oct 2026, 02:47 UTC
SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 7d11e8fd6376	25.0%	32	2s	Inspect run → 17 Oct 2026, 02:48 UTC

¹ Evaluation elapsed time includes model loading after dataset selection. It is not serving latency or a fair speed comparison.

What this measures

Benchmark origin

ARC-Challenge ↗

Dataset / split

allenai/ai2_arc / test

Immutable dataset revision

210d026faf9955653af8916fad021475a3f00453

Upstream license metadata

cc-by-sa-4.0 · consult upstream terms

Protocol

codesota-sub1b-choice-v1 / choice

Selection

First 32 valid examples in the pinned split order. SNLI rows without a valid label are excluded.

Configuration

Seed 42. Greedy generation for math and code. Mean answer-token log likelihood for text classification, completion and declarative multiple-choice recipes. No model judge.

Read scoring and comparison rules →

Planned runs

0 qualified visits in 30 days · priority 1000.1. Interest changes execution order, within the approved baseline list.

System	Status
SlayerLab/Slayer139-1.02	completed
HuggingFaceTB/SmolLM2-360M-Instruct	completed
SlayerLab/Slayer-149M-Decision-v1	completed
SlayerLab/Slayer139-1.01b	completed
HuggingFaceTB/SmolLM2-135M-Instruct	completed
SlayerLab/Slayer149	completed
SlayerLab/Slayer139-1.01	completed
SlayerLab/Slayer149-Balanced-ARC-E-ft-v2	completed
SlayerLab/Slayer149-ARC	completed
SlayerLab/Slayer149-Balanced-ARC-E-ft-45500	completed
Qwen/Qwen2.5-0.5B-Instruct	completed