Reasoning / HellaSwag

HellaSwag / sub-1B cohort

Scenario + four endings → most plausible ending

Measurements Evaluation protocol Planned runs Cost & information

Model size vs score

Favourable model size / score regionObserved Pareto frontier

Green uses cohort medians, not a universal quality threshold. Frontier points have no measured alternative with both smaller model size and higher score. Lines connect observed points; they do not predict intermediate models. Uncertainty is not included in frontier membership.

HuggingFaceTB/SmolLM2-360M-Instruct · 361.8M Qwen/Qwen2.5-0.5B-Instruct · 494.0M SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 149.3M SlayerLab/Slayer139-1.01b · 139.3M SlayerLab/Slayer149-ARC · 149.3M SlayerLab/Slayer149 · 148.9M SlayerLab/Slayer139-1.01 · 139.3M SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 149.3M SlayerLab/Slayer139-1.02 · 139.3M SlayerLab/Slayer-149M-Decision-v1 · 149.3M HuggingFaceTB/SmolLM2-135M-Instruct · 134.5M

11 compatible measurements. Parameter counts come from the architecture instantiated from each pinned model configuration; no weights are loaded for this count. Click a point to inspect its evidence.

Measured results

Pilot comparison · first 32 valid examples · accuracy higher is better. No global SOTA claim.

System	Score	Samples	Elapsed¹	Evidence
HuggingFaceTB/SmolLM2-360M-Instruct a10cc1512eab	46.9%	32	1s	Inspect run → 17 Oct 2026, 01:10 UTC
Qwen/Qwen2.5-0.5B-Instruct 7ae557604adf	40.6%	32	2s	Inspect run → 17 Oct 2026, 01:09 UTC
SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 e7d4b8dea441	37.5%	32	2s	Inspect run → 17 Oct 2026, 01:10 UTC
SlayerLab/Slayer139-1.01b 283ffb6d620b	37.5%	32	2s	Inspect run → 17 Oct 2026, 01:10 UTC
SlayerLab/Slayer149-ARC 0e6793ecfb85	34.4%	32	2s	Inspect run → 17 Oct 2026, 01:09 UTC
SlayerLab/Slayer149 7857dd28ec05	34.4%	32	2s	Inspect run → 17 Oct 2026, 01:11 UTC
SlayerLab/Slayer139-1.01 2e72c696d7c5	34.4%	32	2s	Inspect run → 17 Oct 2026, 01:08 UTC
SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 7d11e8fd6376	34.4%	32	2s	Inspect run → 17 Oct 2026, 01:11 UTC
SlayerLab/Slayer139-1.02 c4298d757ba3	34.4%	32	2s	Inspect run → 17 Oct 2026, 01:10 UTC
SlayerLab/Slayer-149M-Decision-v1 ccfd7d3d91f3	31.3%	32	2s	Inspect run → 17 Oct 2026, 01:09 UTC
HuggingFaceTB/SmolLM2-135M-Instruct 12fd25f77366	28.1%	32	1s	Inspect run → 17 Oct 2026, 01:10 UTC

¹ Evaluation elapsed time includes model loading after dataset selection. It is not serving latency or a fair speed comparison.

What this measures

Benchmark origin

HellaSwag ↗

Dataset / split

Rowan/hellaswag / validation

Immutable dataset revision

218ec52e09a7e7462a5400043bb9a69a41d06b76

Upstream license metadata

See upstream terms · consult upstream terms

Protocol

codesota-sub1b-choice-v1 / commonsense

Selection

First 32 valid examples in the pinned split order. SNLI rows without a valid label are excluded.

Configuration

Seed 42. Greedy generation for math and code. Mean answer-token log likelihood for text classification, completion and declarative multiple-choice recipes. No model judge.

Read scoring and comparison rules →

Planned runs

1 qualified visits in 30 days · priority 1007.0. Interest changes execution order, within the approved baseline list.

System	Status
SlayerLab/Slayer139-1.01	completed
SlayerLab/Slayer149-ARC	completed
Qwen/Qwen2.5-0.5B-Instruct	completed
SlayerLab/Slayer-149M-Decision-v1	completed
SlayerLab/Slayer139-1.01b	completed
SlayerLab/Slayer139-1.02	completed
SlayerLab/Slayer149-Balanced-ARC-E-ft-45500	completed
HuggingFaceTB/SmolLM2-360M-Instruct	completed
HuggingFaceTB/SmolLM2-135M-Instruct	completed
SlayerLab/Slayer149-Balanced-ARC-E-ft-v2	completed
SlayerLab/Slayer149	completed