Language / HellaSwag (full validation, character-normalized accuracy)
HellaSwag / full sub-1B common core (native)
Provided context/question + candidate answers → selected answer
Model size vs score
Static, watermarked figures. SlayerLab models are red. The close-up expands the crowded small-model cluster without changing any coordinates.
Measured results
Full pinned test split · 10,042 examples · accuracy_char_normalized higher is better. No global SOTA claim.
| System | Score | Samples | Elapsed¹ | Evidence |
|---|---|---|---|---|
| HuggingFaceTB/SmolLM2-360M-Instruct a10cc1512eab | 56.8% | 10042 | 242s | Inspect run → 17 Oct 2026, 15:28 UTC |
| Qwen/Qwen2.5-0.5B-Instruct 7ae557604adf | 52.4% | 10042 | 328s | Inspect run → 17 Oct 2026, 15:23 UTC |
| HuggingFaceTB/SmolLM2-135M-Instruct 12fd25f77366 | 42.9% | 10042 | 103s | Inspect run → 17 Oct 2026, 15:10 UTC |
| SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 7d11e8fd6376 | 40.4% | 10042 | 360s | Inspect run → 17 Oct 2026, 15:40 UTC |
| SlayerLab/Slayer149-ARC 0e6793ecfb85 | 40.1% | 10042 | 364s | Inspect run → 17 Oct 2026, 15:09 UTC |
| SlayerLab/Slayer-149M-Decision-v1 ccfd7d3d91f3 | 40.0% | 10042 | 358s | Inspect run → 17 Oct 2026, 15:34 UTC |
| SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 e7d4b8dea441 | 38.6% | 10042 | 370s | Inspect run → 17 Oct 2026, 15:17 UTC |
| SlayerLab/Slayer139-1.02 c4298d757ba3 | 37.2% | 10042 | 321s | Inspect run → 17 Oct 2026, 15:31 UTC |
| SlayerLab/Slayer139-1.01 2e72c696d7c5 | 37.0% | 10042 | 313s | Inspect run → 17 Oct 2026, 15:16 UTC |
| SlayerLab/Slayer149 7857dd28ec05 | 36.5% | 10042 | 347s | Inspect run → 17 Oct 2026, 15:23 UTC |
| SlayerLab/Slayer139-1.01b 283ffb6d620b | 36.3% | 10042 | 323s | Inspect run → 17 Oct 2026, 15:37 UTC |
¹ Evaluation elapsed time includes model loading after dataset selection. It is not serving latency or a fair speed comparison.
What this measures
Benchmark origin
Dataset / split
Rowan/hellaswag / validation
Immutable dataset revision
218ec52e09a7e7462a5400043bb9a69a41d06b76Upstream license metadata
unspecified · consult upstream terms
Protocol
codesota-common-core-native-hellaswag-lm-eval-v2 / choice
Selection
All 10042 examples in the pinned test split. SNLI rows without a valid label are excluded.
Configuration
Seed 42. Greedy generation for math and code. Mean answer-token log likelihood for text classification, completion and declarative multiple-choice recipes. No model judge.
Planned runs
0 qualified visits in 30 days · priority 1006.6. Interest changes execution order, within the approved baseline list.
| System | Status |
|---|---|
| HuggingFaceTB/SmolLM2-135M-Instruct | completed Run needs operator attention. |
| SlayerLab/Slayer149-ARC | completed |
| SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 | completed |
| SlayerLab/Slayer139-1.01 | completed |
| Qwen/Qwen2.5-0.5B-Instruct | completed |
| SlayerLab/Slayer149 | completed |
| HuggingFaceTB/SmolLM2-360M-Instruct | completed |
| SlayerLab/Slayer139-1.02 | completed |
| SlayerLab/Slayer-149M-Decision-v1 | completed |
| SlayerLab/Slayer139-1.01b | completed |
| SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 | completed |