Language / BoolQ (full validation, raw likelihood accuracy)
BoolQ / full sub-1B common core (native)
Provided context/question + candidate answers → selected answer
Model size vs score
Static, watermarked figures. SlayerLab models are red. The close-up expands the crowded small-model cluster without changing any coordinates.
Measured results
Full pinned test split · 3,270 examples · accuracy higher is better. No global SOTA claim.
| System | Score | Samples | Elapsed¹ | Evidence |
|---|---|---|---|---|
| Qwen/Qwen2.5-0.5B-Instruct 7ae557604adf | 68.0% | 3270 | 84s | Inspect run → 17 Oct 2026, 15:40 UTC |
| SlayerLab/Slayer139-1.01b 283ffb6d620b | 60.2% | 3270 | 60s | Inspect run → 17 Oct 2026, 15:46 UTC |
| SlayerLab/Slayer149 7857dd28ec05 | 60.0% | 3270 | 74s | Inspect run → 17 Oct 2026, 15:42 UTC |
| SlayerLab/Slayer-149M-Decision-v1 ccfd7d3d91f3 | 59.9% | 3270 | 75s | Inspect run → 17 Oct 2026, 15:43 UTC |
| SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 e7d4b8dea441 | 59.4% | 3270 | 75s | Inspect run → 17 Oct 2026, 15:38 UTC |
| SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 7d11e8fd6376 | 57.6% | 3270 | 71s | Inspect run → 17 Oct 2026, 16:02 UTC |
| SlayerLab/Slayer149-ARC 0e6793ecfb85 | 55.9% | 3270 | 75s | Inspect run → 17 Oct 2026, 15:24 UTC |
| SlayerLab/Slayer139-1.02 c4298d757ba3 | 54.4% | 3270 | 64s | Inspect run → 17 Oct 2026, 15:45 UTC |
| SlayerLab/Slayer139-1.01 2e72c696d7c5 | 53.6% | 3270 | 60s | Inspect run → 17 Oct 2026, 15:44 UTC |
| HuggingFaceTB/SmolLM2-135M-Instruct 12fd25f77366 | 43.4% | 3270 | 29s | Inspect run → 17 Oct 2026, 15:41 UTC |
| HuggingFaceTB/SmolLM2-360M-Instruct a10cc1512eab | 42.2% | 3270 | 69s | Inspect run → 17 Oct 2026, 15:43 UTC |
¹ Evaluation elapsed time includes model loading after dataset selection. It is not serving latency or a fair speed comparison.
What this measures
Benchmark origin
Dataset / split
aps/super_glue / validation
Immutable dataset revision
3de24cf8022e94f4ee4b9d55a6f539891524d646Upstream license metadata
other · consult upstream terms
Protocol
codesota-common-core-native-boolq-lm-eval-v2 / choice
Selection
All 3270 examples in the pinned test split. SNLI rows without a valid label are excluded.
Configuration
Seed 42. Greedy generation for math and code. Summed conditional answer-token log likelihood; raw accuracy. Character-normalized accuracy is recorded separately. No model judge.
Planned runs
0 qualified visits in 30 days · priority 1000.0. Interest changes execution order, within the approved baseline list.
| System | Status |
|---|---|
| SlayerLab/Slayer149-ARC | completed |
| SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 | completed |
| Qwen/Qwen2.5-0.5B-Instruct | completed |
| SlayerLab/Slayer149 | completed |
| HuggingFaceTB/SmolLM2-135M-Instruct | completed |
| HuggingFaceTB/SmolLM2-360M-Instruct | completed |
| SlayerLab/Slayer-149M-Decision-v1 | completed |
| SlayerLab/Slayer139-1.01 | completed |
| SlayerLab/Slayer139-1.02 | completed |
| SlayerLab/Slayer139-1.01b | completed |
| SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 | completed |