Code / HumanEval (full 164, serial greedy pass@1)
HumanEval / full native and HF code comparison
Function signature + docstring → Python function
Model size vs score
Static, watermarked figures. SlayerLab models are red. The close-up expands the crowded small-model cluster without changing any coordinates.
Measured results
Full pinned test split · 164 examples · pass@1 higher is better. No global SOTA claim.
| System | Score | Samples | Elapsed¹ | Evidence |
|---|---|---|---|---|
| Qwen/Qwen2.5-Coder-1.5B df3ce67c0e24 | 37.2% | 164 | 526s | Inspect run → 17 Oct 2026, 03:35 UTC |
| Qwen/Qwen2.5-1.5B 8faed761d45a | 36.0% | 164 | 517s | Inspect run → 17 Oct 2026, 03:31 UTC |
| Qwen/Qwen2.5-0.5B 060db6499f32 | 28.7% | 164 | 342s | Inspect run → 17 Oct 2026, 03:18 UTC |
| Qwen/Qwen2.5-Coder-0.5B 8123ea2e9354 | 27.4% | 164 | 437s | Inspect run → 17 Oct 2026, 03:26 UTC |
| SlayerLab/Slayer149-ARC 0e6793ecfb85 | 0.6% | 164 | 588s | Inspect run → 17 Oct 2026, 03:22 UTC |
¹ Evaluation elapsed time includes model loading after dataset selection. It is not serving latency or a fair speed comparison.
What this measures
Benchmark origin
Dataset / split
openai/openai_humaneval / test
Immutable dataset revision
7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544Upstream license metadata
mit · consult upstream terms
Protocol
codesota-humaneval-full-v2 / code
Selection
All 164 examples in the pinned test split. SNLI rows without a valid label are excluded.
Configuration
Seed 42. Greedy generation for math and code. Mean answer-token log likelihood for text classification, completion and declarative multiple-choice recipes. No model judge.
Planned runs
0 qualified visits in 30 days · priority 54.3. Interest changes execution order, within the approved baseline list.
| System | Status |
|---|---|
| SlayerLab/Slayer149-ARC | completed |
| Qwen/Qwen2.5-0.5B | completed |
| Qwen/Qwen2.5-Coder-0.5B | completed |
| Qwen/Qwen2.5-1.5B | completed |
| Qwen/Qwen2.5-Coder-1.5B | completed |