Completed / CodeSOTA execution on White
Inspect the measurement.
SlayerLab/Slayer149-ARC · HumanEval (full 164, serial greedy pass@1) · 0.6% pass@1
Full benchmark split · 164 scored examples. Full 164-problem HumanEval; one greedy 256-token completion per problem; pass@1 is fraction passing original isolated unit tests. FP32; no chat template/BOS. Fixed top-level completion extraction; no retries or repair. Native generation recomputes context per token; HF uses a serial KV cache. Not HumanEval+ or an unrestricted-generation score.
Run ID / completed
b5ac56547d5088dab7e49bc4
17 Oct 2026, 03:22 UTC
17 Oct 2026, 03:22 UTC
Model revision
0e6793ecfb8501dccc1c97dcd944731748eb8d17Comparison scope
7388be4229d45cb22b346201154469558e2015772bcb2abde7f2fb7b0b7ad294Evaluator SHA-256
496a63534a2dcc2f9bf860166acaaeb0f15f41abd137c89881b8c7674bb321d6Observed resources
588s elapsed · 0.63 GiB peak torch allocation
Dollar cost: not measured
Dollar cost: not measured
Download evidence
Inspect individual examples
Read the recorded prediction, reference and outcome for each example. Code is shown with its original indentation; the reference is the unit-test criterion, not a reference implementation.
- Scored examples
- 164
- Passed
- 1
- Failed
- 163
- Metric
- pass@1
Examples are downloaded only when requested. Search covers all loaded records; 25 examples are displayed per page. Download all examples · JSONL
Example evidence has not been loaded.