Code / HumanEval

Code generation

Function signature + docstring → Python function

Measurements Evaluation protocol Planned runs Cost & information

Model size vs score

Favourable model size / score regionObserved Pareto frontier

Green uses cohort medians, not a universal quality threshold. Frontier points have no measured alternative with both smaller model size and higher score. Lines connect observed points; they do not predict intermediate models. Uncertainty is not included in frontier membership.

Qwen/Qwen2.5-1.5B · 1543.7M Qwen/Qwen2.5-0.5B · 494.0M

2 compatible measurements. Parameter counts come from the architecture instantiated from each pinned model configuration; no weights are loaded for this count. Click a point to inspect its evidence.

Measured results

Pilot comparison · first 32 valid examples · pass@1 higher is better. No global SOTA claim.

System	Score	Samples	Elapsed¹	Evidence
Qwen/Qwen2.5-1.5B 8faed761d45a	75.0%	32	68s	Inspect run → 16 Oct 2026, 23:27 UTC
Qwen/Qwen2.5-0.5B 060db6499f32	62.5%	32	61s	Inspect run → 16 Oct 2026, 23:24 UTC

¹ Evaluation elapsed time includes model loading after dataset selection. It is not serving latency or a fair speed comparison.

What this measures

Benchmark origin

HumanEval ↗

Dataset / split

openai/openai_humaneval / test

Immutable dataset revision

7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544

Upstream license metadata

mit · consult upstream terms

Protocol

codesota-pilot-v1 / code

Selection

First 32 valid examples in the pinned split order. SNLI rows without a valid label are excluded.

Configuration

Seed 42. Greedy generation for math and code. Mean answer-token log likelihood for text classification, completion and declarative multiple-choice recipes. No model judge.

Read scoring and comparison rules →

Planned runs

6 qualified visits in 30 days · priority 54.5. Interest changes execution order, within the approved baseline list.

System	Status
Qwen/Qwen2.5-0.5B	completed
Qwen/Qwen2.5-1.5B	completed