Language / ARC-Challenge / Qwen3.5 same-base cohort

Qwen3.5 0.8B cohort / full ARC-Challenge

Question + candidate answers → selected answer

Measurements Evaluation protocol Planned runs Cost & information

Model size vs score

Static, watermarked figures. SlayerLab models are red. The close-up expands the crowded small-model cluster without changing any coordinates.

ARC-Challenge / Qwen3.5 same-base cohort: all measured models, with a dotted Pareto frontier and peach favourable region — Overview · Download PNG · Download SVG

ARC-Challenge / Qwen3.5 same-base cohort: small-model close-up, with a dotted Pareto frontier and peach favourable region — Small-model close-up · Download PNG · Download SVG

Measured results

Full pinned test split · 1,172 examples · accuracy higher is better. No global SOTA claim.

System	Score	Samples	Elapsed¹	Evidence
Mapika/decider-0.8b a0a01d6f8135	46.8%	1172	31s	Inspect run → 17 Oct 2026, 04:41 UTC
Qwen/Qwen3.5-0.8B-Base dc7cdfe2ee41	38.5%	1172	35s	Inspect run → 17 Oct 2026, 04:40 UTC
unsloth/Qwen3.5-0.8B-Base 44bc982576bf	38.5%	1172	40s	Inspect run → 17 Oct 2026, 04:45 UTC
NamanAgnih0tri/AlphaRoute-0.8B-v1.5 ef87d71f1105	36.1%	1172	31s	Inspect run → 17 Oct 2026, 04:43 UTC
mithulaartigala/reasoning-summarizer-qwen3.5-0.8b-preview 558fc651bf01	34.9%	1172	32s	Inspect run → 17 Oct 2026, 04:44 UTC
McGill-NLP/AfriqueQwen3.5-0.8B-50Langs f49099d79e51	34.2%	1172	36s	Inspect run → 17 Oct 2026, 04:47 UTC
Qwen/Qwen3.5-0.8B 2fc06364715b	33.3%	1172	35s	Inspect run → 17 Oct 2026, 04:44 UTC
sh0ck0r/Qwen3.5-0.8B-heretic fb35b60374fc	32.8%	1172	40s	Inspect run → 17 Oct 2026, 04:43 UTC
ahmedeltawel/Qwen3.5-0.8B-hsb-dsb 7e781a3e9704	32.6%	1172	32s	Inspect run → 17 Oct 2026, 04:42 UTC
Kewk/Heretical-Qwen3.5-0.8B c16807de35d1	32.3%	1172	32s	Inspect run → 17 Oct 2026, 04:44 UTC
cooperdk/BooruNL-0.8B 14dbcf2e009f	31.7%	1172	31s	Inspect run → 17 Oct 2026, 04:45 UTC
mghafiri/qwen3.5-0.8B-decision-model 4a9939034a27	30.3%	1172	35s	Inspect run → 17 Oct 2026, 04:41 UTC
SeerRay-Lab/Xiaomi-OCR-0 ff990be318c4	27.1%	1172	32s	Inspect run → 17 Oct 2026, 04:46 UTC
OmniJev/PlayJev-0.8B a7348002b1e1	24.7%	1172	41s	Inspect run → 17 Oct 2026, 04:46 UTC
nineninesix/qwen3_5-full-attn-only-14 a26dff91d90e	21.4%	1172	18s	Inspect run → 17 Oct 2026, 04:42 UTC

¹ Evaluation elapsed time includes model loading after dataset selection. It is not serving latency or a fair speed comparison.

What this measures

Benchmark origin

ARC-Challenge / Qwen3.5 same-base cohort ↗

Dataset / split

allenai/ai2_arc / test

Immutable dataset revision

210d026faf9955653af8916fad021475a3f00453

Upstream license metadata

cc-by-sa-4.0 · consult upstream terms

Protocol

codesota-qwen35-arc-challenge-v1 / choice

Selection

All 1172 examples in the pinned test split. SNLI rows without a valid label are excluded.

Configuration

Seed 42. Greedy generation for math and code. Summed conditional answer-token log likelihood; raw accuracy. Character-normalized accuracy is recorded separately. No model judge.

Read scoring and comparison rules →

Planned runs

0 qualified visits in 30 days · priority 1000.0. Interest changes execution order, within the approved baseline list.

System	Status
Qwen/Qwen3.5-0.8B-Base	completed
Mapika/decider-0.8b	completed
mghafiri/qwen3.5-0.8B-decision-model	completed
ahmedeltawel/Qwen3.5-0.8B-hsb-dsb	completed
nineninesix/qwen3_5-full-attn-only-14	completed
sh0ck0r/Qwen3.5-0.8B-heretic	completed
NamanAgnih0tri/AlphaRoute-0.8B-v1.5	completed
Kewk/Heretical-Qwen3.5-0.8B	completed
Qwen/Qwen3.5-0.8B	completed
mithulaartigala/reasoning-summarizer-qwen3.5-0.8b-preview	completed
unsloth/Qwen3.5-0.8B-Base	completed
cooperdk/BooruNL-0.8B	completed
OmniJev/PlayJev-0.8B	completed
SeerRay-Lab/Xiaomi-OCR-0	completed
McGill-NLP/AfriqueQwen3.5-0.8B-50Langs	completed