CodeSOTAβOriginal site ↗

Benchmark diagnostics / cost & information

What does a benchmark tell us?

Compare execution cost with how often measured models differ in correctness—and how stable their score gap is within the pilot.

Exploratory first-32-example pilots. This measures separation for the tested models, not universal benchmark quality. Equal aggregate scores can hide different strengths; a large gap can still be uncertain.
Benchmark cost versus observed discriminationGSM8K: 40.0 seconds; 4 separating examples out of 32. BoolQ: 21.0 seconds; 10 separating examples out of 32. HellaSwag: 9.0 seconds; 8 separating examples out of 32. SST-2: 21.0 seconds; 12 separating examples out of 32. SNLI: 15.0 seconds; 0 separating examples out of 32. AI2 ARC-Challenge (test): 9.0 seconds; 7 separating examples out of 32. OpenBookQA main (test): 12.0 seconds; 4 separating examples out of 32. ARC-Easy: 8.0 seconds; 12 separating examples out of 32. ARC-Easy: 8.0 seconds; 9 separating examples out of 32. ARC-Easy: 9.0 seconds; 12 separating examples out of 32. ARC-Easy: 9.0 seconds; 10 separating examples out of 32. ARC-Easy: 9.0 seconds; 8 separating examples out of 32. ARC-Easy: 9.0 seconds; 10 separating examples out of 32. ARC-Easy: 10.0 seconds; 14 separating examples out of 32. ARC-Easy: 10.0 seconds; 9 separating examples out of 32. ARC-Easy: 10.0 seconds; 12 separating examples out of 32. ARC-Easy: 9.0 seconds; 10 separating examples out of 32. ARC-Easy: 8.0 seconds; 9 separating examples out of 32. ARC-Easy: 9.0 seconds; 14 separating examples out of 32. ARC-Easy: 9.0 seconds; 12 separating examples out of 32. ARC-Easy: 9.0 seconds; 12 separating examples out of 32. ARC-Easy: 9.0 seconds; 12 separating examples out of 32. ARC-Easy: 10.0 seconds; 12 separating examples out of 32. ARC-Easy: 10.0 seconds; 11 separating examples out of 32. ARC-Easy: 10.0 seconds; 14 separating examples out of 32. ARC-Easy: 9.0 seconds; 12 separating examples out of 32. ARC-Easy: 9.0 seconds; 7 separating examples out of 32. ARC-Easy: 9.0 seconds; 7 separating examples out of 32. ARC-Easy: 9.0 seconds; 13 separating examples out of 32. ARC-Easy: 9.0 seconds; 7 separating examples out of 32. ARC-Easy: 10.0 seconds; 9 separating examples out of 32. ARC-Easy: 10.0 seconds; 8 separating examples out of 32. ARC-Easy: 10.0 seconds; 7 separating examples out of 32. ARC-Easy: 9.0 seconds; 5 separating examples out of 32. ARC-Easy: 10.0 seconds; 4 separating examples out of 32. ARC-Easy: 10.0 seconds; 10 separating examples out of 32. ARC-Easy: 10.0 seconds; 4 separating examples out of 32. ARC-Easy: 11.0 seconds; 8 separating examples out of 32. ARC-Easy: 11.0 seconds; 7 separating examples out of 32. ARC-Easy: 11.0 seconds; 2 separating examples out of 32. ARC-Easy: 10.0 seconds; 2 separating examples out of 32. ARC-Easy: 10.0 seconds; 8 separating examples out of 32. ARC-Easy: 10.0 seconds; 0 separating examples out of 32. ARC-Easy: 11.0 seconds; 10 separating examples out of 32. ARC-Easy: 11.0 seconds; 7 separating examples out of 32. ARC-Easy: 11.0 seconds; 2 separating examples out of 32. ARC-Easy: 10.0 seconds; 4 separating examples out of 32. ARC-Easy: 10.0 seconds; 8 separating examples out of 32. ARC-Easy: 11.0 seconds; 8 separating examples out of 32. ARC-Easy: 11.0 seconds; 13 separating examples out of 32. ARC-Easy: 11.0 seconds; 10 separating examples out of 32. ARC-Easy: 10.0 seconds; 10 separating examples out of 32. ARC-Easy: 11.0 seconds; 10 separating examples out of 32. ARC-Easy: 11.0 seconds; 7 separating examples out of 32. ARC-Easy: 11.0 seconds; 2 separating examples out of 32. ARC-Easy: 10.0 seconds; 4 separating examples out of 32. ARC-Easy: 12.0 seconds; 11 separating examples out of 32. ARC-Easy: 12.0 seconds; 10 separating examples out of 32. ARC-Easy: 11.0 seconds; 10 separating examples out of 32. ARC-Easy: 12.0 seconds; 7 separating examples out of 32. ARC-Easy: 11.0 seconds; 5 separating examples out of 32. ARC-Easy: 11.0 seconds; 2 separating examples out of 32. HellaSwag: 8.0 seconds; 8 separating examples out of 32. HellaSwag: 9.0 seconds; 4 separating examples out of 32. HellaSwag: 9.0 seconds; 5 separating examples out of 32. HellaSwag: 9.0 seconds; 4 separating examples out of 32. HellaSwag: 9.0 seconds; 3 separating examples out of 32. HellaSwag: 9.0 seconds; 4 separating examples out of 32. HellaSwag: 9.0 seconds; 4 separating examples out of 32. HellaSwag: 9.0 seconds; 8 separating examples out of 32. HellaSwag: 10.0 seconds; 5 separating examples out of 32. HellaSwag: 10.0 seconds; 6 separating examples out of 32. HellaSwag: 9.0 seconds; 6 separating examples out of 32. HellaSwag: 9.0 seconds; 7 separating examples out of 32. HellaSwag: 9.0 seconds; 10 separating examples out of 32. HellaSwag: 9.0 seconds; 9 separating examples out of 32. HellaSwag: 9.0 seconds; 10 separating examples out of 32. HellaSwag: 9.0 seconds; 10 separating examples out of 32. HellaSwag: 9.0 seconds; 10 separating examples out of 32. HellaSwag: 10.0 seconds; 7 separating examples out of 32. HellaSwag: 10.0 seconds; 8 separating examples out of 32. HellaSwag: 10.0 seconds; 5 separating examples out of 32. HellaSwag: 10.0 seconds; 6 separating examples out of 32. HellaSwag: 10.0 seconds; 5 separating examples out of 32. HellaSwag: 10.0 seconds; 6 separating examples out of 32. HellaSwag: 10.0 seconds; 6 separating examples out of 32. HellaSwag: 10.0 seconds; 8 separating examples out of 32. HellaSwag: 11.0 seconds; 5 separating examples out of 32. HellaSwag: 11.0 seconds; 6 separating examples out of 32. HellaSwag: 10.0 seconds; 5 separating examples out of 32. HellaSwag: 10.0 seconds; 6 separating examples out of 32. HellaSwag: 10.0 seconds; 5 separating examples out of 32. HellaSwag: 10.0 seconds; 7 separating examples out of 32. HellaSwag: 10.0 seconds; 3 separating examples out of 32. HellaSwag: 11.0 seconds; 2 separating examples out of 32. HellaSwag: 11.0 seconds; 1 separating examples out of 32. HellaSwag: 10.0 seconds; 1 separating examples out of 32. HellaSwag: 10.0 seconds; 0 separating examples out of 32. HellaSwag: 10.0 seconds; 6 separating examples out of 32. HellaSwag: 10.0 seconds; 4 separating examples out of 32. HellaSwag: 11.0 seconds; 5 separating examples out of 32. HellaSwag: 11.0 seconds; 4 separating examples out of 32. HellaSwag: 10.0 seconds; 1 separating examples out of 32. HellaSwag: 10.0 seconds; 5 separating examples out of 32. HellaSwag: 10.0 seconds; 5 separating examples out of 32. HellaSwag: 11.0 seconds; 4 separating examples out of 32. HellaSwag: 11.0 seconds; 5 separating examples out of 32. HellaSwag: 10.0 seconds; 6 separating examples out of 32. HellaSwag: 10.0 seconds; 4 separating examples out of 32. HellaSwag: 11.0 seconds; 5 separating examples out of 32. HellaSwag: 11.0 seconds; 4 separating examples out of 32. HellaSwag: 10.0 seconds; 6 separating examples out of 32. HellaSwag: 11.0 seconds; 7 separating examples out of 32. HellaSwag: 11.0 seconds; 8 separating examples out of 32. HellaSwag: 11.0 seconds; 3 separating examples out of 32. HellaSwag: 11.0 seconds; 2 separating examples out of 32. HellaSwag: 12.0 seconds; 1 separating examples out of 32. BoolQ: 14.0 seconds; 6 separating examples out of 32. BoolQ: 15.0 seconds; 22 separating examples out of 32. BoolQ: 15.0 seconds; 27 separating examples out of 32. BoolQ: 14.0 seconds; 19 separating examples out of 32. BoolQ: 14.0 seconds; 27 separating examples out of 32. BoolQ: 15.0 seconds; 19 separating examples out of 32. BoolQ: 15.0 seconds; 25 separating examples out of 32. BoolQ: 15.0 seconds; 17 separating examples out of 32. BoolQ: 14.0 seconds; 24 separating examples out of 32. BoolQ: 14.0 seconds; 21 separating examples out of 32. BoolQ: 15.0 seconds; 18 separating examples out of 32. BoolQ: 15.0 seconds; 21 separating examples out of 32. BoolQ: 14.0 seconds; 15 separating examples out of 32. BoolQ: 14.0 seconds; 23 separating examples out of 32. BoolQ: 15.0 seconds; 15 separating examples out of 32. BoolQ: 15.0 seconds; 25 separating examples out of 32. BoolQ: 15.0 seconds; 11 separating examples out of 32. BoolQ: 14.0 seconds; 20 separating examples out of 32. BoolQ: 14.0 seconds; 15 separating examples out of 32. BoolQ: 16.0 seconds; 15 separating examples out of 32. BoolQ: 15.0 seconds; 15 separating examples out of 32. BoolQ: 15.0 seconds; 13 separating examples out of 32. BoolQ: 16.0 seconds; 15 separating examples out of 32. BoolQ: 16.0 seconds; 15 separating examples out of 32. BoolQ: 16.0 seconds; 13 separating examples out of 32. BoolQ: 15.0 seconds; 16 separating examples out of 32. BoolQ: 15.0 seconds; 17 separating examples out of 32. BoolQ: 15.0 seconds; 10 separating examples out of 32. BoolQ: 15.0 seconds; 6 separating examples out of 32. BoolQ: 16.0 seconds; 10 separating examples out of 32. BoolQ: 16.0 seconds; 6 separating examples out of 32. BoolQ: 16.0 seconds; 12 separating examples out of 32. BoolQ: 15.0 seconds; 5 separating examples out of 32. BoolQ: 15.0 seconds; 6 separating examples out of 32. BoolQ: 14.0 seconds; 8 separating examples out of 32. BoolQ: 15.0 seconds; 0 separating examples out of 32. BoolQ: 15.0 seconds; 12 separating examples out of 32. BoolQ: 15.0 seconds; 8 separating examples out of 32. BoolQ: 14.0 seconds; 11 separating examples out of 32. BoolQ: 14.0 seconds; 10 separating examples out of 32. BoolQ: 15.0 seconds; 8 separating examples out of 32. BoolQ: 15.0 seconds; 6 separating examples out of 32. BoolQ: 15.0 seconds; 14 separating examples out of 32. BoolQ: 14.0 seconds; 7 separating examples out of 32. BoolQ: 14.0 seconds; 12 separating examples out of 32. BoolQ: 16.0 seconds; 12 separating examples out of 32. BoolQ: 16.0 seconds; 8 separating examples out of 32. BoolQ: 15.0 seconds; 11 separating examples out of 32. BoolQ: 15.0 seconds; 10 separating examples out of 32. BoolQ: 16.0 seconds; 16 separating examples out of 32. BoolQ: 15.0 seconds; 9 separating examples out of 32. BoolQ: 15.0 seconds; 10 separating examples out of 32. BoolQ: 15.0 seconds; 11 separating examples out of 32. BoolQ: 15.0 seconds; 6 separating examples out of 32. BoolQ: 14.0 seconds; 5 separating examples out of 320%020%940%1860%2880%37100%46GSM8K · Qwen/Qwen2.5-0.5B vs Qwen/Qwen2.5-1.5B · 40.0s · 4/32 separating examplesBoolQ · Qwen/Qwen2.5-0.5B vs Qwen/Qwen2.5-1.5B · 21.0s · 10/32 separating examplesHellaSwag · Qwen/Qwen2.5-0.5B vs Qwen/Qwen2.5-1.5B · 9.0s · 8/32 separating examplesSST-2 · Qwen/Qwen2.5-0.5B vs Qwen/Qwen2.5-1.5B · 21.0s · 12/32 separating examplesSNLI · Qwen/Qwen2.5-0.5B vs Qwen/Qwen2.5-1.5B · 15.0s · 0/32 separating examplesAI2 ARC-Challenge (test) · Qwen/Qwen2.5-0.5B vs Qwen/Qwen2.5-1.5B · 9.0s · 7/32 separating examplesOpenBookQA main (test) · Qwen/Qwen2.5-0.5B vs Qwen/Qwen2.5-1.5B · 12.0s · 4/32 separating examplesARC-Easy · HuggingFaceTB/SmolLM2-135M-Instruct vs HuggingFaceTB/SmolLM2-360M-Instruct · 8.0s · 12/32 separating examplesARC-Easy · HuggingFaceTB/SmolLM2-135M-Instruct vs Qwen/Qwen2.5-0.5B-Instruct · 8.0s · 9/32 separating examplesARC-Easy · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer-149M-Decision-v1 · 9.0s · 12/32 separating examplesARC-Easy · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer139-1.01 · 9.0s · 10/32 separating examplesARC-Easy · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer139-1.01b · 9.0s · 8/32 separating examplesARC-Easy · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer139-1.02 · 9.0s · 10/32 separating examplesARC-Easy · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer149 · 10.0s · 14/32 separating examplesARC-Easy · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer149-ARC · 10.0s · 9/32 separating examplesARC-Easy · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 10.0s · 12/32 separating examplesARC-Easy · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 9.0s · 10/32 separating examplesARC-Easy · HuggingFaceTB/SmolLM2-360M-Instruct vs Qwen/Qwen2.5-0.5B-Instruct · 8.0s · 9/32 separating examplesARC-Easy · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer-149M-Decision-v1 · 9.0s · 14/32 separating examplesARC-Easy · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer139-1.01 · 9.0s · 12/32 separating examplesARC-Easy · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer139-1.01b · 9.0s · 12/32 separating examplesARC-Easy · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer139-1.02 · 9.0s · 12/32 separating examplesARC-Easy · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer149 · 10.0s · 12/32 separating examplesARC-Easy · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer149-ARC · 10.0s · 11/32 separating examplesARC-Easy · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 10.0s · 14/32 separating examplesARC-Easy · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 9.0s · 12/32 separating examplesARC-Easy · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer-149M-Decision-v1 · 9.0s · 7/32 separating examplesARC-Easy · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer139-1.01 · 9.0s · 7/32 separating examplesARC-Easy · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer139-1.01b · 9.0s · 13/32 separating examplesARC-Easy · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer139-1.02 · 9.0s · 7/32 separating examplesARC-Easy · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer149 · 10.0s · 9/32 separating examplesARC-Easy · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer149-ARC · 10.0s · 8/32 separating examplesARC-Easy · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 10.0s · 7/32 separating examplesARC-Easy · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 9.0s · 5/32 separating examplesARC-Easy · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer139-1.01 · 10.0s · 4/32 separating examplesARC-Easy · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer139-1.01b · 10.0s · 10/32 separating examplesARC-Easy · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer139-1.02 · 10.0s · 4/32 separating examplesARC-Easy · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer149 · 11.0s · 8/32 separating examplesARC-Easy · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer149-ARC · 11.0s · 7/32 separating examplesARC-Easy · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 11.0s · 2/32 separating examplesARC-Easy · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 10.0s · 2/32 separating examplesARC-Easy · SlayerLab/Slayer139-1.01 vs SlayerLab/Slayer139-1.01b · 10.0s · 8/32 separating examplesARC-Easy · SlayerLab/Slayer139-1.01 vs SlayerLab/Slayer139-1.02 · 10.0s · 0/32 separating examplesARC-Easy · SlayerLab/Slayer139-1.01 vs SlayerLab/Slayer149 · 11.0s · 10/32 separating examplesARC-Easy · SlayerLab/Slayer139-1.01 vs SlayerLab/Slayer149-ARC · 11.0s · 7/32 separating examplesARC-Easy · SlayerLab/Slayer139-1.01 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 11.0s · 2/32 separating examplesARC-Easy · SlayerLab/Slayer139-1.01 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 10.0s · 4/32 separating examplesARC-Easy · SlayerLab/Slayer139-1.01b vs SlayerLab/Slayer139-1.02 · 10.0s · 8/32 separating examplesARC-Easy · SlayerLab/Slayer139-1.01b vs SlayerLab/Slayer149 · 11.0s · 8/32 separating examplesARC-Easy · SlayerLab/Slayer139-1.01b vs SlayerLab/Slayer149-ARC · 11.0s · 13/32 separating examplesARC-Easy · SlayerLab/Slayer139-1.01b vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 11.0s · 10/32 separating examplesARC-Easy · SlayerLab/Slayer139-1.01b vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 10.0s · 10/32 separating examplesARC-Easy · SlayerLab/Slayer139-1.02 vs SlayerLab/Slayer149 · 11.0s · 10/32 separating examplesARC-Easy · SlayerLab/Slayer139-1.02 vs SlayerLab/Slayer149-ARC · 11.0s · 7/32 separating examplesARC-Easy · SlayerLab/Slayer139-1.02 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 11.0s · 2/32 separating examplesARC-Easy · SlayerLab/Slayer139-1.02 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 10.0s · 4/32 separating examplesARC-Easy · SlayerLab/Slayer149 vs SlayerLab/Slayer149-ARC · 12.0s · 11/32 separating examplesARC-Easy · SlayerLab/Slayer149 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 12.0s · 10/32 separating examplesARC-Easy · SlayerLab/Slayer149 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 11.0s · 10/32 separating examplesARC-Easy · SlayerLab/Slayer149-ARC vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 12.0s · 7/32 separating examplesARC-Easy · SlayerLab/Slayer149-ARC vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 11.0s · 5/32 separating examplesARC-Easy · SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 11.0s · 2/32 separating examplesHellaSwag · HuggingFaceTB/SmolLM2-135M-Instruct vs HuggingFaceTB/SmolLM2-360M-Instruct · 8.0s · 8/32 separating examplesHellaSwag · HuggingFaceTB/SmolLM2-135M-Instruct vs Qwen/Qwen2.5-0.5B-Instruct · 9.0s · 4/32 separating examplesHellaSwag · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer-149M-Decision-v1 · 9.0s · 5/32 separating examplesHellaSwag · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer139-1.01 · 9.0s · 4/32 separating examplesHellaSwag · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer139-1.01b · 9.0s · 3/32 separating examplesHellaSwag · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer139-1.02 · 9.0s · 4/32 separating examplesHellaSwag · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer149 · 9.0s · 4/32 separating examplesHellaSwag · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer149-ARC · 9.0s · 8/32 separating examplesHellaSwag · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 10.0s · 5/32 separating examplesHellaSwag · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 10.0s · 6/32 separating examplesHellaSwag · HuggingFaceTB/SmolLM2-360M-Instruct vs Qwen/Qwen2.5-0.5B-Instruct · 9.0s · 6/32 separating examplesHellaSwag · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer-149M-Decision-v1 · 9.0s · 7/32 separating examplesHellaSwag · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer139-1.01 · 9.0s · 10/32 separating examplesHellaSwag · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer139-1.01b · 9.0s · 9/32 separating examplesHellaSwag · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer139-1.02 · 9.0s · 10/32 separating examplesHellaSwag · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer149 · 9.0s · 10/32 separating examplesHellaSwag · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer149-ARC · 9.0s · 10/32 separating examplesHellaSwag · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 10.0s · 7/32 separating examplesHellaSwag · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 10.0s · 8/32 separating examplesHellaSwag · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer-149M-Decision-v1 · 10.0s · 5/32 separating examplesHellaSwag · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer139-1.01 · 10.0s · 6/32 separating examplesHellaSwag · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer139-1.01b · 10.0s · 5/32 separating examplesHellaSwag · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer139-1.02 · 10.0s · 6/32 separating examplesHellaSwag · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer149 · 10.0s · 6/32 separating examplesHellaSwag · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer149-ARC · 10.0s · 8/32 separating examplesHellaSwag · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 11.0s · 5/32 separating examplesHellaSwag · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 11.0s · 6/32 separating examplesHellaSwag · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer139-1.01 · 10.0s · 5/32 separating examplesHellaSwag · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer139-1.01b · 10.0s · 6/32 separating examplesHellaSwag · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer139-1.02 · 10.0s · 5/32 separating examplesHellaSwag · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer149 · 10.0s · 7/32 separating examplesHellaSwag · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer149-ARC · 10.0s · 3/32 separating examplesHellaSwag · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 11.0s · 2/32 separating examplesHellaSwag · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 11.0s · 1/32 separating examplesHellaSwag · SlayerLab/Slayer139-1.01 vs SlayerLab/Slayer139-1.01b · 10.0s · 1/32 separating examplesHellaSwag · SlayerLab/Slayer139-1.01 vs SlayerLab/Slayer139-1.02 · 10.0s · 0/32 separating examplesHellaSwag · SlayerLab/Slayer139-1.01 vs SlayerLab/Slayer149 · 10.0s · 6/32 separating examplesHellaSwag · SlayerLab/Slayer139-1.01 vs SlayerLab/Slayer149-ARC · 10.0s · 4/32 separating examplesHellaSwag · SlayerLab/Slayer139-1.01 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 11.0s · 5/32 separating examplesHellaSwag · SlayerLab/Slayer139-1.01 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 11.0s · 4/32 separating examplesHellaSwag · SlayerLab/Slayer139-1.01b vs SlayerLab/Slayer139-1.02 · 10.0s · 1/32 separating examplesHellaSwag · SlayerLab/Slayer139-1.01b vs SlayerLab/Slayer149 · 10.0s · 5/32 separating examplesHellaSwag · SlayerLab/Slayer139-1.01b vs SlayerLab/Slayer149-ARC · 10.0s · 5/32 separating examplesHellaSwag · SlayerLab/Slayer139-1.01b vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 11.0s · 4/32 separating examplesHellaSwag · SlayerLab/Slayer139-1.01b vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 11.0s · 5/32 separating examplesHellaSwag · SlayerLab/Slayer139-1.02 vs SlayerLab/Slayer149 · 10.0s · 6/32 separating examplesHellaSwag · SlayerLab/Slayer139-1.02 vs SlayerLab/Slayer149-ARC · 10.0s · 4/32 separating examplesHellaSwag · SlayerLab/Slayer139-1.02 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 11.0s · 5/32 separating examplesHellaSwag · SlayerLab/Slayer139-1.02 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 11.0s · 4/32 separating examplesHellaSwag · SlayerLab/Slayer149 vs SlayerLab/Slayer149-ARC · 10.0s · 6/32 separating examplesHellaSwag · SlayerLab/Slayer149 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 11.0s · 7/32 separating examplesHellaSwag · SlayerLab/Slayer149 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 11.0s · 8/32 separating examplesHellaSwag · SlayerLab/Slayer149-ARC vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 11.0s · 3/32 separating examplesHellaSwag · SlayerLab/Slayer149-ARC vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 11.0s · 2/32 separating examplesHellaSwag · SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 12.0s · 1/32 separating examplesBoolQ · HuggingFaceTB/SmolLM2-135M-Instruct vs HuggingFaceTB/SmolLM2-360M-Instruct · 14.0s · 6/32 separating examplesBoolQ · HuggingFaceTB/SmolLM2-135M-Instruct vs Qwen/Qwen2.5-0.5B-Instruct · 15.0s · 22/32 separating examplesBoolQ · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer-149M-Decision-v1 · 15.0s · 27/32 separating examplesBoolQ · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer139-1.01 · 14.0s · 19/32 separating examplesBoolQ · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer139-1.01b · 14.0s · 27/32 separating examplesBoolQ · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer139-1.02 · 15.0s · 19/32 separating examplesBoolQ · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer149 · 15.0s · 25/32 separating examplesBoolQ · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer149-ARC · 15.0s · 17/32 separating examplesBoolQ · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 14.0s · 24/32 separating examplesBoolQ · HuggingFaceTB/SmolLM2-135M-Instruct vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 14.0s · 21/32 separating examplesBoolQ · HuggingFaceTB/SmolLM2-360M-Instruct vs Qwen/Qwen2.5-0.5B-Instruct · 15.0s · 18/32 separating examplesBoolQ · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer-149M-Decision-v1 · 15.0s · 21/32 separating examplesBoolQ · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer139-1.01 · 14.0s · 15/32 separating examplesBoolQ · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer139-1.01b · 14.0s · 23/32 separating examplesBoolQ · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer139-1.02 · 15.0s · 15/32 separating examplesBoolQ · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer149 · 15.0s · 25/32 separating examplesBoolQ · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer149-ARC · 15.0s · 11/32 separating examplesBoolQ · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 14.0s · 20/32 separating examplesBoolQ · HuggingFaceTB/SmolLM2-360M-Instruct vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 14.0s · 15/32 separating examplesBoolQ · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer-149M-Decision-v1 · 16.0s · 15/32 separating examplesBoolQ · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer139-1.01 · 15.0s · 15/32 separating examplesBoolQ · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer139-1.01b · 15.0s · 13/32 separating examplesBoolQ · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer139-1.02 · 16.0s · 15/32 separating examplesBoolQ · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer149 · 16.0s · 15/32 separating examplesBoolQ · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer149-ARC · 16.0s · 13/32 separating examplesBoolQ · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 15.0s · 16/32 separating examplesBoolQ · Qwen/Qwen2.5-0.5B-Instruct vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 15.0s · 17/32 separating examplesBoolQ · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer139-1.01 · 15.0s · 10/32 separating examplesBoolQ · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer139-1.01b · 15.0s · 6/32 separating examplesBoolQ · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer139-1.02 · 16.0s · 10/32 separating examplesBoolQ · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer149 · 16.0s · 6/32 separating examplesBoolQ · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer149-ARC · 16.0s · 12/32 separating examplesBoolQ · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 15.0s · 5/32 separating examplesBoolQ · SlayerLab/Slayer-149M-Decision-v1 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 15.0s · 6/32 separating examplesBoolQ · SlayerLab/Slayer139-1.01 vs SlayerLab/Slayer139-1.01b · 14.0s · 8/32 separating examplesBoolQ · SlayerLab/Slayer139-1.01 vs SlayerLab/Slayer139-1.02 · 15.0s · 0/32 separating examplesBoolQ · SlayerLab/Slayer139-1.01 vs SlayerLab/Slayer149 · 15.0s · 12/32 separating examplesBoolQ · SlayerLab/Slayer139-1.01 vs SlayerLab/Slayer149-ARC · 15.0s · 8/32 separating examplesBoolQ · SlayerLab/Slayer139-1.01 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 14.0s · 11/32 separating examplesBoolQ · SlayerLab/Slayer139-1.01 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 14.0s · 10/32 separating examplesBoolQ · SlayerLab/Slayer139-1.01b vs SlayerLab/Slayer139-1.02 · 15.0s · 8/32 separating examplesBoolQ · SlayerLab/Slayer139-1.01b vs SlayerLab/Slayer149 · 15.0s · 6/32 separating examplesBoolQ · SlayerLab/Slayer139-1.01b vs SlayerLab/Slayer149-ARC · 15.0s · 14/32 separating examplesBoolQ · SlayerLab/Slayer139-1.01b vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 14.0s · 7/32 separating examplesBoolQ · SlayerLab/Slayer139-1.01b vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 14.0s · 12/32 separating examplesBoolQ · SlayerLab/Slayer139-1.02 vs SlayerLab/Slayer149 · 16.0s · 12/32 separating examplesBoolQ · SlayerLab/Slayer139-1.02 vs SlayerLab/Slayer149-ARC · 16.0s · 8/32 separating examplesBoolQ · SlayerLab/Slayer139-1.02 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 15.0s · 11/32 separating examplesBoolQ · SlayerLab/Slayer139-1.02 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 15.0s · 10/32 separating examplesBoolQ · SlayerLab/Slayer149 vs SlayerLab/Slayer149-ARC · 16.0s · 16/32 separating examplesBoolQ · SlayerLab/Slayer149 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 15.0s · 9/32 separating examplesBoolQ · SlayerLab/Slayer149 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 15.0s · 10/32 separating examplesBoolQ · SlayerLab/Slayer149-ARC vs SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 · 15.0s · 11/32 separating examplesBoolQ · SlayerLab/Slayer149-ARC vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 15.0s · 6/32 separating examplesBoolQ · SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 vs SlayerLab/Slayer149-Balanced-ARC-E-ft-v2 · 14.0s · 5/32 separating examplesReserved GPU lease seconds / pairPaired correctness disagreement
172 compatible pairs. Click a point for run evidence. Higher disagreement is not automatically better: cost, uncertainty, task relevance and reliability matter.

Cost and coverage

BenchmarkCompatible modelsGPU lease timeEvaluation elapsedProcess GPU-slot timeDemand priorityPaired evidence
HumanEval
32-example pilot · pass@1
2Not reliably recorded
0/2 runs · claim → finish
128.8sNot recorded
0/2 completed runs instrumented
54.5
interest + coverage / freshness
1 aligned pairs
GSM8K
32-example pilot · exact_match
240 GPU-s
2/2 runs · claim → finish
27.0s38.4 GPU-s
2/2 completed runs instrumented
34.7
interest + coverage / freshness
1 aligned pairs
BoolQ
32-example pilot · accuracy
221 GPU-s
2/2 runs · claim → finish
3.3s15.2 GPU-s
2/2 completed runs instrumented
0.0
interest + coverage / freshness
1 aligned pairs
HellaSwag
32-example pilot · accuracy
29 GPU-s
2/2 runs · claim → finish
3.0s9.0 GPU-s
2/2 completed runs instrumented
6.9
interest + coverage / freshness
1 aligned pairs
SST-2
32-example pilot · accuracy
221 GPU-s
2/2 runs · claim → finish
2.7s15.2 GPU-s
2/2 completed runs instrumented
0.0
interest + coverage / freshness
1 aligned pairs
SNLI
32-example pilot · accuracy
215 GPU-s
2/2 runs · claim → finish
2.8s14.4 GPU-s
2/2 completed runs instrumented
0.0
interest + coverage / freshness
1 aligned pairs
STS Benchmark
32-example pilot · spearman
1Not reliably recorded
0/1 runs · claim → finish
0.9sNot recorded
0/1 completed runs instrumented
32.6
interest + coverage / freshness
At least two compatible models are required.
IAM lines
32-example pilot · cer
1Not reliably recorded
0/1 runs · claim → finish
8.4sNot recorded
0/1 completed runs instrumented
35.1
interest + coverage / freshness
At least two compatible models are required.
LibriSpeech test-clean
32-example pilot · wer
1Not reliably recorded
0/1 runs · claim → finish
5.0sNot recorded
0/1 completed runs instrumented
42.0
interest + coverage / freshness
At least two compatible models are required.
CIFAR-10
32-example pilot · accuracy
1Not reliably recorded
0/1 runs · claim → finish
2.0sNot recorded
0/1 completed runs instrumented
35.1
interest + coverage / freshness
At least two compatible models are required.
AI2 ARC-Challenge (test)
32-example pilot · accuracy
29 GPU-s
2/2 runs · claim → finish
3.1s9.0 GPU-s
2/2 completed runs instrumented
7.0
interest + coverage / freshness
1 aligned pairs
OpenBookQA main (test)
32-example pilot · accuracy
212 GPU-s
2/2 runs · claim → finish
4.1s10.3 GPU-s
2/2 completed runs instrumented
0.0
interest + coverage / freshness
1 aligned pairs
ARC-Easy
32-example pilot · accuracy
1155 GPU-s
11/11 runs · claim → finish
20.9s53.3 GPU-s
11/11 completed runs instrumented
1000.0
interest + coverage / freshness
55 aligned pairs
HellaSwag
32-example pilot · accuracy
1155 GPU-s
11/11 runs · claim → finish
20.9s53.4 GPU-s
11/11 completed runs instrumented
1006.9
interest + coverage / freshness
55 aligned pairs
BoolQ
32-example pilot · accuracy
1182 GPU-s
11/11 runs · claim → finish
18.1s80.2 GPU-s
11/11 completed runs instrumented
1006.9
interest + coverage / freshness
55 aligned pairs

Does changing the model change the answer quality?

For binary metrics, align example IDs and reference answers within the same scope. A “separating” example passes for one model and fails for the other. A/B order is fixed by model ID and revision, not chosen by the winning score.

Benchmark / pairSeparating examplesΔ score · B − A95% resampling intervalExact McNemar pPilot reading
HumanEval
A: Qwen/Qwen2.5-0.5B
B: Qwen/Qwen2.5-1.5B
4/32 · 12.5%
A only: 0 · B only: 4
Both pass: 20 · both fail: 8
12.5 pp[3.1, 25.0] pp0.125
exploratory · unadjusted
Exploratory interval only
Resampling excludes zero, but the exact paired test does not resolve the difference.
Mixed difficulty for this pair
GSM8K
A: Qwen/Qwen2.5-0.5B
B: Qwen/Qwen2.5-1.5B
4/32 · 12.5%
A only: 1 · B only: 3
Both pass: 0 · both fail: 28
6.3 pp[-6.3, 18.8] pp0.625
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: Qwen/Qwen2.5-0.5B
B: Qwen/Qwen2.5-1.5B
10/32 · 31.3%
A only: 5 · B only: 5
Both pass: 17 · both fail: 5
0.0 pp[-18.8, 18.8] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: Qwen/Qwen2.5-0.5B
B: Qwen/Qwen2.5-1.5B
8/32 · 25.0%
A only: 2 · B only: 6
Both pass: 10 · both fail: 14
12.5 pp[-3.1, 28.1] pp0.289
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
SST-2
A: Qwen/Qwen2.5-0.5B
B: Qwen/Qwen2.5-1.5B
12/32 · 37.5%
A only: 11 · B only: 1
Both pass: 19 · both fail: 1
-31.3 pp[-50.0, -12.5] pp0.006
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
SNLI
A: Qwen/Qwen2.5-0.5B
B: Qwen/Qwen2.5-1.5B
0/32 · 0.0%
A only: 0 · B only: 0
Both pass: 12 · both fail: 20
0.0 pp[0.0, 0.0] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
AI2 ARC-Challenge (test)
A: Qwen/Qwen2.5-0.5B
B: Qwen/Qwen2.5-1.5B
7/32 · 21.9%
A only: 1 · B only: 6
Both pass: 9 · both fail: 16
15.6 pp[0.0, 31.3] pp0.125
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
OpenBookQA main (test)
A: Qwen/Qwen2.5-0.5B
B: Qwen/Qwen2.5-1.5B
4/32 · 12.5%
A only: 2 · B only: 2
Both pass: 11 · both fail: 17
0.0 pp[-12.5, 12.5] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: HuggingFaceTB/SmolLM2-360M-Instruct
12/32 · 37.5%
A only: 8 · B only: 4
Both pass: 10 · both fail: 10
-12.5 pp[-34.4, 9.4] pp0.388
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: Qwen/Qwen2.5-0.5B-Instruct
9/32 · 28.1%
A only: 4 · B only: 5
Both pass: 14 · both fail: 9
3.1 pp[-15.6, 21.9] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer-149M-Decision-v1
12/32 · 37.5%
A only: 6 · B only: 6
Both pass: 12 · both fail: 8
0.0 pp[-21.9, 21.9] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer139-1.01
10/32 · 31.3%
A only: 4 · B only: 6
Both pass: 14 · both fail: 8
6.3 pp[-12.5, 25.0] pp0.754
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer139-1.01b
8/32 · 25.0%
A only: 7 · B only: 1
Both pass: 11 · both fail: 13
-18.8 pp[-34.4, -3.1] pp0.070
exploratory · unadjusted
Exploratory interval only
Resampling excludes zero, but the exact paired test does not resolve the difference.
Mixed difficulty for this pair
ARC-Easy
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer139-1.02
10/32 · 31.3%
A only: 4 · B only: 6
Both pass: 14 · both fail: 8
6.3 pp[-12.5, 25.0] pp0.754
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer149
14/32 · 43.8%
A only: 9 · B only: 5
Both pass: 9 · both fail: 9
-12.5 pp[-34.4, 9.4] pp0.424
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer149-ARC
9/32 · 28.1%
A only: 1 · B only: 8
Both pass: 17 · both fail: 6
21.9 pp[6.3, 37.5] pp0.039
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
ARC-Easy
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
12/32 · 37.5%
A only: 5 · B only: 7
Both pass: 13 · both fail: 7
6.3 pp[-15.6, 28.1] pp0.774
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
10/32 · 31.3%
A only: 4 · B only: 6
Both pass: 14 · both fail: 8
6.3 pp[-12.5, 25.0] pp0.754
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: Qwen/Qwen2.5-0.5B-Instruct
9/32 · 28.1%
A only: 2 · B only: 7
Both pass: 12 · both fail: 11
15.6 pp[-3.1, 34.4] pp0.180
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer-149M-Decision-v1
14/32 · 43.8%
A only: 5 · B only: 9
Both pass: 9 · both fail: 9
12.5 pp[-9.4, 34.4] pp0.424
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer139-1.01
12/32 · 37.5%
A only: 3 · B only: 9
Both pass: 11 · both fail: 9
18.8 pp[-3.1, 40.6] pp0.146
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer139-1.01b
12/32 · 37.5%
A only: 7 · B only: 5
Both pass: 7 · both fail: 13
-6.3 pp[-28.1, 15.6] pp0.774
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer139-1.02
12/32 · 37.5%
A only: 3 · B only: 9
Both pass: 11 · both fail: 9
18.8 pp[-3.1, 40.6] pp0.146
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer149
12/32 · 37.5%
A only: 6 · B only: 6
Both pass: 8 · both fail: 12
0.0 pp[-21.9, 21.9] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer149-ARC
11/32 · 34.4%
A only: 0 · B only: 11
Both pass: 14 · both fail: 7
34.4 pp[18.8, 50.0] pp<0.001
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
ARC-Easy
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
14/32 · 43.8%
A only: 4 · B only: 10
Both pass: 10 · both fail: 8
18.8 pp[-3.1, 40.6] pp0.180
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
12/32 · 37.5%
A only: 3 · B only: 9
Both pass: 11 · both fail: 9
18.8 pp[-3.1, 40.6] pp0.146
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer-149M-Decision-v1
7/32 · 21.9%
A only: 4 · B only: 3
Both pass: 15 · both fail: 10
-3.1 pp[-18.8, 12.5] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer139-1.01
7/32 · 21.9%
A only: 3 · B only: 4
Both pass: 16 · both fail: 9
3.1 pp[-12.5, 18.8] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer139-1.01b
13/32 · 40.6%
A only: 10 · B only: 3
Both pass: 9 · both fail: 10
-21.9 pp[-43.8, 0.0] pp0.092
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer139-1.02
7/32 · 21.9%
A only: 3 · B only: 4
Both pass: 16 · both fail: 9
3.1 pp[-12.5, 18.8] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer149
9/32 · 28.1%
A only: 7 · B only: 2
Both pass: 12 · both fail: 11
-15.6 pp[-34.4, 3.1] pp0.180
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer149-ARC
8/32 · 25.0%
A only: 1 · B only: 7
Both pass: 18 · both fail: 6
18.8 pp[3.1, 34.4] pp0.070
exploratory · unadjusted
Exploratory interval only
Resampling excludes zero, but the exact paired test does not resolve the difference.
Mixed difficulty for this pair
ARC-Easy
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
7/32 · 21.9%
A only: 3 · B only: 4
Both pass: 16 · both fail: 9
3.1 pp[-12.5, 18.8] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
5/32 · 15.6%
A only: 2 · B only: 3
Both pass: 17 · both fail: 10
3.1 pp[-9.4, 15.6] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer139-1.01
4/32 · 12.5%
A only: 1 · B only: 3
Both pass: 17 · both fail: 11
6.3 pp[-6.3, 18.8] pp0.625
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer139-1.01b
10/32 · 31.3%
A only: 8 · B only: 2
Both pass: 10 · both fail: 12
-18.8 pp[-37.5, 0.0] pp0.109
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer139-1.02
4/32 · 12.5%
A only: 1 · B only: 3
Both pass: 17 · both fail: 11
6.3 pp[-6.3, 18.8] pp0.625
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer149
8/32 · 25.0%
A only: 6 · B only: 2
Both pass: 12 · both fail: 12
-12.5 pp[-31.3, 3.1] pp0.289
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer149-ARC
7/32 · 21.9%
A only: 0 · B only: 7
Both pass: 18 · both fail: 7
21.9 pp[9.4, 37.5] pp0.016
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
2/32 · 6.3%
A only: 0 · B only: 2
Both pass: 18 · both fail: 12
6.3 pp[0.0, 15.6] pp0.500
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
2/32 · 6.3%
A only: 0 · B only: 2
Both pass: 18 · both fail: 12
6.3 pp[0.0, 15.6] pp0.500
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer139-1.01
B: SlayerLab/Slayer139-1.01b
8/32 · 25.0%
A only: 8 · B only: 0
Both pass: 12 · both fail: 12
-25.0 pp[-40.6, -12.5] pp0.008
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer139-1.01
B: SlayerLab/Slayer139-1.02
0/32 · 0.0%
A only: 0 · B only: 0
Both pass: 20 · both fail: 12
0.0 pp[0.0, 0.0] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer139-1.01
B: SlayerLab/Slayer149
10/32 · 31.3%
A only: 8 · B only: 2
Both pass: 12 · both fail: 10
-18.8 pp[-37.5, 0.0] pp0.109
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer139-1.01
B: SlayerLab/Slayer149-ARC
7/32 · 21.9%
A only: 1 · B only: 6
Both pass: 19 · both fail: 6
15.6 pp[0.0, 31.3] pp0.125
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer139-1.01
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
2/32 · 6.3%
A only: 1 · B only: 1
Both pass: 19 · both fail: 11
0.0 pp[-9.4, 9.4] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer139-1.01
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
4/32 · 12.5%
A only: 2 · B only: 2
Both pass: 18 · both fail: 10
0.0 pp[-12.5, 12.5] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer139-1.01b
B: SlayerLab/Slayer139-1.02
8/32 · 25.0%
A only: 0 · B only: 8
Both pass: 12 · both fail: 12
25.0 pp[12.5, 40.6] pp0.008
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer139-1.01b
B: SlayerLab/Slayer149
8/32 · 25.0%
A only: 3 · B only: 5
Both pass: 9 · both fail: 15
6.3 pp[-12.5, 25.0] pp0.727
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer139-1.01b
B: SlayerLab/Slayer149-ARC
13/32 · 40.6%
A only: 0 · B only: 13
Both pass: 12 · both fail: 7
40.6 pp[25.0, 59.4] pp<0.001
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer139-1.01b
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
10/32 · 31.3%
A only: 1 · B only: 9
Both pass: 11 · both fail: 11
25.0 pp[9.4, 43.8] pp0.021
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer139-1.01b
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
10/32 · 31.3%
A only: 1 · B only: 9
Both pass: 11 · both fail: 11
25.0 pp[9.4, 43.8] pp0.021
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer139-1.02
B: SlayerLab/Slayer149
10/32 · 31.3%
A only: 8 · B only: 2
Both pass: 12 · both fail: 10
-18.8 pp[-37.5, 0.0] pp0.109
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer139-1.02
B: SlayerLab/Slayer149-ARC
7/32 · 21.9%
A only: 1 · B only: 6
Both pass: 19 · both fail: 6
15.6 pp[0.0, 31.3] pp0.125
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer139-1.02
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
2/32 · 6.3%
A only: 1 · B only: 1
Both pass: 19 · both fail: 11
0.0 pp[-9.4, 9.4] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer139-1.02
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
4/32 · 12.5%
A only: 2 · B only: 2
Both pass: 18 · both fail: 10
0.0 pp[-12.5, 12.5] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer149
B: SlayerLab/Slayer149-ARC
11/32 · 34.4%
A only: 0 · B only: 11
Both pass: 14 · both fail: 7
34.4 pp[18.8, 50.0] pp<0.001
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer149
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
10/32 · 31.3%
A only: 2 · B only: 8
Both pass: 12 · both fail: 10
18.8 pp[0.0, 37.5] pp0.109
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer149
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
10/32 · 31.3%
A only: 2 · B only: 8
Both pass: 12 · both fail: 10
18.8 pp[0.0, 37.5] pp0.109
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer149-ARC
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
7/32 · 21.9%
A only: 6 · B only: 1
Both pass: 19 · both fail: 6
-15.6 pp[-31.3, 0.0] pp0.125
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer149-ARC
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
5/32 · 15.6%
A only: 5 · B only: 0
Both pass: 20 · both fail: 7
-15.6 pp[-28.1, -3.1] pp0.063
exploratory · unadjusted
Exploratory interval only
Resampling excludes zero, but the exact paired test does not resolve the difference.
Mixed difficulty for this pair
ARC-Easy
A: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
2/32 · 6.3%
A only: 1 · B only: 1
Both pass: 19 · both fail: 11
0.0 pp[-9.4, 9.4] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: HuggingFaceTB/SmolLM2-360M-Instruct
8/32 · 25.0%
A only: 1 · B only: 7
Both pass: 8 · both fail: 16
18.8 pp[3.1, 34.4] pp0.070
exploratory · unadjusted
Exploratory interval only
Resampling excludes zero, but the exact paired test does not resolve the difference.
Mixed difficulty for this pair
HellaSwag
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: Qwen/Qwen2.5-0.5B-Instruct
4/32 · 12.5%
A only: 0 · B only: 4
Both pass: 9 · both fail: 19
12.5 pp[3.1, 25.0] pp0.125
exploratory · unadjusted
Exploratory interval only
Resampling excludes zero, but the exact paired test does not resolve the difference.
Mixed difficulty for this pair
HellaSwag
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer-149M-Decision-v1
5/32 · 15.6%
A only: 2 · B only: 3
Both pass: 7 · both fail: 20
3.1 pp[-9.4, 15.6] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer139-1.01
4/32 · 12.5%
A only: 1 · B only: 3
Both pass: 8 · both fail: 20
6.3 pp[-6.3, 18.8] pp0.625
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer139-1.01b
3/32 · 9.4%
A only: 0 · B only: 3
Both pass: 9 · both fail: 20
9.4 pp[0.0, 21.9] pp0.250
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer139-1.02
4/32 · 12.5%
A only: 1 · B only: 3
Both pass: 8 · both fail: 20
6.3 pp[-6.3, 18.8] pp0.625
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer149
4/32 · 12.5%
A only: 1 · B only: 3
Both pass: 8 · both fail: 20
6.3 pp[-6.3, 18.8] pp0.625
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer149-ARC
8/32 · 25.0%
A only: 3 · B only: 5
Both pass: 6 · both fail: 18
6.3 pp[-12.5, 25.0] pp0.727
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
5/32 · 15.6%
A only: 1 · B only: 4
Both pass: 8 · both fail: 19
9.4 pp[-3.1, 21.9] pp0.375
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
6/32 · 18.8%
A only: 2 · B only: 4
Both pass: 7 · both fail: 19
6.3 pp[-9.4, 21.9] pp0.688
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: Qwen/Qwen2.5-0.5B-Instruct
6/32 · 18.8%
A only: 4 · B only: 2
Both pass: 11 · both fail: 15
-6.3 pp[-21.9, 9.4] pp0.688
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer-149M-Decision-v1
7/32 · 21.9%
A only: 6 · B only: 1
Both pass: 9 · both fail: 16
-15.6 pp[-31.3, 0.0] pp0.125
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer139-1.01
10/32 · 31.3%
A only: 7 · B only: 3
Both pass: 8 · both fail: 14
-12.5 pp[-31.3, 6.3] pp0.344
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer139-1.01b
9/32 · 28.1%
A only: 6 · B only: 3
Both pass: 9 · both fail: 14
-9.4 pp[-28.1, 9.4] pp0.508
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer139-1.02
10/32 · 31.3%
A only: 7 · B only: 3
Both pass: 8 · both fail: 14
-12.5 pp[-31.3, 6.3] pp0.344
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer149
10/32 · 31.3%
A only: 7 · B only: 3
Both pass: 8 · both fail: 14
-12.5 pp[-31.3, 6.3] pp0.344
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer149-ARC
10/32 · 31.3%
A only: 7 · B only: 3
Both pass: 8 · both fail: 14
-12.5 pp[-31.3, 6.3] pp0.344
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
7/32 · 21.9%
A only: 5 · B only: 2
Both pass: 10 · both fail: 15
-9.4 pp[-25.0, 6.3] pp0.453
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
8/32 · 25.0%
A only: 6 · B only: 2
Both pass: 9 · both fail: 15
-12.5 pp[-31.3, 3.1] pp0.289
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer-149M-Decision-v1
5/32 · 15.6%
A only: 4 · B only: 1
Both pass: 9 · both fail: 18
-9.4 pp[-25.0, 3.1] pp0.375
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer139-1.01
6/32 · 18.8%
A only: 4 · B only: 2
Both pass: 9 · both fail: 17
-6.3 pp[-21.9, 9.4] pp0.688
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer139-1.01b
5/32 · 15.6%
A only: 3 · B only: 2
Both pass: 10 · both fail: 17
-3.1 pp[-15.6, 9.4] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer139-1.02
6/32 · 18.8%
A only: 4 · B only: 2
Both pass: 9 · both fail: 17
-6.3 pp[-21.9, 9.4] pp0.688
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer149
6/32 · 18.8%
A only: 4 · B only: 2
Both pass: 9 · both fail: 17
-6.3 pp[-21.9, 9.4] pp0.688
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer149-ARC
8/32 · 25.0%
A only: 5 · B only: 3
Both pass: 8 · both fail: 16
-6.3 pp[-25.0, 9.4] pp0.727
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
5/32 · 15.6%
A only: 3 · B only: 2
Both pass: 10 · both fail: 17
-3.1 pp[-18.8, 9.4] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
6/32 · 18.8%
A only: 4 · B only: 2
Both pass: 9 · both fail: 17
-6.3 pp[-21.9, 9.4] pp0.688
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer139-1.01
5/32 · 15.6%
A only: 2 · B only: 3
Both pass: 8 · both fail: 19
3.1 pp[-9.4, 18.8] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer139-1.01b
6/32 · 18.8%
A only: 2 · B only: 4
Both pass: 8 · both fail: 18
6.3 pp[-6.3, 21.9] pp0.688
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer139-1.02
5/32 · 15.6%
A only: 2 · B only: 3
Both pass: 8 · both fail: 19
3.1 pp[-9.4, 18.8] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer149
7/32 · 21.9%
A only: 3 · B only: 4
Both pass: 7 · both fail: 18
3.1 pp[-12.5, 18.8] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer149-ARC
3/32 · 9.4%
A only: 1 · B only: 2
Both pass: 9 · both fail: 20
3.1 pp[-6.3, 12.5] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
2/32 · 6.3%
A only: 0 · B only: 2
Both pass: 10 · both fail: 20
6.3 pp[0.0, 15.6] pp0.500
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
1/32 · 3.1%
A only: 0 · B only: 1
Both pass: 10 · both fail: 21
3.1 pp[0.0, 9.4] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer139-1.01
B: SlayerLab/Slayer139-1.01b
1/32 · 3.1%
A only: 0 · B only: 1
Both pass: 11 · both fail: 20
3.1 pp[0.0, 9.4] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer139-1.01
B: SlayerLab/Slayer139-1.02
0/32 · 0.0%
A only: 0 · B only: 0
Both pass: 11 · both fail: 21
0.0 pp[0.0, 0.0] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer139-1.01
B: SlayerLab/Slayer149
6/32 · 18.8%
A only: 3 · B only: 3
Both pass: 8 · both fail: 18
0.0 pp[-15.6, 15.6] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer139-1.01
B: SlayerLab/Slayer149-ARC
4/32 · 12.5%
A only: 2 · B only: 2
Both pass: 9 · both fail: 19
0.0 pp[-12.5, 12.5] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer139-1.01
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
5/32 · 15.6%
A only: 2 · B only: 3
Both pass: 9 · both fail: 18
3.1 pp[-9.4, 15.6] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer139-1.01
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
4/32 · 12.5%
A only: 2 · B only: 2
Both pass: 9 · both fail: 19
0.0 pp[-12.5, 12.5] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer139-1.01b
B: SlayerLab/Slayer139-1.02
1/32 · 3.1%
A only: 1 · B only: 0
Both pass: 11 · both fail: 20
-3.1 pp[-9.4, 0.0] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer139-1.01b
B: SlayerLab/Slayer149
5/32 · 15.6%
A only: 3 · B only: 2
Both pass: 9 · both fail: 18
-3.1 pp[-15.6, 9.4] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer139-1.01b
B: SlayerLab/Slayer149-ARC
5/32 · 15.6%
A only: 3 · B only: 2
Both pass: 9 · both fail: 18
-3.1 pp[-18.8, 9.4] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer139-1.01b
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
4/32 · 12.5%
A only: 2 · B only: 2
Both pass: 10 · both fail: 18
0.0 pp[-12.5, 12.5] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer139-1.01b
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
5/32 · 15.6%
A only: 3 · B only: 2
Both pass: 9 · both fail: 18
-3.1 pp[-15.6, 9.4] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer139-1.02
B: SlayerLab/Slayer149
6/32 · 18.8%
A only: 3 · B only: 3
Both pass: 8 · both fail: 18
0.0 pp[-15.6, 15.6] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer139-1.02
B: SlayerLab/Slayer149-ARC
4/32 · 12.5%
A only: 2 · B only: 2
Both pass: 9 · both fail: 19
0.0 pp[-12.5, 12.5] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer139-1.02
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
5/32 · 15.6%
A only: 2 · B only: 3
Both pass: 9 · both fail: 18
3.1 pp[-9.4, 15.6] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer139-1.02
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
4/32 · 12.5%
A only: 2 · B only: 2
Both pass: 9 · both fail: 19
0.0 pp[-12.5, 12.5] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer149
B: SlayerLab/Slayer149-ARC
6/32 · 18.8%
A only: 3 · B only: 3
Both pass: 8 · both fail: 18
0.0 pp[-15.6, 15.6] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer149
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
7/32 · 21.9%
A only: 3 · B only: 4
Both pass: 8 · both fail: 17
3.1 pp[-12.5, 18.8] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer149
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
8/32 · 25.0%
A only: 4 · B only: 4
Both pass: 7 · both fail: 17
0.0 pp[-18.8, 15.6] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer149-ARC
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
3/32 · 9.4%
A only: 1 · B only: 2
Both pass: 10 · both fail: 19
3.1 pp[-6.3, 15.6] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer149-ARC
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
2/32 · 6.3%
A only: 1 · B only: 1
Both pass: 10 · both fail: 20
0.0 pp[-9.4, 9.4] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
HellaSwag
A: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
1/32 · 3.1%
A only: 1 · B only: 0
Both pass: 11 · both fail: 20
-3.1 pp[-9.4, 0.0] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: HuggingFaceTB/SmolLM2-360M-Instruct
6/32 · 18.8%
A only: 1 · B only: 5
Both pass: 6 · both fail: 20
12.5 pp[0.0, 28.1] pp0.219
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: Qwen/Qwen2.5-0.5B-Instruct
22/32 · 68.8%
A only: 3 · B only: 19
Both pass: 4 · both fail: 6
50.0 pp[28.1, 71.9] pp<0.001
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
BoolQ
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer-149M-Decision-v1
27/32 · 84.4%
A only: 6 · B only: 21
Both pass: 1 · both fail: 4
46.9 pp[18.8, 71.9] pp0.006
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
BoolQ
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer139-1.01
19/32 · 59.4%
A only: 4 · B only: 15
Both pass: 3 · both fail: 10
34.4 pp[12.4, 56.3] pp0.019
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
BoolQ
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer139-1.01b
27/32 · 84.4%
A only: 5 · B only: 22
Both pass: 2 · both fail: 3
53.1 pp[25.0, 78.1] pp0.002
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
BoolQ
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer139-1.02
19/32 · 59.4%
A only: 4 · B only: 15
Both pass: 3 · both fail: 10
34.4 pp[12.4, 56.3] pp0.019
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
BoolQ
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer149
25/32 · 78.1%
A only: 5 · B only: 20
Both pass: 2 · both fail: 5
46.9 pp[18.8, 71.9] pp0.004
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
BoolQ
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer149-ARC
17/32 · 53.1%
A only: 4 · B only: 13
Both pass: 3 · both fail: 12
28.1 pp[3.1, 50.0] pp0.049
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
BoolQ
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
24/32 · 75.0%
A only: 3 · B only: 21
Both pass: 4 · both fail: 4
56.3 pp[31.3, 78.1] pp<0.001
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
BoolQ
A: HuggingFaceTB/SmolLM2-135M-Instruct
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
21/32 · 65.6%
A only: 4 · B only: 17
Both pass: 3 · both fail: 8
40.6 pp[15.6, 62.5] pp0.007
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
BoolQ
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: Qwen/Qwen2.5-0.5B-Instruct
18/32 · 56.3%
A only: 3 · B only: 15
Both pass: 8 · both fail: 6
37.5 pp[12.5, 59.4] pp0.008
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
BoolQ
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer-149M-Decision-v1
21/32 · 65.6%
A only: 5 · B only: 16
Both pass: 6 · both fail: 5
34.4 pp[6.3, 59.4] pp0.027
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
BoolQ
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer139-1.01
15/32 · 46.9%
A only: 4 · B only: 11
Both pass: 7 · both fail: 10
21.9 pp[0.0, 43.8] pp0.118
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer139-1.01b
23/32 · 71.9%
A only: 5 · B only: 18
Both pass: 6 · both fail: 3
40.6 pp[12.5, 65.6] pp0.011
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
BoolQ
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer139-1.02
15/32 · 46.9%
A only: 4 · B only: 11
Both pass: 7 · both fail: 10
21.9 pp[0.0, 43.8] pp0.118
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer149
25/32 · 78.1%
A only: 7 · B only: 18
Both pass: 4 · both fail: 3
34.4 pp[6.3, 62.5] pp0.043
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
BoolQ
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer149-ARC
11/32 · 34.4%
A only: 3 · B only: 8
Both pass: 8 · both fail: 13
15.6 pp[-3.1, 34.4] pp0.227
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
20/32 · 62.5%
A only: 3 · B only: 17
Both pass: 8 · both fail: 4
43.8 pp[18.8, 65.6] pp0.003
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
BoolQ
A: HuggingFaceTB/SmolLM2-360M-Instruct
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
15/32 · 46.9%
A only: 3 · B only: 12
Both pass: 8 · both fail: 9
28.1 pp[6.3, 50.0] pp0.035
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
BoolQ
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer-149M-Decision-v1
15/32 · 46.9%
A only: 8 · B only: 7
Both pass: 15 · both fail: 2
-3.1 pp[-25.0, 18.8] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer139-1.01
15/32 · 46.9%
A only: 10 · B only: 5
Both pass: 13 · both fail: 4
-15.6 pp[-37.5, 6.3] pp0.302
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer139-1.01b
13/32 · 40.6%
A only: 6 · B only: 7
Both pass: 17 · both fail: 2
3.1 pp[-18.8, 25.0] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer139-1.02
15/32 · 46.9%
A only: 10 · B only: 5
Both pass: 13 · both fail: 4
-15.6 pp[-37.5, 6.3] pp0.302
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer149
15/32 · 46.9%
A only: 8 · B only: 7
Both pass: 15 · both fail: 2
-3.1 pp[-25.0, 18.8] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer149-ARC
13/32 · 40.6%
A only: 10 · B only: 3
Both pass: 13 · both fail: 6
-21.9 pp[-43.8, 0.0] pp0.092
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
16/32 · 50.0%
A only: 7 · B only: 9
Both pass: 16 · both fail: 0
6.3 pp[-18.8, 31.3] pp0.804
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: Qwen/Qwen2.5-0.5B-Instruct
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
17/32 · 53.1%
A only: 10 · B only: 7
Both pass: 13 · both fail: 2
-9.4 pp[-34.4, 15.6] pp0.629
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer139-1.01
10/32 · 31.3%
A only: 7 · B only: 3
Both pass: 15 · both fail: 7
-12.5 pp[-31.3, 6.3] pp0.344
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer139-1.01b
6/32 · 18.8%
A only: 2 · B only: 4
Both pass: 20 · both fail: 6
6.3 pp[-9.4, 21.9] pp0.688
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer139-1.02
10/32 · 31.3%
A only: 7 · B only: 3
Both pass: 15 · both fail: 7
-12.5 pp[-31.3, 6.3] pp0.344
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer149
6/32 · 18.8%
A only: 3 · B only: 3
Both pass: 19 · both fail: 7
0.0 pp[-15.6, 15.6] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer149-ARC
12/32 · 37.5%
A only: 9 · B only: 3
Both pass: 13 · both fail: 7
-18.8 pp[-40.6, 0.0] pp0.146
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
5/32 · 15.6%
A only: 1 · B only: 4
Both pass: 21 · both fail: 6
9.4 pp[-3.1, 21.9] pp0.375
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer-149M-Decision-v1
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
6/32 · 18.8%
A only: 4 · B only: 2
Both pass: 18 · both fail: 8
-6.3 pp[-21.9, 9.4] pp0.688
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer139-1.01
B: SlayerLab/Slayer139-1.01b
8/32 · 25.0%
A only: 1 · B only: 7
Both pass: 17 · both fail: 7
18.8 pp[3.1, 34.4] pp0.070
exploratory · unadjusted
Exploratory interval only
Resampling excludes zero, but the exact paired test does not resolve the difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer139-1.01
B: SlayerLab/Slayer139-1.02
0/32 · 0.0%
A only: 0 · B only: 0
Both pass: 18 · both fail: 14
0.0 pp[0.0, 0.0] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer139-1.01
B: SlayerLab/Slayer149
12/32 · 37.5%
A only: 4 · B only: 8
Both pass: 14 · both fail: 6
12.5 pp[-6.3, 34.4] pp0.388
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer139-1.01
B: SlayerLab/Slayer149-ARC
8/32 · 25.0%
A only: 5 · B only: 3
Both pass: 13 · both fail: 11
-6.3 pp[-21.9, 12.5] pp0.727
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer139-1.01
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
11/32 · 34.4%
A only: 2 · B only: 9
Both pass: 16 · both fail: 5
21.9 pp[3.1, 40.6] pp0.065
exploratory · unadjusted
Exploratory interval only
Resampling excludes zero, but the exact paired test does not resolve the difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer139-1.01
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
10/32 · 31.3%
A only: 4 · B only: 6
Both pass: 14 · both fail: 8
6.3 pp[-12.5, 25.0] pp0.754
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer139-1.01b
B: SlayerLab/Slayer139-1.02
8/32 · 25.0%
A only: 7 · B only: 1
Both pass: 17 · both fail: 7
-18.8 pp[-34.4, -3.1] pp0.070
exploratory · unadjusted
Exploratory interval only
Resampling excludes zero, but the exact paired test does not resolve the difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer139-1.01b
B: SlayerLab/Slayer149
6/32 · 18.8%
A only: 4 · B only: 2
Both pass: 20 · both fail: 6
-6.3 pp[-21.9, 9.4] pp0.688
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer139-1.01b
B: SlayerLab/Slayer149-ARC
14/32 · 43.8%
A only: 11 · B only: 3
Both pass: 13 · both fail: 5
-25.0 pp[-46.9, -3.1] pp0.057
exploratory · unadjusted
Exploratory interval only
Resampling excludes zero, but the exact paired test does not resolve the difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer139-1.01b
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
7/32 · 21.9%
A only: 3 · B only: 4
Both pass: 21 · both fail: 4
3.1 pp[-12.5, 18.8] pp1.000
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer139-1.01b
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
12/32 · 37.5%
A only: 8 · B only: 4
Both pass: 16 · both fail: 4
-12.5 pp[-34.4, 9.4] pp0.388
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer139-1.02
B: SlayerLab/Slayer149
12/32 · 37.5%
A only: 4 · B only: 8
Both pass: 14 · both fail: 6
12.5 pp[-6.3, 34.4] pp0.388
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer139-1.02
B: SlayerLab/Slayer149-ARC
8/32 · 25.0%
A only: 5 · B only: 3
Both pass: 13 · both fail: 11
-6.3 pp[-21.9, 12.5] pp0.727
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer139-1.02
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
11/32 · 34.4%
A only: 2 · B only: 9
Both pass: 16 · both fail: 5
21.9 pp[3.1, 40.6] pp0.065
exploratory · unadjusted
Exploratory interval only
Resampling excludes zero, but the exact paired test does not resolve the difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer139-1.02
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
10/32 · 31.3%
A only: 4 · B only: 6
Both pass: 14 · both fail: 8
6.3 pp[-12.5, 25.0] pp0.754
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer149
B: SlayerLab/Slayer149-ARC
16/32 · 50.0%
A only: 11 · B only: 5
Both pass: 11 · both fail: 5
-18.8 pp[-43.8, 6.3] pp0.210
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer149
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
9/32 · 28.1%
A only: 3 · B only: 6
Both pass: 19 · both fail: 4
9.4 pp[-9.4, 28.1] pp0.508
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer149
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
10/32 · 31.3%
A only: 6 · B only: 4
Both pass: 16 · both fail: 6
-6.3 pp[-25.0, 12.5] pp0.754
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer149-ARC
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
11/32 · 34.4%
A only: 1 · B only: 10
Both pass: 15 · both fail: 6
28.1 pp[9.4, 46.9] pp0.012
exploratory · unadjusted
Separation supported in this pilot
Resampling excludes zero and the exact paired test has p < 0.05; exploratory, not adjusted.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer149-ARC
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
6/32 · 18.8%
A only: 1 · B only: 5
Both pass: 15 · both fail: 11
12.5 pp[0.0, 28.1] pp0.219
exploratory · unadjusted
Inconclusive gap
Zero remains inside the interval; this pilot does not establish a directional difference.
Mixed difficulty for this pair
BoolQ
A: SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
B: SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
5/32 · 15.6%
A only: 5 · B only: 0
Both pass: 20 · both fail: 7
-15.6 pp[-28.1, -3.1] pp0.063
exploratory · unadjusted
Exploratory interval only
Resampling excludes zero, but the exact paired test does not resolve the difference.
Mixed difficulty for this pair

Plan a meaningful comparison

Choose an improvement for this task before running a confirmatory evaluation. Planning target: 80% power, two-sided α = 0.05. These are plug-in normal approximations from pilot variance, not measured power.

Benchmark / target pairCurrent paired SEApprox. detectable change at current nNominal items for 5 ppProjected evaluation cost
HumanEval
Qwen/Qwen2.5-0.5B ↔ Qwen/Qwen2.5-1.5B
5.94 pp16.6 pp355
Normal approximation; independent representative items assumed.
23.4 pair-minutes
Projected evaluator wall time; fixed setup + mean per-item time.
GSM8K
Qwen/Qwen2.5-0.5B ↔ Qwen/Qwen2.5-1.5B
6.25 pp17.5 pp393
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
Qwen/Qwen2.5-0.5B ↔ Qwen/Qwen2.5-1.5B
10.04 pp28.1 pp1013
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
Qwen/Qwen2.5-0.5B ↔ Qwen/Qwen2.5-1.5B
8.70 pp24.4 pp760
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
SST-2
Qwen/Qwen2.5-0.5B ↔ Qwen/Qwen2.5-1.5B
9.46 pp26.5 pp899
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
SNLI
Qwen/Qwen2.5-0.5B ↔ Qwen/Qwen2.5-1.5B
0.00 ppNot estimable from zero observed varianceNot estimable
No observed variance does not prove equivalence or perfect power.
Not estimated
Requires paired per-item timings.
AI2 ARC-Challenge (test)
Qwen/Qwen2.5-0.5B ↔ Qwen/Qwen2.5-1.5B
7.92 pp22.2 pp630
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
OpenBookQA main (test)
Qwen/Qwen2.5-0.5B ↔ Qwen/Qwen2.5-1.5B
6.35 pp17.8 pp406
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
HuggingFaceTB/SmolLM2-135M-Instruct ↔ HuggingFaceTB/SmolLM2-360M-Instruct
10.77 pp30.2 pp1165
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
HuggingFaceTB/SmolLM2-135M-Instruct ↔ Qwen/Qwen2.5-0.5B-Instruct
9.51 pp26.6 pp909
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer-149M-Decision-v1
11.00 pp30.8 pp1216
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer139-1.01
9.98 pp28.0 pp1001
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer139-1.01b
8.32 pp23.3 pp697
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer139-1.02
9.98 pp28.0 pp1001
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer149
11.67 pp32.7 pp1368
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer149-ARC
8.68 pp24.3 pp757
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
10.94 pp30.7 pp1203
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
9.98 pp28.0 pp1001
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
HuggingFaceTB/SmolLM2-360M-Instruct ↔ Qwen/Qwen2.5-0.5B-Instruct
9.10 pp25.5 pp833
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer-149M-Decision-v1
11.67 pp32.7 pp1368
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer139-1.01
10.47 pp29.3 pp1102
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer139-1.01b
10.94 pp30.7 pp1203
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer139-1.02
10.47 pp29.3 pp1102
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer149
11.00 pp30.8 pp1216
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer149-ARC
8.53 pp23.9 pp732
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
11.39 pp31.9 pp1304
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
10.47 pp29.3 pp1102
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer-149M-Decision-v1
8.38 pp23.5 pp706
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer139-1.01
8.38 pp23.5 pp706
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer139-1.01b
10.75 pp30.1 pp1162
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer139-1.02
8.38 pp23.5 pp706
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer149
9.10 pp25.5 pp833
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer149-ARC
8.32 pp23.3 pp697
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
8.38 pp23.5 pp706
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
7.08 pp19.8 pp504
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer139-1.01
6.25 pp17.5 pp393
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer139-1.01b
9.46 pp26.5 pp899
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer139-1.02
6.25 pp17.5 pp393
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer149
8.70 pp24.4 pp760
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer149-ARC
7.42 pp20.8 pp554
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
4.35 pp12.2 pp190
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
4.35 pp12.2 pp190
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer139-1.01 ↔ SlayerLab/Slayer139-1.01b
7.78 pp21.8 pp608
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer139-1.01 ↔ SlayerLab/Slayer139-1.02
0.00 ppNot estimable from zero observed varianceNot estimable
No observed variance does not prove equivalence or perfect power.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer139-1.01 ↔ SlayerLab/Slayer149
9.46 pp26.5 pp899
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer139-1.01 ↔ SlayerLab/Slayer149-ARC
7.92 pp22.2 pp630
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer139-1.01 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
4.49 pp12.6 pp203
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer139-1.01 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
6.35 pp17.8 pp406
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer139-1.01b ↔ SlayerLab/Slayer139-1.02
7.78 pp21.8 pp608
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer139-1.01b ↔ SlayerLab/Slayer149
8.91 pp25.0 pp798
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer139-1.01b ↔ SlayerLab/Slayer149-ARC
8.82 pp24.7 pp782
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer139-1.01b ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
8.98 pp25.2 pp811
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer139-1.01b ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
8.98 pp25.2 pp811
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer139-1.02 ↔ SlayerLab/Slayer149
9.46 pp26.5 pp899
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer139-1.02 ↔ SlayerLab/Slayer149-ARC
7.92 pp22.2 pp630
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer139-1.02 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
4.49 pp12.6 pp203
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer139-1.02 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
6.35 pp17.8 pp406
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer149 ↔ SlayerLab/Slayer149-ARC
8.53 pp23.9 pp732
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer149 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
9.46 pp26.5 pp899
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer149 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
9.46 pp26.5 pp899
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer149-ARC ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
7.92 pp22.2 pp630
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer149-ARC ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
6.52 pp18.3 pp428
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
ARC-Easy
SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
4.49 pp12.6 pp203
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
HuggingFaceTB/SmolLM2-135M-Instruct ↔ HuggingFaceTB/SmolLM2-360M-Instruct
8.32 pp23.3 pp697
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
HuggingFaceTB/SmolLM2-135M-Instruct ↔ Qwen/Qwen2.5-0.5B-Instruct
5.94 pp16.6 pp355
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer-149M-Decision-v1
7.08 pp19.8 pp504
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer139-1.01
6.25 pp17.5 pp393
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer139-1.01b
5.24 pp14.7 pp276
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer139-1.02
6.25 pp17.5 pp393
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer149
6.25 pp17.5 pp393
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer149-ARC
8.91 pp25.0 pp798
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
6.90 pp19.3 pp478
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
7.70 pp21.6 pp595
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
HuggingFaceTB/SmolLM2-360M-Instruct ↔ Qwen/Qwen2.5-0.5B-Instruct
7.70 pp21.6 pp595
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer-149M-Decision-v1
7.92 pp22.2 pp630
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer139-1.01
9.79 pp27.4 pp963
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer139-1.01b
9.38 pp26.3 pp883
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer139-1.02
9.79 pp27.4 pp963
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer149
9.79 pp27.4 pp963
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer149-ARC
9.79 pp27.4 pp963
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
8.23 pp23.1 pp681
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
8.70 pp24.4 pp760
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer-149M-Decision-v1
6.90 pp19.3 pp478
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer139-1.01
7.70 pp21.6 pp595
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer139-1.01b
7.08 pp19.8 pp504
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer139-1.02
7.70 pp21.6 pp595
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer149
7.70 pp21.6 pp595
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer149-ARC
8.91 pp25.0 pp798
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
7.08 pp19.8 pp504
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
7.70 pp21.6 pp595
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer139-1.01
7.08 pp19.8 pp504
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer139-1.01b
7.70 pp21.6 pp595
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer139-1.02
7.08 pp19.8 pp504
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer149
8.38 pp23.5 pp706
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer149-ARC
5.47 pp15.3 pp301
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
4.35 pp12.2 pp190
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
3.13 pp8.8 pp99
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer139-1.01 ↔ SlayerLab/Slayer139-1.01b
3.13 pp8.8 pp99
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer139-1.01 ↔ SlayerLab/Slayer139-1.02
0.00 ppNot estimable from zero observed varianceNot estimable
No observed variance does not prove equivalence or perfect power.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer139-1.01 ↔ SlayerLab/Slayer149
7.78 pp21.8 pp608
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer139-1.01 ↔ SlayerLab/Slayer149-ARC
6.35 pp17.8 pp406
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer139-1.01 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
7.08 pp19.8 pp504
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer139-1.01 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
6.35 pp17.8 pp406
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer139-1.01b ↔ SlayerLab/Slayer139-1.02
3.13 pp8.8 pp99
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer139-1.01b ↔ SlayerLab/Slayer149
7.08 pp19.8 pp504
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer139-1.01b ↔ SlayerLab/Slayer149-ARC
7.08 pp19.8 pp504
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer139-1.01b ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
6.35 pp17.8 pp406
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer139-1.01b ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
7.08 pp19.8 pp504
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer139-1.02 ↔ SlayerLab/Slayer149
7.78 pp21.8 pp608
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer139-1.02 ↔ SlayerLab/Slayer149-ARC
6.35 pp17.8 pp406
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer139-1.02 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
7.08 pp19.8 pp504
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer139-1.02 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
6.35 pp17.8 pp406
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer149 ↔ SlayerLab/Slayer149-ARC
7.78 pp21.8 pp608
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer149 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
8.38 pp23.5 pp706
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer149 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
8.98 pp25.2 pp811
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer149-ARC ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
5.47 pp15.3 pp301
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer149-ARC ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
4.49 pp12.6 pp203
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
HellaSwag
SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
3.13 pp8.8 pp99
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
HuggingFaceTB/SmolLM2-135M-Instruct ↔ HuggingFaceTB/SmolLM2-360M-Instruct
7.45 pp20.9 pp558
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
HuggingFaceTB/SmolLM2-135M-Instruct ↔ Qwen/Qwen2.5-0.5B-Instruct
11.88 pp33.3 pp1418
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer-149M-Decision-v1
14.19 pp39.7 pp2023
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer139-1.01
12.39 pp34.7 pp1542
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer139-1.01b
13.46 pp37.7 pp1820
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer139-1.02
12.39 pp34.7 pp1542
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer149
13.46 pp37.7 pp1820
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer149-ARC
12.08 pp33.8 pp1466
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
11.83 pp33.1 pp1406
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
HuggingFaceTB/SmolLM2-135M-Instruct ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
12.59 pp35.3 pp1592
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
HuggingFaceTB/SmolLM2-360M-Instruct ↔ Qwen/Qwen2.5-0.5B-Instruct
11.67 pp32.7 pp1368
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer-149M-Decision-v1
13.17 pp36.9 pp1744
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer139-1.01
11.65 pp32.6 pp1365
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer139-1.01b
13.36 pp37.4 pp1795
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer139-1.02
11.65 pp32.6 pp1365
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer149
14.63 pp41.0 pp2149
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer149-ARC
10.15 pp28.4 pp1035
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
11.83 pp33.1 pp1406
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
HuggingFaceTB/SmolLM2-360M-Instruct ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
11.21 pp31.4 pp1263
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer-149M-Decision-v1
12.28 pp34.4 pp1516
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer139-1.01
11.97 pp33.5 pp1441
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer139-1.01b
11.43 pp32.0 pp1314
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer139-1.02
11.97 pp33.5 pp1441
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer149
12.28 pp34.4 pp1516
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer149-ARC
10.75 pp30.1 pp1162
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
12.65 pp35.4 pp1608
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
Qwen/Qwen2.5-0.5B-Instruct ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
12.98 pp36.4 pp1694
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer139-1.01
9.79 pp27.4 pp963
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer139-1.01b
7.70 pp21.6 pp595
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer139-1.02
9.79 pp27.4 pp963
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer149
7.78 pp21.8 pp608
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer149-ARC
10.47 pp29.3 pp1102
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
6.90 pp19.3 pp478
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer-149M-Decision-v1 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
7.70 pp21.6 pp595
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer139-1.01 ↔ SlayerLab/Slayer139-1.01b
8.32 pp23.3 pp697
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer139-1.01 ↔ SlayerLab/Slayer139-1.02
0.00 ppNot estimable from zero observed varianceNot estimable
No observed variance does not prove equivalence or perfect power.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer139-1.01 ↔ SlayerLab/Slayer149
10.77 pp30.2 pp1165
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer139-1.01 ↔ SlayerLab/Slayer149-ARC
8.91 pp25.0 pp798
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer139-1.01 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
9.77 pp27.4 pp959
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer139-1.01 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
9.98 pp28.0 pp1001
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer139-1.01b ↔ SlayerLab/Slayer139-1.02
8.32 pp23.3 pp697
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer139-1.01b ↔ SlayerLab/Slayer149
7.70 pp21.6 pp595
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer139-1.01b ↔ SlayerLab/Slayer149-ARC
11.00 pp30.8 pp1216
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer139-1.01b ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
8.38 pp23.5 pp706
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer139-1.01b ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
10.77 pp30.2 pp1165
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer139-1.02 ↔ SlayerLab/Slayer149
10.77 pp30.2 pp1165
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer139-1.02 ↔ SlayerLab/Slayer149-ARC
8.91 pp25.0 pp798
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer139-1.02 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
9.77 pp27.4 pp959
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer139-1.02 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
9.98 pp28.0 pp1001
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer149 ↔ SlayerLab/Slayer149-ARC
12.25 pp34.3 pp1507
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer149 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
9.38 pp26.3 pp883
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer149 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
9.98 pp28.0 pp1001
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer149-ARC ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-45500
9.24 pp25.9 pp858
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer149-ARC ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
7.45 pp20.9 pp558
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.
BoolQ
SlayerLab/Slayer149-Balanced-ARC-E-ft-45500 ↔ SlayerLab/Slayer149-Balanced-ARC-E-ft-v2
6.52 pp18.3 pp428
Normal approximation; independent representative items assumed.
Not estimated
Requires paired per-item timings.

First-32 examples are not random or necessarily independent. Clustered questions need grouped resampling; stochastic decoding needs repeated generations. Do not treat nominal sample sizes as guarantees. The required number may exceed available independent tasks. A percentage point has a different meaning in different tasks. Item cost projections assume similar lengths, constant model settings and a fixed setup cost; token and task-group metadata are not yet complete.

Evidence profile

DimensionCurrent evidence / next validation
Target cohortThe exact paired baseline models shown above; this is not evidence about nearby 100M checkpoints.
Validity / coverageTask contract, pinned dataset and scoring protocol. Independent target-task validation and contamination auditing are not measured here.
Local discriminationAligned correctness disagreement, paired score difference and exploratory uncertainty. A wider score range alone is insufficient.
Floor / ceilingThe pair tables expose both-pass / both-fail counts for this local cohort; these are not population-wide item difficulties.
ReliabilityOne deterministic pilot per system. Prompt/decoding variation, repeated evaluations and training-seed variability are not estimated.
EfficiencyRecorded lease/process cost where available; sample-size and cost planning is explicitly projected.
ComplementarityIncremental held-out predictive value and suite redundancy require a larger shared model panel. Not estimated.
IRT / adaptive selectionNot fitted. First compare random and stratified subset budgets on held-out model families, then test whether IRT improves efficiency.

How to read these metrics

Cost: GPU lease seconds come from server claim-to-finish timestamps for one reserved GPU slot per job (one-second clock resolution); they include loading, polling and upload overhead. Historical upload-only or inconsistent leases are excluded, not interpreted as zero-cost evaluations. Instrumented GPU-slot seconds are evaluator subprocess wall seconds × one reserved GPU, including setup, data/model loading and CPU scoring. They are not kernel-active seconds. Historical evaluation elapsed is a distinct proxy and not retroactively relabelled. Only completed-run costs are aggregated here; failed-attempt logs remain in the private archive.

Uncertainty: Paired percentile resampling over aligned binary correctness; B minus A; exploratory, no multiple-comparison correction. First valid examples in pinned order, not a random representative sample. The interval describes sensitivity within this pilot, not population confidence on the full benchmark. We use 4000 paired resamples, seed 42. Identical outcomes can produce a zero-width interval without proving equivalence.

Useful information: Correctness disagreement and score separation describe this measured model pair. High disagreement alone does not prove a useful or reliable benchmark. Redundancy, ranking stability and broad information gain need a larger shared model panel and repeated/wider evaluations. This page does not turn these diagnostics into an automatic “best benchmark” score or change queue priority.

Next evidence to collect: a wider shared model panel, repeated runs where decoding is stochastic, broader subsets, ranking stability and incremental coverage beyond existing benchmarks. CER, WER and correlation need their own aggregation-aware diagnostics; binary correctness analysis is not applied to them.

Methodological reading: Adding Error Bars to Evals · tinyBenchmarks · How Reliable is Language Model Micro-Benchmarking? · MetaEval. These motivate further validation; they do not validate our pilot subsets.

Evaluation loop → · Public diagnostics JSON →

Measured batching speedups

Same pinned model, dataset scope and GPU; warmed inference only. Data/model loading and Docker testing excluded. Choice: median of 3 repeats; generation: one run. Numeric changes are possible even with greedy decoding.

Choice batch 16, math batch 8. Code generation remains serial because one 0.5B unit-test outcome changed. Media and embeddings keep existing runners. Agreement on these 32-item pilots is not a guarantee on future examples.

Model / benchmarkBatchSerial → batchedSpeedupChanged predictions / outcomesPeak GPU memory
Qwen/Qwen2.5-0.5B
reading · 32 examples
160.600s → 0.260s2.31×0 / 0
Choice labels
1.09 GiB
Qwen/Qwen2.5-0.5B
commonsense · 32 examples
161.125s → 0.164s6.87×0 / 0
Choice labels
1.13 GiB
Qwen/Qwen2.5-0.5B
sentiment · 32 examples
160.549s → 0.068s8.04×0 / 0
Choice labels
0.96 GiB
Qwen/Qwen2.5-0.5B
entailment · 32 examples
160.851s → 0.106s8.05×0 / 0
Choice labels
0.96 GiB
Qwen/Qwen2.5-0.5B
arc-challenge-choice · 32 examples
161.132s → 0.191s5.93×0 / 0
Choice labels
1.08 GiB
Qwen/Qwen2.5-0.5B
openbookqa-main-choice · 32 examples
161.093s → 0.117s9.32×0 / 0
Choice labels
1.00 GiB
Qwen/Qwen2.5-0.5B
math · 32 examples
831.587s → 11.094s2.85×0 / 0
Generated strings / scored outcomes
0.99 GiB
Qwen/Qwen2.5-0.5B
code-generation · 32 examples
853.382s → 11.159s4.78×3 / 1
Generated strings / scored outcomes
1.00 GiB
Qwen/Qwen2.5-1.5B
reading · 32 examples
161.019s → 0.663s1.54×0 / 0
Choice labels
3.17 GiB
Qwen/Qwen2.5-1.5B
commonsense · 32 examples
161.275s → 0.386s3.30×0 / 0
Choice labels
3.08 GiB
Qwen/Qwen2.5-1.5B
sentiment · 32 examples
160.647s → 0.160s4.05×0 / 0
Choice labels
2.93 GiB
Qwen/Qwen2.5-1.5B
entailment · 32 examples
160.965s → 0.251s3.84×0 / 0
Choice labels
2.93 GiB
Qwen/Qwen2.5-1.5B
arc-challenge-choice · 32 examples
161.320s → 0.471s2.80×0 / 0
Choice labels
3.03 GiB
Qwen/Qwen2.5-1.5B
openbookqa-main-choice · 32 examples
161.274s → 0.252s5.05×0 / 0
Choice labels
2.95 GiB
Qwen/Qwen2.5-1.5B
math · 32 examples
847.123s → 12.950s3.64×0 / 0
Generated strings / scored outcomes
3.00 GiB
Qwen/Qwen2.5-1.5B
code-generation · 32 examples
858.917s → 13.048s4.52×2 / 0
Generated strings / scored outcomes
3.00 GiB

Stable length grouping, a padded-token budget and left-padding masks avoid wasted work. Multiple-choice options share forwards; only answer-position logits are materialised. Math generation uses greedy batches. Models still load per job; resident model reuse and shared-prefix caching remain future optimisations. Raw profiles, repeats and source hashes →