Visionary / proposed direction
Where CodeSOTA goes next.
A model-selection service built around reproducible measurements and useful evidence.
This page is a proposed direction. It is not a list of shipped features, an execution queue or a delivery promise.
See verified changes.
Version 1 · 18 Oct 2026 · 13:19 UTC
Help people choose models and benchmarks with clear evidence
Start from a concrete task: pick a benchmark and a model pair, see cost versus discrimination with pilot scope marked, and leave with one actionable conclusion. Prioritize the weak /benchmark-value path before adding anything new.
Proposal
Make /benchmark-value answer a decision, not list 1645 rows
The page serves 1645 rows and 1.28 MB in 12.2 s with no visible filters, so pilots and full protocols blur together and a conclusion requires scrolling hundreds of rows.
Next experiment
Test a summary view: top separations per benchmark, cost per separating example, benchmark and model-pair filters, capped first page with progressive disclosure. Measure time-to-conclusion for picking a cheaper benchmark that separates two named small models.
Open question
Findings come from structural HTML analysis only; no rendered-browser or user study evidence yet.
Measurement completed: HellaSwag (full validation, character-normalized accuracy) · Readability settings updated
Proposal
Mark every number with its scope: pilot, full split, or cohort
Pilot counts of 32 examples sit beside full-protocol rows with no per-row marker, and home rows mix scopes and outlier interest scales, so a skimmer may treat a 4-of-32 signal as a stable benchmark property.
Next experiment
Test per-row scope badges, headings that name the pilot scope, and grouping rows by scope on home. Verify by asking a reader which figures come from full splits and checking they cite only pilot-marked rows.
Open question
Grouping may already exist below the excerpt; verify the rendered page before changing anything.
Measurement completed: BoolQ (full validation, raw likelihood accuracy) · Usability agent: first recorded completed cycle
Version history
- v1 · 18 Oct 2026 · 13:19 UTC · Help people choose models and benchmarks with clear evidence
Usability findings · Vision JSON · Agent activity
Vision items are proposals, not measured results, commitments, or completed work. Evidence links are supplied by the server.