Ontology / current system
From interest
to measured evidence.
People help choose what matters. Runtime agents find and prepare benchmarks. White runs the evaluations. Every published score points back to evidence.
The evaluation loop
Select a step to explore it. On smaller screens, swipe the diagram or use the step buttons below.
Selected step
01 People explore
Server-side analytics counts public HTML responses and estimates daily unique visitors using a rotating-key HMAC of IP + User-Agent. Counts are public aggregates; no raw IPs or User-Agent strings are stored in the analytics tables. Daily uniques are estimates, and their sum across days is visitor-days. Separately, beta task visits qualify after 2.5 seconds on a visible page, with deduplication, bot filtering and privacy controls. Original CodeSOTA’s top-100 page counters refresh hourly. These are planning signals, not unique-person analytics. Visits to documentation and run pages appear in the popularity table; task-page visits drive task demand.
Public usage leaderboard →What a score means
A score belongs to a model and one exact benchmark scope. The task is the starting point; the run is the evidence behind the result.
| Entity | Meaning / relationship | Example in this beta |
|---|---|---|
| Capability area | Groups tasks for discovery. | Code & Software Engineering |
| Task | Defines inputs → outputs; has benchmarks. | Function signature + description → code completion |
| Dataset | Versioned evaluation examples and reference answers; used by a benchmark. | Pinned HumanEval data, test split |
| Benchmark | Task + dataset revision / split / selection + metric + protocol + evaluator version. | CodeSOTA HumanEval 32-example pilot · fraction passing all tests |
| Model | The exact system evaluated, pinned by model revision. | Qwen2.5-0.5B at a specific model commit |
| Run | One execution attempt linking a model to a benchmark; records configuration and evidence. | White GPU job with timestamps, outputs and evaluator hash |
| Result | Completed model + benchmark + score + configuration + source + date. | Measured passing fraction on the pinned pilot scope |
| Source | Supports a definition or result: upstream benchmark origin and our execution artifacts. | Primary dataset repository; samples, environment and SHA-256 manifest |
| Leaderboard / chart | Compares results within the same compatible scope. | Task result table and model size vs score scatter plot |
V1 direction / distinct from the current pilot registry
Choose a model. Follow the evidence.
The target V1 closes the path from a practical question to comparable results and their sources, within a limited set of task families. It keeps useful guides from the original site.
Three page templates
Task: inputs → outputs, benchmarks and relevant guides.
Benchmark: what it measures, protocol-specific results and sources.
Guide / comparison: practical choices, limitations, examples and links to evidence.
Two independent result labels
Origin: measured by CodeSOTA / reported by a source.
Scope: full benchmark / subset / unknown.
A reported result does not require a CodeSOTA run. Compatible protocols can share a ranking; other results belong in separate context groups.
| V1 element | Minimum relationship |
|---|---|
| Task | Stable ID, name, description, input and output types. Categories are navigation groups. |
| Benchmark | Stable ID, primary source and one or more related tasks. Discovering a benchmark does not define a new task. |
| Protocol | Dataset revision / split / subset, metric, prompt and scoring rules; fixes comparison compatibility. |
| System | A versioned model or pipeline, optionally with execution configuration. |
| Result | System + benchmark/protocol + score + date + source, origin and scope; optional run_id. |
| Guide | Practical content linked to tasks and benchmarks. “OCR for Python” is a guide, not a separate task. |
The current beta publishes our measured pilot runs and represents newly activated benchmarks as registry rows. The separate V1 content model, external-result ingestion and guide migration are not implemented yet.
Start the content migration with OCR / document AI, then embeddings, speech and coding. Preserve useful original URLs and guides while connecting them to the new result views. Existing original pages remain available until their sections are migrated.
Original OCR for Python guide ↗ · Handwriting guide ↗ · TTS guide ↗
Current boundaries
Measured pilots
Results use the first 32 valid examples. They are diagnostic subset scores, not official full-benchmark scores. Popularity changes work priority; it never changes scoring or makes a proposal into a result.
Explicit adapter work
Existing evaluators run the baseline task families. New multiple-choice recipes can be prepared automatically. Other discovered protocols require an implemented adapter and remain in the backlog.
SOTA picks
The beta does not publish a global SOTA pick. The best pilot result is only best among measured models within that exact comparison scope.
Versioning and lineage
Dataset/model commits, protocol settings and evaluator hashes identify versions and comparison scopes. An explicit graph connecting benchmark predecessors and successors is not implemented yet.