Benchmark plan / gsm8k
GSM8K
math · instruct · en
adapter_ready · No sub-1B cohort score yet. Planning cost Ś–W; actual GPU time is not measured.
Pin data, item selection and scoring protocol before scheduling. Existing CodeSOTA pilot adapter is available; loading/scoring native Slayer models still needs validation for this task.
Primary benchmark source ↗ · Existing CodeSOTA adapter/pilot →
Before scheduling
Pin dataset revision, split and item IDs; validate the native-model adapter; set context and generation budgets; freeze prompts, decoding, scoring and any grader/environment. Report every compatible system in the same scope. Long-context or gated data may require further preparation.
Presets: liquid-reference. Qualified detail-page visits affect planning priority, not automatic budget expansion.
Ontology record
{
"number": 8,
"benchmark_id": "gsm8k",
"name": "GSM8K",
"benchmark_family": "gsm8k",
"task_ids": [
"math"
],
"group": "liquid-reference",
"source": "https://github.com/openai/grade-school-math",
"liquid": [
"LFM2"
],
"presets": [
"liquid-reference"
],
"languages": [
"en"
],
"base_or_instruct": "instruct",
"metric": "exact_match",
"direction": "max",
"dataset_revision": null,
"split": null,
"item_ids": null,
"protocol_id": null,
"generation_and_context_budget": {
"max_input_tokens": null,
"max_output_tokens": null,
"sample_count": null
},
"grader_or_environment": "Deterministic scoring adapter to validate",
"measured_cost": null,
"planning_cost": "Ś–W",
"status": "adapter_ready",
"note": "Pin data, item selection and scoring protocol before scheduling. Existing CodeSOTA pilot adapter is available; loading/scoring native Slayer models still needs validation for this task.",
"existing_adapter_task": "math",
"comparison_scope": "No sub-1B cohort result for this catalogue entry yet",
"execution_tier": 2,
"execution_priority_reason": "Liquid reference coverage",
"execution_order": 108,
"visits_30d": 0,
"demand": 0,
"priority": 0
}