Benchmark plan / arc-agi

ARC-AGI-1

grid-reasoning · base + instruct; solver protocol must be fixed · integer grids; language-independent task data

planned · No sub-1B cohort score yet. Planning cost W; actual GPU time is not measured.

Requested addition. Separate from AI2 ARC-Easy/Challenge science QA. Pin a grid serializer, context policy, attempt budget and decoding before running. Upstream task JSON supplies demonstration pairs and test inputs; hide all test outputs from the solver. Public evaluation is distinct from private ARC Prize scores. No executable adapter or measured result yet.

Primary benchmark source ↗

Before scheduling

Pin dataset revision, split and item IDs; validate the native-model adapter; set context and generation budgets; freeze prompts, decoding, scoring and any grader/environment. Report every compatible system in the same scope. Long-context or gated data may require further preparation.

Presets: Optional additional coverage. Qualified detail-page visits affect planning priority, not automatic budget expansion.

Ontology record

{
  "number": 51,
  "benchmark_id": "arc-agi-1",
  "name": "ARC-AGI-1",
  "benchmark_family": "arc-agi",
  "task_ids": [
    "grid-reasoning"
  ],
  "group": "reasoning",
  "source": "https://github.com/fchollet/ARC-AGI",
  "liquid": [],
  "presets": [],
  "languages": [
    "integer grids; language-independent task data"
  ],
  "base_or_instruct": "base + instruct; solver protocol must be fixed",
  "metric": "exact_task_success",
  "direction": "max",
  "dataset_revision": "399030444e0ab0cc8b4e199870fb20b863846f34",
  "split": "data/evaluation (public, 400 tasks)",
  "item_ids": null,
  "protocol_id": null,
  "generation_and_context_budget": {
    "max_input_tokens": null,
    "max_output_tokens": null,
    "sample_count": null
  },
  "grader_or_environment": "Grid-generation adapter required; strict rectangular integer-grid validation and exact equality for every test input",
  "measured_cost": null,
  "planning_cost": "W",
  "status": "planned",
  "note": "Requested addition. Separate from AI2 ARC-Easy/Challenge science QA. Pin a grid serializer, context policy, attempt budget and decoding before running. Upstream task JSON supplies demonstration pairs and test inputs; hide all test outputs from the solver. Public evaluation is distinct from private ARC Prize scores. No executable adapter or measured result yet.",
  "existing_adapter_task": null,
  "comparison_scope": "No sub-1B cohort result for this catalogue entry yet",
  "execution_tier": 1,
  "execution_priority_reason": "User-requested abstract grid reasoning; adapter work first",
  "execution_order": 9,
  "visits_30d": 0,
  "demand": 0,
  "priority": 0
}