Benchmark plan / openbookqa
OpenBookQA
knowledge · base · en
adapter_ready · No sub-1B cohort score yet. Planning cost N; actual GPU time is not measured.
Pin data, item selection and scoring protocol before scheduling. Existing CodeSOTA pilot adapter is available; loading/scoring native Slayer models still needs validation for this task.
Primary benchmark source ↗ · Existing CodeSOTA adapter/pilot →
Before scheduling
Pin dataset revision, split and item IDs; validate the native-model adapter; set context and generation budgets; freeze prompts, decoding, scoring and any grader/environment. Report every compatible system in the same scope. Long-context or gated data may require further preparation.
Presets: small-base. Qualified detail-page visits affect planning priority, not automatic budget expansion.
Ontology record
{
"number": 27,
"benchmark_id": "openbookqa",
"name": "OpenBookQA",
"benchmark_family": "openbookqa",
"task_ids": [
"knowledge"
],
"group": "small-base",
"source": "https://github.com/allenai/OpenBookQA",
"liquid": [],
"presets": [
"small-base"
],
"languages": [
"en"
],
"base_or_instruct": "base",
"metric": "accuracy",
"direction": "max",
"dataset_revision": null,
"split": null,
"item_ids": null,
"protocol_id": null,
"generation_and_context_budget": {
"max_input_tokens": null,
"max_output_tokens": null,
"sample_count": null
},
"grader_or_environment": "Deterministic scoring adapter to validate",
"measured_cost": null,
"planning_cost": "N",
"status": "adapter_ready",
"note": "Pin data, item selection and scoring protocol before scheduling. Existing CodeSOTA pilot adapter is available; loading/scoring native Slayer models still needs validation for this task.",
"existing_adapter_task": "openbookqa-main-choice",
"comparison_scope": "No sub-1B cohort result for this catalogue entry yet",
"execution_tier": 1,
"execution_priority_reason": "Core small-model signal",
"execution_order": 8,
"visits_30d": 0,
"demand": 0,
"priority": 0
}