Benchmark plan / boolq
BoolQ
reading · instruct · en
pilot · 11/11 models measured in a 32-item pilot. Planning cost N–Ś; 80.19 measured GPU-slot seconds.
Pin data, item selection and scoring protocol before scheduling. Existing CodeSOTA pilot adapter is available; loading/scoring native Slayer models still needs validation for this task.
Primary benchmark source ↗ · Sub-1B cohort results & queue → · Existing CodeSOTA adapter/pilot →
Before scheduling
Pin dataset revision, split and item IDs; validate the native-model adapter; set context and generation budgets; freeze prompts, decoding, scoring and any grader/environment. Report every compatible system in the same scope. Long-context or gated data may require further preparation.
Presets: Optional additional coverage. Qualified detail-page visits affect planning priority, not automatic budget expansion.
Ontology record
{
"number": 40,
"benchmark_id": "boolq",
"name": "BoolQ",
"benchmark_family": "boolq",
"task_ids": [
"reading"
],
"group": "reading-context",
"source": "https://github.com/google-research-datasets/boolean-questions",
"liquid": [],
"presets": [],
"languages": [
"en"
],
"base_or_instruct": "instruct",
"metric": "accuracy",
"direction": "max",
"dataset_revision": null,
"split": null,
"item_ids": null,
"protocol_id": null,
"generation_and_context_budget": {
"max_input_tokens": null,
"max_output_tokens": null,
"sample_count": null
},
"grader_or_environment": "Deterministic scoring adapter to validate",
"measured_cost": 80.19,
"planning_cost": "N–Ś",
"status": "pilot",
"note": "Pin data, item selection and scoring protocol before scheduling. Existing CodeSOTA pilot adapter is available; loading/scoring native Slayer models still needs validation for this task.",
"existing_adapter_task": "reading",
"comparison_scope": "bea6197d37b7692e8660c09f3be4bdc84c65bd6d240a6e23f418cc1d58d63ca5",
"execution_tier": 1,
"execution_priority_reason": "Core small-model signal",
"cohort_profile_id": "sub1b-boolq",
"execution_order": 3,
"cohort_progress": {
"completed": 11,
"total": 11,
"running": 0,
"queued": 0,
"failed": 0
},
"measured_cost_unit": "gpu_slot_seconds",
"visits_30d": 0,
"demand": 0,
"priority": 0
}