Benchmark plan / hellaswag

HellaSwag

commonsense · base · en

pilot · 11/11 models measured in a 32-item pilot. Planning cost Ś; 53.44 measured GPU-slot seconds.

Pin data, item selection and scoring protocol before scheduling. Existing CodeSOTA pilot adapter is available; loading/scoring native Slayer models still needs validation for this task.

Primary benchmark source ↗ · Sub-1B cohort results & queue → · Existing CodeSOTA adapter/pilot →

Before scheduling

Pin dataset revision, split and item IDs; validate the native-model adapter; set context and generation budgets; freeze prompts, decoding, scoring and any grader/environment. Report every compatible system in the same scope. Long-context or gated data may require further preparation.

Presets: small-base. Qualified detail-page visits affect planning priority, not automatic budget expansion.

Ontology record

{
  "number": 24,
  "benchmark_id": "hellaswag",
  "name": "HellaSwag",
  "benchmark_family": "hellaswag",
  "task_ids": [
    "commonsense"
  ],
  "group": "small-base",
  "source": "https://huggingface.co/datasets/Rowan/hellaswag",
  "liquid": [],
  "presets": [
    "small-base"
  ],
  "languages": [
    "en"
  ],
  "base_or_instruct": "base",
  "metric": "accuracy",
  "direction": "max",
  "dataset_revision": null,
  "split": null,
  "item_ids": null,
  "protocol_id": null,
  "generation_and_context_budget": {
    "max_input_tokens": null,
    "max_output_tokens": null,
    "sample_count": null
  },
  "grader_or_environment": "Deterministic scoring adapter to validate",
  "measured_cost": 53.44,
  "planning_cost": "Ś",
  "status": "pilot",
  "note": "Pin data, item selection and scoring protocol before scheduling. Existing CodeSOTA pilot adapter is available; loading/scoring native Slayer models still needs validation for this task.",
  "existing_adapter_task": "commonsense",
  "comparison_scope": "b4821eeee94ae676351fa0a7023207ccfb0c15fd7747debe3ffa69d04edf6456",
  "execution_tier": 1,
  "execution_priority_reason": "Core small-model signal",
  "cohort_profile_id": "sub1b-hellaswag",
  "execution_order": 2,
  "cohort_progress": {
    "completed": 11,
    "total": 11,
    "running": 0,
    "queued": 0,
    "failed": 0
  },
  "measured_cost_unit": "gpu_slot_seconds",
  "visits_30d": 0,
  "demand": 0,
  "priority": 0
}