Measurement validity
Match the benchmark to the job.
Model cards now distinguish runtime compatibility, intended-task relevance and evidence of scientific validity. Checks are tied to exact model revisions.
Decision models need native protocols
A decision model may emit option probabilities, a binary probability or an ordinal distribution. Others generate routing JSON. A generic ARC score does not measure routing usefulness.
- Pin the state/question format, output interface and option mapping.
- Freeze prompts and fit calibration only on separate validation data.
- Measure accuracy and probability quality; test label permutations, unseen domains and abstention.
- Compare cost-sensitive routing policies and latency against simple baselines.
- Review benchmark training exposure before interpreting gains.
What the labels mean
- Blocked output interface: the reviewed head/runtime does not support this text evaluator.
- Diagnostic only: generic text results do not establish specialist task quality.
- Task fit unverified: a general diagnostic is available; validity for a user’s target job still needs validation.
- Unreviewed: no contract has been reviewed for this exact revision.
These labels are evidence limits, not automated validity certification. No decision-specific adapter is claimed ready by these checks.