{"api_version":"codesota.v1","data":{"agents":[{"latest":{"finished":1792200471,"id":"scout-20261017012730-7fe4de","role":"scout","runtime":"Runtime","started":1792200450,"status":"completed","summary":"Runtime cycle completed; 9 tool calls.","tool_calls":9},"role":"scout"},{"latest":{"finished":1792202442,"id":"operator-20261017020028-310e37","role":"operator","runtime":"Runtime","started":1792202428,"status":"completed","summary":"Runtime cycle completed; 3 tool calls.","tool_calls":3},"role":"operator"},{"latest":{"finished":1792203907,"id":"debugger-20261017022504-372cc8","role":"debugger","runtime":"Runtime","started":1792203904,"status":"completed","summary":"Runtime diagnosis saved: 11 recent cycles inspected, 1 historical failures. Observation only; GPU jobs unchanged.","tool_calls":1},"role":"debugger"}],"backlog":[{"created":1792195951,"dataset":"princeton-nlp/SWE-bench_Verified","error":"ValueError: Needs a new adapter: automatic execution currently supports declarative multiple-choice schemas","id":"swe-bench-verified-code","parent_task":"code-generation","priority":54.5,"reason":"Code is the #2 original-CodeSOTA demand topic (933 month views; /tasks/swe-bench alone 235) and current coverage stops at HumanEval function synthesis \u2014 no repository-level issue resolution, the capability the /tasks/swe-bench page actually tracks. SWE-bench Verified (princeton-nlp primary authors, 500 OpenAI-validated instances) is the primary benchmark for patch generation from a real issue: model edits the repo at base_commit, then FAIL_TO_PASS must pass and PASS_TO_PASS must not regress. Needs adapter: execution harness must clone each repo at base_commit, apply candidate patch (model diff vs fields patch/test_patch), build env at environment_setup_commit, and run the FAIL_TO_PASS / PASS_TO_PASS test lists per instance \u2014 declarative choice scoring cannot represent this. Inspection (snapshot c104f840) confirms: ungated, 13 fields incl. instance_id, base_commit, patch, test_patch, problem_statement, FAIL_TO_PASS, PASS_TO_PASS, environment_setup_commit, difficulty; single shard data/test-00000-of-00001.parquet (2,096,679 bytes), split test. Access limitation stated honestly: HF card license metadata is null \u2014 the SWE-bench GitHub repo code is MIT but instances derive from 12 upstream repos with differing licenses; downstream runs must honor each instance's repo. Adapter work (repo clone + test execution) is significant; nothing measured until it exists.","recipe":{"config":"default","data_file":"data/test-00000-of-00001.parquet","kind":"needs_adapter","split":"test"},"source":"https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified","status":"needs_adapter","title":"SWE-bench Verified (test)","updated":1792197023},{"created":1792194975,"dataset":"gorilla-llm/Berkeley-Function-Calling-Leaderboard","error":"ValueError: Needs a new adapter: automatic execution currently supports declarative multiple-choice schemas","id":"bfcl-function-calling","parent_task":"code-generation","priority":54.5,"reason":"Emerging-capability slot: structured function/tool calling is absent from coverage; GAIA (needs_adapter, reading) covers general web/file agents but not API-call correctness, which is a distinct agent axis. BFCL is the primary-author (Gorilla team, UC Berkeley) benchmark for function-call generation, scored by executable and AST checks per category incl. multi-turn categories - not declarative choice, so needs_adapter with a live execution monitor harness per category. Parent task code-generation is the closest existing bucket since function-call JSON generation is structured code output. Access confirmed by inspection (snapshot 61fc0608): ungated, license Apache-2.0. Limitation stated honestly: the datasets viewer errored (ValueError) and no data files or ground truths were surfaced by inspection, so data and GT files must be resolved from the Gorilla project repo before pinning; nothing measured yet.","recipe":{"kind":"needs_adapter"},"source":"https://huggingface.co/datasets/gorilla-llm/Berkeley-Function-Calling-Leaderboard","status":"needs_adapter","title":"BFCL function calling (Berkeley Function-Calling Leaderboard)","updated":1792195236},{"created":1792194975,"dataset":"TangRain/SingMOS","error":"ValueError: Needs a new adapter: automatic execution currently supports declarative multiple-choice schemas","id":"singmos-speech-quality","parent_task":"transcription","priority":42.0,"reason":"TTS is a top original-CodeSOTA demand topic (927 month views, /guides/tts-models 471) with zero coverage. SingMOS (TangRain primary authors, SingMOSv2) provides human-annotated MOS for synthesized sung speech; the measurable benchmark metric is MOS-quality prediction per utterance against human MOS (MSE + Pearson/Spearman). Scope discipline: this ranks speech quality ASSESSORS; it is NOT an end-to-end text-to-waveform synthesis score and MUST NOT be substituted by or into speech-transcription WER (TTS is a separate task from ASR per policy); scoring a real TTS generator would additionally need a generation-then-prediction harness. Needs adapter: baked-choice scoring cannot represent regression over audio; the adapter must load clips, obtain predicted MOS, and compare with column mos (MSE/LCC/SRCC). Access and schema verified by inspection at snapshot a0fac9f7: ungated, license CC-BY-4.0, fields utt_id (string), audio (Audio), mos (float64), split (string; sample value valid); the viewer serves config default/split train. Limitation honestly stated: no parquet filename was surfaced by inspection (data_files empty), so the adapter must resolve the parquet path before pinning.","recipe":{"kind":"needs_adapter"},"source":"https://huggingface.co/datasets/TangRain/SingMOS","status":"needs_adapter","title":"SingMOS sung-speech MOS prediction (TTS-side quality)","updated":1792195236},{"created":1792194975,"dataset":"opendatalab/OmniDocBench","error":"ValueError: Needs a new adapter: automatic execution currently supports declarative multiple-choice schemas","id":"omnidocbench-document-parsing","parent_task":"handwriting","priority":35.13,"reason":"Documents/OCR is the top-original-CodeSOTA demand topic (2671 month views; /benchmark/omnidocbench alone 276 views) and current coverage has only handwriting recognition, so a real document-parsing benchmark fills the highest-value gap. OmniDocBench (arXiv:2412.07626, opendatalab primary authors, 46,480 downloads) is the primary-author end-to-end document-parsing benchmark for real-world PDF pages, scored by text/table/formula reconstruction quality against page-level ground truth. Needs adapter: declarative choice scoring cannot represent image->markdown parsing; the harness must render page images, run the parser, and score text plus table/formula reconstruction against ground truth with the paper's normalization. Inspection (snapshot aa1ee96d) shows datasets-viewer does not serve it: only a single Image feature, no parquet data files, no ground-truth fields and license metadata null, while the repo itself is ungated. Ground truth therefore must be sourced outside the viewer (project GitHub per arXiv:2412.07626) and card license must be confirmed from the project repo before any pinning. No score claimed until an adapter exists; access limitation stated honestly: ungated but GT location unconfirmed.","recipe":{"kind":"needs_adapter"},"source":"https://huggingface.co/datasets/opendatalab/OmniDocBench","status":"needs_adapter","title":"OmniDocBench document parsing","updated":1792197023},{"created":1792195951,"dataset":"rafaelpadilla/coco2017","error":"ValueError: Needs a new adapter: automatic execution currently supports declarative multiple-choice schemas","id":"coco2017-detection","parent_task":"image-classification","priority":35.12,"reason":"Vision is a top-4 original-CodeSOTA topic (866 month views) and /browse/computer-vision/object-detection/coco draws 289 month views, yet coverage has only CIFAR-10 classification \u2014 no localization/detection. COCO 2017 val (5k images, 80 classes) is the standard detection benchmark, scored by bbox mAP@[.5:.95] against ground-truth boxes, a fundamentally different axis from classification accuracy. Needs adapter: declarative choice scoring cannot represent per-image box prediction; the adapter must load images, run a detector, and compute COCO-style mAP with the official pycocotools protocol (IoU matching, per-category AP, score averaging). Inspection (snapshot 7d5136d6) confirms: ungated, splits train/val, schema image (Image), image_id (int64), objects sequence with id, area, bbox (4x float32 [x,y,w,h]), label (ClassLabel, 81 names incl. index 0 = 'None'), iscrowd (bool); sample row image_id 147328 with one bbox served by the viewer. Access limitation stated honestly: inspection returned no parquet data_files ({}), so the adapter must resolve underlying image/annotation files before pinning; HF card license metadata is null \u2014 cocodataset.org releases annotations for research and images under Flickr terms of use, which must be honored. No score claimed until an adapter exists.","recipe":{"config":"default","kind":"needs_adapter","split":"val"},"source":"https://huggingface.co/datasets/rafaelpadilla/coco2017","status":"needs_adapter","title":"COCO 2017 object detection (val)","updated":1792198855},{"created":1792200466,"dataset":"allenai/sciq","error":"ValueError: Needs a new adapter: automatic execution currently supports declarative multiple-choice schemas","id":"sciq-knowledge","parent_task":"commonsense","priority":6.95,"reason":"Core sub-1B catalogue entry (tier 1, order 10, planned) still uncovered; catalogue note requires choosing context-supported vs context-free protocol explicitly \u2014 this proposal defers that pin to the Operator. SciQ (allenai primary authors) is crowdsourced 13.7k science MC questions with supporting passages; tests scientific knowledge distinct from ARC exam style. Inspection snapshot 2c94ad3e: ungated, license cc-by-nc-3.0 (noncommercial \u2014 must be honored downstream); single-shard parquet data/{train,validation,test}-00000-of-00001.parquet (test 342,808 bytes). Verified features: question, distractor1/2/3, correct_answer (strings), support. No pre-joined choices array or answer index exists in the parquet, so the declarative choice recipe cannot represent it faithfully; adapter must assemble the 4 options from correct_answer + distractors with a pinned, seeded shuffle and support-context policy (with vs without), then score accuracy. Access: public, ungated.","recipe":{"config":"default","data_file":"data/test-00000-of-00001.parquet","kind":"needs_adapter","split":"test"},"source":"https://huggingface.co/datasets/allenai/sciq","status":"needs_adapter","title":"SciQ science QA (test)","updated":1792200654},{"created":1792200441,"dataset":"allenai/winogrande","error":"ValueError: Needs a new adapter: automatic execution currently supports declarative multiple-choice schemas","id":"winogrande-validation","parent_task":"commonsense","priority":6.95,"reason":"Core sub-1B catalogue entry (tier 1, order 9, planned): Winograd-family 2-choice pronoun resolution; sub-1B pilots cover ARC-Easy/HellaSwag/BoolQ, so this fills the workstream's planned commonsense slot without duplicating a measured profile. Parent commonsense is the exact bucket. Inspection snapshot 01e74176: ungated; parquet shards verified for configs xs/s/m/l/xl/debiased across train/validation/test; scoring file winogrande_xl/validation-00000-of-00001.parquet (85,928 bytes). Features verified: sentence, option1, option2 (strings), answer (1/2). Sampled row shows empty answer, consistent with withheld upstream test labels; protocol scores validation and must assert label presence per row. Needs adapter: options are separate flat fields, not a nested choices list, so the declarative choice schema cannot express them; adapter flattens both options, scores sentence likelihood per filled blank, accuracy on validation. License metadata null on HF card; confirm upstream allenai terms before pinning.","recipe":{"config":"winogrande_xl","data_file":"winogrande_xl/validation-00000-of-00001.parquet","kind":"needs_adapter","split":"validation"},"source":"https://huggingface.co/datasets/allenai/winogrande","status":"needs_adapter","title":"WinoGrande winogrande_xl (validation)","updated":1792200654},{"created":1792200441,"dataset":"TAUR-Lab/MuSR","error":"ValueError: Needs a new adapter: automatic execution currently supports declarative multiple-choice schemas","id":"musr-narrative-reasoning","parent_task":"reading","priority":0.07,"reason":"Emerging-capability slot: multistep soft reasoning over long narratives (murder mystery, object placement, team allocation) \u2014 no current coverage tests long-narrative deductive reasoning; complements planned BBH. TAUR-Lab primary authors (arXiv 2310.16049). Inspection snapshot 7c365b43: ungated, license cc-by-4.0; splits murder_mysteries, object_placements, team_allocation; features narrative, question, choices, answer_index (int64), answer_choice; sample row verified (2-choice murder mystery, answer_index 0). Metric is multiple-choice accuracy by answer_index, scored per split with no pooling. Needs adapter: choices is a stringified Python list, not a list field, and data_files came back empty (no parquet surfaced), so the baked choice recipe cannot score it; adapter must resolve the data files, parse choices, and score answer_index. Parent reading = long-narrative QA; commonsense is the alternate bucket. Floor risk stated honestly: narrative length may not separate sub-1B models; no discriminative claim.","recipe":{"config":"default","kind":"needs_adapter"},"source":"https://huggingface.co/datasets/TAUR-Lab/MuSR","status":"needs_adapter","title":"MuSR narrative multistep reasoning","updated":1792202434},{"created":1792195954,"dataset":"lmms-lab/Video-MME","error":"ValueError: Needs a new adapter: automatic execution currently supports declarative multiple-choice schemas","id":"videomme-video-understanding","parent_task":"reading","priority":0.07,"reason":"Emerging/underrepresented slot: zero video or multimodal coverage exists in the backlog, and every current task is text-, audio-, or static-image-only. Video-MME (lmms-lab primary authors, arXiv:2405.21015) is the primary long-form video-understanding benchmark: 900 YouTube videos, 2,700 multiple-choice QA pairs spanning 6 visual task types (recognition, OCR, object recognition, counting, temporal, plotting), durations 11s-1h with/without subtitles. Schema is declarative MC but scoring is NOT a baked choice recipe: the adapter must fetch each video (external YouTube source, not in the repo), extract frames/subtitles, run the multimodal model, and score the A/B/C/D answer \u2014 video retrieval and frame sampling are outside declarative choice execution. Inspection (snapshot ead1408f) confirms: ungated, one parquet videomme/test-00000-of-00001.parquet (405,167 bytes); limitation stated honestly: the datasets viewer raised ValueError so features, splits and license were NOT surfaced by inspection \u2014 schema must be re-verified before pinning and the parquet likely holds metadata/questions only, with videos fetched from YouTube at run time (deployment cost plus YouTube ToU applies). No score claimed until an adapter exists.","recipe":{"config":"videomme","data_file":"videomme/test-00000-of-00001.parquet","kind":"needs_adapter","split":"test"},"source":"https://huggingface.co/datasets/lmms-lab/Video-MME","status":"needs_adapter","title":"Video-MME video understanding (test)","updated":1792198855},{"created":1792193796,"dataset":"gaia-benchmark/GAIA","error":"ValueError: Needs a new adapter: automatic execution currently supports declarative multiple-choice schemas","id":"gaia-validation-agent","parent_task":"reading","priority":0.07,"reason":"Emerging-capability pick: GAIA (arXiv:2311.12983) is the primary-author general-assistant agent benchmark \u2014 real-world questions requiring web browsing, tool use, multimodal/file reasoning and multi-step planning, scored by exact-match of a single final answer per question. Nothing agentic exists in current coverage, so it fills an underrepresented capability. Declarative choice scoring does not apply: answers are free-form strings and questions need an execution harness (tool/browser access, file handling, answer extraction), hence needs_adapter. Adapter work: orchestrate per-question agent runs against validation metadata.parquet, extract the final answer, compare case-normalized exact match; also verify field names (e.g. Question, Final answer, level, task_name) since inspection could not return schema (viewer HTTPError) and split-level files 2023/validation/metadata.level1.parquet through level3.parquet allow leveled reporting. Access: repository is gated (auto-approval gate; credentials required to download) and no license is declared on the card \u2014 both stated honestly; pilot runs may fail until gate access is granted. data_file 2023/validation/metadata.parquet (116,994 bytes) is a real file confirmed by inspection. Access is gated; adapter/access work may be required.","recipe":{"data_file":"2023/validation/metadata.parquet","kind":"needs_adapter"},"source":"https://huggingface.co/datasets/gaia-benchmark/GAIA","status":"needs_adapter","title":"GAIA (2023 validation)","updated":1792193899},{"created":1792193796,"dataset":"allenai/openbookqa","error":null,"id":"openbookqa-main-choice","parent_task":"reading","priority":0.07,"reason":"Primary-source AI2 open-book science QA: 4-way questions built from a core science fact book plus broad common knowledge; tests retrieval-plus-application, complementary to closed-book ARC-Challenge also proposed this cycle. Declarative choice scored by accuracy on the 500-question main test. Public and ungated; repository license metadata is \"unknown\" (2018 AI2 release, no explicit license tag) and must be honored downstream. data_file main/test-00000-of-00001.parquet confirmed by inspection (55,535 bytes, single shard). Fields question_stem, choices.text, choices.label, answerKey are the canonical openbookqa main-config schema documented on the dataset card; inspection viewer errored (KeyError), so schema should be re-verified at pinning. answer_type label matches answerKey against choices.label (A\u2013D).","recipe":{"answer_field":"answerKey","answer_type":"label","choice_labels_field":"choices.label","choices_field":"choices.text","config":"main","data_file":"main/test-00000-of-00001.parquet","kind":"choice","question_field":"question_stem","split":"test"},"source":"https://huggingface.co/datasets/allenai/openbookqa","status":"measured","title":"OpenBookQA main (test)","updated":1792198034},{"created":1792193711,"dataset":"allenai/ai2_arc","error":null,"id":"arc-challenge-choice","parent_task":"reading","priority":0.07,"reason":"Primary-source AI2 grade-school science exam, Challenge test split: 4-way multiple-choice requiring multi-step science reasoning, an exam-style QA axis not covered by existing tasks. Declarative choice scored by plain accuracy. Public and ungated; license CC-BY-SA-4.0 per repository metadata. data_file ARC-Challenge/test-00000-of-00001.parquet confirmed by inspection (203,808 bytes, single shard). Fields question, choices.text, choices.label, answerKey are the canonical ai2_arc schema; answer_type label matches answerKey against choices.label.","recipe":{"answer_field":"answerKey","answer_type":"label","choice_labels_field":"choices.label","choices_field":"choices.text","config":"ARC-Challenge","data_file":"ARC-Challenge/test-00000-of-00001.parquet","kind":"choice","question_field":"question","split":"test"},"source":"https://huggingface.co/datasets/allenai/ai2_arc","status":"measured","title":"AI2 ARC-Challenge (test)","updated":1792198034}],"cycles":[{"finished":1792203907,"id":"debugger-20261017022504-372cc8","role":"debugger","runtime":"Runtime","started":1792203904,"status":"completed","summary":"Runtime diagnosis saved: 11 recent cycles inspected, 1 historical failures. Observation only; GPU jobs unchanged.","tool_calls":1},{"finished":1792203304,"id":"debugger-20261017021501-872982","role":"debugger","runtime":"Runtime","started":1792203301,"status":"completed","summary":"Runtime diagnosis saved: 11 recent cycles inspected, 1 historical failures. Observation only; GPU jobs unchanged.","tool_calls":1},{"finished":1792202732,"id":"debugger-20261017020528-fb1b19","role":"debugger","runtime":"Runtime","started":1792202728,"status":"completed","summary":"Runtime diagnosis saved: 11 recent cycles inspected, 1 historical failures. Observation only; GPU jobs unchanged.","tool_calls":1},{"finished":1792202442,"id":"operator-20261017020028-310e37","role":"operator","runtime":"Runtime","started":1792202428,"status":"completed","summary":"Runtime cycle completed; 3 tool calls.","tool_calls":3},{"finished":1792202112,"id":"debugger-20261017015509-77a11f","role":"debugger","runtime":"Runtime","started":1792202109,"status":"completed","summary":"Runtime diagnosis saved: 10 recent cycles inspected, 1 historical failures. Observation only; GPU jobs unchanged.","tool_calls":1},{"finished":1792201513,"id":"debugger-20261017014510-cda84f","role":"debugger","runtime":"Runtime","started":1792201510,"status":"completed","summary":"Runtime diagnosis saved: 10 recent cycles inspected, 1 historical failures. Observation only; GPU jobs unchanged.","tool_calls":1},{"finished":1792200929,"id":"debugger-20261017013524-f8d0ce","role":"debugger","runtime":"Runtime","started":1792200924,"status":"completed","summary":"Runtime diagnosis saved: 10 recent cycles inspected, 1 historical failures. Observation only; GPU jobs unchanged.","tool_calls":1},{"finished":1792200658,"id":"operator-20261017013042-6d5888","role":"operator","runtime":"Runtime","started":1792200642,"status":"completed","summary":"Runtime cycle completed; 3 tool calls.","tool_calls":3},{"finished":1792200471,"id":"scout-20261017012730-7fe4de","role":"scout","runtime":"Runtime","started":1792200450,"status":"completed","summary":"Runtime cycle completed; 9 tool calls.","tool_calls":9},{"finished":1792200455,"id":"debugger-20261017012730-2347cd","role":"debugger","runtime":"Runtime","started":1792200450,"status":"completed","summary":"Runtime diagnosis saved: 9 recent cycles inspected, 1 historical failures. Observation only; GPU jobs unchanged.","tool_calls":1},{"finished":1792200438,"id":"debugger-20261017012537-6fcea0","role":"debugger","runtime":"Runtime","started":1792200337,"status":"completed","summary":"Runtime diagnosis saved: 8 recent cycles inspected, 1 historical failures. Observation only; GPU jobs unchanged.","tool_calls":1},{"finished":1792200329,"id":"operator-20261017012417-460c8b","role":"operator","runtime":"Runtime","started":1792200258,"status":"completed","summary":"Runtime cycle completed; 4 tool calls.","tool_calls":4},{"finished":1792200445,"id":"scout-20261017012417-51a655","role":"scout","runtime":"Runtime","started":1792200258,"status":"completed","summary":"Runtime cycle completed; 11 tool calls.","tool_calls":11},{"finished":1792200021,"id":"scout-20261017011621-6bdd46","role":"scout","runtime":"Runtime","started":1792199781,"status":"failed","summary":"Runtime cycle failed; 19 tool calls.","tool_calls":19},{"finished":1792198863,"id":"operator-20261017010019-045e8e","role":"operator","runtime":"Runtime","started":1792198819,"status":"completed","summary":"Runtime cycle completed; 4 tool calls.","tool_calls":4},{"finished":1792197033,"id":"operator-20261017003009-edcaed","role":"operator","runtime":"Runtime","started":1792197009,"status":"completed","summary":"Runtime cycle completed; 4 tool calls.","tool_calls":4},{"finished":1792195960,"id":"scout-20261017001109-1173c4","role":"scout","runtime":"Runtime","started":1792195869,"status":"completed","summary":"Runtime cycle completed; 11 tool calls.","tool_calls":11},{"finished":1792195246,"id":"operator-20261017000027-cf32c6","role":"operator","runtime":"Runtime","started":1792195227,"status":"completed","summary":"Runtime cycle completed; 4 tool calls.","tool_calls":4},{"finished":1792194980,"id":"scout-20261016235437-ee7e2b","role":"scout","runtime":"Runtime","started":1792194877,"status":"completed","summary":"Runtime cycle completed; 12 tool calls.","tool_calls":12},{"finished":1792193907,"id":"operator-20261016233802-19d69f","role":"operator","runtime":"Runtime","started":1792193882,"status":"completed","summary":"Runtime cycle completed; 3 tool calls.","tool_calls":3}],"events":[{"action":"prepared","created":1792202434,"detail":"needs_adapter","item":"musr-narrative-reasoning","role":"operator"},{"action":"prepared","created":1792200654,"detail":"needs_adapter","item":"winogrande-validation","role":"operator"},{"action":"prepared","created":1792200654,"detail":"needs_adapter","item":"sciq-knowledge","role":"operator"},{"action":"discovered","created":1792200466,"detail":"SciQ science QA (test)","item":"sciq-knowledge","role":"scout"},{"action":"discovered","created":1792200441,"detail":"MuSR narrative multistep reasoning","item":"musr-narrative-reasoning","role":"scout"},{"action":"discovered","created":1792200441,"detail":"WinoGrande winogrande_xl (validation)","item":"winogrande-validation","role":"scout"},{"action":"sub1b-prioritized","created":1792199285,"detail":"Explicit operator priority; first 32 pinned examples across eleven under-1B systems","item":"sub1b-boolq","role":"operator"},{"action":"sub1b-prioritized","created":1792199285,"detail":"Explicit operator priority; first 32 pinned examples across eleven under-1B systems","item":"sub1b-hellaswag","role":"operator"},{"action":"sub1b-prioritized","created":1792199285,"detail":"Explicit operator priority; first 32 pinned examples across eleven under-1B systems","item":"sub1b-arc-easy","role":"operator"},{"action":"prepared","created":1792198855,"detail":"needs_adapter","item":"videomme-video-understanding","role":"operator"},{"action":"prepared","created":1792198855,"detail":"needs_adapter","item":"coco2017-detection","role":"operator"},{"action":"protocol-upgraded","created":1792198034,"detail":"Validated length-batched v2; serial results retained in historical scope","item":"openbookqa-main-choice","role":"operator"},{"action":"protocol-upgraded","created":1792198034,"detail":"Validated length-batched v2; serial results retained in historical scope","item":"arc-challenge-choice","role":"operator"},{"action":"prepared","created":1792197023,"detail":"needs_adapter","item":"omnidocbench-document-parsing","role":"operator"},{"action":"prepared","created":1792197023,"detail":"needs_adapter","item":"swe-bench-verified-code","role":"operator"},{"action":"discovered","created":1792195954,"detail":"Video-MME video understanding (test)","item":"videomme-video-understanding","role":"scout"},{"action":"discovered","created":1792195951,"detail":"SWE-bench Verified (test)","item":"swe-bench-verified-code","role":"scout"},{"action":"discovered","created":1792195951,"detail":"COCO 2017 object detection (val)","item":"coco2017-detection","role":"scout"},{"action":"prepared","created":1792195236,"detail":"needs_adapter","item":"bfcl-function-calling","role":"operator"},{"action":"prepared","created":1792195236,"detail":"needs_adapter","item":"singmos-speech-quality","role":"operator"},{"action":"discovered","created":1792194975,"detail":"SingMOS sung-speech MOS prediction (TTS-side quality)","item":"singmos-speech-quality","role":"scout"},{"action":"discovered","created":1792194975,"detail":"OmniDocBench document parsing","item":"omnidocbench-document-parsing","role":"scout"},{"action":"discovered","created":1792194975,"detail":"BFCL function calling (Berkeley Function-Calling Leaderboard)","item":"bfcl-function-calling","role":"scout"},{"action":"prepared","created":1792193899,"detail":"needs_adapter","item":"gaia-validation-agent","role":"operator"},{"action":"scheduled","created":1792193829,"detail":"2 pilot runs","item":"openbookqa-main-choice","role":"operator"},{"action":"prepared","created":1792193828,"detail":"ready","item":"openbookqa-main-choice","role":"operator"},{"action":"discovered","created":1792193796,"detail":"OpenBookQA main (test)","item":"openbookqa-main-choice","role":"scout"},{"action":"discovered","created":1792193796,"detail":"GAIA (2023 validation)","item":"gaia-validation-agent","role":"scout"},{"action":"scheduled","created":1792193749,"detail":"2 pilot runs","item":"arc-challenge-choice","role":"operator"},{"action":"prepared","created":1792193746,"detail":"ready","item":"arc-challenge-choice","role":"operator"}],"policy":{"debugger_interval_minutes":10,"gpu_slots":2,"max_new_jobs_per_day":4,"operator_interval_minutes":30,"pilot_examples":32,"scout_interval_hours":6}}}