Documents / OCRBench v1 / full 1,000 questions — experimental batch-4
Visual text understanding / OCRBench v1 — experimental batch-4
Image + question → answer
Model size vs score
Static, watermarked figures. SlayerLab models are red. The close-up expands the crowded small-model cluster without changing any coordinates.
Measured results
Full pinned test split · 1,000 examples · accuracy higher is better. No global SOTA claim.
| System | Score | Samples | Elapsed¹ | Evidence |
|---|---|---|---|---|
| Qwen/Qwen2.5-VL-3B-Instruct 66285546d2b8 | 60.7% | 1000 | 369s | Inspect run → 18 Oct 2026, 18:17 UTC |
| Qwen/Qwen2-VL-2B-Instruct 895c3a49bc3f | 49.0% | 1000 | 265s | Inspect run → 18 Oct 2026, 18:15 UTC |
¹ Evaluation elapsed time includes model loading after dataset selection. It is not serving latency or a fair speed comparison.
What this measures
Benchmark origin
Dataset / split
echo840/OCRBench / test
Immutable dataset revision
92a54bd1384387c178d5a07140a2d85e0a3d12e1Upstream license metadata
Upstream benchmark code MIT; component data retain their source terms · consult upstream terms
Protocol
codesota-ocrbench-v1-full-bf16-qwen-batch4-v1 / ocrbench
Selection
All 1000 examples in the pinned test split. SNLI rows without a valid label are excluded.
Configuration
Seed 42. Greedy generation for math and code. Mean answer-token log likelihood for text classification, completion and declarative multiple-choice recipes. No model judge.
Planned runs
0 qualified visits in 30 days · priority 0.1. Interest changes execution order, within the approved baseline list.
| System | Status |
|---|---|
| Qwen/Qwen2.5-VL-3B-Instruct | completed |
| Qwen/Qwen2-VL-2B-Instruct | completed |