Benchmark candidate / document parsing

ParseBench.

Evaluate whether PDF parsing preserves usable structure and meaning across five separate dimensions.

Upstream evaluator ↗ · Dataset ↗ · Pinned plan JSON · OCR hub

What it measures

• tables
• charts
• content faithfulness
• semantic formatting
• visual grounding

Default evaluator is deterministic; optional judge-based chart normalization must remain disabled or a separate protocol.

Before we can publish results

1. Pin evaluator, dataset files and split/group identifiers
2. Adapt each local parser to the required structured output and preserve native coordinates
3. Keep all five dimensions separate; do not infer document parsing from line CER
4. Record local model revision, parser configuration, failures, latency and reserved GPU cost
5. Keep upstream vendor-reported scores separate from CodeSOTA runs

Candidate local pipelines

Docling, Dots OCR, Qwen VLM

Candidate pipeline names from upstream documentation; exact models, weights licences, local hosting and adapter compatibility still require validation. No CodeSOTA ParseBench scores yet.

Pinned evidence

Evaluator: 9af4102e26b51f234ab5f037e19fec05c0696ec6 · Apache-2.0.

Dataset: 2805a1d940f95a203e0ae4b88be9934f7765b3fc · licence metadata: Apache-2.0. Preserve provenance and review individual document terms before redistribution.

Upstream scores are vendor-reported context. They are not imported into CodeSOTA rankings.