How to Measure OCR Quality
Summary
The video critiques the current state of benchmarking in OCR and document processing, arguing that many published benchmarks are flawed, biased, or designed to measure adherence to format rather than true underlying quality. The speaker, Vik Paruchuri, emphasizes that robust evaluation requires moving beyond standard metrics like edit distance and adopting a unit-test methodology. He provides a comprehensive guide on how to build custom, reliable evaluation sets tailored to specific downstream use cases, such as RAG pipelines over annual filings.
Key takeaways
-
Benchmarking is often biased and misleading.
5:10
Many benchmarks, including ParsBench and OmniParsBench, can be gamed. Models may improve their scores by simply aligning to the benchmark's expected format rather than improving actual OCR quality. The root cause of differing rankings is that different benchmarks measure fundamentally different things. (00:00, 03:10)
-
Avoid metrics that over-penalize formatting or under-penalize content errors.
10:30
Edit distance is poor because it can assign a high divergence score when only the output format changes (e.g., Markdown to HTML), while assigning a low score when critical content (like a number) is factually incorrect. (06:30)
-
Adopt a unit-test style evaluation for maximum reliability.
15:33
The most reliable method is treating evaluation like unit testing: defining specific, verifiable pass/fail criteria (e.g., 'Is this cell under this header?'). This approach minimizes ambiguity and is superior to relying on LLM judges due to issues like non-determinism. (09:33)
-
Build custom evals by focusing on the use case.
28:13
When designing a benchmark, the first step must be defining the production use case (e.g., financial RAG). Ground truth must be sourced from verifiable methods (human review, exact PDF ground truth, or synthetic documents), and the metric must be invariant to non-critical formatting changes. (16:53, 24:10)
Technical details
-
Document Pipeline Components
57s
The document pipeline involves multiple steps, including scanning, parsing, chunking, and retrieving (RAG). Parsing is identified as a crucial step because downstream quality depends heavily on its accuracy. (00:57)
-
Parsing Failure Modes
57s
Common failures include misreading characters (e.g., '3' read as '8'), outputting values in the wrong script (Cyrillic instead of English), or format changes that break downstream logic. (00:57)
-
Metric Flaws
903s
Layout metrics are problematic because the definition of a 'paragraph' or bounding box can vary slightly between authors. LLM judges are unreliable due to non-determinism and subtle misinterpretations (e.g., marking 'not found' as correct). (09:03, 12:40)
-
Best Practices for Eval Design
535s
A good metric must be valid (correct output passes, wrong output fails), specific, sourced (verifiable ground truth), and discriminating (some models pass, some fail). The unit-test style is recommended for its simplicity and clear pass/fail criteria. (05:35, 11:33)
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.