# How to Measure OCR Quality

## Executive summary

The video critiques the current state of benchmarking in OCR and document processing, arguing that many published benchmarks are flawed, biased, or designed to measure adherence to format rather than true underlying quality. The speaker, Vik Paruchuri, emphasizes that robust evaluation requires moving beyond standard metrics like edit distance and adopting a unit-test methodology. He provides a comprehensive guide on how to build custom, reliable evaluation sets tailored to specific downstream use cases, such as RAG pipelines over annual filings.

## Key takeaways

- Benchmarking is often biased and misleading.: Many benchmarks, including ParsBench and OmniParsBench, can be gamed. Models may improve their scores by simply aligning to the benchmark's expected format rather than improving actual OCR quality. The root cause of differing rankings is that different benchmarks measure fundamentally different things. (00:00, 03:10)
- Avoid metrics that over-penalize formatting or under-penalize content errors.: Edit distance is poor because it can assign a high divergence score when only the output format changes (e.g., Markdown to HTML), while assigning a low score when critical content (like a number) is factually incorrect. (06:30)
- Adopt a unit-test style evaluation for maximum reliability.: The most reliable method is treating evaluation like unit testing: defining specific, verifiable pass/fail criteria (e.g., 'Is this cell under this header?'). This approach minimizes ambiguity and is superior to relying on LLM judges due to issues like non-determinism. (09:33)
- Build custom evals by focusing on the use case.: When designing a benchmark, the first step must be defining the production use case (e.g., financial RAG). Ground truth must be sourced from verifiable methods (human review, exact PDF ground truth, or synthetic documents), and the metric must be invariant to non-critical formatting changes. (16:53, 24:10)

## Technical details

- Document Pipeline Components: The document pipeline involves multiple steps, including scanning, parsing, chunking, and retrieving (RAG). Parsing is identified as a crucial step because downstream quality depends heavily on its accuracy. (00:57)
- Parsing Failure Modes: Common failures include misreading characters (e.g., '3' read as '8'), outputting values in the wrong script (Cyrillic instead of English), or format changes that break downstream logic. (00:57)
- Metric Flaws: Layout metrics are problematic because the definition of a 'paragraph' or bounding box can vary slightly between authors. LLM judges are unreliable due to non-determinism and subtle misinterpretations (e.g., marking 'not found' as correct). (09:03, 12:40)
- Best Practices for Eval Design: A good metric must be valid (correct output passes, wrong output fails), specific, sourced (verifiable ground truth), and discriminating (some models pass, some fail). The unit-test style is recommended for its simplicity and clear pass/fail criteria. (05:35, 11:33)

## Practical implications

- When building a document pipeline, focus evaluation efforts on the parsing step, as it dictates the quality of all subsequent downstream processes (e.g., RAG).
- Always define evaluation metrics based on the specific business use case (e.g., financial vs. scientific), rather than relying on general-purpose benchmarks.
- Prioritize unit-test style evaluations that require explicit pass/fail checks over continuous metrics like edit distance.
- When reviewing benchmarks, scrutinize the ground truth source and look for evidence of bias (e.g., if all strong models fail on the same set of tests, the test may be flawed).

## Topics

OCR, Document Processing, Benchmarking, Evaluation Metrics, RAG, AI Quality Assurance, AI Evals October 2026 cohort, Evals Guide

Source: https://www.youtube.com/watch?v=DeEjQ1r6ip8