AI Model Debugging: Find the First Divergence Across AI Accelerators
Summary
This guide details a systematic approach to debugging AI models that exhibit numerical divergence when moved from a validated reference machine to a new AI accelerator. The core technique involves finding the 'first observed divergence' by comparing intermediate tensor results at selected boundaries (layers or operators). This localization step significantly reduces the search space, allowing engineers to isolate the problem to a smaller region—which could be the operation itself, its inputs, or components like the compiler or runtime—before attempting diagnosis. The process culminates in creating a minimal test case that can be integrated as a regression test into the CI/CD pipeline.
Key takeaways
-
The Scope of AI Model Debugging
When moving a model to a new accelerator, the change affects not just the hardware but the entire software stack (compiler, runtime, and accelerator backend). The problem can appear at any level, including operations, lowering, fusion, scheduling, or precision decisions.
-
Localizing Divergence via Boundaries
7:25
Instead of debugging the entire model, compare intermediate tensor outputs at selected boundaries (e.g., after the embedding layer, after attention, etc.) between the reference and target accelerator. The first boundary where the difference exceeds the allowed tolerance is the 'first observed divergence,' narrowing the investigation to a specific region.
-
Creating Regression Tests
Once the root cause is isolated and fixed (e.g., by adjusting precision or fixing a compiler translation), the minimal test that reproduces the difference must be saved as a regression test. This ensures that future changes to the compiler or runtime do not reintroduce the failure.
Technical details
-
AI Model Software Stack
0s
The model execution passes through several components: the compiler (translates operations), the runtime (manages work and data), and the accelerator (executes the computation). Failures can occur at any of these levels.
-
Numerical Comparison and Tolerances
842s
Since floating-point computations can introduce expected differences (due to changes in operation order or precision), comparison requires more than bit-for-bit matching. Techniques include checking shape, dtype, maximum absolute differences, relative error, and cosine similarity against defined tolerances.
-
Debugging Workflow: Localization to Diagnosis
650s
The recommended workflow is: 1) Run controlled inputs through both reference and target systems. 2) Compare intermediate tensors at boundaries. 3) Identify the first mismatch (the suspect region). 4) Inspect the inputs to the operation, isolate the operation, and perform a minimal test to reproduce the difference. This separates the probability of the bug being in the operation, its inputs, or the surrounding execution components.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.