# AI Model Debugging: Find the First Divergence Across AI Accelerators

## Executive summary

This guide details a systematic approach to debugging AI models that exhibit numerical divergence when moved from a validated reference machine to a new AI accelerator. The core technique involves finding the 'first observed divergence' by comparing intermediate tensor results at selected boundaries (layers or operators). This localization step significantly reduces the search space, allowing engineers to isolate the problem to a smaller region—which could be the operation itself, its inputs, or components like the compiler or runtime—before attempting diagnosis. The process culminates in creating a minimal test case that can be integrated as a regression test into the CI/CD pipeline.

## Key takeaways

- The Scope of AI Model Debugging: When moving a model to a new accelerator, the change affects not just the hardware but the entire software stack (compiler, runtime, and accelerator backend). The problem can appear at any level, including operations, lowering, fusion, scheduling, or precision decisions.
- Localizing Divergence via Boundaries: Instead of debugging the entire model, compare intermediate tensor outputs at selected boundaries (e.g., after the embedding layer, after attention, etc.) between the reference and target accelerator. The first boundary where the difference exceeds the allowed tolerance is the 'first observed divergence,' narrowing the investigation to a specific region.
- Creating Regression Tests: Once the root cause is isolated and fixed (e.g., by adjusting precision or fixing a compiler translation), the minimal test that reproduces the difference must be saved as a regression test. This ensures that future changes to the compiler or runtime do not reintroduce the failure.

## Technical details

- AI Model Software Stack: The model execution passes through several components: the compiler (translates operations), the runtime (manages work and data), and the accelerator (executes the computation). Failures can occur at any of these levels.
- Numerical Comparison and Tolerances: Since floating-point computations can introduce expected differences (due to changes in operation order or precision), comparison requires more than bit-for-bit matching. Techniques include checking shape, dtype, maximum absolute differences, relative error, and cosine similarity against defined tolerances.
- Debugging Workflow: Localization to Diagnosis: The recommended workflow is: 1) Run controlled inputs through both reference and target systems. 2) Compare intermediate tensors at boundaries. 3) Identify the first mismatch (the suspect region). 4) Inspect the inputs to the operation, isolate the operation, and perform a minimal test to reproduce the difference. This separates the probability of the bug being in the operation, its inputs, or the surrounding execution components.

## Practical implications

- Provides a structured methodology for debugging cross-platform AI model failures, moving beyond simple final output comparison.
- Enables build engineers to pinpoint the exact operational boundary (the 'first observed divergence') causing numerical instability, significantly reducing debugging time.
- Establishes a process for converting transient bugs into permanent, automated regression tests for CI/CD pipelines.

## Topics

AI Debugging, AI Accelerators, Tensor Comparison, Compiler Optimization, Regression Testing, Machine Learning Deployment, Debugging Across the Dataflow Stack (Coursera), AI Debugging Information

Source: https://www.youtube.com/watch?v=K6ffYKrlPdU
