IBM Technology

AI Model Debugging: Find the First Divergence Across AI Accelerators

Published 2026-11-15 · Duration 14:30

Summary

This guide details a systematic approach to debugging AI models that exhibit numerical divergence when moved from a validated reference machine to a new AI accelerator. The core technique involves finding the 'first observed divergence' by comparing intermediate tensor results at selected boundaries (layers or operators). This localization step significantly reduces the search space, allowing engineers to isolate the problem to a smaller region—which could be the operation itself, its inputs, or components like the compiler or runtime—before attempting diagnosis. The process culminates in creating a minimal test case that can be integrated as a regression test into the CI/CD pipeline.

Download summary

Key takeaways

  1. The Scope of AI Model Debugging

    When moving a model to a new accelerator, the change affects not just the hardware but the entire software stack (compiler, runtime, and accelerator backend). The problem can appear at any level, including operations, lowering, fusion, scheduling, or precision decisions.

  2. Localizing Divergence via Boundaries 7:25

    Instead of debugging the entire model, compare intermediate tensor outputs at selected boundaries (e.g., after the embedding layer, after attention, etc.) between the reference and target accelerator. The first boundary where the difference exceeds the allowed tolerance is the 'first observed divergence,' narrowing the investigation to a specific region.

  3. Creating Regression Tests

    Once the root cause is isolated and fixed (e.g., by adjusting precision or fixing a compiler translation), the minimal test that reproduces the difference must be saved as a regression test. This ensures that future changes to the compiler or runtime do not reintroduce the failure.

More digests on AI Debugging

Technical details

  • AI Model Software Stack 0s

    The model execution passes through several components: the compiler (translates operations), the runtime (manages work and data), and the accelerator (executes the computation). Failures can occur at any of these levels.

  • Numerical Comparison and Tolerances 842s

    Since floating-point computations can introduce expected differences (due to changes in operation order or precision), comparison requires more than bit-for-bit matching. Techniques include checking shape, dtype, maximum absolute differences, relative error, and cosine similarity against defined tolerances.

  • Debugging Workflow: Localization to Diagnosis 650s

    The recommended workflow is: 1) Run controlled inputs through both reference and target systems. 2) Compare intermediate tensors at boundaries. 3) Identify the first mismatch (the suspect region). 4) Inspect the inputs to the operation, isolate the operation, and perform a minimal test to reproduce the difference. This separates the probability of the bug being in the operation, its inputs, or the surrounding execution components.

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.