Weights & Biases

Using CoreWeave Mission Control to Diagnose and Fix Slow Inference

Published 2026-11-05 · Duration 5:35

Summary

This demo showcases how CoreWeave Mission Control AI diagnoses and resolves performance bottlenecks in a dedicated inference endpoint. When an endpoint running GLM 5.2 on GB200 nodes experienced slowdowns under agentic traffic, the AI identified the root cause as an undersized KV cache capacity. Instead of simply scaling out, the AI recommended and applied a more efficient fix: quantizing the model weights from FP8 to NVFP4. This optimization allowed the endpoint to maintain high throughput (nearly doubling capacity to 18,400 tokens/second) while significantly reducing the Time to First Token (TTFT).

Download summary

Key takeaways

  1. AI-Driven Diagnosis of Inference Bottlenecks 2:06

    CoreWeave Mission Control AI can diagnose complex performance issues, identifying that the slowdown was caused by KV cache saturation and high KV pressure (near 100%) during agentic load, rather than just general compute limitations.

  2. Optimizing Throughput via Quantization

    The AI proposed optimizing the deployment by quantizing the model weights from FP8 (8-bit floats) to NVFP4 (4-bit floats), allowing the system to increase KV cache headroom without adding physical hardware capacity.

  3. Significant Performance Gains

    After applying the quantization fix, the endpoint achieved a new throughput of 18,400 tokens per second (up from ~9600 tokens/second) and reduced the Time to First Token (TTFT) from minutes to 6.1 seconds.

More digests on AI

Technical details

  • Inference Hardware & Model 0s

    The deployment utilized GLM 5.2 running on two GB200 nodes in the EU North O2A region.

  • Performance Metrics 126s

    Key metrics monitored included Time to First Token (TTFT), P95 TTFT, and queue wait time. The initial issue was characterized by increasing TTFT and queue buildup due to cache saturation.

  • Optimization Techniques

    The primary fix involved quantizing model weights from FP8 to NVFP4, which increased the available KV cache headroom and improved overall throughput efficiency.

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.