Weights & Biases

Inference Optimization with CoreWeave Mission Control Agent

Published 2026-11-04 · Duration 9:33

Summary

The CoreWeave Mission Control Agent is an AI-powered tool that diagnoses and optimizes production inference endpoints without requiring manual intervention or new hardware. In a demonstration, the agent analyzed a slow GLM 5.2 deployment suffering from KV cache pressure under agentic traffic. It proposed and applied two optimizations—quantizing the model to NVFP4 and reshaping the deployment from one replica at TP8 to two replicas at TP4—resulting in over 3x the output throughput per GPU on the same eight GB200 GPUs.

Download summary

Key takeaways

  1. Automated Diagnosis of Underperforming Endpoints 0:02

    Mission Control can detect performance degradation (e.g., climbing time to first token) even when standard health checks are green, identifying root causes like KV cache running at 99% hot due to agentic traffic.

  2. Optimization Strategy: Quantization and Reshaping 0:03

    The agent recommended two approaches: scaling up (costly) or quantizing the model (using existing hardware). Quantization to NVFP4 reduced memory footprint, freeing up KV cache space. Further optimization involved reshaping the deployment from one replica at TP8 to two replicas at TP4.

  3. Significant Performance Gains 0:07

    The combined optimizations increased input processing from 28,000 to 56,000 tokens/min/GPU, and output throughput from 3,000 to 5,000 tokens/min/GPU, achieving over 3x useful work per GPU without increasing hardware costs.

More digests on Inference Optimization

Technical details

  • Inference Endpoint Analysis 0s

    The initial deployment was a GLM 5.2 model serving FP8 on GB200 GPUs. The core issue identified was KV cache pressure, which caused GPUs to recompute work instead of caching it.

  • Model Optimization (Quantization) 3s

    The agent quantized the model weights to NVFP4, which reduced the memory required for the weights, thereby unlocking more memory for the strained KV cache.

  • Deployment Reshaping (TP4) 4s

    The system was reshaped from one replica at TP8 to two replicas at TP4 on the same eight GPUs. This doubled the request concurrency and improved failure isolation between replicas.

Mentioned resources

  • CoreWeave Mission Control Agent (AI Tool/Console Feature)
  • GLM 5.2 (Model)

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.