# CoreWeave AI Training: Built for Training at Scale with Confidence

## Executive summary

Model Distillation, implemented via CoreWeave Forge, is an optimization technique for production AI workloads that repeat narrow tasks. Instead of relying solely on large, expensive proprietary models (the 'teacher'), the process uses logged production traffic to train a smaller, open-weight model (the 'student'). This approach significantly reduces inference costs and latency while maintaining or improving performance. The full loop includes data collection through a proxy, data improvement via relabeling, serverless training, head-to-head evaluation, and gradual traffic rollout.

## Key takeaways

- The Need for Distillation: Most production AI calls are not open-ended; they involve repeating the same prompts and tools millions of times a day. Using a massive frontier model for a narrow task is inefficient and costly.
- The Distillation Process: The system captures data through a Model Distillation proxy, which logs all requests. This logged data forms the basis for training a smaller, specialized model.
- Data Improvement via Relabeling: To prevent the student model from learning mistakes, a stronger model (e.g., GPT-4 Astra) can review logged turns and relabel or correct weak answers before training begins.
- Evaluation and Rollout: Student models are evaluated head-to-head against the production model on held-out test sets. Once confidence is established, traffic is rolled out gradually (e.g., starting at 10%) while keeping the original model live as a reference.

## Technical details

- Data Collection and Routing: The process begins by pointing an OpenAI-compatible client to a CoreWeave proxy base URL. This proxy logs every request, allowing the creation of a dataset by selecting a time window and applying filters. The system supports routing traffic across multiple models (weights).
- Training Infrastructure: Training is performed on serverless infrastructure, eliminating the need for manual GPU management. Users select open-source models (e.g., Llama, Mistral, Gemma) and hyperparameters, and the resulting model is automatically deployed for serverless inference.
- Evaluation Methodology: Evaluation is conducted via a head-to-head comparison on a test set. A 'judge' compares each student model's turn against the original data set and the teacher model's turn, providing a quantitative score (e.g., 50% means as good as the dataset).
- Cost and Ownership: The cost of the expensive model is paid only once during the relabeling phase, not on every inference request. The resulting model weights are owned by the user and can be exported.

## Practical implications

- Reduces operational costs by replacing expensive frontier models with smaller, specialized, open-weight models.
- Allows for continuous improvement: Since all traffic is logged, every deployed version provides fresh training data for the next iteration of the model.
- Mitigates risk through controlled deployment: Traffic can be rolled out gradually (e.g., 10%) while retaining the original model as a live reference.
- Provides full transparency: Every evaluation row shows the conversation and the judge's reasoning for preferring one answer over another.

## Topics

Model Distillation, Generative AI, Open-Weight Models, Serverless Inference, MLOps, Cost Optimization, CoreWeave, Weights & Biases

Source: https://www.youtube.com/watch?v=187QjWfps8s
