# Fireside Chat with Erik Bernhardsson, Modal

## Executive summary

Modal provides general-purpose AI infrastructure, powering inference, training, and sandboxes for advanced AI workloads. The platform's growth has been driven by the demand for sandboxes—used for 'vibe coding' apps, background agents, and reinforcement learning (RL) rollouts. Architecturally, Modal is shifting from lightweight runtimes like gVisor to full Virtual Machines (VMs) for greater capability and performance. Furthermore, the company is building out advanced services, including auto and shared inference endpoints, multi-node training clusters (using RDMA), and batch processing capabilities, positioning itself as a comprehensive platform for sophisticated ML engineers.

## Key takeaways

- Sandboxes are becoming ephemeral and decoupled: The optimal architecture for agent harnesses is decoupling: the long-running harness should operate outside the sandbox, while the sandboxes themselves should be short-lived, ideally running for only one tool call. This contrasts with older patterns of keeping the entire agent within a single, long-lived sandbox.
- Infrastructure is evolving from gVisor to VMs: Modal is migrating from gVisor to full Virtual Machines (VMs) to support customers requiring full operating system access (e.g., running Docker or needing advanced firewall rules). This shift also enables support for non-Linux environments like Windows and Mac sandboxes.
- Inference endpoints are becoming more accessible: Modal has introduced auto and shared endpoints to simplify deploying open-source models. This allows users to easily deploy single-tenant or resource-efficient shared endpoints, making advanced inference more accessible without requiring deep knowledge of performance optimization parameters (like vLLM or SGLang).
- Focus on high-scale, complex workloads: The platform is building for sophisticated users, particularly those running RL or large-scale coding models. New products include multi-node training (serverless clusters of 4, 8, 16 nodes, fully interconnected with RDMA) and discounted batch jobs for latency-insensitive tasks.

## Technical details

- Sandboxes and Agent Architecture: Sandboxes are critical for running code generated by LLMs. While the core primitive remains programmatic container creation and execution, the differentiation lies in scaling, reliability, and performance. The shift toward ephemeral sandboxes (one per tool call) is key to modern agent design. The median sandbox lifecycle is noted to be around 12 seconds.
- Compute and Training: Modal is developing a multi-node training product, allowing users to spin up serverless clusters (e.g., 128 GPUs) fully interconnected with RDMA. Additionally, a batch product is being launched to handle large volumes of latency-insensitive jobs (e.g., transcribing thousands of audio files) with significant discounts.
- Inference Optimization: The platform supports all open-source models and offers auto and shared endpoints. The goal is to abstract away the complexity of performance tuning (like those required for vLLM or SGLang) by automatically optimizing performance through monitoring and parameter tweaking.
- Model Fine-Tuning vs. RL: While Supervised Fine-Tuning (SFT) is sufficient for most average customers, complex, large-scale customers (e.g., Cognition) require Reinforcement Learning (RL). RL is noted to be highly complex and resource-intensive, but offers the potential for significant accuracy gains.

## Practical implications

- Build engineers should design agent architectures that decouple the long-running agent harness from the execution environment, utilizing short-lived, ephemeral sandboxes for tool calls.
- When designing high-throughput systems, plan for multi-node, RDMA-connected compute clusters for training and batch processing to maximize efficiency.
- Leverage auto and shared endpoints for open-source model inference to reduce operational overhead and simplify deployment compared to manually optimizing frameworks like vLLM.
- Be aware of the architectural shift from gVisor to full VMs when building sandboxes that require full OS access or non-Linux environments (Windows/Mac).

## Topics

AI Infrastructure, Agent Development, Containerization, Distributed Computing, Machine Learning Operations (MLOps), LLM Inference, Modal, Modal Sandboxes, Deep Agents with Modal sandboxes

Source: https://www.youtube.com/watch?v=1M8vagUOdJM
