Fireside Chat with Erik Bernhardsson, Modal
Summary
Modal provides general-purpose AI infrastructure, powering inference, training, and sandboxes for advanced AI workloads. The platform's growth has been driven by the demand for sandboxes—used for 'vibe coding' apps, background agents, and reinforcement learning (RL) rollouts. Architecturally, Modal is shifting from lightweight runtimes like gVisor to full Virtual Machines (VMs) for greater capability and performance. Furthermore, the company is building out advanced services, including auto and shared inference endpoints, multi-node training clusters (using RDMA), and batch processing capabilities, positioning itself as a comprehensive platform for sophisticated ML engineers.
Key takeaways
-
Sandboxes are becoming ephemeral and decoupled
11:51
The optimal architecture for agent harnesses is decoupling: the long-running harness should operate outside the sandbox, while the sandboxes themselves should be short-lived, ideally running for only one tool call. This contrasts with older patterns of keeping the entire agent within a single, long-lived sandbox.
-
Infrastructure is evolving from gVisor to VMs
8:55
Modal is migrating from gVisor to full Virtual Machines (VMs) to support customers requiring full operating system access (e.g., running Docker or needing advanced firewall rules). This shift also enables support for non-Linux environments like Windows and Mac sandboxes.
-
Inference endpoints are becoming more accessible
Modal has introduced auto and shared endpoints to simplify deploying open-source models. This allows users to easily deploy single-tenant or resource-efficient shared endpoints, making advanced inference more accessible without requiring deep knowledge of performance optimization parameters (like vLLM or SGLang).
-
Focus on high-scale, complex workloads
The platform is building for sophisticated users, particularly those running RL or large-scale coding models. New products include multi-node training (serverless clusters of 4, 8, 16 nodes, fully interconnected with RDMA) and discounted batch jobs for latency-insensitive tasks.
More digests on AI Infrastructure
Technical details
-
Sandboxes and Agent Architecture
247s
Sandboxes are critical for running code generated by LLMs. While the core primitive remains programmatic container creation and execution, the differentiation lies in scaling, reliability, and performance. The shift toward ephemeral sandboxes (one per tool call) is key to modern agent design. The median sandbox lifecycle is noted to be around 12 seconds.
-
Compute and Training
Modal is developing a multi-node training product, allowing users to spin up serverless clusters (e.g., 128 GPUs) fully interconnected with RDMA. Additionally, a batch product is being launched to handle large volumes of latency-insensitive jobs (e.g., transcribing thousands of audio files) with significant discounts.
-
Inference Optimization
The platform supports all open-source models and offers auto and shared endpoints. The goal is to abstract away the complexity of performance tuning (like those required for vLLM or SGLang) by automatically optimizing performance through monitoring and parameter tweaking.
-
Model Fine-Tuning vs. RL
1549s
While Supervised Fine-Tuning (SFT) is sufficient for most average customers, complex, large-scale customers (e.g., Cognition) require Reinforcement Learning (RL). RL is noted to be highly complex and resource-intensive, but offers the potential for significant accuracy gains.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.