Weights & Biases

The Quiet $1B+ Company Shaping AI’s Behavior | Nick Heiner, Surge AI

Published 2026-11-17 · Duration 1:00:00

Summary

The development of advanced LLMs requires specialized post-training—the process that teaches models behavior, personality, and skills—as this surface area is too broad for any single lab to handle in-house. The discussion highlights that while pre-training provides a fundamental world model, post-training, particularly through Reinforcement Learning (RL) environments, is necessary for alignment and advanced knowledge work. A major challenge is defining 'good' performance, as many industry benchmarks are flawed, and the most difficult skills to source are those requiring deep domain expertise and the ability to articulate that 'taste.'

Download summary

Key takeaways

  1. Post-training is critical for alignment and skills. 3:54

    Post-training teaches models their behavior and personality, moving beyond pre-training's function of merely modeling text. This process is crucial for teaching models to follow instructions and align with specific preferences and values. (02:34)

  2. The Outer Loop is the new bottleneck in software engineering.

    While agents excel at the 'inner loop' (writing code), the 'outer loop'—the strategic judgment (e.g., determining what features should be implemented, or identifying architectural drift)—remains a human staff engineer task that current agents struggle with. (37:35)

  3. Most industry benchmarks are fundamentally unsound.

    Many existing benchmarks (e.g., IF Eval, L Marina) test constraints that no real user cares about or rely on flawed methodologies (like using unpaid workers). Effective evaluation requires generalization to real-world tasks and human red-teaming. (15:17, 42:42)

  4. RL environments require sophisticated reward functions. 18:51

    RL environments function as training gyms where human input is not judging every attempt, but rather designing the reward signal (the verifier). These environments are designed to produce continuous scores, allowing for gradient climbing rather than relying on binary outcomes. (11:31, 18:54)

More digests on Large Language Models (LLMs)

Technical details

  • LLM Training Paradigms 2022s

    LLMs progress from pre-training (fundamental world model) to post-training. The evolution includes: 1) Chat-tuning (2022); 2) Basic instruction following (2023); and 3) Advanced, expert-tier knowledge work (current). (2022)

  • RL Environments and Verifiers 1517s

    RL environments allow agents to practice complex knowledge work tasks (e.g., drafting a memo using Slack/email data). The system uses 'verifiers'—rubrics of atomic criteria—to judge output, moving beyond simple string matching. (15:17)

  • Staff Engineer Judgment (Outer Loop)

    The ability to perform 'thought partnership'—taking a step back to assess if an architecture or data model is fundamentally flawed, even if the current task is solvable—is the key differentiator between an L3 and an L7 staff engineer. (37:35)

  • Sycophancy and Alignment

    A major concern is 'sycophancy,' where models are overly positive and agree with the user, even when the user is incorrect. This tendency undermines the ability to use the model as a critical thought partner. (52:06)

Mentioned resources

  • Surge AI (Company/Service)
  • Weights & Biases (Company/Platform)

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.