# The Quiet $1B+ Company Shaping AI’s Behavior | Nick Heiner, Surge AI

## Executive summary

The development of advanced LLMs requires specialized post-training—the process that teaches models behavior, personality, and skills—as this surface area is too broad for any single lab to handle in-house. The discussion highlights that while pre-training provides a fundamental world model, post-training, particularly through Reinforcement Learning (RL) environments, is necessary for alignment and advanced knowledge work. A major challenge is defining 'good' performance, as many industry benchmarks are flawed, and the most difficult skills to source are those requiring deep domain expertise and the ability to articulate that 'taste.'

## Key takeaways

- Post-training is critical for alignment and skills.: Post-training teaches models their behavior and personality, moving beyond pre-training's function of merely modeling text. This process is crucial for teaching models to follow instructions and align with specific preferences and values. (02:34)
- The Outer Loop is the new bottleneck in software engineering.: While agents excel at the 'inner loop' (writing code), the 'outer loop'—the strategic judgment (e.g., determining what features should be implemented, or identifying architectural drift)—remains a human staff engineer task that current agents struggle with. (37:35)
- Most industry benchmarks are fundamentally unsound.: Many existing benchmarks (e.g., IF Eval, L Marina) test constraints that no real user cares about or rely on flawed methodologies (like using unpaid workers). Effective evaluation requires generalization to real-world tasks and human red-teaming. (15:17, 42:42)
- RL environments require sophisticated reward functions.: RL environments function as training gyms where human input is not judging every attempt, but rather designing the reward signal (the verifier). These environments are designed to produce continuous scores, allowing for gradient climbing rather than relying on binary outcomes. (11:31, 18:54)

## Technical details

- LLM Training Paradigms: LLMs progress from pre-training (fundamental world model) to post-training. The evolution includes: 1) Chat-tuning (2022); 2) Basic instruction following (2023); and 3) Advanced, expert-tier knowledge work (current). (2022)
- RL Environments and Verifiers: RL environments allow agents to practice complex knowledge work tasks (e.g., drafting a memo using Slack/email data). The system uses 'verifiers'—rubrics of atomic criteria—to judge output, moving beyond simple string matching. (15:17)
- Staff Engineer Judgment (Outer Loop): The ability to perform 'thought partnership'—taking a step back to assess if an architecture or data model is fundamentally flawed, even if the current task is solvable—is the key differentiator between an L3 and an L7 staff engineer. (37:35)
- Sycophancy and Alignment: A major concern is 'sycophancy,' where models are overly positive and agree with the user, even when the user is incorrect. This tendency undermines the ability to use the model as a critical thought partner. (52:06)

## Practical implications

- For build engineers, the focus must shift from optimizing the 'inner loop' (code generation) to defining and testing the 'outer loop' (system architecture, process flow, and strategic decision-making).
- Evaluation systems must move beyond simple unit tests or string matches, requiring complex, multi-criteria rubrics and human-in-the-loop red-teaming to ensure generalization.
- When designing agentic workflows, the reward function must be designed to produce continuous scores, allowing for measurable progress and gradient improvement rather than relying on binary pass/fail outcomes.

## Topics

Large Language Models (LLMs), Reinforcement Learning (RL), AI Alignment, System Architecture, Benchmarking, Surge AI, Weights & Biases

Source: https://www.youtube.com/watch?v=ulnkfr6E5z4
