AI Engineer

The Agents Were Fine. Everything Around Them Broke. - Joey Wang, Harvey

Published 2026-11-10 · Duration 12:58

Summary

Joey Wang discusses the architecture and scaling challenges of Spectre, an internal cloud agent platform at Harvey. The core thesis is that while LLM agents are powerful, the true product is the 'harness'—the surrounding infrastructure that manages, validates, and merges agent-written code. Key architectural improvements included implementing follow-up loops for automated fixes, making all platform actions callable via CLI (allowing agents to drive other agents), and introducing a fair launch queue to manage massive task bursts and prevent cluster saturation.

Download summary

Key takeaways

  1. The Product is the Harness, Not the Model 2:00

    The value lies in the infrastructure (the 'harness') that manages the agent workflow, including checking out every ripple, opening the PR, and ensuring the merge. This layer is what enables agents to operate on the company's real stack and secrets, which is necessary for scaling beyond local machine capabilities.

  2. Automating the Full Development Lifecycle 3:25

    The most impactful feature is the follow-up loop, which allows an agent to read a CI failure, push a fix, and continue the process automatically, moving beyond just writing initial code. This transforms an agent that merely writes code into one that 'finishes the job.'

  3. Enabling Agent-Driven Workflow via CLI 5:50

    By making every action (e.g., publishing a page, starting a task, turning on automerge) callable from the Command Line Interface (CLI), the platform allows any agent—whether running on a laptop, in a sandbox, or via CI—to drive the entire workflow, significantly increasing the platform's utility.

  4. Scaling for Massive Task Bursts 10:20

    To handle sudden, large-scale task increases (e.g., 900 tasks in four hours), a fair launch queue was implemented. This queue ensures that tasks are processed in a fair, orderly manner, preventing cluster overload and maintaining system stability.

More digests on Cloud Agents

Technical details

  • Agent Platform Architecture (Spectre) 140s

    Spectre is the cloud agent platform built inside Harvey. It is designed to sit underneath various agent interfaces (Cursor, CodeEx, etc.), providing access to the company's real stack, secrets, and alerts.

  • Automated Fix Loops 205s

    The follow-up loop reads CI failures and automatically pushes subsequent fixes, automating the debugging cycle and ensuring the PR moves closer to merge status without manual intervention.

  • Workflow Automation and Callability 350s

    The platform was designed so that if a human could click a function (e.g., 'publish a page'), an agent could call it via the CLI. This expanded the scope from terminal-only tasks to web app and browser-based actions.

  • Resource Management and Scaling 620s

    The implementation of a fair launch queue and dispatcher was critical for handling task bursts, improving task latency from nearly 2 seconds to 70 milliseconds during high load.

  • Agent Guardrails 720s

    To prevent runaway automation, a guardrail was implemented: if an automation agent fails five times in a row, it automatically switches itself off and can be assigned a budget or expiry date.

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.