Agents Made Code Free. Maintaining It Didn't Get Cheaper. - Dmitry Buykin, Maersk
Summary
While AI coding agents have made generating code nearly free, maintaining large, complex production systems remains challenging. Dmitry Buykin outlines a process for managing this entropy, arguing that continuous, small-scale sweeps and robust feedback loops are necessary. The solution involves moving beyond simple code testing to establish human-owned decision tables (the 'pro' layer) and implementing rigorous runtime guardrails to prevent system drift and ensure that changes are validated against real-world tasks and expert consensus.
Key takeaways
-
Shift Focus from Code to Process (Pro)
2:00
Code tests are insufficient for validating complex business logic. The 'pro' layer, which contains human-defined rules and decisions, must be treated as a first-class citizen, requiring specialized validation methods beyond traditional unit testing.
-
Implement Runtime Guardrails and Context Management
7:30
To control AI agents, the system must limit the model's context (the 'mind' stone). The runtime environment must validate context for consistency and ensure the model only sees the necessary, verified facts, preventing over-generalization or hallucination.
-
Establish a Single Source of Truth (Ledger)
11:20
All changes and decisions must be recorded in a persistent ledger. This ledger tracks both 'what happened' and 'what is true now,' ensuring that the next session starts from a verified state and that every change is traceable to its root cause.
-
Adopt Continuous, Incremental Auditing
14:10
Instead of large, infrequent rewrites, the system must operate through weekly, small sweeps. Gauges (like 'blast radius') and real-world task replaying are used to detect subtle drifts and ensure that every change is validated against live business outcomes.
More digests on AI Engineering
Technical details
-
Pro vs. Code Validation
140s
The system distinguishes between 'code' (which can be tested) and 'pro' (process/rules). Validation for 'pro' requires human expertise and specialized review, as agents can only fix what is easy to see, leading to potential consistency breaks.
-
Mutation Testing
160s
Initially, the team borrowed mutation testing to validate the 'pro' layer, but found that for complex agent instructions and tests, human judgment remains critical for determining what must survive a rewrite.
-
Workflow Generation and State Management
330s
When agents repeatedly follow the same path through the 'pro' layer (a 'walk'), that path is converted into a formal workflow tool. The system maintains a boundary where 'code prepares the facts and carries out the decision,' while 'experts own the decision table.'
-
Context and Fact Validation
450s
The runtime environment acts as a guardrail, limiting the model's context to verified facts. The model passes a handle (a short name) for data that the runtime already holds, ensuring the context remains small, consistent, and accurate.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.