# Your Spec Has Bugs. Catch Them Before Your Agent Codes. - Rajath Mahendrappa Kotyal, AWS

## Executive summary

The Keto Specification Analyzer addresses the critical issue of 'quiet bugs' in specifications—ambiguities, conflicts, or missing rules—that can lead to incorrect code when passed to an AI coding agent. The system uses formal methods, specifically pairing LLMs with SMT solvers, to detect these flaws before code generation. This process aims to prevent significant drops in code accuracy and reduce regressions in the codebase.

## Key takeaways

- Formal Verification of Specifications: The tool uses automated reasoning and SMT solvers to translate natural language rules into formal formulas, allowing the detection of logical inconsistencies (e.g., two rules that cannot both hold true).
- Three-Step Analysis Process: The process involves: 1) LLM refinement of wording; 2) Formalization into SMT formulas; 3) Iterative sampling (e.g., 7-10 times) and clustering interpretations based on logical equivalence.
- Identifying Specification Flaws: The system can detect three types of issues: Conflicts (rules contradicting each other), Ambiguities (a single sentence having multiple meanings), and Gaps (missing rules or coverage for specific scenarios).

## Technical details

- Automated Reasoning & SMT Solvers: The core mechanism involves passing English specifications to an SMT solver (Satisfiability Modulo Theories). The solver returns a Boolean value, indicating if the expression is valid, true, or false, thereby verifying the logical consistency of the rules.
- LLM Interpretation Clustering: To resolve ambiguity, the system samples the rule translation multiple times (e.g., 7 or 10 times) and clusters the resulting interpretations based on logical equivalence. This helps determine if the LLM's behavior is inherent or influenced by external factors.
- Specification Failure Rates: The speaker cites research showing that unclear prompts can cause a 20-40% drop in LLM code accuracy, and 60-90% of generated code may still be incorrect, leading to increased regressions.

## Practical implications

- Reduces the risk of deploying code based on flawed or ambiguous requirements.
- Provides a pre-coding review layer for specifications, acting as a 'design doc' verification step.
- Minimizes regressions by catching logical conflicts and gaps before the code enters the codebase.

## Topics

Formal Methods, Automated Reasoning, LLMs, Specification Analysis, Software Engineering, SMT Solvers, Keto Specification Analyzer, AWS

Source: https://www.youtube.com/watch?v=0EnLxIbAvYk
