# Answering your questions about GPT-Live

## Executive summary

This session provides a deep dive into GPT-Live, OpenAI's advanced voice model, detailing its architectural improvements over previous models like GPT-Realtime. Key advancements include 'full duplex' communication (allowing simultaneous listening and speaking) and a decoupled 'delegation' mechanism. For build engineers, the focus is on best practices for robust implementation, including using client delegation for controlled tool parameter collection, managing conversation context, and optimizing prompts for reliable backend handoffs.

## Key takeaways

- GPT-Live vs. GPT-Realtime Architecture: Previous voice models used a slow, cascaded system (STT -> LM -> TTS). GPT-Realtime consolidated this into a single model, making it faster but limited to turn-based interaction. GPT-Live improves this by enabling 'full duplex' (listening and speaking simultaneously) and splitting responsibilities via delegation.
- Delegation and Backend Integration: GPT-Live handles the conversation flow, while complex tasks are delegated to a separate backend model or system. This allows developers to swap out backend models or integrate GPT-Live into existing text-based applications, enhancing the voice experience.
- Handling Complex Environments: GPT-Live excels in noisy or multi-speaker environments by leveraging conversational context, rather than just detecting speech. Developers can refine this behavior by prompting the model to ignore side conversations and only respond when addressed.
- Advanced Tool Parameter Collection: To prevent the model from over-delegating or failing to collect necessary parameters, the recommended approach is 'client delegation.' The backend should provide structured context in the 'thinking' update, explicitly listing all required parameters and instructing the model not to delegate until all are gathered.

## Technical details

- Model Evolution: The evolution moved from a cascaded system (Speech-to-Text -> Language Model -> Text-to-Speech) to GPT-Realtime (single model, faster, but turn-based). GPT-Live introduced 'full duplex' and separated conversation handling from complex work delegation.
- Custom Voice Generation: Custom voices can be generated via two methods: 1) Prompt-to-voice (providing a natural language description of accent, tone, or pacing); or 2) Recording (with speaker consent). Prompting can further influence personality and formality.
- Optimizing Delegation Prompts: For reliable delegation, the GPT-Live prompt should be kept short, focusing only on *when* to delegate. The detailed tool operating procedure belongs in the backend prompt. Using the special `SpawnThinking` token (for GPT-Live 1) and monitoring transcripts via Decisions triggers are recommended best practices.
- Context Management: While OpenAI handles context compaction for the voice model automatically, developers should save critical information in the application layer. For granular control over context passing, 'client delegation' is recommended over built-in Responses/Decisions integration.

## Practical implications

- Migrating from GPT-Realtime to GPT-Live requires updating the architecture to leverage full duplex and delegation for a more natural, less latency-prone voice experience.
- When building voice agents that interact with tools, implement client delegation to gain explicit control over the context and parameters passed to the backend, preventing premature or failed delegation.
- For applications requiring high reliability in tool usage, ensure the backend prompt contains the full operating procedure, while the GPT-Live prompt only dictates the decision to delegate.

## Topics

GPT-Live, Voice AI, Delegation, Full Duplex, Tool Calling, Context Management, GPT Live prompting guide, Building with GPT-Live-1 and Cognition

Source: https://www.youtube.com/watch?v=5YXlrzC_RYE
