# ‘We may release models we can’t control’: Amodei, Altman … and Gemini Carbon

## Executive summary

The video summarizes recent high-profile developments in frontier AI, focusing on debates around model control, capability, and safety. Key discussions include Anthropic's stance on releasing models whose behavior cannot be adequately verified, the shift from absolute to relative safety commitments (Responsible Scaling Policy v3), and the increasing capability of models like Gemini 4 Carbon. The speaker argues that the complexity of modern LLMs, likened to chess or advanced calculators, makes full human understanding or prediction of their rules nearly impossible, suggesting that the focus must shift to robust containment and alignment.

## Key takeaways

- Anthropic's Stance on Unverifiable Models: Anthropic stated that if they become less able to detect or assess misalignment as models advance, they may deploy models whose behavior they cannot adequately verify. This suggests a shift in safety philosophy from absolute control to managing uncertainty. (18:41)
- Amodei's Rationalizations for Strategy: Dario Amodei's strategy involves a 'race to the top,' arguing that Anthropic must compete at the frontier. He also emphasized that the primary risk is superintelligence arising from recursive self-improvement (RSI), rather than human misuse. (04:32)
- Google's Gemini 4 Carbon Benchmark Performance: Gemini 4 Carbon is noted for its strong performance on benchmarks like the GDPx benchmark for agentic capabilities, achieving high scores while being significantly more cost-effective than competitors like Astra or Opus 5.5. (11:43)
- The Limits of Human Understanding in AI: The speaker posits that LLMs are fundamentally complex systems, arguing that predicting their rules or outputs is nearly impossible, even for advanced engines like Stockfish. This suggests that development must focus on containment and bounding, rather than full comprehension. (14:00)

## Technical details

- Responsible Scaling Policy (RSP): Anthropic shifted from an older, stricter policy requiring absolute commitments to checking model capabilities, to a more relative commitment. The new policy requires a strong argument that catastrophic risk is contained, allowing for delayed development and deployment as needed. (09:10)
- Model Architecture & Capabilities: The discussion highlighted the potential shift from tokenization (breaking words into chunks) to modeling on raw bytes, which could allow models to learn abstractions directly from raw data. (12:30)
- Containment and Sandboxing: OpenAI's approach involves securing frontier model training by conducting it with no live internet access. The principle remains that preventing unauthorized arbitrary access to the live internet is paramount, even if the sandbox fails. (15:00)
- Recursive Self-Improvement (RSI): The concept of RSI—a model improving itself—is a central concern, with Anthropic co-founder Jack Clark predicting that the risk from a system a hundred times smarter than the smartest person is low, while the risk from RSI is high. (00:00, 09:00)

## Practical implications

- Safety policies are shifting from absolute commitments to relative risk assessments, acknowledging the practical difficulty and cost of maintaining perfect control at the frontier.
- The industry is moving toward advanced containment methods (e.g., training without live internet access) to mitigate the risk of unauthorized access.
- The increasing complexity of models necessitates a focus on architectural bounding and robust testing, as full human comprehension of model rules is deemed impossible.
- Competition remains intense, with models like Opus 5.5 and Gemini 4 Carbon demonstrating high performance and cost efficiency, driving rapid development cycles.

## Topics

AI Safety, Frontier AI Models, Alignment, System Architecture, AI Policy, Anthropic Staff Survey, Anthropic Responsible Scaling Policy v3.0, OpenAI Securing Frontier Model Training, Google Introducing Gemini 4 Carbon

Source: https://www.youtube.com/watch?v=0Kw6_DTZPrY