# Fable 5.2: The Self-Improvement Loop Keeps Loopin’ … (plus Gemini Carbon)

## Executive summary

The AI model landscape is accelerating rapidly, with Fable 5.2 and Gemini 4 Carbon pushing new boundaries in intelligence. The speaker details advanced benchmarking, noting that while models show exponential gains in specific domains (like coding and cybersecurity), common sense and general reasoning remain challenging. Key benchmarks, such as the Remote Labor Index and BenchCAD, suggest that AI can automate complex, defined tasks, though the ultimate timeline for full automation is debated. The analysis highlights that progress is driven by both model capability and the ability of top researchers to run parallel, complex experiments.

## Key takeaways

- AI Acceleration is Expected to Continue: Anthropic suggests that the rate of progress will continue to quicken, noting that the industry may soon pass the threshold of AI speeding up its own development rate. The speaker cites a 30-page METR report suggesting significant acceleration.
- Fable 5.2 Outperforms Competitors in Key Benchmarks: Fable 5.2 demonstrated impressive scores, notably achieving 41% on the Remote Labor Index, compared to GPT-4 Astra's 21% (at 9:01). It also outperformed Opus 5.5 in BenchCAD 2.1, and scored highly on the STEM Professor Bench (61.8%).
- Common Sense Reasoning Remains a Weak Point: Despite high scores on specialized benchmarks, models still struggle with common sense and real-world context. The speaker notes that the linear progress on Simple Bench suggests common sense improvement is much messier and slower than exponential gains in other domains.
- Cybersecurity and Engineering Gains are Significant: The Remote Labor Index covers tasks like video, graphic design, and architecture. Furthermore, BenchCAD 2.1 (Computer Aided Design) shows AI's capability to understand and analyze complex 3D engineering parameters, suggesting a major impact on manufacturing.

## Technical details

- Remote Labor Index (RLI): The RLI measures automation across diverse tasks (video, graphic design, architecture, etc.). Performance increased from 3.8% (Opus 4.5) to 41% (Fable 5.2). Extrapolations suggest reaching 98% could occur between May next year and October 2028.
- BenchCAD 2.1 (Computer Aided Design): This benchmark tests understanding of 3D structure and engineering parameters (e.g., bevel gears, compression springs). Fable 5.2 outperformed Opus 5.5. OpenAI's internal model, 'Bell,' achieved an average score of 87.5%, potentially leading in this domain.
- Hallucination and Integrity Benchmarking: While Mythos 5.2 scored highest on one measure of hallucinations (72%), the speaker found that Gemini 4 Carbon performed best on the 'hallucination rate' (BSing) and 'Integrity Bench,' which measures if the model's stated confidence matches its actual accuracy.
- Benchmark Integrity (Max Bench): The speaker introduced 'Max Bench' to test for memorization/cheating by tweaking existing benchmarks. Claude models were observed to be less 'bench maxed' compared to others.

## Practical implications

- AI models are rapidly moving from simple automation to complex, multi-domain tasks (e.g., architectural planning, quant trading, chip design).
- For developers, the focus should shift from basic coding tasks to defining complex objectives and managing the AI's iterative, multi-stage output, as the final product may take longer than the initial draft.
- The ability to define and test highly specific, real-world benchmarks (like those created by quant trading or chip design firms) is critical for assessing true model capability.
- The current progress suggests that while full automation of all labor is distant, AI can provide brilliant first drafts for clearly defined, computer-based objectives within the next year.

## Topics

Artificial Intelligence, Model Benchmarking, Software Engineering, Automation, Machine Learning, Claude Fable 5.2 and Mythos 5.2, METR: AI R&D acceleration at Anthropic, Remote Labor Index, Anthropic Capabilities Index, benchcad.com leaderboard

Source: https://www.youtube.com/watch?v=583ThJIqQ7k
