# What Runs Where? Local AI in Production — Ahmad Osman, Osmantic

## Executive summary

The future of AI is shifting toward specialized, locally run models, driven by critical software and systems engineering. The speaker argues that performance is not solely determined by model size or cloud compute, but by the efficiency of the inference engine, quantization methods, and the ability to handle complex agent workflows. Local AI requires a holistic approach, treating the entire pipeline—from token generation to decision-making—as an optimization problem.

## Key takeaways

- Specialization Over Frontier Intelligence: The trend is moving toward small, specialized models (e.g., 1-2 billion parameter models) that excel at specific tasks (like coding or web development) rather than massive, general frontier models.
- The Importance of the Inference Engine: The speaker emphasizes that a custom inference engine, optimized 'per model per chip,' is necessary to extract maximum value from hardware, as off-the-shelf solutions often lack the required software optimizations.
- Local AI Requires a Complete Harness: For agents to function effectively, they require more than just a model; they need a complete harness (like the 'Portal' harness) that includes search capabilities and decision models to prevent the system from becoming 'plotted and useless.'
- Continual Learning is a Local Requirement: True continual learning—where a model adapts to a specific user's data—cannot be achieved on cloud platforms, as providers train for a general user distribution. It must run locally.

## Technical details

- Inference Bottlenecks (Prefill vs. Decode): Prefill is compute bound, while decode is memory bandwidth bound. Hardware selection must be based on the specific workflow (e.g., long prompts are compute-intensive; short prompts are memory-intensive).
- Quantization Techniques: Quantization is not monolithic; dynamic quantization involves assessing which layers are least used (e.g., 0.01%) and compressing them (e.g., to 1-bit) while keeping critical layers at higher precision (e.g., 16-bit).
- Performance Gains via Optimization: Optimizations, such as the Deepseek V4.1 Flash KV cache shrinking, can lead to massive performance increases (e.g., 437 times reduction in memory for a given task).
- Agent Workflow and Parallelism: Agents are increasingly the primary token consumers. Running multiple agents in parallel (batching) is a key use case where high memory bandwidth (like the RTX Pro 6000) significantly outperforms sequential processing on other platforms.

## Practical implications

- When selecting hardware, evaluate the required workflow (compute-bound vs. memory-bound) rather than relying solely on raw specs.
- Focus on building or utilizing specialized inference engines tailored to specific models and hardware platforms to maximize efficiency.
- Treat the AI system as a complete pipeline, requiring tools like search and decision models to move beyond simple markdown-file-based 'harnesses.'
- For high-throughput, parallel tasks (like agent execution), GPUs are often superior to consumer CPUs/MPUs, provided the software stack is optimized.

## Topics

Local AI, Inference Engineering, LLM Optimization, Agentic Systems, Hardware Acceleration, Osmantic (ODS), RTX Spark, Quen 3.5, Deepseek V4.1 Flash, Haiku 5.5

Source: https://www.youtube.com/watch?v=GKSQYbELDzw
