AI Engineer

Your SKILL.md Is 70% Stuff the Model Already Knows - Nikolay Petrov, Intel

Published 2026-11-11 · Duration 14:01

Summary

This presentation details advanced methods for evaluating the necessity and impact of AI skills provided to language models (agents). The core methodology involves splitting large skills into small, deterministic, testable claims (items). By running a two-arm comparison (with skill vs. without skill) at the item level, engineers can precisely map what the model already knows, what the skill teaches, and identify specific instructions that degrade performance. This process is crucial for skill optimization, reducing token usage, and ensuring that skills are adding genuine, non-redundant knowledge.

Download summary

Key takeaways

  1. Item-Level Skill Decomposition 5:40

    Instead of evaluating skills holistically, the process recommends splitting skills into discrete 'items' (pairs of question and expected answer). This allows for granular testing of knowledge contribution, providing a 'simple forward counts matrix' of impact.

  2. Two-Arm Deterministic Evaluation 2:50

    Skills are validated by comparing model performance in two arms: one with no skill loaded, and one with the skill provided in the context. Metrics include the number of solved tasks, token usage, and time taken.

  3. Identifying Skill Degradation 7:00

    A critical finding is identifying items where the model performs correctly without the skill but shows degradation when the skill is added. This pinpoints unnecessary or poorly written instructions that actively hurt model capability.

More digests on AI Agents

Technical details

  • Skill Types and Functionality 25s

    Skills can be 'knowledge type' (bringing new capabilities) or 'steering/process skills' (defining a repeatable flow of actions) [0:25].

  • Evaluation Metrics 170s

    Evaluation compares performance across multiple factors, including the number of solved tasks (e.g., solving all five tasks with skill vs. three without) and implications on token usage and completion time [1:20].

  • Knowledge Assessment Depth 530s

    Evaluation can be conducted using two methods: 'lenient' (checking general idea understanding) and 'strict' (checking for exact details, parameters, or values). The difference in performance between these two modes is often significant, especially with larger frontier models [5:30].

  • Optimization Framework 630s

    The process utilizes a framework (e.g., `docume`) to provide an initial vector for skill optimization. This allows engineers to remove known, redundant information from the skill to reduce size and save tokens while maintaining performance [6:30].

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.