Your SKILL.md Is 70% Stuff the Model Already Knows - Nikolay Petrov, Intel
Summary
This presentation details advanced methods for evaluating the necessity and impact of AI skills provided to language models (agents). The core methodology involves splitting large skills into small, deterministic, testable claims (items). By running a two-arm comparison (with skill vs. without skill) at the item level, engineers can precisely map what the model already knows, what the skill teaches, and identify specific instructions that degrade performance. This process is crucial for skill optimization, reducing token usage, and ensuring that skills are adding genuine, non-redundant knowledge.
Key takeaways
-
Item-Level Skill Decomposition
5:40
Instead of evaluating skills holistically, the process recommends splitting skills into discrete 'items' (pairs of question and expected answer). This allows for granular testing of knowledge contribution, providing a 'simple forward counts matrix' of impact.
-
Two-Arm Deterministic Evaluation
2:50
Skills are validated by comparing model performance in two arms: one with no skill loaded, and one with the skill provided in the context. Metrics include the number of solved tasks, token usage, and time taken.
-
Identifying Skill Degradation
7:00
A critical finding is identifying items where the model performs correctly without the skill but shows degradation when the skill is added. This pinpoints unnecessary or poorly written instructions that actively hurt model capability.
Technical details
-
Skill Types and Functionality
25s
Skills can be 'knowledge type' (bringing new capabilities) or 'steering/process skills' (defining a repeatable flow of actions) [0:25].
-
Evaluation Metrics
170s
Evaluation compares performance across multiple factors, including the number of solved tasks (e.g., solving all five tasks with skill vs. three without) and implications on token usage and completion time [1:20].
-
Knowledge Assessment Depth
530s
Evaluation can be conducted using two methods: 'lenient' (checking general idea understanding) and 'strict' (checking for exact details, parameters, or values). The difference in performance between these two modes is often significant, especially with larger frontier models [5:30].
-
Optimization Framework
630s
The process utilizes a framework (e.g., `docume`) to provide an initial vector for skill optimization. This allows engineers to remove known, redundant information from the skill to reduce size and save tokens while maintaining performance [6:30].
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.