# Your SKILL.md Is 70% Stuff the Model Already Knows - Nikolay Petrov, Intel

## Executive summary

This presentation details advanced methods for evaluating the necessity and impact of AI skills provided to language models (agents). The core methodology involves splitting large skills into small, deterministic, testable claims (items). By running a two-arm comparison (with skill vs. without skill) at the item level, engineers can precisely map what the model already knows, what the skill teaches, and identify specific instructions that degrade performance. This process is crucial for skill optimization, reducing token usage, and ensuring that skills are adding genuine, non-redundant knowledge.

## Key takeaways

- Item-Level Skill Decomposition: Instead of evaluating skills holistically, the process recommends splitting skills into discrete 'items' (pairs of question and expected answer). This allows for granular testing of knowledge contribution, providing a 'simple forward counts matrix' of impact.
- Two-Arm Deterministic Evaluation: Skills are validated by comparing model performance in two arms: one with no skill loaded, and one with the skill provided in the context. Metrics include the number of solved tasks, token usage, and time taken.
- Identifying Skill Degradation: A critical finding is identifying items where the model performs correctly without the skill but shows degradation when the skill is added. This pinpoints unnecessary or poorly written instructions that actively hurt model capability.

## Technical details

- Skill Types and Functionality: Skills can be 'knowledge type' (bringing new capabilities) or 'steering/process skills' (defining a repeatable flow of actions) [0:25].
- Evaluation Metrics: Evaluation compares performance across multiple factors, including the number of solved tasks (e.g., solving all five tasks with skill vs. three without) and implications on token usage and completion time [1:20].
- Knowledge Assessment Depth: Evaluation can be conducted using two methods: 'lenient' (checking general idea understanding) and 'strict' (checking for exact details, parameters, or values). The difference in performance between these two modes is often significant, especially with larger frontier models [5:30].
- Optimization Framework: The process utilizes a framework (e.g., `docume`) to provide an initial vector for skill optimization. This allows engineers to remove known, redundant information from the skill to reduce size and save tokens while maintaining performance [6:30].

## Practical implications

- Use item-based testing to validate that every line or concept in a skill is necessary and adds unique value.
- Implement deterministic two-arm testing (with/without skill) to quantify the exact performance impact of adding or modifying skill context.
- Prioritize skill optimization by removing redundant knowledge to reduce context window size and associated token costs.
- Use the analysis to guide model training, focusing on providing exact details and data where general understanding is insufficient.

## Topics

AI Agents, Large Language Models (LLMs), Prompt Engineering, Skill Optimization, Context Management, Build Validation, Nikolay Petrov, Intel

Source: https://www.youtube.com/watch?v=yzx4IuEh_tU
