# You Can't Tell What an Agent Skill Will Do by Reading It - Roey Ben Chaim, Zenity

## Executive summary

The security analysis of AI agent skills requires moving beyond static code inspection. Because skills can dynamically fetch payloads, install packages, and execute processes whose effects are not visible in the initial manifest (like `SKILL.md`), relying solely on static scanners is insufficient. The solution involves using a controlled 'agent detonation chamber'—a sandbox that simulates the full runtime environment, captures process creation, file system activity, and network traffic to accurately determine a skill's true intent, whether malicious or benign.

## Key takeaways

- Static Analysis is Insufficient: Reading only the `SKILL.md` file is inadequate for determining a skill's true function. Skills often use the manifest only as an entry point, invoking external endpoints or downloading packages that contain the actual payload or instructions.
- The Need for Runtime Behavior Analysis: To understand a skill's true behavior, it must be executed within a controlled 'detonation chamber' (sandbox). This process tracks the complete chain of execution, including payloads fetched from external sources.
- Comprehensive Evidence Capture is Critical: A robust analysis environment must capture more than just agent traces. It must monitor process creation events, file system activity (e.g., post-installation scripts from `npm install`), and network packet capturing to detect info-stealing behavior.

## Technical details

- Agent Skill Architecture: An agent skill is defined as a 'capability of capability on demand,' containing instructions, resources, and potentially code. The risk arises because the code is written by external parties.
- The Detonation Chamber Components: The advanced analysis environment includes: 1) A Sandbox, 2) A World Builder (to supply necessary services, e.g., Notion service), 3) An Operator (to cover all possible execution paths/trajectories), and 4) Instrumentation (for processes, filesystem, and network traffic).
- Malicious Techniques Observed: Observed attacks include: 1) **Chained Skills:** A skill installing a secondary skill that fetches a payload from GitHub. 2) **Misleading Packages:** Using non-official npm registries and packages that act as info-stealers. 3) **Persistence:** Adding instructions to reinstall the skill if it is uninstalled. 4) **Hook Abuse:** Triggering a skill on common hooks, such as `user prompt submit`.

## Practical implications

- Treat agent skills as software dependencies, requiring the same level of scrutiny as third-party libraries.
- Implement strict access controls and privilege limitation for skills to restrict what they can access (e.g., cloud credentials, identity services).
- Mandate the inspection of all loaded components and observed runtime behavior, not just the initial manifest.
- Utilize controlled execution environments (sandboxes) that capture process, file, and network events to detect hidden payloads.

## Topics

AI Safety, Agent Security, Software Supply Chain, Sandboxing, Dependency Management, neoto.io, Zenity Labs blog

Source: https://www.youtube.com/watch?v=_7P580m4kPY
