Vetting a Third-Party Skill

Step 1Two executable surfaces, and most reviews read one of them.

A skill is a folder: a SKILL.md file carrying instructions, plus whatever scripts and reference material those instructions point at. Both halves execute, by different interpreters, and the review habits people bring from software security are aimed at the wrong one.

• The scripts execute in a shell, with the agent's own privileges, when the instructions tell the agent to run them. This is the half that looks like code, so it is the half that gets read. The scale study below found skills bundling executable scripts 2.12 times more likely to contain a vulnerability than instruction-only skills (odds ratio 2.12, p<0.001) — so the instinct is not wrong, just incomplete.
• The prose executes in the model, which was built to follow instructions and cannot distinguish one the author meant helpfully from one they meant otherwise. There is no sandbox around this half, no syntax to audit, and no diff that looks suspicious.

The largest documented campaign to date went through the second surface. In late January 2026, Koi Security audited 2,857 skills on the ClawHub registry and found 341 malicious ones; 335 belonged to a single campaign, ClawHavoc, which delivered the Atomic Stealer macOS infostealer. The payload was not obfuscated code in a bundled script. It was a fake "prerequisites" section in SKILL.md, telling the agent to download and run an installer before starting the task — an entirely ordinary-looking setup step, in the file whose whole purpose is to be read and acted on.

Step 2What the format does not carry, and why that decides the whole problem.

Three fields that would make review tractable are absent from a skill folder, and their absence is not an oversight — it follows from the format's goal, which is that the same folder works identically in every host implementing the standard.

• No version. There is no number to increment, so there is nothing for a review to attach to. "Reviewed" becomes a statement about a moment, and nothing in the folder records which moment.
• No publisher identity. The folder does not say who wrote it in a way anything can verify. Whatever attribution exists lives in the registry or repository around it — a different system, with different access control, which a copied folder leaves behind.
• No capability declaration. Nothing states which tools, network destinations or files the skill expects to touch, so a host cannot grant narrowly even when it wants to. The skill simply inherits whatever the agent already holds, which is the ambient authority problem arriving by a new route.

Compare an MCP server, which has all three. A server is a URL with an identity behind it and an operator on the other end, so a host can pin the endpoint, verify the token audience, and revoke access in one action that takes effect immediately. A skill is a folder somebody copied. There is no endpoint to deauthorise, no token to revoke and no operator to notify — just files already sitting in a repository, loaded by whichever agent next matches the description line. Skills vs subagents vs MCP vs hooks works through why you would pick a skill at all; the point here is that picking one means picking the mechanism with the fewest revocation handles.

One consequence deserves saying plainly, because teams keep rediscovering it during incident review: what decides whether a skill is ever loaded is its description line, not its contents. A skill the agent never matches is inert; a skill with a broadly worded description gets loaded on tasks nobody anticipated. Agent skills argues the description is the load-bearing field for usefulness. It is equally the load-bearing field for exposure — one line of prose that no review checklist treats as security-relevant.

Step 3Read the base rate before you design the process.

Agent Skills in the Wild (arXiv 2601.10338) crawled two skill marketplaces in December 2025, collecting 42,447 skills and analysing 31,132 after deduplication, along with 3,574 bundled executable scripts. Its detector, SkillScan, pairs static analysis with LLM classification and was calibrated against 500 hand-annotated skills. The result: 26.1% — 8,126 skills — matched at least one of fourteen dangerous patterns, split into 12.8% low severity (poor practice), 8.1% medium (ambiguous intent) and 5.2% high severity suggesting malicious intent.

Read that 5.2% carefully. It is a severity judgement produced by a detector, not a count of confirmed malware, and anyone quoting it as "one in twenty skills is malicious" has promoted a classifier output into a finding. The defensible version is narrower and still decisive: on a large public corpus, a pattern-based detector finds something worth a human's attention in roughly a quarter of skills and puts about a twentieth in its most serious tier. At that rate a process built on spot checks will miss steadily — and a process built on "we only install popular ones" leans on install count, a signal ClawHavoc showed to be cheap to manufacture.

Step 4Make the revision the unit, and the review becomes a diff.

All of the above points at one procedural change, and it is cheap. Stop installing skills by name from a registry and start vendoring them: copy the folder into a repository you control, at a commit you record, and take every later update as a pull request. This is the move pinning and verification argues for everywhere else — a name resolves through someone else's namespace, a commit is the content — and for skills it is the only way to make "reviewed" mean anything, because it supplies the version field the format lacks.

What it buys you concretely: the second review is a diff instead of a re-read. The first review of a skill is real work; the twentieth update is four changed lines of prose a reviewer can actually evaluate. Without the pin, every update is either an unreviewed change or a full re-read — and in practice it is the former.

Then make the first review mechanical enough that someone will actually perform it. The enumeration matters more than the judgement, because the failure mode is not misjudging a file. It is not knowing the file was there.

# Enumerate the surface before reading anything. Three questions:
# what executes, what it reaches, and what the prose tells the agent to run.

# 1. Everything that is not prose or reference material
find skill/ -type f ! -name "*.md" ! -name "*.txt" -printf "%M %p\n"

# 2. Every network destination mentioned anywhere in the folder
grep -rEoh "https?://[^ )\"]+" skill/ | sort -u

# 3. Every imperative in the prose that asks for execution or install
grep -rniE "curl|wget|pip install|npm i|chmod|sudo|prerequisite|before you" skill/*.md

# 4. The one line that decides whether any of this is ever loaded
head -20 skill/SKILL.md

Four readings, in this order, and each one has a disposition attached:

• The description line. Is it scoped to the task you want, or is it broad enough to match work you never intended? A description that would plausibly match "help me with files" is a standing invitation. Narrow it in your vendored copy — this is your folder now.
• The prose, read as instructions to an obedient process. Any step that fetches and runs something is the ClawHavoc shape and needs a reason that survives being said out loud. "Install our helper first" is not one.
• The destinations. Every host in the folder should be one you can name a purpose for. An unexplained domain is the finding; you do not need to prove what it does.
• The scripts. Read them last, because by now you know what the prose claims they do, and the interesting defect is the divergence between the two.

Step 5Capability survives mutation. Review does not.

A pinned revision solves the review problem and leaves a harder one: you still had to be right. A scan and a read are both judgements about contents, and contents you judged correctly today can be replaced tomorrow in any copy you did not pin. The control that does not depend on your judgement being right is a limit on what a loaded skill can reach at all.

Three limits carry nearly all of the value, and they are the same three that convert any instruction into an incident:

• Network egress. An allowlist on the agent's outbound traffic turns "fetch and run this installer" into a failed request and "post the file here" into a deny-log entry. This is the highest-leverage single control on the surface, and egress control for agents is the implementation treatment.
• Shell and execution. If the agent running a third-party skill cannot spawn a process, the bundled-script half of the problem is gone outright. Most skills that genuinely need know-how rather than execution lose nothing.
• Credential reach. A skill inherits the agent's credentials, so the question is what those credentials are rather than what the skill asks for. Reference secrets through a broker instead of placing them where the loop can read them, per secrets management for agents.

Put the three together and you have the configuration where a hostile skill is a wasted turn rather than a breach — which is the only state that is stable against the folder changing. Note that this is also the lethal trifecta read from the other direction: untrusted content plus credentials plus an exfiltration channel, where a third-party skill is a particularly direct way for the first leg to arrive, and where removing either of the other two is what actually helps.

Price it honestly. An egress allowlist on an agent that browses is real work and will break things in the first week. No-shell is free for a documentation skill and impossible for a build skill. The recommendation is not "lock everything" but "decide the three per skill class, once", because the alternative is deciding them per skill, never.

Step 6Three numbers, and when none of this is worth it.

The measurements that keep this honest are small and none of them is a scan result.

• Inventory completeness. Can you list every skill loadable by every agent you operate, and the commit each one is pinned at? If not, that is the whole finding and nothing else on this page matters yet. Ecosystem install counts run into the hundreds of millions; yours is a number you can actually know.
• Time to revoke. From "this skill is bad" to "no agent can load it", measured by doing it once in anger. For a vendored folder this is a commit; for skills installed per-developer from a registry it is a message in a channel and a hope, and the gap between those two is the number worth reporting upward.
• Update review rate. What fraction of skill updates in the last quarter went through a diff somebody read? If the honest answer is low, the pin is decorative and you are relying on the publisher.

And the case for doing almost none of this: a skill you wrote, in a repository you control, loaded by an agent with no credentials and no egress, is not a supply-chain problem and does not need a review process. The same is true of a skill that contains only prose about your own conventions, running in an agent whose tool set cannot reach anything external. Most of this page is about the third-party case specifically — and the fastest way to shrink the problem is to notice how many of your skills did not need to come from somewhere else. Tool catalog lifecycle is the adjacent discipline once the count grows.