Pinning the agent skill supply chain

Part of my notes on running coding agents with least privilege.

A skill is a markdown file that tells an agent what to do, often with scripts next to it. The agent runs those steps with my permissions. I would never install a package without a lockfile, but for a long time I installed skills from GitHub with no record of which commit I got. This post covers how I track them now, and how much of that tracking is still fiction.

Part 13 reports 202 entries at consolidation time. Today fd SKILL.md ~/dotfiles/skills | wc -l prints 156, and ls ~/.agents/skills | wc -l prints 253. Of those 253 entries, 86 are symlinks. I did not reconcile the two totals, so I treat them as separate facts.

What a skill can do

The instructions in a skill are text, but the frontmatter can also grant tools. The public playwright-skill from lackeyjb/playwright-skill declares this in its frontmatter:

allowed-tools: Bash(node:*) Bash(npm:*) Read Write

Its setup step tells the agent to run cd "$SKILL_DIR" && npm run setup, which installs Playwright and Chromium, and the frontmatter says that needs network access. So installing this skill means the agent will run npm and node with arbitrary arguments on my behalf, and a later change to the repo changes what those commands do. I copied this skill into my dotfiles after adopting it, but the point holds for any skill: the text I review today is not the text I get after the next pull.

The catalog

skills/catalog.yaml is the source of truth for external skill sources. Its header defines six install methods:

lockfile   tracked in ~/.agents/.skill-lock.json; restored via `npx skills install`
npx-add    installed via `npx skills add <repo>`
git-clone  cloned to ~/.agents/<dir>, skills symlinked to ~/.agents/skills/
dotfiles   committed to ~/dotfiles/skills/, deployed by home-manager switch
skip       evaluated, not worth adding
evaluate   promising but needs deeper review before adoption

A typical entry has a repo, a URL, a status, a note and a list of skills with a category and a one line description. skills/REGISTRY.md sits beside it and records the decision made for each source, so I do not re-evaluate the same repo twice.

I counted entries per section by matching - repo: lines and inline { name: lines. The skill name count includes names under skip, so treat it as approximate.

The dotfiles count is the file count from the start of the post. It includes skills I wrote myself, and I am not listing those.

What is pinned and what is not

Copying a skill into ~/dotfiles/skills/ is the only method that gives me a fixed revision, because my own Git history then holds the text. Home Manager deploys those files read-only from the Nix store, and nix/home/ai-tools/lib.nix builds the home.file entries by enumerating the skill directories.

The other methods track whatever the source serves. The catalog has no field for a commit, tag or hash. I searched it for rev:, commit:, ref: and sha, and the only hits were ordinary words such as "shared" and "share" inside descriptions. So the 20 npx-add repos and the 10 git-clone repos are unpinned, 30 of the 33 installable repos. For git-clone, the restore script runs git clone --depth=1 on the URL, which takes the default branch tip, and tells me to run git pull if the directory already exists.

The lockfile method is supposed to be the pinned one. The catalog says ~/.agents/.skill-lock.json tracks those skills. That file does not exist on this machine, so I cannot tell you what it records. I also did not open the skills CLI's source, so I do not know whether it stores commit hashes or only repo names.

The pin script that no longer pins

scripts/pin-skill-hashes.sh fetches the current HEAD and Nix hash for two public repos, obra/superpowers-skills and nimrodfisher/data-analytics-skills, and prints a rev and hash to paste into nix/home/ai-tools.nix. That file is now the directory nix/home/ai-tools/. I ran rg -n 'rev *=' nix/home/ai-tools and it found nothing. There is no pinned revision anywhere in that directory.

The comment in tools-and-skills.nix still says "Revs are pinned" and points at the script. The code below that comment does something else for data-analytics-skills: an activation step runs git clone into ~/.agents/skills/ if the directory is missing, with the comment that it is a live clone so git pull works. So the script works as a tool, the comment describes a design I abandoned, and nothing consumes the output. I have not fixed either file. I did not find when the pin block was removed.

Restore

scripts/restore-agent-skills.sh reads the catalog and rebuilds ~/.agents/skills/. It takes --dry-run, which prefixes each command with [dry-run] instead of running it, and --section lockfile, npx_add or git_clone. It skips the dotfiles section because Home Manager owns it. If the lockfile is missing it prints a warning and skips npx skills install, which is the case here. I did not run it, because a dry run still touches ~/.agents with mkdir -p before anything else, and I was asked not to change that directory.

Review before install

The repo documents a decision record, not a review procedure. REGISTRY.md tells me to consult it before ingesting a skill, and each entry records adopted, adapted, merged or rejected with a reason. That stops repeat work. It does not say what I read in a skill before adopting it, and I have no written checklist for scripts or allowed-tools. I will not claim one.

What I could not verify

• What ~/.agents/.skill-lock.json records, since the file is absent and I did not read the CLI source.
• Why 202 entries in Part 13 became 253 here, and how the 156 files and 253 entries relate.
• When the Nix revision pins were removed.
• Whether any npx-add or git-clone skill has changed upstream since I installed it. I have no revision to compare against.