How to test an agent skill

Testing a skill is four separate questions, not one: does the file parse, does it install where the docs say, does the host actually load it, and does it make the answer better. Each one can pass while the next fails, so we check them one at a time and report each result on its own. That’s the ladder this site uses everywhere, explained in full at /methodology/. Below is how we worked through the first three, by hand, on one small skill; the fourth is still unmeasured.

The ladder

1. Format: is the file valid against the published spec?
2. Installs: does it land at the documented path, by the documented route?
3. Loads: does the host actually discover it and use it, on a real task, in a fresh session?
4. Helps: does it measurably improve the outcome compared with not using it?

Passing one says nothing about the next. A skill can validate perfectly and still never load; it can load reliably and still not help. We rebuilt this ladder on explain-error, the small skill (explains a pasted error or stack trace, suggests one fix) we used across this set of guides. Its full contents are in the SKILL.md format.

Step 1: format

npx skills-ref validate ./explain-error

We got Valid skill: ./explain-error (skills-ref v0.1.5), and npx skills-ref read-properties ./explain-error parsed name and description cleanly. This only checks frontmatter shape: name pattern and length, description non-empty. It does not check that anything the instructions reference actually exists; see how references and assets work inside a skill for what we found when we broke a reference on purpose and re-ran this same check.

Step 2: installs

We copied the folder to the documented project paths and confirmed the files landed intact:

mkdir -p proj-claude/.claude/skills
cp -R explain-error proj-claude/.claude/skills/explain-error

Same pattern for Codex, at both .agents/skills/explain-error/ (current docs) and .codex/skills/explain-error/ (older path, which still works on Codex CLI 0.157.1). We also ran the installer, npx skills add <path> -a claude-code -y and -a codex -y (skills CLI v1.7.0), which wrote full copies to .claude/skills/ and .agents/skills/ respectively, plus a skills-lock.json recording the source and a content hash; the installer doesn’t target .codex/skills/, which we reached only by hand-copy. Installs only confirms the files exist where a host is documented to look, not that the host reads them.

Step 3: loads

This is where we needed a live host, a fresh session each time, and a way to see inside the trace rather than trust the final answer.

On Codex CLI 0.157.1:

codex exec --skip-git-repo-check -s read-only "List the names and descriptions of every skill you currently have available."
codex exec --skip-git-repo-check -s read-only < positive1.txt
codex exec --skip-git-repo-check -s read-only "Write a haiku about tea."

The discovery listing included explain-error. The positive prompt (a Python traceback) produced cat .agents/skills/explain-error/SKILL.md and cat .agents/skills/explain-error/references/checklist.md in the trace, then a reply that opened with our required marker line, Checked with explain-error, and named the file, line, cause, and one fix. The control prompt (“Write a haiku about tea.”) produced only a haiku: no skill mention, no marker, no file read in the trace. That combination (a real tool-call in the trace, correct on the matching prompt, silent on the unrelated one) is what “checked” means for loads.

On Claude Code 2.1.283, we could not complete this step: claude -p returned Not logged in · Please run /login on every invocation, including a plain claude -p "say hi", even though the test itself was being run from inside an authenticated, interactive Claude Code session. claude auth status confirmed no usable credential was reachable from that shell; this held with the sandbox on and off, and network to api.anthropic.com was unaffected. That’s a credential-isolation problem in our test setup, not a defect in the skill, but it means loads is not tested for Claude Code, and we say so rather than guessing. We did not have Cursor installed, so loads is not tested for Cursor either.

Step 4: helps

We did not run this step for explain-error. Helps means a paired comparison: the same task, the same inputs, a fresh session with the skill available and a fresh session without it, checking a mechanically verifiable property of the output rather than an impression. We only measured whether the skill loaded (step 3), which is a real result but a different claim from “it made the answer better.” /methodology/ covers how we define and run a paired helps comparison when we do one.

What we checked

Format

Checked. skills-ref validate passed.

Installs

Checked. File placement confirmed on Claude Code and Codex documented paths by hand-copy; npx skills add additionally confirmed for .claude/skills/ and .agents/skills/. It doesn’t write .codex/skills/, which we reached only by hand-copy.

Loads

Checked on Codex only (discovery, correct positive trigger with reference file read, no false positive on a control prompt). Not tested on Claude Code (authentication blocker in our setup) or Cursor (not installed).

Helps

Not measured for this skill.

What we didn’t check

Live loading on Claude Code and Cursor, and any paired outcome comparison (“helps”) for explain-error.

Common problems

Stopping at format and calling it “tested” is one mistake: a validator pass is real but narrow, and it says nothing about installs, loads, or helps. Reusing the authoring conversation to “test” loading is another; the official evaluation guidance warns this can hide missing instructions, because the agent still has your corrections in context from writing the skill, so test in a fresh session. A third is judging load success from the final answer alone: a plausible answer can come from the base model’s own knowledge, so check the tool trace for an actual file read.

Sources

• Agent Skills: evaluating skills
• Claude Code: evaluation guidance
• Codex: build skills
• /methodology/