Why does skill-creator run_eval trigger rate drop to about 1/num_workers with parallel workers?

Question

With anthropics/skills skill-creator at commit 3b3fad96, run_eval.py at the default --num-workers 10 reports about 6% recall, while --num-workers 1 gives 80% on the same skill and eval set (Claude Code 2.1.251, macOS 15, Python 3.13, claude-opus-5). Why does the measured trigger rate fall toward 1/num_workers, and is there a fix that keeps parallelism?

Originally asked on GitHub: https://github.com/anthropics/skills/issues/1706

AgentArmorHQ team · Accepted Answer

**Status: open.** The issue is still open and no maintainer fix is recorded in the thread.

The reporter found that run_eval.py writes every worker's temporary command file into one shared `.claude/commands/` directory. Each `claude -p` session sees the other workers' identical skills, and detection only accepts the run's own skill name, so measured recall tracks 1/N. The reporter measured 6% recall at 10 workers, 20% then 0% at 3 workers, and 80% at 1 worker, with precision near 100%. A second issue is that `run_single_query()` returns False on the first tool_use block if it is not `Skill` or `Read`, so a run that starts with `Bash` and then invokes the skill is scored as a miss.

What commenters reported:

1. One commenter gave each worker its own project directory (`projects/worker-<pid>/.claude/commands/`, with leftover `.md` files cleared) and passed it as `project_root`. Their A/B comparison went from 7/36 vs 5/36 with the shared directory to 33/36 vs 35/36 with isolated directories, which reversed the verdict.
2. Another commenter combined per-query isolated roots with drained-tail scoring, where a non-matching tool call resets state and a positive match returns immediately. At 8 workers on a 20-query set, positives scored 3/3 and negatives 0/3. They did not report the two changes separately.

Caveats: `--num-workers 1` avoids the race but is much slower. On Windows, symlink seeding can fail silently without Developer Mode or elevation. If the skill under test is also installed in the user config, the installed copy can be invoked instead of the probe.

AgentArmorHQ team · Answer

**What a GitHub user tried on GitHub** (Partly worked)

Ran the same 20-query eval with --num-workers 10, 3, and 1 on the reporter's skill. Recall was 6%, 20% then 0%, and 80%, with precision near 100%.

Source: https://github.com/anthropics/skills/issues/1706

AgentArmorHQ team · Answer

**What @stillpimpn tried on GitHub** (Didn't work)

Replayed the detector in isolation on a run with tool sequence ['Bash', 'Skill', 'Bash', 'Read']. The skill was invoked as the second tool, but the harness returned False because it stops at the first non-Skill/Read tool_use block.

Source: https://github.com/anthropics/skills/issues/1706#issuecomment-5503618420

AgentArmorHQ team · Answer

**What @Dheirav tried on GitHub** (Worked)

Compared old and new descriptions over 432 runs with 4 workers on a shared commands directory, then again with one project directory per worker. Shared directory gave 7/36 vs 5/36; isolated gave 33/36 vs 35/36.

Source: https://github.com/anthropics/skills/issues/1706#issuecomment-5865954831

AgentArmorHQ team · Answer

**What @C0DECHEF tried on GitHub** (Worked)

Implemented per-query isolated project roots together with drained-tail scoring. At 8 workers on 20 queries, all positives scored 3/3 and all negatives 0/3. The two changes were not reported separately.

Source: https://github.com/anthropics/skills/issues/1706#issuecomment-6184146300

Why does skill-creator run_eval trigger rate drop to about 1/num_workers with parallel workers?

5 answers

Add your answer or comment

Why does skill-creator run_eval trigger rate drop to about 1/num_workers with parallel workers?

5 answers

Add your answer or comment

Related questions