skill-creator run_eval.py: recall drops to 0% with parallel workers (identical clone commands)

4 answers

• Best answerAgentArmor teamTeam3d ago

  Still open as of the last thread update (2026-07-28), but the root cause is confirmed by two independent environments and a serial workaround exists. This is a design flaw in run_single_query(), not stream-format drift: it writes a clone named <skill>-skill-<uuid8>.md into the shared .claude/commands/ directory and counts a trigger only when the accumulated tool_use JSON contains that exact hashed name. With the default --num-workers 10, N clones with identical descriptions but different hashes coexist in the same directory, the model picks one arbitrarily, and each worker's match probability is ~1/N — recall collapses toward 0. Orphaned clones left by killed runs and the skill's canonical name in CLAUDE.md (the model calls Skill("<real-name>") instead of a clone) make it worse.

  What works today: run serialized. A commenter re-ran a 10-query slice (6 positive / 4 negative) with --num-workers 1, swept stale <skill>-skill-*.md clones beforehand, and temporarily removed the skill's canonical-name reference from user-level CLAUDE.md: recall recovered from 0% to 3/6 positives with negatives 4/4. (The reporter's 0/27 figure came from default-parallel runs across three descriptions.) A second environment (WSL2 Ubuntu, Claude Code 2.1.220) reproduced the A/B directly: 10 workers gave ~0/3 triggers per query vs --num-workers 1 = 3/3.

  The thread suggests fixes the maintainers could adopt: match the clone-name prefix f"{skill_name}-skill-" in the accumulated JSON instead of the exact hash, sweep stale clones at eval start plus a cleanup handler for killed runs, or give each worker an isolated project_root (note user-level ~/.claude/commands/ is still shared when project_root resolves to $HOME). A secondary bug: run_single_query() returns False at the first tool_use block that isn't Skill/Read (run_eval.py L133-141) instead of checking the rest of the turn; patching that changed a few individual results (some 0/3 → 1/3) but did not fix the core collapse.

  Did this work for you?WorkedDidn't work
  UpvoteStarReport
• AgentArmor teamTeam3d ago

  What a GitHub user tried on GitHub (Worked)

  Reporter ran a control experiment with exactly one clone present: claude -p with --output-format stream-json --verbose --include-partial-messages showed the model invoking that clone's full hashed name and trigger detection working correctly, confirming the stream-event parsing is fine and the failure is the N-clone ambiguity.

  Source: https://github.com/anthropics/skills/issues/1427 (opens external site) ↗

  Did this work for you?WorkedDidn't work
  UpvoteStarReport
• AgentArmor teamTeam3d ago

  What @Buer2333 tried on GitHub (Worked)

  Commenter re-ran a 10-query slice (6 positive / 4 negative) with --num-workers 1, swept stale clones beforehand, and removed the skill's canonical-name reference from user-level CLAUDE.md; recall recovered from 0% to 50% (3/6 positives) with negatives 4/4.

  Source: https://github.com/anthropics/skills/issues/1427#issuecomment-4947820453 (opens external site) ↗

  Did this work for you?WorkedDidn't work
  UpvoteStarReport
• AgentArmor teamTeam3d ago

  What @surfvand-gif tried on GitHub (Worked)

  Independent A/B on the same 3 should-trigger queries with the same description: parallel --num-workers 10 gave ~0/3 triggers each, serial --num-workers 1 gave 3/3 (100%), on WSL2 Ubuntu with Claude Code 2.1.220.

  Source: https://github.com/anthropics/skills/issues/1427#issuecomment-5110481446 (opens external site) ↗

  Did this work for you?WorkedDidn't work
  UpvoteStarReport