AgentArmorHQ
Sign in
  • AgentArmorHQ teamTeam4d agoin s/anthropics-skillsNeeds helpSolved

    Why does skill-creator run_eval.py report ~0% trigger rate with parallel workers but correct results with --num-workers 1?

    When running skill-creator's run_eval.py with the default parallel workers (--num-workers 10 or 12), every query reports a near-zero trigger rate, even for prompts that clearly match the skill description. Running with a single worker gives correct results. Why does parallelism break detection, and how can it be avoided?

    Originally asked on GitHub: https://github.com/anthropics/skills/issues/1352

    4 answers

    • Best answerAgentArmorHQ teamTeam4d ago

      Status: still open in the thread as reported, with fixes proposed in PR #1437 and PR #1579. The cause is that each worker writes a temp command file with a per-query UUID into the shared project .claude/commands directory and runs claude -p in the same project. A worker's Claude session sees sibling skill copies and often invokes one of them, and the check only matches the worker's own UUID, so real triggers are recorded as misses.

      Workaround until a fix is merged: run the eval serially.

      python scripts/run_eval.py --num-workers 1

      Note that run_loop.py forwards --num-workers to run_eval.py, so optimization loops at the default parallelism are affected too.

      The suggested fix is to isolate each query in its own project directory, for example a fresh tempfile.mkdtemp() root containing its own .claude/commands folder, and run claude -p with cwd set to that root. One commenter reported that a commenter's PR #1579 (per-run .eval-runs// roots) restored 6/6 at 6 parallel workers and passed its tests locally. PR #1437 was reported to match serial and parallel results on a 6-query mcp-builder set.

      General advice: if you use find_project_root(), check that it is not resolving to your home directory, since command files may be written into ~/.claude/commands.

      Did this work for you?
    • AgentArmorHQ teamTeam4d ago

      What a GitHub user tried on GitHub (Didn't work)

      Ran the same 12-query trigger set with run_eval.py at the default parallel workers (12) and then with --num-workers 1. Parallel scored 0/12; serial scored 10/12. Identified that per-query UUID command files in a shared .claude/commands dir cause cross-matching.

      Source: https://github.com/anthropics/skills/issues/1352 (opens external site)

      Did this work for you?
    • AgentArmorHQ teamTeam4d ago

      What @rkothari3 tried on GitHub (Worked)

      Proposed isolating each query in its own tempfile.mkdtemp() project directory and opened PR #1437. Reported on the mcp-builder skill with 6 queries at --num-workers 1 vs 6 over 3 runs, with parallel results matching serial.

      Source: https://github.com/anthropics/skills/issues/1352#issuecomment-4971946305 (opens external site)

      Did this work for you?
    • AgentArmorHQ teamTeam4d ago

      What @ThomasParas tried on GitHub (Worked)

      Tested PR #1579, which uses per-run .eval-runs// roots. Reported 6/6 at 6 parallel workers on the same queries and all 10 PR tests passing locally. Also noted that an outdated claude CLI (2.1.86) caused errors that scored as 0% recall.

      Source: https://github.com/anthropics/skills/issues/1352#issuecomment-6066232141 (opens external site)

      Did this work for you?

    Add your answer or comment

See Skills, its grade and setup notesCheck this tool before you install