Skill-creator run_eval: crashed runs count as "did not trigger" and pass negative queries

Question

In anthropics/skills skill-creator's `scripts/run_eval.py`, a query whose eval run crashes (subprocess error, timeout, API failure) is recorded as `False` — the same value as a legitimate "skill did not trigger". For `should_trigger: false` queries the pass test is `trigger_rate < threshold`, so an infrastructure failure is scored as a pass and can even produce 20/20 passed while 28 of 60 runs failed with `You've hit your session limit` on `main` at commit `dbd4588`. Is this still open, and is there a workaround?

AgentArmorHQ team · Accepted Answer

Status: still open as of this thread — no fix or PR is mentioned. The cause is in `skills/skill-creator/scripts/run_eval.py`: the exception handler appends `False` to `query_triggers` when a run crashes, so a subprocess crash, timeout, or API error becomes indistinguishable from a legitimate "skill did not trigger". For `should_trigger: false` queries the pass test is `trigger_rate < threshold`, so a crashed run scores as a correct result. `run_loop.py` imports `run_eval` directly, so the optimizer sees the same corrupted counts.

Workaround reported from a real-world hit on `main` at commit `dbd4588`: treat a `result` event with `is_error` (or no `result` event at all) as a failure rather than appending `False`. That surfaces the problem as `Warning: query failed` and keeps the failed run out of the trigger value space. The failure occurred with `run_eval.py --runs-per-query 3 --num-workers 10` on 20 queries: 28 of 60 `claude -p` runs failed with `"is_error": true` and `You've hit your session limit`, yet the script reported 20/20 passed with no warning printed.

One commenter argues the fix belongs at the append site — add an `errored` field to the emitted payload — and recommends excluding the failed iteration plus a minimum valid-run count rather than retrying, since retrying biases the eval toward flaky-passing descriptions. Note the asymmetry: the same crash scores as a pass for `should_trigger: false` but as a miss for `should_trigger: true`, so a flaky environment makes negative-query results look better and positive-query results look worse.

AgentArmorHQ team · Answer

**What a GitHub user tried on GitHub** (Worked)

Reproduced the bug with a self-contained script that transcribes run_eval's aggregation and injects a crash as False. The healthy non-trigger payload and the crashed payload are byte-identical, and the crashed case scores 2/2 passed.

Source: https://github.com/anthropics/skills/issues/1478

AgentArmorHQ team · Answer

**What @ScriptAutomate tried on GitHub** (Partly worked)

Ran run_eval.py --runs-per-query 3 --num-workers 10 on 20 queries on main at commit dbd4588. 28 of 60 claude -p runs failed with "is_error": true and "You've hit your session limit", yet the script reported 20/20 passed with no warning. As a workaround, treated a result event with is_error (or no result event) as a failure instead of False, which surfaces Warning: query failed; the thread does not confirm the final aggregate score after the change.

Source: https://github.com/anthropics/skills/issues/1478#issuecomment-6327140747

AgentArmorHQ team · Answer

**What @zzzzz9999 tried on GitHub** (Outcome unclear)

Suggested excluding the failed iteration and enforcing a minimum valid-run count instead of retrying, plus adding an errored field at the append site. Design proposal only; not implemented in the thread.

Source: https://github.com/anthropics/skills/issues/1478#issuecomment-6334930295

Skill-creator run_eval: crashed runs count as "did not trigger" and pass negative queries

4 answers

Add your answer or comment

Skill-creator run_eval: crashed runs count as "did not trigger" and pass negative queries

4 answers

Add your answer or comment

Related questions