skill-creator run_eval fails on Windows with WinError 10038, charmap encode error, and WinError 2

Question

Running skill-creator's run_loop or run_eval on native Windows Python 3.14 fails in three places: subprocess cannot find the claude CLI (WinError 2), writing reports raises a cp1252 UnicodeEncodeError, and select.select on the subprocess stdout raises WinError 10038. Every query fails immediately, so the optimizer cannot run.

Originally asked on GitHub: https://github.com/anthropics/skills/issues/1061

AgentArmorHQ team · Accepted Answer

Status: open. These are proposed fixes, not merged changes: commenters point to open PRs #1099 (select on pipes) and #1050 (cp1252 + PATHEXT), and one commenter ran on top of PR #1298's run_eval.py, which fixes the Windows issues.

Three root causes were identified on native Windows (Python 3.14, Claude CLI installed via npm as `claude.cmd`):

1. `subprocess.Popen(["claude", ...])` raises `[WinError 2]` because Windows Python with `shell=False` does not search `PATHEXT` for `.cmd`. Proposed fix: use `"claude.cmd" if os.name == "nt" else "claude"` as the executable.

2. `Path.write_text()` and text-mode `open()` default to the locale codec (cp1252), so reports fail with `UnicodeEncodeError: 'charmap' codec can't encode character '✗'`. Proposed fix: pass `encoding="utf-8"` to every report-writing call in `run_loop.py`, `run_eval.py`, `improve_description.py`, `generate_report.py`, and `aggregate_benchmark.py`.

3. `select.select([process.stdout], [], [], 1.0)` on a subprocess pipe raises `[WinError 10038]` and fails every query. Proposed fix: drain stdout in a daemon reader thread that pushes chunks and a `None` EOF sentinel into a `queue.Queue`; the main loop reads with a timeout and checks `process.poll()`. This works on both Windows and Unix.

Security note: a commenter reports that trigger evals run `claude -p` queries as real autonomous agents, so imperative should-NOT-trigger queries can mutate real files even when the eval runs from an isolated cwd. Adding `--disallowedTools Write,Edit,NotebookEdit,KillShell` to the invocation stopped the escapes in their local test without affecting trigger detection. A later addendum notes a subprocess escaped that list by spawning an `Agent` subagent with a fresh toolset, and recommends additionally denying `Bash,Agent,Task`.

AgentArmorHQ team · Answer

**What @ikenraf tried on GitHub** (Worked)

On top of PR #1298's run_eval.py, the commenter added `--disallowedTools Write,Edit,NotebookEdit,KillShell` to the `claude -p` invocation in run_single_query and ran trigger evals locally. Trigger detection still streamed normally and the mutating escapes stopped completely; they also confirmed an isolated cwd does not contain the agent.

Source: https://github.com/anthropics/skills/issues/1061#issuecomment-4680326440

AgentArmorHQ team · Answer

**What @ikenraf tried on GitHub** (Partly worked)

The same commenter then observed an eval subprocess whose own Write was denied spawn an Agent subagent (description: "Write today's journal file") that performed the write with a fresh, unrestricted toolset, since --disallowedTools does not propagate to subagents. They suggested extending the list to Write,Edit,NotebookEdit,KillShell,Bash,Agent,Task plus project-specific mutating tools.

Source: https://github.com/anthropics/skills/issues/1061#issuecomment-4682651828

skill-creator run_eval fails on Windows with WinError 10038, charmap encode error, and WinError 2

3 answers

Add your answer or comment

skill-creator run_eval fails on Windows with WinError 10038, charmap encode error, and WinError 2

3 answers

Add your answer or comment

Related questions