← Back to feed

A developer at ScaleX built a browser game that simulates the human-in-the-loop experience of approving AI coding agent commands under time pressure. Players see a stream of permission prompts — some routine (git status, npm test), some malicious (cat ~/.aws/credentials, curl to exfiltration endpoints) — and approve or deny them for score. Over 40,000 runs and 409,000 individual decisions later, the data is out. Humans missed one in three threats.

The game is simple enough to be taken as toy evidence. The numbers are not.

The Breakdown

Miss rates varied significantly by threat category:

Category	Examples	Miss rate
Scope violations	cat ~/.aws/credentials, cat ~/.kube/config	35.0%
Exfiltration / code execution	curl to unknown APIs, typosquatted packages	33.4%
Persistent mutation	crontab injection, git config hijack	23.8%
Obviously destructive	rm -rf /, chmod -R 777 /	11.7%

The pattern is clear: humans catch what looks obviously destructive. They miss what looks routine. The commands that actually exfiltrate credentials are missed three times as often as commands that would wipe the filesystem. Any real attacker knows which side of that split to work on.

The npm run Problem

The single most-missed command was npm run analyze, approved 64.7% of the time. The game shows the actual payload in the history log above the prompt. Two-thirds of players approved it without reading the log.

npm run executes whatever script is defined in the project’s package.json. It can be a standard bundle analyzer or malicious code injected by an earlier file edit. Approving it correctly requires inspecting all modified files first — a verification step that would slow any developer down significantly. Pooled across three npm run variants in the game, the miss rate was 52.5%, against 28.4% for other exfiltration-style attacks. Hiding a payload behind a familiar script name roughly doubles its success rate even when the payload is shown in the log above.

What the Aggregate Says

Across all 40,000+ sessions:

• 32.9% of sessions ended with a net negative score — approved threats outweighed everything done right
• 35.2% of players caught every threat, but only 20.8% did so without blocking more than one in five safe commands
• 7% approved every single prompt (the --dangerously-skip-permissions cohort)

The 7% mass-approval pattern is the most concerning population. These are the users who have adopted blanket approval as their workflow — the behaviour Anthropic’s Auto-Mode was designed to eliminate. Anthropic’s own telemetry from Claude Code shows the broader population isn’t far off: 93% of permission prompts get approved.

The Structural Problem

The game’s creator noted the core dilemma in the blog post: if a human-in-the-loop wants to verify potentially malicious commands, they need to stop and inspect every file the agent has touched. For an agent running hundreds of tool calls, that’s not oversight — it’s a full-time job. The alternative is the approval fatigue pattern the data shows: a gradual drift toward approving everything.

Anthropic’s approach of automated pre-screening (Auto-Mode) tries to resolve this by taking routine safe approvals out of the human loop entirely. But the study shows that what gets flagged for human review is precisely the ambiguous middle category — commands that look plausible but may not be.

The game had a higher threat density than real-world agent sessions (34% of prompts were threats). But the ScaleX developer made the counterpoint on Hacker News: real-world pressure to ship quickly creates fatigue analogous to the time pressure in the game. The threat mix is lower, but the decision-making conditions are not.

Why It Matters Now

The timing is not accidental. Claude Code has over 8 million users. AWS AgentCore, GitHub Copilot agent mode, and Cursor’s parallel subagent architecture are all in production. These systems generate permission prompts at scale. The governance model for all of them rests on the assumption that a human in the loop provides a safety boundary.

The data from 40,000 game runs — not a controlled lab study, but real decisions made by developers — suggests that boundary is substantially weaker than the industry has assumed.

Key Numbers

• 40,000+ runs, 409,000 approve/deny decisions
• Average threat miss rate: 33.7% (accuracy: 66.3%)
• Scope violations (credential access): 35.0% miss rate
• Exfiltration/code execution: 33.4% miss rate
• Persistent mutation: 23.8% miss rate
• Obviously destructive: 11.7% miss rate
• Most-missed command: npm run analyze at 64.7% approval
• npm run variants combined: 52.5% miss rate
• 7% of users: mass-approved every prompt
• Claude Code production telemetry (Anthropic): 93% of prompts approved