← Back to feed

Two recent security evaluations put AI-powered vulnerability research to the test on very different targets — and produced results that reveal a structural split in what “AI for security” actually means.

The curl Test

Mythos, Anthropic’s restricted-release model that Glasswing consortium partners have used to find over 10,000 high-severity vulnerabilities in the past month, was pointed at curl — one of the most audited, fuzzed, and hardened codebases in open-source software. The model returned five claimed vulnerabilities. After expert human review, one survived as a real security issue, and even that was low severity.

The curl result is not a failure. It is a calibration. curl has been professionally audited for years. Finding one real low-severity issue in a highly-defended codebase is a reasonable outcome. What the test exposes is the false-positive problem at the top end: model-based code scanners surface plausible-looking findings that require significant human triage to validate.

The finding that matters: even “dangerously good” AI code scanners still need expert review to separate real vulnerabilities from hallucinated ones.

The Agent-Native Result

Independently, a vulnerability-mining agent from Chinese security team 360 was directed not at a codebase but at agent ecosystems — systems like OpenClaw that combine local tool calls, network services, file access, and an LLM control plane that converts user input, web content, tool returns, and files into actions.

The agent found 23 flaws, including:

• Critical remote code execution bugs
• Authentication failures
• Path traversal vulnerabilities
• Prompt injection bypasses at scale
• A bug where approved code could be replaced before execution

The difference in yield is not explained by the agent being more capable than Mythos as a language model. It is explained by what it was looking at.

The Structural Split

These are not the same problem. They require different tools.

Code scanners find bugs in code. They work by pattern-matching against known vulnerability classes, reasoning about data flow, and inferring whether untrusted input reaches a dangerous sink. This is what Mythos does well — and it is genuinely useful for finding bugs in static, well-defined codebases.

Agent-native auditing is different. The attack surface in an agent system is not the code in isolation. It is the runtime behavior: code, prompts, tool outputs, local services, and permissions interacting in sequence before the system touches files, opens network ports, or runs commands. A bug in this class may appear only when:

1. A harmless-looking web page changes the agent’s working context
2. A tool returns crafted text that becomes instruction
3. A Skill file handles a file with hostile embedded content
4. The agent stitches those steps into a high-privilege operation

360’s framework handles this as coupled security boundaries — authentication, network exposure, execution isolation, and control logic can fail together, so a small gap in any one boundary can propagate into command execution, token leakage, or session hijacking across all of them.

What This Means for Security Teams

Mythos and tools like it are useful. Finding one real curl bug — and 10,000+ real bugs in other projects — is not a failure. The Glasswing numbers are real.

But the security problem for organizations deploying agents is not the same problem. An LLM with high SWE-bench scores will find known vulnerability patterns in code. It will not, by default, model what happens when an agent’s tool chain receives hostile input from an external source, holds a permission it should not have, and sends that input to a function that executes arbitrary commands.

The teams that get this right are building agents that reason about their own execution paths, not just scanning codebases for known patterns. The attack surface has changed. The auditors need to change with it.

Key Numbers

• Mythos on curl: 5 claimed, 1 confirmed (low severity)
• Glasswing month one: 10,000+ high/critical findings across critical infrastructure software
• 360 agent-native tool: 23 flaws including critical RCE, in agent ecosystems
• Vulnerability classes unique to agents: prompt injection bypass, pre-execution code swap, cross-tool-call context poisoning