← Back to feed

YC-backed startup Armature ran 16,893 coding agent sessions across Claude Code, Codex, and Cursor to answer one question: when an agent picks which tool or service to use in a real codebase, what actually determines that choice?

The study was published September 3, 2026. Armature sells growth services to dev tool vendors and was motivated partly by commercial interest in mapping agent selection behavior. The methodology is disclosed and the raw traces are public, so the data can be evaluated directly.

Methodology

75 repositories, 10 programming languages. Each repo used fake company names and fake git histories with real lockfiles validated against live package manager registries — so agents ran in realistic conditions without external brand signals.

Four developer personas drove each session: Vibe-coder (describes symptoms, not categories), Junior engineer (names categories, not constraints), Senior engineer (specifies constraints and requirements), and Enterprise engineer (adds compliance and procurement context).

1,163 prompt variations, with 20–25% explicitly mentioning cost or usage volume to test whether budget signals change picks.

Key Findings

Web search behavior diverges sharply. Claude Code almost never uses web search during tool selection. Codex nearly always does. Cursor sits in the middle. This aligns with Claude Code’s tendency to reason from the existing codebase outward, versus Codex anchoring more heavily on external sources.

Agents disagree more than they agree. Across comparable prompts and repositories, Claude Code and Codex reach the same tool recommendation less than half the time. This is not a noise result — the study ran enough variation to control for prompt sensitivity.

Some tools are always mentioned, almost never chosen. LangChain, Supabase, Netlify, PayPal, and Adyen appeared in agent reasoning in nearly every relevant session across all three agents. Selection rates for these tools were far below their mention rates. Being well-known enough to be in training data doesn’t translate into being chosen.

Repository context dominates brand preference. When Armature modified a repo to contain existing signals — partial implementations, lockfile entries, configuration references — agents shifted their picks accordingly. An agent with a clean slate might pick Neon. The same agent in a repo with Supabase config files picks Supabase.

The Business Implication

Armature’s commercial framing matters for how to read this: they’re telling dev tool vendors that the path to agent adoption isn’t marketing — it’s getting into codebases. The study was designed to validate that thesis, and the results support it.

That said, the raw findings hold independently of the motivation. Vercel reported in April 2026 that over 30% of its deployments were being initiated by coding agents, up from near-zero 6 months prior. As agent-initiated choices constitute a larger share of tool adoption, understanding how those choices get made matters for the vendors on the receiving end — and for developers trusting agents to make those calls.

The divergence in web search behavior is the finding that cuts both ways. Claude Code’s preference for in-context reasoning produces picks more tightly coupled to the existing codebase. Codex’s search behavior introduces more external signal, which could mean better discovery of newer tools or drift toward well-documented options regardless of fit.

Neither is obviously right. They’re just different bets on where the reliable signal lives.