Claude Code, Codex, Cursor: 16,893 Sessions Reveal How Coding Agents Pick Tools

Armature, a startup building evaluation infrastructure for coding agents, published a dataset and analysis of 16,893 agent sessions. The goal: figure out which third-party tools Claude Code, Codex, and Cursor actually choose when left to their own devices. The short version: they disagree a lot, repository context matters more than vendor brand, and being mentioned in a conversation is not the same as being picked.

How the experiment was run

Armature built 75 synthetic repositories across 10 languages, designed to mirror real-world startup stacks. They generated fake companies, fake git histories, fake API keys, and real lockfiles. Then they created 1,163 task variations, each phrased from one of four personas: vibe-coder, junior engineer, senior engineer, and enterprise engineer. A simulated human (played by Gemini 3.7 Flash) was put in the loop to make the interactions more realistic—agents could ask questions and get approval before picking a third-party solution.

Each session ran in an ephemeral sandbox (E2B, Blaxel, or Daytona), and a second Gemini instance judged whether the session was valid and which tool won. Out of 16,893 runs, 5,292 sessions on 51 codebases were deemed valid and published, along with full traces.

Key findings

Agents disagree on sources and picks

• Cursor bases decisions on web search in ~2/3 of sessions.
• Codex uses web search 94% of the time, but 9 out of 10 queries use site: operators to focus on trusted domains.
• Claude Code relies on priors, searching the web only ~30% of the time—but when it does, it browses 3x more pages than Codex.
• All three agents pick the same tool in only 42% of cases. Example: for voice agents, Claude Code picks Twilio, Codex picks OpenAI Realtime API, Cursor picks Vapi.
• Claude Code builds in-house almost twice as often as Codex and Cursor (19% vs 10%).

Repository context is decisive

With the same ask on four repositories in different languages, the email provider winner changed: Resend wins on TypeScript, SendGrid on Python, Postmark on Go, and Azure ACS on Java. Vercel wins on TypeScript repos (100% when Next.js is used) but is never recommended on Python repos, where Render dominates.

Being mentioned isn't winning

PayPal was cited 139 times and never picked (Stripe won 124 of those sessions). LangChain was the most cited framework with 194 mentions but picked only 4 times. Netlify: 152 mentions, 6 picks. Supabase: 242 mentions, still dominated by Neon.

Vendor page details can flip decisions

Mailgun lost to Postmark when agents read “1-day retention” on its free plan. Supabase lost because its bundle pricing presented too many BaaS features when agents only wanted a database. In 388 sessions, platform management overhead was mentioned; in 195, costs—often more about presentation than actual disqualifiers.

Some markets are dominated, others are contested

Stripe wins 9 out of 10 payment sessions. Neon wins 66% of database sessions. S3 dominates file storage with 45%. Resend and Postmark lead email with 35.6% and 27.4% respectively.

What this means for vendors and developers

For vendors: your documentation and pricing page are now part of the agent's decision process. For developers: don't assume the agent's default choice is optimal—it's influenced by the repo context and the agent's priors. Armature has made all traces public, so you can dig into the data yourself.