Migrating a Production AI Agent to GPT-5.6: What Broke and How We Fixed It

Ploy's AI agent builds and edits real marketing websites — planning pages, reading codebases, writing components, generating imagery, and deciding when it's done. For months, Claude Opus held the default slot. Then GPT-5.6 Sol arrived, and the numbers were too good to ignore: builds finishing in less than half the wall-clock time, at 27% lower cost, with comparable or better quality scores. But switching models, even with a universal LLM SDK like Vercel's AI SDK, turned out to be a minefield of provider-specific behaviors.

Step 0: Fix Your Eval Harness First

Ploy's eval suite runs the real agent against hundreds of fixture workspaces, scoring builds on visual checks, content checks, and tool trajectories. Running the same suite across two model families revealed a painful truth: your harness is tuned to your incumbent model, and you don't know it. Tool-call budgets sized for Opus's sequential style were blown by GPT-5.6's parallel calls. The eval executor didn't support batched file reads, which GPT-5.6 uses constantly. Roughly a third of the raw failures in the first cross-model run traced back to harness assumptions, not model behavior. Triage the traces before you trust the pass rate.

Step 1: Tool Call Schemas — The Silent Corruptor

Ploy's code tool has 25 top-level parameters, one required and the rest optional. Claude sends only the two or three it uses. GPT-5.6 sends all 25, every time, inventing plausible values for unused ones: offset: 0, timeout: 120000, siteId: "00000000-0000-0000-0000-000000000000". The problem isn't verbosity — it's that an invented value is indistinguishable from an intended one. offset: 0 caused 52% to 64% of file reads to come back empty. Prompting doesn't fix this; even with per-property "OPTIONAL, omit if unused" hints, GPT-5.6 still sent all 25. The fix: a schema transform at the provider boundary that rewrites every optional property to be required but nullable using anyOf: [T, null], then strips nulls before validation. Empty file reads dropped from 52% to 0%, and the agent needed roughly 30% fewer tool calls.

Step 2: Rebuild Prompt Caching

On the surface, both providers offer "prompt caching," but the implementations are entirely different. Ploy's agent opens with a 29K-token static prefix that cached across the whole organization on Claude with 92-96% hit rates. GPT-5.6 dropped partial-prefix matching; implicit caching now only creates whole-prompt entries keyed on the latest message. A new conversation sharing the same static prefix cached 0% of it. The intended mechanism uses explicit prompt_cache_breakpoint markers plus a mandatory prompt_cache_key, and each key maps to a cache node sustaining roughly 15 requests per minute before traffic fans to cold caches. That turns "enable caching" into a real design decision about key scoping. Before fixing this, GPT-5.6 looked about 50% more expensive than Opus — it wasn't the model's pricing, it was the cache configuration.

Takeaways

Ploy's migration story is a reminder that "the model" is really a bundle of provider-specific behaviors your whole stack has quietly specialized around. Tool argument filling, prompt caching, and reasoning replay all differ between providers. If you're evaluating a challenger model, fix your harness first, then expect to redesign around these differences. The performance gains are real — but they come with engineering cost.