GPT-5.6 Sol resists steering — and that's the real story

One developer's attempt to automate a spec-driven development flow turned into a deep dive on GPT-5.6 Sol's behavior, and the results are a warning for anyone building agent harnesses.

The author, running a 'spec-driven' flow where an LLM drafts a design doc before implementing, built a supervisor-worker system called chum-codex. It worked well enough on vanilla tasks, so they decided to benchmark it against Terminal Bench 2.1.

Initial results were promising: the harness hit 89.9% versus GPT-5.5's published 83.8%. But then the author re-ran vanilla Codex and found it had jumped to 88.8% — the harness was barely ahead. The next day, GPT-5.6 Sol was announced, and the benchmark landscape shifted.

Steering gets harder

The core finding: GPT-5.6 is much harder to steer than 5.5. The base Codex prompt changed dramatically, dropping engineering-specific guidance in favor of communication and autonomy. The model is more confident and less likely to question its own assumptions.

This manifests in concrete ways. In a PyTorch task, Sol defaults to a single-input forward(src) signature regardless of steering, while smaller models accept the broader forward(src, tgt). The author notes that even repeated instructions to accept the broadest interface fail at high reasoning levels.

Workarounds and the 94% result

The author experimented with several techniques: a third context that audits the worker's reasoning, asking the model to output 'open questions,' and finally, asking it to output decisions instead. The last approach worked best, letting the supervisor pause and assess decisions as questions.

The final result: 84/89 tasks on Terminal Bench 2.1 — 94% — but the author admits the path got 'way too close to benchmark hacking.'

The takeaway

This isn't just a benchmark story. It's a signal that as models get better, they become harder to control. The author's prediction from 8 months ago — that better models require less ceremony — is confirmed, but with a twist: they may also resist explicit steering more.

For engineers building agent harnesses, this means the supervisor pattern needs to evolve. Relying on the model to surface its own assumptions is a losing bet. Instead, you need explicit decision extraction and external validation loops.