Why Your Local LLM Feels Dumber Than It Is: Attention Backends Quietly Change Outputs

Every engineer who has pulled a quantized GGUF into Ollama and felt the model underperform has wondered: is the model bad, or is my setup? A new technical post by thr3e on the Level1Techs forum digs into the latter, showing that even with identical weights and hardware, the software stack can change what tokens a model outputs.

The core finding: swapping the attention backend in vLLM—between FlashAttention 2, Flash Inference, and Triton Attention—causes the model to disagree on the greedy next token in later portions of a long prompt. The test used the official BF16 checkpoint of Qwen3.6-27B on an RTX PRO 6000 Blackwell GPU, with a 100k-token context captured from a real agentic workstream. Only the attention backend changed between runs; everything else stayed fixed.

For the first few thousand tokens, all backends agreed. But as the prompt progressed, the backends began to diverge. The post reports that in later 8k-token windows, a measurable percentage of sampled positions had top-1 flips—meaning the backend would have chosen a different token than the baseline. The logits were bit-for-bit identical across repeated runs with the same backend, isolating the divergence to the matrix math in prefill.

The post also warns about KL divergence claims on quantized model cards. A low KLD number is meaningless unless the author discloses the reference checkpoints, runtime environment, evaluation text, context lengths, and how the measurements were aggregated. Methodology matters as much as the number.

For local LLM users, the takeaway is practical: benchmark your actual workload, not just three zero-shot prompts. Use the sampler settings specified on the model card—setting temperature too low can cause looping. And understand that your inference engine's implementation details are part of the equation.