MiMo-V2.5 Inference Optimization: Making Hybrid SWA Actually Work in Production

Xiaomi's MiMo team published a deep-dive on the inference optimizations behind their MiMo-V2.5 and V2.5-Pro models. The headline: Hybrid Sliding Window Attention (SWA) promises a 7× reduction in KVCache storage and compute, but bridging that gap from paper to production required a ground-up refactor of the caching and scheduling stack.

The V2.5 series uses a 70-layer architecture — 10 Full Attention layers and 60 SWA layers with a window size of 128. In theory, that cuts attention compute and KVCache to about 1/7 of a full-attention model. In practice, as the team notes, naive implementations in frameworks like SGLang v0.5.5 stored full KVCache for SWA layers, nullifying the advantage.

Dual-Pool KVCache Design

The core fix was splitting KVCache into two independent pools: one for Full Attention layers (O(N) storage) and one for SWA layers (O(W) storage). At the physical layer, the SWA pool is sized only for the window and supports independent eviction. At the logical layer, a unified sequence view is exposed to the prefix tree and scheduler, with a Full → SWA mapping maintained for transparent tiered storage. This alone recovers the 7× capacity efficiency.

SWA-Aware Prefix Cache Tree

Traditional RadixAttention assumes equal token sequences yield reusable KV. Under SWA, that breaks — a prefix tree node may still logically represent a token sequence, but its SWA KV may have been evicted past the window. The team revised matching rules to a "window-safe length": match length is clipped so that the tail W tokens must have valid slots in the SWA pool. Eviction is tied to request lifecycle (chunk completion, termination, every N decode tokens), keeping SWA pool usage constant at window size rather than growing with sequence length. Nodes now carry dual indices — Full Attention segment index and SWA segment mapping — allowing independent eviction of out-of-window SWA segments while preserving Full Attention segments for reuse.

Tiered Caching and Distributed Consistency

The three-tier HiCache system (device, host, storage backend) was refactored to maintain valid SWA indices per tier. The team found that Full Attention Cache hits with SWA Cache misses caused severe match-length truncation, forcing recomputation. They optimized distributed consistency across deployments, shared prefix lengths, and async data movement to keep the two in sync — critical for maintaining hit rates under the new matching rules.

MoE and Multimodal Bottlenecks

Beyond SWA, the article touches on MoE scheduling and load balancing, and notes that multimodal encoders remain a throughput bottleneck for large-image and long-video inputs. The team optimized the Prefill/Decode execution pipeline and scheduling strategy to handle these, but the piece is primarily focused on the KVCache work.

The takeaway: Hybrid SWA's theoretical efficiency is real, but only after you rebuild your caching system from the ground up. The MiMo team's approach — dual pools, window-safe prefix trees, tiered consistency — is now being adopted by mainstream inference frameworks, which says something about how necessary this work was.