Cloudflare's recipe for serving Kimi and GLM: quantize the KV cache, compress weights, verify the cache

Running large mixture-of-experts models like Moonshot's Kimi K-series and Z.ai's GLM on shared GPU infrastructure is a memory problem before it's a speed problem. Cloudflare's Workers AI team has been attacking that problem with three techniques layered on top of their existing disaggregated prefill/decode setup: quantizing the KV cache, compressing model weights, and adding a safety check for the shared cache those optimizations enable.

FP8 KV cache doubles context capacity

The KV cache—the structure that stores attention keys and values for every processed token—is usually the first thing to fill GPU memory, not the weights. Cloudflare stores it in 8-bit floating point (FP8, e4m3) instead of the default 16-bit (BF16), halving its size. For Kimi K2.6, that raises the context that fits in memory from roughly 686,000 tokens to about 1.37 million.

Quantizing the cache isn't a raw-speed win—the FP8 attention kernel does a bit of extra conversion work per token, so BF16 is a few percent faster at any single concurrency level. The win is capacity: BF16 runs out of cache at 32 concurrent requests, while FP8 keeps going to 64 and hits 2,192 tokens per second, about 41% higher than BF16's peak, at roughly 30% lower cost per token. Because prefill is compute-bound rather than memory-bound, Cloudflare leaves the cache in BF16 there and applies FP8 only to the decode pool.

Accuracy holds up. Across GSM8K, ARC, MMLU, MMLU-Pro, and internal benchmarks, FP8 and BF16 caches are statistically indistinguishable.

INT4 weights shrink GLM 5.2 by 40%

For GLM 5.2, Cloudflare compresses weights from 8-bit floating point to 4-bit integers (INT4). The checkpoint drops from 705 GB to 421 GB, and per-GPU memory across an 8-way tensor-parallel deployment falls from roughly 88 GB to 52 GB—leaving room for about 1.18 million tokens of KV cache on the same hardware.

Smaller weights make decode faster because generating each token means streaming weights out of memory, and decode speed is bandwidth-bound. At low concurrency the effect is dramatic: single-request throughput jumps from 60 to 92 tokens per second, a 55% gain. At higher concurrency the gain settles to 16–27%. Prefill, being compute-bound, is actually slower with INT4 (8,660 vs 10,160 tok/s in FP8), so the disaggregated design lets Cloudflare run INT4 for decode and FP8 for prefill—each where it wins. Accuracy stays within 0.8 points of FP8 across all benchmarks.

Protecting the shared cache

Both optimizations pack more requests onto the same GPU, which means hundreds of requests are reading and writing pages of the same physical KV cache. Paged attention, continuous batching, and cache reuse all depend on perfect bookkeeping. At Cloudflare's request volumes, even a one-in-a-billion mistake would surface regularly.

Their defense: every physical cache page gets a tag that changes on reallocation, and the server records which pages and tags each request expects. Before supported decode operations read from the cache, the mappings are checked; any mismatch aborts the request rather than returning data from the wrong page. The cost is under 1% on both throughput and p95 latency, measured on a production model with 8,192-token inputs and 1,000-token outputs. The check runs as a separate batch validation rather than fused into the attention kernel, avoiding a race between GPU thread groups. It's enabled per deployment, and the default no-op tracker has zero overhead.

What's next

Cloudflare is expanding FP8 KV caches across more of the fleet, validating NVFP4 weights on Blackwell, and working toward leaving integrity checks on everywhere at negligible cost. The work is upstreamed to SGLang, the open-source serving framework they use and contribute to.