Hot Chips 2026: High Bandwidth Flash Is Not the Memory Savior You Think

At Hot Chips 2026, a tutorial session laid out the promise and the pain of High Bandwidth Flash (HBF). The pitch is simple: machine learning workloads are starving for DRAM capacity, and flash is far cheaper per gigabyte. So why not put flash on the package, HBM-style, and give it a wide, fast interface?

HBF is not HBM. It's not even a memory. It's a block device that happens to sit next to your compute die. Software must move data between HBF and DRAM using DMA, in large aligned chunks, and the host has to take on SSD controller duties like wear leveling and data retention. That means no plug-and-play. You can't just map it into your address space and start reading bytes.

The talk walked through how a runtime like vLLM might adapt. One idea: store Mixture-of-Experts (MoE) experts in HBF and DMA the active ones into HBM on demand. Another: park the KV cache in HBF, but only if you're using sparse attention that reads a small subset of tokens per step. The catch is that those reads are scattered, while HBF prefers sequential access—so you'd DMA the top-k rows into DRAM first.

There's also a networking angle. Large models sharded across GPUs spend a lot of time on cross-device scatter/gather. HBF could let you replicate more weights locally, trading flash bandwidth for off-package traffic. DMA off flash isn't free, but it's cheaper than going off-device.

Cost-wise, HBF only wins when you're not bandwidth-bound. Small models, small batches—fine. The moment you saturate HBF's bandwidth, the cost-per-bandwidth disadvantage versus HBM kills the deal. Caching hot experts in HBM helps, but only if the cache hit rate is high enough.

The software burden is the real story. Working with HBF feels like using O_DIRECT or FILE_FLAG_NO_BUFFERING. A single byte write means reading a 64 KB block, modifying it, and writing it back. That's block storage discipline, not memory semantics. Any framework built for DRAM will need a major rewrite, and switching frameworks means redoing that work.

Honestly, the effort to leverage HBF isn't far off from just streaming weights off an SSD—except the OS kernel can hide SSD block alignment behind buffering and caching. HBF offers no such insulation. Until products exist, it's an open question whether the capacity gains justify the engineering cost.