// HACKER NEWS — CYBERSECURITY
Hot Chips 2026: Applying High Bandwidth Flash (HBF)
HBF, or High Bandwidth Flash, uses the same flash memory technology we see in SSDs today. Unlike SSDs, HBF is implemented much like HBM (High Bandwidth Memory). HBF cubes sit on the same package as a compute chip, perhaps even next to HBM. HBF’s idea is to offer much higher capacity than HBM, while still providing decent memory bandwidth. At Hot Chips 2026 tutorials day, Anurag Agarwal and Radhakrishna Giduthuri’s talk explores how HBF could apply to machine learning workloads. No HBF products exist yet, so the talk focuses on simulations, projections, and how software can adapt to take advantage of HBF.
Even though HBF uses a HBM-like form factor, it’s completely different under the hood. It’s not like Intel’s Optane, which could function as another pool of memory. Instead, HBF is almost like a SSD integrated onto a processor. Software uses DMA to move data between HBF and DRAM. HBF accesses must be done in large, aligned chunks as if it were a mass storage device, rather than system memory. Host software also has to take on SSD controller functions like managing write leveling and ensuring data retention. That means HBF can’t be a plug-and-play solution.
Instead, taking advantage of HBF means formulating a special strategy and implementing it into a runtime. Giduthuri takes vLLM as an example. vLLM typically holds model weights in GPU memory, and is already exploring options to reduce VRAM usage. For example, vLLM is investigating putting model weights in pinned CPU memory provided the host has lots of free memory. While that wouldn’t work for HBF because HBF doesn’t support fine-grained random access, other options might have promise. For example, MoE experts can be stored in HBF. Software can DMA active experts into HBM as needed.
vLLM’s KV cache can also be placed into HBF. However, that may only work well in a sparse attention implementation thatonly reads a subset of tokens off the top of the KV cache for each step. That allows most of the KV cache to sit “cold” in flash, which takes advantage of HBF’s capacity while placing less pressure on HBF’s lower bandwidth. A potential caveat is that the top-k read is scattered, while HBF prefers sequential reads. Perhaps software can get around this by DMA-ing the top-k rows into DRAM as needed.
Another opportunity is using HBF capacity to reduce cross-device communication. Large models are often sharded across multiple GPUs, which results in performance being bound by cross-device scatter and gather operations. Cross-device communication can become a bigger performance barrier than compute throughput or memory bandwidth. HBF can mitigate this by replicating more of a model’s weights across different GPUs. DMA-ing data off flash isn’t cheap, but it’s cheaper than going off-device.
Agarwal went over when HBF makes sense from a cost perspective. Basically, HBF is good if a workload doesn’t reach its bandwidth limits. That applies with smaller models and/or smaller batch sizes. If a workload becomes bandwidth bound, it’s bad for HBF’s cost equation because both cost per capacity and cost per bandwidth factor into final cost. HBF is great for cost per capacity, but is worse in cost per bandwidth compared to HBM.
He also discussed using HBM to cache hot experts, but that also seems like a difficult solution. Caching needs to work out well, or HBF bandwidth can throw a wrench into the works for the cost-per-token equation.
HBF may alleviate the DRAM capacity problem to some extent, but the software challenges feel immense. Handling HBF sounds a lot like working with a low level disk access API, like using FILE_FLAG_NO_BUFFERING in Windows or O_DIRECT in Linux. Software has to carry out accesses in large, aligned chunks rather than freely addressing storage with byte-level granularity. Modifying a single byte can mean reading a large 64 KB block into DRAM, modifying that block, and writing the whole block back to flash. That’s more like working with a block storage device than w