// TOM'S HARDWARE US — HARDWARE & GADGET
Hot Chips 2026: Nvidia presents Groq 3 LPX architecture and unveils its first third-party inference benchmark — LP30-based rack already in production, company says
The inference rack built from Nvidia's $20 billion Groq acquisition claims a four-times long-context decode lead.
When you purchase through links on our site, we may earn an affiliate commission. Here’s how it works.
This Tom's Hardware Premium article is free to read with a Tom's Hardware account; no payment necessary. We're offering free access from August 23 to 26 so you can read all of our reporting from Hot Chips.
Groq's former chief architect stood on stage at Hot Chips 2026 and presented his former company's inference chip as Nvidia silicon. Igor Arsovski, now Nvidia's VP of hardware, presented the Groq 3 LPX rack's architecture and published the first third-party benchmark of the hardware: Artificial Analysis measured it at 3,431 output tokens per second on a 100K-context Gemma 4 31B reasoning workload, roughly four times the 870 tokens per second of the next-fastest public endpoint. Arsovski said the rack is already in production, built on the LP30 chip Nvidia obtained through its $20 billion Groq deal in December 2025, the same deal that pushed the Rubin CPX it replaced off Nvidia's roadmap.
Artificial Analysis ran the comparison on a private, pre-release Gemma 4 31B endpoint served through Google Cloud, taking the median of 50 sequential client requests at a concurrency of one, while the public providers it measured against ran shared production serverless endpoints. Serving one request at a time produces the highest per-user token rate the hardware can post, and it's not directly comparable to the multi-tenant conditions the other endpoints run under.
Nvidia's on-stage demo showed a higher figure still, 10,996 tokens per second on the same 31B model, which Igor Arsovski, Nvidia's VP of hardware, flagged on stage as "self-reported" before telling the audience the aim was "third-party verified independent benchmarks that you guys can trust." Gemma 4 31B is also a dense model small enough to sit inside a single LPX rack, and the picture at trillion-parameter mixture-of-experts scale, where memory capacity becomes the main constraint, went unaddressed.
Each LP30 carries roughly 500MB of on-die SRAM and no HBM, so a full LPX rack of 256 chips holds 128GB of memory delivering 40 PB/s of aggregate bandwidth against 315 PFLOPS of FP8 compute, with 350 ns of chip-to-chip latency in a Vera Rubin-compatible, MGX liquid-cooled rack that scales past 1,000 LPUs.
Keeping model weights resident in SRAM rather than streaming them from HBM removes the memory-access latency that dominates single-token decode, and the design drops caches, branch prediction, and out-of-order execution in favor of a fully deterministic pipeline that the compiler schedules at clock-cycle granularity. The architecture descends directly from the Tensor Streaming Processor that Groq, founded by ex-Google TPU engineer Jonathan Ross, described in a 2020 ISCA paper titled Think Fast, the same title Arsovski and Raghavan reused at Hot Chips.
A Rubin GPU carries 288GB of HBM4, roughly 576 times the memory of a single LP30, so a 31-billion-parameter model at FP8 needs on the order of 62 LPUs to hold its weights, and a large mixture-of-experts model runs into four figures of chips across several racks. Capacity is the cost of the SRAM-only design, and it's why Nvidia is describing the LPU as for decode rather than as a general-purpose replacement for its GPUs.
Determinism lets the compiler predict power draw cycle by cycle, which Nvidia uses to pre-order current from the rack's regulators ahead of demand, cutting voltage droop by more than 60% and overshoot by more than 70% against an uncompensated load. The same per-block scheduling lets the hardware equalize heat instead of throttling to the hottest tile, which Arsovski put at roughly 10% to 11% additional performance under a fixed thermal limit. "By doing this, we can actually get more utilization of the chip under the same thermal limit, basically. So we can actually get, again, about 10 to 11% more p