// HACKER NEWS — CYBERSECURITY
What happens when a GPU reads memory
Our previous post followed a
vector-add kernel — c[i] = a[i] + b[i], one thread per float — from nvcc
down to the warps. We went into a lot of detail on how the kernel was launched,
but we also left a lot out.
This time, we’re going to address our omissions, and follow the path the
critical SASS instruction (a global load) takes through the hardware — in this
case, since it’s under my desk, an RTX
4090We do this kind of reverse engineering for performance reasons, at least
in principle (for a great rationale, see 'Why these details matter' in the
Citadel microbenchmarking paper). For the
same work applied to more production-relevant GPUs, watch this space..
Little of the detail of this path is documented by NVIDIA, at least not to
the level that we’d like, so we’ll determine it by running timing
experiments on the hardware itself.
The CUDA kernel we are investigating has two lines in its function body:
If you inspect the compiled SASS, you’ll see the instructions that power
those lines:
They serve to load the elements of the vector bThe instructions are the same for a, we're following b. from global memory into a
register, where they can be added to the elements of a to perform the kernel.
One LDG.E asks for four bytes in each of 32 lanes. Serving it takes four
32-byte sectors, one cache line, one address translation, a crossbar crossing,
one of thirty-six L2 slices, and, when it misses everywhere, an activate and
four column reads at a DRAM chip. It’s this journey of the instruction through
the hardware, and back, that we’ll try to follow.
To set the scene: our warp lives on one of the SM’s four sub-partitions, alongside eleven
other resident warps. Each cycle the sub-partition’s scheduler picks one warp
that is eligible, and issues its next instruction across the 32 lanes at once.
Our warp wins twice: once for the IMAD.WIDE, and a few cycles later (the
addresses now sitting in R4 and R5) for the LDG.
Let’s start with the instruction. LDG.E R4, [R4.64] is a global load of 32
bits from the 64-bit address stored in registers R4 and R5R5 appears because of the .64 annotation: registers are 32 bits in
size., storing
the result in register R4. To load the data itself, we first must go get that
address from those registers.
One row of the register file holds R4 for all 32 lanes at onceThe reads are staged in an operand collector first. The staging is
there for instructions whose sources share a bank of the register file, since a
bank serves one read per cycle. There are two banks, picked by the low bit of
the register number, so an adjacent pair always spans both.. Another
holds R5. The warp reads both entries, yielding 256 bytes read as 32 distinct
64-bit addresses, one address per lane.
The address read adds at most one cycle. A shared-memory load taking its
address from a register takes 24 cycles from issue to first use, and the same
load with the address as an immediate takes 23. (LDG can’t take an immediate).
With all of its addresses resolved, the instruction issues to the
load/store unit (LSU). The LSU takes the instruction and its operand
addresses, does some address arithmetic (if necessary)This unit can add immediate offsets ([R4.64] carries no offset to add),
and scope loads (LDG names the global window directly)., and sends on the
opcode (‘load these addresses’, in binary), a 32-bit mask of active lanes,
its computed addresses, and the number of the register the result belongs in.
The next destination is the coalescer.