Concepts
The glossary for Kernel Ladder. Each entry ends with a Used in line pointing at the module where it does real work.
Execution and memory
Wave
A group of threads that execute in lockstep: 32 or 64 on RDNA, always 64 on CDNA.
Cross-lane operations, reductions and ballots all depend on the wave size, which is why warpSize must never be hard-coded.
Used in: module-1-hip-model, module-2-memory-bound
LDS
Local Data Share: fast on-chip memory shared by the threads of one workgroup.
LDS is where tiles are staged for reuse. It is split into banks, and when several lanes hit the same bank in one access they serialize — a bank conflict.
Used in: module-1-hip-model, module-4-gemm
VGPR
Vector general-purpose register: per-lane storage, the most precious resource in a kernel.
The more VGPRs each thread uses, the fewer waves fit on a SIMD at once.
Used in: module-1-hip-model, module-4-gemm
AGPR
Accumulation register on CDNA, used to hold MFMA results.
On CDNA3 the register budget is shared between VGPRs and AGPRs, so a large MFMA accumulator tile competes with everything else for occupancy.
Used in: module-4-gemm
Occupancy
How many waves are resident on a SIMD compared with the maximum.
Higher occupancy hides memory latency, but a kernel that uses more registers to reuse data can be faster at lower occupancy. Occupancy is a means, not the goal.
Used in: module-1-hip-model
Coalescing
Adjacent lanes reading adjacent addresses, so the hardware can merge their accesses into a few wide transactions.
Used in: module-1-hip-model, module-2-memory-bound
Roofline
A plot of attainable FLOP/s against arithmetic intensity (FLOPs per byte moved).
Below the ridge point a kernel is limited by memory bandwidth; above it, by compute. Placing a kernel on the roofline tells you which optimizations can possibly help.
Used in: index, module-3-profiling
Matrix cores
WMMA
Wave Matrix Multiply-Accumulate: the matrix instructions on RDNA3 and RDNA4.
One instruction multiplies small matrix tiles held across a wave’s registers. RDNA4 adds FP8 inputs. The fragment layout — which lane holds which element — is fixed per instruction.
Used in: module-4-gemm, module-5-attention
MFMA
Matrix Fused Multiply-Add: the matrix instructions on CDNA (Instinct).
Much higher throughput than WMMA, wave64 only, with results in AGPRs. CDNA4 adds scaled FP8/FP6/FP4 variants.
Used in: module-4-gemm, module-6-quantization-moe
GEMM
General matrix multiply, C = αAB + βC: most of the arithmetic in a transformer.
Used in: module-4-gemm
Grouped GEMM
Many independent GEMMs of different sizes in one launch, as in a mixture-of-experts layer where each expert gets a different number of tokens.
Used in: module-6-quantization-moe
Kernels and algorithms
Kernel fusion
Combining several operations into one kernel so intermediate results never round-trip through DRAM.
For memory-bound operators this is often the largest available speedup.
Used in: module-2-memory-bound, module-7-libraries
Online softmax
Computing softmax in one pass by keeping a running maximum and rescaling the running sum when the maximum changes.
It is the trick that lets FlashAttention process attention in tiles without storing the full score matrix.
Used in: module-2-memory-bound, module-5-attention
Split-KV
Splitting a long KV cache across several workgroups during decode and reducing their partial results at the end.
With one query token there is too little parallelism otherwise.
Used in: module-5-attention
Paged KV cache
Storing the KV cache in fixed-size pages addressed through a table, so memory can be allocated per request without fragmentation.
Used in: module-5-attention
Compute-communication overlap
Splitting work into chunks so one chunk’s collective (for example an all-reduce) runs while the next chunk computes.
Used in: module-8-shipping
Numbers
FP8 FNUZ vs OCP
Two incompatible families of 8-bit float encodings.
MI300X (CDNA3) uses the FNUZ variants (finite, no negative zero), which have a different exponent bias and NaN handling from the OCP FP8 formats used by RDNA4 and MI355X. The same byte decodes to a different value, so kernels and conversion code must know which one they are handling.
Used in: module-2-memory-bound, module-6-quantization-moe
MXFP4
A micro-scaled 4-bit float format: blocks of 32 values share one scale factor.
Block scaling keeps 4-bit values usable by letting each small block choose its own range.
Used in: module-6-quantization-moe