Module 3 — Profiling on ROCm

1 week · Part of Kernel Ladder · Prev: module-2-memory-bound · Next: module-4-gemm

What this module is for

A kernel engineer’s real job is explaining why something is slow, with evidence. This module covers the four tools you need and ends with a public write-up: where the time goes in one decode step of a small language model.

ToolQuestion it answersWorks on
rocprofv3 (rocprofiler-sdk)Which kernels ran, for how long, with which hardware countersRadeon and Instinct
rocprof-compute, formerly Omniperf (repo)Why one kernel is slow: roofline, memory chart, cache hit rates, stallsMainly Instinct; check Radeon support for your ROCm release
rocprof-systems, formerly Omnitrace (repo)The timeline: CPU–GPU gaps, launch overhead, stream overlap, collectivesRadeon and Instinct
PyTorch ProfilerWhich operators dominate a model stepAny card with ROCm PyTorch

Concepts: the roofline; launch overhead and HIP Graphs; how counters map onto the memory hierarchy; reading ISA-level hotspots.


3.1 Where the time goes

Goal

A per-operator breakdown of prefill and decode for a ~0.6B–1B parameter model on the 9070 XT.

Generate 256 tokens with ROCm PyTorch and list the top 10 operators by GPU time, split into prefill and decode.

3.2 Launch gaps

Run the same generation under rocprof-systems. Measure the idle time between kernels during decode. Capture decode in a graph (torch.cuda.graphs works on ROCm builds) and measure tokens/s before and after.

3.3 One kernel on the roofline

Use your first hour of Instinct time to run rocprof-compute on your softmax. Place it on the roofline, name the top two limiters, change one thing, and show the profile move.

3.4 Your playbook

Write one page in the repo: which tool answers which question, and the ten counters you check first.

Done when

You have published a write-up titled “Where the time goes in a 1B-parameter decode step” with an annotated timeline and a roofline plot. This is portfolio post #1.

Check your understanding

  • Decode spends 40% of its wall time with the GPU idle. Which tool shows that, and what are the two most likely causes?
  • Why can a kernel at 30% of peak bandwidth still be “done”?