Capstone — a fast embeddings encoder

2 weeks · Part of Kernel Ladder · Prev: module-8-shipping · Next: interview-prep

What this capstone is for

Embedding models are a clean, commercially important target: no autoregressive loop, lots of variable-length batches, and a quality metric (cosine similarity to a reference) that makes numerics errors visible. Production embedding runtimes compete on exactly this workload; for one published account of the engineering involved, see How Baseten built BEI.


The build

Take an LLM-based embedding model (for example Qwen3-Embedding-0.6B) and build its inference path from your own kernels:

The benchmark

On MI300X, compare against Hugging Face Transformers on ROCm and a serving engine’s ROCm embedding path:

MeasureBatch sizesNotes
Throughput (sequences/s)1, 32, 512Mixed sequence lengths
p50 / p99 latency1, 32Client-side and server-side
Cosine similarity vs bf16 referenceAllReport the minimum, not the mean

Bonus: run the same build, more slowly, on the 9070 XT, and explain the gap with a roofline.

Done when

You have published portfolio post #3 with all three measures, the exact commands to reproduce them, and a profile showing where the remaining time goes.

Open-source contributions

Aim for two or three merged pull requests; small ones count.

  • vLLM or SGLang ROCm backends, AITER or Composable Kernel: a tuned configuration for a new shape, a kernel fix or a missing dtype.
  • llama.cpp’s HIP backend on consumer Radeon cards: RDNA performance work is under-served, and the 5700 XT and 9070 XT are exactly the hardware many users have.
  • A GPU MODE leaderboard entry with a public write-up.