Reference Material

Part of Two Pools.

Pull these as you need them, not up front

Reading everything below before you start a single module is a procrastination strategy. Each entry is tagged with the module where it becomes relevant.

---

The SEO problem in this specific topic area

Most "local LLM tok/s" content on the web is unverifiable, and a fair amount of it is generated by the exact kind of model it claims to benchmark

Local-LLM performance is a high-traffic, low-friction topic for content farms: a plausible-looking number, a screenshot of a terminal, zero methodology. Searching for --n-cpu-moe or RX 9070 XT tok/s surfaces pages with precise-looking figures — “127.4 tok/s!” — that state no quant, no context length, no build hash, no thread count, and no variance. A number with four significant figures and zero reproducibility information is not more credible than a rounder number with a command line attached. It is less.

Before treating any benchmark claim you find online as real, check it against this list:

The benchmark-credibility checklist

1. Is there an exact command line? Not “I ran llama-bench” — the actual flags: model file, quant, -ngl, -ncmoe, -fa, context length, thread count.
2. Is there a variance or repeat-count figure? A single run of a bandwidth- or thermal-sensitive workload is a data point, not a measurement. llama-bench reports std-dev by default — a post that strips it out is a red flag, not a simplification.
3. Is the build identified? llama.cpp changes fast; a commit hash or release tag matters, especially for anything ROCm/Vulkan-related given how much backend behavior shifts release to release (see the flash-attention and n-cpu-moe PRs below).
4. Is the hardware fully specified and plausible? GPU model and VRAM and driver/ROCm version and, for anything DDR-bound, DIMM population and speed. “Ryzen 9950X + 9070 XT” without DIMM count and configured speed is not enough to reproduce a MoE-offload number, because DDR5 bandwidth is exactly what changes with population (see module-0-two-pools).
5. Does the claim generalize the way it’s stated? A number measured at one context length, one quant, and one n_cpu_moe value does not transfer to a different combination of the three — and a post that reports one number as “the” throughput for a model is eliding that the roofline is placement-dependent (see the abstract in index).

If a source fails two or more of these, treat its numbers as directionally suggestive at best, and re-derive your own before relying on them for a configuration decision.

---

Primary vendor docs

Topic	Source	Needed by
ROCm platform support, gfx1201 compatibility	The official ROCm documentation site — search for the current compatibility matrix and the Linux system requirements page. gfx1201/RDNA4 support landed around ROCm 7.0, not at RX 9070 XT launch	module-1-toolchain
Qwen3-Coder-30B-A3B-Instruct model card and config.json	Hugging Face — Qwen/Qwen3-Coder-30B-A3B-Instruct	module-4-vram-budget, module-5-expert-placement
AMD-Cline collaboration on agentic coding evaluation	Cline blog — source of the “models smaller than Qwen3-Coder-30B consistently fail” finding; verify the specific post is still current before citing it as a live claim	module-3-eval-instrument, module-10-serving

llama.cpp PRs and issues

Change	Reference	Needed by
--n-cpu-moe/-ncmoe and --cpu-moe/-cmoe (per-layer MoE CPU offload)	ggml-org/llama.cpp PR #15077 (merged Aug 4 2025)	module-5-expert-placement
--override-tensor/-ot (general regex tensor placement, predates -ncmoe)	ggml-org/llama.cpp PR #11397	module-5-expert-placement
-fa tri-state (on/off/auto, default auto)	ggml-org/llama.cpp PR #15434 (merged Aug 30 2025)	module-7-backend-shootout, module-9-kv-quantization
gfx1201 Vulkan token-gen 4.7–6.7x slower than HIP above hidden_size 4096 (open)	ggml-org/llama.cpp issue #26663 — suspected KHR_coopmat shader path issue; smaller models (hidden_size 2560) unaffected	module-7-backend-shootout
GBNF grammar format (logit masking for structural validity)	grammars/ directory in the ggml-org/llama.cpp repository	module-10-serving

Gotcha — issue #26663 is model-shape dependent, not a blanket "Vulkan is bad"

The slowdown is specific to hidden_size ≥ 4096 on gfx1201. A 4B model at hidden_size 2560 shows no such penalty (~183 tok/s), and Vulkan reportedly beats HIP on some smaller models. Backend choice in module-7-backend-shootout is a per-model-shape decision, not a fixed preference — measure both before picking one.

Papers

Paper	What it establishes	Needed by
Modarressi et al., NoLiMa: Long-Context Evaluation Beyond Literal Matching — arXiv 2502.05167 (ICML 2025)	Defines “effective length” as the longest context retaining ≥85% of a model’s base score; measured Llama 4 Scout’s effective length at 1,000 tokens against a marketed 10,000,000	capstone-2-effective-context
Opsahl-Ong et al. / DSPy + GEPA — arXiv 2507.19457 (ICLR 2026 Oral)	Reflective prompt-evolution as a real, documented approach to prompt optimization against a local model	module-11-aider (architecture-search extensions are at the edge of documented practice — treat with more caution than the prompt-optimization core result)

Benchmarks

Benchmark	What it measures	Needed by
Aider Polyglot	225 coding exercises, Docker-based, works against any OpenAI-compatible endpoint. The practical local benchmark for this course	module-3-eval-instrument, module-8-quantization
SWE-bench Verified	Real-world GitHub issue resolution; needs roughly 120GB disk and 8 cores of Docker orchestration per full run — noted here as the harder, heavier alternative, not the course’s working instrument	module-3-eval-instrument (context only)

---

A note on what’s unverified in this course

Three specific claims in this course are predictions, not measurements, until you run the relevant module or capstone: the RX 9070 XT n_cpu_moe roofline curve, the AMD -fa on/off A/B, and any repo-map-vs-RAG-vs-long-context comparison. See the “Three things nobody has published numbers for” section on index for exactly what’s missing and which module fills each gap.