Phase 5 — Everything that isn’t matmul

4 weeks · Part of Zero to NPU · Prev: phase-4-memory · Next: phase-6-sequencer

What this phase is for

Back in Phase 1 you profiled stories260K on the ARM core and found matmul eating 85–95% of every millisecond. In Phase 3 you built an 8×4 array of PEs and made matmul roughly 5× faster. That sounds like a clean win, and for matmul itself it is — but Amdahl’s Law says the rest of the runtime doesn’t shrink just because matmul did. Everything that used to be a rounding error next to matmul — normalization, attention weighting, activation functions, position encoding — is now roughly 35% of your runtime, and none of it runs on your PE array. This phase builds a small, shared “vector unit”: one modest datapath (about 12 DSP slices, a few BRAMs for lookup tables, under 2,000 LUTs) that handles the four operations a transformer layer needs besides matmul — RMSNorm, Softmax, SiLU/SwiGLU, and RoPE. None of these are individually hard. What makes them worth four weeks is that each one hides a fixed-point trap that will silently corrupt your model’s output if you don’t understand why the trick works, not just that it exists.


Amdahl’s Law, concretely

Amdahl’s Law is really just one piece of arithmetic, and it’s worth doing by hand once so the rest of this phase makes sense instead of feeling like a rule someone told you to follow.

Say your baseline run takes 100 units of time total. Based on your Phase 1 profile, about 90 units are matmul and 10 units are everything else — RMSNorm, softmax, SiLU, RoPE, and the residual adds. That “everything else” bucket is what this phase is about.

Now you speed matmul up 5×, exactly like Phase 3 did. Matmul’s 90 units become 90 / 5 = 18 units. The other 10 units haven’t changed at all — you didn’t touch them — so your new total is 18 + 10 = 28 units. Two things happen here that trip people up:

  1. Your overall speedup is not 5×. It’s 100 / 28 ≈ 3.6×. You made one part of the system five times faster and got less than four times faster overall, because the part you didn’t speed up is now a much bigger fraction of what’s left.
  2. The “other” bucket’s share of runtime nearly quadrupled. It was 10% of 100 units before (10/100). Now it’s 10 units out of 28 total — about 36%. Same absolute cost, much bigger relative cost, purely because you shrank the thing next to it.

This is the whole mechanism behind the phase. If you now build a vector unit that makes that “other” bucket 4× faster, it drops from 10 units to 2.5 units, and your total becomes 18 + 2.5 = 20.5 units — an overall speedup of 100/20.5 ≈ 4.9×, much closer to matmul’s own 5×. You cannot get there by making matmul faster again; matmul is already down to 18 units and squeezing it further has rapidly diminishing returns while the vector-unit work is still cheap and unclaimed. That’s the entire argument for spending four weeks on operations that individually look tiny next to a 32-PE systolic array: once matmul stops dominating, everything else becomes the bottleneck by definition, whether or not it “feels” important.


Where these four operations live in a transformer layer

It helps to see exactly where in a single transformer layer these four operations sit, because they are not random — they are the specific glue between matmuls that Phase 1’s reference model needs at every layer, five times over for stories260K (n_layers=5).

x (residual stream, dim=64)
  │
  ├─► RMSNorm ──► [Q,K,V projections]  (matmul, Phase 3)
  │                       │
  │                RoPE on Q and K
  │                       │
  │              [attention scores]    (matmul, Phase 3)
  │                       │
  │                   Softmax
  │                       │
  │            [weighted sum over V]   (matmul, Phase 3)
  │                       │
  └───────────────────► add (residual)
                          │
                    RMSNorm ──► [FFN gate, FFN up]  (matmul, Phase 3)
                          │             │
                          │         SiLU(gate)
                          │             │
                          │      elementwise multiply  (SwiGLU)
                          │             │
                          │       [FFN down]           (matmul, Phase 3)
                          │             │
                          └───────────► add (residual)

Every box labeled “matmul” runs on the 8×4 PE array you built in Phase 3 — that hardware only knows how to multiply-and-accumulate. Every other box — RMSNorm, RoPE, Softmax, SiLU, and the elementwise multiply and adds — is this phase’s job. Notice the pattern: RMSNorm always appears before a block of matmuls (it conditions the input), RoPE modifies Q and K right after they’re produced and before they’re used in attention, softmax turns raw attention scores into weights right after the score matmul, and SiLU/SwiGLU sits between the two halves of the feed-forward network. With n_heads=8, n_kv_heads=4, and head_size=8, this exact sequence repeats once per attention block per layer, and the FFN sequence repeats once per layer — five times end to end for stories260K. None of these operations are optional extras; skip any one of them and the model’s output stops matching the golden reference almost immediately, because each one is doing real numerical work the matmuls depend on.


5.1 RMSNorm

What it does: RMSNorm rescales a vector so its values sit at a consistent, predictable magnitude before the next matmul touches them. You can think of it as a cousin of LayerNorm, but simpler: it skips the mean-subtraction step and skips the learned bias term, using only a per-channel scale. That simplicity is exactly why it’s cheap enough to build in fixed point.

The math:

  • — the -th element of the input vector (here, , matching dim=64)
  • — the vector length being normalized (64)
  • — a tiny constant added for numerical safety, so you never divide by exactly zero
  • — a learned per-channel scale (a weight from the trained model, one per dimension)

Why it’s here: Without it, the scale of activations can drift layer to layer — five layers deep, small biases compound. A matmul fed activations that are sometimes tiny and sometimes huge will alternately lose precision to rounding and overflow its accumulator. RMSNorm keeps every layer’s input in a known, bounded range so the rest of your fixed-point pipeline can be sized correctly instead of defensively oversized.

How to build it in fixed point: Your input is 64 int8 values (dim=64). Square each one: an int8 value maxes out at ±127, and , which fits easily in an int16 but you should accumulate the running sum in int32. Sum all 64 squares: worst case is — about 1.03 million, which fits comfortably inside a 32-bit accumulator (max ~2.1 billion) with enormous headroom. This is the overflow analysis for this op, and it’s the reassuring one in this phase: there is no realistic way for this particular sum to overflow int32.

Divide that sum by to get the mean of squares — since 64 is a power of two, this is a free right-shift by 6 bits, not a real division. Add (represented as a small fixed-point constant), and now you need of the result. Don’t compute a real square root in hardware if you can avoid it — build a small reciprocal-square-root lookup table indexed by the (mean-square + eps) value, get an approximate answer , then sharpen it with one Newton-Raphson iteration, the same trick the famous “fast inverse square root” hack uses in software:

Here is the value you’re taking the reciprocal square root of (mean-square + eps), is your LUT’s initial guess, and is the refined result — one multiply-heavy iteration turns a coarse LUT into something accurate enough to be bit-exact against the golden model after requantization.

// RMSNorm reference, fixed-point style (int32 accumulate, LUT + 1 NR step)
int32_t sumsq = 0;
for (int i = 0; i < 64; i++) {
    int32_t xi = (int32_t)x[i];          // int8 -> int32, sign-extended
    sumsq += xi * xi;                    // max ~1.03M, safe in int32
}
int32_t mean_sq = sumsq >> 6;            // divide by 64 (free shift)
int32_t arg = mean_sq + EPS_FIXED;       // add epsilon
 
int32_t y0 = rsqrt_lut[quantize_index(arg)];   // 256 or 512-entry LUT, coarse
// One Newton-Raphson refinement: y' = y*(3 - arg*y^2)/2
int32_t y1 = fixed_mul(y0, (3*ONE - fixed_mul(arg, fixed_mul(y0, y0)))) / 2;
 
for (int i = 0; i < 64; i++) {
    int32_t scaled = fixed_mul((int32_t)x[i], y1);   // x_i * rsqrt(mean_sq+eps)
    out[i] = requantize(fixed_mul(scaled, g[i]));    // apply learned g_i, requant (Phase 2)
}
// RMSNorm datapath sketch: accumulate squares, then hand off to shared rsqrt LUT+NR unit (5.5)
module rmsnorm_accum #(parameter N = 64) (
    input  wire              clk, rst_n, start,
    input  wire signed [7:0] x_in,       // one element per cycle
    input  wire               x_valid,
    output reg  signed [31:0] sumsq,     // int32 accumulator
    output reg                done
);
    reg [6:0] count;
    always @(posedge clk or negedge rst_n) begin
        if (!rst_n) begin sumsq <= 0; count <= 0; done <= 0; end
        else if (start) begin sumsq <= 0; count <= 0; done <= 0; end
        else if (x_valid && count < N) begin
            sumsq <= sumsq + ($signed(x_in) * $signed(x_in)); // widens to 32b
            count <= count + 1;
            done  <= (count == N-1);
        end
    end
endmodule

Done when

Your RMSNorm output is bit-exact against the golden model for every test vector, and specifically: the int32 sumsq you compute matches the golden model’s sum-of-squares exactly (not just the final normalized output) — that catches accumulator bugs before they hide behind the LUT/NR stage.


5.2 Softmax

What it does: Softmax takes a row of raw attention scores — arbitrary signed numbers — and turns them into a proper probability distribution: values between 0 and 1 that sum to 1, with larger scores getting disproportionately more weight. It’s the step that decides how much attention each token pays to every other token.

The math:

  • — the -th raw attention score in the row
  • — the largest score in that row
  • — the exponentiated, shifted score (always after shifting)
  • the denominator — the sum of all shifted-and-exponentiated scores in the row, used to normalize

Why it’s here: Attention scores by themselves aren’t weights — they’re arbitrary magnitudes. Softmax is what converts “bigger score” into “proportionally more attention,” and it’s what makes the weighted-sum-over-V matmul that follows actually meaningful instead of using raw, unbounded numbers as weights.

How to build it in fixed point: The subtract-the-max step is not optional polish — it is load-bearing. grows explosively: , already brushing up against int32’s ceiling of about , and your raw attention scores can easily exceed that in magnitude before shifting. Without subtracting the row max first, a single large score anywhere in the row overflows your fixed-point exponential and every downstream weight in that row becomes garbage — silently, with no crash, just wrong numbers propagating into the next matmul. Subtracting the max guarantees every exponent argument is , so every is , and overflow simply cannot happen in that step.

Once every value is and bounded below by your fixed-point range, build a 256-entry exp lookup table: index it by the (quantized, non-positive) shifted score, get back a fixed-point approximation of . Sum those 256-or-fewer results in an int32 accumulator (same style of safety margin as RMSNorm), then get the reciprocal of that sum — either with its own small LUT or by reusing the same LUT+Newton-Raphson machinery from 5.1, since a reciprocal is a simpler cousin of reciprocal-square-root. Multiply each exponentiated value by that reciprocal and requantize.

If you want to avoid two full passes over the row (one to find the max, one to exponentiate-and-sum), look at online softmax (Milakov & Gimelshein, “Online normalizer calculation for softmax”): it updates a running max and a running sum together in a single pass, rescaling the running sum whenever a new max is found. That’s a real engineering option here, not just an academic curiosity — it’s exactly the kind of one-pass streaming trick that matters when your row is arriving from BRAM one element per cycle instead of sitting fully resident in registers.

// Two-pass fixed-point softmax with max-subtraction
int8_t row_max = x[0];
for (int i = 1; i < len; i++) if (x[i] > row_max) row_max = x[i];
 
int32_t sum = 0;
int32_t exp_vals[MAX_LEN];
for (int i = 0; i < len; i++) {
    int shifted = (int)x[i] - (int)row_max;      // always <= 0, no overflow risk
    exp_vals[i] = exp_lut[quantize_index(shifted)]; // 256-entry LUT
    sum += exp_vals[i];                          // int32 accumulate
}
int32_t recip = reciprocal_lut_with_nr(sum);     // reuse 5.1's LUT+NR machinery
for (int i = 0; i < len; i++) {
    out[i] = requantize(fixed_mul(exp_vals[i], recip));
}
// Softmax exp-LUT stage: one shifted score in, one LUT-approximated exp value out
module softmax_exp_lut (
    input  wire signed [7:0] shifted_score,  // x_i - row_max, always <= 0
    output wire       [15:0] exp_val         // fixed-point e^(shifted_score)
);
    // 256-entry ROM, initialized at synth time from the golden model's table
    (* rom_style = "block" *) reg [15:0] rom [0:255];
    initial $readmemh("exp_lut.mem", rom);
    assign exp_val = rom[shifted_score[7:0]];  // index by the negative offset
endmodule

Done when

Your softmax output is bit-exact against the golden model row by row, and specifically: feed a row containing your model’s largest realistic raw attention score, confirm your fixed-point sum never overflows int32, and confirm the output row still sums to your fixed-point representation of 1.0 within your defined rounding tolerance.


5.3 SiLU / SwiGLU

What it does: SiLU is a smooth, non-linear activation function — the thing that gives the feed-forward network its actual expressive power. Without a non-linearity between matmuls, stacking multiple matmuls back to back is mathematically equivalent to one single matmul, which would make five layers pointless. SwiGLU, as used in this model, is SiLU applied to a “gate” projection and then multiplied elementwise by a separate “up” projection — a gating mechanism that lets the network learn to selectively pass information through.

The math:

  • — the input value (here, the “gate” projection’s output, per element)
  • — the logistic sigmoid function
  • — the separate “up” projection, same shape as gate
  • — elementwise multiplication

Why it’s here: This is the non-linearity every transformer FFN needs. It’s also the gate in SwiGLU: multiplying by up lets the network scale how much of each channel passes through, which is strictly more expressive than a plain non-gated FFN.

How to build it in fixed point: This is the easy win of the whole phase, and it’s worth saying plainly: because your input is int8, there are only 256 possible input values, full stop. That means you don’t need an approximation of SiLU at all — you can precompute the exact output for every one of those 256 inputs (using the real floating-point SiLU formula, then requantizing to int8 the same way the golden model does) and store the whole function as a 256-entry int8-to-int8 lookup table. That’s one small BRAM, essentially free in terms of DSPs, and it is not an approximation — it is the exact answer for every input your hardware will ever actually see, because the input domain is small enough to enumerate completely. Once you have SiLU(gate) from the LUT, the SwiGLU elementwise multiply against up is a single int8×int8 multiply (one DSP) followed by the usual requantization from Phase 2.

// Building the exact SiLU LUT offline (host-side, once, from the same
// quantization scheme the golden model uses)
void build_silu_lut(int8_t lut[256], float scale, int32_t zero_point) {
    for (int q = -128; q <= 127; q++) {
        float x = dequantize(q, scale, zero_point);       // int8 -> float
        float y = x / (1.0f + expf(-x));                  // real SiLU
        lut[q & 0xFF] = quantize(y, scale, zero_point);    // float -> int8, exact for this input
    }
}
 
// Runtime: SwiGLU = SiLU(gate) * up, elementwise
for (int i = 0; i < hidden_dim; i++) {                     // hidden_dim = 172
    int8_t silu_gate = silu_lut[(uint8_t)gate[i]];
    out[i] = requantize(fixed_mul((int32_t)silu_gate, (int32_t)up[i]));
}
// SiLU LUT + elementwise multiply, one element per cycle
module silu_swiglu (
    input  wire signed [7:0] gate_in,
    input  wire signed [7:0] up_in,
    output wire       [15:0] product_out   // feeds requantize stage (Phase 2)
);
    (* rom_style = "block" *) reg signed [7:0] silu_lut [0:255];
    initial $readmemh("silu_lut.mem", silu_lut);   // exact table, all 256 inputs
 
    wire signed [7:0] silu_val = silu_lut[gate_in[7:0]];
    assign product_out = silu_val * up_in;          // one DSP
endmodule

Done when

Every one of the 256 possible int8 inputs produces the exact int8 output the golden model’s LUT produces (this should be a trivial, total match — if it isn’t, the LUT was built with the wrong quantization parameters), and the elementwise SwiGLU product matches the golden model bit-exactly across a full hidden_dim=172 vector.


5.4 RoPE

What it does: RoPE (Rotary Position Embedding) is how the model knows word order. Plainly: it takes each pair of dimensions inside a query (Q) or key (K) vector and rotates that pair by an angle that depends on the token’s position in the sequence. Tokens at different positions get different rotation angles, so the same word appearing early versus late in a sentence produces measurably different Q/K vectors — that difference is what lets attention scores reflect position, without needing separate positional-embedding vectors added anywhere.

The math: for one pair of dimensions at position , with per-pair angle :

  • — one adjacent pair of dimensions within a head
  • — the token’s position in the sequence (0, 1, 2, …)
  • — which pair within the head this is (pairs are indexed )
  • — dimensions per attention head (8, for this model)
  • — the rotation angle for this pair at this position; lower-indexed pairs rotate faster with position, higher-indexed pairs rotate more slowly

Why it’s here: matmul and the other three ops in this phase are all position-agnostic — they’d produce the identical result if you shuffled the token order. Language obviously isn’t position-agnostic. RoPE is the one piece of the whole pipeline that injects “where in the sequence am I” into Q and K before they’re compared in attention.

How to build it in fixed point: The formula has a and for every (position, pair) combination, and computing trig functions in hardware per-token would be wasteful and slow. The fix is exactly what the phase brief says: precompute every sin/cos value you’ll ever need into a BRAM table once, at initialization — not per token, not per inference call. With head_size=8, each head has pairs, and with a 128-token context window (128-tok context, per your board notes), the table needs fixed-point entries. At, say, 2 bytes per entry, that’s 2,048 bytes — about 2 KB, trivially small next to your ~4 BRAM-block budget, and it comfortably fits in a single block. Compute it once when the model loads, index it by (position, pair_index) at runtime, and apply the rotation formula above as two multiplies and an add/subtract per pair — cheap, and it never touches a trig function at inference time.

// Precompute the RoPE sin/cos table once, at init (NOT per token)
// 128 positions x 4 pairs x {sin, cos}, fixed-point Q-format values
void build_rope_table(int16_t sin_tab[128][4], int16_t cos_tab[128][4]) {
    const int head_size = 8;
    for (int pos = 0; pos < 128; pos++) {
        for (int i = 0; i < head_size / 2; i++) {           // 4 pairs
            float theta = pos / powf(10000.0f, (2.0f * i) / head_size);
            sin_tab[pos][i] = float_to_fixed(sinf(theta));
            cos_tab[pos][i] = float_to_fixed(cosf(theta));
        }
    }
}
 
// Apply RoPE to one (x0, x1) pair at runtime — pure table lookup + rotate
void rope_apply_pair(int8_t *x0, int8_t *x1, int pos, int pair_idx,
                      const int16_t sin_tab[128][4], const int16_t cos_tab[128][4]) {
    int16_t s = sin_tab[pos][pair_idx];
    int16_t c = cos_tab[pos][pair_idx];
    int32_t new_x0 = fixed_mul(*x0, c) - fixed_mul(*x1, s);
    int32_t new_x1 = fixed_mul(*x0, s) + fixed_mul(*x1, c);
    *x0 = requantize(new_x0);
    *x1 = requantize(new_x1);
}
// RoPE table lookup + rotate, one pair per cycle
module rope_rotate (
    input  wire [6:0]        pos,        // 0..127
    input  wire [1:0]        pair_idx,   // 0..3 (head_size/2 - 1)
    input  wire signed [7:0] x0_in, x1_in,
    output wire signed [7:0] x0_out, x1_out
);
    // 1024-entry sin/cos tables, precomputed at init, loaded via $readmemh
    (* rom_style = "block" *) reg signed [15:0] sin_rom [0:511]; // 128*4
    (* rom_style = "block" *) reg signed [15:0] cos_rom [0:511];
    initial begin
        $readmemh("rope_sin.mem", sin_rom);
        $readmemh("rope_cos.mem", cos_rom);
    end
 
    wire [8:0] idx = {pos, pair_idx};          // 128*4 = 512 entries
    wire signed [15:0] s = sin_rom[idx];
    wire signed [15:0] c = cos_rom[idx];
 
    wire signed [23:0] new_x0 = (x0_in * c) - (x1_in * s);
    wire signed [23:0] new_x1 = (x0_in * s) + (x1_in * c);
 
    assign x0_out = requantize24_to_8(new_x0);   // Phase 2 requant stage
    assign x1_out = requantize24_to_8(new_x1);
endmodule

Done when

Rotated Q and K vectors are bit-exact against the golden model at every tested position, and specifically: position 0 must produce an identity rotation (, so , , meaning and exactly) — that’s a free, cheap sanity check before you trust any other position.


5.5 Wiring it together

You don’t need four separate pieces of hardware for four operations — that would burn far more than your ~12-DSP, ~4-BRAM budget. Instead, build one small shared datapath: a modest ALU (a handful of adders and multipliers, reused across ops) plus a single LUT-holding BRAM block, with a mode select signal that decides what that cycle’s ALU/LUT combination actually computes. RMSNorm and softmax’s reciprocal step reuse the same LUT+Newton-Raphson hardware (a coarse LUT plus one refinement multiply-add). Softmax’s exp LUT, SiLU’s LUT, and RoPE’s sin/cos tables are three different contents loaded into the same kind of BRAM lookup structure — same address-in, data-out shape, different tables. The accumulator (int32, used by both RMSNorm’s sum-of-squares and softmax’s sum-of-exponentials) is one more shared resource, cleared and reused between ops. This is why the phase fits its tiny budget: you’re not building four units, you’re building one flexible one and pointing it at different tables and control sequences depending on which of the four operations the command sequencer (Phase 6) asks for next.

// Shared vector-unit top level: one ALU + one LUT BRAM, mode-selected
module vector_unit (
    input  wire [2:0]  mode,     // 0=RMSNorm, 1=Softmax, 2=SiLU, 3=RoPE
    input  wire signed [7:0] a_in, b_in,
    input  wire [8:0] lut_addr,
    output reg  signed [15:0] result
);
    wire [15:0] lut_data;
    lut_bram u_lut (.addr(lut_addr), .mode(mode), .data_out(lut_data));
 
    always @(*) begin
        case (mode)
            3'd0: result = rmsnorm_step(a_in, b_in, lut_data);  // sumsq / rsqrt+NR
            3'd1: result = softmax_step(a_in, b_in, lut_data);  // exp LUT / recip+NR
            3'd2: result = a_in * lut_data[7:0];                // SiLU LUT * up
            3'd3: result = rope_step(a_in, b_in, lut_data);     // sin/cos rotate
            default: result = 16'sd0;
        endcase
    end
endmodule

Check your understanding


What’s next

You now have a working vector unit that handles every non-matmul operation a transformer layer needs, wired to share one ALU and one LUT BRAM across four modes. phase-6-sequencer is where you build the control logic that actually sequences matmul and vector-unit operations correctly, layer after layer, so the whole pipeline runs itself instead of needing you to single-step it.