GEMM optimization

In short: General matrix-multiply (GEMM) dominates AI compute because almost every dense layer and attention block reduces to one, and you make it fast by tiling the operands into cache- and register-sized blocks so each element loaded from DRAM is reused O(N) times instead of once.

Ask any inference-serving team where their FLOPs go and the answer is the same: matrix multiplication. A transformer's projections, feed-forward layers, and attention scores are all GEMMs; a convolution is lowered to one via im2col. If you can make GEMM run close to a processor's peak throughput, you have made the whole model fast. The catch is that a naive triple-loop multiply spends almost all of its time waiting on memory, not computing. This article explains why GEMM is the center of gravity for AI hardware, and the two techniques — tiling and cache blocking — that turn a memory-bound crawl into a compute-bound sprint.

Why matrix-multiply dominates AI compute

A dense layer computing Y = XW for a batch of tokens is literally a GEMM. For a modern LLM, the per-token cost of the feed-forward and projection layers scales with roughly 2 × (parameters) FLOPs; a 7-billion-parameter model therefore burns about 14 GFLOPs of matrix-multiply work for every single token generated. Multiply that by thousands of tokens per request and millions of requests, and you see why GPU vendors quote peak numbers in matrix units: NVIDIA's Tensor Cores and the systolic arrays in Google's TPU exist specifically to accelerate the multiply-accumulate that GEMM is built from.

Arithmetic intensity is the whole game. Multiplying two N×N matrices does 2N³ floating-point operations but touches only 3N² data elements. The ratio — FLOPs per byte — grows with N, which is exactly why GEMM can be made compute-bound while a vector add never can.

The naive algorithm and why it is slow

The textbook version is three nested loops: for each output element C[i][j], sweep across a row of A and a column of B and accumulate the dot product. It is correct and it is a disaster for memory. Consider C = A·B with 1024×1024 single-precision matrices. Each matrix is 4 MB — far larger than a typical 32 KB L1 or even a few-megabyte L2 cache. As the innermost loop strides down a column of B, consecutive accesses are 4 KB apart in memory, so nearly every access is a cache miss. The CPU's arithmetic units sit idle while the memory subsystem fetches cache lines that get evicted before they are reused.

The problem is reuse. In the naive loop, each element of B is re-read from DRAM once for every row of A. At N=1024 that is a thousand wasted trips to main memory for data that could have stayed resident.

Tiling: trade one big miss for many cheap hits

Tiling (also called blocking) restructures the computation so that it works on small sub-blocks that fit in a given cache level. Instead of computing one output element fully before moving on, you load a T×T tile of A and a T×T tile of B, multiply them, and accumulate into a T×T tile of C — then slide the tiles along the shared K dimension. The key property: once a tile is in cache, every element in it is reused T times (once per row or column of the partner tile) before it is evicted.

The right tile size is the one whose three active tiles fit together in the target cache. For 32 KB of L1 in single precision, three tiles of 48×48 floats is about 27 KB — a good fit. Production kernels tile hierarchically: large tiles for L2/L3, medium tiles for L1, and the innermost "micro-kernel" keeps a handful of columns in vector registers and runs fused multiply-add instructions back to back.

Reuse math: with a tile of side T, each byte brought into cache is reused roughly T times. Going from T=1 (naive) to T=64 turns one DRAM read into 64 useful multiply-accumulates, pushing arithmetic intensity past the point where the processor, not the memory bus, is the bottleneck.

Where the loop order matters too: swapping the naive i-j-k order to i-k-j lets the innermost loop stream along contiguous rows of B and C, which hardware prefetchers love. Tiling plus the right loop order is the difference between a few percent of peak and 80–90% of peak on a well-tuned CPU kernel such as those in OpenBLAS or Intel oneMKL.

From cache blocking to systolic silicon

The same principle drives the hardware. A TPU's systolic array is a grid of multiply-accumulate cells; weights are preloaded and activations flow through, so each weight is reused across the whole streamed batch — tiling made physical. NVIDIA's Tensor Cores operate on small matrix fragments (for example 16×16 tiles) and libraries like CUTLASS expose exactly the hierarchical tiling — threadblock, warp, and thread-level tiles — that CPU kernels do in software. When you profile a GEMM-heavy model with Siemens, Synopsys, or Cadence tooling during accelerator design, the figure of merit is always the same: how much data reuse the on-chip SRAM buys you before you have to pay for off-chip HBM bandwidth.

Understanding where a kernel sits relative to the memory-vs-compute boundary is best done with the roofline model for AI, which plots achievable performance against arithmetic intensity. Tiling is literally the lever that slides a kernel rightward across the roofline until it hits the compute ceiling. For the mapping of convolutions onto GEMM and the memory cost of that lowering, see our note on im2col convolution lowering.

Practical takeaways for engineers

• Never hand-roll GEMM in production. Use oneMKL, OpenBLAS, cuBLAS, or CUTLASS — they already tile for every cache level and emit tuned micro-kernels.
• Batch to raise intensity. A single matrix-vector product (GEMV) has intensity near 1 and is memory-bound; batching many tokens into a GEMM is what lets accelerators reach peak.
• Watch your data layout. Row-major vs column-major and padding to vector width decide whether the prefetcher helps or hurts.
• Quantize to shrink the working set. Int8 or FP8 weights halve or quarter the bytes per tile, so more reuse fits in the same cache.

Teams building custom accelerators or optimizing inference stacks often pair this theory with hands-on RTL and flow experience — the kind of silicon-design support offered by Avecas VLSI design services.

Want to go from theory to a tuned kernel you profile yourself? Our AI Hardware course on CourseTron walks through tiling, roofline analysis, and Tensor-Core programming with real lab exercises.

Frequently asked questions

Is tiling the same as blocking?

Yes — "tiling" and "cache blocking" are two names for the same loop transformation: restructuring the computation to operate on sub-blocks that fit in a chosen level of the memory hierarchy so data is reused before eviction.

How do I pick a tile size?

Size the tiles so the three active blocks (A-tile, B-tile, C-tile) fit together in the target cache. Start from the cache capacity, divide by the element size and by three, and take the square root. Then auto-tune around that estimate, because associativity and prefetch behavior shift the optimum.

Why not just add more memory bandwidth instead of tiling?

Bandwidth is expensive and physically limited; HBM already costs far more energy per byte than an on-chip SRAM read. Tiling reduces how many bytes you need to move at all, which improves both speed and energy efficiency — a lever you get in software for free.

Does tiling matter on GPUs, or only CPUs?

It matters even more on GPUs. Libraries like CUTLASS tile at the threadblock, warp, and register levels, staging data through shared memory (a software-managed cache). The reuse principle is identical; only the names of the memory levels change.