In short: A hardware accelerator wins only when the work it removes from the CPU is large, regular and compute-heavy enough to outweigh the time and energy spent moving data to it and the serial code Amdahl's law says you cannot speed up.
What a hardware accelerator actually is
A hardware accelerator is a block of logic built to execute one class of computation faster, or with less energy, than a general-purpose CPU can. It might be a crypto engine inside an SoC, a video codec, a GPU, an FPGA kernel, or a matrix engine inside a neural-network chip. What they share is that they trade generality for efficiency: a CPU spends most of its silicon and energy on fetch, decode, branch prediction, register renaming and caches, while the arithmetic itself is a small slice. An accelerator strips that overhead away and spends the area on parallel datapaths and local memory.
The motivation is physical. In classic 45 nm energy numbers from Mark Horowitz's ISSCC keynote, a 32-bit floating-point add costs around one picojoule, while fetching a 32-bit word from off-chip DRAM costs several hundred. Computation is cheap; moving operands is expensive. Every design decision below follows from that asymmetry. For the large-scale picture of how this plays out in neural-network silicon, see our companion piece on AI accelerator architecture.
Key fact: Arithmetic is cheap, data movement is not. A DRAM access can cost two orders of magnitude more energy than the arithmetic performed on the fetched word.
Fixed-function versus programmable
Accelerators sit on a spectrum. At one end is a fixed-function block such as an AES engine or an H.264 deblocking filter: its datapath is hard-wired, it reaches the best performance per watt and per mm², and it can never run anything else. At the other end is a programmable accelerator such as a GPU, a DSP or a vector engine: it has an instruction stream, so one chip serves many algorithms, but it pays for instruction fetch, scheduling and flexible memory systems. Between them are configurable designs, such as CGRAs and FPGA overlays, and domain-specific processors with narrow ISAs.
| Property | Fixed-function | Programmable |
|---|---|---|
| Efficiency (ops/W) | Highest, often 10x-1000x a CPU | Typically 3x-50x a CPU |
| Flexibility | One algorithm, maybe a few modes | Any code you can compile |
| Risk | Standard changes obsolete it | Software absorbs change |
| Verification | Block-level, golden model compare | Needs compiler, ISA and software stack |
| Typical example | AES, JPEG, CRC engine | GPU, DSP, NPU with a command processor |
The choice is an economics question. If the algorithm is frozen by a standard and shipped in hundreds of millions of devices, fixed-function is almost always right. If the workload is still evolving, as transformer variants are, you pay for programmability. Interviewers like this question because there is no universal answer, only a defensible argument about volume, stability and power budget.
Amdahl sets the ceiling
If a fraction p of runtime can be offloaded and the accelerator runs it s times faster, total speedup is 1 / ((1 − p) + p/s). Suppose 80% of a pipeline is offloadable and the accelerator is 20x faster: speedup is 1 / (0.2 + 0.04) = 4.2x, not 20x. Even an infinitely fast engine on that 80% caps at 5x. At 95% offloadable the ceiling rises to 20x, but the lesson is the same: the serial remainder, which includes parsing, control flow and glue code, dominates. Profile first. A 100x kernel that covers 10% of runtime yields about 1.11x, and rarely justifies a tape-out.
Data-movement cost: the real break-even
Amdahl is optimistic because it ignores the trip to the accelerator. Add transfer time and the model becomes: T_offload = T_in + T_compute + T_out + T_setup. Take a 64 MB tensor that a CPU processes in 5 ms. Over PCIe Gen4 x16, usable bandwidth is roughly 25-28 GB/s, so moving 64 MB takes about 2.4 ms each way. Even if the accelerator computes in 0.2 ms, the total is around 5 ms, which is no gain at all. Raise the work per byte (arithmetic intensity) and the picture flips: a matrix multiply of that size reuses each byte hundreds of times, so compute dominates and the transfer amortises.
Three techniques rescue marginal cases. First, overlap with double-buffering and DMA so transfer of tile N+1 hides behind compute on tile N. Second, keep data resident by chaining several kernels on the accelerator before returning results. Third, shrink the data through quantisation to INT8 or FP8, or compress on the fly. The roofline model captures all this: attainable performance is the lesser of peak compute and bandwidth times arithmetic intensity.
Key fact: Offload wins when T_in + T_compute + T_out + T_setup < T_cpu. If arithmetic intensity is low, the bus, not the engine, sets your speed.
Interface choices: AXI, PCIe and shared memory
How the accelerator attaches determines its latency, bandwidth and software model.
- On-chip AXI. Inside an SoC, accelerators usually connect with AMBA AXI. AXI4 has five independent channels (AW, W, B, AR, R), supports bursts up to 256 beats, and allows multiple outstanding transactions, which hides memory latency. A typical IP exposes AXI4-Lite for control registers, AXI4 full master ports for DMA into memory, and AXI4-Stream for point-to-point data pipes. With a 256-bit bus at 1 GHz, one port peaks near 32 GB/s, with latency measured in tens of cycles.
- PCIe. A discrete card sits across PCIe. Gen4 x16 gives about 32 GB/s raw per direction, Gen5 about 64 GB/s, but a round-trip costs microseconds, so many small transfers are punishing. Drivers pin memory, build descriptors and ring doorbells, adding fixed overhead per launch.
- Shared (coherent) memory. CPU and accelerator see one address space, either through a cache-coherent fabric such as CCIX or CXL, ARM ACE/CHI, or simply a unified DRAM pool as in mobile SoCs. Pointers can be passed rather than buffers copied, which removes the copy cost but exposes you to coherence traffic and cache-line contention.
The offload life-cycle in motion
A well-designed offload follows a repeatable pattern: the CPU writes a descriptor, rings a doorbell, a DMA engine pulls the input into accelerator-local SRAM, the engine computes, results are written back, and an interrupt or polled status bit signals completion. Each step has a cost, and pipelining them is where throughput comes from.
What this means for design and verification
In practice, accelerator teams model first. Architects build a transaction-level model in SystemC or Python, sweep tile size and bandwidth, and only then write RTL. High-level synthesis tools such as Siemens Catapult or Cadence Stratus turn C++ kernels into RTL, and Synopsys VCS, Cadence Xcelium or Siemens Questa run the verification, often with UVM agents for AXI. Protocol checkers matter: AXI deadlocks from mismatched ready/valid handshakes or incorrect burst boundaries (a burst must not cross a 4 KB page) are classic bugs that only appear under back-pressure. Integration is the other trap, so verify the full software path, from driver to interrupt, in emulation before silicon.
If you are planning a custom block and need a partner for RTL, verification or physical design, Avecas design services take accelerators from specification to GDS. For how accelerators behave under realistic memory systems, our follow-up on memory bandwidth and the roofline model goes deeper.
Key fact: A single AXI burst must never cross a 4 KB address boundary, and many accelerator DMA bugs come from violating exactly this rule.
Frequently asked questions
When is a hardware accelerator not worth building?
When the offloadable fraction is small, the arithmetic intensity is low, or the algorithm is still changing. A kernel that is 10% of runtime cannot give more than about 1.11x regardless of how fast the accelerator is.
Is a GPU a fixed-function or programmable accelerator?
Programmable. A GPU runs instruction streams on many parallel cores, though it also contains fixed-function parts such as texture units, video codecs and tensor cores.
Why not always use shared memory instead of PCIe?
Coherent shared memory avoids copies, but it constrains bandwidth to the system DRAM and adds coherence traffic. Discrete cards bring their own high-bandwidth memory such as HBM, which wins for bandwidth-hungry workloads.
What is arithmetic intensity?
It is operations performed per byte moved from memory. High intensity makes a workload compute-bound and good for offload, while low intensity leaves it memory-bound.
Want to design and verify real accelerators yourself? Explore the hands-on CourseTron courses on RTL design, AXI protocols and AI hardware, and for instructor-led VLSI training see ChipXpert.