PCIe TLPs and layers

In short: PCIe moves data as Transaction Layer Packets through three layers — transaction, data link, and physical — where the transaction layer frames read/write requests (posted vs non-posted), the data link layer guarantees delivery with sequence numbers and a 32-bit LCRC, and credit-based flow control keeps a sender from ever overrunning the receiver's buffers.

If you have ever stared at a PCIe analyzer trace and wondered why a single 64-byte DMA write turns into a burst of tiny ACK and flow-control packets, the answer is the layered protocol stack. PCIe is a packet-switched, point-to-point serial interconnect, and every meaningful exchange — a CPU reading a status register, a GPU streaming texture data, an NVMe drive DMA-ing a block — is ultimately a Transaction Layer Packet (TLP) that is wrapped, protected, serialized, and acknowledged by the layers beneath it. Understanding those layers is the difference between guessing at a bandwidth problem and actually fixing it.

The three layers at a glance

PCIe borrows the OSI idea of separation of concerns but keeps it lean. A TLP is born in the transaction layer, handed down, and physically launched onto the lanes. On the receive side it climbs back up, getting checked and stripped at each step.

The transaction layer: where TLPs are born

The transaction layer constructs the TLP header (3 or 4 double-words, i.e. 12 or 16 bytes) plus an optional data payload. The header encodes the type of transaction, the address or ID of the target, the requester ID, a tag, byte-enables, and traffic class. There are four fundamental transaction categories in PCIe: memory, I/O, configuration, and messages. Memory reads and writes dominate real traffic.

A crucial design choice lives here: posted versus non-posted transactions.

• Posted — a Memory Write (MWr) or a Message. The requester fires it and moves on; no completion is expected. This is why a 4 KB DMA write can stream at near line rate — there is no round-trip stall per packet.
• Non-posted — a Memory Read (MRd), I/O Read/Write, or Configuration Read/Write. The requester must receive a Completion TLP (CplD, for a read with data) carrying the requested payload or just a status (Cpl).

Why this matters for performance: a Memory Read is round-trip-latency bound. If one device issues reads one tag at a time and the link latency is 500 ns, you cap out around a few hundred MB/s no matter how wide the link is. The fix is outstanding tags — issuing many non-posted reads concurrently (up to 256 or more tags with 10-bit/14-bit tag extensions) so completions pipeline back.

The transaction layer also implements credit-based flow control, which we return to below, and it is where the receiver's buffering is modeled. When you debug a stall in a Synopsys VC Verification IP or a Cadence PCIe VIP simulation, the transaction-layer log is usually the first place you look for a "no credits available" message.

The data link layer: making an unreliable link reliable

The physical link can and does flip bits. The data link layer's job is to make the link behave as if it were lossless. It does three things to every outgoing TLP:

1. Prepends a 12-bit sequence number so the receiver can detect gaps or duplicates.
2. Appends a 32-bit LCRC (Link CRC) computed over the sequence number and the entire TLP.
3. Stores a copy in the replay buffer until an acknowledgement arrives.

The receiver recomputes the LCRC. If it matches and the sequence number is in order, it schedules an ACK Data Link Layer Packet (DLLP). If the LCRC fails or a sequence number is skipped, it sends a NAK, and the transmitter replays every TLP from the replay buffer starting at the failed sequence number. DLLPs — ACK, NAK, and flow-control updates — are generated and consumed entirely within the data link layer; they never reach the transaction layer and carry no address.

This replay mechanism is invisible when the link is clean but becomes a measurable tax on a marginal channel — exactly the kind of signal-integrity problem you characterize with an eye-diagram and a protocol analyzer from Keysight or Siemens (Tessent/analyzer tooling). A link littered with NAKs is a link losing effective bandwidth to replays.

The physical layer: lanes, framing, and encoding

The physical layer takes the data-link-wrapped packet, adds Start and End framing tokens, scrambles the data (to spread spectral energy), encodes it, and serializes it across one or more lanes. Each lane is a differential pair in each direction. Encoding has changed across generations and directly affects throughput.

Note the jump from Gen 2 to Gen 3: the rate only doubled (5→8 GT/s), yet usable bandwidth nearly doubled again because 8b/10b's 20% encoding tax was replaced by 128b/130b's ~1.5%. Gen 6 goes further, moving to PAM4 signaling with FEC — a change that lives almost entirely in the physical layer while TLP semantics above stay the same.

Credit-based flow control

PCIe never drops a TLP for lack of buffer space, because a sender is forbidden from transmitting unless the receiver has already advertised room. At link-up, each receiver broadcasts its buffer capacity in credits (one header credit = one TLP header; one data credit = 4 double-words = 16 bytes of payload), split across six virtual channels/types: posted, non-posted, and completion, each with header and data pools.

The transmitter maintains running counters. Before sending a TLP it checks it has enough header and data credits for that type; if not, it waits. As the receiver drains its buffers it emits UpdateFC DLLPs returning credits. The elegance is that this is entirely proactive — there is no "pause, retry" handshake per packet, so a well-provisioned receiver sustains full throughput. A receiver that advertises too few completion credits, by contrast, throttles every read-heavy workload, a classic finding when profiling an endpoint in a Cadence or Synopsys verification environment. For a refresher on how these transactions map to real driver behaviour, see our PCIe protocol basics guide and the companion piece on PCIe link training and the LTSSM.

Designing and verifying this credit logic correctly is non-trivial, which is why teams building PCIe controllers lean on mature verification IP and expert design services such as those offered by Avecas VLSI design services.

Putting it together: a Memory Read walk-through

Say a CPU reads 256 bytes from an NVMe BAR. (1) The transaction layer builds an MRd TLP — non-posted, with a tag, the target address, and length = 64 DW. (2) The data link layer stamps a sequence number and LCRC, buffers a copy. (3) The physical layer frames, scrambles, encodes, and sends it. (4) The completer's stack validates the LCRC, ACKs it, and the transaction layer sees a read request. (5) The completer returns one or more CplD TLPs (a single read can split into multiple completions bounded by the Read Completion Boundary and Max Payload Size). (6) Each completion is itself ACK'd and consumes completion credits at the requester. The read is done only when all expected bytes for that tag arrive.

Want to read live PCIe analyzer traces and design TLP-aware RTL with confidence? Our CourseTron protocol engineering track walks you through PCIe, from TLP fields to LTSSM, with real waveforms and lab exercises.

Frequently asked questions

What is the difference between a TLP and a DLLP?

A TLP (Transaction Layer Packet) carries application-level requests and completions — memory reads/writes, config, messages — and travels end-to-end. A DLLP (Data Link Layer Packet) is a small link-local control packet (ACK, NAK, flow-control credit updates, power management) that never leaves the data link layer and carries no address or payload.

Why are Memory Writes posted but Memory Reads non-posted?

A write has nothing to return, so making it posted avoids a wasteful round-trip and lets writes stream at line rate. A read must return data, so it is inherently non-posted — the requester waits for a Completion TLP. This is the single biggest reason reads are latency-bound and writes are bandwidth-bound.

How does credit-based flow control prevent packet loss?

A transmitter may only send a TLP if the receiver has already advertised enough header and data credits for that TLP's type. Because room is guaranteed before transmission, the receiver's buffers can never overflow, so PCIe does not need to drop and retransmit TLPs due to congestion.

Does the LCRC replace the ECRC?

No — they coexist. The LCRC is a 32-bit CRC added by the data link layer and checked hop-by-hop on every link. The optional ECRC is an end-to-end CRC carried inside the TLP digest and checked only by the ultimate completer, protecting against corruption inside switches that the per-hop LCRC would otherwise mask.