In short: CXL (Compute Express Link) rides on the PCIe physical layer but adds cache-coherent protocols — CXL.io, CXL.cache and CXL.mem — that let a CPU and accelerators share memory with load/store semantics, enabling pooled, tiered and expandable memory beyond what DDR channels alone can offer.
If you have ever watched a GPU training job stall because a server ran out of DRAM, or priced out a dual-socket box just to reach a memory capacity target, you have felt the problem CXL solves. Compute Express Link is an open interconnect standard (CXL 1.1 in 2019, 2.0 in 2020, 3.x since 2022) that layers coherent, low-latency memory semantics on top of the ubiquitous PCIe electrical and physical layer. This article unpacks the three sub-protocols, the device types, and why memory expansion and pooling are the headline use cases.
Why PCIe needed a coherent sibling
PCIe moves data in packets. When a CPU talks to a PCIe NVMe drive or a NIC, it uses descriptors, DMA and interrupts — producer/consumer semantics with explicit software-managed transfers. That is fine for block storage but terrible for fine-grained memory sharing. An accelerator that wants to read a single 64-byte cache line the CPU just wrote has no efficient, hardware-coherent way to do it over plain PCIe.
CXL reuses PCIe's PHY — a CXL 2.0 link runs on PCIe 5.0 at 32 GT/s per lane, and CXL 3.0 on PCIe 6.0 at 64 GT/s with PAM4 signalling. A x16 CXL 3.0 link therefore reaches roughly 128 GB/s per direction. But at link-training time the device negotiates (via "Flex Bus" and alternate protocol negotiation) whether it speaks plain PCIe or CXL. If CXL, up to three multiplexed protocols run over the same lanes.
The three sub-protocols
CXL.io is functionally PCIe. It handles device enumeration, configuration space, register access, interrupts and bulk DMA. Every CXL device must support CXL.io, because that is how the OS discovers and initialises it — it is the mandatory "control plane."
CXL.cache lets a device cache host memory coherently. An accelerator can pull a cache line into its local cache, and the host's home agent tracks that line's state so writes stay consistent. This is a device-to-host coherency protocol using MESI-style states (Modified, Exclusive, Shared, Invalid) with request/response/data channels in each direction.
CXL.mem is the inverse: it lets the host issue plain load/store commands to memory that physically lives on the device. The commands (MemRd, MemWr, and in CXL 2.0+ metadata-carrying variants) look to the CPU like ordinary memory access. This is the backbone of memory expansion.
Key fact: CXL.cache and CXL.mem have a fixed, asymmetric latency budget — the spec targets sub-200 ns pin-to-pin on CXL.mem, comparable to a remote NUMA hop, which is what makes device-attached DRAM usable as real main memory rather than a storage tier.
Device types: 1, 2 and 3
| Type | Protocols used | Typical device |
|---|---|---|
| Type 1 | CXL.io + CXL.cache | Accelerator/SmartNIC with no local memory that caches host DRAM (e.g. a crypto or compression engine) |
| Type 2 | CXL.io + CXL.cache + CXL.mem | GPU/FPGA with its own HBM that both caches host memory and exposes its memory to the host |
| Type 3 | CXL.io + CXL.mem | Memory expander / buffer — a card full of DDR5 or persistent memory that the host treats as extra capacity |
Type 3 is where the near-term money is. A single Type 3 expander card can add hundreds of GB over one x16 slot, letting you decouple capacity from the CPU's limited DDR channels (a modern server tops out at, say, 12 DDR5 channels per socket). The OS exposes this as a separate CPU-less NUMA node, and software or the kernel's tiering logic places cold pages there.
Memory pooling and fabrics
CXL 2.0 introduced the single-level switch, which enables pooling: several hosts connect to a switch, behind which sit memory devices carved into logical devices (an MLD — multi-logical device). A fabric manager assigns slices of a pool to whichever host needs them, so stranded DRAM in one server can be lent to a neighbour. CXL 3.0 goes further with multi-level switching, port-based routing, and memory sharing — the same region coherently visible to multiple hosts at once, opening the door to disaggregated, rack-scale memory fabrics.
How engineers actually verify CXL silicon
Designing a CXL controller is a verification-heavy exercise because you must prove coherency under adversarial traffic. Teams lean on commercial VIP (verification IP): Synopsys VC Verification IP for CXL, Cadence's CXL VIP and the Perspec system-level flows, and Siemens EDA's (Mentor) Questa with CXL compliance suites drive constrained-random stimulus and assertion checking against the spec's flit formats and coherency rules. Interoperability is then proven on CXL Consortium plugfests. For physical implementation, the same PCIe 5.0/6.0 SerDes IP and PHY sign-off flows apply, since CXL shares the electricals.
Key fact: CXL 3.0 adopts 256-byte flits with PCIe 6.0's FLIT mode and forward error correction, versus the 68-byte flits of CXL 1.1/2.0 — a change that verification environments must model explicitly, because flit framing affects latency, error handling and bandwidth efficiency.
Where CXL fits against other interconnects
CXL deliberately targets the "node and rack" scope. It is not competing with on-die NoC fabrics, nor with long-haul Ethernet. It overlaps with and has largely absorbed earlier coherent-interconnect efforts (CCIX, Gen-Z, OpenCAPI all folded their assets toward CXL). If you understand PCIe transaction semantics, CXL is the natural next layer — so a solid grounding in PCIe protocol fundamentals pays off directly. For coherency theory, our cache-coherence and MESI walkthrough covers the state machine CXL.cache builds on. If you are moving from protocol study toward building and verifying real IP, teams at Avecas VLSI design services work on exactly these PCIe/CXL controller and verification flows.
Ready to go deeper into memory and interconnect protocols with hands-on labs? Explore the CourseTron Protocols & SoC Interconnect courses.
Frequently asked questions
Is CXL replacing DDR memory?
No. CXL complements DDR. Directly-attached DDR5 remains the fastest, lowest-latency tier; CXL.mem adds a slightly higher-latency expansion and pooling tier. Most systems use both, with the OS tiering hot pages into local DRAM and colder pages onto CXL.
Can any PCIe slot run CXL?
Only if both the host root port and the device support CXL and negotiate it during Flex Bus link training. The slot is electrically PCIe, but the platform (CPU, BIOS) must expose CXL root ports. A CXL device will fall back to plain PCIe if the host cannot do CXL.
What is the practical latency of CXL memory?
Expect roughly 150–250 ns added over local DRAM for a Type 3 expander — similar to a one-hop remote NUMA access. That is why it works as main memory for capacity-bound workloads but is not a drop-in for latency-critical hot data.
Which CXL version should I target?
It depends on the platform. CXL 2.0 (switching, pooling, on PCIe 5.0) is broadly deployable today; CXL 3.0/3.1 (sharing, multi-level fabrics, PCIe 6.0) targets disaggregated rack-scale designs and is the forward-looking choice for new controller IP.