Protocols

NVMe protocol basics

By CourseTron Team ·  6 min read ·  Updated Oct 2026 💬 𝕏 in f
Table of Contents
    🖨  Download / Print this guide (PDF)

    In short: nvme is a lean storage protocol built for solid-state drives that talk directly over PCIe. Instead of one slow command queue inherited from spinning disks, it uses paired submission and completion queues in host memory, so a modern SSD can juggle thousands of commands in flight at once with very low latency.

    What Is nvme?

    nvme (Non-Volatile Memory Express) is a register-level interface and command set designed from scratch for non-volatile storage such as NAND flash SSDs. Earlier drives spoke SATA/AHCI or SAS protocols that were created in the era of mechanical hard disks, where the bottleneck was a physical head seeking across a platter. Those protocols assumed deep queuing was pointless because the disk could only service one request at a time anyway.

    Solid-state media broke that assumption. A flash device has no moving parts and can read thousands of independent locations in parallel across many NAND dies and channels. The protocol wrapping it therefore needed to allow massive parallelism, minimal CPU overhead per command, and a short, efficient path from the application to the media. nvme answers all three by sitting directly on the PCI Express bus and using a queue model that lives in the host's own RAM.

    Key fact: AHCI (the old SATA model) supports a single command queue up to 32 entries deep. nvme supports up to 65,535 queues, each up to 65,536 commands deep — a design that matches how parallel flash really behaves.

    How nvme Uses Submission and Completion Queues

    The heart of the protocol is a pair of circular ring buffers allocated in host memory:

    • Submission Queue (SQ): where the host driver writes 64-byte command entries describing work it wants done — a read, a write, a flush, an identify.
    • Completion Queue (CQ): where the controller posts 16-byte completion entries once it has finished a command, reporting status and the command identifier.

    Queues work in pairs, but the mapping is flexible: several submission queues can share one completion queue. Each queue uses a doorbell register on the controller. When the host adds commands it writes the new tail pointer to the SQ doorbell; the controller writes the CQ head doorbell back after the host has consumed completions. A phase tag bit in each completion entry flips every time the ring wraps, letting the host detect fresh entries without an extra read.

    NVMe queue pair over PCIe Host (CPU + RAM) Submission Queue 64-byte commands Completion Queue 16-byte status + phase PCIe bus SSD Controller + doorbell registers NAND flash dies CourseTron · coursetron.com
    The host posts commands into the submission queue; the controller fetches them over PCIe and returns results to the completion queue.

    The nvme Command Lifecycle, Step by Step

    A single I/O travels through a well-defined sequence. Watching it move makes the queue model concrete:

    1. The driver builds a 64-byte command and places it at the SQ tail.
    2. It rings the SQ tail doorbell, telling the controller new work is waiting.
    3. The controller fetches the command over PCIe using DMA and executes it against the flash.
    4. The controller writes a completion entry into the CQ and (optionally) raises an MSI-X interrupt.
    5. The driver processes the completion, then rings the CQ head doorbell to free the slot.
    One NVMe command, round trip Host SQ SSD controller Host CQ command over PCIe completion over PCIe CourseTron · coursetron.com
    A command (teal) travels host→SSD; a completion (navy) returns SSD→host. Thousands of these overlap concurrently in a real device.

    Why nvme Runs Directly Over PCIe

    PCIe is a packet-based, point-to-point serial interconnect with multiple lanes. A Gen4 x4 link — the common consumer SSD configuration — delivers roughly 8 GB/s of bandwidth, far beyond SATA's 600 MB/s ceiling. By mapping its doorbells and data transfers onto PCIe, nvme skips the legacy host-bus-adapter layer entirely. Commands and data move by direct memory access, and interrupts use MSI-X, so many queues can be serviced across many CPU cores without a single lock becoming a bottleneck.

    This tight coupling is also why nvme scales with the bus: as PCIe generations double in speed, the same protocol keeps up without redesign. The specification further defines namespaces (logical partitions of the media), admin versus I/O queue separation, and optional features like write-zeroes, dataset management (TRIM), and zoned namespaces for newer flash layouts.

    nvme vs. the Old Storage Stack

    • Queues: AHCI has one 32-deep queue; nvme has up to 64K queues, each up to 64K deep.
    • Latency: nvme removes the HBA translation layer, cutting software overhead per command dramatically.
    • Parallelism: per-core queues let the protocol exploit many NAND channels at once.
    • Transport reach: the same command set extends over fabrics (nvme-oF) using RDMA, TCP, or Fibre Channel.
    Queue model: AHCI vs NVMe AHCI (SATA) 1 queue · 32 deep single command queue NVMe Queue 1 · 65,536 deepQueue 2 · 65,536 deepQueue 3 · 65,536 deep … up to 65,535 queues CourseTron · coursetron.com
    AHCI exposes one 32-entry queue; nvme allows up to 65,535 queues of up to 65,536 commands each.
    Learn this hands-on

    Real projects and industry tool flows, with certification.

    Explore CourseTron Courses →

    nvme — FAQ

    Is nvme a connector or a protocol?

    It is a protocol and command set. It commonly runs over the M.2 or U.2 connectors, but those are physical form factors — the nvme logic rides on the PCIe lanes those connectors carry.

    What is the difference between a submission and completion queue?

    The submission queue holds commands the host wants executed; the completion queue holds the controller's status replies. They operate as a pair and are coordinated through doorbell registers and a phase tag.

    Why is nvme faster than SATA SSDs?

    Two reasons: it talks over high-bandwidth PCIe instead of SATA, and its deep multi-queue model lets flash serve thousands of parallel requests, which the single-queue AHCI model could never express.

    Can nvme work over a network?

    Yes. nvme over Fabrics (nvme-oF) carries the identical command set across RDMA, TCP, or Fibre Channel, letting remote storage behave almost like a local drive.

    Related: PCIe protocol basics · Glossary · all courses.

    Advertisement