NVMe‑oF NVMe over Fabrics Low latency storage iSCSI vs FC All‑flash array RDMA

What Is NVMe‑over‑Fabrics (NVMe‑oF) and Why Modern Data Centers Need It

Speed is the currency of the modern data centre. When applications demand microsecond latency and traditional protocols (iSCSI, Fibre Channel) become bottlenecks, NVMe‑over‑Fabrics (NVMe‑oF) steps in. This technology extends the ultra‑fast NVMe protocol from local connections to the network, delivering storage access with latency comparable to direct‑attached drives. In this article, we explain what NVMe‑oF is, how it works, how it compares to iSCSI and FC, and why it is becoming a must‑have for all‑flash infrastructure.

1. What Is NVMe and Why Was It a Revolution?

NVMe (Non‑Volatile Memory Express) is a protocol designed specifically for flash storage. Unlike legacy SATA and SAS, NVMe uses parallel command queues (up to 65,535 queues with 65,535 commands each), delivering:

Ultra‑low latency – below 20 microseconds.
High throughput – up to 7 GB/s and beyond per drive.
Massive parallelism – handling thousands of concurrent I/O operations.

NVMe has become the standard for local drives, but until recently its benefits were confined to a single server. NVMe‑oF solves this by making NVMe accessible over the network.

2. What Is NVMe‑over‑Fabrics (NVMe‑oF)?

NVMe‑oF encapsulates NVMe commands in network packets and transmits them over low‑latency fabrics: Ethernet (with RDMA – RoCE, iWARP), InfiniBand, or Fibre Channel (FC‑NVMe). This allows servers to access remote NVMe storage as if it were locally attached.

The key differentiator from traditional protocols is the absence of command‑translation overhead. iSCSI wraps SCSI commands in TCP/IP, adding latency. NVMe‑oF transfers native NVMe commands directly, with network‑level latency below 10 microseconds.

The main advantage

NVMe‑oF enables high‑performance clusters with shared storage, where access latency is virtually indistinguishable from local NVMe. This is a game‑changer for databases, AI/ML, and high‑frequency trading.

3. NVMe‑oF vs iSCSI vs Fibre Channel (FC)

To appreciate NVMe‑oF, let’s compare it with the two main SAN protocols:

iSCSI – uses TCP/IP, adding latency (typically 100–500 µs). Suitable for moderate workloads but not high‑performance systems.
Fibre Channel (FC) – dedicated network with low latency (50–100 µs). Reliable but expensive and complex, requiring specific switches and HBAs.
NVMe‑oF – sub‑10 µs network latency, bandwidth up to 400 Gbps (with InfiniBand or 400G Ethernet). Supports multi‑path routing and scale‑out.

Moreover, NVMe‑oF can run on Ethernet (with RoCE or iWARP), letting you leverage existing network infrastructure with minimal upgrades, unlike FC which requires a separate network.

4. Where Is NVMe‑oF Used?

High‑throughput databases – OLTP, banking, trading platforms where every microsecond counts.
Artificial intelligence and machine learning – training models on massive datasets requires fast storage access. NVMe‑oF accelerates dataset loading and checkpointing.
Virtualisation and containerisation – Kubernetes, VMware vSphere with high VM/container density, where fast migration and failover are essential.
High‑Performance Computing (HPC) – scientific simulations, financial modelling, image processing.
All‑flash arrays – NVMe‑oF is a natural complement to all‑flash storage systems like Huawei OceanStor Dorado and Dell PowerStore, unlocking their full potential.

5. How to Build an NVMe‑oF Infrastructure

Deploying NVMe‑oF requires:

NVMe‑oF targets – all‑flash arrays or servers with NVMe drives that support the protocol.
Initiators – servers with RDMA‑capable network adapters (RoCE or iWARP).
Low‑latency network – Ethernet (25/100/400 Gbps) with RoCE, InfiniBand, or FC (32/64 Gbps) for FC‑NVMe.
Switches – for Ethernet, choose data centre switches like Cisco Nexus 9300‑GX2 with 400G and RoCE support.

NVMe‑oF configuration is simpler than FC, especially with Ethernet and RoCE. Many vendors offer ready‑to‑use solutions with built‑in NVMe‑oF support.

Deployment tip

Start with a pilot project on a small cluster. Use Ethernet with RoCE – it leverages existing cabling and switches. For maximum performance, choose adapters with native NVMe‑oF support and verify compatibility with your storage system.

6. NVMe‑oF in Huawei, Dell, and Lenovo Products

All leading storage vendors are actively adopting NVMe‑oF:

Huawei – OceanStor Dorado 3000 and Dorado 5000 support NVMe‑oF over Ethernet and FC, delivering 0.05 ms latency.
Dell – PowerStore 1200T features built‑in NVMe‑oF support, including optimisation for container environments.
Lenovo – ThinkSystem DM7100F offers converged file and block storage with NVMe‑oF for versatile workloads.

We supply all these solutions directly from China with full warranty and engineering support.

Planning an NVMe‑oF deployment and need advice? Contact our engineers

7. Frequently Asked Questions (FAQ)

How does NVMe‑oF differ from regular NVMe?

Regular NVMe is a local protocol for connecting SSDs to a server via PCIe. NVMe‑oF extends it over the network, allowing servers to access remote NVMe drives with similarly low latency.

What network is required for NVMe‑oF?

NVMe‑oF supports several fabrics: Ethernet with RDMA (RoCE or iWARP), InfiniBand, and Fibre Channel (FC‑NVMe). For new projects, Ethernet 25/100/400 Gbps with RoCE is often chosen for its flexibility and cost.

Can I use NVMe‑oF with my existing infrastructure?

Yes, if your network supports RDMA (RoCE or iWARP), you can adopt NVMe‑oF without replacing all hardware. For FC networks, you may need to upgrade to FC‑NVMe (32/64 Gbps).

What does it cost?

The cost includes the NVMe‑oF‑ready storage system, network adapters, and switches. It is higher than iSCSI but significantly lower than FC, especially with Ethernet. The long‑term benefits of lower latency and higher performance justify the investment.

Accelerate your storage with NVMe‑oF

We help design and deploy NVMe‑oF infrastructure, select the right hardware, and provide ongoing support. Direct sourcing from China gives you the best value for money.

Get a consultation on NVMe‑oF

In the next article, we will explore scale‑out storage systems and their role in Big Data and AI. Stay tuned!