Skip to the yard
Waxsealai

How Enterprise SSDs Are Tested for Sustained Performance and Latency

Shipped
Shipper
Shane Downing
Readers
884

A custom Debian and fio test bench measures enterprise SSDs under steady-state workloads, with checks for latency, recovery, AI checkpoints and thermal limits.

How Enterprise SSDs Are Tested for Sustained Performance and Latency

Enterprise SSD performance depends on what happens after the drive fills, garbage collection starts and several workloads compete for its resources. A fresh-drive speed test misses those conditions. This test system is designed to measure sustained behavior, expose the host’s own limits and make each result traceable to the settings that produced it.

Intel Xeon 658X

The bench marks a return to in-depth testing of enterprise and data center SSDs after a hiatus of several years. Consumer drives and conventional hard drives often lack the endurance and performance needed for extensive AI workloads, while U.2, U.3, E3.S and E1.S drives are appearing in high-end workstations and home labs. These products offer greater capacity, performance and reliability, but buying one is a substantial commitment for organizations. A useful review needs to show how a drive behaves under real pressure, not just repeat its specification sheet.

A workstation built to keep pace with the drives

The test system centers on Intel’s 24-core, 48-thread Xeon 658X from the Granite Rapids family. Its boost clock reaches 4.9 GHz. Both per-core speed and total core count matter here: clock speed affects how many IOPS one core can handle and how quickly it answers an individual request, while the 24 cores let the team distribute work across eight fio jobs and test multiple drives at once.

Asus Pro WS W890E-Sage SE

An Asus Pro WS W890E-Sage SE motherboard provides the connectivity. Its onboard MCIO x8 link can be split into x4/x4 for two U.2 or U.3 drives. Seven PCIe 5.0 expansion slots—six x16 and one x8—connect to the Xeon’s 128 PCIe 5.0 lanes, leaving room to attach SSDs and graphics cards without making host bandwidth the immediate bottleneck.

Noctua’s NH-U14S DX-4677 air cooler handles CPU heat. It supports LGA4677 and LGA4710 sockets and uses two NF-A15 140 mm fans. The test system does not yet have an RTX Pro graphics card, which would help with real-world AI workload testing. Its current configuration still covers the core storage performance measurements.

Noctua NH-U14S DX-4677

G.Skill supplied a 64GB G5 DDR5-6400 C32 R-DIMM kit, arranged as four 16GB modules. The memory runs at the standard DDR5-5600 rate: memory frequency does not affect these enterprise SSD tests. Corsair supplied the MP700 Pro XT 2TB as the system’s primary drive. This PCIe 5.0 SSD is rated for sequential reads up to 14,900 MB/s and writes up to 14,500 MB/s, keeping the workstation responsive during long test runs.

A Corsair HX1500i power supply provides 1,500W of capacity for the system and potential upgrades. It is rated 80 Plus Platinum, with at least 89% efficiency at full load. Quiet, stable operation matters when a test can keep the machine busy for many hours.

Gigabyte GT 1030 Silent Low Profile 2G

Enterprise SSDs come in form factors that don’t fit a typical desktop slot, so the bench uses adapters and drive enclosures for access and cooling. Icy Dock’s ExpressSlot Elite MB221E5P-B and MB303E5P-B hold one and two E1.S drives, respectively. Their tool-less installation doesn’t require removing the adapter from the PCIe slot, and built-in 50 mm fans cool the drives during testing. A ToughArmor MB218E5P-B enclosure fits in a 5.25-inch bay and can hold up to eight E1.S drives for future RAID testing.

For E3.S drives, a JMT ADT-R27C adapter is paired with a Noctua NF-F12 industrialPPC-24V-3000 IP67 PWM fan. The fan can run at 12V while providing substantial airflow. The hardware sits on a TGDGamer E-ATX open-air test bench. This layout makes drives easy to install and remove, avoids heat buildup from a closed case and supports motherboard sizes up to SSI-EEB.

G.Skill G5 DDR5-6400 C32 64GB (4x16GB)

One software stack for every run

The bench runs a custom Debian 13 image with its own maintained software suite. The kernel is pinned at Linux 7.2.6 from Debian backports; Debian 13’s stock 6.12 kernel predates support for the motherboard’s Intel E610 10GbE controller. The suite can run remotely, though someone still has to change the drives at the bench.

Debian was chosen over Ubuntu because it has fewer background services, does not include Snap and does not enable unattended upgrades by default. The team removes AppArmor and disables scheduled work that could interfere with a run, including the weekly TRIM timer and a drive-health service that polls the SSDs. The suite records any such services that remain active. It also disables NVMe autonomous power-state transitions at the start of each run and checks that they are off, preventing idle gaps from putting a drive into a low-power state and adding wake-up latency.

Corsair MP700 Pro XT 2TB

Every workload runs through fio 3.42 with io_uring and direct I/O against the raw block device. There is no filesystem or page cache between fio and the SSD, so the system’s 64GB of RAM does not cache test data. The team builds fio from source because the Debian package lacks SPRandom, the preconditioning tool used in the suite.

The choice of fio also keeps the workload definitions and latency accounting consistent across the results. Some enterprise reviews use Oracle’s Vdbench, but fio is open source and actively maintained; Vdbench has not had a new release since 2018. The workload patterns can be assembled from common elements such as read/write ratio, block size and random or sequential access. The suite covers mixed read/write ratios at 4K and 8K, though it does not currently test mixed block-size profiles. Using Vdbench would not, by itself, make results comparable with other reviewers, whose hardware, preconditioning and workload definitions differ.

Corsair HX1500i

Preconditioning before measuring

Freshly erased SSDs can post performance that they cannot sustain once their flash is in use. Each run therefore begins by trying to secure-erase the drive with nvme format. If that fails, the suite tries a regular format and then a full-drive discard, recording which method worked.

Next, SPRandom prepares the drive for steady-state testing. The open-source fio contribution from SanDisk uses user capacity and over-provisioning as inputs, aiming to establish a steady-state layout in one pass instead of repeatedly filling the drive for hours. The team estimates over-provisioning from the drive’s capacity or enters it directly. NVMe drives do not report their raw flash capacity; for example, a 7.68TB drive assumed to contain 8TiB of NAND has about 14.5% over-provisioning.

ExpressSlot Elite MB221E5P-B

The suite then verifies that the drive has actually settled. It runs 4K random writes at queue depth 256 (QD256) and checks the last 30 minutes using the two measures in the Storage Networking Industry Association’s steady-state rule, with limits ten times tighter. The window is split into five six-minute periods: average write speeds must stay within 2% of each other, and the trend across the full window must move by no more than 1%.

That check can take time, but a looser earlier version passed the first test drive while its write speed was still rising. The full settling curve is saved with each run. If a drive does not reach steady state within three hours, the suite marks it accordingly and continues. Later, it repeats the four-corner 4K write test as a second check; a difference greater than 3% between the early and late results flags the run.

ExpressSlot Elite MB303E5P-B

The suite also looks for outliers as it works. A test is repeated immediately if its slowest moment falls below 70% of its average. During a random sweep, any point more than 10% away from the curve formed by its neighbors is measured again before the drive is erased for sequential tests. The original and repeated results are retained, and the review notes retests.

Before sequential and AI tests, the drive is erased again and filled end to end with 128K sequential writes at QD64. This prepares it with the kind of I/O those tests will measure. Without that step, sequential write results can swing sharply: on the first test drive, the fastest result exceeded the slowest by more than three times. After a sequential fill, results stayed flat from QD2 through QD1024 at both 128K and 1MB block sizes.

ToughArmor MB218E5P-B

Where supported, drives are formatted with 512-byte logical blocks. That enables tests down to 512-byte transfers, which a 4K-native namespace cannot issue. The SNIA enterprise IOPS test also uses 512-byte transfers, and Kioxia has promoted 512-byte access for its GP Series, aimed at GPU-initiated storage. Drives that cannot use the format are tested as shipped, with unsupported sub-4K results marked accordingly. The format change adds no time because it occurs during the secure-erase command.

Separating drive performance from host limits

A sufficiently fast SSD can expose limits in the test system itself. Early runs identified three sources of distortion: CPU idle states, fio’s per-job capacity and repeated random access patterns.

JMT ADT-R27C

Waking a Xeon 658X core from its deepest idle state can take up to 170 microseconds. At QD1, that delay directly affects measured latency. Keeping the cores awake throughout a run reduced the first drive’s average 4K QD1 read latency by about one-sixth.

A single fio job can also saturate one CPU core before it fully exercises a PCIe 5.0 SSD. Splitting the same test across eight jobs made it several times faster, so random tests now spread their queue depth across as many as eight fio jobs. This is one reason the bench needs both fast cores and enough of them.

Fio’s default random-number seed created another problem: each run repeated the same sequence of offsets, causing successive random-write tests to rewrite the same blocks in the same order. On the first drive, that pattern made 4K random-write performance more than twice the sustained figure. Each random test now uses a fresh seed.

The suite leaves other settings closer to typical application behavior. Completions arrive by interrupt rather than polling, and the IOMMU stays in its default mode. On the first drive, both polling and IOMMU passthrough reduced 4K QD1 write latency, but those settings are not used for the main charts.

Interrupt handling also caps the bench at roughly 2.8 million 4K random-read IOPS. Adding fio jobs does not raise that ceiling, although polling does. The fastest SSDs may therefore score below their specification-sheet figures because of the host’s limit. At low queue depths, particularly for writes, measured latency can also exceed what a tuned server would show, though it better reflects a typical setup.

To show the drive’s potential without interrupt overhead, the suite runs three additional tests with polled I/O at the end of the random stage: peak 4K random reads and 4K random reads and writes at QD1. These results appear in separate charts and are not mixed with the main results.

Workloads that show more than peak speed

The benchmark covers headline performance, scaling, latency and recovery. The four-corner results measure 4K random reads and writes at QD256, plus 128K and 1MB sequential reads and writes at QD64. Because the drive is preconditioned, these numbers represent sustained operation rather than a fresh-erase peak.

Random and mixed-workload sweeps cover 4K and 8K transfers from QD1 to QD4096, across eight read/write ratios ranging from all reads to all writes. Low queue depth matters to latency-bound software, while 8K is a common database page size. A separate sequential sweep measures 128K and 1MB reads and writes from QD1 to QD1024, showing how much parallelism each drive needs to reach its maximum speed. Low queue depths also represent jobs such as a single restore or scan.

A block-size sweep tests random reads and writes from 512 bytes to 1MB at QD32. Small writes below a drive’s mapping unit can trigger read-modify-write work; larger transfers reveal the drive’s bandwidth ceiling. Tail-latency tests measure 4K and 8K reads and writes at QD1, QD4 and QD16, from the median (P50) through the 99.999th percentile (P99.999). An average can conceal the slow requests applications notice. At one million IOPS, the slowest 0.001% still amounts to ten requests per second.

The saturation curve tracks a 4K 70/30 read/write mix at 13 queue depths, QD1 through QD4096. Throughput is expressed as a share of the drive’s own peak, allowing the chart to show where latency begins to climb as load increases. That turning point offers a practical operating ceiling: the load a server can sustain before giving up low latency.

To show behavior over time, the suite runs an hour of 4K random writes at QD256. The result reveals whether performance stays level or oscillates as garbage collection works through the drive. It also records drive temperature against the SSD’s own warning and critical thresholds.

Two recovery tests examine what happens after heavy writes. For write-to-read recovery, the drive receives several minutes of 4K random writes at QD256, followed by four minutes of QD1 reads. If garbage collection is still active, reads may initially slow; the test measures both the size and duration of that drop. For idle recovery, the drive writes for one minute, waits for an idle period ranging from zero to 300 seconds, then runs a 10-second write burst. Logging each second shows how much speed a quiet interval restores.

Simulated database patterns cover three common operations without measuring a database’s RAM cache: 4K QD1 log writes with a sync after each write, a random 8K 70/30 mix at QD16, and a 256K sequential read scan. AI checkpoint tests model saves and restores for 8B- and 70B-class models: 105GiB written across eight writers and 912GiB across 64 writers. Completion time is measured at the slowest writer, since a real checkpoint is not complete until that writer finishes.

A separate interference test measures how checkpoint saves affect other work on the same drive. It runs a fixed 4K read load by itself and again during a checkpoint. Reads are timed from when they were due, not merely from when they were sent, so a request delayed behind a stalled operation still contributes to the result.

Repeatability and scope

A full run takes most of a day on a 7.68TB PCIe 5.0 SSD and writes across the drive many times. The process is automated, can run unattended and resumes after a crash from its last completed stage. A shorter express mode supports bench checks and triage. If a result looks unusual, targeted runs can repeat the relevant tests or change one setting at a time before the team raises the question with the drive maker.

Some qualification work is outside the review’s scope. The tests characterize drives; they do not certify them, so dual-port failover, power-loss validation and single-root I/O virtualization (SR-IOV) are left to qualification labs. The suite can grow as new tests prove useful on the bench. Its central value is already practical: consistent preconditioning, controlled software and recorded parameters make comparisons repeatable, while separate measurements expose where the drive ends and the test host begins.

Source: Shane Downing · www.tomshardware.com

Same vesselRelated cargo