A filesystem benchmark that measures the failure cases: degraded arrays, full disks, fsync tails

Most filesystem comparisons are a table of sequential throughput taken once, on one machine, by someone with a preference. This one is a continuously running CI suite: 593 recorded runs, the latest on 18 September, with the trend for every layout plotted over months rather than captured in an afternoon.

The comparison spans the classic pair and the copy-on-write set — ext4, XFS, ZFS, Btrfs, bcachefs — and, importantly, not as bare filesystems but as layouts: single device, md-RAID10, LVM-RAID10, an LVM integrity variant, XFS on a ZFS volume, ZFS mirror and RAIDZ1/RAIDZ2, Btrfs RAID1 and RAID6, bcachefs with two replicas and with erasure coding, plus a LUKS variant. That axis matters, because in production nobody runs a filesystem — they run a filesystem on top of a particular stack.

What makes it worth reading is which phases it measures. Alongside ordinary throughput there are the cases most benchmarks skip because they are awkward to set up:

• Overwrite after snapshot — writing into a file that was just snapshotted, where every write has to break extent sharing. The unshare penalty is a daily cost in any snapshotting deployment, and it never appears in a throughput table.
• Degraded random write — one device deliberately failed (zpool offline, mdadm --fail, a detached loop device with a degraded mount) and the array measured while it is missing a disk. That is the state you will actually be in when performance suddenly matters.
• ENOSPC behaviour — fill to hard ENOSPC, then delete a file and verify that space comes back and a new write succeeds. Whether a full filesystem can be recovered by deleting something is a real operational property, and CoW filesystems have historically been the interesting cases.
• fsync tail latency at p99 and p99.9, rather than an average — which is the number a database actually lives on.

What it means

The caveat at the top is the most valuable line on the page. The runs use loop devices on shared ephemeral cloud VMs, one VM per filesystem, and the author says plainly: compare shapes and ratios, not absolute MB/s, and each job carries a host-calibration anchor so drift in the underlying hardware is visible. A benchmark that tells you what it cannot support is more useful than one that publishes clean numbers and leaves you to discover the limits yourself.

Measure the failure modes, not the happy path — the same rule applies to your own systems. Nobody is surprised by sequential write speed on an idle array. The surprises arrive when a disk is missing, when the volume is full, when the snapshot you take every hour is making each subsequent write more expensive. If your capacity planning is built on numbers from the healthy case, it is planning for the one state that never needs it.

And a trend line beats a measurement. Because the suite runs repeatedly against the same layouts, a regression in a kernel or a filesystem shows up as a step in the chart rather than as an argument on a mailing list. That is the difference between a benchmark and a monitor, and it is why this one is worth bookmarking rather than reading once.

Code is Apache-2.0, the result data CC BY 4.0, and the methodology and how to run it on real hardware are documented in the repository.