Linux 7.3 To Overcome "Significant Bottleneck" For Small I/O With PCIe Gen5 NVMe SSDs

While the Linux 7.2 feature merge window ended just days ago and the better part of two months now before v7.2 will be released as stable, there are already features beginning to accumulate that will target the Linux 7.3 cycle. The most exciting change I've seen to kick off that dance ahead of Linux 7.3 is addressing a "significant" bottleneck affecting small direct I/O performance with speedy storage such as PCIe Gen5 NVMe SSDs.

Bytedance engineer Fengnan Chang noticed a "significant bottleneck" when carrying out 4K random reads on Gen5 NVMe SSDs. Significant software overhead was tracked down to the Linux kernel's IOmap framework with its direct I/O path. The CPU ends up spending much time on memory allocations and the IOmap state machine. In turn, Fengnan Chang devised a simple direct I/O path for small direct I/O when using IOmap such as with EXT4 and XFS.

This simple DIO path is used to reduce the IOmap overhead when a read request is received where the I/O size is smaller or equal to the inode blocksize. Plus other limitations like the inode not being encrypted. Following this creation, a jump from 1.92M IOPS to 2.19M IOPS for 4K random reads were observed in testing on PCIe Gen5 NVMe SSDs. Additional benchmarks are showing benefits of up to 10% on EXT4 and XFS when using IO_uring with modern PCIe Gen5 NVMe SSD storage and especially with higher queue depths.

This simple DIO path for IOmap is merged into the VFS subsystem's "fvfs-7.3.iomap" Git branch. With it now being queued in vfs/vfs.git for a "7.3" branch, expect it to be submitted for the Linux 7.3 cycle later this year.