Hi all,
I'm attempting to understand a roughly 30% degradation in BTRFS RAID0
for large read I/Os across six disks compared with ext4 atop mdadm RAID0.
Specifically, I achieve performance parity with BTRFS in terms of
single-threaded write and read, and multi-threaded write, but poor
performance for multi-threaded read. The relative discrepancy appears
to grow as one adds disks. At 6 disks in a RAID0 (yes, I know, and I do
not care about data persistence as I have this solved at a different
layer) I see approximately 1.3GB/s for ext4 atop mdadm, but only about
950MB/s for BTRFS, both using four threads to read and write four
different large files. Across a large number of my nodes this
aggregates to a sizable performance loss.
This has been a long and winding road for me, but to keep my question
somewhat succinct, I'm down to the level of block tracing and one thing
that stands out between the two traces is the number of rather small
read I/O's that reach one of the drives in the test is vastly different
for mdadm RAID0 vs BTRFS, which I think explains (in part at least) the
performance drop off. The read queue depth for BTRFS hovers in the
upper single digits while the ext4/mdadm queue depth is towards 20. I'm
unsure right now if this is related or not.
Benchmark: FIO was used with the following command:
fio --name=read --rw=read --bs=1M --direct=0 --size=16G --numjobs=4
--runtime=120 --group_reporting
The block sizes and counts of I/Os at that size I'm seeing for both
cases comes in like the following (my max_segment_kb_size is 4K, hence
the above typical upper-end):
BTRFS:
Count Read I/O Size
21849 128
18 640
9 768
3 1280
9 1408
3 2048
3 2560
1011 2688
507 2816
ext4 on mdadm RAID0:
Count Read I/O Size
9 8
3 16
5 256
5 768
19 1024
716 1536
5 1592
5 2504
695 2560
24 4096
21 6656
477 8192
Before I dive into the BTRFS source or try tracing in a different way, I
wanted to see if this was a well-known artifact of BTRFS RAID0 and, even
better, if there's any tunables available for RAID0 in BTRFS I could
play with. The man page for mkfs.btrfs and btrfstune in the tuning
regard seemed...sparse.
Any help or pointers are greatly appreciated!
Thanks,
ellis
