btrfs-progs-5.6-1.fc32.x86_64
kernel-5.6.8-300.fc32.x86_64
I have one snapshot of a subvolume being sent from one Btrfs
filesystem to another. Both are single device. The HDD scrub
performance is ~90MB/s. The send/receive is averaging less than 2MB/s.
Upon attaching strace to the send PID, it dies.
# btrfs send /mnt/first/scratch/gits.20200503/ | btrfs receive /mnt/aviat
ERROR: command length 1718776930 too big for buffer 65536
#
$ sudo strace -p 5628
strace: Process 5628 attached
ioctl(4, BTRFS_IOC_SEND, {send_fd=6, clone_sources_count=0,
clone_sources=NULL, parent
_root=0, flags=0}) = ?
+++ killed by SIGPIPE +++
During the send/receive, top reports both of them using 90%+ CPU
rsync averages 65MBs
Another subvolume, also one snapshot, has long periods of time (1/2
hour) where the transfer rate is never above 500K/s, but both the
btrfs send and btrfs receive PIDS are pinned at 99%+ CPU.
perf top during high btrfs command CPU consumption and low IO throughput
34.01% [kernel] [k]
mutex_spin_on_owner
28.35% [kernel] [k] mutex_unlock
8.05% [kernel] [k]
mwait_idle_with_hints.constprop.
5.10% [kernel] [k] mutex_lock
perf top during moderate btrfs command CPU consumption and high IO throughput
15.16% [kernel] [k]
fuse_perform_write
10.27% [kernel] [k]
mwait_idle_with_hints.constprop.
7.97% [kernel] [k] mutex_unlock
7.35% [kernel] [k]
mutex_spin_on_owner
5.75% btrfs [.] __crc32c_le
I suspect many fragments for some files when performance is slow, and
somehow btrfs send is more adversely affected by this than cp or
rsync.
--
Chris Murphy
