On 4.05.20 г. 10:43 ч., Chris Murphy wrote:
> btrfs-progs-5.6-1.fc32.x86_64
> kernel-5.6.8-300.fc32.x86_64
>
> I have one snapshot of a subvolume being sent from one Btrfs
> filesystem to another. Both are single device. The HDD scrub
> performance is ~90MB/s. The send/receive is averaging less than 2MB/s.
> Upon attaching strace to the send PID, it dies.
>
> # btrfs send /mnt/first/scratch/gits.20200503/ | btrfs receive /mnt/aviat
> ERROR: command length 1718776930 too big for buffer 65536
> #
>
> $ sudo strace -p 5628
> strace: Process 5628 attached
> ioctl(4, BTRFS_IOC_SEND, {send_fd=6, clone_sources_count=0,
> clone_sources=NULL, parent
> _root=0, flags=0}) = ?
> +++ killed by SIGPIPE +++
>
> During the send/receive, top reports both of them using 90%+ CPU
>
> rsync averages 65MBs
>
> Another subvolume, also one snapshot, has long periods of time (1/2
> hour) where the transfer rate is never above 500K/s, but both the
> btrfs send and btrfs receive PIDS are pinned at 99%+ CPU.
>
> perf top during high btrfs command CPU consumption and low IO throughput
>
> 34.01% [kernel] [k]
> mutex_spin_on_owner
> 28.35% [kernel] [k] mutex_unlock
> 8.05% [kernel] [k]
> mwait_idle_with_hints.constprop.
> 5.10% [kernel] [k] mutex_lock
That doesn't say which lock would be the contended one. Could you
perhahps run with perf top -g -U and see what is the callchain.
>
> perf top during moderate btrfs command CPU consumption and high IO throughput
>
> 15.16% [kernel] [k]
> fuse_perform_write
> 10.27% [kernel] [k]
> mwait_idle_with_hints.constprop.
> 7.97% [kernel] [k] mutex_unlock
> 7.35% [kernel] [k]
> mutex_spin_on_owner
> 5.75% btrfs [.] __crc32c_le
>
>
> I suspect many fragments for some files when performance is slow, and
> somehow btrfs send is more adversely affected by this than cp or
> rsync.
>
