On Tue, Apr 7, 2020 at 2:39 PM kjansen387 <kjansen387@xxxxxxxxx> wrote: >$ grep /export /etc/fstab >UUID=8ce9e167-57ea-4cf8-8678-3049ba028c12 /export btrfs >device=/dev/sde,device=/dev/sdf 0 2 I'd use only noatime for options. There's no advantage of specifying devices. And also fs_passno can be 0. fsck.btrfs is a no op anyway, so it doesn't hurt anything to leave it at 2. > I've attached sysstat info of my disks. What's obvious is that 2 disks > have the load (one is written to, the other one is the mirror), and 3 > are pretty idle. But, it's 2.4MB per second - that's not much! Lots of small file writes maybe? What's iostat show for utilization? Or vmstat for io? Hard drives of course have limited IO per second. > I've just changed the space_cache to v2, but it doesn't seem to help > much. 'sync -f /export/tmp' still takes very long at times (just took 22 > seconds!) I just did a strace on this command and it uses syncfs, not fsync. I'm pretty sure on Btrfs this is a full filesystem sync, which is expensive, all data and metadata. So if it's very dirty, yeah it could take some time to flush everything to disk. Try it without -f and you'll get fsync. > Any way we can find the cause, before I move everything into subvolumes > ? I'd like to avoid that if possible. Sounds a bit overkill for 2.4MB/s > writes, and I think most of it is going to one influxdb database anyway. >From the sysrq... [937013.794093] mysqld D 0 10400 1 0x00004000 [937013.794240] do_fsync+0x38/0x70 [937013.794253] mysqld D 0 10412 1 0x00004000 [937013.794297] do_fsync+0x38/0x70 [937013.794306] mysqld D 0 10421 1 0x00004000 [937013.794353] do_fsync+0x38/0x70 [937013.794788] WTJourn.Flusher D 0 1186978 1186954 0x00004320 [937013.794894] do_fsync+0x38/0x70 [937013.794903] ftdc D 0 1186980 1186954 0x00000320 [937013.794951] ? btrfs_create+0x200/0x200 [btrfs] [937013.795022] influxd D 0 2082782 1 0x00004000 [937013.795086] do_fsync+0x38/0x70 [937013.795098] influxd D 0 2082804 1 0x00004000 [937013.795143] do_fsync+0x38/0x70 [937013.795191] vim D 0 2228648 2223845 0x00004000 [937013.795286] do_fsync+0x38/0x70 Quite a lot of fsync all at once. But I'm not very good at parsing kernel output. Maybe a developer will have input. -- Chris Murphy
