Hi Chris,
$ grep /export /etc/fstab
UUID=8ce9e167-57ea-4cf8-8678-3049ba028c12 /export btrfs
device=/dev/sde,device=/dev/sdf 0 2
The device section is not complete, but I don't think that matters
because btrfs handles this automatically right ?
$ mount | grep /export
/dev/sde on /export type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)
I've attached sysstat info of my disks. What's obvious is that 2 disks
have the load (one is written to, the other one is the mirror), and 3
are pretty idle. But, it's 2.4MB per second - that's not much!
I've just changed the space_cache to v2, but it doesn't seem to help
much. 'sync -f /export/tmp' still takes very long at times (just took 22
seconds!)
Any way we can find the cause, before I move everything into subvolumes
? I'd like to avoid that if possible. Sounds a bit overkill for 2.4MB/s
writes, and I think most of it is going to one influxdb database anyway.
Thanks!
On 07-Apr-20 22:22, Chris Murphy wrote:
On Tue, Apr 7, 2020 at 2:11 PM Chris Murphy <lists@xxxxxxxxxxxxxxxxx> wrote:
On Tue, Apr 7, 2020 at 1:46 PM kjansen387 <kjansen387@xxxxxxxxx> wrote:
Hello,
I'm using btrfs on fedora 31 running 5.5.10-200.fc31.x86_64 .
I've moved my workload from md raid5 to btrfs raid1.
# btrfs filesystem show
Label: none uuid: 8ce9e167-57ea-4cf8-8678-3049ba028c12
Total devices 5 FS bytes used 3.73TiB
devid 1 size 3.64TiB used 2.53TiB path /dev/sde
devid 2 size 3.64TiB used 2.53TiB path /dev/sdf
devid 3 size 1.82TiB used 902.00GiB path /dev/sdb
devid 4 size 1.82TiB used 902.03GiB path /dev/sdc
devid 5 size 1.82TiB used 904.03GiB path /dev/sdd
# btrfs fi df /export
Data, RAID1: total=3.85TiB, used=3.72TiB
System, RAID1: total=32.00MiB, used=608.00KiB
Metadata, RAID1: total=6.00GiB, used=5.16GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
After moving to btrfs, I'm seeing freezes of ~10 seconds very, very
often (multiple times per minute). Mariadb, vim, influxdb, etc.
See attachment for a stacktrace of vim, and the dmesg output of 'echo w
> /proc/sysrq-trigger' also including other hanging processes.
What's going on.. ? Hope someone can help.
How busy are the databases? What are the mount options for this volume?
I think there is some kind of write contention possible if there's
heavy fsync writes, since the tree log is per subvolume? (Maybe a
developer can describe this correctly if I haven't.) A possible work
around is putting each database in its own subvolume.
You could, of course, start with just the busiest database and see if
it makes any difference. By default on Fedora, only /home and / are on
separate subvolumes. The general idea would be to use a subvolume
instead of a directory, e.g.
##shutdown the database
mv /var/lib/mysql /var/lib/mysqlold
btrfs sub create /var/lib/mysql
restorecon -Rv /var/lib/mysql
cp -r --reflink /var/lib/mysqlold/* /var/lib/mysql/
##resume database operation, and clean up mysqlold whenever
Another thing that might help, depending on the workload, is
space_cache=v2. You can safely switch to v2 via:
mount -o remount,clear_cache,space_cache=v2 /
This is a one time command. It will set a feature flag for space cache
v2, and it'll be used automatically on subsequent mounts. It may be
busy for a minute or so while the cache is rebuilt.
You might test these separately so you have an idea of their relative
effects on the workload.
$ sar -dp -s 21:00:00 -e 22:00:00 | egrep -e '(tps|sd[bcdef])'
21:00:01 DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util
21:01:00 sdb 6.21 0.09 101.08 0.00 16.30 0.03 5.52 0.75
21:01:00 sdc 4.15 0.03 10.81 0.00 2.61 0.03 7.05 0.56
21:01:00 sdd 3.95 0.50 9.30 0.00 2.48 0.03 8.18 0.52
21:01:00 sde 44.66 2.08 2178.70 0.00 48.83 8.17 183.46 2.49
21:01:00 sdf 47.07 0.44 2146.21 0.00 45.61 6.53 139.14 2.50
21:02:00 sdb 7.58 0.09 156.24 0.00 20.64 0.05 6.72 0.81
21:02:00 sdc 5.78 0.02 16.38 0.00 2.84 0.05 9.00 0.67
21:02:00 sdd 4.43 0.02 11.72 0.00 2.65 0.04 8.80 0.59
21:02:00 sde 67.12 0.02 2549.52 0.00 37.99 2.39 36.12 3.09
21:02:00 sdf 71.21 0.29 2753.11 0.00 38.66 7.49 105.73 3.19
21:03:00 sdb 6.78 0.02 207.26 0.00 30.59 0.03 5.40 0.58
21:03:00 sdc 2.41 0.02 5.79 0.00 2.41 0.02 7.97 0.33
21:03:00 sdd 2.45 0.09 5.93 0.00 2.46 0.03 11.99 0.32
21:03:00 sde 34.33 0.02 1554.25 0.00 45.27 7.60 221.78 2.11
21:03:00 sdf 36.41 0.02 1740.13 0.00 47.79 8.06 221.85 2.22
21:04:00 sdb 6.73 0.02 265.20 0.00 39.43 0.02 3.77 0.45
21:04:00 sdc 1.65 0.02 4.06 0.00 2.48 0.02 11.16 0.20
21:04:00 sdd 1.71 0.02 4.60 0.00 2.69 0.03 15.21 0.19
21:04:00 sde 39.88 1.16 1927.74 0.00 48.37 10.57 265.53 2.04
21:04:00 sdf 44.11 0.16 2056.74 0.00 46.64 2.24 51.22 2.15
21:05:00 sdb 2.58 0.05 49.74 0.00 19.30 0.02 8.17 0.24
21:05:00 sdc 1.10 0.05 2.20 0.00 2.05 0.01 11.00 0.15
21:05:00 sdd 1.23 0.05 3.46 0.00 2.85 0.04 31.08 0.15
21:05:00 sde 21.69 0.05 1086.47 0.00 50.09 4.37 202.00 1.23
21:05:00 sdf 18.03 0.05 1106.38 0.00 61.37 9.84 546.46 1.22
21:06:00 sdb 1.77 0.02 35.17 0.00 19.94 0.01 7.28 0.16
21:06:00 sdc 0.67 0.02 1.53 0.00 2.34 0.01 10.25 0.08
21:06:00 sdd 0.72 0.02 1.73 0.00 2.45 0.01 20.72 0.08
21:06:00 sde 23.98 1.42 1013.76 0.00 42.33 0.42 17.94 1.25
21:06:00 sdf 20.50 0.29 1055.99 0.00 51.53 15.91 776.58 1.24
21:07:00 sdb 3.66 0.02 91.71 0.00 25.04 0.02 6.30 0.32
21:07:00 sdc 1.35 0.29 3.33 0.00 2.69 0.01 8.98 0.18
21:07:00 sdd 1.43 0.09 4.13 0.00 2.95 0.01 10.57 0.17
21:07:00 sde 29.74 0.49 1274.66 0.00 42.88 3.63 122.68 1.49
21:07:00 sdf 26.39 0.29 1351.98 0.00 51.24 11.08 420.47 1.49
21:08:00 sdb 2.36 4.55 44.61 0.00 20.80 0.02 6.92 0.22
21:08:00 sdc 0.80 0.02 1.86 0.00 2.36 0.01 12.58 0.10
21:08:00 sdd 0.87 0.02 2.33 0.00 2.72 0.01 16.15 0.10
21:08:00 sde 23.97 1.16 1128.15 0.00 47.11 12.84 536.13 1.44
21:08:00 sdf 27.02 0.02 1164.18 0.00 43.09 9.12 338.15 1.52
21:09:00 sdb 6.16 0.02 178.38 0.00 28.96 0.03 5.61 0.50
21:09:00 sdc 2.16 0.02 5.06 0.00 2.35 0.03 13.26 0.28
21:09:00 sdd 2.33 0.09 6.66 0.00 2.90 0.03 11.49 0.28
21:09:00 sde 47.24 0.02 2285.96 0.00 48.39 5.08 108.10 2.19
21:09:00 sdf 46.24 0.29 2438.23 0.00 52.73 9.52 206.36 2.41
21:10:00 sdb 3.70 0.22 113.22 0.00 30.69 0.01 4.37 0.29
21:10:00 sdc 1.12 0.02 2.60 0.00 2.35 0.01 9.06 0.14
21:10:00 sdd 1.42 1.22 3.00 0.00 2.98 0.02 14.45 0.18
21:10:00 sde 29.44 31.86 1348.12 0.00 46.88 0.48 16.75 1.54
21:10:00 sdf 24.08 0.82 1416.18 0.00 58.86 13.47 559.88 1.41
21:11:00 sdb 2.83 0.02 38.36 0.00 13.56 0.02 6.88 0.30
21:11:00 sdc 2.65 0.02 8.92 0.00 3.38 0.02 8.69 0.20
21:11:00 sdd 1.50 0.09 3.66 0.00 2.51 0.02 11.19 0.18
21:11:00 sde 28.97 0.02 1456.10 0.00 50.27 8.24 284.85 1.63
21:11:00 sdf 33.59 0.02 1582.09 0.00 47.09 4.01 119.77 1.71
21:12:00 sdb 5.63 0.02 149.48 0.00 26.56 0.03 5.40 0.52
21:12:00 sdc 2.15 0.02 5.46 0.00 2.55 0.02 7.76 0.30
21:12:00 sdd 2.23 0.02 6.86 0.00 3.09 0.02 11.04 0.28
21:12:00 sde 45.41 0.02 2012.12 0.00 44.31 2.08 46.19 2.18
21:12:00 sdf 43.61 0.02 2108.91 0.00 48.35 12.94 297.30 2.23
21:13:00 sdb 5.71 0.02 82.92 0.00 14.52 0.03 5.87 0.65
21:13:00 sdc 3.51 0.02 7.93 0.00 2.26 0.03 7.89 0.48
21:13:00 sdd 3.60 0.02 8.66 0.00 2.41 0.03 9.21 0.47
21:13:00 sde 47.12 0.02 1945.85 0.00 41.30 1.17 25.40 2.20
21:13:00 sdf 46.29 0.02 2005.59 0.00 43.33 4.54 98.58 2.22
21:14:00 sdb 3.40 5.15 34.37 0.00 11.63 0.02 7.26 0.37
21:14:00 sdc 1.78 0.69 3.86 0.00 2.56 0.01 8.64 0.24
21:14:00 sdd 1.75 0.02 4.20 0.00 2.41 0.02 13.26 0.22
21:14:00 sde 31.34 12.35 1519.93 0.00 48.89 8.77 280.39 1.76
21:14:00 sdf 35.69 6.69 1559.10 0.00 43.88 3.03 85.51 1.71
21:15:00 sdb 6.26 0.06 129.45 0.00 20.69 0.03 5.82 0.58
21:15:00 sdc 3.16 0.06 7.19 0.00 2.29 0.03 9.32 0.42
21:15:00 sdd 3.18 0.06 7.46 0.00 2.36 0.06 19.19 0.40
21:15:00 sde 49.08 0.32 2347.86 0.00 47.85 6.09 124.60 2.56
21:15:00 sdf 51.61 0.46 2328.95 0.00 45.14 4.57 89.12 2.63
21:16:00 sdb 1.90 0.02 45.42 0.00 23.94 0.01 7.19 0.18
21:16:00 sdc 0.98 0.02 2.80 0.00 2.87 0.01 10.24 0.12
21:16:00 sdd 1.07 0.29 2.93 0.00 3.02 0.01 12.55 0.12
21:16:00 sde 16.55 11.88 1008.52 0.00 61.66 15.59 942.31 1.14
21:16:00 sdf 21.99 0.69 1031.37 0.00 46.92 0.27 12.55 1.12
21:17:00 sdb 2.06 0.02 37.44 0.00 18.14 0.01 6.96 0.21
21:17:00 sdc 1.08 0.02 2.73 0.00 2.55 0.01 9.58 0.14
21:17:00 sdd 1.27 0.22 4.00 0.00 3.34 0.02 12.63 0.14
21:17:00 sde 19.22 8.42 1329.36 0.00 69.61 13.69 712.74 1.37
21:17:00 sdf 26.73 2.49 1306.71 0.00 48.98 0.61 23.24 1.34
21:18:01 sdb 6.48 0.02 104.81 0.00 16.19 0.04 7.32 0.75
21:18:01 sdc 4.01 0.02 9.32 0.00 2.33 0.03 7.99 0.56
21:18:01 sdd 4.08 0.02 9.99 0.00 2.46 0.03 7.30 0.53
21:18:01 sde 58.90 1.09 2357.38 0.00 40.04 3.71 63.55 2.75
21:18:01 sdf 59.96 0.16 2372.57 0.00 39.57 1.34 22.83 2.78
21:19:01 sdb 2.30 0.02 20.92 0.00 9.11 0.02 7.54 0.24
21:19:01 sdc 1.37 0.02 3.33 0.00 2.46 0.01 8.41 0.17
21:19:01 sdd 1.42 0.02 3.93 0.00 2.79 0.02 11.29 0.16
21:19:01 sde 31.59 0.02 1686.53 0.00 53.39 7.34 232.92 1.78
21:19:01 sdf 33.12 0.02 1610.12 0.00 48.61 9.48 286.80 1.86
21:20:01 sdb 2.61 0.29 36.16 0.00 13.95 0.01 5.72 0.27
21:20:01 sdc 1.51 0.02 3.66 0.00 2.43 0.02 12.71 0.20
21:20:01 sdd 1.55 0.02 4.20 0.00 2.73 0.02 10.18 0.19
21:20:01 sde 34.09 0.02 1615.58 0.00 47.39 8.43 247.65 1.90
21:20:01 sdf 33.79 0.02 1670.18 0.00 49.42 10.99 325.83 1.88
21:21:01 sdb 2.88 1.29 17.98 0.00 6.69 0.02 6.71 0.35
21:21:01 sdc 2.95 0.02 9.59 0.00 3.26 0.01 5.07 0.26
21:21:01 sdd 1.76 0.02 4.73 0.00 2.69 0.02 9.55 0.22
21:21:01 sde 35.70 8.75 1959.24 0.00 55.13 11.88 333.36 2.13
21:21:01 sdf 43.74 48.91 1973.09 0.00 46.23 3.60 82.75 2.47
21:22:01 sdb 3.10 0.02 16.92 0.00 5.47 0.01 5.12 0.39
21:22:01 sdc 2.18 0.02 5.53 0.00 2.55 0.02 8.05 0.30
21:22:01 sdd 2.23 0.02 5.93 0.00 2.67 0.02 8.52 0.28
21:22:01 sde 33.57 0.02 1472.57 0.00 43.86 3.44 102.90 2.03
21:22:01 sdf 32.07 0.02 1476.04 0.00 46.02 9.75 304.46 1.90
21:23:01 sdb 2.93 0.02 18.58 0.00 6.35 0.02 6.43 0.33
21:23:01 sdc 1.98 0.02 4.73 0.00 2.40 0.02 12.38 0.27
21:23:01 sdd 2.08 0.09 5.46 0.00 2.67 0.02 10.12 0.25
21:23:01 sde 42.64 0.02 1941.59 0.00 45.53 5.33 125.49 2.17
21:23:01 sdf 41.56 0.02 1942.99 0.00 46.75 10.29 248.03 2.09
21:24:01 sdb 3.95 0.09 19.21 0.00 4.89 0.03 6.79 0.50
21:24:01 sdc 3.15 0.03 7.23 0.00 2.31 0.03 9.10 0.45
21:24:01 sdd 3.21 0.03 7.97 0.00 2.49 0.03 9.65 0.43
21:24:01 sde 43.18 0.03 1820.85 0.00 42.17 5.30 123.18 2.40
21:24:01 sdf 43.68 0.03 1826.74 0.00 41.82 6.16 141.60 2.26
21:25:01 sdb 3.28 0.98 15.64 0.00 5.07 0.02 7.61 0.41
21:25:01 sdc 2.48 0.05 5.57 0.00 2.26 0.02 8.09 0.35
21:25:01 sdd 2.67 0.05 7.29 0.00 2.75 0.05 17.55 0.34
21:25:01 sde 36.86 7.07 1739.47 0.00 47.39 5.13 139.82 1.94
21:25:01 sdf 38.50 3.69 1741.46 0.00 45.33 6.78 176.60 1.92
21:26:01 sdb 0.92 0.09 3.00 0.00 3.37 0.01 9.91 0.11
21:26:01 sdc 0.75 0.02 1.80 0.00 2.43 0.01 11.84 0.10
21:26:01 sdd 0.78 0.02 2.00 0.00 2.59 0.01 14.91 0.09
21:26:01 sde 12.74 0.02 876.46 0.00 68.81 11.13 874.11 0.95
21:26:01 sdf 13.54 0.02 939.79 0.00 69.43 12.19 901.14 0.95
21:27:01 sdb 1.57 0.02 9.99 0.00 6.40 0.01 7.76 0.18
21:27:01 sdc 1.10 0.02 2.86 0.00 2.63 0.01 9.20 0.15
21:27:01 sdd 1.17 0.02 3.33 0.00 2.88 0.01 11.43 0.14
21:27:01 sde 23.73 0.02 1364.46 0.00 57.50 3.53 149.12 1.39
21:27:01 sdf 21.90 0.02 1317.10 0.00 60.15 11.87 542.60 1.31
21:28:01 sdb 1.72 0.03 8.41 0.00 4.91 0.01 7.81 0.23
21:28:01 sdc 1.40 0.03 3.27 0.00 2.35 0.01 9.50 0.20
21:28:01 sdd 1.47 0.03 3.74 0.00 2.56 0.02 13.36 0.19
21:28:01 sde 26.21 0.03 1266.63 0.00 48.33 3.38 129.37 1.39
21:28:01 sdf 24.12 0.03 1244.93 0.00 51.62 5.84 242.51 1.39
21:29:01 sdb 5.42 0.03 26.34 0.00 4.87 0.03 6.85 0.68
21:29:01 sdc 4.36 0.03 10.03 0.00 2.30 0.04 8.70 0.62
21:29:01 sdd 4.56 0.09 12.17 0.00 2.69 0.04 8.22 0.59
21:29:01 sde 60.35 0.09 2264.75 0.00 37.53 2.76 46.25 2.90
21:29:01 sdf 61.66 0.03 2203.58 0.00 35.74 3.33 54.54 2.81
21:30:01 sdb 4.32 0.02 25.15 0.00 5.83 0.03 7.85 0.55
21:30:01 sdc 3.31 0.02 7.54 0.00 2.29 0.03 10.39 0.48
21:30:01 sdd 3.36 0.02 7.94 0.00 2.37 0.02 7.23 0.44
21:30:01 sde 51.75 0.02 2065.78 0.00 39.92 0.96 19.05 2.53
21:30:01 sdf 50.07 0.55 2089.21 0.00 41.74 5.24 105.19 2.51
21:31:01 sdb 4.20 0.02 20.85 0.00 4.97 0.03 6.72 0.54
21:31:01 sdc 4.43 0.02 13.19 0.00 2.98 0.02 5.36 0.47
21:31:01 sdd 3.55 1.09 8.66 0.00 2.75 0.04 10.48 0.45
21:31:01 sde 44.56 1.89 2270.60 0.00 51.00 7.86 176.80 2.61
21:31:01 sdf 46.72 0.22 2325.54 0.00 49.78 11.52 247.11 2.62
21:32:01 sdb 5.24 0.03 24.13 0.00 4.61 0.05 9.07 0.66
21:32:01 sdc 4.41 0.03 10.72 0.00 2.44 0.04 9.17 0.62
21:32:01 sdd 4.51 0.03 12.06 0.00 2.68 0.03 7.95 0.57
21:32:01 sde 58.13 0.03 2335.79 0.00 40.18 2.63 45.75 2.89
21:32:01 sdf 58.45 0.03 2340.62 0.00 40.05 4.05 69.71 2.76
21:33:01 sdb 2.36 0.02 11.41 0.00 4.86 0.02 8.26 0.31
21:33:01 sdc 1.92 0.02 4.45 0.00 2.32 0.02 9.39 0.26
21:33:01 sdd 1.99 0.02 4.91 0.00 2.48 0.02 10.93 0.25
21:33:01 sde 28.98 0.02 1349.54 0.00 46.57 5.67 196.25 1.61
21:33:01 sdf 28.58 0.02 1316.44 0.00 46.06 2.29 80.75 1.53
21:34:01 sdb 2.50 0.03 12.89 0.00 5.17 0.02 7.48 0.29
21:34:01 sdc 1.97 0.03 4.66 0.00 2.37 0.02 10.15 0.27
21:34:01 sdd 2.01 0.03 5.19 0.00 2.60 0.02 11.58 0.25
21:34:01 sde 29.60 0.03 1298.84 0.00 43.88 1.46 49.83 1.65
21:34:01 sdf 28.88 0.03 1338.51 0.00 46.36 8.35 289.83 1.64
21:35:01 sdb 1.56 0.05 6.89 0.00 4.46 0.01 8.19 0.18
21:35:01 sdc 1.27 0.05 2.85 0.00 2.27 0.01 10.86 0.16
21:35:01 sdd 1.29 0.05 3.11 0.00 2.45 0.03 21.23 0.15
21:35:01 sde 23.72 0.05 1340.62 0.00 56.53 10.50 443.37 1.43
21:35:01 sdf 28.02 0.05 1343.13 0.00 47.94 4.09 146.62 1.40
21:36:01 sdb 1.55 17.62 6.53 0.00 15.59 0.01 6.68 0.15
21:36:01 sdc 1.40 17.88 1.53 0.00 13.88 0.01 7.00 0.12
21:36:01 sdd 1.38 17.62 1.53 0.00 13.85 0.02 11.42 0.11
21:36:01 sde 18.27 17.62 902.88 0.00 50.37 12.86 704.01 1.31
21:36:01 sdf 17.76 17.62 940.53 0.00 53.96 13.46 758.66 1.30
21:37:01 sdb 8.10 0.09 144.64 0.00 17.88 0.04 5.59 0.86
21:37:01 sdc 31.19 0.03 517.64 0.00 16.60 0.75 24.56 2.07
21:37:01 sdd 31.07 0.03 518.31 0.00 16.68 0.10 3.74 2.13
21:37:01 sde 102.70 0.29 3837.35 0.00 37.37 2.17 21.66 4.14
21:37:01 sdf 100.55 0.29 3683.64 0.00 36.64 1.11 11.53 4.08
21:38:00 sdb 5.93 0.03 26.00 0.00 4.39 0.04 6.97 0.77
21:38:00 sdc 11.69 0.03 139.47 0.00 11.93 0.12 10.68 0.93
21:38:00 sdd 11.71 0.03 140.14 0.00 11.97 0.06 5.82 0.89
21:38:00 sde 95.35 0.23 3413.91 0.00 35.81 0.93 10.23 3.89
21:38:00 sdf 96.39 0.03 3417.87 0.00 35.46 1.06 11.51 3.85
21:39:00 sdb 51.18 0.02 847.68 0.00 16.56 0.59 11.94 2.62
21:39:00 sdc 5.56 0.02 12.78 0.00 2.30 0.04 7.99 0.78
21:39:00 sdd 5.75 0.09 14.64 0.00 2.56 0.04 6.42 0.75
21:39:00 sde 68.13 0.02 3002.38 0.00 44.07 0.50 7.86 3.25
21:39:00 sdf 111.67 0.02 3827.48 0.00 34.27 2.57 23.51 4.65
21:40:01 sdb 74.28 0.09 1347.79 0.00 18.15 1.08 14.98 3.73
21:40:01 sdc 5.86 0.02 13.57 0.00 2.32 0.04 7.91 0.81
21:40:01 sdd 5.96 0.02 14.43 0.00 2.43 0.04 7.85 0.79
21:40:01 sde 63.27 0.02 3366.73 0.00 53.21 0.70 11.50 3.31
21:40:01 sdf 131.17 0.29 4649.23 0.00 35.45 1.36 10.83 5.69
21:41:01 sdb 41.49 0.09 754.77 0.00 18.19 0.69 17.12 2.51
21:41:01 sdc 8.13 0.03 21.25 0.00 2.61 0.05 6.34 1.00
21:41:01 sdd 7.00 0.03 16.70 0.00 2.39 0.04 6.68 0.92
21:41:01 sde 69.88 0.29 3200.33 0.00 45.80 1.30 19.15 3.80
21:41:01 sdf 99.06 0.56 3681.11 0.00 37.16 1.31 13.73 4.74
21:42:00 sdb 6.65 0.03 31.28 0.00 4.71 0.04 5.82 0.86
21:42:00 sdc 5.41 0.03 12.86 0.00 2.38 0.04 7.27 0.77
21:42:00 sdd 5.58 0.03 14.54 0.00 2.61 0.04 6.61 0.73
21:42:00 sde 74.96 0.03 2923.73 0.00 39.00 1.30 17.85 4.10
21:42:00 sdf 76.60 0.03 2924.33 0.00 38.18 1.47 19.71 4.01
21:43:01 sdb 6.14 0.02 25.17 0.00 4.10 0.04 6.78 0.84
21:43:01 sdc 5.23 0.02 12.02 0.00 2.30 0.04 7.64 0.74
21:43:01 sdd 5.32 0.02 12.75 0.00 2.40 0.03 6.62 0.71
21:43:01 sde 68.64 0.02 2657.53 0.00 38.72 1.27 19.07 3.61
21:43:01 sdf 68.11 0.02 2662.81 0.00 39.09 1.12 16.90 3.60
21:44:00 sdb 6.30 0.03 33.27 0.00 5.29 0.04 7.21 0.82
21:44:00 sdc 5.08 0.03 11.52 0.00 2.27 0.04 9.01 0.71
21:44:00 sdd 5.12 0.03 12.05 0.00 2.36 0.03 6.54 0.69
21:44:00 sde 68.80 0.03 2527.21 0.00 36.73 0.89 13.45 3.71
21:44:00 sdf 69.18 0.03 2542.76 0.00 36.76 1.96 28.81 3.66
21:45:01 sdb 6.40 0.06 25.49 0.00 3.99 0.04 7.03 0.85
21:45:01 sdc 5.55 0.06 12.41 0.00 2.25 0.04 8.07 0.77
21:45:01 sdd 5.60 0.32 12.94 0.00 2.37 0.07 12.99 0.72
21:45:01 sde 72.05 0.06 2806.21 0.00 38.95 0.87 12.63 3.51
21:45:01 sdf 72.91 0.32 2812.28 0.00 38.58 0.91 12.96 3.55
21:46:01 sdb 8.73 0.03 117.82 0.00 13.50 0.05 6.13 1.07
21:46:01 sdc 6.55 0.03 15.85 0.00 2.42 0.05 7.44 0.89
21:46:01 sdd 6.97 1.57 16.19 0.00 2.55 0.05 7.69 0.89
21:46:01 sde 78.81 2.71 3419.65 0.00 43.43 1.97 25.52 4.95
21:46:01 sdf 78.27 1.10 3444.50 0.00 44.02 2.33 30.33 4.72
21:47:00 sdb 5.92 0.03 31.38 0.00 5.31 0.03 6.25 0.76
21:47:00 sdc 4.58 0.03 10.46 0.00 2.29 0.03 7.63 0.65
21:47:00 sdd 4.68 0.09 11.26 0.00 2.43 0.03 6.77 0.62
21:47:00 sde 69.63 0.03 2466.38 0.00 35.42 1.98 28.94 4.25
21:47:00 sdf 69.26 0.03 2468.45 0.00 35.64 1.90 27.87 4.15
21:48:01 sdb 7.07 0.02 29.71 0.00 4.21 0.05 6.80 0.95
21:48:01 sdc 5.79 0.02 13.09 0.00 2.26 0.04 7.82 0.81
21:48:01 sdd 5.82 0.02 13.61 0.00 2.34 0.04 6.46 0.78
21:48:01 sde 76.31 0.02 2922.56 0.00 38.30 2.03 27.05 4.66
21:48:01 sdf 78.32 0.02 2932.11 0.00 37.44 2.39 31.06 4.81
21:49:01 sdb 6.45 0.02 26.88 0.00 4.17 0.04 6.20 0.86
21:49:01 sdc 5.29 0.02 11.84 0.00 2.24 0.04 7.35 0.75
21:49:01 sdd 5.32 0.02 12.38 0.00 2.33 0.03 6.19 0.72
21:49:01 sde 68.41 0.02 2402.66 0.00 35.12 2.41 35.66 4.01
21:49:01 sdf 70.06 0.02 2408.98 0.00 34.38 1.49 21.75 4.44
21:50:01 sdb 4.68 0.03 18.72 0.00 4.01 0.03 6.66 0.62
21:50:01 sdc 4.05 0.03 9.19 0.00 2.27 0.03 8.19 0.55
21:50:01 sdd 4.09 0.03 9.73 0.00 2.39 0.03 7.16 0.54
21:50:01 sde 62.16 0.03 2202.13 0.00 35.43 1.04 17.15 3.20
21:50:01 sdf 60.03 0.03 2205.58 0.00 36.74 3.44 57.78 3.12
21:51:01 sdb 7.19 0.03 29.03 0.00 4.04 0.04 6.20 0.97
21:51:01 sdc 6.96 0.03 18.55 0.00 2.67 0.04 5.75 0.82
21:51:01 sdd 5.81 0.03 13.95 0.00 2.41 0.04 6.64 0.76
21:51:01 sde 79.19 0.03 2975.14 0.00 37.57 2.01 25.83 4.79
21:51:01 sdf 80.73 0.03 2988.55 0.00 37.02 3.07 38.59 5.02
21:52:01 sdb 6.89 0.03 65.27 0.00 9.48 0.04 5.58 0.83
21:52:01 sdc 4.72 0.03 11.14 0.00 2.37 0.04 7.92 0.67
21:52:01 sdd 5.06 0.09 21.29 0.00 4.23 0.04 7.57 0.63
21:52:01 sde 76.41 0.03 2972.24 0.00 38.90 3.84 50.79 5.18
21:52:01 sdf 77.69 0.03 3009.41 0.00 38.73 3.57 46.45 5.20
21:53:01 sdb 6.07 0.02 31.33 0.00 5.17 0.04 6.21 0.77
21:53:01 sdc 4.64 0.02 10.58 0.00 2.28 0.03 7.08 0.65
21:53:01 sdd 5.02 0.02 16.36 0.00 3.26 0.04 8.33 0.65
21:53:01 sde 68.11 0.02 2519.26 0.00 36.99 1.33 20.07 3.90
21:53:01 sdf 68.61 0.02 2523.71 0.00 36.79 2.41 35.69 3.84
21:54:01 sdb 6.29 0.02 29.63 0.00 4.71 0.04 6.08 0.80
21:54:01 sdc 5.08 0.02 11.72 0.00 2.31 0.04 8.11 0.70
21:54:01 sdd 5.26 0.02 14.18 0.00 2.70 0.04 7.26 0.68
21:54:01 sde 69.39 0.02 2613.95 0.00 37.67 1.29 19.08 3.70
21:54:01 sdf 67.85 0.02 2618.41 0.00 38.59 1.56 23.50 3.67
21:55:00 sdb 7.03 0.05 30.49 0.00 4.34 0.06 9.19 0.91
21:55:00 sdc 5.81 0.05 13.36 0.00 2.31 0.05 9.83 0.82
21:55:00 sdd 6.09 0.05 16.59 0.00 2.73 0.07 11.96 0.78
21:55:00 sde 66.03 0.12 2727.33 0.00 41.30 0.99 15.43 3.53
21:55:00 sdf 66.49 0.05 2731.70 0.00 41.09 1.24 19.20 3.52
21:56:01 sdb 5.13 0.02 21.36 0.00 4.17 0.03 7.05 0.66
21:56:01 sdc 4.65 0.02 12.07 0.00 2.60 0.03 7.43 0.58
21:56:01 sdd 4.19 0.02 10.09 0.00 2.42 0.04 8.84 0.56
21:56:01 sde 48.02 0.02 2200.92 0.00 45.83 4.83 101.11 3.05
21:56:01 sdf 50.43 0.02 2208.57 0.00 43.80 5.34 106.47 3.13
21:57:01 sdb 5.87 0.02 24.63 0.00 4.20 0.04 7.44 0.76
21:57:01 sdc 4.98 0.02 11.65 0.00 2.35 0.04 8.58 0.70
21:57:01 sdd 5.09 0.02 12.45 0.00 2.45 0.03 7.14 0.67
21:57:01 sde 59.42 0.56 2439.73 0.00 41.07 2.49 42.36 3.10
21:57:01 sdf 61.40 0.02 2446.59 0.00 39.85 0.79 13.36 3.18
21:58:01 sdb 6.36 0.03 30.14 0.00 4.74 0.04 6.25 0.85
21:58:01 sdc 5.30 0.03 12.04 0.00 2.28 0.04 7.80 0.74
21:58:01 sdd 5.40 0.03 12.85 0.00 2.38 0.04 6.80 0.72
21:58:01 sde 58.39 0.29 2092.45 0.00 35.84 0.56 10.04 2.83
21:58:01 sdf 58.92 0.03 2102.81 0.00 35.69 0.60 10.75 2.87
21:59:00 sdb 7.47 0.03 61.00 0.00 8.16 0.04 6.25 0.96
21:59:00 sdc 5.68 0.03 12.87 0.00 2.27 0.04 7.30 0.81
21:59:00 sdd 5.70 0.03 13.41 0.00 2.36 0.04 7.63 0.76
21:59:00 sde 63.15 0.29 2417.43 0.00 38.29 0.68 11.27 3.29
21:59:00 sdf 64.77 0.03 2459.33 0.00 37.97 1.16 18.47 3.38
Average: sdb 7.33 0.54 101.03 0.00 13.86 0.07 9.65 0.68
Average: sdc 4.07 0.35 19.11 0.00 4.78 0.04 10.41 0.50
Average: sdd 4.02 0.42 19.50 0.00 4.95 0.03 8.29 0.48
Average: sde 49.50 1.92 2101.35 0.00 42.49 4.44 90.20 2.66
Average: sdf 52.62 1.49 2167.92 0.00 41.22 5.22 99.74 2.76
