Hi list!I have btrfs pool consisting of 5x 2,72 TiB LUKS (dm-crypt) partitions in RAID1, mounted on Linux 4.4 with btrfs-progs 4.4. I never had any crashes or power loss here, but recently about every 60 - 120 minutes (while in use) btrfs detects corruptions, aborts the transaction and drops to read-only mode. btrfs still mounts normally without any special options (it does take about 60 seconds, which I guess is normal for this kind of size). All LUKS partitions have at least 400GiB of free space.
I don't see any HW problems here; I doubt there is a corruption coming from the LUKS partition. I did test the RAM but it seems fine in multiple memtest86+ amd memtest86 runs.
Are there any known bugs in 4.4? Any suggestions would be greatly appreciated!
I have to admit I did not regularly scrub. Thanks, Lukas --- ~# uname -aLinux srv1-dom0 4.4.0-63-generic #84-Ubuntu SMP Wed Feb 1 17:20:32 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
~# btrfs --version
btrfs-progs v4.4
~# btrfs fi show
Label: 'dom0-os' uuid: e475636c-21e0-4563-87d6-91f03c519a62
Total devices 5 FS bytes used 3.52GiB
devid 1 size 10.00GiB used 3.53GiB path /dev/sda2
devid 2 size 10.00GiB used 4.25GiB path /dev/sdb2
devid 3 size 10.00GiB used 3.28GiB path /dev/sdc2
devid 4 size 10.00GiB used 4.00GiB path /dev/sdd2
devid 5 size 10.00GiB used 4.00GiB path /dev/sde2
Label: 'storage_pool' uuid: f50f980e-7640-49c7-bf8d-20d55cfe6005
Total devices 5 FS bytes used 5.77TiB
devid 1 size 2.72TiB used 2.31TiB path /dev/mapper/sda3_crypt
devid 2 size 2.72TiB used 2.31TiB path /dev/mapper/sdb3_crypt
devid 3 size 2.72TiB used 2.31TiB path /dev/mapper/sdc3_crypt
devid 4 size 2.72TiB used 2.31TiB path /dev/mapper/sdd3_crypt
devid 5 size 2.72TiB used 2.31TiB path /dev/mapper/sde3_crypt
~# btrfs fi df /storage/users/
Data, RAID1: total=5.77TiB, used=5.76TiB
System, RAID1: total=32.00MiB, used=832.00KiB
Metadata, RAID1: total=8.00GiB, used=6.96GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
~#
~#
partial dmesg:
[ 1509.033492] BTRFS: device label storage_pool devid 1 transid 238135
/dev/dm-5
[ 1510.498804] BTRFS: device label storage_pool devid 2 transid 238135
/dev/dm-6
[ 1511.980968] BTRFS: device label storage_pool devid 3 transid 238135
/dev/dm-7
[ 1513.461799] BTRFS: device label storage_pool devid 4 transid 238135
/dev/dm-8
[ 1514.838757] BTRFS: device label storage_pool devid 5 transid 238135
/dev/dm-9
[ 1517.726471] BTRFS info (device dm-9): btrfs: use no compression [ 1517.726477] BTRFS info (device dm-9): disk space caching is enabled [ 1517.726479] BTRFS: has skinny extents [ 1569.598633] BTRFS: checking UUID tree[ 3540.825747] BTRFS critical (device dm-9): corrupt leaf, slot offset bad: block=5242107641856,root=1, slot=39 [ 3540.836168] BTRFS critical (device dm-9): corrupt leaf, slot offset bad: block=5242107641856,root=1, slot=39
[ 3540.846413] ------------[ cut here ]------------[ 3540.846432] WARNING: CPU: 2 PID: 2757 at /build/linux-mPTI9s/linux-4.4.0/fs/btrfs/extent-tree.c:2930 btrfs_run_delayed_refs+0x26b/0x2a0 [btrfs]()
[ 3540.846433] BTRFS: Transaction aborted (error -5)[ 3540.846434] Modules linked in: algif_skcipher af_alg xen_gntdev xen_evtchn xenfs xen_privcmd drbg ansi_cprng dm_crypt nls_iso8859_1 bridge stp llc intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel serio_raw joydev input_leds nuvoton_cir 8250_fintek ie31200_edac mac_hid rc_core lpc_ich edac_core shpchp mei_me mei ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid mxm_wmi i915 i2c_algo_bit drm_kms_helper aesni_intel aes_x86_64 glue_helper syscopyarea sysfillrect firewire_ohci sysimgblt firewire_core fb_sys_fops lrw psmouse [ 3540.846466] tg3 gf128mul ablk_helper cryptd crc_itu_t ptp ahci drm pps_core libahci fjes wmi video [ 3540.846473] CPU: 2 PID: 2757 Comm: btrfs-transacti Not tainted 4.4.0-63-generic #84-Ubuntu [ 3540.846475] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z77 Extreme6, BIOS P2.80 07/01/2013 [ 3540.846476] 0000000000000200 0000000002709bc3 ffff88007615fc90 ffffffff813f8083 [ 3540.846478] ffff88007615fcd8 ffffffffc048d498 ffff88007615fcc8 ffffffff810812d2 [ 3540.846479] ffff8802adf562f8 ffff8802a9c71800 ffff8800056caef0 ffffffffffffffff
[ 3540.846481] Call Trace: [ 3540.846486] [<ffffffff813f8083>] dump_stack+0x63/0x90 [ 3540.846489] [<ffffffff810812d2>] warn_slowpath_common+0x82/0xc0 [ 3540.846491] [<ffffffff8108136c>] warn_slowpath_fmt+0x5c/0x80[ 3540.846500] [<ffffffffc03f16cd>] ? __btrfs_run_delayed_refs+0xcdd/0x1220 [btrfs] [ 3540.846509] [<ffffffffc03f4cab>] btrfs_run_delayed_refs+0x26b/0x2a0 [btrfs] [ 3540.846520] [<ffffffffc04837b2>] commit_cowonly_roots+0x22b/0x2c2 [btrfs] [ 3540.846530] [<ffffffffc040a1b6>] btrfs_commit_transaction+0x576/0xa90 [btrfs]
[ 3540.846533] [<ffffffff810c41e0>] ? wake_atomic_t_function+0x60/0x60 [ 3540.846542] [<ffffffffc04052e9>] transaction_kthread+0x229/0x240 [btrfs][ 3540.846558] [<ffffffffc04050c0>] ? btrfs_cleanup_transaction+0x570/0x570 [btrfs]
[ 3540.846560] [<ffffffff810a0ba8>] kthread+0xd8/0xf0 [ 3540.846562] [<ffffffff810a0ad0>] ? kthread_create_on_node+0x1e0/0x1e0 [ 3540.846564] [<ffffffff8183c98f>] ret_from_fork+0x3f/0x70 [ 3540.846566] [<ffffffff810a0ad0>] ? kthread_create_on_node+0x1e0/0x1e0 [ 3540.846567] ---[ end trace 70830ce6f0e320dd ]---[ 3540.846587] BTRFS: error (device dm-9) in btrfs_run_delayed_refs:2930: errno=-5 IO failure
[ 3540.855086] BTRFS info (device dm-9): forced readonly[ 3540.855088] BTRFS warning (device dm-9): Skipping commit of aborted transaction. [ 3540.855090] BTRFS: error (device dm-9) in cleanup_transaction:1746: errno=-5 IO failure
full dmesg attached and online at: http://pastebin.com/raw/K8FNNEnS
Attachment:
dmesg-4.4.0-59-generic-btrfs-trace2.gz
Description: application/gzip
Attachment:
dmesg-4.4.0-47-generic-btrfs-trace3.gz
Description: application/gzip
Attachment:
dmesg-4.4.0-63-generic-btrfs-trace4.gz
Description: application/gzip
