I updated kernels recently and now am getting a corrupt leaf error.
The drives decrypt and mount, and I can touch a file briefly until the
mount switches over to read-only mode. Extended SMART tests show all 6
of my drives have a healthy status. I have a backup of the data. The
array is configured as RAID10. As the BTRFS filesystem remains
accessible / read-only, I am able to take an additional backup. What
is the best way to recover from this error?
layout: proxmox with direct /dev passthrough to VMs
$ uname -a
VM: Linux server0 5.5.6-arch1-1 #1 SMP PREEMPT Mon, 24 Feb 2020 12:20:16 +0000 x86_64 GNU/Linux
proxmox: Linux pxe 4.15.18-26-pve #1 SMP PVE 4.15.18-54 (Sat, 15 Feb 2020 15:34:24 +0100) x86_64 GNU/Linux
$ btrfs --version (VM)
btrfs-progs v5.4
$ btrfs fi show
Label: none uuid: 8c1dea88-fa40-4e6e-a1a1-214ea6bcdb00
Total devices 6 FS bytes used 2.88TiB
devid 1 size 2.73TiB used 1.02TiB path /dev/mapper/luks0
devid 2 size 2.73TiB used 1.02TiB path /dev/mapper/luks1
devid 3 size 2.73TiB used 1.02TiB path /dev/mapper/luks2
devid 4 size 2.73TiB used 1.02TiB path /dev/mapper/luks3
devid 5 size 2.73TiB used 1.02TiB path /dev/mapper/luks4
devid 6 size 2.73TiB used 1.02TiB path /dev/mapper/luks5
$ btrfs fi df /mnt/raid
Data, RAID10: total=3.05TiB, used=2.87TiB
System, RAID10: total=103.88MiB, used=320.00KiB
Metadata, RAID10: total=6.09GiB, used=4.46GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
$ dmesg | grep BTRFS
[ 19.060581] BTRFS: device fsid 8c1dea88-fa40-4e6e-a1a1-214ea6bcdb00 devid 5 transid 361687 /dev/dm-5 scanned by systemd-udevd (553)
[ 19.061232] BTRFS: device fsid 8c1dea88-fa40-4e6e-a1a1-214ea6bcdb00 devid 1 transid 361687 /dev/dm-0 scanned by systemd-udevd (526)
[ 19.062756] BTRFS: device fsid 8c1dea88-fa40-4e6e-a1a1-214ea6bcdb00 devid 2 transid 361687 /dev/dm-3 scanned by systemd-udevd (538)
[ 19.063265] BTRFS: device fsid 8c1dea88-fa40-4e6e-a1a1-214ea6bcdb00 devid 4 transid 361687 /dev/dm-2 scanned by systemd-udevd (545)
[ 19.071525] BTRFS: device fsid 8c1dea88-fa40-4e6e-a1a1-214ea6bcdb00 devid 6 transid 361687 /dev/dm-1 scanned by systemd-udevd (557)
[ 19.073708] BTRFS: device fsid 8c1dea88-fa40-4e6e-a1a1-214ea6bcdb00 devid 3 transid 361687 /dev/dm-4 scanned by systemd-udevd (533)
[ 19.190159] BTRFS info (device dm-0): enabling auto defrag
[ 19.190172] BTRFS info (device dm-0): disk space caching is enabled
[ 19.190174] BTRFS info (device dm-0): has skinny extents
[ 19.448971] BTRFS info (device dm-0): bdev /dev/mapper/luks0 errs: wr 13790, rd 387, flush 0, corrupt 3532, gen 578
[ 19.448977] BTRFS info (device dm-0): bdev /dev/mapper/luks5 errs: wr 13673, rd 207, flush 0, corrupt 3540, gen 705
[ 130.172956] BTRFS info (device dm-0): the free space cache file (9692905472) is invalid, skip it
[ 130.206490] BTRFS info (device dm-0): the free space cache file (32241483776) is invalid, skip it
[ 130.221862] BTRFS info (device dm-0): the free space cache file (38683934720) is invalid, skip it
[ 130.254926] BTRFS info (device dm-0): the free space cache file (54790062080) is invalid, skip it
[ 130.256586] BTRFS info (device dm-0): the free space cache file (58011287552) is invalid, skip it
[ 130.261085] BTRFS info (device dm-0): the free space cache file (61232513024) is invalid, skip it
[ 130.261771] BTRFS info (device dm-0): the free space cache file (67674963968) is invalid, skip it
[ 130.395696] BTRFS critical (device dm-0): corrupt leaf: root=7 block=2533706842112 slot=5, csum end range (68761223168) goes beyond the start range (68761178112) of the next csum item
[ 130.395829] BTRFS error (device dm-0): block=2533706842112 read time tree block corruption detected
[ 130.406624] BTRFS critical (device dm-0): corrupt leaf: root=7 block=2533706842112 slot=5, csum end range (68761223168) goes beyond the start range (68761178112) of the next csum item
[ 130.406803] BTRFS error (device dm-0): block=2533706842112 read time tree block corruption detected
[ 130.412343] BTRFS critical (device dm-0): corrupt leaf: root=7 block=2533706842112 slot=5, csum end range (68761223168) goes beyond the start range (68761178112) of the next csum item
[ 130.412526] BTRFS error (device dm-0): block=2533706842112 read time tree block corruption detected
[ 130.414847] BTRFS critical (device dm-0): corrupt leaf: root=7 block=2533706842112 slot=5, csum end range (68761223168) goes beyond the start range (68761178112) of the next csum item
[ 130.415056] BTRFS error (device dm-0): block=2533706842112 read time tree block corruption detected
