Hello Fellow BTRFS users.
I have run into the bad key order issue.
corrupt leaf: root=1 block=57567265079296 slot=83, bad key order, prev
(18446744073709551605 0 57707594776576) current (18446726481523507189
0 57709742260224)
The lines repeats over and over..
I read a thread between Hugo Mills and Eric Wolf about a similar issue
and i have gathered the same info.
I understand that it probably is hardware related, i have been running
memtest for 60h+ to see if i could reproduce it.
I also tried to run btrfs check --recover but it did not help.
My questions is if it can be fixed? It looks like that all surrounding
blocks are empty space nodes, so could i reset the empty space using
mount option clear_cache?
If i have to copy the files out, can i then use btrfs send-recive or
should i copy the files using the ordinary cp command?
I hope you can help me, thanks in advance. Below is log snippets etc
that i hope can help with the debugging.
Failing OS: Ubuntu 16.04.5 LTS with the latest updates.
The rest is done from a liveOS Ubuntu 18.10.
Details:
ubuntu@ubuntu:/isodevice$ sudo uname -a
Linux ubuntu 4.18.0-10-generic #11-Ubuntu SMP Thu Oct 11 15:13:55 UTC
2018 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@ubuntu:/isodevice$ sudo btrfs --version
btrfs-progs v4.16.1
ubuntu@ubuntu:/isodevice$ sudo btrfs fi show
Label: none uuid: 90900a21-8a71-4301-b5c2-21dea31f1132
Total devices 4 FS bytes used 6.61TiB
devid 1 size 2.72TiB used 2.30TiB path /dev/sdc4
devid 2 size 2.72TiB used 2.30TiB path /dev/sde4
devid 3 size 2.72TiB used 2.30TiB path /dev/sda4
devid 4 size 5.45TiB used 4.96TiB path /dev/sdb1
ubuntu@ubuntu:/isodevice$ btrfs fi df /mnt
Data, RAID1: total=4.36TiB, used=4.35TiB
Data, RAID5: total=2.25TiB, used=2.23TiB
System, RAID1: total=64.00MiB, used=832.00KiB
Metadata, RAID1: total=33.00GiB, used=31.43GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
DMESG:
[ 451.381275] BTRFS info (device sdc4): disk space caching is enabled
[ 451.531224] BTRFS info (device sdc4): bdev /dev/sdc4 errs: wr 0, rd
0, flush 0, corrupt 0, gen 19
[ 451.531239] BTRFS info (device sdc4): bdev /dev/sde4 errs: wr 0, rd
0, flush 0, corrupt 0, gen 3
[ 451.531249] BTRFS info (device sdc4): bdev /dev/sda4 errs: wr 0, rd
15, flush 0, corrupt 0, gen 16
[ 493.187544] BTRFS info (device sdc4): checking UUID tree
[ 550.070483] BTRFS critical (device sdc4): corrupt leaf: root=1
block=57567265079296 slot=83, bad key order, prev
(18446744073709551605 0 57707594776576) current (18446726481523507189
0 57709742260224)
[ 550.070569] BTRFS warning (device sdc4): btrfs_uuid_scan_kthread failed -5
Debug info:
sudo btrfs inspect-internal dump-tree -b 57567265079296 --follow /dev/sda4
btrfs-progs v4.16.1
leaf 57567265079296 items 246 free space 47 generation 7325984 owner ROOT_TREE
leaf 57567265079296 flags 0x1(WRITTEN) backref revision 1
fs uuid 90900a21-8a71-4301-b5c2-21dea31f1132
chunk uuid b3eed334-68af-43e6-ae1e-c05e504f30cf
item 0 key (FREE_SPACE UNTYPED 57557270921216) itemoff 16242 itemsize 41
location key (74274 INODE_ITEM 0)
cache generation 7325984 entries 308 bitmaps 8
item 1 key (FREE_SPACE UNTYPED 57558344663040) itemoff 16201 itemsize 41
location key (74275 INODE_ITEM 0)
cache generation 7325984 entries 338 bitmaps 8
... [snip] ...
item 81 key (FREE_SPACE UNTYPED 57705447292928) itemoff 12921 itemsize 41
location key (74392 INODE_ITEM 0)
cache generation 2530576 entries 0 bitmaps 0
item 82 key (FREE_SPACE UNTYPED 57707594776576) itemoff 12880 itemsize 41
location key (74393 INODE_ITEM 0)
cache generation 2530576 entries 0 bitmaps 0
item 83 key (18446726481523507189 UNKNOWN.0 57709742260224)
itemoff 12839 itemsize 41
item 84 key (FREE_SPACE UNTYPED 57711889743872) itemoff 12798 itemsize 41
location key (74395 INODE_ITEM 0)
cache generation 2530576 entries 0 bitmaps 0
item 85 key (FREE_SPACE UNTYPED 57714037227520) itemoff 12757 itemsize 41
location key (74396 INODE_ITEM 0)
cache generation 5542000 entries 0 bitmaps 0
... [snip] ...
If i read the block for slot 83 and also 82 and 84 i get:
ubuntu@ubuntu:/isodevice$ sudo btrfs inspect-internal dump-tree -b
57709742260224 --follow /dev/sda4
btrfs-progs v4.16.1
checksum verify failed on 57709742260224 found 957DA941 wanted 46464952
checksum verify failed on 57709742260224 found 957DA941 wanted 46464952
checksum verify failed on 57709742260224 found 1A43BFC8 wanted F8AC1E60
checksum verify failed on 57709742260224 found 1A43BFC8 wanted F8AC1E60
bytenr mismatch, want=57709742260224, have=41175219756947292
ERROR: failed to read 57709742260224
ubuntu@ubuntu:/isodevice$ sudo btrfs inspect-internal dump-tree -b
57711889743872 --follow /dev/sda4
btrfs-progs v4.16.1
checksum verify failed on 57711889743872 found 7C7F3E7E wanted D95F6073
checksum verify failed on 57711889743872 found 7C7F3E7E wanted D95F6073
checksum verify failed on 57711889743872 found 15984D09 wanted 72B7E0E4
checksum verify failed on 57711889743872 found 7C7F3E7E wanted D95F6073
bytenr mismatch, want=57711889743872, have=3894064465997382214
ERROR: failed to read 57711889743872
ubuntu@ubuntu:/isodevice$ sudo btrfs inspect-internal dump-tree -b
57707594776576 --follow /dev/sda4
btrfs-progs v4.16.1
checksum verify failed on 57707594776576 found FD8DDF70 wanted B48A9029
checksum verify failed on 57707594776576 found FD8DDF70 wanted B48A9029
checksum verify failed on 57707594776576 found F7A75766 wanted 563355DA
checksum verify failed on 57707594776576 found F7A75766 wanted 563355DA
bytenr mismatch, want=57707594776576, have=14084120062297221284
ERROR: failed to read 57707594776576
