Adding to last email, being more brave to test out commands now: Different crash than on last report. Command: # btrfs check --init-extent-tree --mode=lowmem -p /dev/sdh ######### $ btrfs check --init-extent-tree --mode=lowmem -p /dev/sdh WARNING: low-memory mode repair support is only partial Opening filesystem to check... Checking filesystem on /dev/sdh UUID: 8b19ff46-3f42-4f51-be6b-5fc8a7d8f2cd Creating a new extent tree Failed to find [6815744, 168, 16384] btrfs unable to find ref byte nr 55431977238528 parent 0 root 10 owner 2 offset 0 Failed to find [6815744, 168, 16384] btrfs unable to find ref byte nr 55431977254912 parent 0 root 10 owner 1 offset 0 Failed to find [6815744, 168, 16384] btrfs unable to find ref byte nr 55431977271296 parent 0 root 10 owner 0 offset 0 Failed to find [6815744, 168, 16384] btrfs unable to find ref byte nr 55431977287680 parent 0 root 10 owner 1 offset 0 Failed to find [6815744, 168, 16384] btrfs unable to find ref byte nr 55431977304064 parent 0 root 10 owner 0 offset 0 parent transid verify failed on 55431977353216 wanted 60235 found 60236 Ignoring transid failure Failed to find [55431977238528, 168, 16384] btrfs unable to find ref byte nr 55431977320448 parent 0 root 1 owner 1 offset 0 Failed to find [55431977238528, 168, 16384] btrfs unable to find ref byte nr 55431977336832 parent 0 root 1 owner 0 offset 0 Failed to find [55431977353216, 168, 16384] btrfs unable to find ref byte nr 55431977369600 parent 0 root 1 owner 0 offset 0 [1/7] checking root items... skipped ERROR: extent[1048576 16384] backref lost (owner: 3, level: 1) root 3 ## (shortened - same output repeating several times) Added an extent item [1048576 16384] Added one tree block ref start 1048576 root 3 ERROR: extent[1064960 16384] backref lost (owner: 3, level: 0) root 3 Added an extent item [1064960 16384] Added one tree block ref start 1064960 root 3 ERROR: Dev extent's total-byte 15123666173952 is not equal to bytes-used 15121518690304 in dev[1, 204, 1] ERROR: Dev extent's total-byte 15123687145472 is not equal to bytes-used 15121539661824 in dev[1, 204, 2] ERROR: Dev extent's total-byte 15123653591040 is not equal to bytes-used 15121506107392 in dev[1, 204, 3] ERROR: Dev extent's total-byte 15123653591040 is not equal to bytes-used 15121506107392 in dev[1, 204, 4] ERROR: extent[1163264 16384] backref lost (owner: 3, level: 0) root 3 Added an extent item [1163264 16384] Added one tree block ref start 1163264 root 3 ERROR: extent[1179648 16384] backref lost (owner: 3, level: 0) root 3 Added an extent item [1179648 16384] Added one tree block ref start 1179648 root 3 ERROR: extent[1196032 16384] backref lost (owner: 3, level: 0) root 3 Added an extent item [55431977877504 16384] Added one tree block ref start 55431977877504 root 1 ERROR: extent[55431977861120 16384] backref lost (owner: 10, level: 0) root 1 Added an extent item [55431977861120 16384] Added one tree block ref start 55431977861120 root 1 ERROR: extent[54142438047744 16384] backref lost (owner: 10, level: 0) root 1 Added an extent item [54142438047744 16384] Added one tree block ref start 54142438047744 root 1 ERROR: extent [54142438047744 16384] referencer bytenr mismatch, wanted: 54142438047744, have: 55432766324736 Delete backref in extent [54142438047744 16384] Failed to find [55432521760768, 168, 16384] (0:00:06 elapsed, 225 items checked) btrfs unable to find ref byte nr 55432762015744 parent 0 root 10 owner 0 offset 0 transaction.c:168: btrfs_commit_transaction: BUG_ON `ret` triggered, value -5 btrfs[0x43ebef] btrfs(btrfs_commit_transaction+0x89)[0x43f07a] btrfs[0x47baec] btrfs(check_chunks_and_extents_lowmem+0x1a4)[0x47c8b2] btrfs(cmd_check+0x1e34)[0x460cd4] btrfs(main+0x24a)[0x40ca02] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f6ce4825b35] btrfs[0x40c509] [1] 1049 abort btrfs check --init-extent-tree --mode=lowmem -p /dev/sdh ### And on another run of the same command, yet another crash: ### ERROR: extent[34096057876480 16384] backref lost (owner: 1, level: 0) root 1 Added an extent item [34096057876480 16384] Added one tree block ref start 34096057876480 root 1 ERROR: extent[3543721410560 16384] backref lost (owner: 1, level: 0) root 1 Added an extent item [3543721410560 16384] Added one tree block ref start 3543721410560 root 1 ERROR: extent[55432521760768 16384] backref lost (owner: 10, level: 0) root 1 Added an extent item [55432521760768 16384] Added one tree block ref start 55432521760768 root 1 ERROR: extent[55431977877504 16384] backref lost (owner: 10, level: 0) root 1 Added an extent item [55431977877504 16384] Added one tree block ref start 55431977877504 root 1 check/mode-lowmem.c:4588: walk_down_tree: Warning: assertion `1` failed, value 1 btrfs[0x478fb4] btrfs(check_chunks_and_extents_lowmem+0x4b)[0x47c759] btrfs(cmd_check+0x1e34)[0x460cd4] btrfs(main+0x24a)[0x40ca02] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f348d9a4b35] btrfs[0x40c509] check/mode-lowmem.c:4588: walk_down_tree: Warning: assertion `1` failed, value 1 btrfs[0x478fb4] btrfs(check_chunks_and_extents_lowmem+0x4b)[0x47c759] btrfs(cmd_check+0x1e34)[0x460cd4] btrfs(main+0x24a)[0x40ca02] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f348d9a4b35] btrfs[0x40c509] ERROR: extent[55432747909120 16384] backref lost (owner: 10, level: 1) root 1 Added an extent item [55432747909120 16384] Added one tree block ref start 55432747909120 root 1 ctree.c:1725: leaf_space_used: Warning: assertion `data_len < 0` failed, value 1 btrfs[0x40cb13] btrfs(btrfs_leaf_free_space+0x7f)[0x40ee06] btrfs[0x474092] btrfs[0x47906f] btrfs(check_chunks_and_extents_lowmem+0x4b)[0x47c759] btrfs(cmd_check+0x1e34)[0x460cd4] btrfs(main+0x24a)[0x40ca02] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f348d9a4b35] btrfs[0x40c509] leaf is not a leaf 55432747909120 ERROR: extent [55431977877504 16384] referencer bytenr mismatch, wanted: 55431977877504, have: 55432766324736 Delete backref in extent [55431977877504 16384] ERROR: extent [55432747909120 16384] referencer bytenr mismatch, wanted: 55432747909120, have: 55432766324736 Delete backref in extent [55432747909120 16384] Failed to find [55432521760768, 168, 16384] btrfs unable to find ref byte nr 55432746516480 parent 0 root 10 owner 0 offset 0 transaction.c:168: btrfs_commit_transaction: BUG_ON `ret` triggered, value -5 btrfs[0x43ebef] btrfs(btrfs_commit_transaction+0x89)[0x43f07a] btrfs[0x47baec] btrfs(check_chunks_and_extents_lowmem+0x1a4)[0x47c8b2] btrfs(cmd_check+0x1e34)[0x460cd4] btrfs(main+0x24a)[0x40ca02] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f348d9a4b35] btrfs[0x40c509] [1] 1652 abort btrfs check --init-extent-tree --mode=lowmem -p /dev/sdh ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Sunday, March 31, 2019 4:21 PM, berodual_xyz <berodual_xyz@xxxxxxxxxxxxxx> wrote: > Dear list, > > following from earlier this weeks case of attempting data rescue on a corrupt FS, we have now cloned all devices and can do potentially dangerous rescue attempts (since we can re-clone the original disks). > > On kernel 4.20.17 and btrfs-progs 4.20.2: > > -- > > $ btrfs inspect-internal tree-stats /dev/sdh > parent transid verify failed on 1048576 wanted 60234 found 60230 > parent transid verify failed on 1048576 wanted 60234 found 60230 > Ignoring transid failure > volumes.c:1762: btrfs_chunk_readonly: BUG_ON `!ce` triggered, value 1 > btrfs[0x426fdc] > btrfs(btrfs_chunk_readonly+0x98)[0x429acd] > btrfs(btrfs_read_block_groups+0x1c1)[0x41cd44] > btrfs(btrfs_setup_all_roots+0x368)[0x416540] > btrfs[0x416a8a] > btrfs(open_ctree_fs_info+0xd0)[0x416bcc] > btrfs(open_ctree+0x75)[0x416c61] > btrfs(cmd_inspect_tree_stats+0xfa)[0x4706ad] > btrfs(handle_command_group+0x5d)[0x40c7b3] > btrfs(cmd_inspect+0x15)[0x4499f1] > btrfs(main+0x24a)[0x40ca02] > /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f0f26c7cb35] > btrfs[0x40c509] > [1] 15273 abort btrfs inspect-internal tree-stats /dev/sdh > > ### > > $ btrfs rescue zero-log /dev/sdh > parent transid verify failed on 1048576 wanted 60234 found 60230 > parent transid verify failed on 1048576 wanted 60234 found 60230 > Ignoring transid failure > volumes.c:1762: btrfs_chunk_readonly: BUG_ON `!ce` triggered, value 1 > btrfs[0x426fdc] > btrfs(btrfs_chunk_readonly+0x98)[0x429acd] > btrfs(btrfs_read_block_groups+0x1c1)[0x41cd44] > btrfs(btrfs_setup_all_roots+0x368)[0x416540] > btrfs[0x416a8a] > btrfs(open_ctree_fs_info+0xd0)[0x416bcc] > btrfs(open_ctree+0x75)[0x416c61] > btrfs[0x4657ff] > btrfs(handle_command_group+0x5d)[0x40c7b3] > btrfs(cmd_rescue+0x15)[0x465b98] > btrfs(main+0x24a)[0x40ca02] > /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f4507ab1b35] > btrfs[0x40c509] > [1] 30211 abort btrfs rescue zero-log /dev/sdh > > ### > > Even if we will not be able to recover the data, adding these traces here hopefully helps enhancing the tools in the future as a crash is most definitely not the "expected behavior". > > Have a nice weekend and I hope to receive some feedback. > > Thanks in advance, > > Marcel
