See attached for requested output. Do I still need to recover the super block of sdb? Could you please point me the right direction for doing the inplace recovery? I have not rebooterd or tried to recover / mount the disc btw. Thanks, Ben On 8 April 2018 at 10:02, Qu Wenruo <quwenruo.btrfs@xxxxxxx> wrote: > > > On 2018年04月08日 07:29, Ben Parsons wrote: >> On 7 April 2018 at 22:09, Qu Wenruo <quwenruo.btrfs@xxxxxxx> wrote: >>> >>> >>> On 2018年04月07日 10:31, Ben Parsons wrote: >>> [snip] >>>>> Pretty common hard power reset. >>>>> >>>>>> looking at journalctl, there is a large stacktrace from kernel: amdgpu >>>>>> (see attached). >>>>>> then when I booted back up the pool (2 disks, 1TB + 2TB) wouldn't mount. >>>>> >>>>> I'd say such corruption is pretty serious. >>>>> >>>>> And what's the profile of the btrfs? If metadata is raid1, we could at >>>>> least try to recovery the superblock from the remaining disk. >>>> >>>> I am not sure what the metadata was but the two disks had no parity >>>> and just appeared as a single disk with total space of the two disks >>> >>> Strangely, for the 2nd disk, it's sdc1, which means it has partition table. >>> While for the 1st disk, it's sda, without partition table at all. >>> Is there any possibility that you just took run partition? >>> (Or did some program uses it incorrectly?) >>> >> >> I dont quite understand what you are asking. >> I was always under the impression I could run mount on either >> partition and it would mount the pool >> >>>> >>>> how would i got about recovering the 2nd disk? attached is >>> >>> The 2nd disk looks good, however it's csum_type is wrong. >>> 41700 looks like garbage. >>> >>> Despite that, incompact_flags also has garbage. >>> >>> The good news is, the system (and metadata) profile is RAID1, so it's >>> highly possible for us to salvage (to be more accurate, rebuild) the >>> superblock for the 1st device. >>> >>> Please dump the superblock of the 2nd device (sdc1) by the following >>> command: >>> >>> # dd if=/dev/sdc1 of=super_dump.sdc1 bs=1 count=4096 skip=64k >>> >> >> See attached. >> >>> >>> Unfortunately, btrfs-sb-mod tool added recently doesn't have all needed >>> fields, so I'm afraid I need to manually modify it. >>> >>> And just in case, please paste the following output to help us verify if >>> it's really sda without offset: >>> >>> # lsblk /dev/sda >>> # grep -obUaP "\x5F\x42\x48\x52\x66\x53\x5F\x4D" >>> >> >> dd if=/dev/sdb of=toGrep.sdb bs=1 count=128M status=progress >> cat toGrep.sdb | grep -obUaP "\x5F\x42\x48\x52\x66\x53\x5F\x4D" >> >> 65600:_BHRfS_M >> 67108928:_BHRfS_M > > Well, the magic number is completely correct, and at correct location. > > Would you please run "btrfs inspect dump-super -fFa /dev/sdb" again? > This time it should provide good data. > >> >>> >>> Above grep could be very slow since it will try to iterate the whole >>> disk. It's recommended to dump the first 128M of the disk and then grep >>> on that 128M image. >>> >>> >>> BTW, with superblock of sdc1 patched, you should be able to mount the fs >>> with -o ro,degraded, and salvage some data. >>> >>> Thanks, >>> Qu >> >> Thank you so much! >> >> I am better off copying the data to another disk and then rebuilding the pool? >> or can I just run a scrub after the super block is fixed? > > According to your latest grep output, strangely the 1st device is not > that corrupted as before. > > So I think in-place recover should save you a lot of time. > > Thanks, > Qu > >> >> For reference here is lsblk: >> >> sda 8:0 0 465.8G 0 disk >> ├─sda1 8:1 0 512M 0 part /boot >> ├─sda2 8:2 0 455.3G 0 part / >> └─sda3 8:3 0 10G 0 part [SWAP] >> >> sdb 8:16 0 931.5G 0 disk >> -- first disk >> >> sdc 8:32 0 1.8T 0 disk >> └─sdc1 8:33 0 1.8T 0 part >> -- 2nd disk >>
superblock: bytenr=65536, device=/dev/sdb --------------------------------------------------------- csum_type 41700 (INVALID) csum_size 32 csum 0x76e0389d00000000000000000000000000000000000000000000000000000000 [match] bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 08e51c76-0068-45ba-bac8-9c1f57363ec6 label generation 1285351 root 6485326479360 sys_array_size 129 chunk_root_generation 1273669 root_level 1 chunk_root 5518540881920 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 2979127844864 bytes_used 2924699414528 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 2 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x5b22400000000169 ( MIXED_BACKREF | COMPRESS_LZO | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA | unknown flag: 0x5b22400000000000 ) cache_generation 1285351 uuid_tree_generation 1285351 dev_item.uuid a409aae3-fafd-49d5-b589-d54eedca628f dev_item.fsid 08e51c76-0068-45ba-bac8-9c1f57363ec6 [match] dev_item.type 0 dev_item.total_bytes 1000204886016 dev_item.bytes_used 979802718208 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 2 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 5518540865536) length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1 io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 1 stripe 0 devid 1 offset 1014723772416 dev_uuid 9a4489cd-5aee-41e9-925e-7c8c354dd53f stripe 1 devid 2 offset 6477053952 dev_uuid a409aae3-fafd-49d5-b589-d54eedca628f backup_roots[4]: backup 0: backup_tree_root: 6485326479360 gen: 1285351 level: 1 backup_chunk_root: 5518540881920 gen: 1273669 level: 1 backup_extent_root: 6485326430208 gen: 1285351 level: 2 backup_fs_root: 6485326364672 gen: 1285351 level: 2 backup_dev_root: 4601730236416 gen: 1281190 level: 1 backup_csum_root: 6485326561280 gen: 1285351 level: 3 backup_total_bytes: 2979127844864 backup_bytes_used: 2924699414528 backup_num_devices: 2 backup 1: backup_tree_root: 6485324480512 gen: 1285348 level: 1 backup_chunk_root: 5518540881920 gen: 1273669 level: 1 backup_extent_root: 6485324398592 gen: 1285348 level: 2 backup_fs_root: 6485324283904 gen: 1285348 level: 2 backup_dev_root: 4601730236416 gen: 1281190 level: 1 backup_csum_root: 6485324333056 gen: 1285348 level: 3 backup_total_bytes: 2979127844864 backup_bytes_used: 2924699414528 backup_num_devices: 2 backup 2: backup_tree_root: 6485324955648 gen: 1285349 level: 1 backup_chunk_root: 5518540881920 gen: 1273669 level: 1 backup_extent_root: 6485324873728 gen: 1285349 level: 2 backup_fs_root: 6485324824576 gen: 1285349 level: 2 backup_dev_root: 4601730236416 gen: 1281190 level: 1 backup_csum_root: 6485325053952 gen: 1285349 level: 3 backup_total_bytes: 2979127844864 backup_bytes_used: 2924699414528 backup_num_devices: 2 backup 3: backup_tree_root: 6485325922304 gen: 1285350 level: 1 backup_chunk_root: 5518540881920 gen: 1273669 level: 1 backup_extent_root: 6485325807616 gen: 1285350 level: 2 backup_fs_root: 6485325381632 gen: 1285350 level: 2 backup_dev_root: 4601730236416 gen: 1281190 level: 1 backup_csum_root: 6485326053376 gen: 1285350 level: 3 backup_total_bytes: 2979127844864 backup_bytes_used: 2924699414528 backup_num_devices: 2 superblock: bytenr=67108864, device=/dev/sdb --------------------------------------------------------- csum_type 41700 (INVALID) csum_size 32 csum 0xd681105300000000000000000000000000000000000000000000000000000000 [DON'T MATCH] bytenr 67108864 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 08e51c76-0068-45ba-bac8-9c1f57363ec6 label generation 1285351 root 6485326479360 sys_array_size 129 chunk_root_generation 1273669 root_level 1 chunk_root 5518540881920 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 2979127844864 bytes_used 2924699414528 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 2 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x5b22400000000169 ( MIXED_BACKREF | COMPRESS_LZO | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA | unknown flag: 0x5b22400000000000 ) cache_generation 1285351 uuid_tree_generation 1285351 dev_item.uuid a409aae3-fafd-49d5-b589-d54eedca628f dev_item.fsid 08e51c76-0068-45ba-bac8-9c1f57363ec6 [match] dev_item.type 0 dev_item.total_bytes 1000204886016 dev_item.bytes_used 979802718208 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 2 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 5518540865536) length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1 io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 1 stripe 0 devid 1 offset 1014723772416 dev_uuid 9a4489cd-5aee-41e9-925e-7c8c354dd53f stripe 1 devid 2 offset 6477053952 dev_uuid a409aae3-fafd-49d5-b589-d54eedca628f backup_roots[4]: backup 0: backup_tree_root: 6485326479360 gen: 1285351 level: 1 backup_chunk_root: 5518540881920 gen: 1273669 level: 1 backup_extent_root: 6485326430208 gen: 1285351 level: 2 backup_fs_root: 6485326364672 gen: 1285351 level: 2 backup_dev_root: 4601730236416 gen: 1281190 level: 1 backup_csum_root: 6485326561280 gen: 1285351 level: 3 backup_total_bytes: 2979127844864 backup_bytes_used: 2924699414528 backup_num_devices: 2 backup 1: backup_tree_root: 6485324480512 gen: 1285348 level: 1 backup_chunk_root: 5518540881920 gen: 1273669 level: 1 backup_extent_root: 6485324398592 gen: 1285348 level: 2 backup_fs_root: 6485324283904 gen: 1285348 level: 2 backup_dev_root: 4601730236416 gen: 1281190 level: 1 backup_csum_root: 6485324333056 gen: 1285348 level: 3 backup_total_bytes: 2979127844864 backup_bytes_used: 2924699414528 backup_num_devices: 2 backup 2: backup_tree_root: 6485324955648 gen: 1285349 level: 1 backup_chunk_root: 5518540881920 gen: 1273669 level: 1 backup_extent_root: 6485324873728 gen: 1285349 level: 2 backup_fs_root: 6485324824576 gen: 1285349 level: 2 backup_dev_root: 4601730236416 gen: 1281190 level: 1 backup_csum_root: 6485325053952 gen: 1285349 level: 3 backup_total_bytes: 2979127844864 backup_bytes_used: 2924699414528 backup_num_devices: 2 backup 3: backup_tree_root: 6485325922304 gen: 1285350 level: 1 backup_chunk_root: 5518540881920 gen: 1273669 level: 1 backup_extent_root: 6485325807616 gen: 1285350 level: 2 backup_fs_root: 6485325381632 gen: 1285350 level: 2 backup_dev_root: 4601730236416 gen: 1281190 level: 1 backup_csum_root: 6485326053376 gen: 1285350 level: 3 backup_total_bytes: 2979127844864 backup_bytes_used: 2924699414528 backup_num_devices: 2 superblock: bytenr=274877906944, device=/dev/sdb --------------------------------------------------------- csum_type 41700 (INVALID) csum_size 32 csum 0x2b06466200000000000000000000000000000000000000000000000000000000 [DON'T MATCH] bytenr 274877906944 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 08e51c76-0068-45ba-bac8-9c1f57363ec6 label generation 1285351 root 6485326479360 sys_array_size 129 chunk_root_generation 1273669 root_level 1 chunk_root 5518540881920 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 2979127844864 bytes_used 2924699414528 sectorsize 4096 nodesize 16384 leafsize (deprecated) 16384 stripesize 4096 root_dir 6 num_devices 2 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x5b22400000000169 ( MIXED_BACKREF | COMPRESS_LZO | BIG_METADATA | EXTENDED_IREF | SKINNY_METADATA | unknown flag: 0x5b22400000000000 ) cache_generation 1285351 uuid_tree_generation 1285351 dev_item.uuid a409aae3-fafd-49d5-b589-d54eedca628f dev_item.fsid 08e51c76-0068-45ba-bac8-9c1f57363ec6 [match] dev_item.type 0 dev_item.total_bytes 1000204886016 dev_item.bytes_used 979802718208 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 2 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 5518540865536) length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1 io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 1 stripe 0 devid 1 offset 1014723772416 dev_uuid 9a4489cd-5aee-41e9-925e-7c8c354dd53f stripe 1 devid 2 offset 6477053952 dev_uuid a409aae3-fafd-49d5-b589-d54eedca628f backup_roots[4]: backup 0: backup_tree_root: 6485326479360 gen: 1285351 level: 1 backup_chunk_root: 5518540881920 gen: 1273669 level: 1 backup_extent_root: 6485326430208 gen: 1285351 level: 2 backup_fs_root: 6485326364672 gen: 1285351 level: 2 backup_dev_root: 4601730236416 gen: 1281190 level: 1 backup_csum_root: 6485326561280 gen: 1285351 level: 3 backup_total_bytes: 2979127844864 backup_bytes_used: 2924699414528 backup_num_devices: 2 backup 1: backup_tree_root: 6485324480512 gen: 1285348 level: 1 backup_chunk_root: 5518540881920 gen: 1273669 level: 1 backup_extent_root: 6485324398592 gen: 1285348 level: 2 backup_fs_root: 6485324283904 gen: 1285348 level: 2 backup_dev_root: 4601730236416 gen: 1281190 level: 1 backup_csum_root: 6485324333056 gen: 1285348 level: 3 backup_total_bytes: 2979127844864 backup_bytes_used: 2924699414528 backup_num_devices: 2 backup 2: backup_tree_root: 6485324955648 gen: 1285349 level: 1 backup_chunk_root: 5518540881920 gen: 1273669 level: 1 backup_extent_root: 6485324873728 gen: 1285349 level: 2 backup_fs_root: 6485324824576 gen: 1285349 level: 2 backup_dev_root: 4601730236416 gen: 1281190 level: 1 backup_csum_root: 6485325053952 gen: 1285349 level: 3 backup_total_bytes: 2979127844864 backup_bytes_used: 2924699414528 backup_num_devices: 2 backup 3: backup_tree_root: 6485325922304 gen: 1285350 level: 1 backup_chunk_root: 5518540881920 gen: 1273669 level: 1 backup_extent_root: 6485325807616 gen: 1285350 level: 2 backup_fs_root: 6485325381632 gen: 1285350 level: 2 backup_dev_root: 4601730236416 gen: 1281190 level: 1 backup_csum_root: 6485326053376 gen: 1285350 level: 3 backup_total_bytes: 2979127844864 backup_bytes_used: 2924699414528 backup_num_devices: 2
