Hi Qu, Thanks for your reply. I will investigate upgrading the kernel, however I worry that future ReadyNAS firmware upgrades would fail on a newer kernel version (I don't have much linux experience so maybe my concerns are unfounded!?). I have attached the output of the dump super command. I did actually run chunk recover before, without the verbose option, it took around 24 hours to finish but did not resolve my issue. Happy to start that again if you need its output. Thanks so much for your help. Kind regards Michael On 28 April 2018 at 09:45, Qu Wenruo <quwenruo.btrfs@xxxxxxx> wrote: > > > On 2018年04月28日 16:30, Michael Wade wrote: >> Hi all, >> >> I was hoping that someone would be able to help me resolve the issues >> I am having with my ReadyNAS BTRFS volume. Basically my trouble >> started after a power cut, subsequently the volume would not mount. >> Here are the details of my setup as it is at the moment: >> >> uname -a >> Linux QAI 4.4.116.alpine.1 #1 SMP Mon Feb 19 21:58:38 PST 2018 armv7l GNU/Linux > > The kernel is pretty old for btrfs. > Strongly recommended to upgrade. > >> >> btrfs --version >> btrfs-progs v4.12 > > So is the user tools. > > Although I think it won't be a big problem, as needed tool should be there. > >> >> btrfs fi show >> Label: '11baed92:data' uuid: 20628cda-d98f-4f85-955c-932a367f8821 >> Total devices 1 FS bytes used 5.12TiB >> devid 1 size 7.27TiB used 6.24TiB path /dev/md127 > > So, it's btrfs on mdraid. > It would normally make things harder to debug, so I could only provide > advice from the respect of btrfs. > For mdraid part, I can't ensure anything. > >> >> Here are the relevant dmesg logs for the current state of the device: >> >> [ 19.119391] md: md127 stopped. >> [ 19.120841] md: bind<sdb3> >> [ 19.121120] md: bind<sdc3> >> [ 19.121380] md: bind<sda3> >> [ 19.125535] md/raid:md127: device sda3 operational as raid disk 0 >> [ 19.125547] md/raid:md127: device sdc3 operational as raid disk 2 >> [ 19.125554] md/raid:md127: device sdb3 operational as raid disk 1 >> [ 19.126712] md/raid:md127: allocated 3240kB >> [ 19.126778] md/raid:md127: raid level 5 active with 3 out of 3 >> devices, algorithm 2 >> [ 19.126784] RAID conf printout: >> [ 19.126789] --- level:5 rd:3 wd:3 >> [ 19.126794] disk 0, o:1, dev:sda3 >> [ 19.126799] disk 1, o:1, dev:sdb3 >> [ 19.126804] disk 2, o:1, dev:sdc3 >> [ 19.128118] md127: detected capacity change from 0 to 7991637573632 >> [ 19.395112] Adding 523708k swap on /dev/md1. Priority:-1 extents:1 >> across:523708k >> [ 19.434956] BTRFS: device label 11baed92:data devid 1 transid >> 151800 /dev/md127 >> [ 19.739276] BTRFS info (device md127): setting nodatasum >> [ 19.740440] BTRFS critical (device md127): unable to find logical >> 3208757641216 len 4096 >> [ 19.740450] BTRFS critical (device md127): unable to find logical >> 3208757641216 len 4096 >> [ 19.740498] BTRFS critical (device md127): unable to find logical >> 3208757641216 len 4096 >> [ 19.740512] BTRFS critical (device md127): unable to find logical >> 3208757641216 len 4096 >> [ 19.740552] BTRFS critical (device md127): unable to find logical >> 3208757641216 len 4096 >> [ 19.740560] BTRFS critical (device md127): unable to find logical >> 3208757641216 len 4096 >> [ 19.740576] BTRFS error (device md127): failed to read chunk root > > This shows it pretty clear, btrfs fails to read chunk root. > And according your above "len 4096" it's pretty old fs, as it's still > using 4K nodesize other than 16K nodesize. > > According to above output, it means your superblock by somehow lacks the > needed system chunk mapping, which is used to initialize chunk mapping. > > Please provide the following command output: > > # btrfs inspect dump-super -fFa /dev/md127 > > Also, please consider run the following command and dump all its output: > > # btrfs rescue chunk-recover -v /dev/md127. > > Please note that, above command can take a long time to finish, and if > it works without problem, it may solve your problem. > But if it doesn't work, the output could help me to manually craft a fix > to your super block. > > Thanks, > Qu > > >> [ 19.783975] BTRFS error (device md127): open_ctree failed >> >> In an attempt to recover the volume myself I run a few BTRFS commands >> mostly using advice from here: >> https://lists.opensuse.org/opensuse/2017-02/msg00930.html. However >> that actually seems to have made things worse as I can no longer mount >> the file system, not even in readonly mode. >> >> So starting from the beginning here is a list of things I have done so >> far (hopefully I remembered the order in which I ran them!) >> >> 1. Noticed that my backups to the NAS were not running (didn't get >> notified that the volume had basically "died") >> 2. ReadyNAS UI indicated that the volume was inactive. >> 3. SSHed onto the box and found that the first drive was not marked as >> operational (log showed I/O errors / UNKOWN (0x2003)) so I replaced >> the disk and let the array resync. >> 4. After resync the volume still was unaccessible so I looked at the >> logs once more and saw something like the following which seemed to >> indicate that the replay log had been corrupted when the power went >> out: >> >> BTRFS critical (device md127): corrupt leaf, non-root leaf's nritems >> is 0: block=232292352, root=7, slot=0 >> BTRFS critical (device md127): corrupt leaf, non-root leaf's nritems >> is 0: block=232292352, root=7, slot=0 >> BTRFS: error (device md127) in btrfs_replay_log:2524: errno=-5 IO >> failure (Failed to recover log tree) >> BTRFS error (device md127): pending csums is 155648 >> BTRFS error (device md127): cleaner transaction attach returned -30 >> BTRFS critical (device md127): corrupt leaf, non-root leaf's nritems >> is 0: block=232292352, root=7, slot=0 >> >> 5. Then: >> >> btrfs rescue zero-log >> >> 6. Was then able to mount the volume in readonly mode. >> >> btrfs scrub start >> >> Which fixed some errors but not all: >> >> scrub status for 20628cda-d98f-4f85-955c-932a367f8821 >> >> scrub started at Tue Apr 24 17:27:44 2018, running for 04:00:34 >> total bytes scrubbed: 224.26GiB with 6 errors >> error details: csum=6 >> corrected errors: 0, uncorrectable errors: 6, unverified errors: 0 >> >> scrub status for 20628cda-d98f-4f85-955c-932a367f8821 >> scrub started at Tue Apr 24 17:27:44 2018, running for 04:34:43 >> total bytes scrubbed: 224.26GiB with 6 errors >> error details: csum=6 >> corrected errors: 0, uncorrectable errors: 6, unverified errors: 0 >> >> 6. Seeing this hanging I rebooted the NAS >> 7. Think this is when the volume would not mount at all. >> 8. Seeing log entries like these: >> >> BTRFS warning (device md127): checksum error at logical 20800943685632 >> on dev /dev/md127, sector 520167424: metadata node (level 1) in tree 3 >> >> I ran >> >> btrfs check --fix-crc >> >> And that brings us to where I am now: Some seemly corrupted BTRFS >> metadata and unable to mount the drive even with the recovery option. >> >> Any help you can give is much appreciated! >> >> Kind regards >> Michael >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >
btrfs inspect dump-super -fFa /dev/md127 superblock: bytenr=65536, device=/dev/md127 --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0xa5e7dc75 [match] bytenr 65536 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 20628cda-d98f-4f85-955c-932a367f8821 label 11baed92:data generation 151800 root 44957696 sys_array_size 355 chunk_root_generation 151777 root_level 1 chunk_root 20800943685632 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 7991637573632 bytes_used 5631993507840 sectorsize 4096 nodesize 32768 leafsize (deprecated) 32768 stripesize 4096 root_dir 6 num_devices 1 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x21 ( MIXED_BACKREF | BIG_METADATA ) cache_generation 18446744073709551615 uuid_tree_generation 151800 dev_item.uuid 7c2324dc-7906-430a-b44f-cbfce3ac5c56 dev_item.fsid 20628cda-d98f-4f85-955c-932a367f8821 [match] dev_item.type 2 dev_item.total_bytes 7991637573632 dev_item.bytes_used 6860241371136 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 1 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 0) length 4194304 owner 2 stripe_len 65536 type SYSTEM io_align 4096 io_width 4096 sector_size 4096 num_stripes 1 sub_stripes 0 stripe 0 devid 1 offset 0 dev_uuid 7c2324dc-7906-430a-b44f-cbfce3ac5c56 item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520) length 8388608 owner 2 stripe_len 65536 type SYSTEM|DUP io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 0 stripe 0 devid 1 offset 20971520 dev_uuid 7c2324dc-7906-430a-b44f-cbfce3ac5c56 stripe 1 devid 1 offset 29360128 dev_uuid 7c2324dc-7906-430a-b44f-cbfce3ac5c56 item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 20800943685632) length 33554432 owner 2 stripe_len 65536 type SYSTEM|DUP io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 1 stripe 0 devid 1 offset 266325721088 dev_uuid 7c2324dc-7906-430a-b44f-cbfce3ac5c56 stripe 1 devid 1 offset 266359275520 dev_uuid 7c2324dc-7906-430a-b44f-cbfce3ac5c56 backup_roots[4]: backup 0: backup_tree_root: 44236800 gen: 151798 level: 1 backup_chunk_root: 20800943685632 gen: 151777 level: 1 backup_extent_root: 44302336 gen: 151798 level: 2 backup_fs_root: 167346176 gen: 150972 level: 0 backup_dev_root: 29523968 gen: 151777 level: 1 backup_csum_root: 37650432 gen: 151775 level: 0 backup_total_bytes: 7991637573632 backup_bytes_used: 5631993507840 backup_num_devices: 1 backup 1: backup_tree_root: 44597248 gen: 151799 level: 1 backup_chunk_root: 20800943685632 gen: 151777 level: 1 backup_extent_root: 44630016 gen: 151799 level: 2 backup_fs_root: 167346176 gen: 150972 level: 0 backup_dev_root: 29523968 gen: 151777 level: 1 backup_csum_root: 37650432 gen: 151775 level: 0 backup_total_bytes: 7991637573632 backup_bytes_used: 5631993507840 backup_num_devices: 1 backup 2: backup_tree_root: 44957696 gen: 151800 level: 1 backup_chunk_root: 20800943685632 gen: 151777 level: 1 backup_extent_root: 45023232 gen: 151800 level: 2 backup_fs_root: 167346176 gen: 150972 level: 0 backup_dev_root: 29523968 gen: 151777 level: 1 backup_csum_root: 37650432 gen: 151775 level: 0 backup_total_bytes: 7991637573632 backup_bytes_used: 5631993507840 backup_num_devices: 1 backup 3: backup_tree_root: 43876352 gen: 151797 level: 1 backup_chunk_root: 20800943685632 gen: 151777 level: 1 backup_extent_root: 43941888 gen: 151797 level: 2 backup_fs_root: 167346176 gen: 150972 level: 0 backup_dev_root: 29523968 gen: 151777 level: 1 backup_csum_root: 37650432 gen: 151775 level: 0 backup_total_bytes: 7991637573632 backup_bytes_used: 5631993507840 backup_num_devices: 1 superblock: bytenr=67108864, device=/dev/md127 --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0x0586f4bb [match] bytenr 67108864 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 20628cda-d98f-4f85-955c-932a367f8821 label 11baed92:data generation 151800 root 44957696 sys_array_size 355 chunk_root_generation 151777 root_level 1 chunk_root 20800943685632 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 7991637573632 bytes_used 5631993507840 sectorsize 4096 nodesize 32768 leafsize (deprecated) 32768 stripesize 4096 root_dir 6 num_devices 1 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x21 ( MIXED_BACKREF | BIG_METADATA ) cache_generation 18446744073709551615 uuid_tree_generation 151800 dev_item.uuid 7c2324dc-7906-430a-b44f-cbfce3ac5c56 dev_item.fsid 20628cda-d98f-4f85-955c-932a367f8821 [match] dev_item.type 2 dev_item.total_bytes 7991637573632 dev_item.bytes_used 6860241371136 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 1 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 0) length 4194304 owner 2 stripe_len 65536 type SYSTEM io_align 4096 io_width 4096 sector_size 4096 num_stripes 1 sub_stripes 0 stripe 0 devid 1 offset 0 dev_uuid 7c2324dc-7906-430a-b44f-cbfce3ac5c56 item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520) length 8388608 owner 2 stripe_len 65536 type SYSTEM|DUP io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 0 stripe 0 devid 1 offset 20971520 dev_uuid 7c2324dc-7906-430a-b44f-cbfce3ac5c56 stripe 1 devid 1 offset 29360128 dev_uuid 7c2324dc-7906-430a-b44f-cbfce3ac5c56 item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 20800943685632) length 33554432 owner 2 stripe_len 65536 type SYSTEM|DUP io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 1 stripe 0 devid 1 offset 266325721088 dev_uuid 7c2324dc-7906-430a-b44f-cbfce3ac5c56 stripe 1 devid 1 offset 266359275520 dev_uuid 7c2324dc-7906-430a-b44f-cbfce3ac5c56 backup_roots[4]: backup 0: backup_tree_root: 44236800 gen: 151798 level: 1 backup_chunk_root: 20800943685632 gen: 151777 level: 1 backup_extent_root: 44302336 gen: 151798 level: 2 backup_fs_root: 167346176 gen: 150972 level: 0 backup_dev_root: 29523968 gen: 151777 level: 1 backup_csum_root: 37650432 gen: 151775 level: 0 backup_total_bytes: 7991637573632 backup_bytes_used: 5631993507840 backup_num_devices: 1 backup 1: backup_tree_root: 44597248 gen: 151799 level: 1 backup_chunk_root: 20800943685632 gen: 151777 level: 1 backup_extent_root: 44630016 gen: 151799 level: 2 backup_fs_root: 167346176 gen: 150972 level: 0 backup_dev_root: 29523968 gen: 151777 level: 1 backup_csum_root: 37650432 gen: 151775 level: 0 backup_total_bytes: 7991637573632 backup_bytes_used: 5631993507840 backup_num_devices: 1 backup 2: backup_tree_root: 44957696 gen: 151800 level: 1 backup_chunk_root: 20800943685632 gen: 151777 level: 1 backup_extent_root: 45023232 gen: 151800 level: 2 backup_fs_root: 167346176 gen: 150972 level: 0 backup_dev_root: 29523968 gen: 151777 level: 1 backup_csum_root: 37650432 gen: 151775 level: 0 backup_total_bytes: 7991637573632 backup_bytes_used: 5631993507840 backup_num_devices: 1 backup 3: backup_tree_root: 43876352 gen: 151797 level: 1 backup_chunk_root: 20800943685632 gen: 151777 level: 1 backup_extent_root: 43941888 gen: 151797 level: 2 backup_fs_root: 167346176 gen: 150972 level: 0 backup_dev_root: 29523968 gen: 151777 level: 1 backup_csum_root: 37650432 gen: 151775 level: 0 backup_total_bytes: 7991637573632 backup_bytes_used: 5631993507840 backup_num_devices: 1 superblock: bytenr=274877906944, device=/dev/md127 --------------------------------------------------------- csum_type 0 (crc32c) csum_size 4 csum 0xf801a28a [match] bytenr 274877906944 flags 0x1 ( WRITTEN ) magic _BHRfS_M [match] fsid 20628cda-d98f-4f85-955c-932a367f8821 label 11baed92:data generation 151800 root 44957696 sys_array_size 355 chunk_root_generation 151777 root_level 1 chunk_root 20800943685632 chunk_root_level 1 log_root 0 log_root_transid 0 log_root_level 0 total_bytes 7991637573632 bytes_used 5631993507840 sectorsize 4096 nodesize 32768 leafsize (deprecated) 32768 stripesize 4096 root_dir 6 num_devices 1 compat_flags 0x0 compat_ro_flags 0x0 incompat_flags 0x21 ( MIXED_BACKREF | BIG_METADATA ) cache_generation 18446744073709551615 uuid_tree_generation 151800 dev_item.uuid 7c2324dc-7906-430a-b44f-cbfce3ac5c56 dev_item.fsid 20628cda-d98f-4f85-955c-932a367f8821 [match] dev_item.type 2 dev_item.total_bytes 7991637573632 dev_item.bytes_used 6860241371136 dev_item.io_align 4096 dev_item.io_width 4096 dev_item.sector_size 4096 dev_item.devid 1 dev_item.dev_group 0 dev_item.seek_speed 0 dev_item.bandwidth 0 dev_item.generation 0 sys_chunk_array[2048]: item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 0) length 4194304 owner 2 stripe_len 65536 type SYSTEM io_align 4096 io_width 4096 sector_size 4096 num_stripes 1 sub_stripes 0 stripe 0 devid 1 offset 0 dev_uuid 7c2324dc-7906-430a-b44f-cbfce3ac5c56 item 1 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520) length 8388608 owner 2 stripe_len 65536 type SYSTEM|DUP io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 0 stripe 0 devid 1 offset 20971520 dev_uuid 7c2324dc-7906-430a-b44f-cbfce3ac5c56 stripe 1 devid 1 offset 29360128 dev_uuid 7c2324dc-7906-430a-b44f-cbfce3ac5c56 item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 20800943685632) length 33554432 owner 2 stripe_len 65536 type SYSTEM|DUP io_align 65536 io_width 65536 sector_size 4096 num_stripes 2 sub_stripes 1 stripe 0 devid 1 offset 266325721088 dev_uuid 7c2324dc-7906-430a-b44f-cbfce3ac5c56 stripe 1 devid 1 offset 266359275520 dev_uuid 7c2324dc-7906-430a-b44f-cbfce3ac5c56 backup_roots[4]: backup 0: backup_tree_root: 44236800 gen: 151798 level: 1 backup_chunk_root: 20800943685632 gen: 151777 level: 1 backup_extent_root: 44302336 gen: 151798 level: 2 backup_fs_root: 167346176 gen: 150972 level: 0 backup_dev_root: 29523968 gen: 151777 level: 1 backup_csum_root: 37650432 gen: 151775 level: 0 backup_total_bytes: 7991637573632 backup_bytes_used: 5631993507840 backup_num_devices: 1 backup 1: backup_tree_root: 44597248 gen: 151799 level: 1 backup_chunk_root: 20800943685632 gen: 151777 level: 1 backup_extent_root: 44630016 gen: 151799 level: 2 backup_fs_root: 167346176 gen: 150972 level: 0 backup_dev_root: 29523968 gen: 151777 level: 1 backup_csum_root: 37650432 gen: 151775 level: 0 backup_total_bytes: 7991637573632 backup_bytes_used: 5631993507840 backup_num_devices: 1 backup 2: backup_tree_root: 44957696 gen: 151800 level: 1 backup_chunk_root: 20800943685632 gen: 151777 level: 1 backup_extent_root: 45023232 gen: 151800 level: 2 backup_fs_root: 167346176 gen: 150972 level: 0 backup_dev_root: 29523968 gen: 151777 level: 1 backup_csum_root: 37650432 gen: 151775 level: 0 backup_total_bytes: 7991637573632 backup_bytes_used: 5631993507840 backup_num_devices: 1 backup 3: backup_tree_root: 43876352 gen: 151797 level: 1 backup_chunk_root: 20800943685632 gen: 151777 level: 1 backup_extent_root: 43941888 gen: 151797 level: 2 backup_fs_root: 167346176 gen: 150972 level: 0 backup_dev_root: 29523968 gen: 151777 level: 1 backup_csum_root: 37650432 gen: 151775 level: 0 backup_total_bytes: 7991637573632 backup_bytes_used: 5631993507840 backup_num_devices: 1
