If this is saying there's no extra space for metadata, is that why adding more files often makes the system hang for 30-90s? Is there anything I should do about that? Thank you so much for all of your help. I love how flexible BTRFS is but when things go wrong it's very hard for me to troubleshoot. On Fri, May 8, 2020 at 1:07 AM Qu Wenruo <quwenruo.btrfs@xxxxxxx> wrote: > > > > On 2020/5/8 下午12:23, Tyler Richmond wrote: > > Something went wrong: > > > > Reinitialize checksum tree > > Unable to find block group for 0 > > Unable to find block group for 0 > > Unable to find block group for 0 > > ctree.c:2272: split_leaf: BUG_ON `1` triggered, value 1 > > btrfs(+0x6dd94)[0x55a933af7d94] > > btrfs(+0x71b94)[0x55a933afbb94] > > btrfs(btrfs_search_slot+0x11f0)[0x55a933afd6c8] > > btrfs(btrfs_csum_file_block+0x432)[0x55a933b19d09] > > btrfs(+0x360b2)[0x55a933ac00b2] > > btrfs(+0x46a3e)[0x55a933ad0a3e] > > btrfs(main+0x98)[0x55a933a9fe88] > > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f263ed550b3] > > btrfs(_start+0x2e)[0x55a933a9fa0e] > > Aborted > > This means no space for extra metadata... > > Anyway the csum tree problem shouldn't be a big thing, you could leave > it and call it a day. > > BTW, as long as btrfs check reports no extra problem for the inode > generation, it should be pretty safe to use the fs. > > Thanks, > Qu > > > > I just noticed I have btrfs-progs 5.6 installed and 5.6.1 is > > available. I'll let that try overnight? > > > > On Thu, May 7, 2020 at 8:11 PM Qu Wenruo <quwenruo.btrfs@xxxxxxx> wrote: > >> > >> > >> > >> On 2020/5/7 下午11:52, Tyler Richmond wrote: > >>> Thank you for helping. The end result of the scan was: > >>> > >>> > >>> [1/7] checking root items > >>> [2/7] checking extents > >>> [3/7] checking free space cache > >>> [4/7] checking fs roots > >> > >> Good news is, your fs is still mostly fine. > >> > >>> [5/7] checking only csums items (without verifying data) > >>> there are no extents for csum range 0-69632 > >>> csum exists for 0-69632 but there is no extent record > >>> ... > >>> ... > >>> there are no extents for csum range 946692096-946827264 > >>> csum exists for 946692096-946827264 but there is no extent record > >>> there are no extents for csum range 946831360-947912704 > >>> csum exists for 946831360-947912704 but there is no extent record > >>> ERROR: errors found in csum tree > >> > >> Only extent tree is corrupted. > >> > >> Normally btrfs check --init-csum-tree should be able to handle it. > >> > >> But still, please be sure you're using the latest btrfs-progs to fix it. > >> > >> Thanks, > >> Qu > >> > >>> [6/7] checking root refs > >>> [7/7] checking quota groups skipped (not enabled on this FS) > >>> found 44157956026368 bytes used, error(s) found > >>> total csum bytes: 42038602716 > >>> total tree bytes: 49688616960 > >>> total fs tree bytes: 1256427520 > >>> total extent tree bytes: 1709105152 > >>> btree space waste bytes: 3172727316 > >>> file data blocks allocated: 261625653436416 > >>> referenced 47477768499200 > >>> > >>> What do I need to do to fix all of this? > >>> > >>> On Thu, May 7, 2020 at 1:52 AM Qu Wenruo <quwenruo.btrfs@xxxxxxx> wrote: > >>>> > >>>> > >>>> > >>>> On 2020/5/7 下午1:43, Tyler Richmond wrote: > >>>>> Well, the repair doesn't look terribly successful. > >>>>> > >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224 > >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224 > >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224 > >>>>> Ignoring transid failure > >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84 > >>>>> parent level=1 > >>>>> child level=4 > >>>> > >>>> This means there are more problems, not only the hash name mismatch. > >>>> > >>>> This means the fs is already corrupted, the name hash is just one > >>>> unrelated symptom. > >>>> > >>>> The only good news is, btrfs-progs abort the transaction, thus no > >>>> further damage to the fs. > >>>> > >>>> Please run a plain btrfs-check to show what's the problem first. > >>>> > >>>> Thanks, > >>>> Qu > >>>> > >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224 > >>>>> Ignoring transid failure > >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84 > >>>>> parent level=1 > >>>>> child level=4 > >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224 > >>>>> Ignoring transid failure > >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84 > >>>>> parent level=1 > >>>>> child level=4 > >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224 > >>>>> Ignoring transid failure > >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84 > >>>>> parent level=1 > >>>>> child level=4 > >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224 > >>>>> Ignoring transid failure > >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84 > >>>>> parent level=1 > >>>>> child level=4 > >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224 > >>>>> Ignoring transid failure > >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84 > >>>>> parent level=1 > >>>>> child level=4 > >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224 > >>>>> Ignoring transid failure > >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84 > >>>>> parent level=1 > >>>>> child level=4 > >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224 > >>>>> Ignoring transid failure > >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84 > >>>>> parent level=1 > >>>>> child level=4 > >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224 > >>>>> Ignoring transid failure > >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84 > >>>>> parent level=1 > >>>>> child level=4 > >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224 > >>>>> Ignoring transid failure > >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84 > >>>>> parent level=1 > >>>>> child level=4 > >>>>> parent transid verify failed on 218620880703488 wanted 6875841 found 6876224 > >>>>> Ignoring transid failure > >>>>> ERROR: child eb corrupted: parent bytenr=225049956061184 item=84 > >>>>> parent level=1 > >>>>> child level=4 > >>>>> ERROR: failed to zero log tree: -17 > >>>>> ERROR: attempt to start transaction over already running one > >>>>> WARNING: reserved space leaked, flag=0x4 bytes_reserved=4096 > >>>>> extent buffer leak: start 225049066086400 len 4096 > >>>>> extent buffer leak: start 225049066086400 len 4096 > >>>>> WARNING: dirty eb leak (aborted trans): start 225049066086400 len 4096 > >>>>> extent buffer leak: start 225049066094592 len 4096 > >>>>> extent buffer leak: start 225049066094592 len 4096 > >>>>> WARNING: dirty eb leak (aborted trans): start 225049066094592 len 4096 > >>>>> extent buffer leak: start 225049066102784 len 4096 > >>>>> extent buffer leak: start 225049066102784 len 4096 > >>>>> WARNING: dirty eb leak (aborted trans): start 225049066102784 len 4096 > >>>>> extent buffer leak: start 225049066131456 len 4096 > >>>>> extent buffer leak: start 225049066131456 len 4096 > >>>>> WARNING: dirty eb leak (aborted trans): start 225049066131456 len 4096 > >>>>> > >>>>> What is going on? > >>>>> > >>>>> On Wed, May 6, 2020 at 9:30 PM Tyler Richmond <t.d.richmond@xxxxxxxxx> wrote: > >>>>>> > >>>>>> Chris, I had used the correct mountpoint in the command. I just edited > >>>>>> it in the email to be /mountpoint for consistency. > >>>>>> > >>>>>> Qu, I'll try the repair. Fingers crossed! > >>>>>> > >>>>>> On Wed, May 6, 2020 at 9:13 PM Qu Wenruo <quwenruo.btrfs@xxxxxxx> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On 2020/5/7 上午5:54, Tyler Richmond wrote: > >>>>>>>> Hello, > >>>>>>>> > >>>>>>>> I looked up this error and it basically says ask a developer to > >>>>>>>> determine if it's a false error or not. I just started getting some > >>>>>>>> slow response times, and looked at the dmesg log to find a ton of > >>>>>>>> these errors. > >>>>>>>> > >>>>>>>> [192088.446299] BTRFS critical (device sdh): corrupt leaf: root=5 > >>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation: > >>>>>>>> has 18446744073709551492 expect [0, 6875827] > >>>>>>>> [192088.449823] BTRFS error (device sdh): block=203510940835840 read > >>>>>>>> time tree block corruption detected > >>>>>>>> [192088.459238] BTRFS critical (device sdh): corrupt leaf: root=5 > >>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation: > >>>>>>>> has 18446744073709551492 expect [0, 6875827] > >>>>>>>> [192088.462773] BTRFS error (device sdh): block=203510940835840 read > >>>>>>>> time tree block corruption detected > >>>>>>>> [192088.464711] BTRFS critical (device sdh): corrupt leaf: root=5 > >>>>>>>> block=203510940835840 slot=4 ino=1311670, invalid inode generation: > >>>>>>>> has 18446744073709551492 expect [0, 6875827] > >>>>>>>> [192088.468457] BTRFS error (device sdh): block=203510940835840 read > >>>>>>>> time tree block corruption detected > >>>>>>>> > >>>>>>>> btrfs device stats, however, doesn't show any errors. > >>>>>>>> > >>>>>>>> Is there anything I should do about this, or should I just continue > >>>>>>>> using my array as normal? > >>>>>>> > >>>>>>> This is caused by older kernel underflow inode generation. > >>>>>>> > >>>>>>> Latest btrfs-progs can fix it, using btrfs check --repair. > >>>>>>> > >>>>>>> Or you can go safer, by manually locating the inode using its inode > >>>>>>> number (1311670), and copy it to some new location using previous > >>>>>>> working kernel, then delete the old file, copy the new one back to fix it. > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Qu > >>>>>>> > >>>>>>>> > >>>>>>>> Thank you! > >>>>>>>> > >>>>>>> > >>>> > >> >
