On Thu, Sep 19, 2019 at 11:38 AM Barnes, Samuel <SABarnes@xxxxxxx> wrote: > > I have a btrfs drive that started have problems, I first noticed some of my old backup directories couldn't be deleted, the automated delete commands removed all the files, but got stuck on the directories. Now if I try to delete I get input/output error: > > rm -rf 20190806-020001-412/ > rm: cannot remove '20190806-020001-412/backup/media/network_mriphysics/HAT_data/Phantoms/Penn State Coll Time1': Input/output error > > dmesg shows the following errors: > > [11655.022724] BTRFS critical (device sdc1): corrupt leaf: root=5 block=1399280648192 slot=69 ino=8904454, name hash mismatch with key, have 0x00000000d8649173 expect 0x00000000f88d2ac3 [11655.027629] BTRFS critical (device sdc1): corrupt leaf: root=5 block=1399280648192 slot=69 ino=8904454, name hash mismatch with key, have 0x00000000d8649173 expect 0x00000000f88d2ac3 [11655.027990] BTRFS critical (device sdc1): corrupt leaf: root=5 block=1399280648192 slot=69 ino=8904454, name hash mismatch with key, have 0x00000000d8649173 expect 0x00000000f88d2ac3 [11655.028311] BTRFS critical (device sdc1): corrupt leaf: root=5 block=1399280648192 slot=69 ino=8904454, name hash mismatch with key, have 0x00000000d8649173 expect 0x00000000f88d2ac3 > What do you get for: # btrfs insp dump-t -b 1399280648192 /dev/sdc1 This likely exposes filenames, it's OK to remove them. > I have tried a scrub, no errors: > > sudo ./btrfs scrub status /dev/sdc1 > UUID: d462aea8-a4e8-47a0-b5e7-ab8ec91af82a > Scrub started: Tue Sep 17 10:43:20 2019 > Status: finished > Duration: 1:46:29 > Total to scrub: 1.18TiB > Rate: 193.46MiB/s > Error summary: no errors found > > a btrfs check gives a very very long list of errors most include "link count wrong" such as: > > root 5 inode 5202109 errors 2000, link count wrong > unresolved ref dir 7424552 index 19 namelen 9 name 12-25.dcm filetype 0 errors 3, no dir item, no dir index > unresolved ref dir 8454942 index 19 namelen 9 name 12-25.dcm filetype 0 errors 3, no dir item, no dir index > unresolved ref dir 8616651 index 19 namelen 9 name 12-25.dcm filetype 0 errors 3, no dir item, no dir index > > another example: > > root 5 inode 7422709 errors 2001, no inode item, link count wrong > unresolved ref dir 261 index 182 namelen 19 name 20190630-020001-715 filetype 2 errors 4, no inode ref > > I ran a balance, that didn't help, and interestingly if I run check --repair it aborts: > > sudo ./btrfs check --repair /dev/sdc1 > enabling repair mode > Opening filesystem to check... > Checking filesystem on /dev/sdc1 > UUID: d462aea8-a4e8-47a0-b5e7-ab8ec91af82a > [1/7] checking root items > Fixed 0 roots. > [2/7] checking extents > incorrect offsets 9191 67118055 > incorrect offsets 9191 67118055 > incorrect offsets 9191 67118055 > incorrect offsets 9191 67118055 > items overlap, can't fix > check/main.c:4265: fix_item_offset: BUG_ON `ret` triggered, value -5 ./btrfs(+0x5e2f0)[0x557b1e3c62f0] ./btrfs(+0x5e396)[0x557b1e3c6396] ./btrfs(+0x6951b)[0x557b1e3d151b] ./btrfs(+0x6a45a)[0x557b1e3d245a] ./btrfs(+0x6cf09)[0x557b1e3d4f09] ./btrfs(main+0x95)[0x557b1e37c690] > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f7aec452b97] > ./btrfs(_start+0x2a)[0x557b1e37c1ea] > Aborted That's a bug. I think overall the file system is probably fine, but there's an older kernel bug that caused this leaf to become corrupt at some point, and increasingly stronger checking is what's exposing it now. So the trick is really how to delete the offending file, because right now it looks like that isn't being allowed. Qu might have an idea. -- Chris Murphy
