Jason, I am not a btrfs developer but I had he same problem as you. In my case the problem went away when I used the mount option to clear the free space cache. From my own experience, whatever is going wrong that causes the checksum error also corrupts this cache but that does no long term harm as, once it is cleared on mount, it gets rebuilt. Steve. On Sun, 29 Mar 2020 at 15:15, Jason Clara <jason@xxxxxxxxxxxxxxx> wrote: > > I had a previous post about when trying to do a device delete that it would cause my whole system to hang. I seem to have got past that issue. > > For that, it seems like even though all the SCRUBs finished without any errors I still had a problem with some files. By forcing a read of every single file I was able to detect the bad files in DMESG. Not sure though why SCRUB didn’t detect this. > BTRFS warning (device sdd1): csum failed root 5 ino 14654354 off 163852288 csum 0 > > > But now when I attempt to delete a device from the array it seems to get stuck. Normally it will show in the log that it has found some extents and then another message saying they were relocated. > > But for the last few days it has just been repeating the same found value and never relocating anything, and the usage of the device doesn’t change at all. > > This line has now been repeating for more then 24 hours, and the previous attempt was similar. > [Sun Mar 29 09:59:50 2020] BTRFS info (device sdd1): found 133 extents > > Prior to this run I had tried with an earlier kernel (5.5.10) and had the same results. It starts with finding and then relocating, but then relocating. So I upgraded my kernel to see if that would help, and it has not. > > System Info > Ubuntu 18.04 > btrfs-progs v5.4.1 > Linux FileServer 5.5.13-050513-generic #202003251631 SMP Wed Mar 25 16:35:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux > > DEVICE USAGE > /dev/sdd1, ID: 1 > Device size: 2.73TiB > Device slack: 0.00B > Data,RAID6: 188.67GiB > Data,RAID6: 1.68TiB > Data,RAID6: 888.43GiB > Unallocated: 1.00MiB > > /dev/sdb1, ID: 2 > Device size: 2.73TiB > Device slack: 2.73TiB > Data,RAID6: 188.67GiB > Data,RAID6: 508.82GiB > Data,RAID6: 2.00GiB > Unallocated: -699.50GiB > > /dev/sdc1, ID: 3 > Device size: 2.73TiB > Device slack: 0.00B > Data,RAID6: 188.67GiB > Data,RAID6: 1.68TiB > Data,RAID6: 888.43GiB > Unallocated: 1.00MiB > > /dev/sdi1, ID: 5 > Device size: 2.73TiB > Device slack: 1.36TiB > Data,RAID6: 188.67GiB > Data,RAID6: 1.18TiB > Unallocated: 1.00MiB > > /dev/sdh1, ID: 6 > Device size: 4.55TiB > Device slack: 0.00B > Data,RAID6: 188.67GiB > Data,RAID6: 1.68TiB > Data,RAID6: 1.23TiB > Data,RAID6: 888.43GiB > Data,RAID6: 2.00GiB > Metadata,RAID1: 2.00GiB > Unallocated: 601.01GiB > > /dev/sda1, ID: 7 > Device size: 7.28TiB > Device slack: 0.00B > Data,RAID6: 188.67GiB > Data,RAID6: 1.68TiB > Data,RAID6: 1.23TiB > Data,RAID6: 888.43GiB > Data,RAID6: 2.00GiB > Metadata,RAID1: 2.00GiB > System,RAID1: 32.00MiB > Unallocated: 3.32TiB > > /dev/sdf1, ID: 8 > Device size: 7.28TiB > Device slack: 0.00B > Data,RAID6: 188.67GiB > Data,RAID6: 1.68TiB > Data,RAID6: 1.23TiB > Data,RAID6: 888.43GiB > Data,RAID6: 2.00GiB > Metadata,RAID1: 8.00GiB > Unallocated: 3.31TiB > > /dev/sdj1, ID: 9 > Device size: 7.28TiB > Device slack: 0.00B > Data,RAID6: 188.67GiB > Data,RAID6: 1.68TiB > Data,RAID6: 1.23TiB > Data,RAID6: 888.43GiB > Data,RAID6: 2.00GiB > Metadata,RAID1: 8.00GiB > System,RAID1: 32.00MiB > Unallocated: 3.31TiB > > > FI USAGE > WARNING: RAID56 detected, not implemented > Overall: > Device size: 33.20TiB > Device allocated: 20.06GiB > Device unallocated: 33.18TiB > Device missing: 0.00B > Used: 19.38GiB > Free (estimated): 0.00B (min: 8.00EiB) > Data ratio: 0.00 > Metadata ratio: 2.00 > Global reserve: 512.00MiB (used: 0.00B) > > Data,RAID6: Size:15.42TiB, Used:15.18TiB (98.44%) > /dev/sdd1 2.73TiB > /dev/sdb1 699.50GiB > /dev/sdc1 2.73TiB > /dev/sdi1 1.36TiB > /dev/sdh1 3.96TiB > /dev/sda1 3.96TiB > /dev/sdf1 3.96TiB > /dev/sdj1 3.96TiB > > Metadata,RAID1: Size:10.00GiB, Used:9.69GiB (96.90%) > /dev/sdh1 2.00GiB > /dev/sda1 2.00GiB > /dev/sdf1 8.00GiB > /dev/sdj1 8.00GiB > > System,RAID1: Size:32.00MiB, Used:1.19MiB (3.71%) > /dev/sda1 32.00MiB > /dev/sdj1 32.00MiB > > Unallocated: > /dev/sdd1 1.00MiB > /dev/sdb1 -699.50GiB > /dev/sdc1 1.00MiB > /dev/sdi1 1.00MiB > /dev/sdh1 601.01GiB > /dev/sda1 3.32TiB > /dev/sdf1 3.31TiB > /dev/sdj1 3.31TiB > > > FI SHOW > Label: 'Pool1' uuid: 99935e27-4922-4efa-bf76-5787536dd71f > Total devices 8 FS bytes used 15.19TiB > devid 1 size 2.73TiB used 2.73TiB path /dev/sdd1 > devid 2 size 0.00B used 699.50GiB path /dev/sdb1 > devid 3 size 2.73TiB used 2.73TiB path /dev/sdc1 > devid 5 size 1.36TiB used 1.36TiB path /dev/sdi1 > devid 6 size 4.55TiB used 3.96TiB path /dev/sdh1 > devid 7 size 7.28TiB used 3.96TiB path /dev/sda1 > devid 8 size 7.28TiB used 3.97TiB path /dev/sdf1 > devid 9 size 7.28TiB used 3.97TiB path /dev/sdj1 > > FI DF > Data, RAID6: total=15.42TiB, used=15.18TiB > System, RAID1: total=32.00MiB, used=1.19MiB > Metadata, RAID1: total=10.00GiB, used=9.69GiB > GlobalReserve, single: total=512.00MiB, used=0.00B
