I had a previous post about when trying to do a device delete that it would cause my whole system to hang. I seem to have got past that issue.
For that, it seems like even though all the SCRUBs finished without any errors I still had a problem with some files. By forcing a read of every single file I was able to detect the bad files in DMESG. Not sure though why SCRUB didn’t detect this.
BTRFS warning (device sdd1): csum failed root 5 ino 14654354 off 163852288 csum 0
But now when I attempt to delete a device from the array it seems to get stuck. Normally it will show in the log that it has found some extents and then another message saying they were relocated.
But for the last few days it has just been repeating the same found value and never relocating anything, and the usage of the device doesn’t change at all.
This line has now been repeating for more then 24 hours, and the previous attempt was similar.
[Sun Mar 29 09:59:50 2020] BTRFS info (device sdd1): found 133 extents
Prior to this run I had tried with an earlier kernel (5.5.10) and had the same results. It starts with finding and then relocating, but then relocating. So I upgraded my kernel to see if that would help, and it has not.
System Info
Ubuntu 18.04
btrfs-progs v5.4.1
Linux FileServer 5.5.13-050513-generic #202003251631 SMP Wed Mar 25 16:35:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
DEVICE USAGE
/dev/sdd1, ID: 1
Device size: 2.73TiB
Device slack: 0.00B
Data,RAID6: 188.67GiB
Data,RAID6: 1.68TiB
Data,RAID6: 888.43GiB
Unallocated: 1.00MiB
/dev/sdb1, ID: 2
Device size: 2.73TiB
Device slack: 2.73TiB
Data,RAID6: 188.67GiB
Data,RAID6: 508.82GiB
Data,RAID6: 2.00GiB
Unallocated: -699.50GiB
/dev/sdc1, ID: 3
Device size: 2.73TiB
Device slack: 0.00B
Data,RAID6: 188.67GiB
Data,RAID6: 1.68TiB
Data,RAID6: 888.43GiB
Unallocated: 1.00MiB
/dev/sdi1, ID: 5
Device size: 2.73TiB
Device slack: 1.36TiB
Data,RAID6: 188.67GiB
Data,RAID6: 1.18TiB
Unallocated: 1.00MiB
/dev/sdh1, ID: 6
Device size: 4.55TiB
Device slack: 0.00B
Data,RAID6: 188.67GiB
Data,RAID6: 1.68TiB
Data,RAID6: 1.23TiB
Data,RAID6: 888.43GiB
Data,RAID6: 2.00GiB
Metadata,RAID1: 2.00GiB
Unallocated: 601.01GiB
/dev/sda1, ID: 7
Device size: 7.28TiB
Device slack: 0.00B
Data,RAID6: 188.67GiB
Data,RAID6: 1.68TiB
Data,RAID6: 1.23TiB
Data,RAID6: 888.43GiB
Data,RAID6: 2.00GiB
Metadata,RAID1: 2.00GiB
System,RAID1: 32.00MiB
Unallocated: 3.32TiB
/dev/sdf1, ID: 8
Device size: 7.28TiB
Device slack: 0.00B
Data,RAID6: 188.67GiB
Data,RAID6: 1.68TiB
Data,RAID6: 1.23TiB
Data,RAID6: 888.43GiB
Data,RAID6: 2.00GiB
Metadata,RAID1: 8.00GiB
Unallocated: 3.31TiB
/dev/sdj1, ID: 9
Device size: 7.28TiB
Device slack: 0.00B
Data,RAID6: 188.67GiB
Data,RAID6: 1.68TiB
Data,RAID6: 1.23TiB
Data,RAID6: 888.43GiB
Data,RAID6: 2.00GiB
Metadata,RAID1: 8.00GiB
System,RAID1: 32.00MiB
Unallocated: 3.31TiB
FI USAGE
WARNING: RAID56 detected, not implemented
Overall:
Device size: 33.20TiB
Device allocated: 20.06GiB
Device unallocated: 33.18TiB
Device missing: 0.00B
Used: 19.38GiB
Free (estimated): 0.00B (min: 8.00EiB)
Data ratio: 0.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Data,RAID6: Size:15.42TiB, Used:15.18TiB (98.44%)
/dev/sdd1 2.73TiB
/dev/sdb1 699.50GiB
/dev/sdc1 2.73TiB
/dev/sdi1 1.36TiB
/dev/sdh1 3.96TiB
/dev/sda1 3.96TiB
/dev/sdf1 3.96TiB
/dev/sdj1 3.96TiB
Metadata,RAID1: Size:10.00GiB, Used:9.69GiB (96.90%)
/dev/sdh1 2.00GiB
/dev/sda1 2.00GiB
/dev/sdf1 8.00GiB
/dev/sdj1 8.00GiB
System,RAID1: Size:32.00MiB, Used:1.19MiB (3.71%)
/dev/sda1 32.00MiB
/dev/sdj1 32.00MiB
Unallocated:
/dev/sdd1 1.00MiB
/dev/sdb1 -699.50GiB
/dev/sdc1 1.00MiB
/dev/sdi1 1.00MiB
/dev/sdh1 601.01GiB
/dev/sda1 3.32TiB
/dev/sdf1 3.31TiB
/dev/sdj1 3.31TiB
FI SHOW
Label: 'Pool1' uuid: 99935e27-4922-4efa-bf76-5787536dd71f
Total devices 8 FS bytes used 15.19TiB
devid 1 size 2.73TiB used 2.73TiB path /dev/sdd1
devid 2 size 0.00B used 699.50GiB path /dev/sdb1
devid 3 size 2.73TiB used 2.73TiB path /dev/sdc1
devid 5 size 1.36TiB used 1.36TiB path /dev/sdi1
devid 6 size 4.55TiB used 3.96TiB path /dev/sdh1
devid 7 size 7.28TiB used 3.96TiB path /dev/sda1
devid 8 size 7.28TiB used 3.97TiB path /dev/sdf1
devid 9 size 7.28TiB used 3.97TiB path /dev/sdj1
FI DF
Data, RAID6: total=15.42TiB, used=15.18TiB
System, RAID1: total=32.00MiB, used=1.19MiB
Metadata, RAID1: total=10.00GiB, used=9.69GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
