On Wed, Mar 13, 2019 at 11:11:02PM +0100, Jakub Husák wrote:
Sorry, fighting with this technology called "email" :)
Hopefully better wrapped outputs:
On 13. 03. 19 22:58, Jakub Husák wrote:
Hi,
I added another disk to my 3-disk raid5 and ran a balance command. After
few hours I looked to output of `fi usage` to see that no data are being
used on the new disk. I got the same result even when balancing my raid5
data or metadata.
Next I tried to convert my raid5 metadata to raid1 (a good idea anyway)
and the new disk started to fill immediately (even though it received
the whole amount of metadata with replicas being spread among the other
drives, instead of being really "balanced". I know why this happened, I
don't like it but I can live with it, let's not go off topic here :)).
Now my usage output looks like this:
# btrfs filesystem usage /mnt/data1
WARNING: RAID56 detected, not implemented
Overall:
Device size: 10.91TiB
Device allocated: 316.12GiB
Device unallocated: 10.61TiB
Device missing: 0.00B
Used: 58.86GiB
Free (estimated): 0.00B (min: 8.00EiB)
Data ratio: 0.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Data,RAID5: Size:4.59TiB, Used:4.06TiB
/dev/mapper/crypt-sdb 2.29TiB
/dev/mapper/crypt-sdc 2.29TiB
/dev/mapper/crypt-sde 2.29TiB
Metadata,RAID1: Size:158.00GiB, Used:29.43GiB
/dev/mapper/crypt-sdb 53.00GiB
/dev/mapper/crypt-sdc 53.00GiB
/dev/mapper/crypt-sdd 158.00GiB
/dev/mapper/crypt-sde 52.00GiB
System,RAID1: Size:64.00MiB, Used:528.00KiB
/dev/mapper/crypt-sdc 32.00MiB
/dev/mapper/crypt-sdd 64.00MiB
/dev/mapper/crypt-sde 32.00MiB
Unallocated:
/dev/mapper/crypt-sdb 393.04GiB
/dev/mapper/crypt-sdc 393.01GiB
/dev/mapper/crypt-sdd 2.57TiB
/dev/mapper/crypt-sde 394.01GiB
I'm now running `fi balance -dusage=10` (and rising the usage limit). I
can see that the unallocated space is rising as it's freeing the little
used chunks but still no data are being stored on the new disk.
That is exactly what is happening: you are moving tiny amounts of data
into existing big empty spaces, so no new chunk allocations (which should
use the new drive) are happening. You have 470GB of data allocated
but not used, so you have up to 235 block groups to fill before the new
drive gets any data.
Also note that you always have to do a full data balance when adding
devices to raid5 in order to make use of all the space, so you might
as well get started on that now. It'll take a while. 'btrfs balance
start -dstripes=1..3 /mnt/data1' will work for this case.
I it some bug? Is `fi usage` not showing me something (as it states
"WARNING: RAID56 detected, not implemented")?
The warning just means the fields in the 'fi usage' output header,
like "Free (estimate)", have bogus values because they're not computed
correctly.
Or is there just too much
free space on the first set of disks that the balancing is not bothering
moving any data?
Yes. ;)
If so, shouldn't it be really balancing (spreading) the data among all
the drives to use all the IOPS capacity, even when the raid5 redundancy
constraint is currently satisfied?
btrfs divides the disks into chunks first, then spreads the data across
the chunks. The chunk allocation behavior spreads chunks across all the
disks. When you are adding a disk to raid5, you have to redistribute all
the old data across all the disks to get balanced IOPS and space usage,
hence the full balance requirement.
If you don't do a full balance, it will eventually allocate data on
all disks, but it will run out of space on sdb, sdc, and sde first,
and then be unable to use the remaining 2TB+ on sdd.
# uname -a
Linux storage 4.19.0-0.bpo.2-amd64 #1 SMP Debian 4.19.16-1~bpo9+1
(2019-02-07) x86_64 GNU/Linux
# btrfs --version
btrfs-progs v4.17
# btrfs fi show
Label: none uuid: xxxxxxxxxxxxxxxxx
Total devices 4 FS bytes used 4.09TiB
devid 2 size 2.73TiB used 2.34TiB path /dev/mapper/crypt-sdc
devid 3 size 2.73TiB used 2.34TiB path /dev/mapper/crypt-sdb
devid 4 size 2.73TiB used 2.34TiB path /dev/mapper/crypt-sde
devid 5 size 2.73TiB used 158.06GiB path /dev/mapper/crypt-sdd
# btrfs fi df .
Data, RAID5: total=4.59TiB, used=4.06TiB
System, RAID1: total=64.00MiB, used=528.00KiB
Metadata, RAID1: total=158.00GiB, used=29.43GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
Thanks
Jakub
