On Fri, Mar 15, 2019 at 07:42:21PM +0100, Jakub Husák wrote:
> Thanks for explanation! actually when I moved forward with the rebalancing
> the fourth disk started to receive some data.
>
> BTW, I was hoping some filter like '-dstripes=1..3' existed and it is!
> Wouldn't it deserve some documentation? :)
It has some, from the man page for btrfs-balance:
stripes=<range>
Balance only block groups which have the given number of stripes. The parameter is a range specified as start..end.
Makes sense for block group profiles that utilize striping, ie. RAID0/10/5/6. The range minimum and maximum are
inclusive.
There are probably some wikis that could benefit from a sentence or
two explaining when you'd use this option. Or a table of which RAID
profiles must be balanced after a device add (always raid0, raid5,
raid6, sometimes raid1 and raid10) and which don't (never single, dup,
sometimes raid1 and raid10).
> Also thanks to Noah Massey for caring!
>
> Cheers
>
>
> On 15. 03. 19 19:01, Zygo Blaxell wrote:
> > On Wed, Mar 13, 2019 at 11:11:02PM +0100, Jakub Husák wrote:
> > > Sorry, fighting with this technology called "email" :)
> > >
> > >
> > > Hopefully better wrapped outputs:
> > >
> > > On 13. 03. 19 22:58, Jakub Husák wrote:
> > >
> > >
> > > > Hi,
> > > >
> > > > I added another disk to my 3-disk raid5 and ran a balance command. After
> > > > few hours I looked to output of `fi usage` to see that no data are being
> > > > used on the new disk. I got the same result even when balancing my raid5
> > > > data or metadata.
> > > >
> > > > Next I tried to convert my raid5 metadata to raid1 (a good idea anyway)
> > > > and the new disk started to fill immediately (even though it received
> > > > the whole amount of metadata with replicas being spread among the other
> > > > drives, instead of being really "balanced". I know why this happened, I
> > > > don't like it but I can live with it, let's not go off topic here :)).
> > > >
> > > > Now my usage output looks like this:
> > > >
> > > # btrfs filesystem usage /mnt/data1
> > > WARNING: RAID56 detected, not implemented
> > > Overall:
> > > Device size: 10.91TiB
> > > Device allocated: 316.12GiB
> > > Device unallocated: 10.61TiB
> > > Device missing: 0.00B
> > > Used: 58.86GiB
> > > Free (estimated): 0.00B (min: 8.00EiB)
> > > Data ratio: 0.00
> > > Metadata ratio: 2.00
> > > Global reserve: 512.00MiB (used: 0.00B)
> > >
> > > Data,RAID5: Size:4.59TiB, Used:4.06TiB
> > > /dev/mapper/crypt-sdb 2.29TiB
> > > /dev/mapper/crypt-sdc 2.29TiB
> > > /dev/mapper/crypt-sde 2.29TiB
> > >
> > > Metadata,RAID1: Size:158.00GiB, Used:29.43GiB
> > > /dev/mapper/crypt-sdb 53.00GiB
> > > /dev/mapper/crypt-sdc 53.00GiB
> > > /dev/mapper/crypt-sdd 158.00GiB
> > > /dev/mapper/crypt-sde 52.00GiB
> > >
> > > System,RAID1: Size:64.00MiB, Used:528.00KiB
> > > /dev/mapper/crypt-sdc 32.00MiB
> > > /dev/mapper/crypt-sdd 64.00MiB
> > > /dev/mapper/crypt-sde 32.00MiB
> > >
> > > Unallocated:
> > > /dev/mapper/crypt-sdb 393.04GiB
> > > /dev/mapper/crypt-sdc 393.01GiB
> > > /dev/mapper/crypt-sdd 2.57TiB
> > > /dev/mapper/crypt-sde 394.01GiB
> > >
> > > > I'm now running `fi balance -dusage=10` (and rising the usage limit). I
> > > > can see that the unallocated space is rising as it's freeing the little
> > > > used chunks but still no data are being stored on the new disk.
> > That is exactly what is happening: you are moving tiny amounts of data
> > into existing big empty spaces, so no new chunk allocations (which should
> > use the new drive) are happening. You have 470GB of data allocated
> > but not used, so you have up to 235 block groups to fill before the new
> > drive gets any data.
> >
> > Also note that you always have to do a full data balance when adding
> > devices to raid5 in order to make use of all the space, so you might
> > as well get started on that now. It'll take a while. 'btrfs balance
> > start -dstripes=1..3 /mnt/data1' will work for this case.
> >
> > > > I it some bug? Is `fi usage` not showing me something (as it states
> > > > "WARNING: RAID56 detected, not implemented")?
> > The warning just means the fields in the 'fi usage' output header,
> > like "Free (estimate)", have bogus values because they're not computed
> > correctly.
> >
> > > > Or is there just too much
> > > > free space on the first set of disks that the balancing is not bothering
> > > > moving any data?
> > Yes. ;)
> >
> > > > If so, shouldn't it be really balancing (spreading) the data among all
> > > > the drives to use all the IOPS capacity, even when the raid5 redundancy
> > > > constraint is currently satisfied?
> > btrfs divides the disks into chunks first, then spreads the data across
> > the chunks. The chunk allocation behavior spreads chunks across all the
> > disks. When you are adding a disk to raid5, you have to redistribute all
> > the old data across all the disks to get balanced IOPS and space usage,
> > hence the full balance requirement.
> >
> > If you don't do a full balance, it will eventually allocate data on
> > all disks, but it will run out of space on sdb, sdc, and sde first,
> > and then be unable to use the remaining 2TB+ on sdd.
> >
> > > # uname -a
> > > Linux storage 4.19.0-0.bpo.2-amd64 #1 SMP Debian 4.19.16-1~bpo9+1
> > > (2019-02-07) x86_64 GNU/Linux
> > > # btrfs --version
> > > btrfs-progs v4.17
> > > # btrfs fi show
> > > Label: none uuid: xxxxxxxxxxxxxxxxx
> > > Total devices 4 FS bytes used 4.09TiB
> > > devid 2 size 2.73TiB used 2.34TiB path /dev/mapper/crypt-sdc
> > > devid 3 size 2.73TiB used 2.34TiB path /dev/mapper/crypt-sdb
> > > devid 4 size 2.73TiB used 2.34TiB path /dev/mapper/crypt-sde
> > > devid 5 size 2.73TiB used 158.06GiB path /dev/mapper/crypt-sdd
> > >
> > > # btrfs fi df .
> > > Data, RAID5: total=4.59TiB, used=4.06TiB
> > > System, RAID1: total=64.00MiB, used=528.00KiB
> > > Metadata, RAID1: total=158.00GiB, used=29.43GiB
> > > GlobalReserve, single: total=512.00MiB, used=0.00B
> > >
> > > > Thanks
> > > >
> > > > Jakub
> > > >
