btrfs balance on mixed capacity drives

[CHerSun]

Hi. Sorry if this is a duplicate, but I wasn't able to find issue on this.

I'm using RAID1 on 5 drives, all of the drives were 3 TB. Recently I've had to switch a failed drive to a 4 TB one. And only then I noted, that btrfs balances based on unallocated space on the drive and not on percentage used (used/total) of the drive. Between 3 and 4 TB drives that won't cause too large skew, but googling I've noted cases where ppl use configurations like 1+1+3+3 drives, and there this causes a huge skew, forcing to basically 2 disks until ~2 TB is written. I'm considering to upgrade to ~18 TB drives one by one - then I will get this skew too.

So the question is if it's possible to switch to percentage-based balancing. If not - could that be added? Is there any doc where I could read more on this decision?

[Forza-tng]

Btrfs currently uses the disk with most unallocated space first, given the constraints of the active profile. You can determine how much is usable across all devices using the btrfs disk space calculator https://carfax.org.uk/btrfs-usage/?c=2&slo=1&shi=1&p=0&dg=1&d=4&d=3&d=3&d=3. With 3+3+3+4 TiB, 1TiB will not be usable with a RAID1 profile. With 4x3TiB + 1x4TiB all space will be usable.

You can reduce the usable size of the new disk to 3TiB by using:

btrfs filesystem resize <id>:<size> <mount-point>

You can list device id using btrfs fi us -T <mnt>.

Just remember that btrfs won't reclaim the 1TiB slack until you resize the filesystem again. You can use <id>:max for that.

[CHerSun]

You can reduce the usable size of the 4th disk to 3TiB by using:

Thank you for idea, but well, that's not why one would increase drives capacity and use btrfs, quite the opposite.

[Forza-tng]

You can reduce the usable size of the 4th disk to 3TiB by using:

Thank you for idea, but well, that's not why one would increase drives capacity and use btrfs, quite the opposite.

Well, perhaps i misunderstood. Btrfs only uses 2 mirrors for any allocation with RAID1. So more disks will not make things faster or more resilient. With RAID10, Btrfs would use stripes across devices, potentially making sequential reads/writes faster.

Some related information:
https://btrfs.readthedocs.io/en/latest/mkfs.btrfs.html#profiles
https://wiki.tnonline.net/w/Btrfs/Profiles

[CHerSun]

Well, perhaps i misunderstood. Btrfs only uses 2 mirrors for any allocation with RAID1. So more disks will not make things faster or more resilient. With RAID10, Btrfs would use stripes across devices, potentially making sequential reads/writes faster.

Yep, 2 mirrors for single block, but next block can use 2 mirrors on different disks, etc. So more disks = more speed, especially for larger transfers (considering 1GiB allocations) or concurrent accesses. Placing all of allocations on larger disks gives a huge skew to both possible performance and drives wear. That's why most hardware storages, that do allow mixing capacities normally do distrubution of allocations based on % - to utilize all of them more evenly.

Just to illustrate, let's say I'd still have 3x3 TiB and replace 2 drives with say 10 TiB drives. With 7 TiB of actual data in RAID1 btrfs would distribute data like this currently:

disk	capacity	data usage
1	3 TiB	0 TiB
2	3 TiB	0 TiB
3	3 TiB	0 TiB
4	10 TiB	7 TiB
5	10 TiB	7 TiB

putting all the workload on the 2 drives.

What % based-distribution would do:

disk	capacity	data usage
1	3 TiB	1.45 TiB
2	3 TiB	1.45 TiB
3	3 TiB	1.45 TiB
4	10 TiB	4.8 TiB
5	10 TiB	4.8 TiB

[Forza-tng]

It would be a good option, had it existed. For now, the easiest way to do this is to use slack, and then reduce as needed.