When --rootdir is used, create the loopback file if it does not exist

[DaanDeMeyer]

I'd like to create minimal btrfs filesystems using --rootdir. Currently, I have to provide the file backed image myself which means I have to come up with a good size for it so that all the files specified with --rootdir will fit. Coming up with the size myself is very hard as I don't have enough information available to know how much space btrfs will need to store the files from --rootdir. I'd like for mkfs.btrfs to create the loopback file if it does not exist and make sure it is just large enough to store all the files from --rootdir.

[adam900710]

mkfs.btrfs already has the --shrink option to shrink the loopback file.

So you don't really need to figure out the proper size by yourself, just pass a large enough loopback file, and call mkfs.btrfs --shrink --rootdir <dir>.

[DaanDeMeyer]

That doesn't actually work, depending on the size of the loopback file, you'll get a bigger or smaller btrfs filesystem after shrink. I want the most minimal filesystem regardless of how large the loopback file is.

Example:

➜  systemd git:(repart-copy-deny-list) mkdir abc
➜  systemd git:(repart-copy-deny-list) truncate -s 1T abc.raw
➜  systemd git:(repart-copy-deny-list) mkfs.btrfs --rootdir abc abc.raw --shrink
btrfs-progs v5.18
See http://btrfs.wiki.kernel.org for more information.

NOTE: several default settings have changed in version 5.15, please make sure
      this does not affect your deployments:
      - DUP for metadata (-m dup)
      - enabled no-holes (-O no-holes)
      - enabled free-space-tree (-R free-space-tree)

Making image is completed.
Label:              (null)
UUID:               8c7734c1-2a81-40f7-a0b6-2150fb9b103c
Node size:          16384
Sector size:        4096
Filesystem size:    2.04GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP               1.00GiB
  System:           DUP               8.00MiB
SSD detected:       no
Zoned device:       no
Incompat features:  extref, skinny-metadata, no-holes
Runtime features:   free-space-tree
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     2.04GiB  abc.raw

[adam900710]

OK, got your point.

The --shrink option only works at chunk level (aka, just shrink the fs to have no unallocated space).

But if your initial image is super large, then we will alloc much larger chunks (according to the allocation policy), resulting larger than expected chunks, thus larger image (2G in your case, mostly caused by that almost empty 1G DUP metadata chunk)

I guess we may try something like iterating the rootdir to get a basic estimation (will have extra buffer room for metadata), then create a loopback file, then go the regular mkfs path.

For now, I believe it's easier to craft a small script to run du -s on the rootdir, do some estimation, create the target file, then go mkfs on it.
As the estimation part can be somewhat complex, with a lot of corner cases, thus it may not be implemented any time soon.

[DaanDeMeyer]

My current workaround (for all filesystems) is to populate once in a 1T sparse file, figure out the actual used size, multiple that by 1.5 and populate again using that size. But I'd still much prefer if filesystems did the estimation for me.