Builder ZFS dedup

[jpds]

Describe the solution you'd like

I only use my test build machine at home for sporadic large builds I do for nixpkgs changes. I enabled dedup as soon as fast dedup had been released:

• https://klarasystems.com/articles/introducing-openzfs-fast-dedup/
• https://klarasystems.com/webinars/fast-dedup-with-zfs-smarter-storage-for-modern-workloads/ # video here where various CI usecases are discussed

Even for the small number of builds I do, ZFS dedup seems to return some good results for the Nix store:

$ zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool   928G  99.1G   829G        -         -    14%    10%  1.32x    ONLINE  -
$ zpool status -D
  pool: rpool
 state: ONLINE
config:

	NAME                                                            STATE     READ WRITE CKSUM
	rpool                                                           ONLINE       0     0     0
	  disk                                                          ONLINE       0     0     0

errors: No known data errors

 dedup: DDT entries 2593277, size 1.19G on disk, 941M in core

bucket              allocated                       referenced
______   ______________________________   ______________________________
refcnt   blocks   LSIZE   PSIZE   DSIZE   blocks   LSIZE   PSIZE   DSIZE
------   ------   -----   -----   -----   ------   -----   -----   -----
     1    1.97M    128G   61.4G   63.7G    1.97M    128G   61.4G   63.7G
     2     306K   11.6G   5.68G   6.14G     681K   24.7G   11.9G   12.9G
     4    96.1K   2.46G    921M   1.05G     471K   11.9G   4.32G   5.05G
     8    84.4K   1.09G    853M   1.00G     945K   11.5G   8.94G   10.9G
    16    24.4K   99.7M   56.2M    110M     503K   1.88G   1.09G   2.20G
    32    3.43K   11.5M   8.38M   14.8M     140K    470M    342M    604M
    64      140   1.74M    422K    720K    11.2K    151M   34.0M   58.4M
   128       37    143K    108K    200K    6.55K   26.4M   19.8M   35.9M
   256        8     16K   8.50K     32K    2.95K   5.59M   3.01M   11.8M
   512        9   27.5K     13K     36K    6.57K   23.0M   10.4M   26.3M
    1K        5      9K      9K     24K    6.83K   12.6M   12.6M   32.9M
   16K        1    512B    512B      4K    16.6K   8.28M   8.28M   66.3M
 Total    2.47M    144G   68.9G   72.0G    4.70M    179G   88.1G   95.6G

$ sudo zdb -DD rpool
...
dedup = 1.33, compress = 2.03, copies = 1.09, dedup * compress / copies = 2.49

Might be worth enabling this on a builder and seeing what savings that gives.

[vcunat]

I'm not sure if we have a problem with that. Not much point trying to cram a lot into nix store on a builder, I'd say.

[jpds]

This wasn't a solution to a problem - but rather optimization/new technology that's trivial to implement. People generally shun ZFS dedup because of the extreme CPU/memory requirements the previous iteration had.

That said - I hadn't even looked at the Grafana instance before suggesting this - and there are builders which are less than a percentage point from the infamous "ZFS performance cliff": https://www.45drives.com/community/articles/zfs-80-percent-rule/

[mweinelt]

https://despairlabs.com/blog/posts/2024-10-27-openzfs-dedup-is-good-dont-use-it/

[jpds]

@mweinelt I read that blog post before and it basically boils down to "there are pros and cons to any technology, know your usecase".

https://www.youtube.com/watch?v=aEnqDSlKagE (19:00 onwards) is a good video where CI workloads have been shown to work well with dedup.

[vcunat]

The space on these builders is temporary. If we collect a thing, it won't be rebuilt but just sent from another place. On the other hand, CPU and sometimes RAM can be scarce on these machines. So I'm skeptical about the tradeoff for this use case.