feat: squashing environments into archives

[MilesCranmer]

Problem description

So, I am using pixi on an institutional filesystem that places strict limits on the number of files. It limits files to 1M in total, which gets quickly overran, since I have a different env for every global tool install. To deal with this, I usually take an overkill approach and put my entire .pixi folder into a .tar archive, and unpack it onto /dev/shm whenever I'm working.

But this is inconvenient and constraining, so I've been looking into other approaches. I was wondering if there is any way I can automatically mount and unmount a SquashFS per environment, triggered upon activation?

For example, I could do:

mksquashfs ~/.pixi/envs/vllm ~/.pixi/envs/vllm.sqsh -comp zstd -Xcompression-level 0 -noappend

And then mounting this onto my /dev/shm and symlink it.

But I'm not sure if this is the cleanest approach. It also seems a bit fragile.

Is there a better way I can do this? Perhaps there is some sort of built-in way to squash environments at deactivation?

[wolfv]

There is an approach being discussed over at rattler: conda/rattler#1182 which might be one way to possibly create a "fake" mountable file system without actually creating "files" on disk.

I think we are very excited about this idea, but currently a bit resource constrained to work on it ourselves. As far as I know, an intern at CERN will soon start playing around more with this, and ideally we would also be able to wire this up automagically in pixi!

[wolfv]

PS: this is also quite interesting for pixi-build / rattler-build because we could just log all writes to any files and those would become the new package :)

[baszalmstra]

Would direnv potentially offer a solution here?

[chrisburr]

Indeed the intern has started today so hopefully there will start to be some movement and I have a ton of ideas for how this could be extended.

@MilesCranmer Would using FUSE be an option in your environment?

[MilesCranmer]

I haven't looked into it. My system is a typical university slurm cluster with a module system, based on CentOS. I have no sudo privileges. I can use singularity but can't use docker due to security reasons.

[chrisburr]

I work on similar systems and FUSE is fine on all of them so I expect what I have in mind would work for you.

[MilesCranmer]

Cool!

[MilesCranmer]

Just confirming I can use FUSE on my system 👍