auto reclaim storage based on user configuration

[dustymabe]

Podman has nice commands around cleaning up containers/volumes/images that are "unused" meaning they aren't in current use by any containers.

I believe these things that are able to be cleaned up represent the RECLAIMABLE field in the output of podman systemd df:

[dustymabe@media ~]$ podman system df
TYPE           TOTAL       ACTIVE      SIZE        RECLAIMABLE
Images         70          6           50.67GB     38.54GB (76%)
Containers     4           2           2.261GB     812.9MB (36%)
Local Volumes  0           0           0B          0B (0%)

What would be a killer feature is to be able to configure podman to use a certain amount of disk for container storage and when that threshold is met podman will periodically reclaim storage that it can.

Now what happens if we go over the threshold and there is no reclaimable storage available? I think we continue to operate over the threshold, but maybe print a warning for interactive commands or something.

[Luap99]

So basically you would want to run a periodic timer that runs podman system prune when a certain amount of disk space is exceeded? And should that prune only reclaim until it gets below the threshold or as much as possible? I guess as much as possible is better because then the implementation would not need to make a decision on what to prune vs what not.

I have a few design problems with that kind of stuff in general, some of which is mentioned here #25864

But one general thing is that most podman commands are not race free when a prune is run in parallel, i.e. you would expect a podman run $remoteImage` to pull the image and create the container based on that. However if you run prune at the right time between podman run pulled the image but before it was able to create the actual container then prune will remove it as unused and then the creation fails randomly.
And it is not limited to that and script doing things like podman volume create x && podman run --volume x, etc... now can all fail when the prune service is running in parallel. So to me personally one should only prune when they can be sure they have nothing else in parallel going on.

And one other thing if we talk about system prune to prune containers, they get pruned when they are not running. That in turn means the important container that just exited might get pruned. And that then likely means its volumes would get pruned too which means data loss for sure. Maybe I am overthinking this but I really do not want to cause any data loss

Then I wonder if this would need to be integrated into podman at all, we could have a small external script that uses something like findmnt --noheadings -oUSE% -b --target /var/lib/containers and then call podman system prune if the value is exceeded.

[dustymabe]

So basically you would want to run a periodic timer that runs podman system prune when a certain amount of disk space is exceeded?

I think the implementation detail is something we should discuss, but I also think the UI is something we should discuss. I think the UI should be a config file somewhere the user updates. By default no pruning is on by default (so this is opt-in). The opt in is to update a config file (probably one of our already existing ones would suffice), with something like:

cat <<EOF >> configfile
prune_when_disk_usage=20G # or maybe can also accept a percentage of total filesystem ?
prune_period=<duration>  # i.e. how often to check
EOF

Once configured podman takes care of things for us. Systemd timer would probably be what we ended up using under the hood, yes.

And should that prune only reclaim until it gets below the threshold or as much as possible? I guess as much as possible is better because then the implementation would not need to make a decision on what to prune vs what not.

ehh. From the user point of view I think it would be preferable to just get it down to the allowed (configured) usable space). i.e. keeping a cache is useful sometimes so let's keep as much of the cache as we are allowed to and drop the oldest stuff first.

I know this makes things more complicated, but from a user perspective that would be preferable IMO.

Now from a developer/maintainer perspective it certainly makes things more complicated. I don't know how things are structured underneath, but if there was a way to gather all of the things that could be pruned and then iterate over them in chronological order until you get under the usage threshold...

Usually the real world pragmatic solution to the problem is a combination of best for user + best for maintainer.

I have a few design problems with that kind of stuff in general, some of which is mentioned here #25864

But one general thing is that most podman commands are not race free when a prune is run in parallel, i.e. you would expect a podman run $remoteImage` to pull the image and create the container based on that. However if you run prune at the right time between podman run pulled the image but before it was able to create the actual container then prune will remove it as unused and then the creation fails randomly. And it is not limited to that and script doing things like podman volume create x && podman run --volume x, etc... now can all fail when the prune service is running in parallel. So to me personally one should only prune when they can be sure they have nothing else in parallel going on.

And one other thing if we talk about system prune to prune containers, they get pruned when they are not running. That in turn means the important container that just exited might get pruned. And that then likely means its volumes would get pruned too which means data loss for sure. Maybe I am overthinking this but I really do not want to cause any data loss

I think this class of "race condition" problems can be eliminated if we put a simple filter to only consider objects that were created greater than "1 minute ago" or something. We do this already in CoreOS where we only prune things older than 48H ago. But we could also make the until duration configurable.

Then I wonder if this would need to be integrated into podman at all, we could have a small external script that uses something like findmnt --noheadings -oUSE% -b --target /var/lib/containers and then call podman system prune if the value is exceeded.

Ehh. I think from the user's perspective, not having to implement this themselves would be a value add. It doesn't mean we should do it in podman (it's a balance of what's worth it and what is not). It just means from the user's perspective having it be built in would be nice.

[Luap99]

I think the implementation detail is something we should discuss, but I also think the UI is something we should discuss. I think the UI should be a config file somewhere the user updates. By default no pruning is on by default (so this is opt-in). The opt in is to update a config file (probably one of our already existing ones would suffice), with something like:

I mean the size threshold is certainly a podman specific config that could go into containers.conf. But for the time I don't see that practical because we are not a daemon so we would end up having a systemd timer defined and that declared the time. And of course one could add drop ins to change the time as needed.

Now from a developer/maintainer perspective it certainly makes things more complicated. I don't know how things are structured underneath, but if there was a way to gather all of the things that could be pruned and then iterate over them in chronological order until you get under the usage threshold...

I mean of course it can be possible to do that but it would need quite far reaching changes so I don't know if we would try to prune just as much. Of course there would be the question with what would you start to prune, containers, images, volumes, etc...?

I think this class of "race condition" problems can be eliminated if we put a simple filter to only consider objects that were created greater than "1 minute ago" or something. We do this already in CoreOS where we only prune things older than 48H ago. But we could also make the until duration configurable.

This sounds like a good idea, podman image prune as a until filter however looking at podman images output the crreated time of an image is actually the time that we got from the registry. So if I pull a two weeks old an image right now created would be set to two weeks ago. So that is something that would need to be figured out before relying on such time filter.

Ehh. I think from the user's perspective, not having to implement this themselves would be a value add. It doesn't mean we should do it in podman (it's a balance of what's worth it and what is not). It just means from the user's perspective having it be built in would be nice.

Yeah, I tend to try to keep things out of podman as much as possible if they can already be done. Logic wise I don't disagree that having things part of podman proper is always nice as you can rely on them then to be present. Also if the goal is to only prune until the limit is reached then it is of course no longer simply done externally and would need some proper podman integration so that is certainly an argument for doing that.

[dustymabe]

I think the implementation detail is something we should discuss, but I also think the UI is something we should discuss. I think the UI should be a config file somewhere the user updates. By default no pruning is on by default (so this is opt-in). The opt in is to update a config file (probably one of our already existing ones would suffice), with something like:

I mean the size threshold is certainly a podman specific config that could go into containers.conf. But for the time I don't see that practical because we are not a daemon so we would end up having a systemd timer defined and that declared the time. And of course one could add drop ins to change the time as needed.

Yep. At some point a timer would need to be generated (i.e. systemd-generator) based on the config.

Now from a developer/maintainer perspective it certainly makes things more complicated. I don't know how things are structured underneath, but if there was a way to gather all of the things that could be pruned and then iterate over them in chronological order until you get under the usage threshold...

I mean of course it can be possible to do that but it would need quite far reaching changes so I don't know if we would try to prune just as much. Of course there would be the question with what would you start to prune, containers, images, volumes, etc...?

Ideally you'd iterate over each object in the "objects list" (that is of any type, container, image, volume, etc) and in chronlogical order, but that's super wishful thinking based on an outsider's black box view of podman.

More realistically we could come up with a heuristic that approaches this if we wanted. Something like:

• find oldest object
• run prune for each object type with a filter that gives a time range of oldest <-> oldest/2
• repeat if space still exceeded, but now the range is oldest/2 <-> oldest/4

This is just an idea and probably not a great one. But it gives you an idea on how it could be implemented probably without vastly changing the code base. Though the iterations where we slowly get closer to $today would probably mean a performance hit.

I think this class of "race condition" problems can be eliminated if we put a simple filter to only consider objects that were created greater than "1 minute ago" or something. We do this already in CoreOS where we only prune things older than 48H ago. But we could also make the until duration configurable.

This sounds like a good idea, podman image prune as a until filter however looking at podman images output the crreated time of an image is actually the time that we got from the registry. So if I pull a two weeks old an image right now created would be set to two weeks ago. So that is something that would need to be figured out before relying on such time filter.

Agreed. Good catch.

Ehh. I think from the user's perspective, not having to implement this themselves would be a value add. It doesn't mean we should do it in podman (it's a balance of what's worth it and what is not). It just means from the user's perspective having it be built in would be nice.

Yeah, I tend to try to keep things out of podman as much as possible if they can already be done. Logic wise I don't disagree that having things part of podman proper is always nice as you can rely on them then to be present. Also if the goal is to only prune until the limit is reached then it is of course no longer simply done externally and would need some proper podman integration so that is certainly an argument for doing that.

👍

[Luap99]

Ideally you'd iterate over each object in the "objects list" (that is of any type, container, image, volume, etc) and in chronlogical order, but that's super wishful thinking based on an outsider's black box view of podman.

More realistically we could come up with a heuristic that approaches this if we wanted. Something like:

* find oldest object

* run prune for each object type with a filter that gives a time range of `oldest` <-> `oldest/2`

* repeat if space still exceeded, but now the range is `oldest/2` <-> `oldest/4`

This is just an idea and probably not a great one. But it gives you an idea on how it could be implemented probably without vastly changing the code base. Though the iterations where we slowly get closer to $today would probably mean a performance hit.

I think implementation wise that is likely not to bad, we can easily list all resources and the sort base on the time. But then at this point I don't see the point of calling prune anymore (we still should share the is "resource in use" code though) because prune is always just a loop operation over remove X basically so you might as well call remove X one by one based on the time order.
I think the biggest issue with these size checks that getting the actual resource size (aside from images) means walking the full container layer or volume to stat each file. Although maybe we could even avoid that. If we delete based on time we can just call remove X then check the fs size of the volume and if we are below exit. There would be no need to gather the actual size of each resource which means it should be quite fast as well, well as fast as the normal removal take.