Skip to content

gc: Determine which config options should be exposed and their defaults #13061

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
ehuss opened this issue Nov 28, 2023 · 0 comments
Open

gc: Determine which config options should be exposed and their defaults #13061

ehuss opened this issue Nov 28, 2023 · 0 comments
Labels
A-configuration Area: cargo config files and env vars S-needs-design Status: Needs someone to work further on the design for the feature or fix. NOT YET accepted. Z-gc Nightly: garbage collection

Comments

@ehuss
Copy link
Contributor

ehuss commented Nov 28, 2023

The current implementation from #12634 exposes a large number of low-level config options for configuring how automatic gc works. I'm uncomfortable having so many low-level options, and I would like to slim it down if possible. However, I'm uncertain what use cases would dictate what the user would actually want to change.

I think gc.auto.frequency needs to stay, but the rest should probably change? One idea I had was to separate based on "things that can be recreated" from "things that require downloading", which is essentially how the 1 month/3 month distinction happens.

# Example config.toml file.

# This table defines the behavior for automatic garbage collection.
[gc.auto]
# The maximum frequency that automatic garbage collection happens.
# Can be "never" to disable automatic-gc, or "always" to run on every command.
frequency = "1 day"
# Anything older than this duration will be deleted in the source cache.
max-src-age = "1 month"
# Anything older than this duration will be deleted in the compressed crate cache.
max-crate-age = "3 months"
# Any index older than this duration will be deleted from the index cache.
max-index-age = "3 months"
# Any git checkout older than this duration will be deleted from the checkout cache.
max-git-co-age = "1 month"
# Any git clone older than this duration will be deleted from the git cache.
max-git-db-age = "3 months"
@ehuss ehuss added A-configuration Area: cargo config files and env vars S-needs-design Status: Needs someone to work further on the design for the feature or fix. NOT YET accepted. Z-gc Nightly: garbage collection labels Nov 28, 2023
@ehuss ehuss changed the title gc: Determine which config options should be exposed gc: Determine which config options should be exposed and their defaults Dec 3, 2023
This was referenced Jul 22, 2024
Merged
Closed
github-merge-queue bot pushed a commit that referenced this issue Apr 27, 2025
@weihanglo
Stabilize automatic garbage collection. (#14287)
12555be 
This proposes to stabilize automatic garbage collection of Cargo's
global cache data in the cargo home directory.

### What is being stabilized?

This PR stabilizes automatic garbage collection, which is triggered at
most once per day by default. This automatic gc will delete old, unused
files in cargo's home directory.

It will delete files that need to be downloaded from the network after 3
months, and files that can be generated without network access after 1
month. These thresholds are intended to balance the intent of reducing
cargo's disk usage versus deleting too often forcing cargo to do extra
work when files are missing.

Tracking of the last-use data is stored in a sqlite database in the
cargo home directory. Cargo updates timestamps in that database whenever
it accesses a file in the cache. This part is already stabilized.

This PR also stabilizes the `gc.auto.frequency` configuration option.
The primary use case for when a user may want to set that is to set it
to "never" to disable gc should the need arise to avoid it.

When gc is initiated, and there are files to delete, there will be a
progress bar while it is deleting them. The progress bar will disappear
when it finishes. If the user runs with `-v` verbose option, then cargo
will also display which files it deletes.

If there is an error while cleaning, cargo will only display a warning,
and otherwise continue.

### What is not being stabilized?

The manual garbage collection option (via `cargo clean gc`) is not
proposed to be stabilized at this time. That still needs some design
work. This is tracked in
#13060.

Additionally, there are several low-level config options currently
implemented which define the thresholds for when it will delete files. I
think these options are probably too low-level and specific. This is
tracked in #13061.

Garbage collection of build artifacts is not yet implemented, and
tracked in #13136.

### Background

This feature is tracked in
#12633 and was implemented in a
variety of PRs, primarily #12634.

The tests for this feature are located in
https://github.com/rust-lang/cargo/blob/master/tests/testsuite/global_cache_tracker.rs.

Cargo started tracking the last-use data on stable via
#13492 in 1.78 which was released
2024-05-02. This PR is proposing to stabilize automatic deletion in 1.82
which will be released in 2024-10-17.

### Risks

Users who frequently use versions of Rust older than 1.78 will not have
the last-use data tracking updated. If they infrequently use 1.78 or
newer, and use the same cache files, then the last-use tracking will
only be updated by the newer versions. If that time frame is more than 1
month (or 3 months for downloaded data), then cargo will delete files
that the older versions are still using. This means the next time they
run the older version, it will have to re-download or re-extract the
files.

The effects of deleting cache data in environments where cargo's cache
is modified by external tools is not fully known. For example, CI
caching systems may save and restore cargo's cache. Similarly, things
like Docker images that try to save the cache in a layer, or mount the
cache in a read-only filesystem may have undesirable interactions.

The once-a-day performance hit might be noticeable to some people. I've
been using this for several months, and almost never notice it. However,
slower systems, or situations where there is a lot of data to delete
might take a while (on the order of seconds hopefully).
@ehuss ehuss mentioned this issue Apr 28, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-configuration Area: cargo config files and env vars S-needs-design Status: Needs someone to work further on the design for the feature or fix. NOT YET accepted. Z-gc Nightly: garbage collection
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ehuss