Add cache garbage collection #12634

ehuss · 2023-09-07T05:07:06Z

What does this PR try to resolve?

This introduces a new garbage collection system which can track the last time files were used in cargo's global cache, and delete old, unused files either automatically or manually.

How should we test and review this PR?

This is broken up into a large number of commits, and each commit should have a short overview of what it does. I am breaking some of these out into separate PRs as well (unfortunately GitHub doesn't really support stacked pull requests). I expect to reduce the size of this PR if those other PRs are accepted.

I would first review unstable.md to give you an idea of what the user side of this looks like. I would then skim over each commit message to give an overview of all the changes. The core change is the introduction of the GlobalCacheTracker which is an interface to a sqlite database which is used for tracking the timestamps.

Additional information

I think the interface for this will almost certainly change over time. This is just a stab to create a starting point where we can start testing and discussing what actual user flags should be exposed. This is also intended to start the process of getting experience using sqlite, and getting some testing in real-world environments to see how things might fail.

I'd like to ask for the review to not focus too much on bikeshedding flag names and options. I expect them to change, so this is by no means a concrete proposal for where it will end up. For example, the options are very granular, and I would like to have fewer options. However, it isn't clear how that might best work. The size-tracking options almost certainly need to change, but I do not know exactly what the use cases for size-tracking are, so that will need some discussion with people who are interested in that.

I decided to place the gc commands in cargo's cargo clean command because I would like to have a single place for users to go for deleting cache artifacts. It may be possible that they get moved to another command, however introducing new subcommands is quite difficult (due to shadowing existing third-party commands). Other options might be cargo gc, cargo maintenance, cargo cache, etc. But there are existing extensions that would interfere with.

There are also more directions to go in the future. For example, we could add a cargo clean info subcommand which could be used for querying cache information (like the sizes and such). There is also the rest of the steps in the original proposal at https://hackmd.io/U_k79wk7SkCQ8_dJgIXwJg for rolling out sqlite support.

See #12633 for the tracking issue

rustbot · 2023-09-07T05:07:11Z

r? @epage

(rustbot has picked a reviewer for you, use r? to override)

Add with_stdout_unordered. This adds the `with_stdout_unordered` method to cargo's test system so that tests can use it to check stdout but ignoring the order of lines. Nothing in this PR actually uses this method, but it is added to support #12634. I also expect it could potentially be useful in other cases in the future.

Add wrappers around std::fs::metadata This adds wrappers around `std::fs::metadata` and `std::fs::symlink_metadata` which provide better error messages indicating the path that caused the error. This just helps clean up some duplicated code, and is also going to be used to assist with some code changes in #12634.

epage · 2023-11-11T20:41:01Z

src/bin/cargo/commands/clean.rs

+                // FIXME: arg_quiet doesn't work because `config_configure`
+                // doesn't know about subcommands.


Side note: what do we do about cargo report? I assume this would also affect #11879?

So far, cargo report doesn't have a need for the -q flag because its only purpose is to display something on the console, so there isn't any output that needs suppressing. But it is something we should consider moving forward.

I think it will be a problem for #11879. I don't really know the best way to fix this. I don't suppose clap has a way to say "if this arg appears anywhere"? Perhaps another option is to have each subcommand manually handle setting quiet/verbose settings? Do you have any ideas?

clap has global(true) which propagates a flag definition down through the commands and then propagates the parsed value back up and is used by all of the flags in cli.rs's cli().

I opened #12957 to continue this conversation. We currently don't use global(true) because we want to be able to customize the help text per-command.

epage · 2023-11-11T20:43:55Z

Thanks!

@bors r+

bors · 2023-11-11T20:43:57Z

📌 Commit 0cd970b has been approved by epage

It is now in the queue for this repository.

bors · 2023-11-11T20:45:05Z

⌛ Testing commit 0cd970b with merge 9a1b092...

bors · 2023-11-11T21:17:58Z

☀️ Test successful - checks-actions
Approved by: epage
Pushing 9a1b092 to master...

weihanglo · 2023-11-11T22:49:46Z

Thank you @ehuss and @epage for your efforts for this!

This fixes an issue where some last-use tests would fail sporadically because the code that populates missing database entries was using the current time as it was iterating over the files. If the clock happened to roll over while it was iterating, then different files would get different timestamps non-deterministically. The fix is to snapshot the current time when it starts, and reuse that time while repopulating. I didn't do this originally just because I was reluctant to pass yet another argument around. However, it seems like this is necessary. In rust-lang#12634, we discussed other options such as having some kind of process-global "now" snapshot (like in `Config`), but I'm reluctant to do that because I am uneasy dealing with long-lived programs, or handling before/after relationships (like different parts of the code not considering that all timestamps might be equal). It might be something that we could consider in the future, but I'm not sure I want to try right now.

Fix non-deterministic behavior in last-use repopulation This fixes an issue where some last-use tests would fail sporadically because the code that populates missing database entries was using the current time as it was iterating over the files. If the clock happened to roll over while it was iterating, then different files would get different timestamps non-deterministically. The fix is to snapshot the current time when it starts, and reuse that time while repopulating. I didn't do this originally just because I was reluctant to pass yet another argument around. However, it seems like this is necessary. In #12634, we discussed other options such as having some kind of process-global "now" snapshot (like in `Config`), but I'm reluctant to do that because I am uneasy dealing with long-lived programs, or handling before/after relationships (like different parts of the code not considering that all timestamps might be equal). It might be something that we could consider in the future, but I'm not sure I want to try right now.

Rustin170506 · 2023-11-13T05:42:17Z

Thanks for working on it! ❤️

Update cargo 15 commits in 6790a5127895debec95c24aefaeb18e059270df3..87ee3e96285e0142b71d8c11c02b18647e43974d 2023-11-10 17:09:35 +0000 to 2023-11-14 18:00:46 +0000 - fix error message for duplicate links (rust-lang/cargo#12973) - Only filter out target if its in the package root (rust-lang/cargo#12944) - Ignore changing_spec_relearns_crate_types on windows-gnu (rust-lang/cargo#12972) - fix: do not panic when failed to parse rustc commit-hash (rust-lang/cargo#12965) - query{_vec} use IndexSummary (rust-lang/cargo#12970) - Bump to 0.77.0; update changelog (rust-lang/cargo#12966) - Improve about information of `cargo search` (rust-lang/cargo#12962) - Fix --quiet being used with nested subcommands. (rust-lang/cargo#12959) - make some debug assertion failures more informative (rust-lang/cargo#12963) - refactor(toml): Consistently lead with 'Toml' prefix (rust-lang/cargo#12960) - refactor(toml): Remove unused method (rust-lang/cargo#12961) - Fix non-deterministic behavior in last-use repopulation (rust-lang/cargo#12958) - Add cache garbage collection (rust-lang/cargo#12634) - refactor(toml): Improve consistency (rust-lang/cargo#12954) - Fix typo (rust-lang/cargo#12956) r? ghost

Update cargo 19 commits in 6790a5127895debec95c24aefaeb18e059270df3..2c03e0e2dcd05dd064fcf10cc1050d342eaf67e3 2023-11-10 17:09:35 +0000 to 2023-11-16 04:21:44 +0000 - docs(ref): Find a place to comment on --cap-lints (rust-lang/cargo#12976) - Switch from AtomicU64 to Mutex. (rust-lang/cargo#12981) - If the only path is a loop then counted as the shortest path. (rust-lang/cargo#12977) - fix(resolver): Prefer MSRV, rather than ignore incompatible (rust-lang/cargo#12950) - fix error message for duplicate links (rust-lang/cargo#12973) - Only filter out target if its in the package root (rust-lang/cargo#12944) - Ignore changing_spec_relearns_crate_types on windows-gnu (rust-lang/cargo#12972) - fix: do not panic when failed to parse rustc commit-hash (rust-lang/cargo#12965) - query{_vec} use IndexSummary (rust-lang/cargo#12970) - Bump to 0.77.0; update changelog (rust-lang/cargo#12966) - Improve about information of `cargo search` (rust-lang/cargo#12962) - Fix --quiet being used with nested subcommands. (rust-lang/cargo#12959) - make some debug assertion failures more informative (rust-lang/cargo#12963) - refactor(toml): Consistently lead with 'Toml' prefix (rust-lang/cargo#12960) - refactor(toml): Remove unused method (rust-lang/cargo#12961) - Fix non-deterministic behavior in last-use repopulation (rust-lang/cargo#12958) - Add cache garbage collection (rust-lang/cargo#12634) - refactor(toml): Improve consistency (rust-lang/cargo#12954) - Fix typo (rust-lang/cargo#12956)

I missed adding these in rust-lang#12634.

Add more doc comments for gc changes. I missed adding these in #12634.

This proposes to stabilize automatic garbage collection of Cargo's global cache data in the cargo home directory. ### What is being stabilized? This PR stabilizes automatic garbage collection, which is triggered at most once per day by default. This automatic gc will delete old, unused files in cargo's home directory. It will delete files that need to be downloaded from the network after 3 months, and files that can be generated without network access after 1 month. These thresholds are intended to balance the intent of reducing cargo's disk usage versus deleting too often forcing cargo to do extra work when files are missing. Tracking of the last-use data is stored in a sqlite database in the cargo home directory. Cargo updates timestamps in that database whenever it accesses a file in the cache. This part is already stabilized. This PR also stabilizes the `gc.auto.frequency` configuration option. The primary use case for when a user may want to set that is to set it to "never" to disable gc should the need arise to avoid it. When gc is initiated, and there are files to delete, there will be a progress bar while it is deleting them. The progress bar will disappear when it finishes. If the user runs with `-v` verbose option, then cargo will also display which files it deletes. If there is an error while cleaning, cargo will only display a warning, and otherwise continue. ### What is not being stabilized? The manual garbage collection option (via `cargo clean gc`) is not proposed to be stabilized at this time. That still needs some design work. This is tracked in #13060. Additionally, there are several low-level config options currently implemented which define the thresholds for when it will delete files. I think these options are probably too low-level and specific. This is tracked in #13061. Garbage collection of build artifacts is not yet implemented, and tracked in #13136. ### Background This feature is tracked in #12633 and was implemented in a variety of PRs, primarily #12634. The tests for this feature are located in https://github.com/rust-lang/cargo/blob/master/tests/testsuite/global_cache_tracker.rs. Cargo started tracking the last-use data on stable via #13492 in 1.78 which was released 2024-05-02. This PR is proposing to stabilize automatic deletion in 1.82 which will be released in 2024-10-17. ### Risks Users who frequently use versions of Rust older than 1.78 will not have the last-use data tracking updated. If they infrequently use 1.78 or newer, and use the same cache files, then the last-use tracking will only be updated by the newer versions. If that time frame is more than 1 month (or 3 months for downloaded data), then cargo will delete files that the older versions are still using. This means the next time they run the older version, it will have to re-download or re-extract the files. The effects of deleting cache data in environments where cargo's cache is modified by external tools is not fully known. For example, CI caching systems may save and restore cargo's cache. Similarly, things like Docker images that try to save the cache in a layer, or mount the cache in a read-only filesystem may have undesirable interactions. The once-a-day performance hit might be noticeable to some people. I've been using this for several months, and almost never notice it. However, slower systems, or situations where there is a lot of data to delete might take a while (on the order of seconds hopefully).

rustbot assigned epage Sep 7, 2023

ehuss force-pushed the last-use branch 2 times, most recently from fd1b671 to 2577bdb Compare September 7, 2023 05:23

rustbot added the A-infrastructure Area: infrastructure around the cargo repo, ci, releases, etc. label Sep 7, 2023

epage reviewed Nov 11, 2023

View reviewed changes

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 11, 2023

epage mentioned this pull request Nov 11, 2023

Support cargo owner add #11879

Closed

bors merged commit 9a1b092 into rust-lang:master Nov 11, 2023

ehuss mentioned this pull request Nov 12, 2023

Fix non-deterministic behavior in last-use repopulation #12958

Merged

weihanglo mentioned this pull request Nov 14, 2023

Update cargo rust-lang/rust#117916

Merged

ehuss added a commit to ehuss/cargo that referenced this pull request Nov 28, 2023

Add more doc comments for gc changes.

6950059

I missed adding these in rust-lang#12634.

ehuss mentioned this pull request Nov 28, 2023

Add more doc comments for gc changes. #13055

Merged

bors added a commit that referenced this pull request Nov 28, 2023

Auto merge of #13055 - ehuss:document-more-gc-fields, r=epage

f101fa0

Add more doc comments for gc changes. I missed adding these in #12634.

This was referenced Nov 28, 2023

gc: Verify performance numbers #13058

Open

gc: Determine CLI design for manual cleaning #13060

Open

gc: Determine which config options should be exposed and their defaults #13061

Open

bors added a commit that referenced this pull request Nov 28, 2023

Auto merge of #13055 - ehuss:document-more-gc-fields, r=epage

15a35e5

Add more doc comments for gc changes. I missed adding these in #12634.

ehuss mentioned this pull request Nov 28, 2023

gc: Determine max-size use cases, and consider design changes #13062

Open

ehuss added this to the 1.76.0 milestone Dec 6, 2023

epage mentioned this pull request Dec 8, 2023

Pin cache entries still in use #13137

Open

ehuss mentioned this pull request Jul 22, 2024

Stabilize automatic garbage collection. #14287

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cache garbage collection #12634

Add cache garbage collection #12634

ehuss commented Sep 7, 2023 •

edited by epage

Loading

rustbot commented Sep 7, 2023

epage Nov 11, 2023

ehuss Nov 11, 2023

epage Nov 11, 2023

ehuss Nov 11, 2023

epage commented Nov 11, 2023

bors commented Nov 11, 2023

bors commented Nov 11, 2023

bors commented Nov 11, 2023

weihanglo commented Nov 11, 2023

Rustin170506 commented Nov 13, 2023

		// FIXME: arg_quiet doesn't work because `config_configure`
		// doesn't know about subcommands.

Add cache garbage collection #12634

Add cache garbage collection #12634

Conversation

ehuss commented Sep 7, 2023 • edited by epage Loading

What does this PR try to resolve?

How should we test and review this PR?

Additional information

rustbot commented Sep 7, 2023

epage Nov 11, 2023

Choose a reason for hiding this comment

ehuss Nov 11, 2023

Choose a reason for hiding this comment

epage Nov 11, 2023

Choose a reason for hiding this comment

ehuss Nov 11, 2023

Choose a reason for hiding this comment

epage commented Nov 11, 2023

bors commented Nov 11, 2023

bors commented Nov 11, 2023

bors commented Nov 11, 2023

weihanglo commented Nov 11, 2023

Rustin170506 commented Nov 13, 2023

ehuss commented Sep 7, 2023 •

edited by epage

Loading