Skip to content

Cargo and/or other tools add Cargo.lock files and other build artifacts to registry #5956

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mykmelez opened this issue Aug 31, 2018 · 2 comments
Labels
A-caching Area: caching of dependencies, repositories, and build artifacts

Comments

@mykmelez
Copy link

mykmelez commented Aug 31, 2018

As described in bug 1461553, comment 2 and later comments, Cargo and/or other tools that are using Cargo's global registry sometimes add Cargo.lock files to the crate directories (i.e. unpacked tarballs) in the registry. And they've been known to add other files as well, f.e. the lalrpop crate directory sometimes sprouts the file src/parser/lrgrammar.rs.

Cargo.lock and src/parser/lrgrammar.rs are both build artifacts, so presumably this is caused by a tool that is building the crate directly within the registry directory. Unfortunately, Cargo and tools like cargo-vendor assume that the registry is immutable, so these added files cause unwanted effects, like unexpected changes to already-vendored crates when vendoring unrelated ones (or even simply re-running cargo-vendor after adding no new crates).

Over in that bug, @alexcrichton notes a few options for resolving the issue:

One change would be to have cargo-vendor work directly from the tarballs instead of the checked-out state, but such a change isn't entirely trivial to implement today. Otherwise Cargo could also do something like periodically execute a sanity check to make sure checked out crates look alright.

I'd personally prefer to dig in and discover what crates are modifying the registry. I suspect it's actually very few but perhaps heavily depended on ones that have consequences for the ecosystem.

In the meantime, a workaround is to delete the registry (rm -rf ~/.cargo/registry), which Cargo will then repopulate as needed.

@alexcrichton alexcrichton added the A-caching Area: caching of dependencies, repositories, and build artifacts label Sep 2, 2018
alexcrichton added a commit to alexcrichton/cargo-vendor that referenced this issue Oct 16, 2018
Work around vendoring stale contents by forcing Cargo to re-unpack all
crates which we vendor.

Closes #130
alexcrichton added a commit to alexcrichton/cargo-vendor that referenced this issue Oct 18, 2018
@epage
Copy link
Contributor

epage commented Oct 23, 2023

The reasond packages may be doing this

  • build.rs files that don't follow the rules of writing to OUT_DIR
  • build.rs running cargo commands within the PACKAGE_DIR to do various forms of introspection.

A combination of the following would fix this

However, as the underlying need was resolved in cargo-vendor#131, I'm going to go ahead and close this.

@epage epage closed this as not planned Won't fix, can't repro, duplicate, stale Oct 23, 2023
@weihanglo
Copy link
Member

The other relevant issue is #9455 for making unpacked tarballs and directories "read-only".

weihanglo added a commit to weihanglo/cargo that referenced this issue Mar 4, 2025
cargo-vendor has a workaround that to mitigate rust-lang#5956,
it removes all cached sources in order to trigger a re-unpack.
It was meant for dealing with registry sources only, but accidentally
applied to directory source kind.

While directory source kind was invented for vendoring,
and vendoring from on vendored directory to the other seems unusual,
Cargo IMO should not delete any real sources.

It does not mean that registry sources are cached so is okay to delete,
In long term, we should explore a way that unpacks `.crate` files
directly, without any removal. See
rust-lang#12509 (comment)
weihanglo added a commit to weihanglo/cargo that referenced this issue Mar 4, 2025
cargo-vendor has a workaround that to mitigate rust-lang#5956,
it removes all cached sources in order to trigger a re-unpack.
It was meant for dealing with registry sources only, but accidentally
applied to directory source kind.

While directory source kind was invented for vendoring,
and vendoring from on vendored directory to the other seems unusual,
Cargo IMO should not delete any real sources.

It does not mean that registry sources are cached so is okay to delete,
In long term, we should explore a way that unpacks `.crate` files
directly, without any removal. See
rust-lang#12509 (comment)
weihanglo added a commit to weihanglo/cargo that referenced this issue Mar 4, 2025
cargo-vendor has a workaround that to mitigate rust-lang#5956,
it removes all cached sources in order to trigger a re-unpack.
It was meant for dealing with registry sources only, but accidentally
applied to directory source kind.

While directory source kind was invented for vendoring,
and vendoring from one vendored directory to the other seems unusual,
Cargo IMO should not delete any real sources.

It does not mean that registry sources are okay to delete,
In long term, we should explore a way that unpacks `.crate` files
directly, without any removal. See
rust-lang#12509 (comment)
github-merge-queue bot pushed a commit that referenced this issue Mar 4, 2025
### What does this PR try to resolve?

Fixes #15244

With this fix,
`cargo vendor` will not delete original sources,
if you want to vendor things from one directory sources to the other

#### Background

cargo-vendor has a workaround that to mitigate #5956:
it removes all cached sources in order to trigger a re-unpack.
It was meant for dealing with registry sources only,
but accidentally applied to directory source kind.

While directory source kind was invented for vendoring,
and vendoring from one vendored directory to the other seems unusual,
Cargo IMO should not delete any real sources.

It does not mean that registry sources are okay to delete,
In long term, we should explore a way that unpacks `.crate` files
directly, without any removal. See
#12509 (comment)

### How should we test and review this PR?

The added test should suffice.

Also, although this is for fixing #15244,
`cargo vendor` still doesn't support vendor from and to the same
location.
Unless we figure out an `rsync`-like solutin to update vendor sources,
it is not going to support in short term.
(And I also doubt the real world use case of it)

### Additional information
weihanglo added a commit to weihanglo/cargo that referenced this issue May 11, 2025
By doing direct extractions, source code is kept in its original state
The old workaround is irrelevant now.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-caching Area: caching of dependencies, repositories, and build artifacts
Projects
None yet
Development

No branches or pull requests

4 participants