Allow linking to native library multiple times through build dependencies

[bsteinb]

Recently the mpi crate has stopped building because it transitively depends on different versions of the clang-sys crate which links to the native libclang. Here is the error message:

error: multiple packages link to native library `clang`, but a native library can be linked only once

package `clang-sys v0.22.0`
    ... which is depended on by `bindgen v0.33.2`
    ... which is depended on by `mpi-sys v0.1.1 (file:///Users/bsteinb/Documents/Programming/rust/rsmpi/mpi-sys)`
    ... which is depended on by `mpi v0.5.2 (file:///Users/bsteinb/Documents/Programming/rust/rsmpi)`
links to native library `clang`

package `clang-sys v0.21.2`
    ... which is depended on by `bindgen v0.31.3`
    ... which is depended on by `libffi-sys v0.6.0`
    ... which is depended on by `libffi v0.6.3`
    ... which is depended on by `mpi v0.5.2 (file:///Users/bsteinb/Documents/Programming/rust/rsmpi)`
also links to native library `clang`

I understand that this would be a problem if both versions of the clang-sys crate were to end up being linked to by the mpi crate, but they do not. bindgen (and transitively clang-sys) in both cases is a build dependency of mpi-sys and libffi-sys and is only linked to their respective build scripts and does not end up in the mpi crate. Should this really be an error, or is cargo being too cautious here?

[lukaslueg]

For the moment you can make america mpi great again by having mpi-sys depend on bindgen = "0.31.3", which is the exact same version that libffi-sys uses. As the dependencies converge, the conflict no longer occurs.

[bsteinb]

I know. I am doing that at the moment.

My question is whether the constellation where different dependencies of mpi have build dependencies on different versions of clang-sys should actually be considered to be a problem by cargo at all. The way I see it, mpi-sys builds an executable from its build.rs that contains clang-sys v0.33.2 and libffi-sys builds an executable from its build.rs that contains clang-sys v0.22.0. These are completely distinct executables, why is it a problem that they depend on different versions of clang-sys?

[alexcrichton]

cc @Eh2406

Thanks for the report! Unfortunately this may be an exceedingly tricky one to solve. The resolver doesn't understand host/target distinctions as well as linkage, so it can't necessarily conclude the that the linkage here may actually work out..

[bsteinb]

You are quite welcome. If we are honest, I am not being completely selfless here.

I know nothing about cargo internals, but I am not sure this is a host vs. target issue. Both build executables should be host executables, no? The way I see it, this check for linking the same native library should not be applied across build dependencies of different crates (such as mpi-sys and libffi-sys in my case).

[Eh2406]

Ya this is going to be tricky to solve. It is related to #2589 but in this case it is a conflict between two different packages build dependencies. If we had a solution to rust-lang/rust#44663 then we would just list all build deps as private dependencies. But that is going to be hard to implement as we don't have any support for path of visibility.

[alexcrichton]

Oh sorry @bsteinb I misunderstood! It's true that these are both host dependencies which means that I'm not sure Cargo has ever supported this (I thought this may have been an accidenal recent regression)

[bsteinb]

Well this is something that recently showed up for the mpi crate, but probably not due to a change in cargo. mpi-sys depends on bindgen 0.33 which as of 11 days ago resolves to bindgen 0.33.2 which depends on clang-sys 0.22. Meanwhile libffi-sys has its bindgen dependency pinned to bindgen 0.31.3 which still depends on clang-sys 0.21. As @lukaslueg suggested, I can work around this issue by pinning my dependencies to specific versions that work together.

Even if this was not and is not supported by cargo, I am trying to raise the question whether this behaviour might be too strict/pessimistic.

[Eh2406]

I can confirm that this is not the new error we just added. So I don't think it is a regression.
An ideal cargo would allow this, but I don't think it is going to be easy to fix.
I think an implementation of RFC 1977: public & private dependencies would include infrastructure to make this an easy fix, but that is something of a tautology. That RFC is not implemented because no one has designed the infrastructure.

[bsteinb]

All right, for now I can live with the workaround of pinning my dependencies. When cargo is changed to loosen these restrictions, I can loosen the version specifications again.

[alexcrichton]

Thanks for the clarifications @bsteinb and @Eh2406!

[klausi]

Running into the same problem when building crates with their minimal version:

$ cargo build -Z minimal-versions
error: multiple packages link to native library `z`, but a native library can be linked only once

package `libz-sys v0.0.1`
    ... which is depended on by `libssh2-sys v0.2.4`
    ... which is depended on by `libgit2-sys v0.7.1 (file:///home/klausi/workspace/git2-rs/libgit2-sys)`
    ... which is depended on by `git2 v0.7.1 (file:///home/klausi/workspace/git2-rs)`
links to native library `z`

package `libz-sys v0.1.2`
    ... which is depended on by `libgit2-sys v0.7.1 (file:///home/klausi/workspace/git2-rs/libgit2-sys)`
    ... which is depended on by `git2 v0.7.1 (file:///home/klausi/workspace/git2-rs)`
also links to native library `z`

cargo should be able to solve this: it should select the higher version only of libz-sys in this case because both libgit2-sys and libssh2-sys allow it. Then libz-sys can be linked only once and all is well.

Can we modify cargo's resolver to take into account links = "z" in Cargo.toml files? If 2+ versions of the same package are selected and that package links to a native library then try to select only one version that does not violate the version range specified by all dependents.

[alexcrichton]

@klausi oh that's actually already implemented, but it requires changes to the index which are happening with newer Cargos but old entries in the index aren't retroactively changed with this information

[klausi]

Hm, I ran rm -rf ~/.cargo/registry and cargo downloaded the index again, but still having the same problem when building https://github.com/alexcrichton/git2-rs with cargo build -Z minimal-versions.

Maybe it does not work with the -Z minimal-versions switch?

[sfackler]

Deleting and redownloading the index isn't going to change what's inside of it.