Fails to connect through trusted SSL proxy #15376
Comments
Cargo uses openssl-probe to try to find the certificates on the system. Are you specifically running on Nix, and is it placing the certificates in an unpredictable location, or not in one of the known locations? |
|
@ehuss On my NixOS, when building self.cainfo("/etc/pki/tls/certs/ca-bundle.crt");
self.capath("/etc/pki/tls/certs");It correctly detects and sets the CA info/path, thus $ export CARGO_HTTP_PROXY="http://localhost:8002"
$ cargo fetch
Downloaded unicode-ident v1.0.18
Downloaded proc-macro2 v1.0.94
Downloaded 2 crates (100.1 KB) in 0.29sHowever, no information about proxy CA was provided, thus $ export CARGO_HTTP_PROXY="https://localhost:8002"
$ cargo fetch
error: failed to get `proc-macro2` as a dependency of package `my-app v0.1.0 (/home/andrew/Downloads/tmp/my-app2)`
Caused by:
download of config.json failed
Caused by:
failed to download from `https://index.crates.io/config.json`
Caused by:
[60] SSL peer certificate or SSH remote key was not OK (SSL certificate problem: unable to get local issuer certificate)The following change in diff --git a/src/easy/handler.rs b/src/easy/handler.rs
index e944b87..ab68b77 100644
--- a/src/easy/handler.rs
+++ b/src/easy/handler.rs
@@ -691,9 +691,11 @@ impl<H: Handler> Easy2<H> {
if let Some(ref path) = probe.cert_file {
let _ = self.cainfo(path);
+ let _ = self.proxy_cainfo(&format!("{}", path.display()));
}
if let Some(ref path) = probe.cert_dir {
let _ = self.capath(path);
+ let _ = self.proxy_capath(path);
}
}NOTE: However, I still have a strong feeling that option to configure proxy certs directly in If I am not mistaken, |
weihanglo
added
S-needs-design
|
@koxu1996 Would you mind open an issue against curl crate proposing the fallback proxy CA solution? |
|
We talked about this in today's @rust-lang/cargo meeting, and we talked about doing this in |
|
@weihanglo I could have opened a PR, but I was not entirely convinced that @joshtriplett Implementing the fallback directly in |
|
On Wed, Apr 16, 2025 at 02:14:02AM -0700, koxu1996 wrote:
@joshtriplett Implementing the fallback directly in `cargo` makes sense, but I still believe there is no harm in exposing `CARGO_HTTP_PROXY_CAINFO`. I can come up with valid use cases if that is a requirement 😁.
Please do. Nobody on the team was *opposed* to it, we just wanted to
make sure it was independently needed even after having the fallback in
Cargo.
|
|
@joshtriplett In CI pipelines behind a TLS-inspecting corporate proxy, it is much easier and more maintainable to pass a My case is a little bit different though - I am running crates.io proxy cache using kellnr. |
|
Hit by this as well. Any iother workaround other than waiting for the change? |
|
@MuthuKMu You can just build |
Problem Description / User Story
Let us suppose you want to use an SSL proxy, which generates certificates signed by a custom CA, and this CA is trusted by your OS.
The following
curlcommand confirms the setup is done correctly:However, when calling
cargoit MIGHT surprisingly fail, for example:Code 60 is a well known case and the solution is to simply add the CA certificate to the trust store.
Wait... you already have CA trusted and curl worked. So why Cargo fails?
Solving this mystery involved diffing Wireshark packets, hooking into libcurl calls, and doing bunch of other thing I would prefer to avoid, but finally I found the answer!
Who is Affected?
This is a little spoiler, but if your
cargohas libcurl vendored then you are affected!NOTE: I am talking about Linux, not sure about other systems.
You can check it with
ldd $(which cargo) | grep libcurl- if the output is empty, then it means libcurl was vendored, instead of being dynamically linked.However, if you see output like:
then you are NOT affected.
Reproducing the Issue
Before jumping to the answer, I would like to give you a simple way of reproducing the issue.
Getting Cargo
If your
cargois dynamically linked withlibcurl, then you need to get a static one; otherwise skip this section.It can be done by compiling sources and making sure
curl-syswill not be able to find the system'slibcurllibrary viapkg-config:Binary will be located at
./target/debug/cargo.Running Proxy
To run MITM proxy we will use
mitmdumptool, but firstly let us prepare CA certificates:CAUTION: Remember the location of
./certs/mitmproxy-ca-cert.ceras it will be needed in the next section.Now we run the proxy (including predefined certificates):
docker run \ --detach \ --name mitmproxy \ --rm \ --publish 0.0.0.0:8002:8000 \ --volume $(pwd)/certs:/home/mitmproxy/.mitmproxy \ mitmproxy/mitmproxy:11.1.3 \ mitmdump \ --set mode=regular \ --set listen_port=8000 \ --set block_global=falseIt will be running in the background on port
8002.At this point you should be able to use the proxy, but certificates are not trusted yet. All below calls should succeed:
TIP: Use
--verboseto see more details about the connection.Trusting CA
To avoid using
--(proxy-)insecureflags we need to add the CA certificate to the system's trust store.Remember the location of
./certs/mitmproxy-ca-cert.cer? Now it is time to use it.Instructions depend on the system, but on Arch Linux / Fedora you can trust the certificate with:
sudo trust anchor --store `./certs/mitmproxy-ca-cert.cer`TIP: On Debian / Ubuntu you probably have to use
update-ca-certificates.At this point you should be able to use SSL proxy with HTTPS URLs:
Let Cargo Fail
Create small Rust project:
Enable the use of our SSL proxy:
Try adding dependency:
Boom! It fails with
[60] SSL peer certificate or SSH remote key was not OK (SSL certificate problem: unable to get local issuer certificate).Why Cargo Fails? Why Curl Works?
Now you should be curious why cargo fails even though curl works just fine.
The answer is simple: system
curlhas extra CA configuration!On Arch Linux, package is configured with
--with-ca-bundle, pointing to the system trust store. Therefore updating the trust store is enough to make systemcurlaware of the custom CA.On NixOS, package is configured with
--with-ca-fallback, which allows using the built-in CA store of the SSL library.NOTE: I have not tested this with other Linux distributions, but I guess the same applies.
When
libcurlis vendored in Cargo, it has no extra CA configuration that would allow discovering certificates from the system.Workarounds / Solutions
Use
cargowith dynamically linkedlibcurl- use system package (might work) or build from sources making sure thatpkgconfigis able to find the library; do NOT userustup!Wait until
CARGO_HTTP_PROXY_CAINFOis supported - cargo#15374 - then use it along withCARGO_HTTP_CAINFOto provide certificate path.Extra Notes
Fallback of Proxy CA
curldoes an interesting thing: when--proxy-capathis not set, then--capathvalue is used for it.This behavior was introduced as part of curl#1257, but work has stalled.
Compiling Cargo on NixOS
On NixOS, to compile
cargofrom sources with dynamically linkedlibcurl, you have to make sure the following command succeeds:In my case I had to:
curl.devpackage, andexport PKG_CONFIG_PATH=/nix/store/4nra0djq6vb9j7x37qwr56ac18lbbvrv-curl-8.12.1-dev/lib/pkgconfig:$PKG_CONFIG_PATH.That allowed
curl-systo finally use the system'slibcurllibrary instead of vendoring it from source.NOTE: It is possible to use
cargopackage (it does NOT use vendoredlibcurl, which fixes the problem), but it conflicts with therustup.The text was updated successfully, but these errors were encountered: