Integrate Rust into the build process properly

[alex]

To be fully working

• Get rust on the PATH in the docker containers
• Install Rust on Zuul
• PyO3 doesn't build on musl targets (Alpine linux) PyO3/pyo3#1109
• Add Rust to the docker image? readthedocs/readthedocs-docker-images#135
• https://review.opendev.org/746423
• Complete abi3 support PyO3/pyo3#1125
• Enable building abi3 shared objects PyO3/setuptools-rust#82

To land

• 0.13 Release PyO3/pyo3#1306
• Python 2.7 support timeline #5359

[alex]

@ianw Do you have an idea for what the best way to install Rust in the builders is? I can rustup.sh easily enough, but I'm not sure what I need to change to add things to the PATH.

[ianw]

@ianw Do you have an idea for what the best way to install Rust in the builders is? I can rustup.sh easily enough, but I'm not sure what I need to change to add things to the PATH.

Seems like we should add a role to zuul-jobs to install rust so that everyone can benefit :)

I've started with that @ https://review.opendev.org/#/c/746423/ which should be sufficient for this case.

To use that we can just add to .zuul.playbooks/playbooks/tox/main.yaml

- name: Install rust
  include_role:
    name: ensure-rust

anywhere before the tox run.

This has to pass review. However, until it is committed you should be able to speculatively test a change like above by adding to your change description

Depends-On: https://review.opendev.org/746423

This will instruct Zuul to apply this change before running (Zuul would not allow the change to merge with a Depends-On that isn't committed, but in this case you should be careful not to push it without us either merging the change or taking some other approach).

[ianw]

This has to pass review. However, until it is committed you should be able to speculatively test a change like above by adding to your change description

Depends-On: https://review.opendev.org/746423

It has been pointed out to me that this needs to be in the PR description, not the change description (that's a bit different to Gerrit).

[StefanCristian]

Hello @alex, @reaperhulk , is this going to be optional? My containers (& implicitly also our whole work even on non-containers) depend exclusively on Python in production, and unfortunately I add cannot additional languages since it would break the development, build & release cycles a bit too much.

Our end goal is to deliver as less as possible, strictly python. There are some horrible and stressful security guidelines that would be required to add a additional language & compilers to work with and it would take some years, possibly, judging by the companies necessities.

[alex]

You already posted this, and received an answer, on c84d6ee

[StefanCristian]

This is optional for the 3.4.x release, but required afterwards. Rust is only required at build time, so you should be able to install rust in a build container, but not include in a release container.

@alex thank you for replying. When speaking about very-high secured medium, we cannot have additional compiled binaries with other programming languages, especially not optimized ones and not ones under our control, shipped into the release medium. I gave the container as a example, as I'm not speaking only of containers.
It's going to be a security issue that we will have no possibility of handling.

Is there any other solution? Will you include a option to remove in case things go wrong?

[alex]

I don't know what your approach to security is, but the idea that introducing a new memory safe language, to replace C, reduces your security is lunacy. If this is something that some auditor or compliance regime is imposing on you, please share that so we can shame this for this absurdity.

…

On Fri, Feb 12, 2021 at 11:39 PM Stefan Cristian B. < ***@***.***> wrote: This is optional for the 3.4.x release, but required afterwards. Rust is only required at build time, so you should be able to install rust in a build container, but not include in a release container. @alex <https://github.com/alex> thank you for replying. When speaking about very-high secured medium, we cannot have additional compiled binaries with other programming languages, especially not optimized ones and not ones under our control, shipped into the release medium. I gave the container as a example, as I'm not speaking only of containers. It's going to be a security issue that we will have no possibility of handling. Is there any other solution? Will you include a option to remove in case things go wrong? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5410 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBFXA5P4SQXDMW65Y4LS6X7A7ANCNFSM4QBA6NNA> .

-- All that is necessary for evil to succeed is for good people to do nothing.

[reaperhulk]

There will not be any option to use cryptography without rust being part of the compilation process starting with our next release, no.

[StefanCristian]

@alex I have shared with you all I could. Please find that the reasons behind are holistic and not resume to only security.

@reaperhulk what would it take to have on/off switch for rust compilation? Do you plan on replacing the C code completely?

[reaperhulk]

You can see more information about what our plans for rust are in #5810 (comment). There will not be an on/off switch, no. The core cryptographic primitives will remain OpenSSL and there are no plans to change that.

[StefanCristian]

@reaperhulk Thank you kindly. I will not be participating in the discussion since your plans are already decided.

Even though a decision on our side is not yet here, we might be forced to abandon the library entirely, in time.

[luke-jr]

Rust isn't bootstrappable on non-x86, so not available for security-minded folks on other platforms.

Guess I'm going to have to figure out a way to avoid pyca :/

[alex]

Debian/RHEL/Alpine manage to bootstrap Rust on a variety of other architectures, I suspect if you look into what they're doing you'll find a solution to your problem.

[luke-jr]

Likely they bootstrapped an ancient version and just continued to use newer versions to build each subsequent. Not practical to do from scratch today.

[reaperhulk]

Ignoring the existence proof isn’t a productive path. If there exists an issue with bootstrapping enumerating the challenges and then working towards their elimination with the rust and llvm community sounds like a worthy project though. Rust’s prominence continues to increase so just asking people not to use it isn’t a long term viable path.