RBE: Build buck2 with BuildBuddy RBE #979
Conversation
Provide an example to show building Buck2 using Buck2 on BuildBuddy RBE. --- TLDR: 1. Step one, build buck2 with Cargo. This would embed a version of the patched prelude inside the newly built buck2 binary. ``` cargo install --locked --git https://github.com/facebookincubator/reindeer reindeer reindeer --third-party-dir shim/third-party/rust buckify cargo install --path app/buck2 -Z unstable-options buck2 --version ``` 2. Create an account on https://app.buildbuddy.io and obtain an API key ``` export BUILDBUDDY_API_KEY=<your-api-key-here> ``` 3. Build Buck2 with RBE ``` buck2 build :buck2 ``` --- With that said, here are some notes: First, it worth noting that Buck2 is currently using the system toolchains in shim/BUCK with the most important ones being @shim//:cxx and @shim//:rust. When using these toolchains to build buck2, we need to ensure the same versions are available in the execution environment. That means the same clang, the same rustc, the same libc etc... On most RBE server implementation today, including BuildBuddy, container image is how users can customize the execution environment. For this reason, we use the Dockerfile in this PR to build a container image with clang, git, unzip inside. We also take extra caution to make sure that the rustc installed on the container image is the same as the one declared in //:rust-toolchain file. If the clang version or the //:rust-toolchain file is updated, the container image need to be rebuild(!!!) before the remote build can run successfully. Secondly, the repo currently depends on 3 git_fetch targets generated by reindeer. git_fetch actions require networking, which is why currently they are marked as local_only and thus, not RBE compatible. Since BuildBuddy RBE workers can have external networking enabled, we patch git_fetch to remove the local_only clause. The Execution Platform is also tuned accordingly to have local and hybrid execution disabled. Finally, as all actions are now RBE compatible, we can also turn on all the deferred materializations flags Buck2 has to provide. This means that there are no intermediary artifacts download through out the build. Only the final "buck2" binary artifact should be downloaded from BuildBuddy server, keeping the network impact relatively minimal. A successful RBE build without Action Cache hits should look something like this. ```bash > buck2 log summary Showing summary from: buck2 build :buck2 build ID: 7ffdd713-27fd-4a19-9033-946b8b2afe83 total files materialized: 1 total bytes materialized: 341721472 total bytes uploaded: 4539317 local actions: 0 remote actions: 4140 cached actions: 0 other actions: 4893 targets analysed: 1889 peak process memory: 1.1GiB out of 16GiB peak used disk space: 167GiB out of 234GiB max download speed: 0B/s max upload speed: 0B/s duration: 3:35.6s has local changes: unknown ``` and this is how it should look with full cache hits ``` > buck2 log summary Showing summary from: buck2 build :buck2 build ID: 5372b752-f88c-4485-8bb0-c8ecccc7a76d total files materialized: 1 total bytes materialized: 341721472 total bytes uploaded: 0 local actions: 0 remote actions: 0 cached actions: 4173 other actions: 4860 targets analysed: 1915 peak process memory: 727MiB out of 16GiB peak used disk space: 167GiB out of 234GiB max download speed: 0B/s max upload speed: 0B/s duration: 42.8s has local changes: unknown ``` Note the 3m35.6s duration vs the 42.8s duration.
facebook-github-bot
added
the
CLA Signed
|
@facebook-github-bot has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. (Because this pull request was imported automatically, there will not be any future comments.) |
|
Uh... it goes without saying, but do review the limits of free tier accounts here https://www.buildbuddy.io/pricing before attempting this 🤗 |
|
cc: @alexlian @samkevich |
|
Oh, and all benchmarks are done on my crappy 2018 laptop running on bad Wifi, so take them with a grain of salt. |
|
@sluongng Can you please rebase on master. CI seems unhappy. I fixed those issues yesterday. |
|
@samkevich This PR is currently not really well made for your CI. As I mentioned in the PR Summary, for RBE to be enabled in CI, you would need to have a process to build and push the container image using the Dockerfile first. That would enable the RBE setup to have the same The other option is to create a Which approach would you prefer? |
|
Thank you @sluongng, that is useful! Since I am using NixOS, I went ahead and added a Docker nix image to the flake, on top of your changes: sluongng/buck2@sluongng/buck2-bootstrap-re...avdv:buck2:cb/buck2-bootstrap-re-nix To build using nix:
No need to build buck2 with cargo first, since we can use the prelude directly by commenting out the Also, the Docker image needs to be available in a registry somewhere. I published it to Github's container registry, using this command: |
Maybe off-topic for this PR, but I am keenly interested in figuring a good solution for remote toolchains. I am currently building an RBE solution that relies on bespoke images baked from our nix flake and the turn-around time/coordination of building and shipping these is a bit more painful than I would like. Have you spent any time on a remote toolchain solution? I'd be up for collaborating. |
|
I want to avoid off-topic discussion here, so @sprutton1, feel free to create a separate GH issue and tag me there. |
5 participants
DO NOT MERGE
Provide an example to show building Buck2 using Buck2 on BuildBuddy RBE.
TLDR:
With that said, here are some notes:
First, it worth noting that Buck2 is currently using the system toolchains in shim/BUCK with the most important ones being @shim//:cxx and @shim//:rust. When using these toolchains to build buck2, we need to ensure the same versions are available in the execution environment. That means the same clang, the same rustc, the same libc etc...
On most RBE server implementation today, including BuildBuddy, container image is how users can customize the execution environment. For this reason, we use the Dockerfile in this PR to build a container image with clang, git, unzip inside. We also take extra caution to make sure that the rustc installed on the container image is the same as the one declared in //:rust-toolchain file. If the clang version or the //:rust-toolchain file is updated, the container image need to be rebuild(!!!) before the remote build can run successfully.
Secondly, the repo currently depends on 3 git_fetch targets generated by reindeer. git_fetch actions require networking, which is why currently they are marked as local_only and thus, not RBE compatible. Since BuildBuddy RBE workers can have external networking enabled, we patch git_fetch to remove the local_only clause. The Execution Platform is also tuned accordingly to have local and hybrid execution disabled.
Finally, as all actions are now RBE compatible, we can also turn on all the deferred materializations flags Buck2 has to provide. This means that there are no intermediary artifacts download through out the build. Only the final "buck2" binary artifact should be downloaded from BuildBuddy server, keeping the network impact relatively minimal. A successful RBE build without Action Cache hits should look something like this.
and this is how it should look with full cache hits
Note the 3m35.6s duration vs the 42.8s duration.