bazel: Remove hardcoded dependency on `//:protoc` from language runtimes #19679

fmeum · 2024-12-16T14:57:27Z

Without this change, language runtimes still result in a build of //:protoc even with a prebuilt proto_toolchain registered or --proto_compiler set to a precompiled protoc. Removing this hardcoded dependency allows a (fast) build of java_proto_library targets without a C++ toolchain assuming a prebuilt protoc.

Work towards #19558

Without this change, language runtimes still result in a build of `//:protoc` even with a prebuilt `proto_toolchain` registered or `--proto_compiler` set to a precompiled protoc.

fmeum · 2024-12-16T15:25:35Z

CC @comius @alexeagle

fmeum · 2024-12-16T15:26:09Z

@shaod2 Would you be available to review this?

mkruskal-google · 2024-12-19T19:21:03Z

Hey Fabian,

So IIUC, this PR is trying to make the protoc binary used by proto_library customizable. While that might be a possible long-term solution, it opens the door to a lot of potential misuse. Notably, mixing alternate implementations or different versions of protoc is typically unsupported and can have unexpected results. We have poison pills to avoid some of these situations outside of Bazel, but they haven't been added to all our languages yet. A more conservative change would be to just have some kind of flag that toggles between build-from-source and the prebuilt releases we publish to github, without allowing arbitrary binaries to be injected.

Additionally, this PR is touching protobuf.bzl, which just contains our legacy internal macros/rules (which we want to delete). These have been replaced by the rules in //bazel, which were recently moved from other Bazel repos.

fmeum · 2024-12-30T09:34:46Z

Additionally, this PR is touching protobuf.bzl, which just contains our legacy internal macros/rules (which we want to delete). These have been replaced by the rules in //bazel, which were recently moved from other Bazel repos.

These macros are still used to bootstrap the Java well-known protos for the default Java proto_lang_toolchain, which is why protoc ends up being compiled from source even when using a precompiled protoc binary.

So IIUC, this PR is trying to make the protoc binary used by proto_library customizable. While that might be a possible long-term solution, it opens the door to a lot of potential misuse. Notably, mixing alternate implementations or different versions of protoc is typically unsupported and can have unexpected results.

Isn't the protoc binary used by proto_library already configurable via --proto_compiler or toolchain registration (the latter with --incompatible_enable_proto_toolchain_resolution)? The aim of this PR is specifically to ensure that if users use custom protoc binary, it's the only protoc used in the entire build. Right now, when configuring a non-default protoc binary in any of the existing ways, the //:protoc from this repo still ends up being used to bootstrap protos.

@crt-31

Keeps the `README` guidance in sync with what we're actually using in `WORKSPACE` for consistency's sake. @crt-31 and I found that the Windows build failure for bazel-contrib#1710 mentioned in the earlier commit is related to the Windows/MSVC file path length limit. `src/google/protobuf/compiler/java/java_features.pb.h`, the path specified in the error message, doesn't exist until `protobuf` v25.0. - protocolbuffers/protobuf#12947 Furthermore, the Protobuf team currently plans to just drop MSVC support: - https://protobuf.dev/news/v30/#poison-msvc--bazel - protocolbuffers/protobuf#20085 I plan to experiment again with "Protobuf Toolchainization", which I'd tried in October when beginning the Bzlmod experiment. Here are some interesting background resources before I dig in on that: - bazelbuild/rules_proto#213 - bazelbuild/rules_proto#179 - https://github.com/bazelbuild/rules_proto/releases/tag/6.0.0 - https://github.com/aspect-build/toolchains_protoc/ - protocolbuffers/protobuf#20182 - protocolbuffers/protobuf#19679 - protocolbuffers/protobuf#19558

Registers a precompiled protocol compiler toolchain when `--incompatible_enable_proto_toolchain_resolution` is `True`. Part of bazel-contrib#1482 and bazel-contrib#1652. Stops `protoc` recompilation, and fixes the build breakage in bazel-contrib#1710 due to `protobuf` include paths exceeding the Visual Studio path length limit. The updates to `scala_proto/scala_proto_toolchain.bzl` were inspired by: - protocolbuffers/protobuf: bazel: Remove hardcoded dependency on //:protoc from language runtimes #19679 protocolbuffers/protobuf#19679 The `proto_lang_toolchain` call was inspired by the `README` from: - https://github.com/aspect-build/toolchains_protoc/ Adds `scripts/update_protoc_integrity.py` to automatically update `scala/private/protoc/protoc_integrity.bzl`. This should make builds of `rules_scala` much faster all around. Given the fact that this feature depends on recent `protobuf` versions, and the Windows `protobuf` build breaks without it, we have a catch-22. It likely can't be separated from the rest of bazel-contrib#1710, though I would prefer that. It also seems likely that we'd eventually need to do this to continue supporting Windows, per: - protocolbuffers/protobuf#12947 - https://protobuf.dev/news/v30/#poison-msvc--bazel - protocolbuffers/protobuf#20085 More background on proto toolchainization: - Proto Toolchainisation Design Doc https://docs.google.com/document/d/1CE6wJHNfKbUPBr7-mmk_0Yo3a4TaqcTPE0OWNuQkhPs/edit - bazelbuild/bazel: Protobuf repo recompilation sensitivity bazelbuild/bazel#7095 - bazelbuild/rules_proto: Implement proto toolchainisation bazelbuild/rules_proto#179 - rules_proto 6.0.0 release notes mentioning Protobuf Toolchainisation https://github.com/bazelbuild/rules_proto/releases/tag/6.0.0

This is an experiment to see if applying protocolbuffers/protobuf#19679 via a patch and registering `@rules_scala_toolchains//protoc/...:all` will fix Windows MSVC builds. Local experiments suggest that this does indeed enable the toolchainized `protoc` to take precedence and avoid compiling protobuf.

@simuons

Bumps dependencies to versions that are compatible with both Bazel 7.5.0 and 8.0.0, and adds protocol compiler toolchainization in `//protoc` for `protobuf` v29 and later. Closes bazel-contrib#1652. Part of bazel-contrib#1482. - ScalaPB jars: 0.11.17 => 1.0.0-alpha.1 - Scalafmt: 3.9.2 => 3.9.3 - `rules_python`: 0.38.0 => 1.2.0 - `rules_cc`: 0.0.9 => 0.1.1 - `rules_java`: 7.12.4 => 8.10.0 - `protobuf`: 21.7 => 30.0 - `rules_proto`: 6.0.2 => 7.1.0 Bazel 6 is officially unsupported as of this change and the upcoming `rules_scala` 7.0.0 release. Updates `.bazelci/presubmit.yml` to bump the `7.x` build to `last_rc`. Registers a precompiled protocol compiler toolchain when `--incompatible_enable_proto_toolchain_resolution` is `True`. Otherwise, `register_toolchains(//protoc:all)` toolchains is a no-op, as it will be empty. `scripts/update_protoc_integrity.py` automatically updates `scala/private/protoc/protoc_integrity.bzl`. The `protobuf` patch is the `git diff` output from protocolbuffers/protobuf#19679, which also inspired the updates to `scala_proto/scala_proto_toolchain.bzl`. The `proto_lang_toolchain` call in the `BUILD` file generated by `protoc/private/protoc_toolchain.bzl` was inspired by the `README` from: - https://github.com/aspect-build/toolchains_protoc/ Loads `java_proto_library` from `com_google_protobuf`, replacing the officially deprecated version from `rules_java`. Bumps Scalafmt to 3.9.3 out of convenience. Updates to `README.md`, and updates to `WORKSPACE` and `third_party/repositories` files precipitated by the dependency updates, comprise the remainder of this change. --- We're no longer planning to support Bazel 6 in the next major release per @simuons's decision in: - bazel-contrib#1482 (comment) The plan is now to land the Bazel 7 and 8 compatibility updates first, then land the Bzlmod change. This enables us to make only one new major version release, instead of two (whereby the first release would've continued supporting Bazel 6). It turns out the two major version plan wouldn't've been possible. Bazel 8 and `rules_java` 8 require `protobuf` >= v29, but this bump caused Windows builds to break when compiling `protoc` in bazel-contrib#1710. `src/google/protobuf/compiler/java/java_features.pb.h`, the path specified in the error message, doesn't exist until `protobuf` v25.0. @crt-31 and I found that this was related to the Windows/MSVC 260 character file path length limit. What's more, the `protobuf` team plans to drop MSVC support specifically because of this path length limit. The protocol compiler toolchain prevents `protoc` recompilation, which fixes the Windows breakage while making all builds faster. Since Windows builds break since at least `protobuf` v25, but `protoc` toolchainization requires v29, the version bump and the `protoc` toolchain must land together. --- I tried several things to get protocol compiler toolchainization to work with `protobuf` v28.2, described below. However, each path only led to the same suffering described in the new "Why this requires `protobuf` v29 or later" section of the README. I discovered along the way that `protobuf` v30 isn't compatible with Bazel 6.5.0 at all. I added an explanation to the "Limited Bazel 6.5.0 compatibility" section of `README.md`. --- I experimented with using `protobuf` v28.2, `rules_proto` 6.0.2, and `rules_java` 7.12.4 and 8.10.0. I updated the `protobuf` patch for v28.2 with the following statements: ```py load("//bazel/common:proto_common.bzl", "proto_common") load("@rules_proto//proto:proto_common.bzl", "toolchains") _PROTO_TOOLCHAIN = "@rules_proto//proto:toolchain_type" _PROTO_TOOLCHAIN_ATTR = "INCOMPATIBLE_ENABLE_PROTO_TOOLCHAIN_RESOLUTION" _PROTOC_TOOLCHAINS = toolchains.use_toolchain(_PROTO_TOOLCHAIN) def _protoc_files_to_run(ctx): if getattr(proto_common, _PROTO_TOOLCHAIN_ATTR, False): ``` I updated `protoc/private/protoc_toolchain.bzl` to use `proto_common` from `rules_proto`. I also created a `rules_proto` 6.0.2 patch for `proto_toolchain()` to fix a "no such package: //proto" breakage: ```diff 6.0.2 patch for `proto_toolchain()`: ```diff diff --git i/proto/private/rules/proto_toolchain.bzl w/proto/private/rules/proto_toolchain.bzl index a091b80..def2699 100644 --- i/proto/private/rules/proto_toolchain.bzl +++ w/proto/private/rules/proto_toolchain.bzl @@ -33,7 +33,7 @@ def proto_toolchain(*, name, proto_compiler, exec_compatible_with = []): native.toolchain( name = name + "_toolchain", - toolchain_type = "//proto:toolchain_type", + toolchain_type = Label("//proto:toolchain_type"), exec_compatible_with = exec_compatible_with, target_compatible_with = [], toolchain = name, ``` I tried adding combinations of the following `--incompatible_autoload_externally` flag values to .bazelrc`: ```txt common --incompatible_autoload_externally=+@protobuf,+@rules_java ``` Nothing worked. --- After the `protobuf` v29 bump, and before the ScalaPB 1.0.0-alpha.1 bump, `scala_proto` targets would fail with the following error: ```txt ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14: ProtoScalaPBRule external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329) [ ...snip... ] java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14 Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ```

@simuons

Bumps dependencies to versions that are compatible with both Bazel 7.5.0 and 8.0.0, and adds protocol compiler toolchainization in `//protoc` for `protobuf` v29 and later. Closes bazel-contrib#1652. Part of bazel-contrib#1482. - ScalaPB jars: 0.11.17 => 1.0.0-alpha.1 - Scalafmt: 3.9.2 => 3.9.3 - `rules_python`: 0.38.0 => 1.2.0 - `rules_cc`: 0.0.9 => 0.1.1 - `rules_java`: 7.12.4 => 8.10.0 - `protobuf`: 21.7 => 30.0 - `rules_proto`: 6.0.2 => 7.1.0 Bazel 6 is officially unsupported as of this change and the upcoming `rules_scala` 7.0.0 release. Updates `.bazelci/presubmit.yml` to bump the `7.x` build to `last_rc`. Registers a precompiled protocol compiler toolchain when `--incompatible_enable_proto_toolchain_resolution` is `True`. Otherwise, `register_toolchains(//protoc:all)` toolchains is a no-op, as it will be empty. `scripts/update_protoc_integrity.py` automatically updates `scala/private/protoc/protoc_integrity.bzl`. The `protobuf` patch is the `git diff` output from protocolbuffers/protobuf#19679, which also inspired the updates to `scala_proto/scala_proto_toolchain.bzl`. The `proto_lang_toolchain` call in the `BUILD` file generated by `protoc/private/protoc_toolchain.bzl` was inspired by the `README` from: - https://github.com/aspect-build/toolchains_protoc/ Loads `java_proto_library` from `com_google_protobuf`, replacing the officially deprecated version from `rules_java`. Bumps Scalafmt to 3.9.3 out of convenience. Updates to `README.md`, and updates to `WORKSPACE` and `third_party/repositories` files precipitated by the dependency updates, comprise the remainder of this change. --- We're no longer planning to support Bazel 6 in the next major release per @simuons's decision in: - bazel-contrib#1482 (comment) The plan is now to land the Bazel 7 and 8 compatibility updates first, then land the Bzlmod change. This enables us to make only one new major version release, instead of two (whereby the first release would've continued supporting Bazel 6). It turns out the two major version plan wouldn't've been possible. Bazel 8 and `rules_java` 8 require `protobuf` >= v29, but this bump caused Windows builds to break when compiling `protoc` in bazel-contrib#1710. `src/google/protobuf/compiler/java/java_features.pb.h`, the path specified in the error message, doesn't exist until `protobuf` v25.0. @crt-31 and I found that this was related to the Windows/MSVC 260 character file path length limit. What's more, the `protobuf` team plans to drop MSVC support specifically because of this path length limit. The protocol compiler toolchain prevents `protoc` recompilation, which fixes the Windows breakage while making all builds faster. Since Windows builds break since at least `protobuf` v25, but `protoc` toolchainization requires v29, the version bump and the `protoc` toolchain must land together. --- I tried several things to get protocol compiler toolchainization to work with `protobuf` v28.2, described below. However, each path only led to the same suffering described in the new "Why this requires `protobuf` v29 or later" section of the README. I discovered along the way that `protobuf` v30 isn't compatible with Bazel 6.5.0 at all. I added an explanation to the "Limited Bazel 6.5.0 compatibility" section of `README.md`. --- I experimented with using `protobuf` v28.2, `rules_proto` 6.0.2, and `rules_java` 7.12.4 and 8.10.0. I updated the `protobuf` patch for v28.2 with the following statements: ```py load("//bazel/common:proto_common.bzl", "proto_common") load("@rules_proto//proto:proto_common.bzl", "toolchains") _PROTO_TOOLCHAIN = "@rules_proto//proto:toolchain_type" _PROTO_TOOLCHAIN_ATTR = "INCOMPATIBLE_ENABLE_PROTO_TOOLCHAIN_RESOLUTION" _PROTOC_TOOLCHAINS = toolchains.use_toolchain(_PROTO_TOOLCHAIN) def _protoc_files_to_run(ctx): if getattr(proto_common, _PROTO_TOOLCHAIN_ATTR, False): ``` I updated `protoc/private/protoc_toolchain.bzl` to use `proto_common` from `rules_proto`. I also created a `rules_proto` 6.0.2 patch for `proto_toolchain()` to fix a "no such package: //proto" breakage: ```diff 6.0.2 patch for `proto_toolchain()`: ```diff diff --git i/proto/private/rules/proto_toolchain.bzl w/proto/private/rules/proto_toolchain.bzl index a091b80..def2699 100644 --- i/proto/private/rules/proto_toolchain.bzl +++ w/proto/private/rules/proto_toolchain.bzl @@ -33,7 +33,7 @@ def proto_toolchain(*, name, proto_compiler, exec_compatible_with = []): native.toolchain( name = name + "_toolchain", - toolchain_type = "//proto:toolchain_type", + toolchain_type = Label("//proto:toolchain_type"), exec_compatible_with = exec_compatible_with, target_compatible_with = [], toolchain = name, ``` I tried adding combinations of the following `--incompatible_autoload_externally` flag values to .bazelrc`: ```txt common --incompatible_autoload_externally=+@protobuf,+@rules_java ``` Nothing worked. --- After the `protobuf` v29 bump, and before the ScalaPB 1.0.0-alpha.1 bump, `scala_proto` targets would fail with the following error: ```txt ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14: ProtoScalaPBRule external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329) [ ...snip... ] java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14 Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ```

comius · 2025-03-19T13:09:34Z

If I understand correctly protobuf team would like to eventually remove --protocol_compiler flag and also isn't completely decided to go in the direction of --incompatible_enable_toolchain_resolution. That's fine.

The PR is in my opinion an improvement, because it improves the consistency of protobuf code (protoc is always obtained in the same way), even if only of the legacy and to be removed code. @mkruskal would you mind accepting it as such? The (Google's) standard for code reviews is that the PR is an improvement of the code.

It doesn't get us much closer to precompiled protoc binaries, but it also doesn't hurt us, because the legacy code may still be removed as well as --protocol_compiler and --incompatible_enable_toolchain_resolution flags.

…mes (#19679) Without this change, language runtimes still result in a build of `//:protoc` even with a prebuilt `proto_toolchain` registered or `--proto_compiler` set to a precompiled protoc. Removing this hardcoded dependency allows a (fast) build of `java_proto_library` targets without a C++ toolchain assuming a prebuilt protoc. Work towards #19558 Closes #19679 COPYBARA_INTEGRATE_REVIEW=#19679 from fmeum:19558-dont-hardcode-protoc 3bd939d FUTURE_COPYBARA_INTEGRATE_REVIEW=#19679 from fmeum:19558-dont-hardcode-protoc 3bd939d PiperOrigin-RevId: 739382145

@simuons

Bumps dependencies to versions that are compatible with both Bazel 7.6.0 and 8.0.0, and adds protocol compiler toolchainization in `//protoc` for `protobuf` v29 and later. Closes bazel-contrib#1652. Part of bazel-contrib#1482. - ScalaPB jars: 0.11.17 => 1.0.0-alpha.1 - `rules_python`: 0.38.0 => 1.2.0 - `rules_cc`: 0.0.9 => 0.1.1 - `rules_java`: 7.12.4 => 8.11.0 - `protobuf`: 21.7 => 30.1 - `rules_proto`: 6.0.2 => 7.1.0 Bazel 6 is officially unsupported as of this change and the upcoming `rules_scala` 7.0.0 release. Updates `.bazelci/presubmit.yml` to bump the `7.x` build to `last_rc`. Registers a precompiled protocol compiler toolchain when `--incompatible_enable_proto_toolchain_resolution` is `True`. Otherwise, `register_toolchains("@rules_scala_protoc_toolchains//:all")` is a no-op, as it will be empty. `scripts/update_protoc_integrity.py` automatically updates `scala/private/protoc/protoc_integrity.bzl`. The `protobuf` patch is the `git diff` output from protocolbuffers/protobuf#19679, which also inspired the updates to `scala_proto/scala_proto_toolchain.bzl`. The `proto_lang_toolchain` call in the `BUILD` file generated by `protoc/private/protoc_toolchain.bzl` was inspired by the `README` from: - https://github.com/aspect-build/toolchains_protoc/ Loads `java_proto_library` from `com_google_protobuf`, replacing the officially deprecated version from `rules_java`. Adds the `scala` parameter to `scala_toolchains()` to control whether it instantiates the builtin Scala toolchains. Removes the `if len(toolchains) == 0` check from `_scala_toolchains_repo_impl`. The Scala version check will now happen only when both `scala` and `validate_scala_version` are `True`, which is essentially how the previous API worked. Updates to `README.md`, and updates to `WORKSPACE` and `third_party/repositories` files precipitated by the dependency updates, comprise the remainder of this change. --- We're no longer planning to support Bazel 6 in the next major release per @simuons's decision in: - bazel-contrib#1482 (comment) The plan is now to land the Bazel 7 and 8 compatibility updates first, then land the Bzlmod change. This enables us to make only one new major version release, instead of two (whereby the first release would've continued supporting Bazel 6). It turns out the two major version plan wouldn't've been possible. Bazel 8 and `rules_java` 8 require `protobuf` >= v29, but this bump caused Windows builds to break when compiling `protoc` in bazel-contrib#1710. `src/google/protobuf/compiler/java/java_features.pb.h`, the path specified in the error message, doesn't exist until `protobuf` v25.0. @crt-31 and I found that this was related to the Windows/MSVC 260 character file path length limit. What's more, the `protobuf` team plans to drop MSVC support specifically because of this path length limit. The protocol compiler toolchain prevents `protoc` recompilation, which fixes the Windows breakage while making all builds faster. Since Windows builds break since at least `protobuf` v25, but `protoc` toolchainization requires v29, the version bump and the `protoc` toolchain must land together. Disabling the default Scala toolchain via `scala_toolchains(scala = False)` avoids instantiating any builtin compiler JAR repos or validating the Scala version. This enables users defining custom Scala toolchains using their own JARs to still use other builtin toolchains. This was prompted by: bazel-contrib#1710 (comment) Removing the `if len(toolchains) == 0` covers the case in the upcoming Bzlmod implementation whereby the root module may explicitly disable all builtin toolchains. This avoids potential breakage of the `register_toolchains("@rules_scala_toolchains//...:all")` call from the upcoming `MODULE.bazel` file. Removing the `scala_register_toolchains()` calls from the `dt_patches/test_dt_patches*/WORKSPACE` files proves that those calls were harmless, but ultimately unnecessary. --- I tried several things to get protocol compiler toolchainization to work with `protobuf` v28.2, described below. However, each path only led to the same suffering described in the new "Why this requires `protobuf` v29 or later" section of the README. I discovered along the way that `protobuf` v30 isn't compatible with Bazel 6.5.0 at all. I added an explanation to the "Limited Bazel 6.5.0 compatibility" section of `README.md`. --- I experimented with using `protobuf` v28.2, `rules_proto` 6.0.2, and `rules_java` 7.12.4 and 8.10.0. I updated the `protobuf` patch for v28.2 with the following statements: ```py load("//bazel/common:proto_common.bzl", "proto_common") load("@rules_proto//proto:proto_common.bzl", "toolchains") _PROTO_TOOLCHAIN = "@rules_proto//proto:toolchain_type" _PROTO_TOOLCHAIN_ATTR = "INCOMPATIBLE_ENABLE_PROTO_TOOLCHAIN_RESOLUTION" _PROTOC_TOOLCHAINS = toolchains.use_toolchain(_PROTO_TOOLCHAIN) def _protoc_files_to_run(ctx): if getattr(proto_common, _PROTO_TOOLCHAIN_ATTR, False): ``` I updated `protoc/private/protoc_toolchain.bzl` to use `proto_common` from `rules_proto`. I also created a `rules_proto` 6.0.2 patch for `proto_toolchain()` to fix a "no such package: //proto" breakage: ```diff 6.0.2 patch for `proto_toolchain()`: ```diff diff --git i/proto/private/rules/proto_toolchain.bzl w/proto/private/rules/proto_toolchain.bzl index a091b80..def2699 100644 --- i/proto/private/rules/proto_toolchain.bzl +++ w/proto/private/rules/proto_toolchain.bzl @@ -33,7 +33,7 @@ def proto_toolchain(*, name, proto_compiler, exec_compatible_with = []): native.toolchain( name = name + "_toolchain", - toolchain_type = "//proto:toolchain_type", + toolchain_type = Label("//proto:toolchain_type"), exec_compatible_with = exec_compatible_with, target_compatible_with = [], toolchain = name, ``` I tried adding combinations of the following `--incompatible_autoload_externally` flag values to .bazelrc`: ```txt common --incompatible_autoload_externally=+@protobuf,+@rules_java ``` Nothing worked. --- After the `protobuf` v29 bump, and before the ScalaPB 1.0.0-alpha.1 bump, `scala_proto` targets would fail with the following error: ```txt ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14: ProtoScalaPBRule external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329) [ ...snip... ] java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14 Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ```

@simuons

Bumps dependencies to versions that are compatible with both Bazel 7.6.0 and 8.0.0, and adds protocol compiler toolchainization in `//protoc` for `protobuf` v29 and later. Closes bazel-contrib#1652. Part of bazel-contrib#1482. - ScalaPB jars: 0.11.17 => 1.0.0-alpha.1 - `rules_python`: 0.38.0 => 1.2.0 - `rules_cc`: 0.0.9 => 0.1.1 - `rules_java`: 7.12.4 => 8.11.0 - `protobuf`: 21.7 => 30.1 - `rules_proto`: 6.0.2 => 7.1.0 Bazel 6 is officially unsupported as of this change and the upcoming `rules_scala` 7.0.0 release. Updates `.bazelci/presubmit.yml` to bump the `7.x` build to `last_rc`. Registers a precompiled protocol compiler toolchain when `--incompatible_enable_proto_toolchain_resolution` is `True`. Otherwise, `register_toolchains("@rules_scala_protoc_toolchains//:all")` is a no-op, as it will be empty. `scripts/update_protoc_integrity.py` automatically updates `scala/private/protoc/protoc_integrity.bzl`. The `protobuf` patch is the `git diff` output from protocolbuffers/protobuf#19679, which also inspired the updates to `scala_proto/scala_proto_toolchain.bzl`. The `proto_lang_toolchain` call in the `BUILD` file generated by `protoc/private/protoc_toolchain.bzl` was inspired by the `README` from: - https://github.com/aspect-build/toolchains_protoc/ Loads `java_proto_library` from `com_google_protobuf`, replacing the officially deprecated version from `rules_java`. Adds the `scala` parameter to `scala_toolchains()` to control whether it instantiates the builtin Scala toolchains. Removes the `if len(toolchains) == 0` check from `_scala_toolchains_repo_impl`. The Scala version check will now happen only when both `scala` and `validate_scala_version` are `True`, which is essentially how the previous API worked. Updates to `README.md`, and updates to `WORKSPACE` and `third_party/repositories` files precipitated by the dependency updates, comprise the remainder of this change. --- We're no longer planning to support Bazel 6 in the next major release per @simuons's decision in: - bazel-contrib#1482 (comment) The plan is now to land the Bazel 7 and 8 compatibility updates first, then land the Bzlmod change. This enables us to make only one new major version release, instead of two (whereby the first release would've continued supporting Bazel 6). It turns out the two major version plan wouldn't've been possible. Bazel 8 and `rules_java` 8 require `protobuf` >= v29, but this bump caused Windows builds to break when compiling `protoc` in bazel-contrib#1710. `src/google/protobuf/compiler/java/java_features.pb.h`, the path specified in the error message, doesn't exist until `protobuf` v25.0. @crt-31 and I found that this was related to the Windows/MSVC 260 character file path length limit. What's more, the `protobuf` team plans to drop MSVC support specifically because of this path length limit. The protocol compiler toolchain prevents `protoc` recompilation, which fixes the Windows breakage while making all builds faster. Since Windows builds break since at least `protobuf` v25, but `protoc` toolchainization requires v29, the version bump and the `protoc` toolchain must land together. Disabling the default Scala toolchain via `scala_toolchains(scala = False)` avoids instantiating any builtin compiler JAR repos or validating the Scala version. This enables users defining custom Scala toolchains using their own JARs to still use other builtin toolchains. This was prompted by: bazel-contrib#1710 (comment) Removing the `if len(toolchains) == 0` covers the case in the upcoming Bzlmod implementation whereby the root module may explicitly disable all builtin toolchains. This avoids potential breakage of the `register_toolchains("@rules_scala_toolchains//...:all")` call from the upcoming `MODULE.bazel` file. Removing the `scala_register_toolchains()` calls from the `dt_patches/test_dt_patches*/WORKSPACE` files proves that those calls were harmless, but ultimately unnecessary. --- I tried several things to get protocol compiler toolchainization to work with `protobuf` v28.2, described below. However, each path only led to the same suffering described in the new "Why this requires `protobuf` v29 or later" section of the README. I discovered along the way that `protobuf` v30 isn't compatible with Bazel 6.5.0 at all. I added an explanation to the "Limited Bazel 6.5.0 compatibility" section of `README.md`. --- I experimented with using `protobuf` v28.2, `rules_proto` 6.0.2, and `rules_java` 7.12.4 and 8.10.0. I updated the `protobuf` patch for v28.2 with the following statements: ```py load("//bazel/common:proto_common.bzl", "proto_common") load("@rules_proto//proto:proto_common.bzl", "toolchains") _PROTO_TOOLCHAIN = "@rules_proto//proto:toolchain_type" _PROTO_TOOLCHAIN_ATTR = "INCOMPATIBLE_ENABLE_PROTO_TOOLCHAIN_RESOLUTION" _PROTOC_TOOLCHAINS = toolchains.use_toolchain(_PROTO_TOOLCHAIN) def _protoc_files_to_run(ctx): if getattr(proto_common, _PROTO_TOOLCHAIN_ATTR, False): ``` I updated `protoc/private/protoc_toolchain.bzl` to use `proto_common` from `rules_proto`. I also created a `rules_proto` 6.0.2 patch for `proto_toolchain()` to fix a "no such package: //proto" breakage: ```diff 6.0.2 patch for `proto_toolchain()`: ```diff diff --git i/proto/private/rules/proto_toolchain.bzl w/proto/private/rules/proto_toolchain.bzl index a091b80..def2699 100644 --- i/proto/private/rules/proto_toolchain.bzl +++ w/proto/private/rules/proto_toolchain.bzl @@ -33,7 +33,7 @@ def proto_toolchain(*, name, proto_compiler, exec_compatible_with = []): native.toolchain( name = name + "_toolchain", - toolchain_type = "//proto:toolchain_type", + toolchain_type = Label("//proto:toolchain_type"), exec_compatible_with = exec_compatible_with, target_compatible_with = [], toolchain = name, ``` I tried adding combinations of the following `--incompatible_autoload_externally` flag values to .bazelrc`: ```txt common --incompatible_autoload_externally=+@protobuf,+@rules_java ``` Nothing worked. --- After the `protobuf` v29 bump, and before the ScalaPB 1.0.0-alpha.1 bump, `scala_proto` targets would fail with the following error: ```txt ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14: ProtoScalaPBRule external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329) [ ...snip... ] java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14 Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ```

@simuons

Bumps dependencies to versions that are compatible with both Bazel 7.6.0 and 8.0.0, and adds protocol compiler toolchainization in `//protoc` for `protobuf` v29 and later. Closes bazel-contrib#1652. Part of bazel-contrib#1482. - ScalaPB jars: 0.11.17 => 1.0.0-alpha.1 - `rules_python`: 0.38.0 => 1.2.0 - `rules_cc`: 0.0.9 => 0.1.1 - `rules_java`: 7.12.4 => 8.11.0 - `protobuf`: 21.7 => 30.1 - `rules_proto`: 6.0.2 => 7.1.0 Bazel 6 is officially unsupported as of this change and the upcoming `rules_scala` 7.0.0 release. Updates `.bazelci/presubmit.yml` to bump the `7.x` build to `last_rc`. Registers a precompiled protocol compiler toolchain when `--incompatible_enable_proto_toolchain_resolution` is `True`. Otherwise, `register_toolchains("@rules_scala_protoc_toolchains//:all")` is a no-op, as it will be empty. `scripts/update_protoc_integrity.py` automatically updates `scala/private/protoc/protoc_integrity.bzl`. The `protobuf` patch is the `git diff` output from protocolbuffers/protobuf#19679, which also inspired the updates to `scala_proto/scala_proto_toolchain.bzl`. The `proto_lang_toolchain` call in the `BUILD` file generated by `protoc/private/protoc_toolchains.bzl` was inspired by the `README` from: - https://github.com/aspect-build/toolchains_protoc/ Loads `java_proto_library` from `com_google_protobuf`, replacing the officially deprecated version from `rules_java`. Adds the `scala` parameter to `scala_toolchains()` to control whether it instantiates the builtin Scala toolchains. Removes the `if len(toolchains) == 0` check from `_scala_toolchains_repo_impl`. The Scala version check will now happen only when both `scala` and `validate_scala_version` are `True`, which is essentially how the previous API worked. Updates to `README.md`, and updates to `WORKSPACE` and `third_party/repositories` files precipitated by the dependency updates, comprise the remainder of this change. --- We're no longer planning to support Bazel 6 in the next major release per @simuons's decision in: - bazel-contrib#1482 (comment) The plan is now to land the Bazel 7 and 8 compatibility updates first, then land the Bzlmod change. This enables us to make only one new major version release, instead of two (whereby the first release would've continued supporting Bazel 6). It turns out the two major version plan wouldn't've been possible. Bazel 8 and `rules_java` 8 require `protobuf` >= v29, but this bump caused Windows builds to break when compiling `protoc` in bazel-contrib#1710. `src/google/protobuf/compiler/java/java_features.pb.h`, the path specified in the error message, doesn't exist until `protobuf` v25.0. @crt-31 and I found that this was related to the Windows/MSVC 260 character file path length limit. What's more, the `protobuf` team plans to drop MSVC support specifically because of this path length limit. The protocol compiler toolchain prevents `protoc` recompilation, which fixes the Windows breakage while making all builds faster. Since Windows builds break since at least `protobuf` v25, but `protoc` toolchainization requires v29, the version bump and the `protoc` toolchain must land together. Disabling the default Scala toolchain via `scala_toolchains(scala = False)` avoids instantiating any builtin compiler JAR repos or validating the Scala version. This enables users defining custom Scala toolchains using their own JARs to still use other builtin toolchains. This was prompted by: bazel-contrib#1710 (comment) Removing the `if len(toolchains) == 0` covers the case in the upcoming Bzlmod implementation whereby the root module may explicitly disable all builtin toolchains. This avoids potential breakage of the `register_toolchains("@rules_scala_toolchains//...:all")` call from the upcoming `MODULE.bazel` file. Removing the `scala_register_toolchains()` calls from the `dt_patches/test_dt_patches*/WORKSPACE` files proves that those calls were harmless, but ultimately unnecessary. --- I tried several things to get protocol compiler toolchainization to work with `protobuf` v28.2, described below. However, each path only led to the same suffering described in the new "Why this requires `protobuf` v29 or later" section of the README. I discovered along the way that `protobuf` v30 isn't compatible with Bazel 6.5.0 at all. I added an explanation to the "Limited Bazel 6.5.0 compatibility" section of `README.md`. --- I experimented with using `protobuf` v28.2, `rules_proto` 6.0.2, and `rules_java` 7.12.4 and 8.10.0. I updated the `protobuf` patch for v28.2 with the following statements: ```py load("//bazel/common:proto_common.bzl", "proto_common") load("@rules_proto//proto:proto_common.bzl", "toolchains") _PROTO_TOOLCHAIN = "@rules_proto//proto:toolchain_type" _PROTO_TOOLCHAIN_ATTR = "INCOMPATIBLE_ENABLE_PROTO_TOOLCHAIN_RESOLUTION" _PROTOC_TOOLCHAINS = toolchains.use_toolchain(_PROTO_TOOLCHAIN) def _protoc_files_to_run(ctx): if getattr(proto_common, _PROTO_TOOLCHAIN_ATTR, False): ``` I tried using `proto_common` from `rules_proto`. I also created a `rules_proto` 6.0.2 patch for `proto_toolchain()` to fix a "no such package: //proto" breakage: ```diff diff --git i/proto/private/rules/proto_toolchain.bzl w/proto/private/rules/proto_toolchain.bzl index a091b80..def2699 100644 --- i/proto/private/rules/proto_toolchain.bzl +++ w/proto/private/rules/proto_toolchain.bzl @@ -33,7 +33,7 @@ def proto_toolchain(*, name, proto_compiler, exec_compatible_with = []): native.toolchain( name = name + "_toolchain", - toolchain_type = "//proto:toolchain_type", + toolchain_type = Label("//proto:toolchain_type"), exec_compatible_with = exec_compatible_with, target_compatible_with = [], toolchain = name, ``` I tried adding combinations of the following `--incompatible_autoload_externally` flag values to .bazelrc`: ```txt common --incompatible_autoload_externally=+@protobuf,+@rules_java ``` Nothing worked. --- After the `protobuf` v29 bump, and before the ScalaPB 1.0.0-alpha.1 bump, `scala_proto` targets would fail with the following error: ```txt ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14: ProtoScalaPBRule external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329) [ ...snip... ] java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14 Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ```

@simuons

Bumps dependencies to versions that are compatible with both Bazel 7.6.0 and 8.0.0, and adds protocol compiler toolchainization in `//protoc` for `protobuf` v29 and later. Closes bazel-contrib#1652. Part of bazel-contrib#1482. - ScalaPB jars: 0.11.17 => 1.0.0-alpha.1 - `rules_python`: 0.38.0 => 1.2.0 - `rules_cc`: 0.0.9 => 0.1.1 - `rules_java`: 7.12.4 => 8.11.0 - `protobuf`: 21.7 => 30.1 - `rules_proto`: 6.0.2 => 7.1.0 Bazel 6 is officially unsupported as of this change and the upcoming `rules_scala` 7.0.0 release. Updates `.bazelci/presubmit.yml` to bump the `7.x` build to `last_rc`. Registers a precompiled protocol compiler toolchain when `--incompatible_enable_proto_toolchain_resolution` is `True`. Otherwise, `register_toolchains("@rules_scala_protoc_toolchains//:all")` is a no-op, as it will be empty. `scripts/update_protoc_integrity.py` automatically updates `scala/private/protoc/protoc_integrity.bzl`. The `protobuf` patch is the `git diff` output from protocolbuffers/protobuf#19679, which also inspired the updates to `scala_proto/scala_proto_toolchain.bzl`. The `proto_lang_toolchain` call in the `BUILD` file generated by `protoc/private/protoc_toolchains.bzl` was inspired by the `README` from: - https://github.com/aspect-build/toolchains_protoc/ Loads `java_proto_library` from `com_google_protobuf`, replacing the officially deprecated version from `rules_java`. Adds the `scala` parameter to `scala_toolchains()` to control whether it instantiates the builtin Scala toolchains. Removes the `if len(toolchains) == 0` check from `_scala_toolchains_repo_impl`. The Scala version check will now happen only when both `scala` and `validate_scala_version` are `True`, which is essentially how the previous API worked. Updates to `README.md`, and updates to `WORKSPACE` and `third_party/repositories` files precipitated by the dependency updates, comprise the remainder of this change. --- We're no longer planning to support Bazel 6 in the next major release per @simuons's decision in: - bazel-contrib#1482 (comment) The plan is now to land the Bazel 7 and 8 compatibility updates first, then land the Bzlmod change. This enables us to make only one new major version release, instead of two (whereby the first release would've continued supporting Bazel 6). It turns out the two major version plan wouldn't've been possible. Bazel 8 and `rules_java` 8 require `protobuf` >= v29, but this bump caused Windows builds to break when compiling `protoc` in bazel-contrib#1710. `src/google/protobuf/compiler/java/java_features.pb.h`, the path specified in the error message, doesn't exist until `protobuf` v25.0. @crt-31 and I found that this was related to the Windows/MSVC 260 character file path length limit. What's more, the `protobuf` team plans to drop MSVC support specifically because of this path length limit. The protocol compiler toolchain prevents `protoc` recompilation, which fixes the Windows breakage while making all builds faster. Since Windows builds break since at least `protobuf` v25, but `protoc` toolchainization requires v29, the version bump and the `protoc` toolchain must land together. Disabling the default Scala toolchain via `scala_toolchains(scala = False)` avoids instantiating any builtin compiler JAR repos or validating the Scala version. This enables users defining custom Scala toolchains using their own JARs to still use other builtin toolchains. This was prompted by: bazel-contrib#1710 (comment) Removing the `if len(toolchains) == 0` covers the case in the upcoming Bzlmod implementation whereby the root module may explicitly disable all builtin toolchains. This avoids potential breakage of the `register_toolchains("@rules_scala_toolchains//...:all")` call from the upcoming `MODULE.bazel` file. Removing the `scala_register_toolchains()` calls from the `dt_patches/test_dt_patches*/WORKSPACE` files proves that those calls were harmless, but ultimately unnecessary. --- I tried several things to get protocol compiler toolchainization to work with `protobuf` v28.2, described below. However, each path only led to the same suffering described in the new "Why this requires `protobuf` v29 or later" section of the README. I discovered along the way that `protobuf` v30 isn't compatible with Bazel 6.5.0 at all. I added an explanation to the "Limited Bazel 6.5.0 compatibility" section of `README.md`. --- I experimented with using `protobuf` v28.2, `rules_proto` 6.0.2, and `rules_java` 7.12.4 and 8.10.0. I updated the `protobuf` patch for v28.2 with the following statements: ```py load("//bazel/common:proto_common.bzl", "proto_common") load("@rules_proto//proto:proto_common.bzl", "toolchains") _PROTO_TOOLCHAIN = "@rules_proto//proto:toolchain_type" _PROTO_TOOLCHAIN_ATTR = "INCOMPATIBLE_ENABLE_PROTO_TOOLCHAIN_RESOLUTION" _PROTOC_TOOLCHAINS = toolchains.use_toolchain(_PROTO_TOOLCHAIN) def _protoc_files_to_run(ctx): if getattr(proto_common, _PROTO_TOOLCHAIN_ATTR, False): ``` I tried using `proto_common` from `rules_proto`. I also created a `rules_proto` 6.0.2 patch for `proto_toolchain()` to fix a "no such package: //proto" breakage: ```diff diff --git i/proto/private/rules/proto_toolchain.bzl w/proto/private/rules/proto_toolchain.bzl index a091b80..def2699 100644 --- i/proto/private/rules/proto_toolchain.bzl +++ w/proto/private/rules/proto_toolchain.bzl @@ -33,7 +33,7 @@ def proto_toolchain(*, name, proto_compiler, exec_compatible_with = []): native.toolchain( name = name + "_toolchain", - toolchain_type = "//proto:toolchain_type", + toolchain_type = Label("//proto:toolchain_type"), exec_compatible_with = exec_compatible_with, target_compatible_with = [], toolchain = name, ``` I tried adding combinations of the following `--incompatible_autoload_externally` flag values to .bazelrc`: ```txt common --incompatible_autoload_externally=+@protobuf,+@rules_java ``` Nothing worked. --- After the `protobuf` v29 bump, and before the ScalaPB 1.0.0-alpha.1 bump, `scala_proto` targets would fail with the following error: ```txt ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14: ProtoScalaPBRule external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329) [ ...snip... ] java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14 Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ```

@simuons

Bumps dependencies to versions that are compatible with both Bazel 7.6.0 and 8.0.0, and adds protocol compiler toolchainization in `//protoc` for `protobuf` v29 and later. Closes #1652. Part of #1482. - ScalaPB jars: 0.11.17 => 1.0.0-alpha.1 - `rules_python`: 0.38.0 => 1.2.0 - `rules_cc`: 0.0.9 => 0.1.1 - `rules_java`: 7.12.4 => 8.11.0 - `protobuf`: 21.7 => 30.1 - `rules_proto`: 6.0.2 => 7.1.0 Bazel 6 is officially unsupported as of this change and the upcoming `rules_scala` 7.0.0 release. Updates `.bazelci/presubmit.yml` to bump the `7.x` build to `last_rc`. Registers a precompiled protocol compiler toolchain when `--incompatible_enable_proto_toolchain_resolution` is `True`. Otherwise, `register_toolchains("@rules_scala_protoc_toolchains//:all")` is a no-op, as it will be empty. `scripts/update_protoc_integrity.py` automatically updates `scala/private/protoc/protoc_integrity.bzl`. The `protobuf` patch is the `git diff` output from protocolbuffers/protobuf#19679, which also inspired the updates to `scala_proto/scala_proto_toolchain.bzl`. The `proto_lang_toolchain` call in the `BUILD` file generated by `protoc/private/protoc_toolchains.bzl` was inspired by the `README` from: - https://github.com/aspect-build/toolchains_protoc/ Loads `java_proto_library` from `com_google_protobuf`, replacing the officially deprecated version from `rules_java`. Adds the `scala` parameter to `scala_toolchains()` to control whether it instantiates the builtin Scala toolchains. Removes the `if len(toolchains) == 0` check from `_scala_toolchains_repo_impl`. The Scala version check will now happen only when both `scala` and `validate_scala_version` are `True`, which is essentially how the previous API worked. Updates to `README.md`, and updates to `WORKSPACE` and `third_party/repositories` files precipitated by the dependency updates, comprise the remainder of this change. --- We're no longer planning to support Bazel 6 in the next major release per @simuons's decision in: - #1482 (comment) The plan is now to land the Bazel 7 and 8 compatibility updates first, then land the Bzlmod change. This enables us to make only one new major version release, instead of two (whereby the first release would've continued supporting Bazel 6). It turns out the two major version plan wouldn't've been possible. Bazel 8 and `rules_java` 8 require `protobuf` >= v29, but this bump caused Windows builds to break when compiling `protoc` in #1710. `src/google/protobuf/compiler/java/java_features.pb.h`, the path specified in the error message, doesn't exist until `protobuf` v25.0. @crt-31 and I found that this was related to the Windows/MSVC 260 character file path length limit. What's more, the `protobuf` team plans to drop MSVC support specifically because of this path length limit. The protocol compiler toolchain prevents `protoc` recompilation, which fixes the Windows breakage while making all builds faster. Since Windows builds break since at least `protobuf` v25, but `protoc` toolchainization requires v29, the version bump and the `protoc` toolchain must land together. Disabling the default Scala toolchain via `scala_toolchains(scala = False)` avoids instantiating any builtin compiler JAR repos or validating the Scala version. This enables users defining custom Scala toolchains using their own JARs to still use other builtin toolchains. This was prompted by: #1710 (comment) Removing the `if len(toolchains) == 0` covers the case in the upcoming Bzlmod implementation whereby the root module may explicitly disable all builtin toolchains. This avoids potential breakage of the `register_toolchains("@rules_scala_toolchains//...:all")` call from the upcoming `MODULE.bazel` file. Removing the `scala_register_toolchains()` calls from the `dt_patches/test_dt_patches*/WORKSPACE` files proves that those calls were harmless, but ultimately unnecessary. --- I tried several things to get protocol compiler toolchainization to work with `protobuf` v28.2, described below. However, each path only led to the same suffering described in the new "Why this requires `protobuf` v29 or later" section of the README. I discovered along the way that `protobuf` v30 isn't compatible with Bazel 6.5.0 at all. I added an explanation to the "Limited Bazel 6.5.0 compatibility" section of `README.md`. --- I experimented with using `protobuf` v28.2, `rules_proto` 6.0.2, and `rules_java` 7.12.4 and 8.10.0. I updated the `protobuf` patch for v28.2 with the following statements: ```py load("//bazel/common:proto_common.bzl", "proto_common") load("@rules_proto//proto:proto_common.bzl", "toolchains") _PROTO_TOOLCHAIN = "@rules_proto//proto:toolchain_type" _PROTO_TOOLCHAIN_ATTR = "INCOMPATIBLE_ENABLE_PROTO_TOOLCHAIN_RESOLUTION" _PROTOC_TOOLCHAINS = toolchains.use_toolchain(_PROTO_TOOLCHAIN) def _protoc_files_to_run(ctx): if getattr(proto_common, _PROTO_TOOLCHAIN_ATTR, False): ``` I tried using `proto_common` from `rules_proto`. I also created a `rules_proto` 6.0.2 patch for `proto_toolchain()` to fix a "no such package: //proto" breakage: ```diff diff --git i/proto/private/rules/proto_toolchain.bzl w/proto/private/rules/proto_toolchain.bzl index a091b80..def2699 100644 --- i/proto/private/rules/proto_toolchain.bzl +++ w/proto/private/rules/proto_toolchain.bzl @@ -33,7 +33,7 @@ def proto_toolchain(*, name, proto_compiler, exec_compatible_with = []): native.toolchain( name = name + "_toolchain", - toolchain_type = "//proto:toolchain_type", + toolchain_type = Label("//proto:toolchain_type"), exec_compatible_with = exec_compatible_with, target_compatible_with = [], toolchain = name, ``` I tried adding combinations of the following `--incompatible_autoload_externally` flag values to .bazelrc`: ```txt common --incompatible_autoload_externally=+@protobuf,+@rules_java ``` Nothing worked. --- After the `protobuf` v29 bump, and before the ScalaPB 1.0.0-alpha.1 bump, `scala_proto` targets would fail with the following error: ```txt ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14: ProtoScalaPBRule external/com_google_protobuf/src/google/protobuf/any_proto_jvm_extra_protobuf_generator_scalapb.srcjar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/.../bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) --jvm_extra_protobuf_generator_out: java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$FieldOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)' at scalapb.compiler.DescriptorImplicits$ExtendedFieldDescriptor.fieldOptions(DescriptorImplicits.scala:329) [ ...snip... ] java.lang.RuntimeException: Exit with code 1 at scala.sys.package$.error(package.scala:30) at scripts.ScalaPBWorker$.work(ScalaPBWorker.scala:44) at io.bazel.rulesscala.worker.Worker.persistentWorkerMain(Worker.java:96) at io.bazel.rulesscala.worker.Worker.workerMain(Worker.java:49) at scripts.ScalaPBWorker$.main(ScalaPBWorker.scala:39) at scripts.ScalaPBWorker.main(ScalaPBWorker.scala) ERROR: .../external/com_google_protobuf/src/google/protobuf/BUILD.bazel:23:14 Building source jar external/com_google_protobuf/src/google/protobuf/any_proto_scalapb-src.jar failed: (Exit 1): scalapb_worker failed: error executing ProtoScalaPBRule command (from target @@com_google_protobuf//src/google/protobuf:any_proto) bazel-out/darwin_arm64-opt-exec-ST-a828a81199fe/bin/src/scala/scripts/scalapb_worker ... (remaining 2 arguments skipped) ```

adincebic · 2025-04-30T18:42:17Z

@fmeum do you know what's required for this to land?

fmeum · 2025-04-30T18:54:02Z

@mkruskal-google Friendly ping

mbland · 2025-04-30T19:56:19Z

As another data point, I made a patch from this pull request for bazel-contrib/rules_scala. I created it while working towards the upcoming Bzlmod-enabled rules_scala release (bazel-contrib/rules_scala#1482), when our Windows builds failed after upgrading past protobuf v21.7 in bazel-contrib/rules_scala#1710 .

The breakage was due to later protobuf versions pushing some file paths past the cursed 260 character limit, breaking Windows MSVC source builds. Since protobuf support for MSVC is officially dead for precisely this reason (references below), this patch became essential to fixing our Windows builds via --incompatible_enable_proto_toolchain_resolution.

The precompiled protoc toolchain mechanism is only mandatory for Windows builds, but also shaves minutes off the build for Linux and macOS. Though we'll have to maintain the patch as long as current protobuf releases are in widespread use, it would be nice to not have to require it for future releases.

mkruskal-google · 2025-04-30T20:09:23Z

Sorry for the delay in responding here. While we do understand that this is a strict improvement to the current situation, it only further endorses a deprecated set of Bazel features we want to turn down completely. We would prefer to focus on providing a future-facing solution to the root problem, and are working towards providing a supported mechanism to use prebuilt binaries with the protobuf rules.

Note: we have no intention of killing MSVC+Bazel support before this solution is in place. There is currently an opt-out flag to allow building from source with MSVC that will be in place until at least 34.0. Setting --output_user_root=C:/ should get you past the 255 character limit issue, as we've have CI to prevent further accidental breakages for a while now

fmeum · 2025-04-30T20:28:52Z

@mkruskal-google Could you elaborate on what these deprecated Bazel features are? As far as I can tell, proto toolchainization is a much more robust and modern approach than --proto_compiler (which should definitely just go away).

Please also keep in mind that prebuilt binaries are not the end of the story. You won't be able to supply prebuilt binaries for all OS/architecture pairs, but users on such platforms can't be limited to from source builds.

mkruskal-google · 2025-04-30T20:51:07Z

--incompatible_enable_proto_toolchain_resolution and --proto_compiler are the deprecated features. The first one is marked "incompatible" today and we do not intend to promote it.

We're still in design discussions atm (with aspect.build), and generic toolchains vs just prebuilts is a key piece that hasn't been decided yet. You make a good point about arbitrary OS/arch pairs, but OTOH the problem with toolchains is just that there's so many ways to shoot yourself in the foot because of our cross-version compatibility constraints. If we did go that way there would need to be a lot of design consideration into ways to make that safer (or at least guide users away from bad patterns)

mbland · 2025-04-30T21:28:13Z

Note: we have no intention of killing MSVC+Bazel support before this solution is in place. There is currently an opt-out flag to allow building from source with MSVC that will be in place until at least 34.0. Setting --output_user_root=C:/ should get you past the 255 character limit issue, as we've have CI to prevent further accidental breakages for a while now

rules_scala uses Bazel's Buildkite CI system, bazelbuild/continuous-integration. bazel-contrib/rules_scala#1710 (comment) shows the Windows MSVC build breaking on protobuf v29.3:

ERROR: C:/tools/msys64/home/b/_bazel_b/xknd5zlq/external/com_google_protobuf/src/google/protobuf/compiler/java/BUILD.bazel:87:11:
  Compiling src/google/protobuf/compiler/java/java_features.pb.cc [for tool] failed: (Exit 2): cl.exe failed:
  error executing CppCompile command
  (from target @@com_google_protobuf//src/google/protobuf/compiler/java:java_features_bootstrap)
  C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.39.33519\bin\HostX64\x64\cl.exe ...
  (remaining 1 argument skipped)
--
  | external/com_google_protobuf/src/google/protobuf/compiler/java/java_features.pb.cc(6): fatal error C1083:
    Cannot open include file: 'google/protobuf/compiler/java/java_features.pb.h': No such file or directory

The full build results show bazel --output_user_root=C:/b. Perhaps we could try setting it to C:/ to see if that one character makes a difference. Trying to run under cmd.exe instead of MSYS2 may also help (since C:/tools/msys64/home/b/... appears in the ERROR: message), and be good for other reasons. That might take some work, since the bulk of rules_scala tests are Bash scripts invoking bazel in various ways. But both potential experiments are effectively moot, at least for the moment. Right now we're using the precompiled protoc to make progress, and it's making both CI and local development faster.

FWIW, I'm totally on board with replacing the old and busted with the new hotness. (This explains my commitment to completing bazel-contrib/rules_scala#1482.) So if there's a better solution around the corner, I'll be amongst the first to adopt it. But until then, or until this pull request lands, we have to keep patching protobuf to unblock Windows builds of rules_scala. We'll also keep patching it for the overall benefit of avoiding protoc recompiles on all platforms.

mkruskal-google · 2025-04-30T23:24:23Z

O yea msys2 is definitely the problem there. You're getting extra characters from tools/msys64/home, which is going to push you over the limit for sure. We're very close to the limit and I think C:\tmp is the extent of what Bazel+MSVC can handle today (it's what our CI covers).

bazel: Remove hardcoded dependency on //:protoc from language runtimes

bc1a0e3

Without this change, language runtimes still result in a build of `//:protoc` even with a prebuilt `proto_toolchain` registered or `--proto_compiler` set to a precompiled protoc.

fmeum marked this pull request as ready for review December 16, 2024 15:25

Switch to toolchains

3bd939d

fmeum force-pushed the 19558-dont-hardcode-protoc branch from d92eae2 to 3bd939d Compare December 16, 2024 21:20

zhangskz added the bazel label Dec 18, 2024

zhangskz requested a review from mkruskal-google December 18, 2024 16:57

mkruskal-google added the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Dec 18, 2024

github-actions bot removed the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Dec 18, 2024

JasonLunn added the wait for user action label Dec 23, 2024

fmeum mentioned this pull request Jan 15, 2025

chore: add Bazel 8 testing aspect-build/toolchains_protoc#39

Merged

alexeagle mentioned this pull request Jan 16, 2025

feat: re-enable Java example of using pre-built protoc aspect-build/toolchains_protoc#41

Closed

tonyliaoss removed the wait for user action label Feb 19, 2025

mbland mentioned this pull request Mar 1, 2025

Bazel 8 + rules_java 8 updates, protoc toolchain bazel-contrib/rules_scala#1710

Merged

mbland mentioned this pull request Mar 3, 2025

Support Bzlmod and add rules_scala to bazel-central-registry bazel-contrib/rules_scala#1482

Open

copybara-service bot mentioned this pull request Mar 22, 2025

bazel: Remove hardcoded dependency on //:protoc from language runtimes (#19679) #20794

Closed

Merge remote-tracking branch 'upstream' into 19558-dont-hardcode-protoc

2d67e82

mkruskal-google added the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Mar 22, 2025

github-actions bot removed the 🅰️ safe for tests Mark a commit as safe to run presubmits over label Mar 22, 2025

alexeagle mentioned this pull request Apr 1, 2025

[FR]: Add example for cc_proto_library aspect-build/toolchains_protoc#21

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bazel: Remove hardcoded dependency on `//:protoc` from language runtimes #19679

bazel: Remove hardcoded dependency on `//:protoc` from language runtimes #19679

fmeum commented Dec 16, 2024 •

edited

Loading

fmeum commented Dec 16, 2024

fmeum commented Dec 16, 2024

mkruskal-google commented Dec 19, 2024

fmeum commented Dec 30, 2024

comius commented Mar 19, 2025

adincebic commented Apr 30, 2025

fmeum commented Apr 30, 2025

mbland commented Apr 30, 2025

mkruskal-google commented Apr 30, 2025 •

edited

Loading

fmeum commented Apr 30, 2025

mkruskal-google commented Apr 30, 2025 •

edited

Loading

mbland commented Apr 30, 2025

mkruskal-google commented Apr 30, 2025 •

edited

Loading

bazel: Remove hardcoded dependency on //:protoc from language runtimes #19679

Are you sure you want to change the base?

bazel: Remove hardcoded dependency on //:protoc from language runtimes #19679

Conversation

fmeum commented Dec 16, 2024 • edited Loading

fmeum commented Dec 16, 2024

fmeum commented Dec 16, 2024

mkruskal-google commented Dec 19, 2024

fmeum commented Dec 30, 2024

comius commented Mar 19, 2025

adincebic commented Apr 30, 2025

fmeum commented Apr 30, 2025

mbland commented Apr 30, 2025

mkruskal-google commented Apr 30, 2025 • edited Loading

fmeum commented Apr 30, 2025

mkruskal-google commented Apr 30, 2025 • edited Loading

mbland commented Apr 30, 2025

mkruskal-google commented Apr 30, 2025 • edited Loading

bazel: Remove hardcoded dependency on `//:protoc` from language runtimes #19679

bazel: Remove hardcoded dependency on `//:protoc` from language runtimes #19679

fmeum commented Dec 16, 2024 •

edited

Loading

mkruskal-google commented Apr 30, 2025 •

edited

Loading

mkruskal-google commented Apr 30, 2025 •

edited

Loading

mkruskal-google commented Apr 30, 2025 •

edited

Loading