Move rules_scala_dependencies to scala/deps.bzl

mbland · mbland · commit 433bc0c5fded · 2024-12-03T11:18:09.000-05:00
Ensures we get all the versions of dependencies we want in `WORKSPACE`, while providing a new API to consumers. Part of bazel-contrib#1652. Bumps several packages as high as they can go and still be compatible with Bazel 6 and 7: - `bazel_skylib`: 1.4.1 => 1.7.1 - `rules_cc`: 0.0.6 => 0.0.9 - `rules_python`: 0.36.0 => 0.38.0 - `rules_go`: 0.50.1 The following packages are at the maximum version to prevent breakages under Bazel 6.5.0. --- `abseil-cpp` and `protobuf` have to stay at 20220623.1 and v27.1, respectively, for Bazel 6 compatibility per bazel-contrib#1647. `protobuf` up to v25.5 is compatible with Bazel 6 provided users set the compiler flags mentioned in that issue. --- `rules_python` 0.38.0 => 0.39.0 requires at least `rules_cc` 0.0.10, which introduced `cc/common/cc_info.bzl`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:19:6: Label '@rules_cc//cc/common:cc_info.bzl' is invalid because 'cc/common' is not a package; perhaps you meant to put the colon here: '@rules_cc//cc:common/cc_info.bzl'? ``` --- `rules_cc` 0.0.9 => 0.0.10 requires Bazel 7, which defines `CcSharedLibraryHintInfo`: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_cc/cc/private/rules_impl/native.bzl:40:33: name 'CcSharedLibraryHintInfo' is not defined (did you mean 'CcSharedLibraryInfo'?) [ ...snip... ] ERROR: error loading package under directory 'test': error loading package 'test': at .../external/rules_python/python/defs.bzl:17:6: at .../external/rules_python/python/py_binary.bzl:18:6: at .../external/rules_python/python/private/py_binary_macro.bzl:16:6: at .../external/rules_python/python/private/common_bazel.bzl:18:6: at .../external/rules_cc/cc/common/cc_common.bzl:17:6: compilation of module 'cc/private/rules_impl/native.bzl' failed ``` --- `rules_java` remains at 7.9.0 due to the more complicated situation it presents. The current version that could be compatible with Bazel 6 and 7 is 7.12.2. As it turns out, we could include `rules_java` versions up to and including 7.12.2 if we don't call the following in `WORKSPACE`: ```py load("@rules_java//java:repositories.bzl", "rules_java_dependencies", "rules_java_toolchains") rules_java_dependencies() rules_java_toolchains() ``` In fact, __we don't do that today__. When we do, Bazel 7.4.1 builds successfully, but Bazel 6.5.2 fails with: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/toolchains/java_toolchain_alias.bzl:83:34: Use of Starlark transition without allowlist attribute '_allowlist_function_transition'. See Starlark transitions documentation for details and usage: @rules_java//toolchains:java_toolchain_alias.bzl NORMAL ERROR: .../src/java/io/bazel/rulesscala/coverage/instrumenter/BUILD:3:12: While resolving toolchains for target //src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter: invalid registered toolchain '//toolchains:all': while parsing '//toolchains:all': error loading package '@rules_java//toolchains': initialization of module 'toolchains/java_toolchain_alias.bzl' failed ERROR: Analysis of target '//src/java/io/bazel/rulesscala/coverage/instrumenter:instrumenter' failed; build aborted: ``` When we bump to rules_java 7.10.0, which contains bazelbuild/rules_java#210, Bazel 7.4.1 fails with: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java_builtin/toolchains/BUILD:254:14: While resolving toolchains for target @@rules_java_builtin//toolchains:platformclasspath (096dcc8): No matching toolchains found for types @@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type. To debug, rerun with --toolchain_resolution_debug='@@bazel_tools//tools/jdk:bootstrap_runtime_toolchain_type' If platforms or toolchains are a new concept for you, we'd encourage reading https://bazel.build/concepts/platforms-intro. ERROR: Analysis of target '//test/toolchains:java21_toolchain' failed; build aborted: Analysis failed ``` This is the error I described in bazel-contrib#1619, under "Bump to rules_java 7.9.0 for Bazel 7 compatibility" in the message for commit cd22d88. Today I rediscovered: - bazelbuild/rules_java: Regression with @@rules_java//toolchains:bootstrap_runtime_toolchain_type in 7.10.0 bazelbuild/rules_java#214 But even worse, Bazel 6.5.0 fails with: ```txt $ bazel build //{src,test,third_party,scala_proto}/... [ ...snip... ] ERROR: .../test/src/main/resources/java_sources/BUILD:5:14: Building test/src/main/resources/java_sources/CompiledWithJava8_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) at com.google.devtools.build.buildjar.ReducedClasspathJavaLibraryBuilder.compileSources(ReducedClasspathJavaLibraryBuilder.java:57) at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.compileJavaLibrary(SimpleJavaLibraryBuilder.java:110) at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:118) at com.google.devtools.build.buildjar.BazelJavaBuilder.build(BazelJavaBuilder.java:111) at com.google.devtools.build.buildjar.BazelJavaBuilder.parseAndBuild(BazelJavaBuilder.java:91) at com.google.devtools.build.buildjar.BazelJavaBuilder.lambda$main$0(BazelJavaBuilder.java:52) at com.google.devtools.build.lib.worker.WorkRequestHandler$WorkRequestCallback.apply(WorkRequestHandler.java:252) at com.google.devtools.build.lib.worker.WorkRequestHandler.respondToRequest(WorkRequestHandler.java:480) at com.google.devtools.build.lib.worker.WorkRequestHandler.lambda$startResponseThread$1(WorkRequestHandler.java:433) at java.base/java.lang.Thread.run(Thread.java:829) ---8<---8<--- End of log ---8<---8<--- ERROR: .../test/src/main/resources/java_sources/BUILD:11:14: Building test/src/main/resources/java_sources/CompiledWithJava11_java.jar (1 source file) failed: Worker process did not return a WorkResponse: ---8<---8<--- Start of log, file at .../bazel-workers/multiplex-worker-18-Javac.log ---8<---8<--- Error thrown by worker thread, shutting down worker. java.lang.NoSuchMethodError: 'java.lang.Iterable com.google.devtools.build.buildjar.javac.BlazeJavacMain$ClassloaderMaskingFileManager.getJavaFileObjectsFromPaths(java.util.Collection)' at com.google.devtools.build.buildjar.javac.BlazeJavacMain.compile(BlazeJavacMain.java:142) at com.google.devtools.build.buildjar.ReducedClasspathJavaLibraryBuilder.compileSources(ReducedClasspathJavaLibraryBuilder.java:57) at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.compileJavaLibrary(SimpleJavaLibraryBuilder.java:110) at com.google.devtools.build.buildjar.SimpleJavaLibraryBuilder.run(SimpleJavaLibraryBuilder.java:118) at com.google.devtools.build.buildjar.BazelJavaBuilder.build(BazelJavaBuilder.java:111) at com.google.devtools.build.buildjar.BazelJavaBuilder.parseAndBuild(BazelJavaBuilder.java:91) at com.google.devtools.build.buildjar.BazelJavaBuilder.lambda$main$0(BazelJavaBuilder.java:52) at com.google.devtools.build.lib.worker.WorkRequestHandler$WorkRequestCallback.apply(WorkRequestHandler.java:252) at com.google.devtools.build.lib.worker.WorkRequestHandler.respondToRequest(WorkRequestHandler.java:480) at com.google.devtools.build.lib.worker.WorkRequestHandler.lambda$startResponseThread$1(WorkRequestHandler.java:433) at java.base/java.lang.Thread.run(Thread.java:829) ---8<---8<--- End of log ---8<---8<--- ``` In bazel-contrib#1619, I thought we wanted to keep `@bazel_tools//tools/jdk:` as the canonical Java toolchains package, instead of `@rules_java//toolchains:`. But based on this comment from @fmeum, I'm inclined to believe switching to `@rules_java//toolchains:` actually is the preferred, futureproof approach: - bazelbuild/rules_java#214 (comment) --- Bazel 8 requires `rules_java` 8, per bazel-contrib#1652. We can't update to `rules_java` 8 right now, unless we decided to do the following: - Tell Bazel 6 users to add C++ compiler flags to support the newer `abseil-cpp` versions required by newer `protobuf` versions. - Abandon Bazel 6 support in favor of supporting Bazel 7 at a minimum. - In addition to either of the above cases, update ScalaPB to 1.0.0-alpha.1 or higher to support `protobuf` v28. Here are the details explaining why `rules_java` 8 currently breaks. `rules_java` 8.0.0 requires Bazel >= 7.3.2, which provides the `subrule` API. Compatibility with Bazel >= 6.3.0 isn't restored until `rules_java` 8.3.2. ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: .../external/rules_java/java/common/rules/android_lint.bzl:142:24: name 'subrule' is not defined (did you mean 'rule'?) ERROR: Error computing the main repository mapping: at .../scala/private/extensions/dev_deps.bzl:8:6: at .../external/rules_java/java/repositories.bzl:20:6: at .../external/rules_java/toolchains/local_java_repository.bzl:17:6: at .../external/rules_java/java/defs.bzl:18:6: at .../external/rules_java/java/java_library.bzl:16:6: at .../external/rules_java/java/bazel/rules/bazel_java_library.bzl:21:6: compilation of module 'java/common/rules/android_lint.bzl' failed ``` `rules_java` 8.3.0 is broken, as it can't find its own `@compatibility_proxy` repo: ```txt $ bazel build //{src,test,third_party,scala_proto}/... ERROR: error loading package under directory 'src': error loading package 'src/protobuf/io/bazel/rules_scala': at .../external/rules_java/java/defs.bzl:22:6: at .../external/rules_java/java/java_test.bzl:16:6: Unable to find package for @compatibility_proxy//:proxy.bzl: The repository '@compatibility_proxy' could not be resolved: Repository '@compatibility_proxy' is not defined. ``` `rules_java` 8.3.1 seems to fix this, presumably by importing the `protobuf` repo as `com_google_protobuf`. However, it now requires at least `protobuf` v27.0, which adds `bazel/java/lite_proto_library.bzl`. Per bazel-contrib#1647, we'd have to update to ScalaPB 1.0.0-alpha.1 to support `protobuf` v28, abandoning users of previous `protobuf` versions or forcing them to upgrade. ```txt $ bazel build //{src,test,third_party,scala_proto}/... [...snip...] ERROR: error loading package under directory 'src': error loading package 'src/java/io/bazel/rulesscala/worker': at .../external/rules_java/java/defs.bzl:16:6: Label '@com_google_protobuf//bazel:java_lite_proto_library.bzl' is invalid because 'bazel' is not a package; perhaps you meant to put the colon here: '@com_google_protobuf//:bazel/java_lite_proto_library.bzl'? ```
diff --git a/WORKSPACE b/WORKSPACE
@@ -1,31 +1,41 @@
 workspace(name = "io_bazel_rules_scala")
 
 load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
+load("//scala:deps.bzl", "rules_scala_dependencies")
 
-http_archive(
-    name = "bazel_skylib",
-    sha256 = "b8a1527901774180afc798aeb28c4634bdccf19c4d98e7bdd1ce79d1fe9aaad7",
-    urls = [
-        "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.4.1/bazel-skylib-1.4.1.tar.gz",
-        "https://github.com/bazelbuild/bazel-skylib/releases/download/1.4.1/bazel-skylib-1.4.1.tar.gz",
-    ],
-)
+rules_scala_dependencies()
 
 load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace")
 
 bazel_skylib_workspace()
 
 http_archive(
     name = "rules_python",
-    sha256 = "ca77768989a7f311186a29747e3e95c936a41dffac779aff6b443db22290d913",
-    strip_prefix = "rules_python-0.36.0",
-    url = "https://github.com/bazelbuild/rules_python/releases/download/0.36.0/rules_python-0.36.0.tar.gz",
+    sha256 = "ca2671529884e3ecb5b79d6a5608c7373a82078c3553b1fa53206e6b9dddab34",
+    strip_prefix = "rules_python-0.38.0",
+    url = "https://github.com/bazelbuild/rules_python/releases/download/0.38.0/rules_python-0.38.0.tar.gz",
 )
 
 load("@rules_python//python:repositories.bzl", "py_repositories")
 
 py_repositories()
 
+load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")
+
+protobuf_deps()
+
+load("@rules_proto//proto:repositories.bzl", "rules_proto_dependencies")
+
+rules_proto_dependencies()
+
+load("@rules_proto//proto:setup.bzl", "rules_proto_setup")
+
+rules_proto_setup()
+
+load("@rules_proto//proto:toolchains.bzl", "rules_proto_toolchains")
+
+rules_proto_toolchains()
+
 load("@io_bazel_rules_scala//:scala_config.bzl", "scala_config")
 
 scala_config(enable_compiler_dependency_tracking = True)
@@ -47,22 +57,6 @@ register_toolchains(
     "@io_bazel_rules_scala_toolchains//...:all",
 )
 
-load("@rules_proto//proto:repositories.bzl", "rules_proto_dependencies")
-
-rules_proto_dependencies()
-
-load("@rules_proto//proto:setup.bzl", "rules_proto_setup")
-
-rules_proto_setup()
-
-load("@rules_proto//proto:toolchains.bzl", "rules_proto_toolchains")
-
-rules_proto_toolchains()
-
-load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")
-
-protobuf_deps()
-
 # needed for the cross repo proto test
 local_repository(
     name = "proto_cross_repo_boundary",
@@ -81,10 +75,10 @@ local_repository(
 
 http_archive(
     name = "io_bazel_rules_go",
-    sha256 = "6dc2da7ab4cf5d7bfc7c949776b1b7c733f05e56edc4bcd9022bb249d2e2a996",
+    sha256 = "f4a9314518ca6acfa16cc4ab43b0b8ce1e4ea64b81c38d8a3772883f153346b8",
     urls = [
-        "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.39.1/rules_go-v0.39.1.zip",
-        "https://github.com/bazelbuild/rules_go/releases/download/v0.39.1/rules_go-v0.39.1.zip",
+        "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.50.1/rules_go-v0.50.1.zip",
+        "https://github.com/bazelbuild/rules_go/releases/download/v0.50.1/rules_go-v0.50.1.zip",
     ],
 )
 
@@ -96,7 +90,7 @@ load(
 
 go_rules_dependencies()
 
-go_register_toolchains(version = "1.19.5")
+go_register_toolchains(version = "1.23.0")
 
 http_archive(
     name = "bazelci_rules",
diff --git a/scala/deps.bzl b/scala/deps.bzl
@@ -0,0 +1,58 @@
+"""Macro for instantiating repos required for core functionality."""
+
+load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
+load("@bazel_tools//tools/build_defs/repo:utils.bzl", "maybe")
+
+def rules_scala_dependencies():
+    """Instantiates repos needed by rules provided by `rules_scala`."""
+    maybe(
+        http_archive,
+        name = "bazel_skylib",
+        sha256 = "bc283cdfcd526a52c3201279cda4bc298652efa898b10b4db0837dc51652756f",
+        urls = [
+            "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.7.1/bazel-skylib-1.7.1.tar.gz",
+            "https://github.com/bazelbuild/bazel-skylib/releases/download/1.7.1/bazel-skylib-1.7.1.tar.gz",
+        ],
+    )
+
+    maybe(
+        http_archive,
+        name = "rules_cc",
+        urls = ["https://github.com/bazelbuild/rules_cc/releases/download/0.0.9/rules_cc-0.0.9.tar.gz"],
+        sha256 = "2037875b9a4456dce4a79d112a8ae885bbc4aad968e6587dca6e64f3a0900cdf",
+        strip_prefix = "rules_cc-0.0.9",
+    )
+
+    # Needed by protobuf-21.7 and Bazel 6.5.0, as later versions require C++14.
+    maybe(
+        http_archive,
+        name = "com_google_absl",
+        sha256 = "91ac87d30cc6d79f9ab974c51874a704de9c2647c40f6932597329a282217ba8",
+        strip_prefix = "abseil-cpp-20220623.1",
+        url = "https://github.com/abseil/abseil-cpp/archive/refs/tags/20220623.1.tar.gz",
+    )
+
+    maybe(
+        http_archive,
+        name = "rules_java",
+        urls = [
+            "https://github.com/bazelbuild/rules_java/releases/download/7.9.0/rules_java-7.9.0.tar.gz",
+        ],
+        sha256 = "41131de4417de70b9597e6ebd515168ed0ba843a325dc54a81b92d7af9a7b3ea",
+    )
+
+    maybe(
+        http_archive,
+        name = "com_google_protobuf",
+        sha256 = "75be42bd736f4df6d702a0e4e4d30de9ee40eac024c4b845d17ae4cc831fe4ae",
+        strip_prefix = "protobuf-21.7",
+        url = "https://github.com/protocolbuffers/protobuf/archive/refs/tags/v21.7.tar.gz",
+    )
+
+    maybe(
+        http_archive,
+        name = "rules_proto",
+        sha256 = "6fb6767d1bef535310547e03247f7518b03487740c11b6c6adb7952033fe1295",
+        strip_prefix = "rules_proto-6.0.2",
+        url = "https://github.com/bazelbuild/rules_proto/releases/download/6.0.2/rules_proto-6.0.2.tar.gz",
+    )
diff --git a/scala/private/macros/scala_repositories.bzl b/scala/private/macros/scala_repositories.bzl
@@ -1,3 +1,4 @@
+load("//scala:deps.bzl", "rules_scala_dependencies")
 load(
     "//scala:scala_cross_version.bzl",
     "extract_major_version",
@@ -117,59 +118,6 @@ def dt_patched_compiler_setup(scala_version, scala_compiler_srcjar = None):
             integrity = srcjar.get("integrity"),
         )
 
-def load_rules_dependencies():
-    if not native.existing_rule("bazel_skylib"):
-        http_archive(
-            name = "bazel_skylib",
-            sha256 = "b8a1527901774180afc798aeb28c4634bdccf19c4d98e7bdd1ce79d1fe9aaad7",
-            urls = [
-                "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.4.1/bazel-skylib-1.4.1.tar.gz",
-                "https://github.com/bazelbuild/bazel-skylib/releases/download/1.4.1/bazel-skylib-1.4.1.tar.gz",
-            ],
-        )
-
-    # Needed by protobuf-21.7 and Bazel 6.5.0, as later versions require C++14.
-    if not native.existing_rule("com_google_absl"):
-        http_archive(
-            name = "com_google_absl",
-            sha256 = "91ac87d30cc6d79f9ab974c51874a704de9c2647c40f6932597329a282217ba8",
-            strip_prefix = "abseil-cpp-20220623.1",
-            url = "https://github.com/abseil/abseil-cpp/archive/refs/tags/20220623.1.tar.gz",
-        )
-
-    if not native.existing_rule("com_google_protobuf"):
-        http_archive(
-            name = "com_google_protobuf",
-            sha256 = "75be42bd736f4df6d702a0e4e4d30de9ee40eac024c4b845d17ae4cc831fe4ae",
-            strip_prefix = "protobuf-21.7",
-            url = "https://github.com/protocolbuffers/protobuf/archive/refs/tags/v21.7.tar.gz",
-        )
-
-    if not native.existing_rule("rules_cc"):
-        http_archive(
-            name = "rules_cc",
-            urls = ["https://github.com/bazelbuild/rules_cc/releases/download/0.0.6/rules_cc-0.0.6.tar.gz"],
-            sha256 = "3d9e271e2876ba42e114c9b9bc51454e379cbf0ec9ef9d40e2ae4cec61a31b40",
-            strip_prefix = "rules_cc-0.0.6",
-        )
-
-    if not native.existing_rule("rules_java"):
-        http_archive(
-            name = "rules_java",
-            urls = [
-                "https://github.com/bazelbuild/rules_java/releases/download/7.9.0/rules_java-7.9.0.tar.gz",
-            ],
-            sha256 = "41131de4417de70b9597e6ebd515168ed0ba843a325dc54a81b92d7af9a7b3ea",
-        )
-
-    if not native.existing_rule("rules_proto"):
-        http_archive(
-            name = "rules_proto",
-            sha256 = "6fb6767d1bef535310547e03247f7518b03487740c11b6c6adb7952033fe1295",
-            strip_prefix = "rules_proto-6.0.2",
-            url = "https://github.com/bazelbuild/rules_proto/releases/download/6.0.2/rules_proto-6.0.2.tar.gz",
-        )
-
 def setup_scala_compiler_sources(srcjars = {}):
     """Generates Scala compiler source repos used internally by rules_scala.
 
@@ -188,7 +136,7 @@ def setup_scala_compiler_sources(srcjars = {}):
     )
 
 def rules_scala_setup(scala_compiler_srcjar = None):
-    load_rules_dependencies()
+    rules_scala_dependencies()
     setup_scala_compiler_sources({
         version: scala_compiler_srcjar
         for version in SCALA_VERSIONS
@@ -260,7 +208,7 @@ def scala_repositories(
         scala_compiler_srcjars = {}):
     if load_dep_rules:
         # When `WORKSPACE` goes away, so can this case.
-        load_rules_dependencies()
+        rules_scala_dependencies()
 
     setup_scala_compiler_sources(scala_compiler_srcjars)
 
