Skip to content

Commit 7b7ab2a

Browse files
mblandmbland
committed
Add precompiled protoc toolchain
Registers a precompiled protocol compiler toolchain when `--incompatible_enable_proto_toolchain_resolution` is `True`. Part of bazel-contrib#1482 and bazel-contrib#1652. Stops `protoc` recompilation, and fixes the build breakage in bazel-contrib#1710 due to `protobuf` include paths exceeding the Visual Studio path length limit. The updates to `scala_proto/scala_proto_toolchain.bzl` were inspired by: - protocolbuffers/protobuf: bazel: Remove hardcoded dependency on //:protoc from language runtimes #19679 protocolbuffers/protobuf#19679 The `proto_lang_toolchain` call was inspired by the `README` from: - https://github.com/aspect-build/toolchains_protoc/ Adds `scripts/update_protoc_integrity.py` to automatically update `scala/private/protoc/protoc_integrity.bzl`. This should make builds of `rules_scala` much faster all around. Given the fact that this feature depends on recent `protobuf` versions, and the Windows `protobuf` build breaks without it, we have a catch-22. It likely can't be separated from the rest of bazel-contrib#1710, though I would prefer that. It also seems likely that we'd eventually need to do this to continue supporting Windows, per: - protocolbuffers/protobuf#12947 - https://protobuf.dev/news/v30/#poison-msvc--bazel - protocolbuffers/protobuf#20085 More background on proto toolchainization: - Proto Toolchainisation Design Doc https://docs.google.com/document/d/1CE6wJHNfKbUPBr7-mmk_0Yo3a4TaqcTPE0OWNuQkhPs/edit - bazelbuild/bazel: Protobuf repo recompilation sensitivity bazelbuild/bazel#7095 - bazelbuild/rules_proto: Implement proto toolchainisation bazelbuild/rules_proto#179 - rules_proto 6.0.0 release notes mentioning Protobuf Toolchainisation https://github.com/bazelbuild/rules_proto/releases/tag/6.0.0
1 parent b55ab59 commit 7b7ab2a

File tree

23 files changed

+595
-9
lines changed

23 files changed

+595
-9
lines changed

‎.bazelrc

+5
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,11 @@
22
# https://github.com/bazelbuild/rules_scala/issues/1482
33
common --enable_workspace --noenable_bzlmod
44

5+
# Remove once proto toolchainization becomes the default
6+
# - https://bazel.build/reference/command-line-reference#flag--incompatible_enable_proto_toolchain_resolution
7+
# - https://docs.google.com/document/d/1CE6wJHNfKbUPBr7-mmk_0Yo3a4TaqcTPE0OWNuQkhPs/edit
8+
common --incompatible_enable_proto_toolchain_resolution
9+
510
build --enable_platform_specific_config
611

712
#Windows needs --worker_quit_after_build due to workers not being shut down when the compiler tools need to be rebuilt (resulting in 'file in use' errors). See Bazel Issue#10498.

‎README.md

+60
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,13 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
6262

6363
rules_scala_dependencies()
6464

65+
# The next two calls instantiate the `@host_platform` repo to work around:
66+
# - https://github.com/bazelbuild/bazel/issues/22558
67+
# Only required if using `--incompatible_enable_proto_toolchain_resolution`.
68+
load("@platforms//host:extension.bzl", "host_platform_repo")
69+
70+
host_platform_repo(name = "host_platform")
71+
6572
load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
6673

6774
rules_java_dependencies()
@@ -168,6 +175,40 @@ load(
168175
)
169176
```
170177

178+
### <a id="protoc"></a>Using a precompiled protocol compiler
179+
180+
`rules_scala` now supports
181+
[`--incompatible_enable_proto_toolchain_resolution`][]. When using this flag,
182+
`rules_scala` will use a precompiled protocol compiler binary by default. To set
183+
it in your `.bazelrc` file:
184+
185+
[`--incompatible_enable_proto_toolchain_resolution`]: https://bazel.build/reference/command-line-reference#flag--incompatible_enable_proto_toolchain_resolution
186+
187+
```txt
188+
common --incompatible_enable_proto_toolchain_resolution
189+
```
190+
191+
Set the `protoc_platforms` attribute of `scala_toolchains()` if you need to
192+
configure protocol compilers for platforms other than the host platform.
193+
194+
Windows builds now require the precompiled protocol compiler toolchain. See the
195+
[Windows MSVC builds of protobuf broken by default](#protoc-msvc) section
196+
below for details.
197+
198+
More background on proto toolchainization:
199+
200+
- [Proto Toolchainisation Design Doc](
201+
https://docs.google.com/document/d/1CE6wJHNfKbUPBr7-mmk_0Yo3a4TaqcTPE0OWNuQkhPs/edit)
202+
203+
- [bazelbuild/bazel#7095: Protobuf repo recompilation sensitivity](
204+
https://github.com/bazelbuild/bazel/issues/7095)
205+
206+
- [bazelbuild/rules_proto#179: Implement proto toolchainisation](
207+
https://github.com/bazelbuild/rules_proto/issues/179)
208+
209+
- [rules_proto 6.0.0 release notes mentioning Protobuf Toolchainization](
210+
https://github.com/bazelbuild/rules_proto/releases/tag/6.0.0)
211+
171212
### Persistent workers
172213

173214
To run with a persistent worker (much faster), add the following to
@@ -567,6 +608,25 @@ http_archive(
567608
)
568609
```
569610

611+
### <a id="protoc-msvc"></a>Windows MSVC builds of `protobuf` broken by default
612+
613+
MSVC builds of recent `protobuf` versions started failing, as first noted in
614+
bazelbuild/rules_scala#1710. On top of that, `protobuf` is planning to stop
615+
supporting Bazel + MSVC builds per:
616+
617+
- [protocolbuffers/protobuf#12947: src build on windows not working](
618+
https://github.com/protocolbuffers/protobuf/issues/12947)
619+
620+
- [protobuf.dev News Announcements for Version 30.x:Poison MSVC + Bazel](
621+
https://protobuf.dev/news/v30/#poison-msvc--bazel)
622+
623+
- [protocolbuffers/protobuf#20085: Breaking Change: Dropping support for
624+
Bazel+MSVC](https://github.com/protocolbuffers/protobuf/issues/20085)
625+
626+
Enable [protocol compiler toolchainization](#protoc) to avoid recompiling
627+
`@com_google_protobuf//:protoc` on Windows platforms and fix broken Windows
628+
builds.
629+
570630
### Bzlmod configuration (coming soon!)
571631

572632
The upcoming Bzlmod implementation will funnel through the `scala_toolchains()`

‎WORKSPACE

+7
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,13 @@ load("//scala:deps.bzl", "rules_scala_dependencies")
55

66
rules_scala_dependencies()
77

8+
# The next two calls instantiate the `@host_platform` repo to work around:
9+
# - https://github.com/bazelbuild/bazel/issues/22558
10+
# Only required if using `--incompatible_enable_proto_toolchain_resolution`.
11+
load("@platforms//host:extension.bzl", "host_platform_repo")
12+
13+
host_platform_repo(name = "host_platform")
14+
815
load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
916

1017
rules_java_dependencies()

‎dt_patches/test_dt_patches/WORKSPACE

+4
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,10 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
1111

1212
rules_scala_dependencies()
1313

14+
load("@platforms//host:extension.bzl", "host_platform_repo")
15+
16+
host_platform_repo(name = "host_platform")
17+
1418
load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
1519

1620
rules_java_dependencies()

‎dt_patches/test_dt_patches_user_srcjar/WORKSPACE

+4
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,10 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
1111

1212
rules_scala_dependencies()
1313

14+
load("@platforms//host:extension.bzl", "host_platform_repo")
15+
16+
host_platform_repo(name = "host_platform")
17+
1418
load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
1519

1620
rules_java_dependencies()

‎examples/crossbuild/WORKSPACE

+4
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,10 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
1111

1212
rules_scala_dependencies()
1313

14+
load("@platforms//host:extension.bzl", "host_platform_repo")
15+
16+
host_platform_repo(name = "host_platform")
17+
1418
load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
1519

1620
rules_java_dependencies()

‎examples/overridden_artifacts/WORKSPACE

+4
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,10 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
1111

1212
rules_scala_dependencies()
1313

14+
load("@platforms//host:extension.bzl", "host_platform_repo")
15+
16+
host_platform_repo(name = "host_platform")
17+
1418
load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
1519

1620
rules_java_dependencies()

‎examples/scala3/WORKSPACE

+4
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,10 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
1111

1212
rules_scala_dependencies()
1313

14+
load("@platforms//host:extension.bzl", "host_platform_repo")
15+
16+
host_platform_repo(name = "host_platform")
17+
1418
load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
1519

1620
rules_java_dependencies()

‎examples/semanticdb/WORKSPACE

+4
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,10 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
1111

1212
rules_scala_dependencies()
1313

14+
load("@platforms//host:extension.bzl", "host_platform_repo")
15+
16+
host_platform_repo(name = "host_platform")
17+
1418
load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
1519

1620
rules_java_dependencies()

‎examples/testing/multi_frameworks_toolchain/WORKSPACE

+4
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,10 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
1111

1212
rules_scala_dependencies()
1313

14+
load("@platforms//host:extension.bzl", "host_platform_repo")
15+
16+
host_platform_repo(name = "host_platform")
17+
1418
load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
1519

1620
rules_java_dependencies()

‎examples/testing/scalatest_repositories/WORKSPACE

+4
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,10 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
1111

1212
rules_scala_dependencies()
1313

14+
load("@platforms//host:extension.bzl", "host_platform_repo")
15+
16+
host_platform_repo(name = "host_platform")
17+
1418
load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
1519

1620
rules_java_dependencies()

‎examples/testing/specs2_junit_repositories/WORKSPACE

+4
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,10 @@ load("@rules_scala//scala:deps.bzl", "rules_scala_dependencies")
1111

1212
rules_scala_dependencies()
1313

14+
load("@platforms//host:extension.bzl", "host_platform_repo")
15+
16+
host_platform_repo(name = "host_platform")
17+
1418
load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
1519

1620
rules_java_dependencies()

‎scala/deps.bzl

+10
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,16 @@ def rules_scala_dependencies():
1515
],
1616
)
1717

18+
maybe(
19+
http_archive,
20+
name = "platforms",
21+
urls = [
22+
"https://mirror.bazel.build/github.com/bazelbuild/platforms/releases/download/0.0.11/platforms-0.0.11.tar.gz",
23+
"https://github.com/bazelbuild/platforms/releases/download/0.0.11/platforms-0.0.11.tar.gz",
24+
],
25+
sha256 = "29742e87275809b5e598dc2f04d86960cc7a55b3067d97221c9abbc9926bff0f",
26+
)
27+
1828
maybe(
1929
http_archive,
2030
name = "rules_cc",

‎scala/private/protoc/protoc_integrity.bzl

+148
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
"""Protocol compiler build and integrity metadata.
2+
3+
Generated and updated by scripts/update_protoc_integrity.py.
4+
"""
5+
6+
PROTOC_RELEASES_URL = "https://github.com/protocolbuffers/protobuf/releases"
7+
PROTOC_DOWNLOAD_URL = (
8+
PROTOC_RELEASES_URL +
9+
"/download/v{version}/protoc-{version}-{platform}.zip"
10+
)
11+
12+
PROTOC_VERSIONS = [
13+
"29.3",
14+
"29.2",
15+
"29.1",
16+
"29.0",
17+
"28.3",
18+
"28.2",
19+
]
20+
21+
PROTOC_BUILDS = {
22+
"linux-aarch_64": {
23+
"exec_compat": [
24+
"@platforms//os:linux",
25+
"@platforms//cpu:aarch64",
26+
],
27+
"integrity": {
28+
"29.3": "sha256-ZCc0kUDgHwbgSecHpYcJpPIhrnOrmgQlvEoAyNDhqzI=",
29+
"29.2": "sha256-Kc9IPi+yGCfl+sSWTjXq5HKiOOKMdi8C+xfc2T/4uJ8=",
30+
"29.1": "sha256-H3Sj8zVd58Bma8ElYRwTUywlmPhTUh0NPmIaWwnyR5k=",
31+
"29.0": "sha256-MF8b5a57LzlFGHCzErRcHguiaZAcg7oW2F+fnRRBs0g=",
32+
"28.3": "sha256-HeUiAyqLGUAC/jXKuG10eEgji15N5PmWSDcgefW0b5o=",
33+
"28.2": "sha256-kdglPNwPDw/FHCtpyAZ3mWYy9SWthFBL+ltO44rT5Jw=",
34+
},
35+
},
36+
"linux-ppcle_64": {
37+
"exec_compat": [
38+
"@platforms//os:linux",
39+
"@platforms//cpu:ppc64le",
40+
],
41+
"integrity": {
42+
"29.3": "sha256-DpiU7C45krFNGD586sFkZdam7nPh0jRpXYDm0elHAUw=",
43+
"29.2": "sha256-uiCJWht/NKb/ql5Rw0ExbErrxMFEN3hjQITpn262f/k=",
44+
"29.1": "sha256-B1vWZq1B60BKkjv6+pCkr6IHSiyaqLnHfERF5hbo75s=",
45+
"29.0": "sha256-EJAnjNB1e3AsNrA+6t9KvTYCAnm7B7DfX4C9iW8+IDM=",
46+
"28.3": "sha256-dSKdPN5z5wYZcXgU9R+m9K16NiwzUe5ZGXuxV8sAgsY=",
47+
"28.2": "sha256-xcFrR2f/iGYJDuEOhwkDABBSAbXbHs7zgO3XPCM03Ng=",
48+
},
49+
},
50+
"linux-s390_64": {
51+
"exec_compat": [
52+
"@platforms//os:linux",
53+
"@platforms//cpu:s390x",
54+
],
55+
"integrity": {
56+
"29.3": "sha256-Y3hX/bqwsTNL2ysIcz8L5JaF5pMBG2EEgJSRrGL71NU=",
57+
"29.2": "sha256-LwpVmdprgpMqCNQz+nkTux1oWIl8zTxrwyhvAYt2xOA=",
58+
"29.1": "sha256-J5fNVlyn/7/ALaVsygE7/0+X5oMkizuZ3PfUuxf27GE=",
59+
"29.0": "sha256-LhXZqwaFbCXKbeYi4RR4XZGHHb25HCwQPrYouH8m3Ew=",
60+
"28.3": "sha256-jhtvqCX7CVlqiS5d6bgRLXoJtL3ftxzLv1+GgKT1aKc=",
61+
"28.2": "sha256-ESIsQ4+G6Hsv2vaKKmzB2ytiKBP9btiRUJ4vOw4F7hs=",
62+
},
63+
},
64+
"linux-x86_32": {
65+
"exec_compat": [
66+
"@platforms//os:linux",
67+
"@platforms//cpu:x86_32",
68+
],
69+
"integrity": {
70+
"29.3": "sha256-VGzx5pHOc/ZuKZzCLN1aF8YaEf5LbV34UHUHg4nx/Ck=",
71+
"29.2": "sha256-FU+NR+6YO8bxoa4tsUl+53E8Qt7vY1EXDxhmG/vNouw=",
72+
"29.1": "sha256-nd/EAbEqC4dHHg4POg0MKpByRj8EFDUUnpSfiiCCu+s=",
73+
"29.0": "sha256-tKyBCfKrSGLV5WuH4/cVMPug46YeyLnZQOKUdskGAQE=",
74+
"28.3": "sha256-DJ6zLLnl06rHLGfPc38In+Mr8kVssBc+EpWQfNGIXhI=",
75+
"28.2": "sha256-ucjToo5Lq5WcwQ5smjjxfFlGc3Npv+AT9RqG19Ns//E=",
76+
},
77+
},
78+
"linux-x86_64": {
79+
"exec_compat": [
80+
"@platforms//os:linux",
81+
"@platforms//cpu:x86_64",
82+
],
83+
"integrity": {
84+
"29.3": "sha256-PoZmIMW+J2ZPPS+i1la18+CbUVK0Lxvtv0J7Mz6QAho=",
85+
"29.2": "sha256-Uunn7OVcfjDn6LvSVLSyG0CKUwm8qCZ2PHEktpahMuk=",
86+
"29.1": "sha256-AMg/6XIthelsgblBsp8Xp0SzO0zmbg8YAJ/Yk33iLGA=",
87+
"29.0": "sha256-PFEGWvO5pgbZ4Yob9igUNzT/S55pcl1kWYV0MLp6eN8=",
88+
"28.3": "sha256-CtlJ8EpqF02oPNy9s23uCkklJypbbYP3mmv5hSB21T8=",
89+
"28.2": "sha256-L+v9QrWc6Too63iQGaRwo90ESWGbwE+E2tEzPaJh3sE=",
90+
},
91+
},
92+
"osx-aarch_64": {
93+
"exec_compat": [
94+
"@platforms//os:osx",
95+
"@platforms//cpu:aarch64",
96+
],
97+
"integrity": {
98+
"29.3": "sha256-K4o0A80Jf5XzumVuFLdscytrJtfxgzMLEeNu8rwCh2U=",
99+
"29.2": "sha256-DhU6ONbaGVlMmA5/fNPqDd1SydoQaMA8DYUzNp+/6yA=",
100+
"29.1": "sha256-uP1ZdpJhmKfE6lxutL94lZ1frtJ7/GGCVMqhBD93BEU=",
101+
"29.0": "sha256-srWfA7AwyKdIYj1oKotbycwJnkvP0GuJZM6J7AZbMQM=",
102+
"28.3": "sha256-ks7v2mpyk+wBTm7KyC1kcZNXFFy2/ChlutreteYsBDE=",
103+
"28.2": "sha256-e7BI9ShBeJ2exhmDvgzkyeT7O9mhQ0YoILqaO+CgN5c=",
104+
},
105+
},
106+
"osx-x86_64": {
107+
"exec_compat": [
108+
"@platforms//os:osx",
109+
"@platforms//cpu:x86_64",
110+
],
111+
"integrity": {
112+
"29.3": "sha256-mniANtj5hU97A8MF30d3zw5U5bCB4lvxUlLah+DpCHU=",
113+
"29.2": "sha256-uivZg7XwbsONZjtgKISll96jmQpDgD1+FT7Y98VCaeE=",
114+
"29.1": "sha256-2wK0uG3k1MztPqmTQ0faKNyV5/OIY//EzjzCYoMCjaY=",
115+
"29.0": "sha256-56HP/ILiHapngzARRJxw3f8eujsRWTQ4fm6BQe+rCS8=",
116+
"28.3": "sha256-l/5dRCCQtNvCPNE4T7m0RPodxuZ9FbteH+TeDadjiyA=",
117+
"28.2": "sha256-Iy8H0Sv0gGIHp57CxzeDAcUuby9+/dIcDdQW8L2hA+w=",
118+
},
119+
},
120+
"win32": {
121+
"exec_compat": [
122+
"@platforms//os:windows",
123+
"@platforms//cpu:x86_32",
124+
],
125+
"integrity": {
126+
"29.3": "sha256-x8gCjBxNgBxTYCkg8shokgVAhr2WW2sjpLqV0hHcsdQ=",
127+
"29.2": "sha256-73CfcaUbOompsm3meCBc7zyV4h0IGLFn+/WwOackr9E=",
128+
"29.1": "sha256-EQXg+mRFnwsa9e5NWHfauGTy4Q2K6wRhjytpxsOm7QM=",
129+
"29.0": "sha256-154nzOTEAXRUERc8XraBFsPiBACxzEwt0stHeneUGP4=",
130+
"28.3": "sha256-sI/m/M9DE+LMxv1ybchV7okFM+JH9MVDyYf/2YDP1bI=",
131+
"28.2": "sha256-V6hpbqvtUgl19PpGkUGwuC+iEIYj0YtyOqi+yae0u1g=",
132+
},
133+
},
134+
"win64": {
135+
"exec_compat": [
136+
"@platforms//os:windows",
137+
"@platforms//cpu:x86_64",
138+
],
139+
"integrity": {
140+
"29.3": "sha256-V+pZ6fVRrY1x/6qbXPvgyh9Ocglyodt+wtEqtEv/k4M=",
141+
"29.2": "sha256-Weph77JLnYohQXHiyj/sVcPxUX7/BnZWyHXYoc0Gzk8=",
142+
"29.1": "sha256-fqSCJYV//BIkWIwzXCsa+deKGK+dV8BSjMoxk+M26c4=",
143+
"29.0": "sha256-0DuSGYWLikyogGO3i/Clzec7UYCLkwxLZvBuhILDq+Y=",
144+
"28.3": "sha256-zmT0m97d70nOS9MTqPWbz5L89ntYMe+/ZhcDhtLmaUg=",
145+
"28.2": "sha256-S94ZJx7XyrkANXDyjG5MTXGWPq8SEahr87sl2biVF3o=",
146+
},
147+
},
148+
}

0 commit comments

Comments
 (0)