Skip to content

Commit d2ad2ea

Browse files
authored
Buck2 golang example (#355)
- Add an example of RE with Buck2 for a simple golang project. - See readme for more details.
1 parent 79b143f commit d2ad2ea

File tree

14 files changed

+419
-1
lines changed

14 files changed

+419
-1
lines changed

buck2/golang/.buckconfig

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
[cells]
2+
root = .
3+
prelude = prelude
4+
toolchains = toolchains
5+
none = none
6+
7+
[cell_aliases]
8+
config = prelude
9+
fbcode = none
10+
fbsource = none
11+
buck = none
12+
13+
[external_cells]
14+
prelude = bundled
15+
16+
[parser]
17+
target_platform_detector_spec = target:root//...->prelude//platforms:default
18+
19+
[buck2]
20+
digest_algorithms = SHA256
21+
22+
[buck2_re_client]
23+
engine_address = <CLUSTER_NAME>.cluster.engflow.com
24+
action_cache_address = <CLUSTER_NAME>.cluster.engflow.com
25+
cas_address = <CLUSTER_NAME>.cluster.engflow.com
26+
http_headers = x-engflow-auth-method:jwt-v0,x-engflow-auth-token:LONG_JWT_STRING
27+
28+
[build]
29+
execution_platforms = root//platforms:remote_platform
30+
31+
[project]
32+
ignore = .git

buck2/golang/.buckroot

Whitespace-only changes.

buck2/golang/.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
/buck-out

buck2/golang/README.md

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# EngFlow RE + Buck2 Go example
2+
3+
This example demonstrates use of EngFlow RE for a simple Go lang project built with [Buck2](https://github.com/facebook/buck2) using the prelude.
4+
5+
It is based on three existing samples in the Buck2 upstream repo and the EngFlow samples repo:
6+
7+
* Simple go project with prelude - https://github.com/facebook/buck2/tree/main/examples/with_prelude/go
8+
* Buck2 remote execution integration with EngFlow - https://github.com/facebook/buck2/tree/main/examples/remote_execution/engflow
9+
* EngFlow go example - https://github.com/EngFlow/example/tree/main/go
10+
11+
### Example structure
12+
13+
In the `platforms` cell we specify:
14+
* The platform used for remote execution in this project `root//platforms:remote_platform`, which includes the definition of the Docker image used for remote execution, and that defines constraints for targets to run in the remote execution environment. This platform provides an `ExecutionPlatformRegistrationInfo`.
15+
* The action keys `root//platforms:remote_execution_action_keys`, which provides a default `BuildModeInfo` that is needed for RE of tests to function properly.
16+
* The platform `image` configured in `platforms/defs.bzl`, notably, uses a different image than other Buck2 samples in this repo. Specifically, it uses a public AWS image that has `go` preinstalled. This is because, unlike Bazel go rules, Buck2 go rules do not include a hermetic go binary.
17+
18+
In the `toolchains` cell we specify:
19+
20+
* The remote go toolchain `root//toolchains:remote_go_toolchain` and remote go bootstrap toolchain `root//toolchains:remote_go_bootstrap_toolchain`, which are compatible with the remote execution environment. These toolchains are configured for `go_arch` set to `amd64` and `go_os` set to `linux`.
21+
* The remote test execution toolchain, `root//toolchains:remote_test_execution_toolchain`. This toolchain defines platform options in the form of `capabilities`. Critically these include the `container-image`. This toolchain is identical to the one in the `buck2/cpp` sample in this repo.
22+
23+
The `main` cell and `library` cell:
24+
25+
* Contains an copied version of https://github.com/facebook/buck2/tree/main/examples/with_prelude/go that works with Buck2 and RE as configured in this sample project.
26+
27+
To test this cell with RE run (after setting up `.buckconfig` as indicated below):
28+
29+
```
30+
buck2 test //go/greeting:greeting_test
31+
```
32+
33+
You can also build the `main` for this sample by running:
34+
35+
```
36+
buck2 build //go:hello
37+
```
38+
39+
### Relevant configs in `.buckconfig`
40+
41+
The EngFlow endpoint and certificate should be configured as the
42+
following:
43+
44+
```ini
45+
[buck2_re_client]
46+
engine_address = <CLUSTER_NAME>.cluster.engflow.com
47+
action_cache_address = <CLUSTER_NAME>.cluster.engflow.com
48+
cas_address = <CLUSTER_NAME>.cluster.engflow.com
49+
http_headers = x-engflow-auth-method:jwt-v0,x-engflow-auth-token:LONG_JWT_STRING
50+
```
51+
52+
To obtain the value of `LONG_JWT_STRING`, log into https://<CLUSTER_NAME>.cluster.engflow.com/gettingstarted and use the value of `x-engflow-auth-token` in section `Method 2: JWT`.

buck2/golang/go/BUCK

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Copyright 2022 EngFlow Inc. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
go_binary(
16+
name = "hello",
17+
srcs = ["main.go"],
18+
deps = [
19+
"//go/greeting:greeting",
20+
],
21+
)

buck2/golang/go/greeting/BUCK

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Copyright 2022 EngFlow Inc. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
go_library(
16+
name = "greeting",
17+
srcs = glob(["*.go"]),
18+
visibility = ["PUBLIC"],
19+
)
20+
21+
go_test(
22+
name = "greeting_test",
23+
srcs = glob(["*.go"]),
24+
remote_execution_action_key_providers = "//platforms:remote_execution_action_keys",
25+
)

buck2/golang/go/greeting/greeting.go

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
2+
package greeting
3+
4+
// Greeting returns a greeting message
5+
func Greeting() string {
6+
return "Hello, world!"
7+
}
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
package greeting
2+
3+
import "testing"
4+
5+
func TestHello(t *testing.T) {
6+
if Greeting() != "Hello, world!" {
7+
t.Errorf("Greeting() = %v, want \"Hello, world!\"", Greeting())
8+
}
9+
}

buck2/golang/go/main.go

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
package main
2+
3+
import (
4+
"fmt"
5+
6+
"go/greeting"
7+
)
8+
9+
func main() {
10+
fmt.Println(greeting.Greeting())
11+
}

buck2/golang/platforms/BUCK

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Copyright 2022 EngFlow Inc. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
load(":defs.bzl", "platforms")
16+
load(":defs.bzl", "action_keys")
17+
18+
# This platform configures details of remote execution.
19+
platforms(
20+
name = "remote_platform",
21+
)
22+
23+
# This action_key provides a default BuildModeInfo that is needed for RE of tests to function properly.
24+
# The values in `cell` and `mode` can be used, in practice, to create cache silos. Any values can be given to these attributes.
25+
action_keys(
26+
name = "remote_execution_action_keys",
27+
cell = "standard",
28+
mode = "standard",
29+
visibility = ["PUBLIC"],
30+
)

buck2/golang/platforms/defs.bzl

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# Copyright 2022 EngFlow Inc. All rights reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
# This platform is essentially the same as the one provided in https://github.com/facebook/buck2/blob/804d62242214455d51787f7c8c96a1e12c75ec32/examples/remote_execution/engflow/platforms/defs.bzl
16+
# The main difference is we enable passing CPU and OS constraints and we use the sample EngFlow RE image.
17+
load("@prelude//:build_mode.bzl", "BuildModeInfo")
18+
19+
def _platforms(ctx):
20+
constraints = dict()
21+
configuration = ConfigurationInfo(
22+
constraints = constraints,
23+
values = {},
24+
)
25+
26+
# A bookworm image with go pre-installed.
27+
# Unlike Bazel go_toolchain, Buck2 go_toolchain does not include a hermetic go binary.
28+
image = "docker://public.ecr.aws/docker/library/golang:1.23.3-bookworm@sha256:3f3b9daa3de608f3e869cd2ff8baf21555cf0fca9fd34251b8f340f9b7c30ec5"
29+
name = ctx.label.raw_target()
30+
platform = ExecutionPlatformInfo(
31+
label = ctx.label.raw_target(),
32+
configuration = configuration,
33+
executor_config = CommandExecutorConfig(
34+
local_enabled = False,
35+
remote_enabled = True,
36+
use_limited_hybrid = False,
37+
remote_execution_properties = {
38+
"container-image": image,
39+
},
40+
remote_execution_use_case = "buck2-default",
41+
# TODO: Use output_paths
42+
remote_output_paths = "strict",
43+
),
44+
)
45+
46+
return [
47+
DefaultInfo(),
48+
ExecutionPlatformRegistrationInfo(platforms = [platform]),
49+
]
50+
51+
def _action_keys(ctx):
52+
return [
53+
DefaultInfo(),
54+
BuildModeInfo(cell = ctx.attrs.cell, mode = ctx.attrs.mode),
55+
]
56+
57+
platforms = rule(
58+
attrs = {},
59+
impl = _platforms
60+
)
61+
62+
action_keys = rule(
63+
attrs = {
64+
"cell": attrs.string(),
65+
"mode": attrs.string(),
66+
},
67+
impl = _action_keys
68+
)

buck2/golang/toolchains/BUCK

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
2+
# Copyright 2022 EngFlow Inc. All rights reserved.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
load("defs.bzl", "remote_go_toolchain", "remote_go_bootstrap_toolchain")
17+
load("@prelude//toolchains:cxx.bzl", "cxx_tools_info_toolchain")
18+
load("@prelude//toolchains:python.bzl", "system_python_bootstrap_toolchain")
19+
load("@prelude//toolchains:remote_test_execution.bzl", "remote_test_execution_toolchain")
20+
load("@prelude//tests:test_toolchain.bzl", "noop_test_toolchain")
21+
22+
# Python toolchain used in scripts that bootstrap other aspects of the Buck2 prelude.
23+
system_python_bootstrap_toolchain(
24+
name = "python_bootstrap",
25+
visibility = ["PUBLIC"],
26+
)
27+
28+
remote_go_bootstrap_toolchain(
29+
name = "go_bootstrap",
30+
visibility = ["PUBLIC"],
31+
)
32+
33+
# c++ toolchain required implicitly for py targets.
34+
cxx_tools_info_toolchain(
35+
name = "cxx",
36+
visibility = ["PUBLIC"],
37+
)
38+
39+
remote_go_toolchain(
40+
name = "go",
41+
visibility = ["PUBLIC"],
42+
)
43+
44+
# Default toolchain for remote execution of tests.
45+
# Note it defines a profile with a capability that defines the `container-image` that matches the one defined in //platforms:remote_platform.
46+
# Capabilities are passed to the RE service to find workers that match them as Platform options.
47+
remote_test_execution_toolchain(
48+
name = "remote_test_execution",
49+
visibility = ["PUBLIC"],
50+
default_profile = "cxx_re_toolchain",
51+
profiles = {
52+
"cxx_re_toolchain": {
53+
"use_case": "cxx-testing",
54+
"capabilities": {
55+
"container-image" : "docker://gcr.io/bazel-public/ubuntu2004-java11@sha256:69a78f121230c6d5cbfe2f4af8ce65481aa3f2acaaaf8e899df335f1ac1b35b5",
56+
},
57+
}
58+
},
59+
)
60+
61+
# In some cases the execution of test can fail looking for this `noop_test_toolchain`.
62+
noop_test_toolchain(
63+
name = "test",
64+
visibility = ["PUBLIC"],
65+
)

0 commit comments

Comments
 (0)