Enable running sccache-dist in Docker. #1922
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Would it be possible to add a new test job with docker to verify that it works ? |
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #1922 +/- ##
==========================================
- Coverage 30.89% 30.46% -0.43%
==========================================
Files 53 51 -2
Lines 20033 19170 -863
Branches 9717 9188 -529
==========================================
- Hits 6189 5841 -348
+ Misses 7947 7740 -207
+ Partials 5897 5589 -308 ☔ View full report in Codecov by Sentry. |
|
@fslongjin ping ? |
|
pong.
Sorry I'm busy these days. I'll push to my repo and paste link here tomorrow~
…---Original---
From: "Sylvestre ***@***.***>
Date: Thu, Nov 2, 2023 18:24 PM
To: ***@***.***>;
Cc: ***@***.******@***.***>;
Subject: Re: [mozilla/sccache] Enable running sccache-dist in Docker. (PR#1922)
@fslongjin ping ?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
Hello~I push the example scripts to my repo: |
|
ping |
|
Sorry if I wasn't clear but I would like to see your change tested in sccache CI :) Ie a new job/test verifying that it works and we don't regress in the future. |
Maybe we cannot run docker in github workflow beacuse the gh workflow runs in docker without |
|
ping |
sylvestre
force-pushed
the
patch-add-dist-bind-addr
branch
from
fbc6176 to
ba4e816
Compare
|
ping |
|
Hi @sylvestre, this PR looks good to me. Do you have other comments? |
|
Yeah, could you please write some more docs ? |
|
ping doc ? |
1. Add 'bind_addr' param to 'sccache-dist server'. 2. Set `X-real-ip` header when 'bind_addr' is set.
sylvestre
force-pushed
the
patch-add-dist-bind-addr
branch
from
ba4e816 to
7da1a75
Compare
|
please reopen when the doc is ready |
|
@fslongjin ping ? |
|
This PR seems necessary to deploy sccache-dist build clusters with k8s. If all that's required is an example of using sccache-dist in docker, here's one way: mkdir /tmp/sccache-secrets
docker build --target scheduler -f Dockerfile -t sccache-dist:scheduler .
docker run --rm --init -d -v /tmp/sccache-secrets:/run/secrets --name scheduler -e PUBLIC_ADDR=scheduler:10600 sccache-dist:scheduler
docker build --target worker -f Dockerfile -t sccache-dist:worker .
docker run --rm --init -d -v /tmp/sccache-secrets:/run/secrets --name worker -e PUBLIC_ADDR=worker:10501 -e SCHEDULER_URL=scheduler:10600 --privileged sccache-dist:worker
# Dockerfile
FROM debian:stable-slim AS dist
RUN <<EOF_RUN
export DEBIAN_FRONTEND=noninteractive;
apt update
apt install -y --no-install-recommends wget libssl-dev ca-certificates
apt install -y bubblewrap libcap-dev
apt clean
rm -rf /var/lib/apt/lists/* /var/tmp/* /tmp/*
update-ca-certificates
EOF_RUN
COPY --chown=root:root target/x86_64-unknown-linux-musl/release/sccache-dist /usr/bin/sccache-dist
WORKDIR /sccache
ENV SCCACHE_LOG=debug
ENV SCCACHE_NO_DAEMON=1
ENV SCCACHE_DIST_SCHEDULER_KEY_FILE=/run/secrets/sccache_dist_scheduler_key
FROM dist as worker
ENV BIND_ADDR=0.0.0.0:10501
ENV PUBLIC_ADDR="$BIND_ADDR"
ENV SCHEDULER=http://127.0.0.1:10600
ENV SCCACHE_BUILDS=/sccache/builds
ENV SCCACHE_TOOLCHAINS=/sccache/toolchains
RUN <<EOF_RUN
cat <<"EOF_ENTRY" > /sccache/entrypoint.sh
#! /usr/bin/env bash
if ! test -s "$SCCACHE_DIST_SCHEDULER_KEY_FILE"; then
cat <<EOF_ERR >&2
Err: Missing scheduler secret key.
The dist-scheduler entrypoint can create one automatically that you can share between the scheduler and workers:
$ touch "$(basename "$SCCACHE_DIST_SCHEDULER_KEY_FILE")"
$ docker run --rm -d -v "$(basename "$SCCACHE_DIST_SCHEDULER_KEY_FILE"):$SCCACHE_DIST_SCHEDULER_KEY_FILE" sccache-dist:scheduler
$ docker run --rm -d -v "$(basename "$SCCACHE_DIST_SCHEDULER_KEY_FILE"):$SCCACHE_DIST_SCHEDULER_KEY_FILE" --privileged sccache-dist:worker
EOF_ERR
exit 1
fi
mkdir -p "$SCCACHE_BUILDS" "$SCCACHE_TOOLCHAINS"
if ! test -s /sccache/sccache.conf; then
cat <<EOF_SCCACHE_CONF > /sccache/sccache.conf
cache_dir = "$SCCACHE_TOOLCHAINS"
scheduler_url = "$SCHEDULER"
bind_addr = "$BIND_ADDR"
public_addr = "$PUBLIC_ADDR"
[builder]
type = "overlay"
build_dir = "$SCCACHE_BUILDS"
bwrap_path = "/usr/bin/bwrap"
[scheduler_auth]
type = "jwt_token"
token = "$(sccache-dist auth generate-jwt-hs256-server-token --secret-key "$(cat "$SCCACHE_DIST_SCHEDULER_KEY_FILE")" --server "$PUBLIC_ADDR")"
EOF_SCCACHE_CONF
fi
exec "$@"
EOF_ENTRY
chmod +x /sccache/entrypoint.sh
EOF_RUN
ENTRYPOINT ["/sccache/entrypoint.sh"]
CMD ["sccache-dist", "server", "--config", "/sccache/sccache.conf"]
FROM dist AS scheduler
ENV BIND_ADDR=0.0.0.0:10600
ENV PUBLIC_ADDR="$BIND_ADDR"
RUN <<EOF_RUN
cat <<"EOF_ENTRY" > /sccache/entrypoint.sh
#! /usr/bin/env bash
if ! test -s "$SCCACHE_DIST_SCHEDULER_KEY_FILE"; then
mkdir -p "$(dirname "$SCCACHE_DIST_SCHEDULER_KEY_FILE")"
cat <<EOF_WARN >&2
Scheduler secret key was not found at:
$SCCACHE_DIST_SCHEDULER_KEY_FILE
I have automatically generated this one for you:
$(sccache-dist auth generate-jwt-hs256-key | tee "$SCCACHE_DIST_SCHEDULER_KEY_FILE")
EOF_WARN
fi
if ! test -s /sccache/sccache.conf; then
cat <<EOF_SCCACHE_CONF > /sccache/sccache.conf
bind_addr = "$BIND_ADDR"
public_addr = "$PUBLIC_ADDR"
[client_auth]
type = "token"
token = "client_token"
[server_auth]
type = "jwt_hs256"
secret_key = "$(cat "$SCCACHE_DIST_SCHEDULER_KEY_FILE")"
EOF_SCCACHE_CONF
fi
exec "$@"
EOF_ENTRY
chmod +x /sccache/entrypoint.sh
EOF_RUN
ENTRYPOINT ["/sccache/entrypoint.sh"]
CMD ["sccache-dist", "scheduler", "--config", "/sccache/sccache.conf"] |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Enable running sccache-dist in Docker.
Description
When deploying sccache-dist using Docker on a machine, I noticed that the
public_addrfield passed from the build server to the scheduler is the same as the local address that sccache-dist is listening on. However, since the IP address of the Docker container is not the same as the physical machine's IP address, it cannot bind to it. To address this issue, I added an optional field calledbind_addr. If this field is empty, its default value is set topublic_addr. If sccache-dist is running inside a Docker container, users only need to set it to0.0.0.0:port.I believe this is helpful for exposing the sccache-dist compilation cluster behind an access gateway using port mapping because it no longer requires the sccache-dist build server to run on a specific separate public IP address.
What I do
X-real-ipheader when 'bind_addr' is set.