Skip to content

Ubuntu 20.04 #120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Ubuntu 20.04 #120

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 61 additions & 48 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Use Ubuntu 18.04 LTS as our base image.
FROM ubuntu:18.04
# syntax=docker/dockerfile:1

Comment on lines +1 to +2
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can add this, since Docker uses it in all their latest examples.

# Use Ubuntu 20.04 LTS as our base image.
FROM ubuntu:20.04

# The Rust toolchain to use when building our image. Set by `hooks/build`.
ARG TOOLCHAIN=stable
Expand All @@ -10,6 +12,10 @@ ARG TOOLCHAIN=stable
#
# ALSO UPDATE hooks/build!
ARG OPENSSL_VERSION=1.1.1m
# Needs evaluation;
# fails for libpq with:
# "configure: error: library 'crypto' is required for OpenSSL"
# ARG OPENSSL_VERSION=3.0.0
Comment on lines +15 to +18
Copy link
Owner

@emk emk Jan 2, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any support of OpenSSL 3.0.0 should happen in a separate PR, and it should wait until all the examples in the test suite support dynamic linking of OpenSSL. (They may do so already.)

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll also need to consider how many existing users this breaks in the short term. We try to avoid breaking working CI setups, though in a 6-year-old project, that occasionally happens to people who pin to :stable instead of a version.


# Versions for other dependencies. Here are the places to check for new
# releases:
Expand Down Expand Up @@ -38,8 +44,8 @@ ARG POSTGRESQL_VERSION=11.14
#
# We also set up a `rust` user by default. This user has sudo privileges if you
# need to install any more software.
RUN apt-get update && \
export DEBIAN_FRONTEND=noninteractive && \
RUN export DEBIAN_FRONTEND=noninteractive && \
apt-get update && \
Comment on lines -41 to +48
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please leave out any minor changes that don't change anything, unless the current version is clearly inelegant.

apt-get install -yq \
build-essential \
cmake \
Expand All @@ -61,32 +67,6 @@ RUN apt-get update && \
apt-get clean && rm -rf /var/lib/apt/lists/* && \
useradd rust --user-group --create-home --shell /bin/bash --groups sudo

# - `mdbook` is the standard Rust tool for making searchable HTML manuals.
# - `mdbook-graphviz` allows using inline GraphViz drawing commands to add illustrations.
# - `cargo-about` generates a giant license file for all dependencies.
# - `cargo-audit` checks for security vulnerabilities. We include it for backwards compat.
# - `cargo-deny` does everything `cargo-audit` does, plus check licenses & many other things.
RUN curl -fLO https://github.com/rust-lang-nursery/mdBook/releases/download/v$MDBOOK_VERSION/mdbook-v$MDBOOK_VERSION-x86_64-unknown-linux-gnu.tar.gz && \
tar xf mdbook-v$MDBOOK_VERSION-x86_64-unknown-linux-gnu.tar.gz && \
mv mdbook /usr/local/bin/ && \
rm -f mdbook-v$MDBOOK_VERSION-x86_64-unknown-linux-gnu.tar.gz && \
Comment on lines -64 to -72
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please leave this whole section at the top of the file. Thank you!

curl -fLO https://github.com/dylanowen/mdbook-graphviz/releases/download/v$MDBOOK_GRAPHVIZ_VERSION/mdbook-graphviz_v${MDBOOK_GRAPHVIZ_VERSION}_x86_64-unknown-linux-musl.zip && \
unzip mdbook-graphviz_v${MDBOOK_GRAPHVIZ_VERSION}_x86_64-unknown-linux-musl.zip && \
mv mdbook-graphviz /usr/local/bin/ && \
rm -f mdbook-graphviz_v${MDBOOK_GRAPHVIZ_VERSION}_x86_64-unknown-linux-musl.zip && \
curl -fLO https://github.com/EmbarkStudios/cargo-about/releases/download/$CARGO_ABOUT_VERSION/cargo-about-$CARGO_ABOUT_VERSION-x86_64-unknown-linux-musl.tar.gz && \
tar xf cargo-about-$CARGO_ABOUT_VERSION-x86_64-unknown-linux-musl.tar.gz && \
mv cargo-about-$CARGO_ABOUT_VERSION-x86_64-unknown-linux-musl/cargo-about /usr/local/bin/ && \
rm -rf cargo-about-$CARGO_ABOUT_VERSION-x86_64-unknown-linux-musl.tar.gz cargo-about-$CARGO_ABOUT_VERSION-x86_64-unknown-linux-musl && \
curl -fLO https://github.com/rustsec/rustsec/releases/download/cargo-audit%2Fv${CARGO_AUDIT_VERSION}/cargo-audit-x86_64-unknown-linux-gnu-v${CARGO_AUDIT_VERSION}.tgz && \
tar xf cargo-audit-x86_64-unknown-linux-gnu-v${CARGO_AUDIT_VERSION}.tgz && \
cp cargo-audit-x86_64-unknown-linux-gnu-v${CARGO_AUDIT_VERSION}/cargo-audit /usr/local/bin/ && \
rm -rf cargo-audit-x86_64-unknown-linux-gnu-v${CARGO_AUDIT_VERSION}.tgz cargo-audit-x86_64-unknown-linux-gnu-v${CARGO_AUDIT_VERSION} && \
curl -fLO https://github.com/EmbarkStudios/cargo-deny/releases/download/$CARGO_DENY_VERSION/cargo-deny-$CARGO_DENY_VERSION-x86_64-unknown-linux-musl.tar.gz && \
tar xf cargo-deny-$CARGO_DENY_VERSION-x86_64-unknown-linux-musl.tar.gz && \
mv cargo-deny-$CARGO_DENY_VERSION-x86_64-unknown-linux-musl/cargo-deny /usr/local/bin/ && \
rm -rf cargo-deny-$CARGO_DENY_VERSION-x86_64-unknown-linux-musl cargo-deny-$CARGO_DENY_VERSION-x86_64-unknown-linux-musl.tar.gz

# Static linking for C++ code
RUN ln -s "/usr/bin/g++" "/usr/bin/musl-g++"

Expand All @@ -97,36 +77,36 @@ RUN ln -s "/usr/bin/g++" "/usr/bin/musl-g++"
# necessarily the right ones) in an effort to compile OpenSSL 1.1's "engine"
# component. It's possible that this will cause bizarre and terrible things to
# happen. There may be "sanitized" header
RUN echo "Building OpenSSL" && \
RUN echo "Building OpenSSL ${OPENSSL_VERSION}" && \
Comment on lines -100 to +80
Copy link
Owner

@emk emk Jan 2, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's a great idea to add the version numbers!

As for variable syntax, according to the spec:

Environment variables are notated in the Dockerfile either with $variable_name or ${variable_name}. They are treated equivalently and the brace syntax is typically used to address issues with variable names with no whitespace, like ${foo}_bar.

This project prefers $variable_name when unambiguous, and only uses ${variable_name} when the next character would otherwise be interpreted as part of the name. Let's use $variable_name here and elsewhere.

ls /usr/include/linux && \
mkdir -p /usr/local/musl/include && \
ln -s /usr/include/linux /usr/local/musl/include/linux && \
ln -s /usr/include/x86_64-linux-gnu/asm /usr/local/musl/include/asm && \
ln -s /usr/include/asm-generic /usr/local/musl/include/asm-generic && \
cd /tmp && \
short_version="$(echo "$OPENSSL_VERSION" | sed s'/[a-z]$//' )" && \
curl -fLO "https://www.openssl.org/source/openssl-$OPENSSL_VERSION.tar.gz" || \
short_version="$(echo "${OPENSSL_VERSION}" | sed s'/[a-z]$//' )" && \
curl -fLO "https://www.openssl.org/source/openssl-${OPENSSL_VERSION}.tar.gz" || \
curl -fLO "https://www.openssl.org/source/old/$short_version/openssl-$OPENSSL_VERSION.tar.gz" && \
tar xvzf "openssl-$OPENSSL_VERSION.tar.gz" && cd "openssl-$OPENSSL_VERSION" && \
tar xvzf "openssl-${OPENSSL_VERSION}.tar.gz" && cd "openssl-${OPENSSL_VERSION}" && \
env CC=musl-gcc ./Configure no-shared no-zlib -fPIC --prefix=/usr/local/musl -DOPENSSL_NO_SECURE_MEMORY linux-x86_64 && \
env C_INCLUDE_PATH=/usr/local/musl/include/ make depend && \
env C_INCLUDE_PATH=/usr/local/musl/include/ make && \
make install && \
rm /usr/local/musl/include/linux /usr/local/musl/include/asm /usr/local/musl/include/asm-generic && \
rm -r /tmp/*

RUN echo "Building zlib" && \
RUN echo "Building zlib ${ZLIB_VERSION}" && \
cd /tmp && \
curl -fLO "http://zlib.net/zlib-$ZLIB_VERSION.tar.gz" && \
tar xzf "zlib-$ZLIB_VERSION.tar.gz" && cd "zlib-$ZLIB_VERSION" && \
curl -fLO "http://zlib.net/zlib-${ZLIB_VERSION}.tar.gz" && \
tar xzf "zlib-${ZLIB_VERSION}.tar.gz" && cd "zlib-${ZLIB_VERSION}" && \
CC=musl-gcc ./configure --static --prefix=/usr/local/musl && \
make && make install && \
rm -r /tmp/*

RUN echo "Building libpq" && \
RUN echo "Building libpq ${POSTGRESQL_VERSION}" && \
cd /tmp && \
curl -fLO "https://ftp.postgresql.org/pub/source/v$POSTGRESQL_VERSION/postgresql-$POSTGRESQL_VERSION.tar.gz" && \
tar xzf "postgresql-$POSTGRESQL_VERSION.tar.gz" && cd "postgresql-$POSTGRESQL_VERSION" && \
curl -fLO "https://ftp.postgresql.org/pub/source/v${POSTGRESQL_VERSION}/postgresql-${POSTGRESQL_VERSION}.tar.gz" && \
tar xzf "postgresql-${POSTGRESQL_VERSION}.tar.gz" && cd "postgresql-${POSTGRESQL_VERSION}" && \
CC=musl-gcc CPPFLAGS=-I/usr/local/musl/include LDFLAGS=-L/usr/local/musl/lib ./configure --with-openssl --without-readline --prefix=/usr/local/musl && \
cd src/interfaces/libpq && make all-static-lib && make install-lib-static && \
cd ../../bin/pg_config && make && make install && \
Expand All @@ -139,7 +119,7 @@ RUN echo "Building libpq" && \
# Install a `git credentials` helper for using GH_USER and GH_TOKEN to access
# private repositories if desired. We make sure this is configured for root,
# here, and for the `rust` user below.
ADD git-credential-ghtoken /usr/local/bin/ghtoken
COPY git-credential-ghtoken /usr/local/bin/ghtoken
Comment on lines -142 to +122
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, it looks like COPY is officially preferred, so we can use it when it works.

RUN git config --global credential.https://github.com.helper ghtoken

# Set up our path with all our binary directories, including those for the
Expand All @@ -155,16 +135,23 @@ ENV RUSTUP_HOME=/opt/rust/rustup \
# interact with the user or fool around with TTYs. We also set the default
# `--target` to musl so that our users don't need to keep overriding it
# manually.
RUN curl https://sh.rustup.rs -sSf | \
RUN echo "Installing Rust (${TOOLCHAIN} toolchain)" && \
curl https://sh.rustup.rs -sSf | \
Comment on lines -158 to +139
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 for logging the toolchain.

env CARGO_HOME=/opt/rust/cargo \
sh -s -- -y --default-toolchain $TOOLCHAIN --profile minimal --no-modify-path && \
sh -s -- -y --default-toolchain ${TOOLCHAIN} --profile minimal --no-modify-path && \
env CARGO_HOME=/opt/rust/cargo \
rustup component add rustfmt && \
env CARGO_HOME=/opt/rust/cargo \
rustup component add clippy && \
env CARGO_HOME=/opt/rust/cargo \
rustup target add x86_64-unknown-linux-musl
ADD cargo-config.toml /opt/rust/cargo/config
# if we also install other target's build dependencies (like tools, libs, and linkers)
# rustup target add x86_64-unknown-linux-musl \
# rustup target add aarch64-unknown-linux-musl \
# rustup target add armv7-unknown-linux-musleabihf \
# rustup target add armv7-unknown-linux-musleabi
Comment on lines +148 to +152
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are definitely not going to try to support multiple targets in a single image. And we won't be shipping any targets that can't link diesel and sqlx. I tried this in the past and it was maintenance headache and backwards compatibility guarantee I didn't want.

Honestly, people should use cross unless they need C libraries that are not supported by cross. I've already converted as many projects as I can, and it's great. (But it won't work for everything.)


COPY cargo-config.toml /opt/rust/cargo/config

# Set up our environment variables so that we cross-compile using musl-libc by
# default.
Expand All @@ -181,12 +168,38 @@ ENV X86_64_UNKNOWN_LINUX_MUSL_OPENSSL_DIR=/usr/local/musl/ \
# slow down image builds). This will use the static linking toolchain, but that
# should be OK.
#
# - `cargo-deb` builds Debian packages.
RUN env CARGO_HOME=/opt/rust/cargo cargo install -f cargo-deb && \
rm -rf /opt/rust/cargo/registry/
# - `cargo-deb` is a helper command which automatically creates binary Debian packages (.deb) from Cargo projects.
# - `mdbook` is the standard Rust tool for making searchable HTML manuals.
# - `mdbook-graphviz` allows using inline GraphViz drawing commands to add illustrations.
# - `cargo-about` generates a giant license file for all dependencies.
# - `cargo-audit` checks for security vulnerabilities. We include it for backwards compat.
# - `cargo-deny` does everything `cargo-audit` does, plus check licenses & many other things.
RUN echo "Installing cargo tools" && \
env CARGO_HOME=/opt/rust/cargo cargo install -f cargo-deb && \
rm -rf /opt/rust/cargo/registry/ && \
curl -fLO https://github.com/rust-lang-nursery/mdBook/releases/download/v${MDBOOK_VERSION}/mdbook-v${MDBOOK_VERSION}-x86_64-unknown-linux-gnu.tar.gz && \
tar xf mdbook-v${MDBOOK_VERSION}-x86_64-unknown-linux-gnu.tar.gz && \
mv mdbook /usr/local/bin/ && \
rm -f mdbook-v${MDBOOK_VERSION}-x86_64-unknown-linux-gnu.tar.gz && \
curl -fLO https://github.com/dylanowen/mdbook-graphviz/releases/download/v${MDBOOK_GRAPHVIZ_VERSION}/mdbook-graphviz_v${MDBOOK_GRAPHVIZ_VERSION}_x86_64-unknown-linux-musl.zip && \
unzip mdbook-graphviz_v${MDBOOK_GRAPHVIZ_VERSION}_x86_64-unknown-linux-musl.zip && \
mv mdbook-graphviz /usr/local/bin/ && \
rm -f mdbook-graphviz_v${MDBOOK_GRAPHVIZ_VERSION}_x86_64-unknown-linux-musl.zip && \
curl -fLO https://github.com/EmbarkStudios/cargo-about/releases/download/${CARGO_ABOUT_VERSION}/cargo-about-${CARGO_ABOUT_VERSION}-x86_64-unknown-linux-musl.tar.gz && \
tar xf cargo-about-${CARGO_ABOUT_VERSION}-x86_64-unknown-linux-musl.tar.gz && \
mv cargo-about-${CARGO_ABOUT_VERSION}-x86_64-unknown-linux-musl/cargo-about /usr/local/bin/ && \
rm -rf cargo-about-${CARGO_ABOUT_VERSION}-x86_64-unknown-linux-musl.tar.gz cargo-about-${CARGO_ABOUT_VERSION}-x86_64-unknown-linux-musl && \
curl -fLO https://github.com/rustsec/rustsec/releases/download/cargo-audit%2Fv${CARGO_AUDIT_VERSION}/cargo-audit-x86_64-unknown-linux-gnu-v${CARGO_AUDIT_VERSION}.tgz && \
tar xf cargo-audit-x86_64-unknown-linux-gnu-v${CARGO_AUDIT_VERSION}.tgz && \
cp cargo-audit-x86_64-unknown-linux-gnu-v${CARGO_AUDIT_VERSION}/cargo-audit /usr/local/bin/ && \
rm -rf cargo-audit-x86_64-unknown-linux-gnu-v${CARGO_AUDIT_VERSION}.tgz cargo-audit-x86_64-unknown-linux-gnu-v${CARGO_AUDIT_VERSION} && \
curl -fLO https://github.com/EmbarkStudios/cargo-deny/releases/download/${CARGO_DENY_VERSION}/cargo-deny-${CARGO_DENY_VERSION}-x86_64-unknown-linux-musl.tar.gz && \
tar xf cargo-deny-${CARGO_DENY_VERSION}-x86_64-unknown-linux-musl.tar.gz && \
mv cargo-deny-${CARGO_DENY_VERSION}-x86_64-unknown-linux-musl/cargo-deny /usr/local/bin/ && \
rm -rf cargo-deny-${CARGO_DENY_VERSION}-x86_64-unknown-linux-musl cargo-deny-${CARGO_DENY_VERSION}-x86_64-unknown-linux-musl.tar.gz

# Allow sudo without a password.
ADD sudoers /etc/sudoers.d/nopasswd
COPY sudoers /etc/sudoers.d/nopasswd

# Run all further code as user `rust`, create our working directories, install
# our config file, and set up our credential helper.
Expand Down
8 changes: 8 additions & 0 deletions examples/linking-with-git2/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,11 @@ ADD --chown=rust:rust . ./

# Build our application.
RUN cargo build

# Now, we need to build our _real_ Docker container, copying in `linking-with-git2`.
FROM debian:stretch-slim
RUN apt-get update && apt-get install -y ca-certificates
COPY --from=builder \
/home/rust/src/target/x86_64-unknown-linux-musl/debug/linking-with-git2 \
/usr/local/bin/
CMD /usr/local/bin/linking-with-git2
Comment on lines +14 to +21
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is probably unnecessary for libgit2. We have other examples anyways.

4 changes: 2 additions & 2 deletions examples/using-sqlx/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ ADD --chown=rust:rust . ./
RUN cargo build --release

# Now, we need to build our _real_ Docker container, copying in `using-sqlx`.
FROM alpine:latest
RUN apk --no-cache add ca-certificates
FROM debian:stretch-slim
RUN apt-get update && apt-get install -y ca-certificates
Comment on lines -20 to +21
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All images should continue to use Alpine, now that we do ldd checks on the host.

Many thanks for figuring out ldd, and for preparing this PR!

COPY --from=builder \
/home/rust/src/target/x86_64-unknown-linux-musl/release/using-sqlx \
/usr/local/bin/
Expand Down
12 changes: 8 additions & 4 deletions test-image
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,14 @@ echo "==== Verifying static linking"

# Make sure we can build a static executable using `sqlx`.
docker build -t rust-musl-builder-using-sqlx examples/using-sqlx
docker run --rm rust-musl-builder-using-sqlx sh -c "
docker run --rm rust-musl-builder-using-sqlx bash -c "
set -euo pipefail

echo -e '--- Test case for sqlx:'
echo 'ldd says:'
if ldd /usr/local/bin/using-sqlx; then
lddresult=\$(ldd /usr/local/bin/using-sqlx)
echo \$lddresult
if [[ "\$lddresult" != *statically\ linked* ]]; then
echo '[FAIL] Executable is not static!' 1>&2
exit 1
fi
Expand All @@ -33,14 +35,16 @@ echo -e '[PASS] using-sqlx binary is statically linked.\n'
docker build -t rust-musl-builder-linking-with-git2 examples/linking-with-git2
docker run --rm rust-musl-builder-linking-with-git2 bash -c "
set -euo pipefail
cd /home/rust/src

echo -e '--- Test case for libgit2:'
echo 'ldd says:'
if ldd target/x86_64-unknown-linux-musl/debug/linking-with-git2; then
lddresult=\$(ldd /usr/local/bin/linking-with-git2)
echo \$lddresult
if [[ "\$lddresult" != *statically\ linked* ]]; then
echo '[FAIL] Executable is not static!' 1>&2
exit 1
fi

echo -e '[PASS] libgit2 binary is statically linked.\n'
"

Expand Down