Move sysext image creation to shared helper

The code to package the folder in filesystem image was shared.
To ease maintenance and adding new features like systemd-repart for
GPT images with dm-verity, move it to an own helper. While doing so
we can also support the new properties that we added in systemd 252.

README.md

@@ -5,9 +5,9 @@ Flatcar Container Linux as an OS without a package manager is a good fit for ext
 The tools in this repository help you to create your own sysext images bundeling software to extend your base OS.
 The current focus is on Docker and containerd, contributions are welcome for other software.
 
-## Systemd-sysext on Flatcar
+## Systemd-sysext
 
-The `NAME.raw` sysext images (or `NAME` sysext directories) can be placed under `/etc/extensions/` or `/var/lib/extensions` in Flatcar to be activated on boot.
+The `NAME.raw` sysext images (or `NAME` sysext directories) can be placed under `/etc/extensions/` or `/var/lib/extensions` to be activated on boot by `systemd-sysext.service`.
 While systemd-sysext images are not really meant to also include the systemd service, Flatcar ships `ensure-sysext.service` as workaround to automatically load the image's services.
 This helper service is bound to `systemd-sysext.service` which activates the sysext images on boot.
 Currently it reloads the unit files from disk and reevaluates `multi-user.target`, `sockets.target`, and `timers.target`, making sure your enabled systemd units run.
@@ -20,6 +20,7 @@ The compatibility mechanism of sysext images requires a metadata file in the ima
 It needs to contain a matching OS `ID`, and either a matching `VERSION_ID` or `SYSEXT_LEVEL`.
 Since the rapid release cycle and automatic updates of Flatcar Container Linux make it hard to rely on particular OS libraries by specifying a dependency of the sysext image to the OS version, it is not recommended to match by `VERSION_ID`.
 Instead, Flatcar defined the `SYSEXT_LEVEL` value `1.0` to match for.
+With systemd 252 you can also use `ID=_any` and then neither `SYSEXT_LEVEL` nor `VERSION_ID` are needed.
 The sysext image should only include static binaries.
 
 Inside the image, binaries should be placed under `usr/bin/` and systemd units under `usr/lib/systemd/system/`.
@@ -65,7 +66,8 @@ SYSEXT_LEVEL=1.0
 ```
 
 This means other distributions will reject to load the sysext image by default.
-Use the configuration parameters in the tools to build for your distribution.
+Use the configuration parameters in the tools to build for your distribution (pass `OS=` to be the OS ID from `/etc/os-release`) or to build for any distribution (pass `OS=_any`).
+You can also set the architecture to be arm64 to fetch the right binaries and encode this information in the sysext image metadata.
 
 To add the automatic systemd unit loading to your distribution, store [`ensure-sysext.service`](https://raw.githubusercontent.com/flatcar/init/ccade77b6d568094fb4e4d4cf71b867819551798/systemd/system/ensure-sysext.service) in your systemd folder (e.g., `/etc/systemd/system/`) and enable the units: `systemctl enable --now ensure-sysext.service systemd-sysext.service`.
 
@@ -83,10 +85,10 @@ To ease the process, the `create_docker_sysext.sh` helper script takes care of d
 [… writes mydocker.raw into current directory …]
 ```
 
-Pass the `OS` or `ARCH` environment variables to build for another target than Flatcar amd64, e.g., for Fedora arm64:
+Pass the `OS` or `ARCH` environment variables to build for another target than Flatcar amd64, e.g., for any distro with arm64:
 
 ```
-OS=fedora ARCH=aarch64 ./create_docker_sysext.sh 20.10.13 mydocker
+OS=_any ARCH=aarch64 ./create_docker_sysext.sh 20.10.13 mydocker
 [… writes mydocker.raw into current directory …]
 ```
 

bake.sh

@@ -0,0 +1,60 @@
+#!/bin/bash
+set -euo pipefail
+
+OS="${OS-flatcar}"
+FORMAT="${FORMAT:-squashfs}"
+ARCH="${ARCH-}"
+
+# This script is to be called as helper by other scripts but can also be used standalone
+if [ $# -lt 1 ]; then
+  echo "Usage: $0 SYSEXTNAME"
+  echo "The script will make a SYSEXTNAME.raw image of the folder SYSEXTNAME, and create an os-release file in it, run with --help for the list of supported environment variables."
+  exit 1
+elif [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
+  echo "If ARCH is specified as environment variable the sysext image will be required to run on the given architecture."
+  echo "To build for another OS than Flatcar, pass 'OS=myosid' as environment variable (current value is '${OS}'), e.g., 'fedora' as found in 'ID' under '/etc/os-release', or pass 'OS=_any' for any OS."
+  echo "The '/etc/os-release' file of your OS has to include 'SYSEXT_LEVEL=1.0' as done in Flatcar (not needed for 'OS=_any')."
+  echo "If the mksquashfs tool is missing you can pass FORMAT=btrfs, FORMAT=ext4, or FORMAT=ext2 as environment variable (current value is '${FORMAT}') but the script won't change the ownership of the files in the SYSEXTNAME directory, so make sure that they are owned by root before creating the sysext image to avoid any problems."
+  echo
+  exit 1
+fi
+
+SYSEXTNAME="$1"
+
+if [ "${FORMAT}" != "squashfs" ] && [ "${FORMAT}" != "btrfs" ] && [ "${FORMAT}" != "ext4" ] && [ "${FORMAT}" != "ext2" ]; then
+  echo "Expected FORMAT=squashfs, FORMAT=btrfs, FORMAT=ext4, or FORMAT=ext2, got '${FORMAT}'" >&2
+  exit 1
+fi
+
+# Map to valid values for https://www.freedesktop.org/software/systemd/man/os-release.html#ARCHITECTURE=
+if [ "${ARCH}" = "amd64" ] || [ "${ARCH}" = "x86_64" ]; then
+  ARCH="x86-64"
+elif [ "${ARCH}" = "aarch64" ]; then
+  ARCH="arm64"
+fi
+
+mkdir -p "${SYSEXTNAME}/usr/lib/extension-release.d"
+{
+  echo "ID=${OS}"
+  if [ "${OS}" != "_any" ]; then
+    echo "SYSEXT_LEVEL=1.0"
+  fi
+  if [ "${ARCH}" != "" ]; then
+    echo "ARCHITECTURE=${ARCH}"
+  fi
+} > "${SYSEXTNAME}/usr/lib/extension-release.d/extension-release.${SYSEXTNAME}"
+rm -f "${SYSEXTNAME}".raw
+if [ "${FORMAT}" = "btrfs" ]; then
+  # Note: We didn't chown to root:root, meaning that the file ownership is left as is
+  mkfs.btrfs --mixed -m single -d single --shrink --rootdir "${SYSEXTNAME}" "${SYSEXTNAME}".raw
+  # This is for testing purposes and makes not much sense to use because --rootdir doesn't allow to enable compression
+elif [ "${FORMAT}" = "ext4" ] || [ "${FORMAT}" = "ext2" ]; then
+  # Assuming that 1 GB is enough
+  truncate -s 1G "${SYSEXTNAME}".raw
+  # Note: We didn't chown to root:root, meaning that the file ownership is left as is
+  mkfs."${FORMAT}" -E root_owner=0:0 -d "${SYSEXTNAME}" "${SYSEXTNAME}".raw
+  resize2fs -M "${SYSEXTNAME}".raw
+else
+  mksquashfs "${SYSEXTNAME}" "${SYSEXTNAME}".raw -all-root
+fi
+echo "Created ${SYSEXTNAME}.raw"

convert_torcx_image.sh

@@ -1,12 +1,15 @@
 #!/bin/bash
 set -euo pipefail
 
+SCRIPTFOLDER="$(dirname "$(readlink -f "$0")")"
+
 if [ $# -lt 2 ] || [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
   echo "Usage: $0 TORCXTAR SYSEXTNAME"
   echo "The script will unpack the Torcx tar ball and create a sysext squashfs image with the name SYSEXTNAME.raw in the current folder."
   echo "A temporary directory named SYSEXTNAME in the current folder will be created and deleted again."
   echo "All files in the sysext image will be owned by root."
   echo "The conversion is done on best effort, and only the TORCX_BINDIR (/usr/bin), TORCX_UNPACKDIR (/), and TORCX_IMAGEDIR ('') env vars in the systemd units get replaced."
+  "${SCRIPTFOLDER}"/bake.sh --help
   exit 1
 fi
 
@@ -38,9 +41,6 @@ for ENTRY in "${SYSEXTNAME}/usr/lib/systemd/system/"*.wants/*; do
   mkdir -p "${SYSEXTNAME}/usr/lib/systemd/system/${TARGET}.d"
   { echo "[Unit]"; echo "Upholds=${UNIT}"; } > "${SYSEXTNAME}/usr/lib/systemd/system/${TARGET}.d/10-${UNIT/./-}.conf"
 done
-mkdir -p "${SYSEXTNAME}/usr/lib/extension-release.d"
-{ echo "ID=flatcar" ; echo "SYSEXT_LEVEL=1.0" ; } > "${SYSEXTNAME}/usr/lib/extension-release.d/extension-release.${SYSEXTNAME}"
-rm -f "${SYSEXTNAME}".raw
-mksquashfs "${SYSEXTNAME}" "${SYSEXTNAME}".raw -all-root
+
+"${SCRIPTFOLDER}"/bake.sh "${SYSEXTNAME}"
 rm -rf "${SYSEXTNAME}"
-echo "Created ${SYSEXTNAME}.raw"

create_docker_sysext.sh

@@ -1,11 +1,10 @@
 #!/bin/bash
 set -euo pipefail
 
-ARCH="${ARCH-x86_64}"
-OS="${OS-flatcar}"
+export ARCH="${ARCH-x86_64}"
+SCRIPTFOLDER="$(dirname "$(readlink -f "$0")")"
 ONLY_CONTAINERD="${ONLY_CONTAINERD:-0}"
 ONLY_DOCKER="${ONLY_DOCKER:-0}"
-FORMAT="${FORMAT:-squashfs}"
 
 if [ $# -lt 2 ] || [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
   echo "Usage: $0 VERSION SYSEXTNAME"
@@ -16,21 +15,14 @@ if [ $# -lt 2 ] || [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
   echo "To only package containerd without Docker, pass ONLY_CONTAINERD=1 as environment variable (current value is '${ONLY_CONTAINERD}')."
   echo "To only package Docker without containerd and runc, pass ONLY_DOCKER=1 as environment variable (current value is '${ONLY_DOCKER}')."
   echo "To use arm64 pass 'ARCH=aarch64' as environment variable (current value is '${ARCH}')."
-  echo "To build for another OS than Flatcar, pass 'OS=myosid' as environment variable (current value is '${OS}'), e.g., 'fedora' as found in 'ID' under '/etc/os-release'."
-  echo "The '/etc/os-release' file of your OS has to include 'SYSEXT_LEVEL=1.0' as done in Flatcar."
-  echo "If the mksquashfs tool is missing you can pass FORMAT=btrfs, FORMAT=ext4, or FORMAT=ext2 as environment variable but the files won't be owned by root."
-  echo
+  "${SCRIPTFOLDER}"/bake.sh --help
   exit 1
 fi
 
 if [ "${ONLY_CONTAINERD}" = 1 ] && [ "${ONLY_DOCKER}" = 1 ]; then
   echo "Cannot set both ONLY_CONTAINERD and ONLY_DOCKER" >&2
   exit 1
 fi
-if [ "${FORMAT}" != "squashfs" ] && [ "${FORMAT}" != "btrfs" ] && [ "${FORMAT}" != "ext4" ] && [ "${FORMAT}" != "ext2" ]; then
-  echo "Expected FORMAT=squashfs, FORMAT=btrfs, FORMAT=ext4, or FORMAT=ext2 got '${FORMAT}'" >&2
-  exit 1
-fi
 
 VERSION="$1"
 SYSEXTNAME="$2"
@@ -135,21 +127,6 @@ EOF
 EOF
   sed 's/SystemdCgroup = true/SystemdCgroup = false/g' "${SYSEXTNAME}/usr/share/containerd/config.toml" > "${SYSEXTNAME}/usr/share/containerd/config-cgroupfs.toml"
 fi
-mkdir -p "${SYSEXTNAME}/usr/lib/extension-release.d"
-{ echo "ID=${OS}" ; echo "SYSEXT_LEVEL=1.0" ; } > "${SYSEXTNAME}/usr/lib/extension-release.d/extension-release.${SYSEXTNAME}"
-rm -f "${SYSEXTNAME}".raw
-if [ "${FORMAT}" = "btrfs" ]; then
-  # Note: We didn't chown to root:root, meaning that the file ownership is left as is
-  mkfs.btrfs --mixed -m single -d single --shrink --rootdir "${SYSEXTNAME}" "${SYSEXTNAME}".raw
-  # This is for testing purposes and makes not much sense to use because --rootdir doesn't allow to enable compression
-elif [ "${FORMAT}" = "ext4" ] || [ "${FORMAT}" = "ext2" ]; then
-  # Assuming that 1 GB is enough
-  truncate -s 1G "${SYSEXTNAME}".raw
-  # Note: We didn't chown to root:root, meaning that the file ownership is left as is
-  mkfs."${FORMAT}" -E root_owner=0:0 -d "${SYSEXTNAME}" "${SYSEXTNAME}".raw
-  resize2fs -M "${SYSEXTNAME}".raw
-else
-  mksquashfs "${SYSEXTNAME}" "${SYSEXTNAME}".raw -all-root
-fi
+
+"${SCRIPTFOLDER}"/bake.sh "${SYSEXTNAME}"
 rm -rf "${SYSEXTNAME}"
-echo "Created ${SYSEXTNAME}.raw"

create_wasmtime_sysext.sh

@@ -1,25 +1,16 @@
 #!/bin/bash
 set -euo pipefail
 
-ARCH="${ARCH-x86_64}"
-OS="${OS-flatcar}"
-FORMAT="${FORMAT:-squashfs}"
+export ARCH="${ARCH-x86_64}"
+SCRIPTFOLDER="$(dirname "$(readlink -f "$0")")"
 
 if [ $# -lt 2 ] || [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
   echo "Usage: $0 VERSION SYSEXTNAME"
   echo "The script will download the wasmtime release tar ball (e.g., for 4.0.0) and create a sysext squashfs image with the name SYSEXTNAME.raw in the current folder."
   echo "A temporary directory named SYSEXTNAME in the current folder will be created and deleted again."
   echo "All files in the sysext image will be owned by root."
   echo "To use arm64 pass 'ARCH=aarch64' as environment variable (current value is '${ARCH}')."
-  echo "To build for another OS than Flatcar, pass 'OS=myosid' as environment variable (current value is '${OS}'), e.g., 'fedora' as found in 'ID' under '/etc/os-release'."
-  echo "The '/etc/os-release' file of your OS has to include 'SYSEXT_LEVEL=1.0' as done in Flatcar."
-  echo "If the mksquashfs tool is missing you can pass FORMAT=btrfs, FORMAT=ext4, or FORMAT=ext2 as environment variable but the files won't be owned by root."
-  echo
-  exit 1
-fi
-
-if [ "${FORMAT}" != "squashfs" ] && [ "${FORMAT}" != "btrfs" ] && [ "${FORMAT}" != "ext4" ] && [ "${FORMAT}" != "ext2" ]; then
-  echo "Expected FORMAT=squashfs, FORMAT=btrfs, FORMAT=ext4, or FORMAT=ext2, got '${FORMAT}'" >&2
+  "${SCRIPTFOLDER}"/bake.sh --help
   exit 1
 fi
 
@@ -35,21 +26,5 @@ rm "wasmtime-${VERSION}.tar.xz"
 mkdir -p "${SYSEXTNAME}"/usr/bin
 mv "${SYSEXTNAME}"/"wasmtime-v${VERSION}-${ARCH}-linux"/wasmtime "${SYSEXTNAME}"/usr/bin/
 rm -r "${SYSEXTNAME}"/"wasmtime-v${VERSION}-${ARCH}-linux"
-mkdir -p "${SYSEXTNAME}/usr/lib/extension-release.d"
-{ echo "ID=${OS}" ; echo "SYSEXT_LEVEL=1.0" ; } > "${SYSEXTNAME}/usr/lib/extension-release.d/extension-release.${SYSEXTNAME}"
-rm -f "${SYSEXTNAME}".raw
-if [ "${FORMAT}" = "btrfs" ]; then
-  # Note: We didn't chown to root:root, meaning that the file ownership is left as is
-  mkfs.btrfs --mixed -m single -d single --shrink --rootdir "${SYSEXTNAME}" "${SYSEXTNAME}".raw
-  # This is for testing purposes and makes not much sense to use because --rootdir doesn't allow to enable compression
-elif [ "${FORMAT}" = "ext4" ] || [ "${FORMAT}" = "ext2" ]; then
-  # Assuming that 1 GB is enough
-  truncate -s 1G "${SYSEXTNAME}".raw
-  # Note: We didn't chown to root:root, meaning that the file ownership is left as is
-  mkfs."${FORMAT}" -E root_owner=0:0 -d "${SYSEXTNAME}" "${SYSEXTNAME}".raw
-  resize2fs -M "${SYSEXTNAME}".raw
-else
-  mksquashfs "${SYSEXTNAME}" "${SYSEXTNAME}".raw -all-root
-fi
+"${SCRIPTFOLDER}"/bake.sh "${SYSEXTNAME}"
 rm -rf "${SYSEXTNAME}"
-echo "Created ${SYSEXTNAME}.raw"