Skip to content

Commit 42cea7d

Browse files
bors[bot]Alexhuszagh
bors[bot]
and
Alexhuszagh
authored
Merge #753
753: Add scripts to validate target info. r=Emilgardis a=Alexhuszagh Attempts to recreate the table present in README.md, which can extract the target information for a single target or for all targets. Should currently support the following targets: - Android - glibc - musl - MinGW - FreeBSD - NetBSD - newlib Closes #686. This produces the following table: | Target | libc | GCC | C++ | QEMU | |:-:|:-:|:-:|:-:|:-:| | `aarch64-linux-android` | 9.0.8 | 9.0.8 | ✓ | 5.1.0 | | `aarch64-unknown-linux-gnu` | 2.23 | 5.4.0 | ✓ | 5.1.0 | | `aarch64-unknown-linux-musl` | 1.2.0 | 9.2.0 | ✓ | 5.1.0 | | `arm-linux-androideabi` | 9.0.8 | 9.0.8 | ✓ | 5.1.0 | | `arm-unknown-linux-gnueabi` | 2.23 | 5.4.0 | ✓ | 5.1.0 | | `arm-unknown-linux-gnueabihf` | 2.17 | 8.3.0 | ✓ | 5.1.0 | | `arm-unknown-linux-musleabi` | 1.2.0 | 9.2.0 | ✓ | 5.1.0 | | `arm-unknown-linux-musleabihf` | 1.2.0 | 9.2.0 | ✓ | 5.1.0 | | `armv5te-unknown-linux-gnueabi` | 2.27 | 7.5.0 | ✓ | 5.1.0 | | `armv5te-unknown-linux-musleabi` | 1.2.0 | 9.2.0 | ✓ | 5.1.0 | | `armv7-linux-androideabi` | 9.0.8 | 9.0.8 | ✓ | 5.1.0 | | `armv7-unknown-linux-gnueabihf` | 2.23 | 5.4.0 | ✓ | 5.1.0 | | `armv7-unknown-linux-musleabihf` | 1.2.0 | 9.2.0 | ✓ | 5.1.0 | | `i586-unknown-linux-gnu` | 2.23 | 5.4.0 | ✓ | N/A | | `i586-unknown-linux-musl` | 1.2.0 | 9.2.0 | ✓ | N/A | | `i686-linux-android` | 9.0.8 | 9.0.8 | ✓ | 5.1.0 | | `i686-pc-windows-gnu` | N/A | 7.5 | ✓ | N/A | | `i686-unknown-freebsd` | 1.5 | 6.4.0 | ✓ | N/A | | `i686-unknown-linux-gnu` | 2.23 | 5.4.0 | ✓ | 5.1.0 | | `i686-unknown-linux-musl` | 1.2.0 | 9.2.0 | ✓ | N/A | | `mips-unknown-linux-gnu` | 2.23 | 5.4.0 | ✓ | 5.1.0 | | `mips-unknown-linux-musl` | 1.2.0 | 9.2.0 | ✓ | 5.1.0 | | `mips64-unknown-linux-gnuabi64` | 2.23 | 5.4.0 | ✓ | 5.1.0 | | `mips64el-unknown-linux-gnuabi64` | 2.23 | 5.4.0 | ✓ | 5.1.0 | | `mipsel-unknown-linux-gnu` | 2.23 | 5.4.0 | ✓ | 5.1.0 | | `mipsel-unknown-linux-musl` | 1.2.0 | 9.2.0 | ✓ | 5.1.0 | | `powerpc-unknown-linux-gnu` | 2.23 | 5.4.0 | ✓ | 5.1.0 | | `powerpc64-unknown-linux-gnu` | 2.23 | 5.4.0 | ✓ | 5.1.0 | | `powerpc64le-unknown-linux-gnu` | 2.23 | 5.4.0 | ✓ | N/A | | `riscv64gc-unknown-linux-gnu` | 2.27 | 7.5.0 | ✓ | 5.1.0 | | `s390x-unknown-linux-gnu` | 2.23 | 5.4.0 | ✓ | 5.1.0 | | `sparc64-unknown-linux-gnu` | 2.23 | 5.4.0 | ✓ | 5.1.0 | | `thumbv6m-none-eabi` | 2.2.0 | 4.9.3 | | N/A | | `thumbv7em-none-eabi` | 2.2.0 | 4.9.3 | | N/A | | `thumbv7em-none-eabihf` | 2.2.0 | 4.9.3 | | N/A | | `thumbv7m-none-eabi` | 2.2.0 | 4.9.3 | | N/A | | `x86_64-linux-android` | 9.0.8 | 9.0.8 | ✓ | 5.1.0 | | `x86_64-pc-windows-gnu` | N/A | 7.3 | ✓ | N/A | | `x86_64-unknown-freebsd` | 1.5 | 6.4.0 | ✓ | N/A | | `x86_64-unknown-linux-gnu` | 2.17 | 4.8.5 | ✓ | 4.2.1 | | `x86_64-unknown-linux-musl` | 1.2.0 | 9.2.0 | ✓ | N/A | | `x86_64-unknown-netbsd` | 9.2.0 | 9.4.0 | ✓ | N/A | There's a few mild differences here, but I think they're correct: - FreeBSD uses the [symbol versioning](https://wiki.freebsd.org/SymbolVersioning) for libc. - Android libc uses the LLVM version, and the compiler version isn't the native GCC toolchain either. A few notes on how this works: - C++ support is checked by there being a C++ executable, and it can compile a simple program (see below): - Compiler version is detected by the output of GCC or Clang by a regular expression. - Qemu version is detected by the presence of a Qemu binary and extracted via a regular expression. - libc verson is quite intricate, but interesting: - If Android, the libc version is just the compiler version (Clang == LLVM). - If musl, we execute the binary and extract the version via a regular expression. - If glibc, we use the versioned filenames to get the version (`libc-2.17.so`). - If newlib, we use `dpkg` to get the package version which has the newlib version. - For FreeBSD, we wrote the FreeBSD version to file, and then match that to the libc version. - For NetBSD, we can read the version from the `libc.so` symbols, grepped for `NetBSD`. - For Windows, intentional passthrough. **C++ Program** This may fail in the case we cannot compile a C program to begin with (AKA, newlib, which we have missing startfiles, etc.). This really doesn't matter right now, since none of these bare-metal targets support C++, although in the future we may want to add `-nostartfiles` during compilation. ```cpp #include <iostream> int main() { std::cout << "Testing this" << std::endl; } ``` Co-authored-by: Alex Huszagh <ahuszagh@gmail.com>
2 parents cc40325 + e45ddaf commit 42cea7d

2 files changed

Lines changed: 392 additions & 0 deletions

File tree

‎ci/extract_image_info.sh‎

Lines changed: 312 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,312 @@
1+
#!/usr/bin/env bash
2+
#
3+
# this script can be invoked as follows:
4+
# export TARGET=aarch64-unknown-linux-musl
5+
# docker run -e TARGET ghcr.io/cross-rs/"$TARGET":main bash -c "`cat extract_target_info.sh`"
6+
#
7+
# the output will be similar to the following:
8+
# | `aarch64-unknown-linux-musl` | 1.2.0 | 9.2.0 | ✓ | 5.1.0 |
9+
#
10+
# in short, it recreates the table except for the test section in README.md.
11+
12+
set -eo pipefail
13+
14+
# shellcheck disable=SC2153
15+
target="${TARGET}"
16+
arch="${target//-*/}"
17+
18+
extract_regex_version() {
19+
# executing shared libraries outputs to stderr, rest to stdout
20+
version="$($1 --version 2>&1)"
21+
if [[ "${version}" =~ $2 ]]; then
22+
echo "${BASH_REMATCH[1]}"
23+
else
24+
echo "Unable to match $3 version info for ${target}." 1>&2
25+
exit 1
26+
fi
27+
}
28+
29+
max_glibc_version() {
30+
# glibc versions have the following format:
31+
# `libc-$major-$minor.so.$abi`, where the `.$abi` may be optional.
32+
# shellcheck disable=SC2207
33+
local -a paths=( $(ls "${1}"/libc-[0-9]*.[0-9]*.so*) )
34+
local major=0
35+
local minor=0
36+
local version
37+
local x
38+
local y
39+
local is_larger
40+
41+
for i in "${!paths[@]}"; do
42+
file=$(basename "${paths[$i]}")
43+
version="${file//libc-/}"
44+
x=$(echo "${version}" | cut -d '.' -f 1)
45+
y=$(echo "${version}" | cut -d '.' -f 2)
46+
is_larger=
47+
48+
if [ "${x}" -gt "${major}" ]; then
49+
is_larger=1
50+
elif [ "${x}" -eq "${major}" ] && [ "${y}" -gt "${minor}" ]; then
51+
is_larger=1
52+
fi
53+
54+
if [ -n "${is_larger}" ]; then
55+
major="${x}"
56+
minor="${y}"
57+
fi
58+
done
59+
60+
echo "${major}.${minor}"
61+
}
62+
63+
# output variables
64+
libc=
65+
cc=
66+
cxx=
67+
qemu=
68+
69+
# select toolchain information
70+
compiler_suffix="${target//-/_}"
71+
cc_var="CC_${compiler_suffix}"
72+
cxx_var="CXX_${compiler_suffix}"
73+
cc_regex=
74+
case "${target}" in
75+
*-*-android*)
76+
cc_regex=".* clang version ([0-9]+.[0-9]+.[0-9]+) .*"
77+
;;
78+
*-*-*-musl*)
79+
cc_regex=".*gcc \(GCC\) ([0-9]+.[0-9]+.[0-9]+).*"
80+
;;
81+
*-*-linux-gnu*)
82+
cc_regex=".*gcc \(.*\) ([0-9]+.[0-9]+.[0-9]+).*"
83+
;;
84+
*-*-windows-gnu*)
85+
# MinGW only reports major/minor versions, and can
86+
# have a -posix or -win32 suffix, eg: 7.5-posix
87+
cc_regex=".*gcc.* \(GCC\) ([0-9]+.[0-9]+).*"
88+
;;
89+
*-*-freebsd)
90+
cc_regex=".*gcc \(GCC\) ([0-9]+.[0-9]+.[0-9]+).*"
91+
;;
92+
*-*-netbsd)
93+
cc_regex=".*gcc \(.*\) ([0-9]+.[0-9]+.[0-9]+).*"
94+
;;
95+
*-none-*)
96+
cc_regex=".*gcc \(.*\) ([0-9]+.[0-9]+.[0-9]+).*"
97+
;;
98+
*)
99+
echo "TODO: Currently unsupported" 1>&2
100+
exit 1
101+
;;
102+
esac
103+
104+
# select qemu arch
105+
qarch="${arch}"
106+
case "${arch}" in
107+
arm*)
108+
qarch="arm"
109+
;;
110+
i*86)
111+
qarch="i386"
112+
;;
113+
powerpc)
114+
qarch="ppc"
115+
;;
116+
powerpc64)
117+
qarch="ppc64"
118+
;;
119+
powerpc64le)
120+
if [ "${CROSS_RUNNER}" = "qemu-user" ]; then
121+
qarch="ppc64le"
122+
else
123+
qarch="ppc64"
124+
fi
125+
;;
126+
riscv64*)
127+
qarch="riscv64"
128+
;;
129+
esac
130+
qemu_regex="qemu-${qarch} version ([0-9]+.[0-9]+.[0-9]+).*"
131+
132+
# evaluate our toolchain info
133+
cc_bin=
134+
cxx_bin=
135+
case "${target}" in
136+
i*86-unknown-linux-gnu | x86_64-unknown-linux-gnu)
137+
cc_bin="gcc"
138+
cxx_bin="g++"
139+
;;
140+
thumb*-none-eabi* | arm*-none-eabi*)
141+
# the ARM/THUMB targets don't have a CC_${compiler_suffix}
142+
cc_bin=arm-none-eabi-gcc
143+
cxx_bin=arm-none-eabi-g++
144+
;;
145+
*)
146+
cc_bin="${!cc_var}"
147+
cxx_bin="${!cxx_var}"
148+
;;
149+
150+
esac
151+
cc=$(extract_regex_version "${cc_bin}" "${cc_regex}" compiler)
152+
if command -v "${cxx_bin}" &>/dev/null; then
153+
# test we can compile a c++ program that requires the c++ stdlib
154+
cat <<EOT >> main.cc
155+
#include <iostream>
156+
int main() {
157+
std::cout << "Testing this" << std::endl;
158+
}
159+
EOT
160+
cxx_flags=()
161+
if [[ "${target}" == *-none-* ]]; then
162+
cxx_flags=("${cxx_flags[@]}" "-nostartfiles")
163+
fi
164+
if "${cxx_bin}" "${cxx_flags[@]}" main.cc >/dev/null 2>&1; then
165+
cxx=1
166+
fi
167+
fi
168+
169+
case "${target}" in
170+
*-*-android*)
171+
libc="${cc}"
172+
;;
173+
*-*-*-musl*)
174+
toolchain_prefix="${!cc_var//-gcc/}"
175+
libdir="/usr/local/${toolchain_prefix}/lib"
176+
libc_regex=".*Version ([0-9]+.[0-9]+.[0-9]+).*"
177+
if [[ "${arch}" = i[3-7]86 ]] || [ "${arch}" == x86_64 ]; then
178+
libc_cmd="${libdir}/libc.so"
179+
else
180+
libc_cmd="qemu-${qarch} ${libdir}/libc.so"
181+
if ! command -v "qemu-${qarch}" &>/dev/null; then
182+
echo "Unable to get qemu version for ${target}: qemu not found." 1>&2
183+
exit 1
184+
fi
185+
fi
186+
libc=$(extract_regex_version "${libc_cmd}" "${libc_regex}" libc)
187+
;;
188+
arm-unknown-linux-gnueabihf)
189+
# this is for crosstool-ng-based images with glibc
190+
libdir="/x-tools/${target}/${target}/sysroot/lib/"
191+
libc=$(max_glibc_version "${libdir}")
192+
;;
193+
i*86-unknown-linux-gnu)
194+
libdir="/lib/x86_64-linux-gnu/"
195+
libc=$(max_glibc_version "${libdir}")
196+
;;
197+
x86_64-unknown-linux-gnu)
198+
libdir="/lib64/"
199+
libc=$(max_glibc_version "${libdir}")
200+
;;
201+
*-*-linux-gnu*)
202+
toolchain_prefix="${!cc_var//-gcc/}"
203+
libdir="/usr/${toolchain_prefix}/lib"
204+
libc=$(max_glibc_version "${libdir}")
205+
;;
206+
*-*-windows-gnu)
207+
# no libc, intentionally omitted.
208+
;;
209+
*-*-freebsd)
210+
# we write the FreeBSD version to /opt/freebsd-version
211+
# the symbol versioning can be found here:
212+
# https://wiki.freebsd.org/SymbolVersioning
213+
version=$(cat /opt/freebsd-version)
214+
if [[ "${version}" =~ ([0-9]+)\.([0-9]+)" ("[A-Za-z]+")" ]]; then
215+
major_version="${BASH_REMATCH[1]}"
216+
minor_version="${BASH_REMATCH[2]}"
217+
case "${major_version}" in
218+
7)
219+
libc="1.0"
220+
;;
221+
8)
222+
libc="1.1"
223+
;;
224+
9)
225+
libc="1.2"
226+
;;
227+
10)
228+
libc="1.3"
229+
;;
230+
11)
231+
libc="1.4"
232+
;;
233+
12)
234+
libc="1.5"
235+
;;
236+
13)
237+
libc="1.6"
238+
;;
239+
*)
240+
echo "Invalid FreeBSD version, got ${major_version}.${minor_version}." 1>&2
241+
exit 1
242+
;;
243+
esac
244+
else
245+
echo "Unable to get libc version for ${target}: invalid FreeBSD release found." 1>&2
246+
exit 1
247+
fi
248+
;;
249+
*-*-netbsd)
250+
# We can read the NetBSD version from the libc symbols.
251+
# The output follows:
252+
# NetBSD 0x00000004 IDENT 902000000 (9.2.0)
253+
libdir="/usr/local/${target}/lib"
254+
version=$(readelf -a "${libdir}"/libc.so | grep NetBSD | head -n 1)
255+
if [[ "${version}" =~ .+" ("([0-9]+)"."([0-9]+)"."([0-9]+)")" ]]; then
256+
major_version="${BASH_REMATCH[1]}"
257+
minor_version="${BASH_REMATCH[2]}"
258+
patch_version="${BASH_REMATCH[3]}"
259+
libc="${major_version}.${minor_version}.${patch_version}"
260+
else
261+
echo "Unable to get libc version for ${target}: invalid NetBSD release found." 1>&2
262+
exit 1
263+
fi
264+
;;
265+
thumb*-none-eabi* | arm*-none-eabi*)
266+
# newlib kinda sucks. just query for the install package
267+
pkg=$(dpkg --get-selections | grep -v deinstall | grep newlib | head -n 1)
268+
pkg=$(echo "${pkg}" | cut -f 1)
269+
version=$(dpkg-query --showformat='${Version}' --show "${pkg}")
270+
if [[ "${version}" =~ ([0-9]+)"."([0-9]+)"."([0-9]+)"+".* ]]; then
271+
major_version="${BASH_REMATCH[1]}"
272+
minor_version="${BASH_REMATCH[2]}"
273+
patch_version="${BASH_REMATCH[3]}"
274+
libc="${major_version}.${minor_version}.${patch_version}"
275+
else
276+
echo "Unable to get libc version for ${target}: invalid NetBSD release found." 1>&2
277+
exit 1
278+
fi
279+
;;
280+
*)
281+
echo "TODO: Currently unsupported" 1>&2
282+
exit 1
283+
;;
284+
esac
285+
286+
if command -v "qemu-${qarch}" &>/dev/null; then
287+
qemu=$(extract_regex_version "qemu-${qarch}" "${qemu_regex}" qemu)
288+
fi
289+
290+
# format our output
291+
printf "| %-36s |" "\`${target}\`"
292+
if [ "$libc" != "" ]; then
293+
printf " %-6s |" "${libc}"
294+
else
295+
printf " N/A |"
296+
fi
297+
if [ "$cc" != "" ]; then
298+
printf " %-7s |" "${cc}"
299+
else
300+
printf " N/A |"
301+
fi
302+
if [ "$cxx" != "" ]; then
303+
printf " ✓ |"
304+
else
305+
printf " |"
306+
fi
307+
if [ "$qemu" != "" ]; then
308+
printf " %-5s |" "${qemu}"
309+
else
310+
printf " N/A |"
311+
fi
312+
printf "\n"

‎ci/extract_target_info.sh‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
#!/usr/bin/env bash
2+
#
3+
# this script can be customized with the following env vars:
4+
# CROSS_IMAGE: defaults to `ghcr.io/cross-rs`
5+
# CROSS_CONTAINER_ENGINE: defaults to `docker` or `podman`
6+
# CROSS_IMAGE_VERSION: defaults to `main`.
7+
#
8+
# if no arguments are provided, this script will process all
9+
# images. you can extract target info for specific targets
10+
# by providing them as arguments after the script, for example,
11+
# `./extract_target_info.sh i686-linux-android`.
12+
#
13+
# the output will be similar to the following:
14+
# | `aarch64-unknown-linux-musl` | 1.2.0 | 9.2.0 | ✓ | 5.1.0 |
15+
# | `i686-linux-android` | 9.0.8 | 9.0.8 | ✓ | 5.1.0 |
16+
# | `i686-unknown-linux-musl` | 1.2.0 | 9.2.0 | ✓ | N/A |
17+
# ...
18+
#
19+
# in short, it recreates the table except for the test section in README.md.
20+
21+
# shellcheck disable=SC2207
22+
23+
set -eo pipefail
24+
25+
scriptdir=$(dirname "${BASH_SOURCE[0]}")
26+
scriptdir=$(realpath "${scriptdir}")
27+
project_dir=$(dirname "${scriptdir}")
28+
29+
if [[ -z "$CROSS_IMAGE" ]]; then
30+
CROSS_IMAGE="ghcr.io/cross-rs"
31+
fi
32+
if [[ -z "$CROSS_CONTAINER_ENGINE" ]]; then
33+
if command -v "docker" &>/dev/null; then
34+
CROSS_CONTAINER_ENGINE="docker"
35+
elif command -v "podman" &>/dev/null; then
36+
CROSS_CONTAINER_ENGINE="podman"
37+
else
38+
echo "Unable to find suitable container engine." 1>&2
39+
exit 1
40+
fi
41+
fi
42+
if [[ -z "$CROSS_IMAGE_VERSION" ]]; then
43+
CROSS_IMAGE_VERSION="main"
44+
fi
45+
46+
pull() {
47+
"${CROSS_CONTAINER_ENGINE}" pull "${1}"
48+
}
49+
50+
run() {
51+
TARGET="${1}" "${CROSS_CONTAINER_ENGINE}" run --rm -e TARGET \
52+
-v "${scriptdir}:/ci:ro" "${2}" \
53+
bash -c "/ci/extract_image_info.sh"
54+
}
55+
56+
# parse our CI list, so updating our CI automatically updates our target list.
57+
if [[ $# -eq "0" ]]; then
58+
ci="${project_dir}"/.github/workflows/ci.yml
59+
matrix=$(yq '."jobs"."generate-matrix"."steps".0."env"."matrix"' "${ci}")
60+
targets=($(yq '.[]."target"' <<< "${matrix}"))
61+
else
62+
targets=("${@}")
63+
fi
64+
for target in "${targets[@]}"; do
65+
# can't do MSVC, Darwin, or iOS images.
66+
case "${target}" in
67+
*-msvc | *-darwin | *-apple-ios)
68+
continue
69+
;;
70+
esac
71+
72+
image="${CROSS_IMAGE}"/"${target}":"${CROSS_IMAGE_VERSION}"
73+
if [[ -z "$DEBUG" ]]; then
74+
pull "${image}" >/dev/null 2>&1
75+
run "${target}" "${image}" 2>/dev/null
76+
else
77+
pull "${image}"
78+
run "${target}" "${image}"
79+
fi
80+
done

0 commit comments

Comments
 (0)