Improve support for different Android versions.

We provide the following arguments to configure the Android version:
- `ANDROID_NDK`
- `ANDROID_SDK` (note that this is dependent on `ANDROID_VERSION)
- `ANDROID_VERSION`
- `ANDROID_SYSTEM_COMPLETE`: do a complete Android build

Next, we've improved the removal of unittests during the build process, to ensure fast builds while maintaining compatibility with various newer Android versions. To do this, we've implemented a Python library and a script. The Python library contains a complete parser (correctly parses all valid input) for Soong blueprint files, using an LALR grammar, and a rudimentary parser for Makefiles.

The Soong parser removes any `gtest` dependencies, as well as any subdirectories or scope names containing `test`. For example:

```
cc_library {
    name: "lib",
    srcs: ["lib.cc",],
}
cc_test {
    name: "test",
    defaults: ["target"],
    srcs: ["test.cc"],
}
```

Will become:

```
cc_library {
    name: "lib",
    srcs: ["lib.cc",],
}
```

The Makefile parser first splits the file based on conditional directives (`ifeq`, `endif`, etc.) to ensure any subsequent processing doesn't lead to unbalanced directives. Next, we split the text within each directive based on comment sections used in the Android source tree. For example:

```
test_tags := tests

include $(call subdir,$(LOCAL_PATH))

c_flags := \
  -g \
  -Wall
```

We can therefore remove the `Benchmarks` and `Unit tests` sections without removing the `Other section`.

The Python library is reasonably performant (it adds no noticeable overhead to the build process) and is compact (in total, < 60KB). Also, it is much more resilient than a series of `sed` scripts.

Finally, extensive unittests have been added for the Python library, for both code linting (`flake8`) and unittests (via `tox`). Since we cannot assume the presence of Python on the host machine, the tests can be optionally enabled via the `--python` flag (or `PYTHON` environment variable, to hook into the git hooks), and custom overrides for the `flake8` and `tox` commands are provided (since the user may wish to specify a specific Python version, such as `python3.7 -m flake8`).

Author: Alexhuszagh

.changes/1023.json

@@ -0,0 +1,4 @@
+{
+    "description": "support different Android NDK, API, and Android versions using Docker build args.",
+    "type": "added"
+}

.gitignore

@@ -4,3 +4,12 @@
 **/*.log
 /cargo-timing*.html
 CHANGELOG.md.draft
+
+# python stuff
+__pycache__/
+.pytest_cache/
+*.py[cod]
+*$py.class
+*.egg-info/
+*.egg
+.tox

Cargo.lock

@@ -0,0 +1,11 @@
+# don't copy any of the python artifacts to the docker context
+__pycache__/
+.pytest_cache/
+*.py[cod]
+*$py.class
+**/*.egg-info/
+*.egg
+.tox
+
+# also skip our test suite
+android/tests/

docker/.dockerignore

@@ -10,31 +10,52 @@ RUN /cmake.sh
 COPY xargo.sh /
 RUN /xargo.sh
 
-COPY android-ndk.sh /
-RUN /android-ndk.sh arm64 28
-ENV PATH=$PATH:/android-ndk/bin
-
-COPY android-system.sh /
-RUN /android-system.sh arm64
-
 COPY qemu.sh /
 RUN /qemu.sh aarch64
 
-RUN cp /android-ndk/sysroot/usr/lib/aarch64-linux-android/28/libz.so /system/lib/
+ARG ANDROID_NDK=r21d
+ARG ANDROID_SDK=28
+ARG ANDROID_VERSION=9.0.0_r1
+ARG ANDROID_SYSTEM_COMPLETE=0
+ARG PYTHON_TMPDIR=/tmp/android
 
-COPY android-runner /
+COPY android-ndk.sh /
+RUN /android-ndk.sh arm64
+ENV PATH=$PATH:/android-ndk/bin
 
-# Libz is distributed in the android ndk, but for some unknown reason it is not
-# found in the build process of some crates, so we explicit set the DEP_Z_ROOT
-ENV CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=aarch64-linux-android-gcc \
-    CARGO_TARGET_AARCH64_LINUX_ANDROID_RUNNER="/android-runner aarch64" \
-    CC_aarch64_linux_android=aarch64-linux-android-gcc \
-    CXX_aarch64_linux_android=aarch64-linux-android-g++ \
-    BINDGEN_EXTRA_CLANG_ARGS_aarch64_linux_android="--sysroot=/android-ndk/sysroot" \
-    DEP_Z_INCLUDE=/android-ndk/sysroot/usr/include/ \
-    RUST_TEST_THREADS=1 \
-    HOME=/tmp/ \
-    TMPDIR=/tmp/ \
-    ANDROID_DATA=/ \
-    ANDROID_DNS_MODE=local \
-    ANDROID_ROOT=/system
+# TODO(ahuszagh) Restore...
+COPY android-system.sh /
+RUN mkdir -p $PYTHON_TMPDIR
+COPY android $PYTHON_TMPDIR
+#RUN /android-system.sh arm64
+#
+#COPY android-symlink.sh /
+#RUN /android-symlink.sh aarch64 aarch64-linux-android
+#
+#COPY android-runner /
+#
+## Libz is distributed in the android ndk, but for some unknown reason it is not
+## found in the build process of some crates, so we explicit set the DEP_Z_ROOT
+#ENV CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=aarch64-linux-android-gcc \
+#    CARGO_TARGET_AARCH64_LINUX_ANDROID_RUNNER="/android-runner aarch64" \
+#    AR_aarch64_linux_android=aarch64-linux-android-ar \
+#    AS_aarch64_linux_android=aarch64-linux-android-as \
+#    CC_aarch64_linux_android=aarch64-linux-android-gcc \
+#    CXX_aarch64_linux_android=aarch64-linux-android-g++ \
+#    LD_aarch64_linux_android=aarch64-linux-android-ld \
+#    NM_aarch64_linux_android=aarch64-linux-android-nm \
+#    OBJCOPY_aarch64_linux_android=aarch64-linux-android-objcopy \
+#    OBJDUMP_aarch64_linux_android=aarch64-linux-android-objdump \
+#    RANLIB_aarch64_linux_android=aarch64-linux-android-ranlib \
+#    READELF_aarch64_linux_android=aarch64-linux-android-readelf \
+#    SIZE_aarch64_linux_android=aarch64-linux-android-size \
+#    STRINGS_aarch64_linux_android=aarch64-linux-android-strings \
+#    STRIP_aarch64_linux_android=aarch64-linux-android-strip \
+#    BINDGEN_EXTRA_CLANG_ARGS_aarch64_linux_android="--sysroot=/android-ndk/sysroot" \
+#    DEP_Z_INCLUDE=/android-ndk/sysroot/usr/include/ \
+#    RUST_TEST_THREADS=1 \
+#    HOME=/tmp/ \
+#    TMPDIR=/tmp/ \
+#    ANDROID_DATA=/ \
+#    ANDROID_DNS_MODE=local \
+#    ANDROID_ROOT=/system

docker/Dockerfile.aarch64-linux-android

@@ -0,0 +1,57 @@
+FROM ubuntu:20.04
+ARG DEBIAN_FRONTEND=noninteractive
+
+COPY common.sh lib.sh /
+RUN /common.sh
+
+COPY cmake.sh /
+RUN /cmake.sh
+
+COPY xargo.sh /
+RUN /xargo.sh
+
+COPY qemu.sh /
+RUN /qemu.sh aarch64
+
+ARG ANDROID_NDK=r21d
+ARG ANDROID_SDK=28
+ARG ANDROID_VERSION=9.0.0_r1
+ARG ANDROID_SYSTEM_COMPLETE=0
+
+COPY android-ndk.sh /
+RUN /android-ndk.sh arm64
+ENV PATH=$PATH:/android-ndk/bin
+
+COPY android-system.sh remove_android_tests.py /
+RUN /android-system.sh arm64
+
+COPY android-symlink.sh /
+RUN /android-symlink.sh aarch64 aarch64-linux-android
+
+COPY android-runner /
+
+# Libz is distributed in the android ndk, but for some unknown reason it is not
+# found in the build process of some crates, so we explicit set the DEP_Z_ROOT
+ENV CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=aarch64-linux-android-gcc \
+    CARGO_TARGET_AARCH64_LINUX_ANDROID_RUNNER="/android-runner aarch64" \
+    AR_aarch64_linux_android=aarch64-linux-android-ar \
+    AS_aarch64_linux_android=aarch64-linux-android-as \
+    CC_aarch64_linux_android=aarch64-linux-android-gcc \
+    CXX_aarch64_linux_android=aarch64-linux-android-g++ \
+    LD_aarch64_linux_android=aarch64-linux-android-ld \
+    NM_aarch64_linux_android=aarch64-linux-android-nm \
+    OBJCOPY_aarch64_linux_android=aarch64-linux-android-objcopy \
+    OBJDUMP_aarch64_linux_android=aarch64-linux-android-objdump \
+    RANLIB_aarch64_linux_android=aarch64-linux-android-ranlib \
+    READELF_aarch64_linux_android=aarch64-linux-android-readelf \
+    SIZE_aarch64_linux_android=aarch64-linux-android-size \
+    STRINGS_aarch64_linux_android=aarch64-linux-android-strings \
+    STRIP_aarch64_linux_android=aarch64-linux-android-strip \
+    BINDGEN_EXTRA_CLANG_ARGS_aarch64_linux_android="--sysroot=/android-ndk/sysroot" \
+    DEP_Z_INCLUDE=/android-ndk/sysroot/usr/include/ \
+    RUST_TEST_THREADS=1 \
+    HOME=/tmp/ \
+    TMPDIR=/tmp/ \
+    ANDROID_DATA=/ \
+    ANDROID_DNS_MODE=local \
+    ANDROID_ROOT=/system

docker/Dockerfile.aarch64-linux-android.bak

@@ -10,26 +10,43 @@ RUN /cmake.sh
 COPY xargo.sh /
 RUN /xargo.sh
 
+COPY qemu.sh /
+RUN /qemu.sh arm
+
+ARG ANDROID_NDK=r21d
+ARG ANDROID_SDK=28
+ARG ANDROID_VERSION=9.0.0_r1
+ARG ANDROID_SYSTEM_COMPLETE=0
+
 COPY android-ndk.sh /
-RUN /android-ndk.sh arm 28
+RUN /android-ndk.sh arm
 ENV PATH=$PATH:/android-ndk/bin
 
-COPY android-system.sh /
+COPY android-system.sh remove_android_tests.py /
 RUN /android-system.sh arm
 
-COPY qemu.sh /
-RUN /qemu.sh arm
-
-RUN cp /android-ndk/sysroot/usr/lib/arm-linux-androideabi/28/libz.so /system/lib/
+COPY android-symlink.sh /
+RUN /android-symlink.sh arm arm-linux-androideabi
 
 COPY android-runner /
 
 # Libz is distributed in the android ndk, but for some unknown reason it is not
 # found in the build process of some crates, so we explicit set the DEP_Z_ROOT
 ENV CARGO_TARGET_ARM_LINUX_ANDROIDEABI_LINKER=arm-linux-androideabi-gcc \
     CARGO_TARGET_ARM_LINUX_ANDROIDEABI_RUNNER="/android-runner arm" \
+    AR_arm_linux_androideabi=arm-linux-androideabi-ar \
+    AS_arm_linux_androideabi=arm-linux-androideabi-as \
     CC_arm_linux_androideabi=arm-linux-androideabi-gcc \
     CXX_arm_linux_androideabi=arm-linux-androideabi-g++ \
+    LD_arm_linux_androideabi=arm-linux-androideabi-ld \
+    NM_arm_linux_androideabi=arm-linux-androideabi-nm \
+    OBJCOPY_arm_linux_androideabi=arm-linux-androideabi-objcopy \
+    OBJDUMP_arm_linux_androideabi=arm-linux-androideabi-objdump \
+    RANLIB_arm_linux_androideabi=arm-linux-androideabi-ranlib \
+    READELF_arm_linux_androideabi=arm-linux-androideabi-readelf \
+    SIZE_arm_linux_androideabi=arm-linux-androideabi-size \
+    STRINGS_arm_linux_androideabi=arm-linux-androideabi-strings \
+    STRIP_arm_linux_androideabi=arm-linux-androideabi-strip \
     BINDGEN_EXTRA_CLANG_ARGS_arm_linux_androideabi="--sysroot=/android-ndk/sysroot" \
     DEP_Z_INCLUDE=/android-ndk/sysroot/usr/include/ \
     RUST_TEST_THREADS=1 \

docker/Dockerfile.arm-linux-androideabi

@@ -10,26 +10,43 @@ RUN /cmake.sh
 COPY xargo.sh /
 RUN /xargo.sh
 
+COPY qemu.sh /
+RUN /qemu.sh arm
+
+ARG ANDROID_NDK=r21d
+ARG ANDROID_SDK=28
+ARG ANDROID_VERSION=9.0.0_r1
+ARG ANDROID_SYSTEM_COMPLETE=0
+
 COPY android-ndk.sh /
-RUN /android-ndk.sh arm 28
+RUN /android-ndk.sh arm
 ENV PATH=$PATH:/android-ndk/bin
 
-COPY android-system.sh /
+COPY android-system.sh remove_android_tests.py /
 RUN /android-system.sh arm
 
-COPY qemu.sh /
-RUN /qemu.sh arm
-
-RUN cp /android-ndk/sysroot/usr/lib/arm-linux-androideabi/28/libz.so /system/lib/
+COPY android-symlink.sh /
+RUN /android-symlink.sh arm arm-linux-androideabi
 
 COPY android-runner /
 
 # Libz is distributed in the android ndk, but for some unknown reason it is not
 # found in the build process of some crates, so we explicit set the DEP_Z_ROOT
 ENV CARGO_TARGET_ARMV7_LINUX_ANDROIDEABI_LINKER=arm-linux-androideabi-gcc \
     CARGO_TARGET_ARMV7_LINUX_ANDROIDEABI_RUNNER="/android-runner arm" \
+    AR_armv7_linux_androideabi=arm-linux-androideabi-ar \
+    AS_armv7_linux_androideabi=arm-linux-androideabi-as \
     CC_armv7_linux_androideabi=arm-linux-androideabi-gcc \
     CXX_armv7_linux_androideabi=arm-linux-androideabi-g++ \
+    LD_armv7_linux_androideabi=arm-linux-androideabi-ld \
+    NM_armv7_linux_androideabi=arm-linux-androideabi-nm \
+    OBJCOPY_armv7_linux_androideabi=arm-linux-androideabi-objcopy \
+    OBJDUMP_armv7_linux_androideabi=arm-linux-androideabi-objdump \
+    RANLIB_armv7_linux_androideabi=arm-linux-androideabi-ranlib \
+    READELF_armv7_linux_androideabi=arm-linux-androideabi-readelf \
+    SIZE_armv7_linux_androideabi=arm-linux-androideabi-size \
+    STRINGS_armv7_linux_androideabi=arm-linux-androideabi-strings \
+    STRIP_armv7_linux_androideabi=arm-linux-androideabi-strip \
     BINDGEN_EXTRA_CLANG_ARGS_armv7_linux_androideabi="--sysroot=/android-ndk/sysroot" \
     DEP_Z_INCLUDE=/android-ndk/sysroot/usr/include/ \
     RUST_TEST_THREADS=1 \

docker/Dockerfile.armv7-linux-androideabi

@@ -10,33 +10,51 @@ RUN /cmake.sh
 COPY xargo.sh /
 RUN /xargo.sh
 
-COPY android-ndk.sh /
-RUN /android-ndk.sh x86 28
-ENV PATH=$PATH:/android-ndk/bin
-
-COPY android-system.sh /
-RUN /android-system.sh x86
-
 # We could supposedly directly run i686 binaries like we do for x86_64, but
 # doing so generates an assertion failure:
 #   ... assertion failed: signal(libc::SIGPIPE, libc::SIG_IGN) != libc::SIG_ERR
 #   ... src/libstd/sys/unix/mod.rs
 #   fatal runtime error: failed to initiate panic, error 5
 #
-# Running with qemu works as expected
+# Running with qemu works as expected. it also ensures that're we're
+# running on a CPU common 32-bit x86 systems.
 COPY qemu.sh /
 RUN /qemu.sh i386
 
-RUN cp /android-ndk/sysroot/usr/lib/i686-linux-android/28/libz.so /system/lib/
+ARG ANDROID_NDK=r21d
+ARG ANDROID_SDK=28
+ARG ANDROID_VERSION=9.0.0_r1
+ARG ANDROID_SYSTEM_COMPLETE=0
+
+COPY android-ndk.sh /
+RUN /android-ndk.sh x86
+ENV PATH=$PATH:/android-ndk/bin
+
+COPY android-system.sh remove_android_tests.py /
+RUN /android-system.sh x86
+
+COPY android-symlink.sh /
+RUN /android-symlink.sh i386 i686-linux-android
 
 COPY android-runner /
 
 # Libz is distributed in the android ndk, but for some unknown reason it is not
 # found in the build process of some crates, so we explicit set the DEP_Z_ROOT
 ENV CARGO_TARGET_I686_LINUX_ANDROID_LINKER=i686-linux-android-gcc \
     CARGO_TARGET_I686_LINUX_ANDROID_RUNNER="/android-runner i686" \
+    AR_i686_linux_android=i686-linux-android-ar \
+    AS_i686_linux_android=i686-linux-android-as \
     CC_i686_linux_android=i686-linux-android-gcc \
     CXX_i686_linux_android=i686-linux-android-g++ \
+    LD_i686_linux_android=i686-linux-android-ld \
+    NM_i686_linux_android=i686-linux-android-nm \
+    OBJCOPY_i686_linux_android=i686-linux-android-objcopy \
+    OBJDUMP_i686_linux_android=i686-linux-android-objdump \
+    RANLIB_i686_linux_android=i686-linux-android-ranlib \
+    READELF_i686_linux_android=i686-linux-android-readelf \
+    SIZE_i686_linux_android=i686-linux-android-size \
+    STRINGS_i686_linux_android=i686-linux-android-strings \
+    STRIP_i686_linux_android=i686-linux-android-strip \
     BINDGEN_EXTRA_CLANG_ARGS_i686_linux_android="--sysroot=/android-ndk/sysroot" \
     DEP_Z_INCLUDE=/android-ndk/sysroot/usr/include/ \
     LIBZ_SYS_STATIC=1 \

docker/Dockerfile.i686-linux-android

@@ -10,17 +10,23 @@ RUN /cmake.sh
 COPY xargo.sh /
 RUN /xargo.sh
 
+COPY qemu.sh /
+RUN /qemu.sh arm
+
+ARG ANDROID_NDK=r21d
+ARG ANDROID_SDK=28
+ARG ANDROID_VERSION=9.0.0_r1
+ARG ANDROID_SYSTEM_COMPLETE=0
+
 COPY android-ndk.sh /
-RUN /android-ndk.sh arm 28
+RUN /android-ndk.sh arm
 ENV PATH=$PATH:/android-ndk/bin
 
-COPY android-system.sh /
+COPY android-system.sh remove_android_tests.py /
 RUN /android-system.sh arm
 
-COPY qemu.sh /
-RUN /qemu.sh arm
-
-RUN cp /android-ndk/sysroot/usr/lib/arm-linux-androideabi/28/libz.so /system/lib/
+COPY android-symlink.sh /
+RUN /android-symlink.sh arm arm-linux-androideabi
 
 COPY android-runner /