Merge pull request #864 from vaikas/test-pipeline

Add Test pipelines

.github/workflows/melange-test-pipelines.yaml

@@ -0,0 +1,73 @@
+name: Test melange test command
+
+on:
+  pull_request:
+  push:
+    branches:
+      - 'main'
+
+jobs:
+  build-melange:
+    name: Build melange and add to artifact cache
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
+
+      - uses: actions/setup-go@93397bea11091df50f3d7e59dc26a7711a8bcfbe # v4.1.0
+        with:
+          go-version: '1.21'
+          check-latest: true
+
+      - name: build
+        run: |
+          make melange
+
+      - uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32 # v3.1.3
+        with:
+          name: melange-${{github.run_id}}
+          path: ${{github.workspace}}/melange
+          retention-days: 1
+
+  test-packages:
+    name: Test packages
+    needs:
+      - build-melange
+    # TODO: Set up a larger runner for this.
+    runs-on: ubuntu-latest
+
+    # This is a list of packages which we want to test against.
+    # Feel free to add additional packages to this matrix which exercise
+    # Melange `test` in new ways (e.g. new pipelines, etc.)
+    # Each test file is of the form <package-name>-test.yaml and gets
+    # constructed from the package name.
+    strategy:
+      fail-fast: false
+      matrix:
+        package:
+          - php-8.2-msgpack
+          - py3-pandas
+
+    steps:
+      # Grab the melange we uploaded above, and install it.
+      - uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a # v3.0.2
+        with:
+          name: melange-${{github.run_id}}
+          path: ${{github.workspace}}/.melange-dir
+
+      - run: |
+          sudo mv ${{github.workspace}}/.melange-dir/melange /usr/bin/melange
+          sudo chmod a+x /usr/bin/melange
+          melange version
+
+      - run: |
+          sudo apt-get -y install bubblewrap
+
+      # Make sure we have our tests files here.
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
+
+      - run: |
+          testfile="${{matrix.package}}-test.yaml"
+          echo "Testing $testfile"
+
+          melange test --arch x86_64 --source-dir ./e2e-tests/test-fixtures ./e2e-tests/$testfile ${{matrix.package}} --repository-append https://packages.wolfi.dev/os --keyring-append https://packages.wolfi.dev/os/wolfi-signing.rsa.pub

docs/TESTING.md

@@ -0,0 +1,209 @@
+# Testing
+
+Melange provides an ability to test packages with a `test` command. Tests are
+implemented using the same `pipeline`, and `subpackages` centric ways to define
+tests, so it should be very familiar continuation from `build` to `test`.
+
+## Overview
+
+The keyword `test:` starts a new test block, and can be embedded either at the
+top level, or inside a subpackage. You can test the 'main' package, and
+subpackages, or any combination of them. Each `test` block has a section for
+specifying the test configuration including the necessary packages, (partly to
+ensure the minimal set of packages, and therefore testing runtime dependencies
+definitions), as well as any environmental variables. This section again, looks
+exactly like a build pipelines, and therefore should feel very familiar.
+
+### Test environment (workspace)
+
+Just like with the build, there is a single shared `workspace` that gets mounted
+as the `CWD` for each of the `test` runs. You can add any test fixtures, for
+example, if you are testing some python packages, you could create `foo-test.py`
+file, and by using a `--source-dir` pointing to the directory, the files in that
+directory will then be available for your tests in the current directory. For
+example, say you are testing `py3-pandas` package, and would like to exercise
+some data transformations, you could create a file
+`/tmp/testfiles/pandas-test.py`:
+
+```python
+import numpy as np
+import pandas as pd
+s = pd.Series([1,3,5,np.nan, 6, 8])
+dates = pd.date_range("20130101", periods=6)
+df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list("ABCD"))
+```
+
+Then you could make sure this file ends up in your workspace as
+`./pandas-test.py` by specifying `--source-dir /tmp/testfiles`
+
+### Execution environment (guest)
+
+Unlike a `build` guest, each `test` will get their own "fresh" container built
+using apko that only contains the Package Under Test (PUT), that is defaulted to
+each container depending on the context, as well as any additional packages
+specified in the `test.environment.contents.packages`. For example, the "main"
+`test` pipeline will get the main package added by default. For subpackages, the
+subpackage that has the `test` block gets added by default.
+
+For example, with a test environment like this for the main package
+(for example, `py3-pandas`):
+```yaml
+package:
+  name: py3-pandas
+# Stuff omitted for readability
+test:
+  environment:
+    contents:
+      packages:
+        - busybox
+        - python-3
+```
+
+Will get a test execution containing the following packages (and their
+transitive dependencies as per apk solver):
+ * py3-pandas
+ * busybox
+ * python-3
+
+And a subpackage test like this:
+```yaml
+package:
+  name: php-8.2-msgpack
+# Again stuff omitted for readability
+subpackages:
+  - name: ${{package.name}}-dev
+    description: PHP 8.2 msgpack development headers
+    test:
+      environment:
+        contents:
+          packages:
+            - busybox
+            - wolfi-base
+      pipeline:
+        - runs: |
+            # Just make sure the expected define is in the expected file
+            # location.
+            grep PHP_MSGPACK_VERSION /usr/include/php/ext/msgpack/php_msgpack.h
+```
+
+This will get a test execution containing the following packages (and their
+transitive dependencies as per apk solver):
+ * php-8.2-msgpack
+ * busybox
+ * wolfi-base
+
+### Execution environment, repo configuration
+
+Because we use the apk solver, and apko to build the guest containers, it's easy
+to configure if you are testing either local, or remote, or combination of the
+packages by simply configuring the appropriate repos. By using these, you can
+easily add tests to existing packages without having to rebuild them, or fetch
+them directly and juggle them. You can also iterate on the package and tests at
+the same time, by rebuilding your local package (and ofc adding the repo
+configuration for it). Because we rely on the normal apk solver rules for
+figuring out which package to install into the guest context, it is flexible
+enough to test whatever combinations of packages that you want to test with.
+
+As discussed above, you can specify which packages are tested, as well as which
+packages a particular test needs to perform the tests (for example, you could
+try to `curl` a URL to test a package, so that would require curl). You can
+configure these with `--keyring-append` as well as `--repository-append`
+variables. As usual by default the "highest" one wins, so if you are testing
+local changes to a package, you can build a local version, and by bumping the
+epoch it will become the PUT, or you can configure/test PUT dependencies this
+way (build a local copy, and it will be picked up by APK resolver). This is very
+similar to how we build/test images with local versions of packages, so again,
+this should feel very natural.
+
+### Where to define the tests?
+
+So, this is one open question, but the short answer is that you can add these
+tests inline with the existing yaml files that specify the build, OR you can
+define them in alternate location. Because of the way the melange configuration
+parsing currently works, you may need to add some "placeholders" to satisfy
+the configuration parser. For example, here's a simple test file that I've been
+using to test things that has some "placeholder" fields that are not really
+actually used, but will allow one to decouple the test/build file if that's the
+direction we want to go:
+
+```yaml
+package:
+  name: php-8.2-msgpack
+  version: 2.2.0
+  epoch: 0
+  description: "Tests for PHP extension msgpack"
+  copyright:
+    - license: BSD-3-Clause
+
+# This is mandatory, so just put an empty one there. Otherwise, config parsing
+# will fail.
+pipeline:
+
+test:
+  environment:
+    contents:
+      packages:
+        - wolfi-base
+        - apk-tools
+  pipeline:
+    - runs: |
+        # Stuff goes here.
+```
+
+## Using pipelines
+
+Not surprisingly you can also use predefined pipelines, just like in the build
+step by using `uses:` instead of `runs:`. You can specify the location of the
+predefined pipelines using the `--pipeline-dir` to point to the directory where
+the custom pipelines are located.
+
+## Specifying package to test / reusing tests
+
+You can leave out the package name from the command line if you want, in which
+case the PUT is pulled from the package.Name. However, for versioned packages,
+say for example, php-8.1, php-8.2, php-8.3, it is beneficial to reuse some of
+the tests. In those cases, you can specify the testfile and also which package
+to use for testing by providing a second argument to the `test` command that is
+the name of the package used for the tests.
+
+Lastly, if you want to test a specific version of the package, you can specify
+the constraint in the argument. For example:
+
+ * Use package.Name
+
+ ```shell
+ melange test ./testfile.yaml
+ ```
+
+ * Use above testfile, but a different package to run tests against
+
+ ```shell
+ melange test ./testfile.yaml mypackage
+ ```
+
+ * Use above testfile, but specify a particular version of the package
+
+ ```shell
+ melange test ./testfile.yaml mypackage=2.2.0-r2
+ ```
+
+## Full example
+
+Here's a full example invocation, where I'm testing with my local mac, so just
+testing the aarch64 (hence `--arch aarch64`` flag), and I'm pulling in the
+abovementioned `py3-pandas-test.yaml` file as specified from the current
+directory, and I do want to test the py3-pandas package (second argument), and
+the keyring/repository append flags pull in my local changes, so that I can
+iterate on package building, as well as testing at the same time. If you are
+only writing tests for existing packages, you could drop the "local"
+keyring/repository, and then only released packages would be pulled in for
+testing.
+
+```shell
+melange test ./py3-pandas-test.yaml py3-pandas \
+--source-dir /tmp/testfiles --arch aarch64 \
+--keyring-append /Users/vaikas/projects/go/src/github.com/wolfi-dev/os/local-melange.rsa.pub \
+--repository-append /Users/vaikas/projects/go/src/github.com/wolfi-dev/os/packages \
+--repository-append https://packages.wolfi.dev/os \
+--keyring-append https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
+```

docs/md/melange.md

@@ -30,6 +30,7 @@ toc: true
 * [melange query](/docs/md/melange_query.md)	 - Query a Melange YAML file for information
 * [melange sign](/docs/md/melange_sign.md)	 - Sign an APK package
 * [melange sign-index](/docs/md/melange_sign-index.md)	 - Sign an APK index
+* [melange test](/docs/md/melange_test.md)	 - Test a package with a YAML configuration file
 * [melange update-cache](/docs/md/melange_update-cache.md)	 - Update a source artifact cache
 * [melange version](/docs/md/melange_version.md)	 - Prints the version
 

docs/md/melange_test.md

@@ -0,0 +1,53 @@
+---
+title: "melange test"
+slug: melange_test
+url: /docs/md/melange_test.md
+draft: false
+images: []
+type: "article"
+toc: true
+---
+## melange test
+
+Test a package with a YAML configuration file
+
+### Synopsis
+
+Test a package from a YAML configuration file containing a test pipeline.
+
+```
+melange test [flags]
+```
+
+### Examples
+
+```
+  melange test <test.yaml> [package-name]
+```
+
+### Options
+
+```
+      --apk-cache-dir string        directory used for cached apk packages (default is system-defined cache directory)
+      --arch strings                architectures to build for (e.g., x86_64,ppc64le,arm64) -- default is all, unless specified in config
+      --cache-dir string            directory used for cached inputs
+      --cache-source string         directory or bucket used for preloading the cache
+      --debug                       enables debug logging of test pipelines (sets -x for steps)
+      --debug-runner                when enabled, the builder pod will persist after the build succeeds or fails
+      --guest-dir string            directory used for the build environment guest
+  -h, --help                        help for test
+  -k, --keyring-append strings      path to extra keys to include in the build environment keyring
+      --log-policy strings          logging policy to use (default [builtin:stderr])
+      --overlay-binsh string        use specified file as /bin/sh overlay in build environment
+      --pipeline-dirs strings       directories used to extend defined built-in pipelines
+  -r, --repository-append strings   path to extra repositories to include in the build environment
+      --runner string               which runner to use to enable running commands, default is based on your platform. Options are ["bubblewrap" "docker" "lima" "kubernetes"] (default "bubblewrap")
+      --source-dir string           directory used for included sources
+      --test-option strings         build options to enable
+      --workspace-dir string        directory used for the workspace at /home/build
+```
+
+### SEE ALSO
+
+* [melange](/docs/md/melange.md)	 - 
+

e2e-tests/php-8.2-msgpack-test.yaml

@@ -0,0 +1,52 @@
+# This is an example test file that shows how 'melange test' works.
+# It has been pulled into its own file, to try to clearly show what the
+# test file looks like.
+# Note, that these tests can also be baked into the package file itself.
+#
+package:
+  name: php-8.2-msgpack
+  version: 2.2.0
+  epoch: 0
+  description: "Tests for PHP extension msgpack"
+  copyright:
+    - license: BSD-3-Clause
+
+# This is mandatory, so just put an empty one there. Otherwise, config parsing
+# will fail.
+pipeline:
+
+test:
+  environment:
+    contents:
+      packages:
+        - wolfi-base
+  pipeline:
+    - runs: |
+        # Make sure msgpack is correctly loaded and listed by modules
+        php -m | grep msgpack
+
+subpackages:
+  - name: ${{package.name}}-config
+    description: PHP 8.2 msgpack tests
+    test:
+      environment:
+        contents:
+          packages:
+            - wolfi-base
+            - busybox
+      pipeline:
+        - runs: |
+            grep msgpack.so /etc/php/conf.d/msgpack.ini
+
+  - name: ${{package.name}}-dev
+    description: PHP 8.2 msgpack development headers tests
+    test:
+      environment:
+        contents:
+          packages:
+            - wolfi-base
+            - busybox
+      pipeline:
+        - runs: |
+            # Just make sure this define is there.
+            grep PHP_MSGPACK_VERSION /usr/include/php/ext/msgpack/php_msgpack.h