Integration Testing: Start performing full rattler-build integration tests

[schuylermartin45]

Overview

I've gotten some requests to perform actual (non dry-run) builds with rattler-build. To start, we will need to select a small subset of the recipe files in the test data set. I'm think maybe 10-30 files found in the conda-forge and AnacondaRecipes folders.

We should maintain a file in Conda Recipe Manager that contains a list of the files we would like to run a full build test on. Maybe something like:
integration_build_test_config.json

{
  "v0_test_files": [
      "/anaconda_recipes_01/types-toml/recipe/meta.yaml",
      "/conda_forge_recipes_01/click-params-feedstock/recipe/meta.yaml"
  ]
}

This new test workflow should:

• Initialize one (or more, depending on how this scales) GitHub Action containers with a development version of Conda Recipe Manager and the latest version of rattler-build from the conda-forge channel, using conda.
• Convert the V0 file to a V1 file using crm convert
• Run a full (NOT a dry-run) rattler-build on any successfully converted files.
• Utilize parallelism where possible to speed up the tests.
• Generate a final summary report should track all meaningful statistics like:
  • How many recipes failed to convert? (Exact number and percentage)
  • How many recipes were able to be built with rattler-build? (Exact number and percentage)
  • How many recipes failed to build? (Exact number and percentage)

Current Integration Test Status

Here is our current integration test: https://github.com/conda-incubator/conda-recipe-manager/blob/main/.github/workflows/integration_tests.yaml

Here is an outline of what currently happens (for 1 of the 6 containers initialized):

• The container sets up conda, CRM, rattler-build, and all the other required tools
• The container performs a sparse checkout action of ~2,000 recipe files in one of the 6 test folders from the test data repo.
• Then those V0 recipes are bulk-converted with crm convert. Statistics about this process are written to a JSON file and dumped to the console.
• If a recipe was converted successfully (without exceptions being thrown), crm convert will leave behind a recipe.yaml file in the same directory as the meta.yaml file.
• All test files that were successfully converted then perform a dry-run build of rattler-build. This is done with the crm rattler-bulk-build script. Statistics about this process are written to a JSON file and dumped to the console.
• The JSON reports are then uploaded as artifacts with the artifact upload action.

When all 6 containers complete, one last phase begins. A new container downloads all the JSON statistics artifacts generated by the 6 tests. There are 12 artifacts in total, one for the convert phase, and one for the rattler-build phase. This script aggregates and averages the statistics in those artifacts and provides a final report (an example can be found here).

Additional Notes

• As we scale this problem, we may need to ask for additional compute resources. I worry that the free GitHub containers will not scale with this request. We'll have to make a proposal to Anaconda/the conda community if it comes to that.
• There will be a follow-up task in the future to validate the artifacts generated by these automated builds. Wolf might have started this work. See the attached sub-ticket for more details.
• rattler-build should be spinning up multiple threads per build. Be aware of this when dispatching build jobs. I don't think the threading model used in crm rattler-bulk-build will work for full-builds, but feel free to give it a try.