🌱 E2E: Image refactor

[lentzi90]

What this PR does / why we need it:

This makes the images configurable through the e2e_conf.yaml.
The images are applied together with the cluster-template, instead of in
the JustBefore code.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

1. Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

TODOs:

• squashed commits
• if necessary:
  • includes documentation
  • adds unit tests

/hold

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[netlify]

✅ Deploy Preview for kubernetes-sigs-cluster-api-openstack ready!

Name	Link
🔨 Latest commit	6d65885
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-cluster-api-openstack/deploys/6836a0cd5c13670008b51f1b
😎 Deploy Preview	https://deploy-preview-2570--kubernetes-sigs-cluster-api-openstack.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign justinsb for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mdbooth]

The way the code is currently written was intended to reduce the number of image creations. Before we used ORC for image downloading we downloaded the images once during initialisation, and effectively referenced them from the Machine specs. When we added ORC for images I was trying hard not to regress the runtime or cost of running the e2e tests, so I was basically trying to do the same thing.

Specifically:

• We define managed images explicitly in the default namespace, meaning that however many times they are applied they are only actually downloaded once
• We define 'non-core' images at their point of use, so if, e.g. we don't run the flatcar tests, we don't download the flatcar images at all
• We only directly reference unmanaged in machine specs because we expect them to have been downloaded exactly once by the managed versions in default

I don't think we pay ingress for images downloaded from a GCP source, so perhaps the financial motivation for this isn't so strong as long as we have enough disk space for multiple copies of the images. Pretty sure we'd pay ingress for the flatcar download, though, so ideally we'd continue to minimise downloads of that.

That said, I still think these are reasonable goals. What's not great is that the download locations themselves moved in to code, making them hard to modify when running locally. I see this PR moves them back into e2e_conf. How would you feel about keeping that part of this change, and just updating the references centrally?

[mdbooth]

Where does this syntax come from, btw? I don't think I've seen it before.

[lentzi90]

Thanks for checking already in WIP state!
I'm still playing around and seeing what works here. The main goal is to make it easier to configure the e2e suite for different environments, and I do also want to keep the behavior of only downloading images once. My idea is to try the onDelete: detach option and see if that achieves the same.

The other thing I would want to do is to make image downloading optional, i.e. use only unmanaged images throughout. Basically, I want to be able to write a config file for an openstack cloud where some images already exists, and just use them in the tests.

[lentzi90]

/test pull-cluster-api-provider-openstack-e2e

[k8s-ci-robot]

@lentzi90: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test pull-cluster-api-provider-openstack-build

/test pull-cluster-api-provider-openstack-e2e-test

/test pull-cluster-api-provider-openstack-test

The following commands are available to trigger optional jobs:

/test pull-cluster-api-provider-openstack-conformance-test

/test pull-cluster-api-provider-openstack-e2e-full-test

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-provider-openstack-build

pull-cluster-api-provider-openstack-e2e-test

pull-cluster-api-provider-openstack-test

In response to this:

/test pull-cluster-api-provider-openstack-e2e

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[lentzi90]

/test pull-cluster-api-provider-openstack-e2e-test

[k8s-ci-robot]

@lentzi90: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-provider-openstack-e2e-test	6d65885	link	true	/test pull-cluster-api-provider-openstack-e2e-test

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.