Add e2e tests for runtimes v2

[akagami-harsh]

What this PR does / why we need it:

• added e2e tests for training runtimes v2 using kubernetes golang client

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):

• part of KEP-2170: Add E2E tests for Kubeflow Training V2 #2213

Checklist:

• Docs included if any changes are user facing

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign johnugeorge for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[akagami-harsh]

Hey @andreyvelich I have added e2e tests for runtimes v2. Would appreciate your feedback on the test scenarios and structure. Let me know if I'm on the right track!

[andreyvelich]

Thank you for working on this and sorry for the late reply @akagami-harsh!
I think, we should also refactor these script to trigger E2Es:

• https://github.com/kubeflow/trainer/blob/master/hack/e2e-setup-cluster.sh
• https://github.com/kubeflow/trainer/blob/master/.github/workflows/test-e2e.yaml

[akagami-harsh]

Thank you for working on this and sorry for the late reply @akagami-harsh!

np

I think, we should also refactor these script to trigger E2Es:
https://github.com/kubeflow/trainer/blob/master/hack/e2e-setup-cluster.sh
https://github.com/kubeflow/trainer/blob/master/.github/workflows/test-e2e.yaml

working on it