WIP Upgrade Karpenter to 1.0.0

[rifelpet]

TODO list:

• Set the EC2NodeClass userdata - this is a bit tricky because userdata is normally generated via fi.CloudupTask dependencies but none of the bootstrapchannelbuilder code uses fi.Resource
• Update awsmodel to not add awstasks.LaunchTemplates for Karpenter instance groups
• Review rolling-update, delete cluster, and toolbox dump code to confirm it works without kops-managed launch templates
• Add release note and update documentation
• Figure out why the --kubeconfig flag is being removed from the CLI options
• Reuse the root volume defaulting logic from pkg/awsmodel

Note that this doesn't follow the required karpenter upgrade procedure - therefore existing kops clusters with karpenter enabled may not have an upgrade path to this version of karpenter. We'll need to figure out a low-effort workaround to include in the upgrade instructions.

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from rifelpet. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rifelpet]

looks like reuse the awsmodel's defaulting logic for root volumes:

kops/pkg/model/awsmodel/autoscalinggroup.go

Lines 146 to 171 in 5d4d867

	rootVolumeSize, err := defaults.DefaultInstanceGroupVolumeSize(ig.Spec.Role)
	if err != nil {
	return nil, err
	}
	var rootVolumeType ec2types.VolumeType
	rootVolumeEncryption := DefaultVolumeEncryption
	rootVolumeKmsKey := ""
	if ig.Spec.RootVolume != nil {
	if fi.ValueOf(ig.Spec.RootVolume.Size) > 0 {
	rootVolumeSize = fi.ValueOf(ig.Spec.RootVolume.Size)
	}
	rootVolumeType = ec2types.VolumeType(fi.ValueOf(ig.Spec.RootVolume.Type))
	if ig.Spec.RootVolume.Encryption != nil {
	rootVolumeEncryption = fi.ValueOf(ig.Spec.RootVolume.Encryption)
	}
	if fi.ValueOf(ig.Spec.RootVolume.Encryption) && ig.Spec.RootVolume.EncryptionKey != nil {
	rootVolumeKmsKey = *ig.Spec.RootVolume.EncryptionKey
	}
	}
	if rootVolumeType == "" {
	rootVolumeType = DefaultVolumeType
	}

[justinsb]

I had a look at the userdata problem. As part of the bare metal WIP, I've refactored the bootstrap script building so it isn't as tricky as it used to be: https://github.com/kubernetes/kops/blob/master/pkg/commands/toolbox_enroll.go#L401

I had a go at piping that through, it isn't too bad except for the wellKnownAddresses. But getting those would indeed be tricky...

But - I was wondering if we should build this in the template anyway, vs uploading the kops InstanceGroups and having kops-controller (for example) expand them. This is what I've been playing with for cluster-api, and it seems to work I think (though I should probably revive and clean that up)

Another advantage of doing it that way is that users could then dynamically create InstanceGroups. The gotcha is that those would not be stored in the state store so would have to be managed differently, but we don't have to support (or encourage) dynamic InstanceGroups.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.