eks-prow-build-cluster: Reconsider instance type selection

[tzneal]

What should be cleaned up or changed:

Some changes were made to the EKS cluster to attempt to resolve an issue with test flakes. These changes also increased the per-node cost. We should consider reverting these changes to reduce cost.

a) Changing to an instance type without instance storage.

b) Changing back to an AMD CPU type

c) Changing to a roughly 8 CPU / 64GB type to more closely match the existing GCP cluster nodes

The cluster currently uses an r5d.4xlarge (16 CPU/ 128 GB) with an on-demand cost of 1.152

An r5a.4xlarge (16 CPU / 128 GB) has an on-demand cost of 0.904 per hour

An r5a.2xlarge (8 CPU / 64 GB) has an on-demand cost of 0.45 per hour

Provide any links for context:

[tzneal]

/sig k8s-infra

[xmudrii]

I'm going to transfer this issue to k/k8s.io as other issues related to this cluster are already there.
/transfer-issue k8s.io

[xmudrii]

/assign @xmudrii @pkprzekwas

[BenTheElder]

One thing to consider: Because kubernetes doesn't have IO/IOPS isolation, sizing really large nodes changes the CPU : I/O ratio. (Though this will also not be 1:1 between GCP and AWS anyhow), so while really large nodes can allow high core count jobs OR bin packing more jobs per node ... the latter can cause issues by over-packing for I/O throughput.

This is less of an issue today than when we ran bazel builds widely, but it's still something that can cause performance issues. The existing size is semi-arbitrary though, and may be somewhat GCP specific, but right now tests that are likely to be IO heavy sometimes reserve that IO by reserving ~all of the CPU at our current node sizes.

[xmudrii]

xref #4686

[xmudrii]

To add to what @BenTheElder said: we already had issues with GOMAXPROCS for unit tests. We "migrated" 5 jobs so far and one was affected (potentially one more). To avoid such issues, we might want to have instances close to what we have on GCP. We can't have 1:1 mapping, but we can try using similar instances based on what AWS offers.

Not having to deal with stuff such as GOMAXPROCS is going to make the migration more smooth and we'll avoid spending a lot of time on debugging such issues.

[dims]

@xmudrii fyi kubernetes/kubernetes#117016