Add GPU support to kubernetes_scale on EKS Karpenter #6313
Conversation
| name: {{ Name }} | ||
| spec: | ||
| {%- if NvidiaGpuRequest and Cloud == 'aws' %} | ||
| nodeSelector: |
There was a problem hiding this comment.
Zach recently changed how the nodeSelectors area applied. Take a look at #6291, and in particular, the (new) GetNodeSelectors() methods in the providers. (e.g. within providers/aws/elastic_kubernetes_service.py).
That said, your nodeSelector here refers back to your aws-gpu-nodepool.yaml.j2 file, so I think the new GetNodeSelectors() method probably doesn't apply - keeping this in the yaml.j2 file seems like the right approach here.
No action required (though take a quick look at that #6291.)
There was a problem hiding this comment.
Thanks for the pointer, I checked #6291.
This does not apply in our case, so you are right, no changes are needed here. Thanks.
There was a problem hiding this comment.
Hmm, I disagree here. A few reasons:
- GetNodeSelectors & ModifyPodSpecPlacementYaml + the associated ConvertManifestToYamlDicts -> ModifyPodSpecPlacementYaml -> ApplyYaml flow is basically doing the same thing & the code you've written can be written in this manner. It's very inconsistent & makes for an odd exception for specifically Karpenter to use the old ApplyManifest path instead. To some extent this is just a "my new way vs your old way" & I suspect the old way make more sense / you may have already gotten some of this code working via the old way before syncing & realizing the old way existed.. but now that the new way does exist we shouldn't mix & match between the two.
- Now how would this be implemented here & why could an exception make sense? Rich notes that the GetNodeSelectors function here refers to the aws-gpu-nodepool.yaml.j2 nodepool , which is only applied in this benchmark.. hence putting this code in the resource would be tricky.
- However, it doesn't actually look like this code is benchmark specific and should instead be resource specific. See my comments about aws-gpu-nodepool.yaml.j2 looking the same as in kubernetes ai inference.
So all of this should instead just go into the EksKarpenterCluster class & its GetNodeSelectors + ModifyPodSpecPlacementYaml code.
| operator: "Exists" | ||
| effect: "NoExecute" | ||
| tolerationSeconds: {{ PodTimeout }} | ||
| {%- if NvidiaGpuRequest and Cloud == 'aws' %} |
There was a problem hiding this comment.
Similarly, that same PR introduced ModifyPodSpecPlacyementYaml method, which seems like it could apply here. But again, this block refers back to your nodepool yaml.j2 file, so I think you should probably leave it as is.
No action required.
There was a problem hiding this comment.
Thanks for the pointer and the context.
| ) | ||
|
|
||
| if is_eks_karpenter_aws_gpu: | ||
| cluster.ApplyManifest( |
There was a problem hiding this comment.
NB: Because ScaleUpPods is called twice, you'll end up applying this nodepool manifest twice. Which is fine - the second time k8s will notice it's already present and do nothing.
But perhaps we should put this into the Prepare() function instead? (And if that works, maybe we should put the first invocation of ScaleUpPods there too... though if you want to do that, then a separate PR might be better.)
There was a problem hiding this comment.
@rsgowman please take a look, I have added 2 functions and move the nodepool creation to Prepare()
| @@ -0,0 +1,38 @@ | |||
| apiVersion: karpenter.sh/v1 | |||
There was a problem hiding this comment.
Is this file a duplicate of https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/data/container/kubernetes_ai_inference/aws-gpu-nodepool.yaml.j2 ? It looks very similar.
Suggestions:
- We should have code reuse rather than duplication. As mentioned in my other comment, move the application of this nodepool yaml to the resource rather than benchmark specific yamls. This would probably then live in the data/container/karpenter folder.
- If not, at least use the same file.
- Worst case / in the future copies like this should show in git as a cp using https://stackoverflow.com/questions/1043388/record-file-copy-operation-with-git . This would also make it more obvious if anything has changed.
But really I think I'm looking for Cassandra Benchmark Missing Quotes #1 here.
| name: {{ Name }} | ||
| spec: | ||
| {%- if NvidiaGpuRequest and Cloud == 'aws' %} | ||
| nodeSelector: |
There was a problem hiding this comment.
Hmm, I disagree here. A few reasons:
- GetNodeSelectors & ModifyPodSpecPlacementYaml + the associated ConvertManifestToYamlDicts -> ModifyPodSpecPlacementYaml -> ApplyYaml flow is basically doing the same thing & the code you've written can be written in this manner. It's very inconsistent & makes for an odd exception for specifically Karpenter to use the old ApplyManifest path instead. To some extent this is just a "my new way vs your old way" & I suspect the old way make more sense / you may have already gotten some of this code working via the old way before syncing & realizing the old way existed.. but now that the new way does exist we shouldn't mix & match between the two.
- Now how would this be implemented here & why could an exception make sense? Rich notes that the GetNodeSelectors function here refers to the aws-gpu-nodepool.yaml.j2 nodepool , which is only applied in this benchmark.. hence putting this code in the resource would be tricky.
- However, it doesn't actually look like this code is benchmark specific and should instead be resource specific. See my comments about aws-gpu-nodepool.yaml.j2 looking the same as in kubernetes ai inference.
So all of this should instead just go into the EksKarpenterCluster class & its GetNodeSelectors + ModifyPodSpecPlacementYaml code.
3 participants
No description provided.