gpu: study and implement a way to limit Pod's i915 count

[tkatila]

In #1377 it was identified that it would be required in some clusters to limit the Pod's i915 resource count to 1 (or some other value). The idea is to allow setting shared-dev-num to >1 and to prevent users from accessing more GPU resource than designed.

Webhook might be a good way to implement this, but it would be good to study other solutions as well.

[eero-t]

K8s supports limit ranges: https://kubernetes.io/docs/concepts/policy/limit-range/

@uniemimu thought that it should support extended resources in addition to core ones, but somebody needs to check (also) whether it actually works when one omits default and min values (i.e. whether it allows pods that do not request a GPU):

apiVersion: v1
kind: LimitRange
metadata:
  name: gpu-count-constraint
spec:
  limits:
  - max:
      gpu.intel.com/i915: 1
    type: Container

@uMartinXu can you try whether that does what you wanted?

[vbedida79]

Hi @eero-t, tried limitrange. supports extended resources.
It does not accept pods with gpu.intel.com/i915 > 1.
Warning FailedCreate 2m55s job-controller Error creating: pods "intel-dgpu-clinfo-gphs2" is forbidden: maximum gpu.intel.com/i915 usage per Container is 1, but limit is 3
In case of a pods that do not request gpu resource- whether scheduled on GPU node or others, the pod does allocate i915 resource requests and limits:

LimitRanger plugin set: gpu.intel.com/i915 request for container httpd;
      gpu.intel.com/i915 limit for container httpd

containers:
    - resources:
        limits:
          gpu.intel.com/i915: '1'
        requests:
          gpu.intel.com/i915: '1'

[eero-t]

Thanks for testing, good to hear that (max) limiting part works!

Will it still try to add GPU resource requests for non-GPU pods, if you add to LimitRange:

min:
      gpu.intel.com/i915: 0

?

[vbedida79]

Yes, added min limit. It still adds request and limit i915 resource to non-GPU pods.

I understand its cause of the default limit and default request, is that right? Irrespective of what default value is set, it does add that to non GPU pods.

    default: 
      gpu.intel.com/i915: 1
    defaultRequest:
      gpu.intel.com/i915: 1

[eero-t]

Do you mean that even if you specify gpu.intel.com/i915: 0 default, non-GPU pods get gpu.intel.com/i915: 1?

That sounds like bug which should be filed to upstream kubernetes. Either as "LimitRange does not respect specified default value", or "LimitRange adds resource request even when default and min resource requests are specified as zero".

[vbedida79]

If default values are added to limitrange it is added to non GPU pods. If no default values are added to limit range, it allocates below values to non GPU pods. Is this expected?

- resources:
        limits:
          gpu.intel.com/i915: '1'
        requests:
          gpu.intel.com/i915: '1'

[eero-t]

If default values are added to limitrange it is added to non GPU pods.

so if you specify default as gpu.intel.com/i915: 0, is the resource request zero for non-GPU pods?

[vbedida79]

Yes works as expected. If default is gpu.intel.com/i915: 0 it shows the same for non GPU pods

[vbedida79]

With max and min in limitrange, for GPU pods requesting resource > 1, it has a forbidden error.
If default values are not added in limirange, it ends up adding max values as limit and request to non GPU pods as well.
After adding default limit and request both 0- for non GPU pods, it just shows up as gpu.intel.com/i915: 0 in the spec.

[eero-t]

After adding default limit and request both 0- for non GPU pods, it just shows up as gpu.intel.com/i915: 0 in the spec.

@uniemimu You've been looking more into scheduler. Do you see any practical problem with zero extended resource request being added to non-GPU processes 8by limitRange)?

I.e. does it just look funny, but work fine in practice?

[vbedida79]

Also, as limitrange is namespace scoped- we only add it in the namespace where workloads are deployed right?

[eero-t]

Yes, to namespaces where your cluster config RBAC rules allow given k8s users access to GPU resources.

[vbedida79]

Got it, thanks.

I.e. does it just look funny, but work fine in practice?

Apart from this, can we assume limitrange as an efficient solution? Then we could add the yaml in the project 1.0.0 GA as deployment step after GPU deviceplugin is created- for publishing certified operator on OCP 4.12.

[eero-t]

I'm also assuming this would be for operator configuration option for whether multi-GPU jobs are allowed.

But does operator component know which namespaces in given cluster are allowed to access GPU resources? Aren't such RBAC rules rather specific to cluster?