Add option to override driver + executor memory limit using webhook

[netanellevine]

Purpose of this PR

Add an option to override the driver and executor memory limit as by default spark uses equal limit and request.
It is Done by webhook as the pod template does not support overriding the memory limit.

Change Category

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Rationale

For many use cases, it makes sense to share memory between pods as they don't all peak simultaneously. Limit allows better resources allocation

Checklist

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign andreyvelich for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[bnetzi]

Hi there, I'm from @netanellevine team, this feature was proven to be very important to us, as instead of defining for each app the max memory amount used, we were able to define a much lower number and save both time and money.

[danielrsfreitas]

@netanellevine @bnetzi I am glad you were able to take that on. It seems like a very useful enhancement. I am wondering if you would need UT as well. I would be happy to help on that (it would be my first contribution, so let me know if I am a distraction ).

[bnetzi]

Hi @danielrsfreitas we will be glad if you can add UT, we didn't got to it (although we did build it and tested it in our env)