Skip to content

Latest commit

 

History

History
232 lines (177 loc) · 23 KB

gardenlet_api_access.md

File metadata and controls

232 lines (177 loc) · 23 KB
Raw

Scoped API Access for Gardenlets

By default, gardenlets have administrative access in the garden cluster. They are able to execute any API request on any object independent of whether the object is related to the seed cluster the gardenlet is responsible fto. As RBAC is not powerful enough for fine-grained checks and for the sake of security, Gardener provides two optional but recommended configurations for your environments that scope the API access for gardenlets.

Similar to the Node authorization mode in Kubernetes, Gardener features a SeedAuthorizer plugin. It is a special-purpose authorization plugin that specifically authorizes API requests made by the gardenlets.

Likewise, similar to the NodeRestriction admission plugin in Kubernetes, Gardener features a SeedRestriction plugin. It is a special-purpose admission plugin that specifically limits the Kubernetes objects gardenlets can modify.

📚 You might be interested to look into the design proposal for scoped Kubelet API access from the Kubernetes community. It can be translated to Gardener and Gardenlets with their Seed and Shoot resources.

Flow Diagram

The following diagram shows how the two plugins are included in the request flow of a gardenlet. When they are not enabled then the kube-apiserver is internally authorizing the request via RBAC before forwarding the request directly to the gardener-apiserver, i.e., the gardener-admission-controller would not be consulted (this is not entirely correct because it also serves other admission webhook handlers, but for simplicity reasons this document focuses on the API access scope only).

When enabling the plugins, there is one additional step for each before the gardener-apiserver responds to the request.

Flow Diagram

Please note that the example shows a request to an object (Shoot) residing in one of the API groups served by gardener-apiserver. However, the gardenlet is also interacting with objects in API groups served by the kube-apiserver (e.g., Secret,ConfigMap, etc.). In this case, the consultation of the SeedRestriction admission plugin is performed by the kube-apiserver itself before it forwards the request to the gardener-apiserver.

Today, the following rules are implemented:

Resource Verbs Path(s) Description
BackupBucket get, list, watch, create, update, patch, delete BackupBucket -> Seed Allow get, list, watch requests for all BackupBuckets. Allow only create, update, patch, delete requests for BackupBuckets assigned to the gardenlet's Seed.
BackupEntry get, list, watch, create, update, patch BackupEntry -> Seed Allow get, list, watch requests for all BackupEntrys. Allow only create, update, patch requests for BackupEntrys assigned to the gardenlet's Seed and referencing BackupBuckets assigned to the gardenlet's Seed.
Bastion get, list, watch, create, update, patch Bastion -> Seed Allow get, list, watch requests for all Bastions. Allow only create, update, patch requests for Bastions assigned to the gardenlet's Seed.
CertificateSigningRequest get, create CertificateSigningRequest -> Seed Allow only get, create requests for CertificateSigningRequests related to the gardenlet's Seed.
CloudProfile get CloudProfile -> Shoot -> Seed Allow only get requests for CloudProfiles referenced by Shoots that are assigned to the gardenlet's Seed.
ClusterRoleBinding create, get, update, patch, delete ClusterRoleBinding -> ManagedSeed -> Shoot -> Seed Allow create, get, update, patch requests for ManagedSeeds in the bootstrapping phase assigned to the gardenlet's Seeds. Allow delete requests from gardenlets bootstrapped via ManagedSeeds.
ConfigMap get ConfigMap -> Shoot -> Seed Allow only get requests for ConfigMaps referenced by Shoots that are assigned to the gardenlet's Seed. Allows reading the kube-system/cluster-identity ConfigMap.
ControllerRegistration get, list, watch ControllerRegistration -> ControllerInstallation -> Seed Allow get, list, watch requests for all ControllerRegistrations.
ControllerDeployment get ControllerDeployment -> ControllerInstallation -> Seed Allow get requests for ControllerDeploymentss referenced by ControllerInstallations assigned to the gardenlet's Seed.
ControllerInstallation get, list, watch, update, patch ControllerInstallation -> Seed Allow get, list, watch requests for all ControllerInstallations. Allow only update, patch requests for ControllerInstallations assigned to the gardenlet's Seed.
Event create, patch none Allow to create or patch all kinds of Events.
ExposureClass get ExposureClass -> Shoot -> Seed Allow get requests for ExposureClasses referenced by Shoots that are assigned to the gardenlet's Seed. Deny get requests to other ExposureClasses.
Lease create, get, watch, update Lease -> Seed Allow create, get, update, and delete requests for Leases of the gardenlet's Seed.
ManagedSeed get, list, watch, update, patch ManagedSeed -> Shoot -> Seed Allow get, list, watch requests for all ManagedSeeds. Allow only update, patch requests for ManagedSeeds referencing a Shoot assigned to the gardenlet's Seed.
Namespace get Namespace -> Shoot -> Seed Allow get requests for Namespaces of Shoots that are assigned to the gardenlet's Seed. Always allow get requests for the garden Namespace.
Project get Project -> Namespace -> Shoot -> Seed Allow get requests for Projects referenced by the Namespace of Shoots that are assigned to the gardenlet's Seed.
SecretBinding get SecretBinding -> Shoot -> Seed Allow only get requests for SecretBindings referenced by Shoots that are assigned to the gardenlet's Seed.
Secret create, get, update, patch, delete(, list, watch) Secret -> Seed, Secret -> Shoot -> Seed, Secret -> SecretBinding -> Shoot -> Seed, BackupBucket -> Seed Allow get, list, watch requests for all Secrets in the seed-<name> namespace. Allow only create, get, update, patch, delete requests for the Secrets related to resources assigned to the gardenlet's Seed`s.
Seed get, list, watch, create, update, patch, delete Seed Allow get, list, watch requests for all Seeds. Allow only create, update, patch, delete requests for the gardenlet's Seeds. [1]
ServiceAccount create, get, update, patch, delete ServiceAccount -> ManagedSeed -> Shoot -> Seed Allow create, get, update, patch requests for ManagedSeeds in the bootstrapping phase assigned to the gardenlet's Seeds. Allow delete requests from gardenlets bootstrapped via ManagedSeeds.
Shoot get, list, watch, update, patch Shoot -> Seed Allow get, list, watch requests for all Shoots. Allow only update, patch requests for Shoots assigned to the gardenlet's Seed.
ShootState get, create, update, patch ShootState -> Shoot -> Seed Allow only get, create, update, patch requests for ShootStates belonging by Shoots that are assigned to the gardenlet's Seed.

[1] If you use ManagedSeed resources then the gardenlet reconciling them ("parent gardenlet") may be allowed to submit certain requests for the Seed resources resulting out of such ManagedSeed reconciliations (even if the "parent gardenlet" is not responsible for them):

  • ℹ️ It is allowed to delete the Seed resources if the corresponding ManagedSeed objects already have a deletionTimestamp (this is secure as gardenlets themselves don't have permissions for deleting ManagedSeeds).
  • ⚠ It is allowed to create or update Seed resources if the corresponding ManagedSeed objects use a seed template, i.e., .spec.seedTemplate != nil. In this case, there is at least one gardenlet in your system which is responsible for two or more Seeds. Please keep in mind that this use case is not recommended for production scenarios (you should only have one dedicated gardenlet per seed cluster), hence, the security improvements discussed in this document might be limited.

SeedAuthorizer Authorization Webhook Enablement

The SeedAuthorizer is implemented as Kubernetes authorization webhook and part of the gardener-admission-controller component running in the garden cluster.

🎛 In order to activate it, you have to follow these steps:

  1. Set the following flags for the kube-apiserver of the garden cluster (i.e., the kube-apiserver whose API is extended by Gardener):

    • --authorization-mode=RBAC,Node,Webhook (please note that Webhook should appear after RBAC in the list [1]; Node might not be needed if you use a virtual garden cluster)
    • --authorization-webhook-config-file=<path-to-the-webhook-config-file>
    • --authorization-webhook-cache-authorized-ttl=0
    • --authorization-webhook-cache-unauthorized-ttl=0

  2. The webhook config file (stored at <path-to-the-webhook-config-file>) should look as follows:

    apiVersion: v1
kind: Config
clusters:
- name: garden
  cluster:
    certificate-authority-data: base64(CA-CERT-OF-GARDENER-ADMISSION-CONTROLLER)
    server: https://gardener-admission-controller.garden/webhooks/auth/seed
users:
- name: kube-apiserver
  user: {}
contexts:
- name: auth-webhook
  context:
    cluster: garden
    user: kube-apiserver
current-context: auth-webhook

  3. When deploying the Gardener controlplane Helm chart, set .global.rbac.seedAuthorizer.enabled=true. This will prevent that the RBAC resources granting global access for all gardenlets will be deployed.

  4. Delete the existing RBAC resources granting global access for all gardenlets by running:

    kubectl delete \
  clusterrole.rbac.authorization.k8s.io/gardener.cloud:system:seeds \
  clusterrolebinding.rbac.authorization.k8s.io/gardener.cloud:system:seeds \
  --ignore-not-found

Please note that you should activate the SeedRestriction admission handler as well.

[1] The reason for the fact that Webhook authorization plugin should appear after RBAC is that the kube-apiserver will be depending on the gardener-admission-controller (serving the webhook). However, the gardener-admission-controller can only start when gardener-apiserver runs, but gardener-apiserver itself can only start when kube-apiserver runs. If Webhook is before RBAC then gardener-apiserver might not be able to start, leading to a deadlock.

Authorizer Decisions

As mentioned earlier, it's the authorizer's job to evaluate API requests and return one of the following decisions:

  • DecisionAllow: The request is allowed, further configured authorizers won't be consulted.
  • DecisionDeny: The request is denied, further configured authorizers won't be consulted.
  • DecisionNoOpinion: A decision cannot be made, further configured authorizers will be consulted.

For backwards compatibility, no requests are denied at the moment, so that they are still deferred to a subsequent authorizer like RBAC. Though, this might change in the future.

First, the SeedAuthorizer extracts the Seed name from the API request. This requires a proper TLS certificate the gardenlet uses to contact the API server and is automatically given if TLS bootstrapping is used. Concretely, the authorizer checks the certificate for name gardener.cloud:system:seed:<seed-name> and group gardener.cloud:system:seeds. In cases where this information is missing e.g., when a custom Kubeconfig is used, the authorizer cannot make any decision. Thus, RBAC is still a considerable option to restrict the gardenlet's access permission if the above explained preconditions are not given.

With the Seed name at hand, the authorizer checks for an existing path from the resource that a request is being made for to the Seed belonging to the gardenlet. Take a look at the Implementation Details section for more information.

Implementation Details

Internally, the SeedAuthorizer uses a directed, acyclic graph data structure in order to efficiently respond to authorization requests for gardenlets:

  • A vertex in this graph represents a Kubernetes resource with its kind, namespace, and name (e.g., Shoot:garden-my-project/my-shoot).
  • An edge from vertex u to vertex v in this graph exists when
    • (1) v is referred by u and v is a Seed, or when
    • (2) u is referred by v, or when
    • (3) u is strictly associated with v.

For example, a Shoot refers to a Seed, a CloudProfile, a SecretBinding, etc., so it has an outgoing edge to the Seed (1) and incoming edges from the CloudProfile and SecretBinding vertices (2). However, there might also be a ShootState or a BackupEntry resource strictly associated with this Shoot, hence, it has incoming edges from these vertices (3).

Resource Dependency Graph

In above picture the resources that are actively watched have are shaded. Gardener resources are green while Kubernetes resources are blue. It shows the dependencies between the resources and how the graph is built based on above rules.

ℹ️ Above picture shows all resources that may be accessed by gardenlets, except for the Quota resource which is only included for completeness.

Now, when a gardenlet wants to access certain resources then the SeedAuthorizer uses a Depth-First traversal starting from the vertex representing the resource in question, e.g., from a Project vertex. If there is a path from the Project vertex to the vertex representing the Seed the gardenlet is responsible for then it allows the request.

Metrics

The SeedAuthorizer registers the following metrics related to the mentioned graph implementation:

Metric Description
gardener_admission_controller_seed_authorizer_graph_update_duration_seconds Histogram of duration of resource dependency graph updates in seed authorizer, i.e., how long does it take to update the graph's vertices/edges when a resource is created, changed, or deleted.
gardener_admission_controller_seed_authorizer_graph_path_check_duration_seconds Histogram of duration of checks whether a path exists in the resource dependency graph in seed authorizer.

Debug Handler

When the .server.enableDebugHandlers field in the gardener-admission-controller's component configuration is set to true then it serves a handler that can be used for debugging the resource dependency graph under /debug/resource-dependency-graph.

🚨 Only use this setting for development purposes as it enables unauthenticated users to view all data if they have access to the gardener-admission-controller component.

The handler renders an HTML page displaying the current graph with a list of vertices and its associated incoming and outgoing edges to other vertices. Depending on the size of the Gardener landscape (and consequently, the size of the graph), it might not be possible to render it in its entirety. If there are more than 2000 vertices then the default filtering will selected for kind=Seed to prevent overloading the output.

Example output:

-------------------------------------------------------------------------------
|
| # Seed:my-seed
|   <- (11)
|     BackupBucket:73972fe2-3d7e-4f61-a406-b8f9e670e6b7
|     BackupEntry:garden-my-project/shoot--dev--my-shoot--4656a460-1a69-4f00-9372-7452cbd38ee3
|     ControllerInstallation:dns-external-mxt8m
|     ControllerInstallation:extension-shoot-cert-service-4qw5j
|     ControllerInstallation:networking-calico-bgrb2
|     ControllerInstallation:os-gardenlinux-qvb5z
|     ControllerInstallation:provider-gcp-w4mvf
|     Secret:garden/backup
|     Shoot:garden-my-project/my-shoot
|
-------------------------------------------------------------------------------
|
| # Shoot:garden-my-project/my-shoot
|   <- (5)
|     CloudProfile:gcp
|     Namespace:garden-my-project
|     Secret:garden-my-project/my-dns-secret
|     SecretBinding:garden-my-project/my-credentials
|     ShootState:garden-my-project/my-shoot
|   -> (1)
|     Seed:my-seed
|
-------------------------------------------------------------------------------
|
| # ShootState:garden-my-project/my-shoot
|   -> (1)
|     Shoot:garden-my-project/my-shoot
|
-------------------------------------------------------------------------------

... (etc., similarly for the other resources)

There are anchor links to easily jump from one resource to another, and the page provides means for filtering the results based on the kind, namespace, and/or name.

Pitfalls

When there is a relevant update to an existing resource, i.e., when a reference to another resource is changed, then the corresponding vertex (along with all associated edges) is first deleted from the graph before it gets added again with the up-to-date edges. However, this does only work for vertices belonging to resources that are only created in exactly one "watch handler". For example, the vertex for a SecretBinding can either be created in the SecretBinding handler itself or in the Shoot handler. In such cases, deleting the vertex before (re-)computing the edges might lead to race conditions and potentially renders the graph invalid. Consequently, instead of deleting the vertex, only the edges the respective handler is responsible for are deleted. If the vertex ends up with no remaining edges then it also gets deleted automatically. Afterwards, the vertex can either be added again or the updated edges can be created.

SeedRestriction Admission Webhook Enablement

The SeedRestriction is implemented as Kubernetes admission webhook and part of the gardener-admission-controller component running in the garden cluster.

🎛 In order to activate it, you have to set .global.admission.seedRestriction.enabled=true when using the Gardener controlplane Helm chart. This will add an additional webhook in the existing ValidatingWebhookConfiguration of the gardener-admission-controller which contains the configuration for the SeedRestriction handler. Please note that it should only be activated when the SeedAuthorizer is active as well.

Admission Decisions

The admission's purpose is to perform extended validation on requests which require the body of the object in question. Additionally, it handles CREATE requests of gardenlets (above discussed resource dependency graph cannot be used in such cases because there won't be any vertex/edge for non-existing resources).

Gardenlets are restricted to only create new resources which are somehow related to the seed clusters they are responsible for.