Document vpc peering between management cluster vpc and tenant cluster vpc

[nab-gha]

/kind feature

Describe the solution you'd like
See #2465, the management cluster requires access to the tenant cluster api and if that is configured to be private access only the management clusters vpc needs to be peered with the tenant cluster's vpc.

The establishing of the required vpc peering needs to be performed by the user. We should document this process and the reasons why it is required.

[nab-gha]

When creating an EKS cluster, if you specify that the API server should only be accessible via the private network then the CAPI/CAPA controllers running on the management cluster will not be able to access the API server of the tenant cluster being created. The CAPI controller on the management cluster requires access to the tenant cluster's API in order to deploy ClusterResourceSets and the CAPA cluster requires access to complete the cluster deployment.

Therefore it is necessary to ensure the management cluster has access to the tenant clusters private API endpoint. If the management cluster is running outside the AWS environment then it will be necessary to provide access to the AWS VPC private network using VPN access. This scenario is out of scope for this ticket, see #2504

If the management cluster is running in AWS then the recommended approach is to establish VPC peering between the management cluster vpc and the tenant cluster VPC. This can be achieved by following the instructions at https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html

Request a peering connection between the management cluster VPC and the tenant cluster VPC then accept the peering request using the account the tenant cluster is running under.

Note that the private address blocks for the management and tenant cluster VPCs must not overlap. This can be set in the AWSManagedControlPlane specification...

kind: AWSManagedControlPlane
apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
metadata:
  name: "tenant01-control-plane"
spec:
  networkSpec:
    vpc:
      cidrBlock: "10.201.0.0/16"
  endpointAccess:
    public: false
    private: true

Routes from the management cluster VPC to the tenant cluster VPC and tenant cluster VPC to management cluster VPC must be established.

For each private subnet in each subnet in the management cluster VPC add an additional rule to the existing route specifying the CIDR of the tenant VPC and the peering connection as the target.

For each private subnet in each subnet in the tenant cluster VPC add an additional rule to the existing route specifying the CIDR of the management VPC and the peering connection as the target.

An additional ingress rule will need to be added to the tenant cluster control plane security group to allow access from the management cluster.

#2465 will provide a mechanism for adding the security group access automatically

[nab-gha]

This needs to be added to docs, when I or someone else has time

[sedefsavas]

/help
/milestone v0.7.x

[k8s-ci-robot]

@sedefsavas:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help
/milestone v0.7.x

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sayantani11]

I can add it to the docs if required.

[sedefsavas]

/assign @sayantani11