Sync change from Klutch upstream (#35)

* Update Makefile with crossplane upstream

* Fix provider health check timeout preventing status patch
Our confighealth reconciler performs a health check HTTP request to the anynines service broker and writes the result to the providerconfig's status field.

Previously, one Golang context with a timeout was used for both the health check and the subsequent patching of providerconfig's status. However, if the health check request exceeded the timeout, the patching of the status field could never succeed. client-go's Patch() method would error with a "context deadline exceeded" error message.

This commit introduces a separate sub-context for the health check request, which has an explicit timeout, while using the parent context (which does not time out) for patching the status field.

* Add documentation for dynamic kubernetes clients

* Add makefiles for internal pipelines

* Replace native patch&transform with composite functions

* Update pg e2e test with latest service plans

* Update changelog

* Update provider-anynines and klutch dataservices to v1.3.1

* Update Klutch-Bind to v1.3.1

Author: abdulhaseeb3

CHANGELOG.md

@@ -2,6 +2,22 @@
 
 ## Unreleased
 
+## [1.3.1] - 2025-01-14
+
+### Added
+
+- Added documentation for dynamic Kubernetes clients, providing guidance on usage and best practices.
+
+### Changed
+
+- Aligned PostgreSQL plan names with a9s-dataservices for improved consistency and integration.
+
+- Adjusted health check timeouts for provider-anynines to improve reliability and reduce errors.
+
+- Enhanced internal processes by updating the Makefile for optimized pipeline management.
+
+- Replaced native patch and transform functionality with composite functions from Crossplane, ensuring better modularity and maintainability.
+
 ## [1.3.0] - 2025-01-14
 
 ### Changed
@@ -34,7 +50,7 @@
 
 - Crossplane Provider named `provider-anynines` for leveraging a9s Data Services.
 - Health checks and readinessProbe for ProviderConfigs.
-- Klunch-bind for cross-cluster service management.
+- Klutch-bind for cross-cluster service management.
 - Crossplane APIs for a9s Data services (provider-anynines), a8s Data Services (provider-kubernetes) and AWS s3 buckets (provider-aws-s3).
 - Documentation content for the Docusaurus-powered Klutchio website.
 - End-to-end tests for Klutch.

bind/docs/dynamic-client.md

@@ -0,0 +1,20 @@
+# Dynamic Client
+
+Klutch Bind uses client-go's `dynamic` client, a dynamic client is a kubernetes client that does not
+have an associated type. Klutch-bind uses it in the Konnector to sync arbitrary resources between
+clusters. The resources to be synced are configured at runtime by the APIs that the user has bound,
+so they cannot be pre-compiled.
+
+Instead of working on objects of a struct, it takes and returns `map[string]interface{}`. The string
+key is the field name, and the interface is the value of the field. For example: `obj["spec"]` will
+return the spec, which will be another `map[string]interface{}`. Because the client does not have an
+associated type, it needs to be parameterized with the `GroupVersionKind` for operations.
+
+Inside the Konnector, klutch-bind automatically configures and starts new controllers for each
+resource to be synchronized based on dynamic client-go clients. To learn more about how controllers
+are constructed using client-go you can check out the following resources:
+
+- [kubernetes sample controller using
+  client-go](https://github.com/kubernetes/sample-controller/blob/master/docs/controller-client-go.md)
+- [client-go dynamic](https://github.com/kubernetes/client-go/tree/master/examples/dynamic-create-update-delete-deployment)
+- [kubecon talk about client-go controllers](https://www.youtube.com/watch?v=_BuqPMlXfpE)

crossplane-api/Makefile

@@ -6,9 +6,13 @@ providerconfig:
 ifdef postgresInstanceName
 	@export PG_SERVICE_INSTANCE_NAME=$$postgresInstanceName; \
 	if [[ $$GET_BROKER_IP == "true" ]]; then \
-		export PG_SERVICEBROKER_IP=$$(ssh aws-s1-inception ". /var/vcap/store/jumpbox/home/a9s/bosh/envs/dsf2;bosh -d $$PG_SERVICE_INSTANCE_NAME instances" | grep "broker/" | sed 's/\t/ /g' | tr -s " " | cut -d " " -f4); \
+		if [[ $$PG_SERVICE_INSTANCE_NAME == "" ]]; then \
+			echo "ERROR: postgresInstanceName must not be empty!"; \
+			exit 1; \
+		fi; \
+		export PG_SERVICEBROKER_IP=$$(ssh aws-s1-inception ". /var/vcap/store/jumpbox/home/a9s/bosh/envs/dsf2;bosh -d $$PG_SERVICE_INSTANCE_NAME instances" --json | yq -p=json '.Tables.0.Rows.[] | select ( .instance == "broker/*") | .ips'); \
 		export PG_SERVICEBROKER_HOST="http://$$PG_SERVICEBROKER_IP:3000"; \
-		export PG_BACKUP_MANAGER_IP=$$(ssh aws-s1-inception ". /var/vcap/store/jumpbox/home/a9s/bosh/envs/dsf2;bosh -d $$PG_SERVICE_INSTANCE_NAME instances" | grep "backup-manager/" | sed 's/\t/ /g' | tr -s " " | cut -d " " -f4); \
+		export PG_BACKUP_MANAGER_IP=$$(ssh aws-s1-inception ". /var/vcap/store/jumpbox/home/a9s/bosh/envs/dsf2;bosh -d $$PG_SERVICE_INSTANCE_NAME instances" --json | yq -p=json '.Tables.0.Rows.[] | select ( .instance == "backup-manager/*") | .ips'); \
 		export PG_BACKUP_MANAGER_HOST="http://$$PG_BACKUP_MANAGER_IP:3000"; \
 		echo "Backup Manager IP: $$PG_BACKUP_MANAGER_IP"; \
 		echo "Service Broker IP: $$PG_SERVICEBROKER_IP"; \

crossplane-api/README.md

@@ -285,7 +285,7 @@ To install the configuration package (containing definitions and compositions),
 1. Install the package via crossplane:
 
 ```bash
-crossplane xpkg install configuration public.ecr.aws/w5n9a2g2/klutch/dataservices:v1.3.0
+crossplane xpkg install configuration public.ecr.aws/w5n9a2g2/klutch/dataservices:v1.3.1
 ```
 
 2. Install files directly:
@@ -308,6 +308,21 @@ Crossplane won't be able to manage RBAC dynamically. As a result, Compositions
 will not be able to configure the provider-anynines managed resources due to
 authorization issues.
 
+### Install Crossplane Functions
+
+Additionally, we install composition functions. Composition functions (or simply “functions”) are Crossplane extensions that template Crossplane resources. Crossplane uses these functions to determine which resources to create when a composite resource (XR) is created. To verify that the composition functions are correctly installed, use the following command:
+
+```bash
+kubectl get function
+```
+
+Expected output:
+
+```text
+NAME                           INSTALLED   HEALTHY   PACKAGE                                                                  AGE
+function-patch-and-transform   True        True      xpkg.upbound.io/crossplane-contrib/function-patch-and-transform:v0.1.4
+```
+
 #### Install ProviderConfig for provider-anynines
 
 To configure the provider, based on your development environment, make sure to
@@ -394,10 +409,10 @@ to refer to a valid service and plan.
 | postgresql-single-nano    | 1       | 3Gi         | 2   | 1 Gi   |
 | postgresql-single-small   | 1       | 10Gi        | 2   | 2 Gi   |
 | postgresql-single-medium  | 1       | 50Gi        | 2   | 4 Gi   |
-| postgresql-single-large   | 1       | 200Gi       | 4   | 16 Gi  |
-| postgresql-cluster-small  | 3       | 10Gi        | 2   | 2 Gi   |
-| postgresql-cluster-medium | 3       | 50Gi        | 2   | 4 Gi   |
-| postgresql-cluster-large  | 3       | 200Gi       | 4   | 16 Gi  |
+| postgresql-single-big   | 1       | 200Gi       | 4   | 16 Gi  |
+| postgresql-replicas-small  | 3       | 10Gi        | 2   | 2 Gi   |
+| postgresql-replicas-medium | 3       | 50Gi        | 2   | 4 Gi   |
+| postgresql-replicas-big  | 3       | 200Gi       | 4   | 16 Gi  |
 
 ```bash
 kubectl apply -f ./crossplane-api/examples/a8s/postgresql-claim.yaml
@@ -560,23 +575,23 @@ field in [definition.yaml](https://github.com/anynines/klutchio/blob/main/crossp
 Within this field you can see a list of supported Plans:
 
     ```yaml
-    plans: &pgPlans ["postgresql-cluster-small",
-        "postgresql-cluster-medium", "postgresql-cluster-large",
+    plans: &pgPlans ["postgresql-replicas-small",
+        "postgresql-replicas-medium", "postgresql-replicas-big",
         "postgresql-single-nano","postgresql-single-small",
-        "postgresql-single-medium", "postgresql-single-large"]
+        "postgresql-single-medium", "postgresql-single-big"]
     ```
 
 3. Update the "plans" list with the new Plan to be supported.
 
-    For example, suppose the new plan "postgresql-single-extralarge" is
+    For example, suppose the new plan "postgresql-single-huge" is
     introduced, so the list will be updated to:
 
     ```yaml
-    plans: &pgPlans ["postgresql-cluster-small",
-        "postgresql-cluster-medium", "postgresql-cluster-large",
+    plans: &pgPlans ["postgresql-replicas-small",
+        "postgresql-replicas-medium", "postgresql-replicas-big",
         "postgresql-single-nano","postgresql-single-small",
-        "postgresql-single-medium", "postgresql-single-large",
-        "postgresql-single-extralarge"]
+        "postgresql-single-medium", "postgresql-single-big",
+        "postgresql-single-huge"]
     ```
 
 4. Update the validation rules.
@@ -586,7 +601,7 @@ Within this field you can see a list of supported Plans:
     [definition.yaml](https://github.com/anynines/klutchio/blob/main/crossplane-api/api/common/postgresql_definition.yaml#L30)
     file under the x-kubernetes-validations.rule field.
 
-    Continuing the example with the "postgresql-single-extralarge", the
+    Continuing the example with the "postgresql-single-huge", the
     validation in this case should be updated with the following rules that
     prohibit the transition from extralarge to smaller dataservice instances.
 
@@ -600,21 +615,21 @@ Within this field you can see a list of supported Plans:
     field in the [composition file](https://github.com/anynines/klutchio/blob/main/crossplane-api/api/a8s/postgresql/composition.yaml)
     should also be updated.
 
-    For the "postgresql-single-extralarge" example, we could add something
+    For the "postgresql-single-huge" example, we could add something
     similar to:
 
     ```yaml
-    volumeSizeExtraLarge: &volumeSizeExtraLarge "1000Gi"
-    CPUExtraLarge: &CPUExtraLarge "8"
-    MemoryExtraLarge: &MemoryExtraLarge "32Gi"
+    volumeSizeHuge: &volumeSizeHuge "1000Gi"
+    CPUHuge: &CPUHuge "8"
+    MemoryHuge: &MemoryHuge "32Gi"
     ```
 
 6. Finally, the [maps](https://github.com/anynines/klutchio/blob/main/crossplane-api/api/a8s/postgresql/composition.yaml#L53)
     used for patching the disk, cpu and memory resources in the
     [composition file](https://github.com/anynines/klutchio/blob/main/crossplane-api/api/a8s/postgresql/composition.yaml)
     should also be updated.
 
-    For our favorite "postgresql-single-extralarge" example this could mean
+    For our favorite "postgresql-single-huge" example this could mean
     adding to the maps something like:
 
     ```yaml
@@ -624,26 +639,26 @@ Within this field you can see a list of supported Plans:
             nano: *volumeSizeNano
             small: *volumeSizeSmall
             medium: *volumeSizeMedium
-            large: *volumeSizeLarge
-            extralarge: *volumeSizeExtraLarge
+            big: *volumeSizeLarge
+            huge: *volumeSizeHuge
 
     ...
     - type: map
         map:
             nano: *CPUNano
             small: *CPUSmall
             medium: *CPUMedium
-            large: *CPULarge
-            extralarge: *CPUExtraLarge
+            big: *CPULarge
+            huge: *CPUHuge
 
     ...
     - type: map
         map:
             nano: *MemoryNano
             small: *MemorySmall
             medium: *MemoryMedium
-            large: *MemoryLarge
-            extralarge: *MemoryExtraLarge
+            big: *MemoryLarge
+            huge: *MemoryHuge
     ...
     ```
 

crossplane-api/api/a8s/backup/composition.yaml

@@ -6,53 +6,60 @@ spec:
   compositeTypeRef:
     apiVersion: anynines.com/v1
     kind: XBackup
-  resources:
-    - name: a8s-backup
-      base:
-        apiVersion: kubernetes.crossplane.io/v1alpha1
-        kind: Object
-        spec:
-          forProvider:
-            manifest:
-              apiVersion: backups.anynines.com/v1beta3
-              kind: Backup
-          providerConfigRef:
-            name: kubernetes-provider
-      patches:
-        - fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
-          toFieldPath: spec.forProvider.manifest.metadata.namespace
-        - fromFieldPath: metadata.labels[crossplane.io/claim-name]
-          toFieldPath: spec.forProvider.manifest.metadata.name
-        - fromFieldPath: spec.serviceInstanceType
-          toFieldPath: spec.forProvider.manifest.spec.serviceInstance.apiGroup
-          transforms:
-            - type: map
-              map:
-                postgresql: postgresql.anynines.com
-        - fromFieldPath: spec.serviceInstanceType
-          toFieldPath: spec.forProvider.manifest.spec.serviceInstance.kind
-          transforms:
-            - type: map
-              map:
-                postgresql: PostgreSQL
-        - fromFieldPath: spec.instanceRef
-          toFieldPath: spec.forProvider.manifest.spec.serviceInstance.name
-        - type: ToCompositeFieldPath
-          fromFieldPath: status.atProvider.manifest.status.backupID
-          toFieldPath: status.managed.backupID
-        - type: ToCompositeFieldPath
-          fromFieldPath: status.atProvider.manifest.status.conditions
-          toFieldPath: status.managed.conditions
-        - type: ToCompositeFieldPath
-          fromFieldPath: status.atProvider.manifest.status.lastObservationTime
-          toFieldPath: status.managed.lastObservationTime
-        - type: ToCompositeFieldPath
-          fromFieldPath: status.atProvider.manifest.status.podUsedNamespacedName
-          toFieldPath: status.managed.podUsedNamespacedName
-        - type: ToCompositeFieldPath
-          fromFieldPath: status.atProvider.manifest.status.podUsedUID
-          toFieldPath: status.managed.podUsedUID
-        - type: ToCompositeFieldPath
-          fromFieldPath: status.atProvider.manifest.status.retries
-          toFieldPath: status.managed.retries                                        
-
+  mode: Pipeline
+  pipeline:
+    - step: patch-and-transform
+      functionRef:
+        name: function-patch-and-transform
+      input:
+        apiVersion: pt.fn.crossplane.io/v1beta1
+        kind: Resources
+        resources:
+          - name: a8s-backup
+            base:
+              apiVersion: kubernetes.crossplane.io/v1alpha1
+              kind: Object
+              spec:
+                forProvider:
+                  manifest:
+                    apiVersion: backups.anynines.com/v1beta3
+                    kind: Backup
+                providerConfigRef:
+                  name: kubernetes-provider
+            patches:
+              - fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
+                toFieldPath: spec.forProvider.manifest.metadata.namespace
+              - fromFieldPath: metadata.labels[crossplane.io/claim-name]
+                toFieldPath: spec.forProvider.manifest.metadata.name
+              - fromFieldPath: spec.serviceInstanceType
+                toFieldPath: spec.forProvider.manifest.spec.serviceInstance.apiGroup
+                transforms:
+                  - type: map
+                    map:
+                      postgresql: postgresql.anynines.com
+              - fromFieldPath: spec.serviceInstanceType
+                toFieldPath: spec.forProvider.manifest.spec.serviceInstance.kind
+                transforms:
+                  - type: map
+                    map:
+                      postgresql: PostgreSQL
+              - fromFieldPath: spec.instanceRef
+                toFieldPath: spec.forProvider.manifest.spec.serviceInstance.name
+              - type: ToCompositeFieldPath
+                fromFieldPath: status.atProvider.manifest.status.backupID
+                toFieldPath: status.managed.backupID
+              - type: ToCompositeFieldPath
+                fromFieldPath: status.atProvider.manifest.status.conditions
+                toFieldPath: status.managed.conditions
+              - type: ToCompositeFieldPath
+                fromFieldPath: status.atProvider.manifest.status.lastObservationTime
+                toFieldPath: status.managed.lastObservationTime
+              - type: ToCompositeFieldPath
+                fromFieldPath: status.atProvider.manifest.status.podUsedNamespacedName
+                toFieldPath: status.managed.podUsedNamespacedName
+              - type: ToCompositeFieldPath
+                fromFieldPath: status.atProvider.manifest.status.podUsedUID
+                toFieldPath: status.managed.podUsedUID
+              - type: ToCompositeFieldPath
+                fromFieldPath: status.atProvider.manifest.status.retries
+                toFieldPath: status.managed.retries