feat: Improve export-terraform function to handle additional resources (

#767)

* feat: add support for organization and folder IAM

* extract adding files into a separate function

* attach versions file to exported Terraform

* add support for exporting projects

* add variable for org_id

* Add README to exported Terraform

* update example output

* Update README

* Add default value for variables

* Always output org_id for projects

* Make billing_account into a variable

* Add documentation to terraformResource

* fix switch in resources.go

* Stop hiding Kind in path

* Add support for IAMPartialPolicy

* Add support for IAMPolicy

* add test for resource parentRef finder

* Add comment about refPath

examples/export-terraform-advanced/.expected/config.yaml

@@ -0,0 +1,2 @@
+testType: eval
+image: gcr.io/kpt-fn/export-terraform:unstable

examples/export-terraform-advanced/.expected/diff.patch

@@ -0,0 +1,80 @@
+diff --git a/terraform.yaml b/terraform.yaml
+new file mode 100644
+index 0000000..c79a7cd
+--- /dev/null
++++ b/terraform.yaml
+@@ -0,0 +1,74 @@
++apiVersion: v1
++kind: ConfigMap
++metadata:
++  name: terraform
++  annotations:
++    blueprints.cloud.google.com/flavor: terraform
++    blueprints.cloud.google.com/syntax: hcl
++    config.kubernetes.io/local-config: "true"
++data:
++  README.md: |
++    # Google Cloud Foundation Blueprint
++
++    This directory contains Terraform configuration for a foundational environment on Google Cloud.
++
++    It includes a subset of resources configured via the [setup checklist](https://cloud.google.com/docs/enterprise/setup-checklist)
++    and is based on the [security foundations blueprint](https://cloud.google.com/architecture/security-foundations).
++
++    ## Prerequisites
++
++    To run the commands described in this document, you need the following:
++
++    1. Install the [Google Cloud SDK](https://cloud.google.com/sdk/install) version 319.0.0 or later
++    1. Install [Terraform](https://www.terraform.io/downloads.html) version 0.13.7 or later.
++    1. Set up a Google Cloud
++       [organization](https://cloud.google.com/resource-manager/docs/creating-managing-organization).
++    1. Set up a Google Cloud
++       [billing account](https://cloud.google.com/billing/docs/how-to/manage-billing-account).
++    1. For the user who will run the Terraform install, grant the
++       following roles:
++       -  The `roles/billing.admin` role on the billing account.
++       -  The `roles/resourcemanager.organizationAdmin` role on the Google
++          Cloud organization.
++       -  The `roles/resourcemanager.folderCreator` role on the Google
++          Cloud organization.
++       -  The `roles/resourcemanager.projectCreator` role on the Google
++          Cloud organization.
++
++    ## Deploying
++
++    1. Run `terraform init`.
++    1. Run `terraform plan` and review the output.
++    1. Run `terraform apply`.
++
++    ## Next steps
++
++    Once you have the basic foundation deployed, you should explore:
++    1. Building an [advanced foundation](https://github.com/terraform-google-modules/terraform-example-foundation) using the security blueprint
++    2. Automatically deploying Terraform with [Cloud Build](https://cloud.google.com/architecture/managing-infrastructure-as-code)
++  folders.tf: |
++    resource "google_folder" "test" {
++      display_name = "Test Display"
++      parent       = "organizations/${var.org_id}"
++    }
++  iam.tf: "module \"organization-iam\" {\n  source  = \"terraform-google-modules/iam/google//modules/organizations_iam\"\n  version = \"~> 7.4\"\n\n  organizations = [\"11111111111\"]\n\n  bindings = {\n    \n    \"roles/editor\" = [\n      \"group:[email protected]\",\n      \"group:[email protected]\",\n    ]\n    \n    \"roles/orgpolicy.policyAdmin\" = [\n      \"group:[email protected]\",\n    ]\n    \n  }\n}\n\n\nmodule \"folder-1-iam\" {\n  source  = \"terraform-google-modules/iam/google//modules/folders_iam\"\n  version = \"~> 7.4\"\n\n  folders = [\"folders/335620346181\"]\n\n  bindings = {\n    \n    \"roles/viewer\" = [\n      \"group:[email protected]\",\n    ]\n    \n  }\n}\n\n\nmodule \"test-iam\" {\n  source  = \"terraform-google-modules/iam/google//modules/folders_iam\"\n  version = \"~> 7.4\"\n\n  folders = [google_folder.test.name]\n\n  bindings = {\n    \n    \"roles/viewer\" = [\n      \"group:[email protected]\",\n    ]\n    \n  }\n}\n"
++  variables.tf: |
++    variable "org_id" {
++      description = "The organization id for the associated resources"
++      type        = string
++      default     = "11111111111"
++    }
++  versions.tf: |
++    terraform {
++      required_version = ">=0.13"
++
++      required_providers {
++        google = {
++          source  = "hashicorp/google"
++          version = ">= 4.0.0"
++        }
++      }
++      provider_meta "google" {
++        module_name = "blueprints/terraform/exported-krm/v0.1.0"
++      }
++    }

examples/export-terraform-advanced/.expected/results.yaml

@@ -0,0 +1,8 @@
+apiVersion: kpt.dev/v1
+kind: FunctionResultList
+metadata:
+  name: fnresults
+exitCode: 0
+items:
+  - image: gcr.io/kpt-fn/export-terraform:unstable
+    exitCode: 0

examples/export-terraform-advanced/.krmignore

@@ -0,0 +1 @@
+.expected

examples/export-terraform-advanced/README.md

@@ -0,0 +1,30 @@
+# export-terraform: Advanced Example
+
+### Overview
+
+In this example, we will see how to export Terraform configuration from a complex blueprint with many KCC resources.
+
+### Fetch the example package
+
+Get the example package by running the following commands:
+
+```shell
+$ kpt pkg get https://github.com/GoogleContainerTools/kpt-functions-catalog.git/examples/export-terraform-advanced
+```
+
+### Function invocation
+
+Invoke the function by running the following commands:
+
+```shell
+$ kpt fn eval export-terraform-advanced --image gcr.io/kpt-fn/export-terraform:unstable
+```
+
+### Expected result
+The function should export successfully
+```shell
+[RUNNING] "gcr.io/kpt-fn/export-terraform:unstable"
+[PASS] "gcr.io/kpt-fn/export-terraform:unstable" in 1.5s
+```
+
+A `ConfigMap` will be placed in `terraform.yaml` which contains the converted Terraform code.

examples/export-terraform-advanced/folder_external.yaml

@@ -0,0 +1,25 @@
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+apiVersion: iam.cnrm.cloud.google.com/v1beta1
+kind: IAMPolicyMember
+metadata:
+  name: external-viewer
+  namespace: config-control
+spec:
+  resourceRef:
+    apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
+    kind: Folder
+    external: "335620346181"
+  role: roles/viewer
+  member: group:[email protected]

examples/export-terraform-advanced/folder_ref.yaml

@@ -0,0 +1,36 @@
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
+kind: Folder
+metadata:
+  name: test
+  namespace: hierarchy
+spec:
+  displayName: Test Display
+  organizationRef:
+    external: '11111111111'
+---
+apiVersion: iam.cnrm.cloud.google.com/v1beta1
+kind: IAMPolicyMember
+metadata:
+  name: folder-viewer
+  namespace: config-control
+spec:
+  resourceRef:
+    apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
+    kind: Folder
+    name: test
+    namespace: hierarchy
+  role: roles/viewer
+  member: group:[email protected]

examples/export-terraform-advanced/org_iam.yaml

@@ -0,0 +1,38 @@
+apiVersion: iam.cnrm.cloud.google.com/v1beta1
+kind: IAMPolicyMember
+metadata:
+  name: gcp-organization-adminsorgpolicy-policyAdmin
+  namespace: config-control
+spec:
+  resourceRef:
+    apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
+    kind: Organization
+    external: "11111111111"
+  role: roles/orgpolicy.policyAdmin
+  member: group:[email protected]
+---
+apiVersion: iam.cnrm.cloud.google.com/v1beta1
+kind: IAMPolicyMember
+metadata:
+  name: gcp-organization-adminsorgpolicy-editor
+  namespace: config-control
+spec:
+  resourceRef:
+    apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
+    kind: Organization
+    external: "11111111111"
+  role: roles/editor
+  member: group:[email protected]
+---
+apiVersion: iam.cnrm.cloud.google.com/v1beta1
+kind: IAMPolicyMember
+metadata:
+  name: gcp-devs-editor
+  namespace: config-control
+spec:
+  resourceRef:
+    apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
+    kind: Organization
+    external: "11111111111"
+  role: roles/editor
+  member: group:[email protected]

examples/export-terraform-imperative/.expected/diff.patch

@@ -1,9 +1,9 @@
 diff --git a/terraform.yaml b/terraform.yaml
 new file mode 100644
-index 0000000..71490e8
+index 0000000..60fa1a3
 --- /dev/null
 +++ b/terraform.yaml
-@@ -0,0 +1,20 @@
+@@ -0,0 +1,78 @@
 +apiVersion: v1
 +kind: ConfigMap
 +metadata:
@@ -13,14 +13,72 @@ index 0000000..71490e8
 +    blueprints.cloud.google.com/syntax: hcl
 +    config.kubernetes.io/local-config: "true"
 +data:
-+  folders.tf: |+
++  README.md: |
++    # Google Cloud Foundation Blueprint
++
++    This directory contains Terraform configuration for a foundational environment on Google Cloud.
++
++    It includes a subset of resources configured via the [setup checklist](https://cloud.google.com/docs/enterprise/setup-checklist)
++    and is based on the [security foundations blueprint](https://cloud.google.com/architecture/security-foundations).
++
++    ## Prerequisites
++
++    To run the commands described in this document, you need the following:
++
++    1. Install the [Google Cloud SDK](https://cloud.google.com/sdk/install) version 319.0.0 or later
++    1. Install [Terraform](https://www.terraform.io/downloads.html) version 0.13.7 or later.
++    1. Set up a Google Cloud
++       [organization](https://cloud.google.com/resource-manager/docs/creating-managing-organization).
++    1. Set up a Google Cloud
++       [billing account](https://cloud.google.com/billing/docs/how-to/manage-billing-account).
++    1. For the user who will run the Terraform install, grant the
++       following roles:
++       -  The `roles/billing.admin` role on the billing account.
++       -  The `roles/resourcemanager.organizationAdmin` role on the Google
++          Cloud organization.
++       -  The `roles/resourcemanager.folderCreator` role on the Google
++          Cloud organization.
++       -  The `roles/resourcemanager.projectCreator` role on the Google
++          Cloud organization.
++
++    ## Deploying
++
++    1. Run `terraform init`.
++    1. Run `terraform plan` and review the output.
++    1. Run `terraform apply`.
++
++    ## Next steps
++
++    Once you have the basic foundation deployed, you should explore:
++    1. Building an [advanced foundation](https://github.com/terraform-google-modules/terraform-example-foundation) using the security blueprint
++    2. Automatically deploying Terraform with [Cloud Build](https://cloud.google.com/architecture/managing-infrastructure-as-code)
++  folders.tf: |
 +    resource "google_folder" "child-folder" {
 +      display_name = "child-folder"
 +      parent       = google_folder.parent-folder.name
 +    }
 +
 +    resource "google_folder" "parent-folder" {
 +      display_name = "parent folder"
-+      parent       = "organizations/123456789012"
++      parent       = "organizations/${var.org_id}"
 +    }
++  variables.tf: |
++    variable "org_id" {
++      description = "The organization id for the associated resources"
++      type        = string
++      default     = "123456789012"
++    }
++  versions.tf: |
++    terraform {
++      required_version = ">=0.13"
 +
++      required_providers {
++        google = {
++          source  = "hashicorp/google"
++          version = ">= 4.0.0"
++        }
++      }
++      provider_meta "google" {
++        module_name = "blueprints/terraform/exported-krm/v0.1.0"
++      }
++    }

examples/export-terraform-imperative/.expected/results.yaml

@@ -5,7 +5,4 @@ metadata:
 exitCode: 0
 items:
   - image: gcr.io/kpt-fn/export-terraform:unstable
-    stderr: |-
-      found matching template for Folder/child-folder: true
-      found matching template for Folder/parent-folder: true
     exitCode: 0

functions/go/export-terraform/README.md

@@ -13,6 +13,13 @@ This function generates idiomatic Terraform configuration by parsing [Config Con
 Where appropriate, the generated Terraform references [Cloud Foundation Toolkit modules](https://g.co/dev/terraformfoundation).
 The goal is to make the generated output as close to possible as what a human would have written.
 
+The following KCC resources are supported:
+- Folder
+- Project
+- IAMPartialPolicy
+- IAMPolicy
+- IAMPolicyMember
+
 The output Terraform will be saved to a `ConfigMap` in `terraform.yaml` at the root of the package.
 Each key in the `ConfigMap` corresponds to a different file which is part of the Terraform module.
 

functions/go/export-terraform/metadata.yaml

@@ -21,6 +21,7 @@ tags:
 sourceURL: https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/master/functions/go/export-terraform
 examplePackageURLs:
   - https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/master/examples/export-terraform-imperative
+  - https://github.com/GoogleContainerTools/kpt-functions-catalog/tree/master/examples/export-terraform-advanced
 emails:
   - [email protected]
 license: Apache-2.0