feat(terraform): support for remote terraform state, introduce managed redis and secrets manager (#274)

This pull request introduces several major improvements and new
resources to the Terraform infrastructure codebase, focusing on enhanced
state management, expanded cloud resource provisioning, and improved
secrets handling. The changes add support for remote Terraform state via
S3-compatible object storage, automate backend bootstrapping, and
introduce managed Redis and Secrets Manager resources. Additional
updates improve cluster configuration and documentation.

**Terraform State Management & Automation:**

- Added support for using an S3-compatible backend for Terraform state,
including a new `tfstate` object storage bucket, output wiring, and
documentation on state management. A helper script (`init-backend.sh`)
was introduced to automate backend initialization and migration,
generating a `.backend.hcl` file with the necessary credentials and
configuration.
(`[[1]](diffhunk://#diff-40e942f521b179f4b67af29e0186e895becd783b1d994f74afdaa204a4007eafR1-R16)`,
`[[2]](diffhunk://#diff-40e942f521b179f4b67af29e0186e895becd783b1d994f74afdaa204a4007eafL33-R44)`,
`[[3]](diffhunk://#diff-951d6ab4b0142466865bc9a073ac82641fd19b6fa1267f65e82d5b827922eaecR95-R157)`,
`[[4]](diffhunk://#diff-be3ec119082ecec13a5ec2e74162fd5d059cb933742745167663003e8f5ccd55R1-R66)`,
`[[5]](diffhunk://#diff-b56e9e8eb752928fb506809cc8881dfda6490b1ea830ac6cc0024e37f543c572R2)`,
`[[6]](diffhunk://#diff-2cfe3e1ceb805f812736573a76b766c3cb8da0ea0ac4931d15bb75dc566a846aL4-R5)`)
- Updated `.gitignore` to exclude backend configuration and kubeconfig
files from version control.
(`[infrastructure/.gitignoreL4-R5](diffhunk://#diff-2cfe3e1ceb805f812736573a76b766c3cb8da0ea0ac4931d15bb75dc566a846aL4-R5)`)

**New Cloud Resources:**

- Added managed Redis provisioning, including instance and credential
resources, and corresponding input variables for version and plan
selection.
(`[[1]](diffhunk://#diff-f116a20752cd128cc4f5a85ea3b01e4acbf9fabbfdcdd04dfc9901e3def7b326R1-R18)`,
`[[2]](diffhunk://#diff-9772d64123f334ac306e54c19018864cc1451e7e4fe5f14658783372750250f1L39-R63)`)
- Added support for STACKIT Secrets Manager, provisioning an instance
and user, with outputs for integration.
(`[[1]](diffhunk://#diff-800eb980bf14a2c09d182f500ef8cb884eeb18ebe50eadb0b682463c61ba2f58R1-R24)`,
`[[2]](diffhunk://#diff-9772d64123f334ac306e54c19018864cc1451e7e4fe5f14658783372750250f1L39-R63)`)
- Added a model serving token resource and output for AI Model Serving
API integration.
(`[infrastructure/terraform/model_serving.tfR1-R12](diffhunk://#diff-12cf4786858eaf9635d3e45f439444fc5e956e3fb7407b09cf512cad83d2bda5R1-R12)`)

**Secrets Management & Seeding:**

- Introduced a new `seed-secrets` Terraform module with documentation,
example variables, and configuration to seed the Secrets Manager with
required secrets for External Secrets integration.
(`[[1]](diffhunk://#diff-b0300cdd94aa57163b0041cb50ea4990acb4bb8a351079693c26ee64d61fcd72R1-R31)`,
`[[2]](diffhunk://#diff-cb4b240b14f3d9aa644d4260872c9586f77d824aa8f36b910503942f028c1d88R1-R26)`,
`[[3]](diffhunk://#diff-d201981cfa7cef09bee51e1359d030e8fa78f73164e37a306c378c9d6f2d3eb8R1-R29)`,
`[[4]](diffhunk://#diff-2bf98bb86073642173af57afd63a434d44fa3ce87d6ef61916be35585a0ab94fR1-R39)`)

**Cluster & Networking Enhancements:**

- Upgraded the Kubernetes cluster minimum version and improved node pool
specs (larger machine type and disk). Added automatic kubeconfig
generation and output, including writing to `kubeconfig.yaml`.
(`[[1]](diffhunk://#diff-60c4ff86f01efedc7e7e4e8c1cee2e772e458b6b71f9980b342196216bbc0a8dL4-R15)`,
`[[2]](diffhunk://#diff-60c4ff86f01efedc7e7e4e8c1cee2e772e458b6b71f9980b342196216bbc0a8dR31-R56)`)
- Improved DNS zone resource configuration with contact email and
explicit type.
(`[infrastructure/terraform/dns.tfR5-R6](diffhunk://#diff-1c935b36cdab82f9bdd925fecea18d7225ec865f99937585f4897155bd9935f9R5-R6)`)

**Other Improvements:**

- Updated variable descriptions for clarity and adjusted the default
deployment timestamp for resource naming.
(`[[1]](diffhunk://#diff-9772d64123f334ac306e54c19018864cc1451e7e4fe5f14658783372750250f1L7-R7)`,
`[[2]](diffhunk://#diff-9772d64123f334ac306e54c19018864cc1451e7e4fe5f14658783372750250f1L39-R63)`)

---

**Most important changes:**

**Terraform State Management & Automation**
- Added S3-compatible backend support for Terraform state, including a
dedicated `tfstate` bucket, outputs, and documentation. Introduced the
`init-backend.sh` script for automated backend setup and state
migration, generating `.backend.hcl` for credentials/config.
(`[[1]](diffhunk://#diff-40e942f521b179f4b67af29e0186e895becd783b1d994f74afdaa204a4007eafR1-R16)`,
`[[2]](diffhunk://#diff-40e942f521b179f4b67af29e0186e895becd783b1d994f74afdaa204a4007eafL33-R44)`,
`[[3]](diffhunk://#diff-951d6ab4b0142466865bc9a073ac82641fd19b6fa1267f65e82d5b827922eaecR95-R157)`,
`[[4]](diffhunk://#diff-be3ec119082ecec13a5ec2e74162fd5d059cb933742745167663003e8f5ccd55R1-R66)`,
`[[5]](diffhunk://#diff-b56e9e8eb752928fb506809cc8881dfda6490b1ea830ac6cc0024e37f543c572R2)`,
`[[6]](diffhunk://#diff-2cfe3e1ceb805f812736573a76b766c3cb8da0ea0ac4931d15bb75dc566a846aL4-R5)`)

**New Cloud Resources**
- Added managed Redis instance and credential resources, with
configurable version and plan variables.
(`[[1]](diffhunk://#diff-f116a20752cd128cc4f5a85ea3b01e4acbf9fabbfdcdd04dfc9901e3def7b326R1-R18)`,
`[[2]](diffhunk://#diff-9772d64123f334ac306e54c19018864cc1451e7e4fe5f14658783372750250f1L39-R63)`)
- Added STACKIT Secrets Manager instance and user resources, with
outputs for integration.
(`[[1]](diffhunk://#diff-800eb980bf14a2c09d182f500ef8cb884eeb18ebe50eadb0b682463c61ba2f58R1-R24)`,
`[[2]](diffhunk://#diff-9772d64123f334ac306e54c19018864cc1451e7e4fe5f14658783372750250f1L39-R63)`)
- Added model serving token resource and output for AI Model Serving
API.
(`[infrastructure/terraform/model_serving.tfR1-R12](diffhunk://#diff-12cf4786858eaf9635d3e45f439444fc5e956e3fb7407b09cf512cad83d2bda5R1-R12)`)

**Secrets Management & Seeding**
- Introduced the `seed-secrets` module for seeding Secrets Manager with
required secrets, including documentation, example variables, and
configuration for External Secrets.
(`[[1]](diffhunk://#diff-b0300cdd94aa57163b0041cb50ea4990acb4bb8a351079693c26ee64d61fcd72R1-R31)`,
`[[2]](diffhunk://#diff-cb4b240b14f3d9aa644d4260872c9586f77d824aa8f36b910503942f028c1d88R1-R26)`,
`[[3]](diffhunk://#diff-d201981cfa7cef09bee51e1359d030e8fa78f73164e37a306c378c9d6f2d3eb8R1-R29)`,
`[[4]](diffhunk://#diff-2bf98bb86073642173af57afd63a434d44fa3ce87d6ef61916be35585a0ab94fR1-R39)`)

**Cluster & Networking Enhancements**
- Upgraded Kubernetes cluster version and node pool specs, and added
automated kubeconfig generation/output to `kubeconfig.yaml`.
(`[[1]](diffhunk://#diff-60c4ff86f01efedc7e7e4e8c1cee2e772e458b6b71f9980b342196216bbc0a8dL4-R15)`,
`[[2]](diffhunk://#diff-60c4ff86f01efedc7e7e4e8c1cee2e772e458b6b71f9980b342196216bbc0a8dR31-R56)`)
- Improved DNS zone resource with contact email and explicit type.
(`[infrastructure/terraform/dns.tfR5-R6](diffhunk://#diff-1c935b36cdab82f9bdd925fecea18d7225ec865f99937585f4897155bd9935f9R5-R6)`)

Author: a-klos

infrastructure/.gitignore

@@ -1,7 +1,8 @@
 **/cr-secret.yaml
 **/customer-values.yaml
 **/auth
-
+**/.backend.hcl
+**/kubeconfig.yaml
 **/*.lock.*
 
 auth

infrastructure/terraform/README.md

@@ -92,6 +92,69 @@ terraform output -json | jq -r .cluster_name.value
 - **Security**: The sa_key.json file contains sensitive credentials and should never be committed to version control
 - **State Management**: Consider using remote state storage for team environments
 
+## Using the bucket as a Terraform S3 backend (optional)
+
+If you want Terraform state to be stored in the object storage bucket, add a backend block to your root module.
+Note: backend blocks cannot reference resources, so you must hardcode or pass the values via variables/partials.
+
+### Bootstrap script (recommended)
+
+Note: `backend "s3" {}` is already defined in `main.tf`. The bootstrap step still works because it runs `terraform init -backend=false`, which ignores the backend block.
+
+Use the helper script to bootstrap the backend in two phases:
+1) Run a local-only apply to create the bucket + credentials.
+2) Generate `.backend.hcl` and migrate state to S3.
+
+```bash
+./scripts/init-backend.sh
+```
+
+This writes `infrastructure/terraform/.backend.hcl` (contains credentials) and runs `terraform init -force-copy`.
+You can re-run the script at any time; it reuses the existing backend config if present.
+If you want remote state from the start, run this script before your first full `terraform apply`.
+
+If you want non-interactive bootstrap:
+
+```bash
+BOOTSTRAP_AUTO_APPROVE=1 ./scripts/init-backend.sh
+```
+
+Manual phase 1 (if you want to see the exact commands the script runs):
+
+```bash
+terraform init -backend=false
+terraform apply \
+  -target=stackit_objectstorage_bucket.tfstate \
+  -target=stackit_objectstorage_credentials_group.rag_creds_group \
+  -target=stackit_objectstorage_credential.rag_creds
+```
+
+### Manual backend block
+
+```hcl
+terraform {
+  backend "s3" {
+    bucket = "<BUCKET_NAME>"
+    key    = "terraform.tfstate"
+    region = "eu01"
+
+    # Use the same credentials as above
+    access_key = "<ACCESS_KEY>"
+    secret_key = "<SECRET_KEY>"
+
+    endpoints = {
+      s3 = "https://object.storage.eu01.onstackit.cloud"
+    }
+
+    # AWS-specific checks must be disabled for STACKIT
+    skip_credentials_validation = true
+    skip_region_validation      = true
+    skip_s3_checksum            = true
+    skip_requesting_account_id  = true
+  }
+}
+```
+
 ## Cleanup
 
 To destroy all resources:

infrastructure/terraform/dns.tf

@@ -1,7 +1,9 @@
 resource "stackit_dns_zone" "rag_zone" {
-  project_id = var.project_id
-  name       = "${var.name_prefix}-zone"
-  dns_name   = var.dns_name
+  project_id    = var.project_id
+  name          = "${var.name_prefix}-zone"
+  dns_name      = var.dns_name
+  contact_email = "data-ai@stackit.cloud"
+  type          = "primary"
 }
 
 output "dns_nameservers" {

infrastructure/terraform/main.tf

@@ -1,4 +1,5 @@
 terraform {
+  backend "s3" {}
   required_providers {
     stackit = {
       source  = "stackitcloud/stackit"
@@ -9,4 +10,5 @@ terraform {
 
 provider "stackit" {
   service_account_key_path = "sa_key.json"
+  default_region           = "eu01"
 }

infrastructure/terraform/model_serving.tf

@@ -0,0 +1,12 @@
+resource "stackit_modelserving_token" "rag_modelserving" {
+  project_id = var.project_id
+  name       = "${var.name_prefix}-modelserving-token"
+
+  # No ttl_duration set -> token does not expire.
+}
+
+output "model_serving_bearer_token" {
+  description = "Bearer token for AI Model Serving API"
+  value       = stackit_modelserving_token.rag_modelserving.token
+  sensitive   = true
+}

infrastructure/terraform/object_storage.tf

@@ -1,8 +1,19 @@
+# This resource stays stable for 365 days, then changes
+resource "time_rotating" "key_rotation" {
+  rotation_days = 365
+}
+
 resource "stackit_objectstorage_bucket" "documents" {
   name       = "${var.name_prefix}-documents-${var.deployment_timestamp}"
   project_id = var.project_id
 }
 
+resource "stackit_objectstorage_bucket" "tfstate" {
+  name       = "${var.name_prefix}-tfstate-${var.deployment_timestamp}"
+  project_id = var.project_id
+  depends_on = [stackit_objectstorage_credentials_group.rag_creds_group]
+}
+
 resource "stackit_objectstorage_bucket" "langfuse" {
   name       = "${var.name_prefix}-langfuse-${var.deployment_timestamp}"
   project_id = var.project_id
@@ -16,7 +27,7 @@ resource "stackit_objectstorage_credentials_group" "rag_creds_group" {
 resource "stackit_objectstorage_credential" "rag_creds" {
   project_id           = var.project_id
   credentials_group_id = stackit_objectstorage_credentials_group.rag_creds_group.credentials_group_id
-  expiration_timestamp = timeadd(timestamp(), "8760h") # Expires after 1 year
+  expiration_timestamp = timeadd(time_rotating.key_rotation.rfc3339, "8760h")
 }
 
 output "object_storage_access_key" {
@@ -30,5 +41,5 @@ output "object_storage_secret_key" {
 }
 
 output "object_storage_bucket" {
-  value = stackit_objectstorage_bucket.documents.name
+  value = stackit_objectstorage_bucket.tfstate.name
 }

infrastructure/terraform/redis.tf

@@ -0,0 +1,18 @@
+resource "stackit_redis_instance" "rag_redis" {
+  project_id = var.project_id
+  name       = "${var.name_prefix}-redis"
+  version    = var.redis_version
+  plan_name  = var.redis_plan_name
+
+  parameters = {
+    sgw_acl                 = join(",", stackit_ske_cluster.rag_cluster.egress_address_ranges)
+    enable_monitoring       = false
+    down_after_milliseconds = 30000
+  }
+}
+
+
+resource "stackit_redis_credential" "rag_redis_cred" {
+  project_id  = var.project_id
+  instance_id = stackit_redis_instance.rag_redis.instance_id
+}

infrastructure/terraform/scripts/init-backend.sh

@@ -0,0 +1,66 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+root_dir="$(cd "${script_dir}/.." && pwd)"
+
+backend_config_file="${BACKEND_CONFIG_FILE:-${root_dir}/.backend.hcl}"
+auto_approve="${BOOTSTRAP_AUTO_APPROVE:-0}"
+
+cd "${root_dir}"
+
+if ! command -v terraform >/dev/null 2>&1; then
+  echo "terraform is not installed or not in PATH." >&2
+  exit 1
+fi
+
+if [ -f "${backend_config_file}" ]; then
+  terraform init -backend-config="${backend_config_file}"
+  exit 0
+fi
+
+echo "Bootstrapping object storage for Terraform state (local backend)."
+terraform init -backend=false
+
+if ! bucket="$(terraform output -raw object_storage_bucket 2>/dev/null)"; then
+  apply_args=(
+    "-target=stackit_objectstorage_bucket.tfstate"
+    "-target=stackit_objectstorage_credentials_group.rag_creds_group"
+    "-target=stackit_objectstorage_credential.rag_creds"
+    "-target=time_rotating.key_rotation"       # <--- Add this (needed for creds)
+    "-target=output.object_storage_bucket"     # <--- Add this
+    "-target=output.object_storage_access_key" # <--- Add this
+    "-target=output.object_storage_secret_key" # <--- Add this
+  )
+  if [ "${auto_approve}" = "1" ]; then
+    terraform apply -auto-approve "${apply_args[@]}"
+  else
+    terraform apply "${apply_args[@]}"
+  fi
+  bucket="$(terraform output -raw object_storage_bucket)"
+fi
+
+access_key="$(terraform output -raw object_storage_access_key)"
+secret_key="$(terraform output -raw object_storage_secret_key)"
+
+cat > "${backend_config_file}" <<EOF
+bucket = "${bucket}"
+key    = "terraform.tfstate"
+region = "eu01"
+
+access_key = "${access_key}"
+secret_key = "${secret_key}"
+
+endpoints = {
+  s3 = "https://object.storage.eu01.onstackit.cloud"
+}
+
+skip_credentials_validation = true
+skip_region_validation      = true
+skip_s3_checksum            = true
+skip_requesting_account_id  = true
+EOF
+
+chmod 600 "${backend_config_file}"
+
+terraform init -backend-config="${backend_config_file}" -force-copy

infrastructure/terraform/secretsmanager.tf

@@ -0,0 +1,24 @@
+resource "stackit_secretsmanager_instance" "rag_secrets" {
+  project_id = var.project_id
+  name       = "${var.name_prefix}-secrets"
+}
+
+resource "stackit_secretsmanager_user" "rag_secrets_user" {
+  project_id    = var.project_id
+  instance_id   = stackit_secretsmanager_instance.rag_secrets.instance_id
+  description   = var.secretsmanager_user_description
+  write_enabled = var.secretsmanager_user_write_enabled
+}
+
+output "secretsmanager_instance_id" {
+  value = stackit_secretsmanager_instance.rag_secrets.instance_id
+}
+
+output "secretsmanager_username" {
+  value = stackit_secretsmanager_user.rag_secrets_user.username
+}
+
+output "secretsmanager_password" {
+  value     = stackit_secretsmanager_user.rag_secrets_user.password
+  sensitive = true
+}

infrastructure/terraform/seed-secrets/README.md

@@ -0,0 +1,31 @@
+# Seed Secrets Manager data with Terraform
+
+This folder writes the `rag-secrets` KV secret used by External Secrets.
+
+Why this is a separate step:
+- The Secrets Manager instance ID and user credentials are created by the main Terraform stack.
+- Provider blocks cannot depend on resources, so we pass them in via variables and run a second apply.
+
+## Steps
+
+1. Apply the main stack to create the Secrets Manager instance and user.
+2. Copy `terraform.tfvars.example` to `terraform.tfvars` and fill in the values:
+   - `vault_mount_path` is the Secrets Manager instance ID.
+   - `vault_username`/`vault_password` are from the `secretsmanager_*` outputs.
+   - `rag_secrets` should include all keys referenced by your ExternalSecret resources.
+     For the cert-manager webhook, store the service account key JSON under `STACKIT_CERT_MANAGER_SA_JSON` (use a heredoc to avoid escaping).
+3. Run:
+   ```bash
+   terraform init
+   terraform plan
+   terraform apply
+   ```
+
+## Security note
+
+All values written by `vault_kv_secret_v2` are stored in Terraform state. Use a secure backend and restrict access.
+
+## Troubleshooting
+
+If you see a 404 from `auth/token/create`, the backend does not allow child token creation.
+This module sets `skip_child_token = true` so Vault uses the login token directly.