[FEATURE] Automate dependency upgrade with popular solutions (renovate, github dependabot) inside terramate code generation

[luong-komorebi]

Is your feature request related to a problem? Please describe.

• As a user, I want my dependencies to be automatically updated and comprehended by an automation system so that my infrastructure components are updated with latest versions (security, bug fixes, new features)
• As a user, I want to generate "required_providers" code by terramate so that versions are consistent through out stacks

Combining the two stories above, I believe it makes sense for us to support terramate as a language in big dependency automation system.
Right now, the main blocker is when versions are put into terramate code generation, when these systems automatically update dependencies, they can only updated the generated code, not the main code, leading to automation mostly fails with terramate run because generated codes now diverged from source

Describe the solution you'd like
The best solution would be contributing to the upstream of these automation system.
For inspiration,

• Rennovate and terragrunt: docs, relevant codes
• Github dependabot: relevant codes

Describe alternatives you've considered
Read a terraform string from a file then somehow passed it into "content" block, so that we can keep dependency a raw hcl block which can be read by the above system. This is not successful
Leave versions outside of generated code, probably in lockfile or somewhere else. This defeats purpose of versions, especially required_versions

Additional context
Add any other context or screenshots about the feature request here.

[luong-komorebi]

With the suggestion of a user in our community discord named shmilee, I have successfully created a config of Renovate to make this work.

It looks like this :
(config filename renovate.json)

  "packageRules": [
    {
      "matchManagers": [
        "terraform",
        "regex"
      ],
      "groupName": "Terraform for terramate stacks",
      "matchDatasources": [
        "terraform-provider"
      ],
      "matchDepTypes": [
        "required_provider"
      ]
    }
  ],
  "customManagers": [
    {
      "customType": "regex",
      "description": "Update version in terramate",
      "fileMatch": [
        "generate_providers.tm.hcl$"
      ],
      "matchStringsStrategy": "any",
      "matchStrings": [
        "source\\s*=\\s*\"(?<namespace>.*?)/(?<depName>.*?)\"\\r?\\n\\s*version\\s*=\\s*\"(?<currentValue>.*?)\""
      ],
      "datasourceTemplate": "terraform-provider",
      "depTypeTemplate": "required_provider",
      "packageNameTemplate": "{{namespace}}/{{depName}}"
    }
  ]

With the manipulation of a custom manager and grouping with package rules , I was able to upgrade terramate code generation along with generated codes

---

Given a terramate import like this

(filename generated_providers.tm.hcl) - inspired by https://github.com/terramate-io/terramate-examples/blob/b8180a3be50cf6049e0e78ae0ef48a9e4e1aeaae/01-keep-terraform-dry/imports/generate_providers.tm.hcl

generate_hcl "_terramate_generated_provider.tf" {
  content {
    terraform {
      required_version = global.terraform.version

      required_providers {
        aws = {
          source  = "hashicorp/aws"
          version = "~> 5.67.0"
        }
        kubectl = {
          source  = "alekc/kubectl"
          version = "~> 2.0.4"
        }
        kubernetes = {
          source  = "hashicorp/kubernetes"
          version = "~> 2.32.0"
        }
        random = {
          source  = "hashicorp/random"
          version = "~> 3.6.0"
        }
        helm = {
          source  = "hashicorp/helm"
          version = "~> 2.15.0"
        }
      }
    }
    provider "random" {}
  }
}

the version upgrade is able to understand and generate a pull request that looks like this

which automatically updates the version in the original codes and the lockfile as well

Terramate Before	Terramate After
	Everything works well

[luong-komorebi]

Another possible solution I can think of is to put providers in tmgen

and then extends the original terraform matcher of renovate to match tmgen file

{
  "terraform": {
    "fileMatch": [".tf.tmgen$"]
  }
}

I havent tried this out to guarantee that this will work. I have tried extending terraform filematch to match .tm.hcl but it was not working