Can't read inventory from remote http state with terraform_state

[Lyannic]

SUMMARY

I can't read the inventory with the terraform_state plugin from a remote Terraform state that is stored at GitLab. The resources are created with the Hetzner Terraform provider. I don't get any error messages, but the resulting inventory is just empty.

ISSUE TYPE

• Bug Report

COMPONENT NAME

cloud.terraform.terraform_state

ANSIBLE VERSION

ansible [core 2.17.3]
  config file = None
  configured module search path = ['/home/yannic/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/yannic/.local/lib/python3.10/site-packages/ansible
  ansible collection location = /home/yannic/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/yannic/.local/bin/ansible
  python version = 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] (/usr/bin/python3)
  jinja version = 3.1.3
  libyaml = True

COLLECTION VERSION

# /home/yannic/.ansible/collections/ansible_collections
Collection      Version   
--------------- ----------
cloud.terraform 4.0.0-dev0

CONFIGURATION

CONFIG_FILE() = None

OS / ENVIRONMENT

Ubuntu 22.04.4 LTS

STEPS TO REPRODUCE

I created a minimal example for my use-case.

main.tf

resource "hcloud_server" "server" {
  name         = "dyn-inv-test-server"
  image        = "docker-ce"
  server_type  = "cx22"
  location     = "fsn1"
}

resource "ansible_group" "docker_host_group" {
  name     = "docker_host"
}

resource "ansible_host" "server" {
  name   = hcloud_server.server.ipv4_address
  groups = [ansible_group.docker_host_group.name]
}

providers.tf

terraform {
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = "1.47.0"
    }
    ansible = {
      source = "ansible/ansible"
      version = "1.3.0"
    }
  }
}

provider "hcloud" {
  token = var.hcloud_token
}

backend.tf

terraform {
  backend "http" {
  }
}

terraform_state.yml

plugin: cloud.terraform.terraform_state
binary_path: /usr/bin/tofu
backend_type: http
search_child_modules: true
backend_config:
  address: https://gitlab.com/api/v4/projects/CENSORED/terraform/state/default
  lock_address: https://gitlab.com/api/v4/projects/CENSORED/terraform/state/default/lock
  unlock_address: https://gitlab.com/api/v4/projects/CENSORED/terraform/state/default/lock
  lock_method: POST
  unlock_method: DELETE
  retry_wait_min: 5
  username: CENSORED
  password: CENSORED

EXPECTED RESULTS

An inventory with the server that I created in main.tf. Here is the result that I get from cloud.terraform.terraform_provider

terraform_provider.yml

plugin: cloud.terraform.terraform_provider
binary_path: /usr/bin/tofu

ansible-inventory [core 2.17.3]
  config file = None
  configured module search path = ['/home/yannic/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/yannic/.local/lib/python3.10/site-packages/ansible
  ansible collection location = /home/yannic/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/yannic/.local/bin/ansible-inventory
  python version = 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] (/usr/bin/python3)
  jinja version = 3.1.3
  libyaml = True
No config file found; using defaults
setting up inventory plugins
Loading collection ansible.builtin from 
host_list declined parsing /home/yannic/projects/server-provisioning-dynamic-inventory-test/terraform_provider.yml as it did not pass its verify_file() method
script declined parsing /home/yannic/projects/server-provisioning-dynamic-inventory-test/terraform_provider.yml as it did not pass its verify_file() method
Loading collection cloud.terraform from /home/yannic/.ansible/collections/ansible_collections/cloud/terraform
Using inventory plugin 'ansible_collections.cloud.terraform.plugins.inventory.terraform_provider' to process inventory source '/home/yannic/projects/server-provisioning-dynamic-inventory-test/terraform_provider.yml'
Parsed /home/yannic/projects/server-provisioning-dynamic-inventory-test/terraform_provider.yml inventory source with auto plugin
{
    "_meta": {
        "hostvars": {}
    },
    "all": {
        "children": [
            "ungrouped",
            "docker_host"
        ]
    },
    "docker_host": {
        "hosts": [
            "76.32.12.154"
        ]
    }
}

Since I don't specify a state_file, terraform_provider is using the remote state too, which has the same config values as terraform_state.yml

terraform.tfstate

{
    "version": 3,
    "serial": 1,
    "lineage": "949ca244-3db7-4aaf-c5e5-03fb7126674b",
    "backend": {
        "type": "http",
        "config": {
            "address": "https://gitlab.com/api/v4/projects/CENSORED/terraform/state/default",
            "client_ca_certificate_pem": null,
            "client_certificate_pem": null,
            "client_private_key_pem": null,
            "headers": null,
            "lock_address": "https://gitlab.com/api/v4/projects/CENSORED/terraform/state/default/lock",
            "lock_method": "POST",
            "password": "CENSORED",
            "retry_max": null,
            "retry_wait_max": null,
            "retry_wait_min": 5,
            "skip_cert_verification": null,
            "unlock_address": "https://gitlab.com/api/v4/projects/CENSORED/terraform/state/default/lock",
            "unlock_method": "DELETE",
            "update_method": null,
            "username": "CENSORED"
        },
        "hash": 2984870146
    },
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {},
            "resources": {},
            "depends_on": []
        }
    ]
}

ACTUAL RESULTS

output from ansible-inventory -vvvvvv -i terraform_state.yml --list

ansible-inventory [core 2.17.3]
  config file = None
  configured module search path = ['/home/yannic/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/yannic/.local/lib/python3.10/site-packages/ansible
  ansible collection location = /home/yannic/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/yannic/.local/bin/ansible-inventory
  python version = 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] (/usr/bin/python3)
  jinja version = 3.1.3
  libyaml = True
No config file found; using defaults
setting up inventory plugins
Loading collection ansible.builtin from 
host_list declined parsing /home/yannic/projects/server-provisioning-dynamic-inventory-test/terraform_state.yml as it did not pass its verify_file() method
script declined parsing /home/yannic/projects/server-provisioning-dynamic-inventory-test/terraform_state.yml as it did not pass its verify_file() method
Loading collection cloud.terraform from /home/yannic/.ansible/collections/ansible_collections/cloud/terraform
Using inventory plugin 'ansible_collections.cloud.terraform.plugins.inventory.terraform_state' to process inventory source '/home/yannic/projects/server-provisioning-dynamic-inventory-test/terraform_state.yml'
Parsed /home/yannic/projects/server-provisioning-dynamic-inventory-test/terraform_state.yml inventory source with auto plugin
{
    "_meta": {
        "hostvars": {}
    },
    "all": {
        "children": [
            "ungrouped"
        ]
    }
}

[dr-acul]

I think you are using a custom provider without specifying the provider_mapping (#146).

Adding the following code to my terraform_state.yml made it partially work:

hostnames:
  - ipv4_address
provider_mapping:
  - provider_name: registry.opentofu.org/hetznercloud/hcloud
    types:
      - hcloud_server

This does not use the ansible_host resource, and you probably have to play around with the mapping.