Issues creating inventory from state file with terraform_state plugin

[seanw7]

SUMMARY

I'm running into issues when attempting to retrieve a GitLab remote state file with the cloud.terraform.terraform_state inventory plugin.

I'm able to retrieve it with terraform show cmd without issue, however, I receive an error when attempting to use the terraform_state inventory plugin stderr: Failed to marshal state to json: unsupported attribute "disks", please see example terraform show output below

Any help would be appreciated, thank you!

ISSUE TYPE

• Bug Report

COMPONENT NAME

cloud.terraform.terraform_state

ANSIBLE VERSION

ansible [core 2.16.7]
  config file = None
  configured module search path = ['/home/username/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/username/Documents/lab_automation/venv/lib/python3.11/site-packages/ansible
  ansible collection location = /home/username/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/username/Documents/lab_automation/venv/bin/ansible
  python version = 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] (/home/username/Documents/lab_automation/venv/bin/python3)
  jinja version = 3.1.4
  libyaml = True

COLLECTION VERSION

ansible-galaxy collection list cloud.terraform

# /home/username/.ansible/collections/ansible_collections
Collection      Version
--------------- -------
cloud.terraform 3.0.0 

CONFIGURATION

CONFIG_FILE() = None

OS / ENVIRONMENT

Raspbian running Terraform & Ansible against Proxmox v8

STEPS TO REPRODUCE

# Example terraform_state.yaml
plugin: cloud.terraform.terraform_state
backend_type: http
backend_config:
  address: https://gitlab.com/api/v4/projects/foo/terraform/state/default
  username: REDACTED
  password: REDACTED

# Exerpt from proxmox main.tf where disks block exists
...
  disks {
    scsi {
      scsi0 {
        disk {
          size    = "40"
          storage = "local-lvm"
        }
      }
    }
  }
...

# Example terraform show output 
terraform show -json
{"format_version":"1.0","terraform_version":"1.8.4","values":{"outputs":{"vm_ips":{"sensitive":false,"value":{"k3s-01-agent-01":"192.168.1.144","k3s-01-server-01":"192.168.1.143"},"type":["object",{"k3s-01-agent-01":"string","k3s-01-server-01":"string"}]}},"root_module":{"resources":[{"address":"data.terraform_remote_state.dev-tfstate","mode":"data","type":"terraform_remote_state","name":"dev-tfstate","provider_name":"terraform.io/builtin/terraform","schema_version":0,"values":{"backend":"http","config":{"address":"https://gitlab.com/api/v4/projects/foo/terraform/state/default","password":"REDACTED","username":"username"},"defaults":null,"outputs":{"vm_ips":{"k3s-01-agent-01":"192.168.1.144","k3s-01-server-01":"192.168.1.143"}},"workspace":null},"sensitive_values":{"config":{},"outputs":{"vm_ips":{}}}}],"child_modules":[{"resources":[{"address":"module.terraform-proxmox-k3s.proxmox_vm_qemu.cloudinit-vm[\"k3s-01-agent-01\"]","mode":"managed","type":"proxmox_vm_qemu","name":"cloudinit-vm","index":"k3s-01-agent-01","provider_name":"registry.terraform.io/telmate/proxmox","schema_version":0,"values":{"additional_wait":5,"agent":1,"args":"","automatic_reboot":true,"balloon":4096,"bios":"seabios","boot":"order=scsi0;net0;ide2","bootdisk":"","bridge":null,"ci_wait":null,"cicustom":"","cipassword":"password","ciuser":"username","clone":"vm101","clone_wait":10,"cloudinit_cdrom_storage":"local-lvm","cores":4,"cpu":"x86-64-v2-AES","default_ipv4_address":"192.168.1.144","define_connection_info":true,"desc":"k3s-01-agent-01","disk":[],"disk_gb":null,"disks":[{"ide":[],"sata":[],"scsi":[{"scsi0":[{"cdrom":[],"disk":[{"asyncio":"","backup":true,"cache":"","discard":false,"emulatessd":false,"format":"raw","id":0,"iops_r_burst":0,"iops_r_burst_length":0,"iops_r_concurrent":0,"iops_wr_burst":0,"iops_wr_burst_length":0,"iops_wr_concurrent":0,"iothread":false,"linked_disk_id":-1,"mbps_r_burst":0,"mbps_r_concurrent":0,"mbps_wr_burst":0,"mbps_wr_concurrent":0,"readonly":false,"replicate":false,"serial":"","size":40,"storage":"local-lvm","wwn":""}],"passthrough":[]}],"scsi1":[],"scsi10":[],"scsi11":[],"scsi12":[],"scsi13":[],"scsi14":[],"scsi15":[],"scsi16":[],"scsi17":[],"scsi18":[],"scsi19":[],"scsi2":[],"scsi20":[],"scsi21":[],"scsi22":[],"scsi23":[],"scsi24":[],"scsi25":[],"scsi26":[],"scsi27":[],"scsi28":[],"scsi29":[],"scsi3":[],"scsi30":[],"scsi4":[],"scsi5":[],"scsi6":[],"scsi7":[],"scsi8":[],"scsi9":[]}],"virtio":[]}],"efidisk":[],"force_create":false,"force_recreate_on_change_of":null,"full_clone":true,"guest_agent_ready_timeout":100,"hagroup":"","hastate":"","hostpci":[],"hotplug":"network,disk,usb","id":"proxmox-01/qemu/104","ipconfig0":"ip=192.168.1.144/24,gw=192.168.1.254","ipconfig1":"","ipconfig10":"","ipconfig11":"","ipconfig12":"","ipconfig13":"","ipconfig14":"","ipconfig15":"","ipconfig2":"","ipconfig3":"","ipconfig4":"","ipconfig5":"","ipconfig6":"","ipconfig7":"","ipconfig8":"","ipconfig9":"","iso":"","kvm":true,"linked_vmid":0,"mac":null,"machine":"","memory":4096,"name":"k3s-01-agent-01","nameserver":"192.168.1.253","network":[{"bridge":"vmbr0","firewall":false,"link_down":false,"macaddr":"FOO","model":"virtio","mtu":0,"queues":0,"rate":0,"tag":-1}],"nic":null,"numa":false,"onboot":false,"oncreate":false,"os_network_config":null,"os_type":"cloud-init","pool":"","preprovision":true,"pxe":null,"qemu_os":"l26","reboot_required":false,"scsihw":"virtio-scsi-single","searchdomain":"home.arpa","serial":[],"smbios":[{"family":"","manufacturer":"","product":"","serial":"","sku":"","uuid":"foo","version":""}],"sockets":2,"ssh_forward_ip":null,"ssh_host":"192.168.1.144","ssh_port":"22","ssh_private_key":null,"ssh_user":null,"sshkeys":"ssh-rsa ","startup":"","storage":null,"storage_type":null,"tablet":true,"tags":"k3s-01;k3s-agent","target_node":"proxmox-01","target_nodes":null,"timeouts":null,"unused_disk":[],"usb":[],"vcpus":0,"vga":[],"vlan":-1,"vm_state":"running","vmid":null},"sensitive_values":{"cipassword":true,"disk":[],"disks":[{"ide":[],"sata":[],"scsi":[{"scsi0":[{"cdrom":[],"disk":[{}],"passthrough":[]}],"scsi1":[],"scsi10":[],"scsi11":[],"scsi12":[],"scsi13":[],"scsi14":[],"scsi15":[],"scsi16":[],"scsi17":[],"scsi18":[],"scsi19":[],"scsi2":[],"scsi20":[],"scsi21":[],"scsi22":[],"scsi23":[],"scsi24":[],"scsi25":[],"scsi26":[],"scsi27":[],"scsi28":[],"scsi29":[],"scsi3":[],"scsi30":[],"scsi4":[],"scsi5":[],"scsi6":[],"scsi7":[],"scsi8":[],"scsi9":[]}],"virtio":[]}],"efidisk":[],"hostpci":[],"network":[{}],"serial":[],"smbios":[{}],"ssh_private_key":true,"unused_disk":[],"usb":[],"vga":[]},"tainted":true},
...
}

EXPECTED RESULTS

Expected to see an inventory of the two VMs that are in my remote state file

ACTUAL RESULTS

ansible-inventory [core 2.16.7]
  config file = None
  configured module search path = ['/home/username/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/username/Documents/venv/lib/python3.11/site-packages/ansible
  ansible collection location = /home/username/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/username/Documents/venv/bin/ansible-inventory
  python version = 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] (/home/username/Documents/venv/bin/python3)
  jinja version = 3.1.4
  libyaml = True
No config file found; using defaults
setting up inventory plugins
Loading collection ansible.builtin from 
host_list declined parsing /home/username/Documents/tf/applications/k3s-module/terraform_state.yaml as it did not pass its verify_file() method
script declined parsing /home/username/Documents/tf/applications/k3s-module/terraform_state.yaml as it did not pass its verify_file() method
Loading collection cloud.terraform from /home/username/.ansible/collections/ansible_collections/cloud/terraform
Using inventory plugin 'ansible_collections.cloud.terraform.plugins.inventory.terraform_state' to process inventory source '/home/username/Documents/tf/applications/k3s-module/terraform_state.yaml'
toml declined parsing /home/username/Documents/tf/applications/k3s-module/terraform_state.yaml as it did not pass its verify_file() method
[WARNING]:  * Failed to parse /home/username/Documents/tf/applications/k3s-module/terraform_state.yaml with auto plugin: Could not get
Terraform show from path: /tmp/tmphg4ifq1m.  stdout:  stderr: Failed to marshal state to json: unsupported attribute "disks"
  File "/home/username/Documents/venv/lib/python3.11/site-packages/ansible/inventory/manager.py", line 293, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/home/username/Documents/venv/lib/python3.11/site-packages/ansible/plugins/inventory/auto.py", line 59, in parse
    plugin.parse(inventory, loader, path, cache=cache)
  File "/home/username/.ansible/collections/ansible_collections/cloud/terraform/plugins/inventory/terraform_state.py", line 573, in parse
    instances = self._query(
                ^^^^^^^^^^^^
  File "/home/username/.ansible/collections/ansible_collections/cloud/terraform/plugins/inventory/terraform_state.py", line 507, in _query
    raise TerraformError(e.message)
[WARNING]:  * Failed to parse /home/username/Documents/tf/applications/k3s-module/terraform_state.yaml with yaml plugin: Plugin configuration
YAML file, not YAML inventory
  File "/home/username/Documents/venv/lib/python3.11/site-packages/ansible/inventory/manager.py", line 293, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/home/username/Documents/venv/lib/python3.11/site-packages/ansible/plugins/inventory/yaml.py", line 114, in parse
    raise AnsibleParserError('Plugin configuration YAML file, not YAML inventory')
[WARNING]:  * Failed to parse /home/username/Documents/tf/applications/k3s-module/terraform_state.yaml with ini plugin: Invalid host pattern
'plugin:' supplied, ending in ':' is not allowed, this character is reserved to provide a port.
  File "/home/username/Documents/venv/lib/python3.11/site-packages/ansible/inventory/manager.py", line 293, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/home/username/Documents/venv/lib/python3.11/site-packages/ansible/plugins/inventory/ini.py", line 138, in parse
    raise AnsibleParserError(e)
[WARNING]: Unable to parse /home/username/Documents/tf/applications/k3s-module/terraform_state.yaml as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
@all:
  |--@ungrouped:

[gravesm]

Would you mind retrying with main branch? The way the terraform_state plugin handles getting and reading state was changed a couple weeks ago. It's possible this may also fix your problem.