empty ethtool stats values for intel's virtual function

[morpig]

Happy holidays, and sorry for the new issue.

We're trying out Intel's virtual function (VF) feature which allows virtual machines to have direct NIC access without any network bridge to the host OS.

With the VF feature, we're unable to collect network stats for each individual VM's as usual.

Intel with their ixgbe driver allows us to retrieve each VF stats by using ethtool, unforunately, node_exporter is unable to parse any of the VF stats values shown in ethtool.

Anything I can do to debug further?

ethtool -S enp13s0f0
     VF 0 Rx Packets: 54659374
     VF 0 Rx Bytes: 78420771894
     VF 0 Tx Packets: 20317044
     VF 0 Tx Bytes: 15179104572
     VF 0 MC Packets: 3
     VF 1 Rx Packets: 13497449
     VF 1 Rx Bytes: 14482172702
     VF 1 Tx Packets: 20547911
     VF 1 Tx Bytes: 27729733501
     VF 1 MC Packets: 2
     VF 2 Rx Packets: 17775535
     VF 2 Rx Bytes: 22464318558
     VF 2 Tx Packets: 26378658
     VF 2 Tx Bytes: 38787481388
     VF 2 MC Packets: 0

# HELP node_ethtool_vf_0_mc_packets Network interface VF_0_MC_Packets
# TYPE node_ethtool_vf_0_mc_packets untyped
node_ethtool_vf_0_mc_packets{device="enp3s0f0"} 0
# HELP node_ethtool_vf_0_received_bytes Network interface VF_0_Rx_Bytes
# TYPE node_ethtool_vf_0_received_bytes untyped
node_ethtool_vf_0_received_bytes{device="enp3s0f0"} 0
# HELP node_ethtool_vf_0_received_packets Network interface VF_0_Rx_Packets
# TYPE node_ethtool_vf_0_received_packets untyped
node_ethtool_vf_0_received_packets{device="enp3s0f0"} 0
# HELP node_ethtool_vf_0_transmitted_bytes Network interface VF_0_Tx_Bytes
# TYPE node_ethtool_vf_0_transmitted_bytes untyped
node_ethtool_vf_0_transmitted_bytes{device="enp3s0f0"} 0
# HELP node_ethtool_vf_0_transmitted_packets Network interface VF_0_Tx_Packets
# TYPE node_ethtool_vf_0_transmitted_packets untyped
node_ethtool_vf_0_transmitted_packets{device="enp3s0f0"} 0
# HELP node_ethtool_vf_1_mc_packets Network interface VF_1_MC_Packets
# TYPE node_ethtool_vf_1_mc_packets untyped
node_ethtool_vf_1_mc_packets{device="enp3s0f0"} 0
# HELP node_ethtool_vf_1_received_bytes Network interface VF_1_Rx_Bytes
# TYPE node_ethtool_vf_1_received_bytes untyped
node_ethtool_vf_1_received_bytes{device="enp3s0f0"} 0
# HELP node_ethtool_vf_1_received_packets Network interface VF_1_Rx_Packets
# TYPE node_ethtool_vf_1_received_packets untyped
node_ethtool_vf_1_received_packets{device="enp3s0f0"} 0
# HELP node_ethtool_vf_1_transmitted_bytes Network interface VF_1_Tx_Bytes
# TYPE node_ethtool_vf_1_transmitted_bytes untyped
node_ethtool_vf_1_transmitted_bytes{device="enp3s0f0"} 0
# HELP node_ethtool_vf_1_transmitted_packets Network interface VF_1_Tx_Packets
# TYPE node_ethtool_vf_1_transmitted_packets untyped
node_ethtool_vf_1_transmitted_packets{device="enp3s0f0"} 0
# HELP node_ethtool_vf_2_mc_packets Network interface VF_2_MC_Packets
# TYPE node_ethtool_vf_2_mc_packets untyped
node_ethtool_vf_2_mc_packets{device="enp3s0f0"} 0
# HELP node_ethtool_vf_2_received_bytes Network interface VF_2_Rx_Bytes
# TYPE node_ethtool_vf_2_received_bytes untyped
node_ethtool_vf_2_received_bytes{device="enp3s0f0"} 0
# HELP node_ethtool_vf_2_received_packets Network interface VF_2_Rx_Packets
# TYPE node_ethtool_vf_2_received_packets untyped
node_ethtool_vf_2_received_packets{device="enp3s0f0"} 0
# HELP node_ethtool_vf_2_transmitted_bytes Network interface VF_2_Tx_Bytes
# TYPE node_ethtool_vf_2_transmitted_bytes untyped
node_ethtool_vf_2_transmitted_bytes{device="enp3s0f0"} 0
# HELP node_ethtool_vf_2_transmitted_packets Network interface VF_2_Tx_Packets
# TYPE node_ethtool_vf_2_transmitted_packets untyped
node_ethtool_vf_2_transmitted_packets{device="enp3s0f0"} 0

Debug logs related to ethtool (VF stats are only available on main NIC, enp13s0f0, but no related information is shown.)

ts=2024-12-27T18:46:06.975Z caller=ethtool_linux.go:431 level=debug collector=ethtool msg="ethtool stats error" err="operation not supported" device=enxbe3af2b6059f errno=95
ts=2024-12-27T18:46:06.975Z caller=ethtool_linux.go:398 level=debug collector=ethtool msg="ethtool link info error" err="operation not supported" device=lo errno=95
ts=2024-12-27T18:46:06.975Z caller=ethtool_linux.go:415 level=debug collector=ethtool msg="ethtool driver info error" err="operation not supported" device=lo errno=95
ts=2024-12-27T18:46:06.975Z caller=ethtool_linux.go:431 level=debug collector=ethtool msg="ethtool stats error" err="operation not supported" device=lo errno=95
ts=2024-12-27T18:46:06.975Z caller=ethtool_linux.go:431 level=debug collector=ethtool msg="ethtool stats error" err="operation not supported" device=vmbr0 errno=95
ts=2024-12-27T18:46:06.977Z caller=ethtool_linux.go:431 level=debug collector=ethtool msg="ethtool stats error" err="operation not supported" device=tailscale0 errno=95

Linux pve-sin-2 6.8.12-5-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-5 (2024-12-03T10:26Z) x86_64 GNU/Linux

node_exporter, version 1.8.2 (branch: HEAD, revision: f1e0e8360aa60b6cb5e5cc1560bed348fc2c1895)
  build user:       root@03d440803209
  build date:       20240714-11:53:45
  go version:       go1.22.5
  platform:         linux/amd64
  tags:             unknown