Skip to content

Conversation

@0xf09f95b4
Copy link
Contributor

@0xf09f95b4 0xf09f95b4 commented Aug 4, 2025

Hi,

this PR attempts to address #2805.

The issue we encountered is a case in which a path is bind-mounted to itself, the same issue you can see in this latest comment.

NixOS bind-mounts the /nix/store path to /nix/store to remount the same mount read-only and (nosuid, nodev), as you can see here. If the /nix/store path is already a mount from some device, this will lead to metrics being collected twice for the same (mountPoint, device) combination if the mount options differ.

This issue only occurs, when the filesystem options of the two mounts are different. The core issue is, that the existing filter that wants to prevent this issue here takes the entire labels struct into account, including mount options.

The mount options themselves are not used or exposed by node_exporter. So this minimal fix only removes the filesystem options after they are used to extract the ro gauge. This way, the seen := map[filesystemLabels]bool{} filter works correctly (as the options are always "").

This has a side-effect: Previously, the ro gauge was exported twice in this case, if the two mounts were read only and read-write (once ro, once rw). Now, the gauge is exported only once and in the example, it would be exported rw, even though the mount actually used by the kernel (the upper mount), is ro.

Additional options we came up with to fix this:

  • Filter this earlier, like the previous commit in this PR tried.
  • Remove the options from fileSystemLabels alltogether (only used in Linux anyway).
  • Add more logic to export the "correct" ro state.
  • Add the ro option to gauges as a label.

@discordianfish You were involved in the original issue. What do you think?

@0xf09f95b4 0xf09f95b4 force-pushed the filter-repeated-mountpoints branch from c347b31 to 83efcfe Compare August 4, 2025 09:17
@0xf09f95b4
Copy link
Contributor Author

@SuperQ do you have an opinion on this? Do you think this minimal fix is a good approach or would you like a more obvious/clearer implementation here?

@SuperQ SuperQ requested a review from discordianfish August 15, 2025 07:22
@0xf09f95b4
Copy link
Contributor Author

@SuperQ @discordianfish Just pinging for an update :) Are you interested in this change?

I'm pretty sure this issue still exists. @aequitas are you still seeing these errors in your log messages? Is it possible for you to test if this change fixes the issue for you?

@aequitas
Copy link

@0xf09f95b4 I'll try to test this change this week and let you know if it fixes things

@SuperQ
Copy link
Member

SuperQ commented Oct 24, 2025

This needs a rebase after #3387.

Yea, this is tricky to fix cleanly as we're overloading the "labels" with non-label filesystem information.

@0xf09f95b4 0xf09f95b4 force-pushed the filter-repeated-mountpoints branch from 83efcfe to de310e3 Compare October 24, 2025 12:52
@0xf09f95b4
Copy link
Contributor Author

0xf09f95b4 commented Oct 24, 2025

Thanks for your feedback! I just tested 1.10.0 and I can still see the issue happening.

I rebased the commit. I had to modify the patch to now zero our both mount option strings. I also checked and as far as I can see, those options are still not used anywhere after the changed line.

Copy link
Member

@SuperQ SuperQ left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. This seems like a reasonable workaround for now. Fixing this cleanly would probably require some serious thought / refactoring of the whole filesystem collector flow.

@SuperQ SuperQ merged commit ef5efb7 into prometheus:master Oct 24, 2025
13 checks passed
SuperQ added a commit that referenced this pull request Oct 25, 2025
* [BUGFIX] filesystem: Fix mount points being collected multiple times #3376
* [BUGFIX] filesystem: Refactor mountinfo parsing #3452
* [BUGFIX] meminfo: Add Zswap/Zswapped metrics #3453

Signed-off-by: Ben Kochie <[email protected]>
@SuperQ SuperQ mentioned this pull request Oct 25, 2025
SuperQ added a commit that referenced this pull request Oct 25, 2025
* [BUGFIX] filesystem: Fix mount points being collected multiple times #3376
* [BUGFIX] filesystem: Refactor mountinfo parsing #3452
* [BUGFIX] meminfo: Add Zswap/Zswapped metrics #3453

Signed-off-by: Ben Kochie <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants