Report more information about keystore contents on reload

[DaveCTurner]

Today when invoking POST _nodes/reload_secure_settings the response includes just a list of node IDs and names, which is not all that useful for diagnosing problems related to secure settings. Could we include more information about the keystore that was loaded on each node in the response? All of the following would be helpful, as long as they do not compromise security:

• absolute path to keystore file
• last-modified date of keystore file
• list of setting keys loaded on each node

I don't see an obvious reason why keeping this information hidden has any security benefits (but ofc security has lots of nonobvious concerns too).

---

Relevant forum post

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[tvernum]

absolute path to keystore file

I wonder whether the absolute path is sensitive, my gut feel is that it is not, but we'd want to think that through. To my knowledge there is no other API that leaks the location of the config directory (I may be wrong though).

list of setting keys loaded on each node

We may need to respect filtered settings for this (maybe - I'm not sure whether we advertise that the filtering behaviour applies to key names), and I don't think we would want to report whether the value was change by the reload, just that it was read.

--

We can definitely do better than we do today - even if all we do is report the number of settings that were found of each node, that would be progress.

[DaveCTurner]

Maybe we could report the path in logs instead, if it's too sensitive to return in an API?

I do not believe that secure settings can be marked as filtered, at least org.elasticsearch.common.settings.SecureSetting#ALLOWED_PROPERTIES does not include Property.Filtered. I do agree that we don't care whether the value was changed with this particular reload or not.

I would be concerned that simply reporting the number of settings is still not sufficient for the troubleshooting cases we've had in this area in the past. Yes with sufficient effort you can work out if you're manipulating the wrong keystore by adding/removing dummy settings and seeing if the count changes, but this is asking rather a lot of our users IMO. I am struggling to think of how the keys themselves could be sensitive - you can deduce the things that should be in the keystore by looking at unprotected settings in most cases anyway.

[DaveCTurner]

with sufficient effort you can work out if you're manipulating the wrong keystore

An alternative idea that would also help without exposing too much extra info would be to report some easily-computed hash of the keystore file as loaded on each node. Still not quite as nice as reporting the names of the settings it loaded, but would at least make it easier to correlate the file that ES read against the one the user thinks it should have read.

[DaveCTurner]

#128097 is a great example of the kind of thing we cannot track down today. My hunch in that case is that ES is reloading settings from a keystore that is missing some items (for reasons outside of ES's control) but it's really hard to tell. The keystore in this case is some kind of virtual secret file managed by Kubernetes so it's pretty hard for the user to get hold of exactly the data that ES is seeing.

[DaveCTurner]

#128658 is another good example - the obvious explanation for this user's troubles is that the keystore contents are not as expected, but again there's layers of abstraction between what the user configures and what ES sees that make it hard to make progress.