Implement OpenMetrics

[stgraber]

It would be useful for IncusOS itself to also provide some metrics data which we could then combine with the Incus and node-exporter ones for a full view of the system.

At minimum we'd get the Go internal metrics (memory, go routines, ...) but we can also export some extra data where it makes sense.

[AmandaCameron]

One thing that would be good to export is if the update "needs reboot" flag is set, that way we can make an alert out of it

[Triiltz]

I'd like to take a look on this one, could you please assign it to me?

[stgraber]

Assigned it to you now!

I think for now, we'd just want an /1.0/metrics endpoint in the IncusOS API, so accessible as /os/1.0/metrics through Incus. We can then tweak Incus to pull in those metrics autoamtically into its own set (same it does with node-exporter).

[Triiltz]

Ok, tks!

One question about the format, should I use prometheus/client_golang for the metrics handler or implement the OpenMetrics output manually?

[stgraber]

I don't mind us pulling prometheus/client_golang as a dependency if it makes life easier.

For LXD/Incus, I believe we wanted more flexibility so we did our own thing with our own caching and forwarding to instances, but I don't think that'd be needed here so using the prometheus Go package will likely make things easier.