Observability on external rate limit module

[eguzki]

In order to be able to observe what is happening at runtime via tools such as prometheus and grafana being able to expose some metrics around the WASM module would be useful.

As initial suggest, Envoy external rate limit filter statistics module (source), exports the following:

Name	Type	Description
ok	Counter	Total under limit responses from the rate limit service
error	Counter	Total errors contacting the rate limit service
over_limit	Counter	total over limit responses from the rate limit service
failure_mode_allowed	Counter	Total requests that were error(s) but were allowed through because of failure_mode_deny set to false.

Note: even if Limitador (the external rate limiting service) exposes metrics like ok and over_limit, the metrics from the gateway are more reliable as they take into account external rate limiting service downtimes and timeouts.

Related #6

[eguzki]

@slopezz ^^

[david-martin]

The proxy_wasm module has some functions to define and increment metrics (which I believe end up as 'stats' in envoy proxy)
I played around with it a little here https://github.com/david-martin/my-envoy-wasm-filter/blob/main/src/lib.rs#L1
The metrics can then be scraped on the /stats/prometheus endpoint.

I had looked into adding custom labels to metrics, but because it's backed by stats, it wasn't clear how to do that.
I think a solution is to rename the metric to include any extra labels, then use 'stats tags' configuration to parse them out of the name and into labels.
https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/metrics/v3/stats.proto#config-metrics-v3-statsconfig