|
| 1 | +# Expose Prometheus metrics |
| 2 | + |
| 3 | +Production services require monitoring to operate reliably and efficiently. In |
| 4 | +a production setup, you may want to record a variety of things like the number |
| 5 | +of access to a feature when doing some A-B tests, the duration of database queries to |
| 6 | +optimize performance when needed, the number of third-party API calls to avoid |
| 7 | +hitting rate-limits, the number of failed logins to report suspicious |
| 8 | +activity, etc. Observability is the umbrella term for techniques and |
| 9 | +technologies concerned with exposing such _metrics_ and _traces_ about |
| 10 | +internals of services. |
| 11 | +A prevalent tool and format to expose metrics is |
| 12 | +[Prometheus](https://prometheus.io/). |
| 13 | + |
| 14 | +Prometheus proposes a simple mechanism: services who want to expose metrics add |
| 15 | +a web-API route returning a series of metrics in a well-known text-format. |
| 16 | +Prometheus collectors then periodically (often at a short interval) request |
| 17 | +this route for one or many services. |
| 18 | + |
| 19 | +This cookbook shows how to expose Prometheus counters using Servant so that a |
| 20 | +Prometheus collector can then collect metrics about your application. We |
| 21 | +leverage the [`prometheus-client`](https://hackage.haskell.org/package/prometheus-client) package to provide most of the instrumentation |
| 22 | +primitives. While packages exist to direcly expose Prometheus counters, this |
| 23 | +cookbook guides you to write your own exposition handler. Writing your own |
| 24 | +handler allows you to tailor the metrics endpoint to your needs. Indeed, you |
| 25 | +may want to re-use Servant combinators to expose different subsets of metrics |
| 26 | +onto different endpoints. Another usage of Servant combinators would be to |
| 27 | +protect the endpoint so that only trusted clients can read the metrics. Here we |
| 28 | +propose to augment the endpoint with a |
| 29 | +[CORS](https://en.wikipedia.org/wiki/Cross-origin_resource_sharing) header so |
| 30 | +that a browser clients (such as |
| 31 | +[prometheus-monitor](https://dicioccio.fr/prometheus-monitor.html)) can query the Prometheus |
| 32 | +metrics endpoint. |
| 33 | + |
| 34 | +First, the imports. |
| 35 | + |
| 36 | +``` haskell |
| 37 | +{-# LANGUAGE DataKinds #-} |
| 38 | +{-# LANGUAGE OverloadedStrings #-} |
| 39 | +{-# LANGUAGE TypeOperators #-} |
| 40 | +{-# LANGUAGE MultiParamTypeClasses #-} |
| 41 | +{-# LANGUAGE GeneralizedNewtypeDeriving #-} |
| 42 | +
|
| 43 | +import Control.Monad (forever) |
| 44 | +import Control.Concurrent (ThreadId, forkIO, threadDelay) |
| 45 | +import Control.Monad.IO.Class (liftIO) |
| 46 | +import Data.ByteString.Lazy (ByteString) |
| 47 | +import Data.Text (Text) |
| 48 | +import Network.Wai.Handler.Warp (run) |
| 49 | +import qualified Prometheus as Prometheus |
| 50 | +import Prometheus.Metric.GHC (ghcMetrics) |
| 51 | +import Servant |
| 52 | +``` |
| 53 | +
|
| 54 | +In this cookbook we will write a dummy "hello-world" API route. A counter will |
| 55 | +count accesses to this API route. For the purpose of this cookbook the |
| 56 | +"hello-world" API route will count how many times whom got a hello. For |
| 57 | +instance "Hello, Bob" and "Hello, Alice" means we got "1 for Bob" and "1 for |
| 58 | +Alice", in short, we record a counter _breakdown_. Another counter will report |
| 59 | +values counted in a background-thread, here, a counter in a sleep-loop. Such |
| 60 | +counters can serve as watchdog for other applications: if the counter stops |
| 61 | +increasing, then something is amiss. |
| 62 | + |
| 63 | +In a real-application you may want to avoid |
| 64 | +exposing counters broken-down by a value chosen by an untrusted-user (i.e., if |
| 65 | +our hello-world API is public, you open the door to unbounded |
| 66 | +memory-requirements as counter breakdowns persist in memory). However, for the |
| 67 | +purpose of this cookbook, we assume the risk is mitigated. |
| 68 | + |
| 69 | +The Prometheus library we use requires us to register the counters ahead of |
| 70 | +time. Let's define a datatype to refer to all the counters needed |
| 71 | +in this cookbook. An `initCounters` function performs all the needed |
| 72 | +registration. |
| 73 | + |
| 74 | +``` haskell |
| 75 | +data Counters |
| 76 | + = Counters |
| 77 | + { countHellos :: Prometheus.Vector (Text) Prometheus.Counter |
| 78 | + , countBackground :: Prometheus.Counter |
| 79 | + } |
| 80 | +
|
| 81 | +initCounters :: IO Counters |
| 82 | +initCounters = |
| 83 | + Counters |
| 84 | + <$> Prometheus.register |
| 85 | + (Prometheus.vector "who" |
| 86 | + (Prometheus.counter |
| 87 | + (Prometheus.Info "cookbook_hello" "breakdown of hello worlds"))) |
| 88 | + <*> Prometheus.register |
| 89 | + (Prometheus.counter |
| 90 | + (Prometheus.Info "cookbook_background" "number of background thread steps")) |
| 91 | +``` |
| 92 | +
|
| 93 | +We next implement the dummy "hello-world" API route. We add a Servant type and |
| 94 | +implement a Servant Handler. We want the API route to have a `who` query-param |
| 95 | +so that one can call `hello?who=Alice` and get a greeting like "hello, Alice" |
| 96 | +in return. We use our first Prometheus counter to record how many times Alice, |
| 97 | +Bob, or anyone got a greeting. |
| 98 | +
|
| 99 | +The handler will defaults to `n/a` as a magic value to represent the absence of |
| 100 | +`who` query-parameter. |
| 101 | +
|
| 102 | +``` haskell |
| 103 | +type Greeting = Text |
| 104 | +
|
| 105 | +newtype HelloWho = HelloWho { getWho :: Text } |
| 106 | + deriving FromHttpApiData |
| 107 | +
|
| 108 | +type HelloAPI = |
| 109 | + Summary "a dummy hello-world route" |
| 110 | + :> "api" |
| 111 | + :> "hello" |
| 112 | + :> QueryParam "who" HelloWho |
| 113 | + :> Get '[JSON] Greeting |
| 114 | +
|
| 115 | +-- | A function to turn an input object into a key that we use as breakdown for |
| 116 | +-- the `countHellos` counter. In a real-world setting you want to ponder |
| 117 | +-- security and privacy risks of recording user-controlled values as breakdown values. |
| 118 | +helloWhoToCounterBreakdown :: HelloWho -> Text |
| 119 | +helloWhoToCounterBreakdown (HelloWho txt) = txt |
| 120 | +
|
| 121 | +handleHello :: Counters -> Maybe HelloWho -> Handler Greeting |
| 122 | +handleHello counters Nothing = do |
| 123 | + let breakdown = "n/a" |
| 124 | + liftIO $ Prometheus.withLabel (countHellos counters) breakdown Prometheus.incCounter |
| 125 | + pure "hello, world" |
| 126 | +handleHello counters (Just who) = do |
| 127 | + let breakdown = helloWhoToCounterBreakdown who |
| 128 | + liftIO $ Prometheus.withLabel (countHellos counters) breakdown Prometheus.incCounter |
| 129 | + pure $ "hello, " <> getWho who |
| 130 | +``` |
| 131 | +
|
| 132 | +We further instrument our program with a second metrics. This second metrics |
| 133 | +consist of a simple counter. A background thread will increment the counter |
| 134 | +every second. |
| 135 | +
|
| 136 | +``` haskell |
| 137 | +startBackgroundThread :: Counters -> IO ThreadId |
| 138 | +startBackgroundThread counters = forkIO $ forever go |
| 139 | + where |
| 140 | + go :: IO () |
| 141 | + go = do |
| 142 | + Prometheus.incCounter (countBackground counters) |
| 143 | + threadDelay 1000000 |
| 144 | +``` |
| 145 | +
|
| 146 | +Now we need to implement the part where we expose the Prometheus metrics on the |
| 147 | +web API. |
| 148 | +Let's define an API route: it's a simple `HTTP GET` returning some _Metrics_. |
| 149 | +In this example we also add the CORS header we discussed in intro. |
| 150 | +
|
| 151 | +``` haskell |
| 152 | +type PrometheusAPI = |
| 153 | + Summary "Prometheus metrics" |
| 154 | + :> "metrics" |
| 155 | + :> Get '[PlainText] |
| 156 | + (Headers '[Header "Access-Control-Allow-Origin" CORSAllowOrigin] Metrics) |
| 157 | +``` |
| 158 | +
|
| 159 | +With this API type, we now need to fill-in the blanks. We define a `Metrics` |
| 160 | +object that serializes as the Prometheus text format. We want to keep it |
| 161 | +simple and use `Prometheus.exportMetricsAsText` to collect the metrics as a |
| 162 | +text-formatted payload. This function is an IO object returning the whole text |
| 163 | +payload, thus, our `Metrics` object contains the raw payload in a |
| 164 | +"pre-rendered" format for the MimeRender instance. |
| 165 | +
|
| 166 | +``` haskell |
| 167 | +newtype Metrics = Metrics {getMetrics :: ByteString} |
| 168 | +
|
| 169 | +instance MimeRender PlainText Metrics where |
| 170 | + mimeRender _ = getMetrics |
| 171 | +``` |
| 172 | +
|
| 173 | +We define the CORS header helper. |
| 174 | +
|
| 175 | +``` haskell |
| 176 | +newtype CORSAllowOrigin = CORSAllowOrigin Text |
| 177 | + deriving ToHttpApiData |
| 178 | +``` |
| 179 | +
|
| 180 | +Finally, we define the Prometheus collection endpoint proper. The magic |
| 181 | +function `Prometheus.exportMetricsAsText` provided by our Prometheus library |
| 182 | +ensures that we collect all registered metrics in a process-global variable. |
| 183 | +The implementation uses a where-clause to recall the Handler data-type so that everything is front of our eyes. |
| 184 | +
|
| 185 | +``` haskell |
| 186 | +handlePrometheus :: CORSAllowOrigin -> Server PrometheusAPI |
| 187 | +handlePrometheus corsAllow = handleMetrics |
| 188 | + where |
| 189 | + handleMetrics :: Handler (Headers '[Header "Access-Control-Allow-Origin" CORSAllowOrigin] Metrics) |
| 190 | + handleMetrics = do |
| 191 | + metrics <- liftIO $ Prometheus.exportMetricsAsText |
| 192 | + pure $ addHeader corsAllow $ Metrics metrics |
| 193 | +``` |
| 194 | +
|
| 195 | +Finally, we bundle everything in an application. |
| 196 | +
|
| 197 | +The complete API consists in the "hello-world" api aside the Prometheus metrics |
| 198 | +endpoint. As a bonus we register the `ghcMetrics` (from the |
| 199 | +`prometheus-metrics-ghc` package) which allows to collects a lot of runtime |
| 200 | +information (such as the memory usage of the program), provided that your |
| 201 | +binary is compiled with `rtsopts` and run your progam with `+RTS -T` (cf. the [GHC docs about the RTS](https://downloads.haskell.org/ghc/latest/docs/users_guide/runtime_control.html)). |
| 202 | +
|
| 203 | +``` haskell |
| 204 | +type API |
| 205 | + = HelloAPI |
| 206 | + :<|> PrometheusAPI |
| 207 | +
|
| 208 | +api :: Proxy API |
| 209 | +api = Proxy |
| 210 | +
|
| 211 | +server :: CORSAllowOrigin -> Counters -> Server API |
| 212 | +server policy counters = handleHello counters :<|> handlePrometheus policy |
| 213 | +
|
| 214 | +runApp :: Counters -> IO () |
| 215 | +runApp counters = do |
| 216 | + let allowEveryone = CORSAllowOrigin "*" |
| 217 | + run 8080 (serve api (server allowEveryone counters)) |
| 218 | + |
| 219 | +main :: IO () |
| 220 | +main = do |
| 221 | + _ <- Prometheus.register ghcMetrics |
| 222 | + counters <- initCounters |
| 223 | + _ <- startBackgroundThread counters |
| 224 | + runApp counters |
| 225 | +``` |
| 226 | +
|
| 227 | +Now you can navigate to various pages: |
| 228 | +- http://localhost:8080/metrics |
| 229 | +- http://localhost:8080/api/hello |
| 230 | +- http://localhost:8080/api/hello?who=prometheus |
| 231 | +- http://localhost:8080/metrics |
| 232 | +- http://localhost:8080/api/hello?who=prometheus again |
| 233 | +- http://localhost:8080/api/hello?who=world |
| 234 | +- http://localhost:8080/metrics |
| 235 | +
|
| 236 | +You can also see counter increases live by installing a Prometheus collector. |
| 237 | +For development and troubleshooting purpose, you can use [prometheus-monitor](https://dicioccio.fr/prometheus-monitor.html) also available as [Firefox extension](https://addons.mozilla.org/en-GB/firefox/addon/prometheus-monitor/). |
0 commit comments