Allow for servers with basic auth

[oliver006]

I have a prometheus server sitting behind nginx and use basic auth to restrict access.
Currently, promdash fails when I try to add datasources from that server.
It'd be great if basic auth would work.

[juliusv]

Just a thought: does it work to encode the username/password into your server URL that you configure in PromDash? I.e. as the Prometheus server URL that you configure in PromDash, provide http://<user>:<pass>@1.2.3.4:9090/

[ericflo]

I just tried that on my install, and it didn't seem to work.

[oliver006]

Yeah, tried that but it didn't work.
I checked the browser console, it's doing a straight request to the url without the basic auth credentials which results in the 401.

[juliusv]

Indeed, I can confirm that. But in general, configuring username/password pairs for each server in PromDash and storing the password plaintext in the PromDash database would be ok? (I don't really see a way around that?)

[juliusv]

So I've been playing with AngularJS/CORS/BasicAuth/nginx locally for a few hours now and haven't been able to tweak the settings on all parts of the chain in such a way that the combination works. I suspect that nginx's basic auth 401 is messing up the CORS flow somehow, which in turn prevents sending any Authorization headers at all... but not sure. If anyone wants to experiment with this or take this on, I'd be greatful.

[oliver006]

Thanks for looking into this!
I played around with the nginx side of things and I got it to work by adding an nginx location for /api and make that proxy-pass through to my prometheus server at localhost:9090 now I cna use the same servername as the dashboard for the prometheus server and auth and CORS both work.
Once #343 lands it'd much easier and cleaner to have different locations proxy to different prometheus servers but this works for me for now.

[juliusv]

@oliver006 Oh interesting. Did you modify PromDash to add basic auth? Also, are you actually doing CORS or avoiding it by having both PromDash and Prometheus going through the same server name (via the proxy)? Would be interesting to hear more about your solution.

#343 becomes most useful together with supporting different roots in the Prometheus server itself, but that PR (prometheus/prometheus#528) is currently stuck because it would break some other things like having console templates in the server at arbitrary directory depths (relative asset paths would then change depending on directory depth). But actually, that latter PR should only be relevant for using the Prometheus web UI, not the API. If all you want to do is use the API from PromDash, that should work with #343 alone.

[oliver006]

Yes, i have both promdash and the prometheus server going through the same server name which addresses the CORS problem. Both also use the same basic auth realm which I think takes care of the auth issue. Right now this works cause promdash doesn't use the api/ endpoint so I can "map" that to the prometheus server. Here's my nginx config:

server {
        listen 80;
        server_name promdash.servername.com;
        location / {
            proxy_pass http://localhost:3000;
            auth_basic "Restricted";
            auth_basic_user_file /etc/nginx/.htpasswd;
            proxy_set_header    X-Host       $host;
        }
        location /api/ {
            proxy_pass http://localhost:9090/api/;
            auth_basic "Restricted";
            auth_basic_user_file /etc/nginx/.htpasswd;
            proxy_set_header    X-Host       $host;
        }
}

[juliusv]

Ah, I see. That naturally takes care of both issues at once. Even if PromDash had an /api endpoint, you could map all of Prometheus to some /prom subpath.

[oliver006]

I tried that but it didn't work.
I mapped /prom/api/ to /api/ of the prometheus server and used http://promdash.servername.com/prom/ as the prom server name in promdash but when I open a dashboard the /prom/ part is stripped from the ajax calls and naturally they fail.

[juliusv]

Oh yeah, right, that only becomes possible with #343.

[grobie]

@oliver006 Does #343 solve the issue?

[oliver006]

Yessir!

[danpalmer]

I apologise if I'm missing something, but this doesn't appear to solve the issue?

I have PromDash running on dashboard.example.com and Prometheus on prometheus.example.com behind nginx with HTTP basic auth. I can add the server URL http://user:[email protected] to PromDash, but the requests made from the dashboard don't include the basic auth details, so I just receive a "401 Authorization Required" from Prometheus.

Is there a solution to this? If not, I suggest re-opening the issue, or I can create a separate issue.

[danpalmer]

To follow up, it appears that Angular JS's $http API which is used for these requests doesn't support HTTP basic auth in URLs, and that's the reason it's not working.

I've never used Angular, so I wouldn't know where to begin fixing this. The examples of how to fix it online are a little conflicting (at least to someone who doesn't know Angular) so I'd appreciate any help in solving this.

[juliusv]

@danpalmer Thanks for that feedback. I'm reopening the issue for now. I don't have the necessary knowledge myself off the top of my head or the time to investigate this at the moment, but maybe someone else is interested enough to tackle this.

[oliver006]

@danpalmer you're right, this never solved the issue of the basic auth angular http requests.
To make this work, I proxied both promdash and prometheus via nginx through the same (sub) domain (see comment from Feb 15th) - #343 helps with that as it allows several prometheus servers to happily coexist on one domain.
Does that make sense/help ?

[danpalmer]

@oliver006 I thought that was what your workaround was in the end. While it works, it is definitely just a workaround, and not a fix, so I think this should be re-opened. I've worked around the issue by removing HTTP basic auth and instead putting Prometheus at a long unpredictable URL, but it would be nice to add auth back to it.

[juliusv]

Agreed, the issue is reopened.

[MrMMorris]

enthusiastic 👍 to this!

[ne0h]

If have the same issue: prometheus and promdash behind apache proxies with basic auth. It would be great if promdash could connect to prometheus this way.

[juliusv]

@ne0h Again, if anyone feels qualified to make AngularJS's AJAX requests work with basic auth, contributions are welcome :)

[stuartnelson3]

don't really have the time right now, but this should work:

1. create an http interceptor
2. have the interceptor check the request url for basic auth
3. if it's there, parse it out and set it as a basic auth header:

// pseudo-ish code
configObj.headers['Authorization'] = 'Basic ' + btoa('admin' + ':' + 'password');

this would be the best way. the other option is to do steps 2 and 3 above but just for the request made to prometheus in the graph refresher. this would have to be duplicated elsewhere though for requests to prometheus so.... first option is better.

[jonnenauha]

I know this is pre 1.0 stuff. It all works wonderfully if you ignore the fact that anyone on the internet can look at you internal stats, metrics and servers. I don't think people are asking for too much to get this stuff secure for production envs. On to the issue...

Yeah, got bit by this. It's pretty incredible the prometheus project in general says "we don't handle auth, hide your stuff behind a reverse proxy and do auth there" but it does not provide any documentation and once you try this stuff out (esp. with the provided docker run docs) nothing works.

Well ok... Now to properly secure both servers that raw IP:PORT is not exposed without the reverse proxy auth mechanisms, I need/want to run

1. The reverse proxy with 2 subdomains prometheus./dash.examples.com (as OP)
2. Run prom localhost:9090
   • This part is fine as it lets you pass in the web ui listen string to golang directly, good flexibility.
   • You can run it as a raw process or run it with docker -p 127.0.0.1:9090:9090. Both ways wont expose the raw IP:PORT.
3. Run promdash localhost:3030
   • I have no idea how to accoplish this without docker. There is no documentation how the port can be changed on the ruby thin server. I tried to google it, read config files, read the code, can't find it.
   • ok ill run it with docker then -p 127.0.0.1:3000:3000. Great its up and raw IP:PORT does not work, youll have to use the domain (that again has the auth). Victory.

At this stage I'm quite happy but it turns out that promdash configured with http://localhost:900/ is just giving errors. No way to know what is happening, I'm looking at the console as there is no indication what happened, why it fails. I try curl localhost:9090/metrcis and it works fine.

Next I read up on docker how the network stuff works (mind you this is the first project I've ever used docker so consider me a noob :). Running promdash with --net host does the trick, but now you are exposing promdash at 0.0.0.0:3000 and your basic auth is out the window if someone uses the raw IP:PORT that bypasses the reverse proxy.

So my summary would be

• Please document the complete steps how to host prom and relevant frequently used components like promdash behind a reverse proxy. So that there is no possible way to get to either server by outsiders without having to do a simple basic auth.
  • I don't care if its a super simple a quick way like hosting everything in the same box, its a start.
  • This really needs to happen if your final answer to auth/security is "use a reverse proxy". I've seen this response both here and in the main server project. There was a meta issue to produce these docs somewhere as well but without much progress.
  • The problem with your response is that people will give up. I almost did and spent a good ~7h fighting with your software and trying my best to get things up. I'm sure its possible but it should not require this much work or to be HAPRoxy/gnix/whatever expert to do it. You can rectify this with a simple dedicated doc page in your docs.
• Please let promdash cmdline or env variable define what host and port it listens to. Maybe you already have, then document it clearly in your main .md file and at prometheus docs in the promdash section.
  • This would solve all my remaining issues. If I could make promdash listen to localhost:3000 i could safely run it with docker --net host.

P.S. My rant started by finding this issue because I tried also what OP tried, putting the basic auth into the URL directly, I thought I had finally found a way to make all this stuff work together, but no. Really hope @juliusv has some kind of communication with the main prom team (or whoever writes/upkeeps the docs) to get something going.

P.S.S. I'm using HAProxy and can provide config sections if its needed. I also tried skipping basic auth if src ip of the request is same as the server itself. This did not make it work either with promdash, I suspect docker might not expose the real outward facing ip in the requests?

---

As a side point, maybe the error when prom cant be connected to could be something better in the UI than server response status undefined which is probably really just a js undefined value as string. I'm sure whatever lib you are using to execute the HTTP request can give something better than that if status code (I'm assuming this is what you are reading) is 'undefined'.

[juliusv]

On Sun, Sep 13, 2015 at 2:15 PM, Jonne Nauha [email protected]
wrote:

I know this is pre 1.0 stuff. It all works wonderfully if you ignore the
fact that anyone on the internet can look at you internal stats, metrics
and servers. I don't think people are asking for too much to get this stuff
secure for production envs.

Keep in mind that Prometheus's main use case was not to be put "on the
internet", but to be used inside private datacenters / VPCs / clouds to
monitor processes there, all behind the firewall. This might explain a bit
of your surprise. Yes, you can put it on the internet, but then you have to
know what you're doing, and we don't document that case well yet.

We would love to provide more documentation around this, but it is not our
main focus at the moment.

Yeah, got bit by this. It's pretty incredible the prometheus project in
general says "we don't handle auth, hide your stuff behind a reverse proxy
and do auth there" but it does not provide any documentation and once you
try this stuff out (esp. with the provided docker run docs) nothing works.

Why is it incredible? We are not a company trying to sell Prometheus, we're
doing the best we can do build a great open-source project under the
constraints we have, and mostly optimized for environments not quite like
yours. If you need commercial support to help with your use case, there are
some first third-party options springing up, see
http://prometheus.io/community/.

Well ok... Now to properly secure both servers that raw IP:PORT is not
exposed without the reverse proxy auth mechanisms, I need to run

1. The reverse proxy with 2 subdomains prometheus./dash.examples.com
   (as OP)
2. Run prom localhost:9090
   • This part is fine as it lets you pass in the web ui listen string
     to golang directly, good flexibility.
   • You can run it as a raw process or run it with docker -p
     127.0.0.1:3000:3000. Both ways wont expose the raw IP:PORT.
3. Run promdash localhost:3030
   • I have no idea how to accoplish this without docker. There is
     no documentation how the port can be changed on the ruby thin server. I
     tried to google it, read config files, read the code, can't find it.

First hit on Google for "thin server port" explains it quite well it
seems:
http://stackoverflow.com/questions/8008436/starting-thin-server-on-different-ports

Even our README.md explains how to set the port:
https://github.com/prometheus/promdash#production-mode

...so I can't quite follow you here about how this is not findable at all.

1. • ok ill run it with docker then -p 127.0.0.1:3000:3000. Great its
     up and raw IP:PORT does not work, youll have to use the domain (with again
     has the auth). Victory.

At this stage I'm quite happy but it turns out that promdash configured
with http://localhost:900/ is just giving errors. No way to know what is
happening, I'm looking at the console as there is no indication what
happened, why it fails. I try curl localhost:9090/metrcis and it works
fine.

I'm not sure what you mean here with "promdash configured with
http://localhost:900/" (I assume the port is a typo). This indeed might not
be clear immediately, but PromDash doesn't need to connect to anything
(except perhaps its SQL server if you're not using sqlite3). It is your
browser that connects via AJAX to the Prometheus servers and fetches data
from them directly. So if you created a server entry in PromDash that
points to localhost, your browser would try to connect to your own
laptop/workstation, even if PromDash is running somewhere else. You'll need
to configure the externally reachable Prometheus URL instead.

Do keep in mind the comments from
#341 (comment) and
onwards about CORS issues when hosting PromDash and Prometheus through
different external server names together with basic auth. We don't have a
good solution for this yet, only the stated workaround (use the same server
name for both and the same basic auth realm).

• Please document the complete steps to host prom and relevant
  frequently used components like promdash behind a reverse proxy. So that
  there is no possible way to get to either server by outsiders without
  having to do a simple basic auth.
  • I don't care if its a super simple a quick way like hosting
    everything in the same box, its a start.
  • This really needs to happen if your final answer to auth/security
    is "use a reverse proxy". I've seen this response both here and in the main
    server project. There was a meta issue to produce these docs somewhere as
    well but without much progress.

So far, Robust Perception wrote this blog post:
http://www.robustperception.io/adding-basic-auth-to-prometheus-with-nginx/

It'd be great to get similar and more extensive documentation for this use
case on prometheus.io at some point, I agree. Since it's an open-source
project, anyone can contribute it :)

• Please let promdash cmdline or env variable define what host and
  port it listens to. Maybe you already have, then document it clearly in
  your main .md file and at prometheus docs in the promdash section.

P.S. My rant started by finding this issue because I tried also what OP
tried, putting the basic auth into the URL directly, I thought I had
finally found a way to make all this stuff work together, but no. Really
hope @juliusv https://github.com/juliusv has some kind of communication
with the main prom team (or whoever writes/upkeeps the docs) to get
something going.

We all know each other well (I co-started Prometheus). I wrote a lot of the
docs, and so did others. The docs are maintained here, for anyone to
contribute improvements: https://github.com/prometheus/docs.

[ne0h]

This might help: We use an openvpn network to monitor different standalone server in the internet. So we do not have to care about authentication and SSL. It works quite well.

[jonnenauha]

Thanks for the reply. I think my post could be read as hostile, sorry that was not my intention. Should probably have not written that after battling with this stuff all day and night yesterday :) I've worked no plenty of open source projects and did not intend to diminish the effort in any way, like I said in the first paragraph its really awesome system. Got a big kick out of seeing a real time graph of cpu, mem usage of all our servers, once I got it to work :)

It was fairly trivial to grok how to put stuff behind a reverse proxy, once I found the doc bits that said that is what you should do. Doing that independently with prom or promdash was easy. But I had to fight a lot with the docker stuff to do what I wanted and make them play nice together. So, its not really your bug in your stuff, more of pointing users in the right direction. As you said this is open source so I will work on tonight on a pull request to affect the thin -a 127.0.0.1 -p 3000 start for the docker image.

I did miss that port config in the readme because I only read the docker parts (it was right above it, doh!), sorry about that. Not familiar with ruby at all so I decided to not install all that stuff by hand and use the repo sources.

That :900 was a tired me typo/copy paste. I mean I used the http://localhost:9090/ inside the web ui to add the prom server. I tried all kinds of variations but anything except the hostname or public IP (with prom --net host) did not work. And then I was back to non restricted access. That basic auth to work inside the URL will be a nice addition that would have solved my issue instantly. I would even suggest that you make those user/pass separate fields in the server edit web UI so its 100% clear to people that might not know that user:pass@ stuff, or realize its possible to put there (unless documented :). I'm really suprised if its the angular http lib doing that as suggested there, very weird.

Good choise having the docs in a separate repo. Makes it much more approachable for outsiders. Do you think it would be good to document at least the "docker way" of installing prom dash there. I know there is a direct link to the readme now, so would probably not bring any benefit in duplicating it there. Although then that doc site has more about how to use prom than prom readme itself. I suppose its a good split, install/develop stuff in readme and higher level usage stuff in doc site.

Thanks again for your work. I'll start booting up my linux VM and see if I can get that pull request for you.