Introduction
Today I was fighting quite a bit with the integration of a software called Teslamate with Grafana through nginx reverse proxy, that Christer recommended to me. The issue I had was how to get Grafana to play nicely in a sub-path like https://example.com/grafana/
There are many explanations online on how to do this, and most of them are old and outdated. What is important is to pass on the proper headers from nginx and to configure grafana properly to accept being served under a sub path. This posed some issues for me as I use our “standard stack” of haproxy ---haproxy-protocol---> nginx -> teslamate/grafana.
As a quick reminder, haproxy integrates with nginx over something called “haproxy protocol”, which in principle just pads 150 bytes to the beginning of whatever protocol it connects with (for example http/https) to pass on some metadata. We do this for https on port 444.
This in turn made it a bit tricky to see where in the chain an issue would be. In my case I had to properly forward some X-headers, and I did not know how to easily see what I “actually” set in the nginx config.
Enter the simple container “whoami” from Traefik.
I shut down the teslamate, started whoami on the same port as Grafana, then used curl:
- Config for
whoami
ops@wue-docker-l02:~/docker/teslamate$ cat docker-compose.test.yml
services:
whoami:
ports:
- 3002:80 #<--- using the port I proxy grafana to
image: "traefik/whoami"
- Shut down Teslamate and Grafana
ops@wue-docker-l02:~/docker/teslamate$ docker compose down
[+] Running 1/1
✔ Network teslamate_default Removed 0.3s
- Starting
whoami
ops@wue-docker-l02:~/docker/teslamate$ docker compose -f docker-compose.test.yml up
[+] Running 2/2
✔ Network teslamate_default Created 0.1s
✔ Container teslamate-whoami-1 Created 0.1s
Attaching to whoami-1
whoami-1 | 2024/10/19 13:26:03 Starting up on port 80
- In a different terminal on my laptop:
15:25 $ curl https://teslamate.example.com/grafana/
Hostname: 8c50a13f857c
IP: 127.0.0.1
IP: ::1
IP: 192.168.25.2
RemoteAddr: 192.168.9.22:34456
GET /grafana/ HTTP/1.1
Host: teslamate.example.com
User-Agent: curl/8.7.1
Accept: */*
Connection: close
X-Forwarded-For: 192.168.4.234
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Real-Ip: 192.168.4.234
And there it is, the headers that I was not sure were correct! Now I could edit the nginx config and figure out the proper configuration in my setting.
You can also see that haproxy and nginx are correctly configured. If I, for example, remove the set_real_ip_from clause in the nginx config for my application:
#set_real_ip_from 192.168.9.0/24;
Then, curl says the following for X-Forwarded-For (classic misconfiguration in a haproxy environment):
X-Forwarded-For: 192.168.9.21 # <--- the haproxy server ip
Instead of the proper:
X-Forwarded-For: 192.168.4.234 # <--- my laptop's ip
Extra credit
Outside the scope of using traefik/whoami, the magic trick to get grafana to work the way I wanted, is to:
- Configure nginx properly
- Configure grafana in the teslamate docker-compose.yml file properly
nginx config:
upstream teslamate.example.com {
server 192.168.4.80:4000 fail_timeout=0;
}
#--- extra config from playbook pre
upstream grafana {
server 192.168.4.80:3002;
}
server {
listen 192.168.9.22:444 ssl proxy_protocol;
...
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Client-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Original-URI $request_uri;
#--- only needed for haproxy
set_real_ip_from 192.168.9.0/24;
real_ip_header proxy_protocol;
location = /grafana {
return 301 $scheme://$host/grafana/;
}
location /grafana/ {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Port 443;
proxy_pass http://grafana;
}
# Proxy Grafana Live WebSocket connections.
location /grafana/api/live/ {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Port 443;
proxy_pass http://grafana;
}
...
}
The grafana part of teslamate/docker-comopse.yml. Anything you set as environment variables for grafana with the prefix GF_ will be considered as parts of the grafana.ini file. For example, GF_SERVER_SERVE_FROM_SUB_PATH=truw will expand to serve_from_sub_path = true in the [server] section of grafana.ini:
grafana:
image: teslamate/grafana:latest
restart: always
environment:
- DATABASE_HOST=${DATABASE_HOST:-database}
- DATABASE_NAME=${DATABASE_NAME:-teslamate}
- DATABASE_USER=${DATABASE_USER:-teslamate}
- DATABASE_PASS=${DATABASE_PASS:-teslamate}
- GF_SERVER_DOMAIN=${FQDN_TM}
#--- default
#- GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s:%(http_port)s/grafana
#--- hard code https for redirection, but protocol remains http between nginx and grafana. (note: one can use https between nginx and grafana also)
- GF_SERVER_ROOT_URL=https://%(domain)s/grafana/
- GF_SERVER_SERVE_FROM_SUB_PATH=true
#---
- GF_SERVER_PROTOCOL=http
#- GF_LOG_LEVEL=debug
ports:
- "${GRAFANA_HTTP_PORT:-3000}:3000"
volumes:
- ./data/teslamate-grafana-data:/var/lib/grafana
The .env file:
TESLAMATE_HTTP_PORT=4000
GRAFANA_HTTP_PORT=3002
FQDN_TM=teslamate.example.com
TM_TZ=Europe/Zurich
DATABASE_HOST=database
DATABASE_NAME=teslamate
DATABASE_USER=teslamate
DATABASE_PASS=supersecretpassword