# Grafana NO DATA debugging + monitoring hygiene (2026-09-02 session)

Symptom: dashboard panels show only "NO DATA" while Prometheus itself has data.
Three distinct root causes hit in one session — check in this order.

## 1. Datasource `uid` not pinned → Grafana generates a random one

If `uid:` is missing from the datasource in
`grafana/provisioning/datasources/datasources.yml`, Grafana assigns a RANDOM uid
at provisioning time (visible in its sqlite: `SELECT uid FROM data_source` →
e.g. `P8E80F9AEF21F6940`). Dashboards reference datasources by `"uid"` in each
panel (`"uid": "Prometheus"`), so panels silently find nothing → NO DATA.

Fix: pin the uid to match the dashboards:
```yaml
- name: Prometheus
  type: prometheus
  uid: Prometheus        # must match panel "uid" refs
  url: http://172.28.0.1:9090
  isDefault: true
```

## 2. Prometheus on host network → `http://prometheus:9090` unresolvable from Grafana

Prometheus + blackbox run `network_mode: host` (needed to reach WireGuard
10.8.0.0/24), while Grafana is on the docker bridge network. The service name
`prometheus` does NOT resolve inside the bridge net
(`docker exec grafana getent hosts prometheus` → nothing). Loki, being on the
same bridge net, resolves fine — so one datasource works and the other doesn't.

Fix: point the datasource at the host via the bridge gateway:
```yaml
url: http://172.28.0.1:9090   # gateway IP = host as seen from Grafana's bridge net
```
Find the gateway: `docker inspect grafana --format '{{range .NetworkSettings.Networks}}{{.Gateway}} {{end}}'`
Verify from inside: `docker exec grafana wget -qO- http://172.28.0.1:9090/-/healthy`
Caveat: the bridge subnet is stable per docker daemon, but recreating the network
can change it.

## 3. Panels/alerts reference NON-EXISTENT metric names

Garage admin-API metrics (:3903) have NO `garage_` prefix. Only
`garage_build_info`, `garage_local_disk_avail/total`, `garage_replication_factor`
carry the prefix. Everything else is bare: `api_s3_request_counter`,
`block_resync_queue_length`, `block_resync_errored_blocks`, `cluster_*`
(connected_nodes, healthy, partitions_all_ok, layout_node_connected), `table_*`, `rpc_*`.

Phantom names that silently render NO DATA (never fire / never plot):
- `garage_block_count`         → real: `block_resync_queue_length` + `block_resync_errored_blocks`
- `garage_rpc_node_health_is_up` → real: `cluster_layout_node_connected`
- `garage_api_s3_request_counter` → real: `api_s3_request_counter` (use `rate(api_s3_request_counter[5m])`)
- `garage_block_resync_error_count` → real: `block_resync_errored_blocks`

Check what actually exists before writing queries:
`curl 'http://127.0.0.1:9090/api/v1/label/__name__/values'` filtered by `job=garage`,
or `curl 'http://127.0.0.1:9090/api/v1/query?query={job="garage"}'` and list `__name__`.

## Provisioning reload semantics (Grafana)

- Dashboards: re-read automatically (~30s, updateIntervalSeconds).
- Datasources: provisioned ONLY at Grafana startup → after editing
  datasources.yml you MUST `docker compose restart grafana`.

## Readable legends: relabel instance → hostname (not the raw scrape address)

By default `instance` = scrape address: tproxy shows `127.0.0.1:18081` (the LOCAL
end of the SSH tunnel — NOT "monitoring the local interface", just the transport),
garage shows `10.8.0.x:3903`. Fix at the SOURCE in prometheus.yml relabel_configs,
not per-panel legend, so panels/explore/alerts are consistent:
```yaml
relabel_configs:
  - target_label: instance
    replacement: vps03:8081                 # tproxy
  - source_labels: [__address__]            # garage / node IP→hostname mapping
    regex: 10.8.0.2:3903
    target_label: instance
    replacement: bigbox:3903
```
Result legends: `vps01:3903 / bigbox:3903 / vps02:3903` (garage),
`vps03:8081` (tproxy), `vps01:9100` etc (node). After changing relabel, restart
Prometheus and re-check `/api/v1/targets` — new labels apply immediately.

## Alert rules: same phantom-name trap

alerts.yml had `garage_block_resync_error_count` and `garage_rpc_node_health_is_up`
(never fire). Fixed to `block_resync_errored_blocks > 0` and
`cluster_layout_node_connected == 0`. Validate with
`docker exec prometheus promtool check config /etc/prometheus/prometheus.yml`
→ expect "6 rules found", all rules eval.

## Grafana sqlite inspection (no CLI auth needed)

Grafana's admin password may be unknown (changed in UI). Inspect state directly:
```
docker cp grafana:/var/lib/grafana/grafana.db /tmp/grafana.db
python3 -c "import sqlite3; print(sqlite3.connect('/tmp/grafana.db').execute('SELECT name,uid,url FROM data_source').fetchall())"
# dashboard JSON lives in table dashboard, column data
```