Files
xmpp-server-prosody/references/docker-ip-shift-nginx-stale-dns.md
2026-09-06 13:51:11 +00:00

71 lines
3.2 KiB
Markdown

# Docker IP shift: nginx stale upstream → WS 502 → Converse spinner
Symptom (2026-08-30, chat.nixg.ru): page loads fine, the login form flashes briefly,
then Converse shows only the pulsing white "connecting" circle forever. No config
change was made — the break appeared after container recreation.
## Root cause
`icq-webchat`'s nginx resolves `proxy_pass http://prosody:5280` ONCE at config load and
caches the IP for the life of the process. Docker bridge IPs are assigned at container
creation; after `docker compose up` recreates containers (or adds a service) any
container can move to a different IP. In this incident:
- webchat started 42h ago → cached the OLD mapping (prosody = 172.27.0.2)
- prosody recreated 26h ago, slidgram created 22h ago → IPs shifted
- new mapping: slidgram = 172.27.0.2, webchat = 172.27.0.3, prosody = 172.27.0.4
Every `/xmpp-websocket` request was proxied to Slidge's port 5280 (closed) →
Connection refused → 502 → client never connects → spinner forever.
## Diagnostic trail (fast)
```bash
docker logs icq-webchat --tail 40 | grep -E 'xmpp-websocket|refused'
# → connect() failed (111: Connection refused) while connecting to upstream
# upstream: "http://172.27.0.2:5280/xmpp-websocket" ← STALE IP in the error line
docker compose ps --format '{{.Name}}\t{{.Status}}' # "Up X hours" reveals who was recreated
# get ACTUAL container IPs on the compose network:
docker network inspect icq_default --format '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{"\n"}}{{end}}'
```
Compare the upstream IP in the nginx error with the current IP of `prosody`: mismatch =
stale DNS cache. Note: `docker logs icq-prosody` can show 0 lines because Prosody logs
to the mounted volume — read `/opt/icq/logs/prosody.log` instead (it is the live log,
debug-level, presence/roster traffic from the Telegram bridge is visible there).
## Fix
```bash
cd /opt/icq && docker compose restart webchat # nginx re-resolves "prosody" → new IP
```
Verify: `docker logs icq-webchat --since 2m | grep -cE '502|refused'` → 0. The user must
F5 the tab — an already-open tab stuck on the spinner does NOT auto-reconnect.
## Permanent hardening (avoid recurrence)
In `webchat/nginx.conf`, force per-request DNS via Docker's embedded resolver + a
variable in `proxy_pass` (the documented nginx pattern: a static hostname in proxy_pass
is resolved only at config load; a variable makes nginx consult the resolver per request):
```nginx
location /xmpp-websocket {
resolver 127.0.0.11 valid=30s; # Docker embedded DNS, re-resolve every 30s
set $prosody prosody:5280;
proxy_pass http://$prosody/xmpp-websocket;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
}
```
## Related
- Blank page (NOTHING renders, ESM `import.meta` SyntaxError) → `conversejs-blank-page-and-auth-testing.md`
- Caddy bind-mount inode trap (edit silently ignored) → `webchat-conversejs-and-caddy-trap.md`
- This is the same class as any nginx-in-Docker stale-upstream issue; the tell is the IP
in the 502 line belonging to a different container than the one named in proxy_pass.