xmpp-server-prosody/references/docker-ip-shift-nginx-stale-dns.md

Docker IP shift: nginx stale upstream → WS 502 → Converse spinner

Symptom (2026-08-30, chat.nixg.ru): page loads fine, the login form flashes briefly, then Converse shows only the pulsing white "connecting" circle forever. No config change was made — the break appeared after container recreation.

Root cause

icq-webchat's nginx resolves proxy_pass http://prosody:5280 ONCE at config load and caches the IP for the life of the process. Docker bridge IPs are assigned at container creation; after docker compose up recreates containers (or adds a service) any container can move to a different IP. In this incident:

• webchat started 42h ago → cached the OLD mapping (prosody = 172.27.0.2)
• prosody recreated 26h ago, slidgram created 22h ago → IPs shifted
• new mapping: slidgram = 172.27.0.2, webchat = 172.27.0.3, prosody = 172.27.0.4

Every /xmpp-websocket request was proxied to Slidge's port 5280 (closed) → Connection refused → 502 → client never connects → spinner forever.

Diagnostic trail (fast)

docker logs icq-webchat --tail 40 | grep -E 'xmpp-websocket|refused'
# → connect() failed (111: Connection refused) while connecting to upstream
#   upstream: "http://172.27.0.2:5280/xmpp-websocket"   ← STALE IP in the error line
docker compose ps --format '{{.Name}}\t{{.Status}}'   # "Up X hours" reveals who was recreated
# get ACTUAL container IPs on the compose network:
docker network inspect icq_default --format '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{"\n"}}{{end}}'

Compare the upstream IP in the nginx error with the current IP of prosody: mismatch = stale DNS cache. Note: docker logs icq-prosody can show 0 lines because Prosody logs to the mounted volume — read /opt/icq/logs/prosody.log instead (it is the live log, debug-level, presence/roster traffic from the Telegram bridge is visible there).

Fix

cd /opt/icq && docker compose restart webchat   # nginx re-resolves "prosody" → new IP

Verify: docker logs icq-webchat --since 2m | grep -cE '502|refused' → 0. The user must F5 the tab — an already-open tab stuck on the spinner does NOT auto-reconnect.

Permanent hardening (avoid recurrence)

In webchat/nginx.conf, force per-request DNS via Docker's embedded resolver + a variable in proxy_pass (the documented nginx pattern: a static hostname in proxy_pass is resolved only at config load; a variable makes nginx consult the resolver per request):

location /xmpp-websocket {
    resolver 127.0.0.11 valid=30s;           # Docker embedded DNS, re-resolve every 30s
    set $prosody prosody:5280;
    proxy_pass http://$prosody/xmpp-websocket;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;
}

Related

• Blank page (NOTHING renders, ESM import.meta SyntaxError) → conversejs-blank-page-and-auth-testing.md
• Caddy bind-mount inode trap (edit silently ignored) → webchat-conversejs-and-caddy-trap.md
• This is the same class as any nginx-in-Docker stale-upstream issue; the tell is the IP in the 502 line belonging to a different container than the one named in proxy_pass.