garage-s3-cluster/SKILL.md

name, description

name	description
garage-s3-cluster	Garage S3 cluster status checks on vps01/bigbox/vps02.

Garage S3 cluster (vps01 / bigbox / vps02)

Garage = self-hosted S3-compatible storage cluster, replication_factor=3, read/write_quorum=2. Runs in Docker on bigbox from /opt/garage. Bucket "obsidian" holds obsidian-vault backups.

Topology (WireGuard 10.8.0.0/24)

• vps01: 10.8.0.1:3901 (hostname 5599453-kpa39l, zone vps01)
• bigbox: 10.8.0.2:3901 (zone home) — /opt/garage, wg0 = 10.8.0.2/24
• vps02: 10.8.0.4:3901 (zone vps02)
• S3 API (incl. admin API): 0.0.0.0:3900 on every node; RPC: WG IP:3901
• Config: /opt/garage/garage.toml (rpc_secret, admin_token, bootstrap_peers inside)

Status check — the reliable way

Container dxflrs/garage:v2.1.0 is SCRATCH: no shell, no CLI in PATH. docker exec garage garage ... and docker exec garage sh ... both fail with "executable file not found". The CLI binary lives at /garage INSIDE the image — run it with --entrypoint, mounting BOTH config and meta (the node key lives in meta/; skipping the meta mount gives "Unable to read node key. It will be generated..."):

docker run --rm \
  -v /opt/garage/garage.toml:/etc/garage.toml:ro \
  -v /opt/garage/meta:/var/lib/garage/meta \
  --entrypoint /garage dxflrs/garage:v2.1.0 status

Subcommands that work at top level: status (health table — HEALTHY NODES), stats (block manager + cluster-wide usage), bucket list. Pitfall: garage cluster status does NOT exist in v2.1 — "Found argument 'cluster' which wasn't expected". There is no cluster subcommand; status/stats are top-level.

Healthy signals in garage stats: "resync queue length: 0", "blocks with resync errors: 0", MklTodo/GcTodo/InsQueue all 0.

Admin HTTP API (port 3900) — do not rely on it

• Authorization: Bearer <admin_token> → "Unsupported authorization method" (S3-style XML error)
• X-Garage-Admin-Token: <admin_token> → AccessDenied "anonymous access"
• /v2/status, /v1/status, /cluster/status all same behavior.
• The CLI route above is the dependable path; no working HTTP admin call was found.

Files in /opt/garage (bigbox)

• docker-compose.yml — service garage, network_mode: host, volumes: garage.toml, meta/, data/
• garage.toml — full config; admin_token duplicated here and in admin_token file
• admin_token — admin token (65 hex chars); secret — RPC secret (not for admin API)
• cli/ — BROKEN: garage-v2.1.0-linux-x86_64.tar.gz is a 10-byte "Not Found" placeholder. Ignore; use the in-image CLI above.
• meta/, data/ — LMDB storage (db_engine = "lmdb")

Reachability probe (over WG)

for ip in 10.8.0.1 10.8.0.2 10.8.0.4; do
  timeout 3 bash -c "echo > /dev/tcp/$ip/3901" 2>/dev/null && echo "$ip:3901 OK" || echo "$ip:3901 FAIL"
done

Script: scripts/garage-status.sh — runs the CLI status command with correct mounts.