---
name: garage-s3-cluster
description: Garage S3 cluster status checks on vps01/bigbox/vps02.
---

# Garage S3 cluster (vps01 / bigbox / vps02)

Garage = self-hosted S3-compatible storage cluster, replication_factor=3, read/write_quorum=2.
Runs in Docker on bigbox from /opt/garage. Bucket "obsidian" holds obsidian-vault backups.

## Topology (WireGuard 10.8.0.0/24)
- vps01: 10.8.0.1:3901 (hostname 5599453-kpa39l, zone vps01)
- bigbox: 10.8.0.2:3901 (zone home) — /opt/garage, wg0 = 10.8.0.2/24
- vps02: 10.8.0.4:3901 (zone vps02)
- S3 API (incl. admin API): 0.0.0.0:3900 on every node; RPC: WG IP:3901
- Config: /opt/garage/garage.toml (rpc_secret, admin_token, bootstrap_peers inside)

## Status check — the reliable way
Container `dxflrs/garage:v2.1.0` is SCRATCH: no shell, no CLI in PATH.
`docker exec garage garage ...` and `docker exec garage sh ...` both fail with
"executable file not found". The CLI binary lives at `/garage` INSIDE the image — run it
with `--entrypoint`, mounting BOTH config and meta (the node key lives in meta/; skipping
the meta mount gives "Unable to read node key. It will be generated..."):

```
docker run --rm \
  -v /opt/garage/garage.toml:/etc/garage.toml:ro \
  -v /opt/garage/meta:/var/lib/garage/meta \
  --entrypoint /garage dxflrs/garage:v2.1.0 status
```

Subcommands that work at top level: `status` (health table — HEALTHY NODES),
`stats` (block manager + cluster-wide usage), `bucket list`.
Pitfall: `garage cluster status` does NOT exist in v2.1 — "Found argument 'cluster' which
wasn't expected". There is no `cluster` subcommand; status/stats are top-level.

Healthy signals in `garage stats`: "resync queue length: 0", "blocks with resync errors: 0",
MklTodo/GcTodo/InsQueue all 0.

## Admin HTTP API (port 3900) — do not rely on it
- `Authorization: Bearer <admin_token>` → "Unsupported authorization method" (S3-style XML error)
- `X-Garage-Admin-Token: <admin_token>` → AccessDenied "anonymous access"
- `/v2/status`, `/v1/status`, `/cluster/status` all same behavior.
- The CLI route above is the dependable path; no working HTTP admin call was found.

## Files in /opt/garage (bigbox)
- docker-compose.yml — service garage, network_mode: host, volumes: garage.toml, meta/, data/
- garage.toml — full config; admin_token duplicated here and in admin_token file
- admin_token — admin token (65 hex chars); secret — RPC secret (not for admin API)
- cli/ — BROKEN: garage-v2.1.0-linux-x86_64.tar.gz is a 10-byte "Not Found" placeholder. Ignore; use the in-image CLI above.
- meta/, data/ — LMDB storage (db_engine = "lmdb")

## Reachability probe (over WG)
```
for ip in 10.8.0.1 10.8.0.2 10.8.0.4; do
  timeout 3 bash -c "echo > /dev/tcp/$ip/3901" 2>/dev/null && echo "$ip:3901 OK" || echo "$ip:3901 FAIL"
done
```

Script: `scripts/garage-status.sh` — runs the CLI status command with correct mounts.