| 1 | # Substation |
| 2 | |
| 3 | `https://substation.ai.paperclover.net` serves the Substation dashboard and its |
| 4 | native APIs. Snow OAuth protects the dashboard and requires `infra-admin`. |
| 5 | Substation's trusted-proxy mode accepts the identity header from the local |
| 6 | OAuth proxy. Caddy sends `/v1/*` and `/v0/management/*` directly to the hub, |
| 7 | which checks its own client and management keys. |
| 8 | |
| 9 | ## Set up accounts |
| 10 | |
| 11 | 1. Open the dashboard and sign in through Snow. |
| 12 | 2. Open **Settings → Hub accounts** and add a Claude or Codex account. |
| 13 | 3. Follow the provider's sign-in link. For Codex, enter the displayed device |
| 14 | code at the provider's page; for Claude, paste the returned code into |
| 15 | Substation. Repeat for each account. |
| 16 | 4. Check that each account shows its quota and can take requests. |
| 17 | |
| 18 | The container includes both providers' login tools. Login scratch directories |
| 19 | are disposable; the hub saves and refreshes the resulting account files in its |
| 20 | own store. Production starts empty and imports no CPA accounts or history. |
| 21 | |
| 22 | ## Connect clients |
| 23 | |
| 24 | The client key is saved on Sandwich at |
| 25 | `~/.config/substation/client-key`, readable only by Clover. SSO mode deliberately |
| 26 | does not reveal or rotate keys through Substation's dashboard. Nomad variable |
| 27 | `nomad/jobs/substation` owns the initial `api_key` and `management_key`. |
| 28 | |
| 29 | For Claude Code: |
| 30 | |
| 31 | ```sh |
| 32 | export SUBSTATION_CLIENT_KEY="$(cat ~/.config/substation/client-key)" |
| 33 | export ANTHROPIC_BASE_URL=https://substation.ai.paperclover.net |
| 34 | export ANTHROPIC_AUTH_TOKEN="$SUBSTATION_CLIENT_KEY" |
| 35 | claude |
| 36 | ``` |
| 37 | |
| 38 | For Codex, use a separate `CODEX_HOME` to keep its normal sign-in and settings: |
| 39 | |
| 40 | ```sh |
| 41 | mkdir -p ~/.codex-substation |
| 42 | ``` |
| 43 | |
| 44 | Save this as `~/.codex-substation/config.toml`: |
| 45 | |
| 46 | ```toml |
| 47 | model = "gpt-6.1-sol" |
| 48 | model_provider = "substation" |
| 49 | |
| 50 | [model_providers.substation] |
| 51 | name = "Substation" |
| 52 | base_url = "https://substation.ai.paperclover.net/v1" |
| 53 | env_key = "SUBSTATION_CLIENT_KEY" |
| 54 | wire_api = "responses" |
| 55 | ``` |
| 56 | |
| 57 | Then run: |
| 58 | |
| 59 | ```sh |
| 60 | export SUBSTATION_CLIENT_KEY="$(cat ~/.config/substation/client-key)" |
| 61 | CODEX_HOME="$HOME/.codex-substation" codex |
| 62 | ``` |
| 63 | |
| 64 | The [official OpenAI configuration documentation](https://learn.chatgpt.com/docs/config-file/config-advanced) |
| 65 | describes custom providers. Choose a model the connected account supports. |
| 66 | Substation serves Anthropic Messages and OpenAI Responses; it does not implement |
| 67 | `/v1/chat/completions`. |
| 68 | |
| 69 | T3 Code's hub quota integration uses the same base address and the separate |
| 70 | management key from `~/.config/substation/management-key` on Sandwich. The |
| 71 | management API is `/v0/management`; it remains behind its own bearer key. The |
| 72 | public reverse proxy reaches the hub over loopback, so the hub's remote |
| 73 | management switch is unnecessary. A client key grants no management access. |
| 74 | |
| 75 | ## Persistence and deployment |
| 76 | |
| 77 | Production state lives under `/srv/prod/substation/data`: `auth` holds provider |
| 78 | accounts, `hub` holds routing state, `dashboard` holds SQLite history and |
| 79 | settings, `home` holds the login tools' home, and `run` holds build identities. |
| 80 | Nomad owns the generated keys and OAuth client credentials. Existing numeric |
| 81 | service identities remain reserved when a service is retired. |
| 82 | |
| 83 | `service/substation/build-source.json` pins a local Jujutsu revision of |
| 84 | `../substation` and its release version. `service/substation/Dockerfile` builds |
| 85 | the program and includes pinned Claude and Codex login tools. The exporter |
| 86 | includes its source manifest as build provenance. Snow Globe receives source and builds locally; |
| 87 | it receives no GitHub SSH key or repository credentials. |
| 88 | |
| 89 | Stage with `python3 tools/deploy.py stage substation`. Previews start with empty |
| 90 | storage, their own keys, and a separate Snow OAuth client. Updates preserve |
| 91 | that preview's state. Use disposable accounts and prompts when testing. |
| 92 | Production uses described, conflict-free main and `python3 tools/deploy.py prod`. |
| 93 | The in-app updater is unavailable inside the immutable image. |
| 94 | |
| 95 | CPA's live dataset, preview, keys, and routing are retired after the replacement |
| 96 | passes production checks. Restricted off-server recovery copies retain the old |
| 97 | store; restoring them requires a separate explicit request. |
| 98 | |
| 99 | ## Request diagnostics |
| 100 | |
| 101 | API responses include a Caddy-generated `X-Request-ID`. API access records in |
| 102 | Caddy's journal contain that ID, trace ID, path, method, protocol, status, bytes, |
| 103 | duration, declared content length, upstream address, upstream status, and time |
| 104 | to upstream headers. Other headers, query strings, cookies, and request/response |
| 105 | bodies are excluded from these access records. The log shipper retains them in |
| 106 | Victoria Logs. |
| 107 | |
| 108 | The `monitor` task checks anonymous rejection, client-key model access, and |
| 109 | management-key version access through HTTPS every minute. It requires the edge |
| 110 | request ID and rejects redirects, so an OAuth login page cannot pass as a healthy |
| 111 | API. It sends no generation requests and logs only check outcomes and IDs. |
| 112 | |
| 113 | Inspect `substation` task `monitor` in the service logs, and search Caddy's |
| 114 | journal for the response's request ID. A 499 with no upstream status can mean |
| 115 | client cancellation or Caddy's request-body idle timeout; an upstream 499 is |
| 116 | recorded separately. |
| 117 | API 499 responses default to `X-Should-Retry: true` so Claude can recover from |
| 118 | request-body timeouts; an upstream's explicit retry decision takes precedence. |
| 119 | Caddy-generated API 499/504 errors close HTTP/1 connections and include the |
| 120 | request ID and retry hint. A timed-out upload can still have queued body bytes; |
| 121 | reusing that connection lets those bytes become an invalid next request and |
| 122 | can produce an unlogged 431 before routing. Upstream responses keep their |
| 123 | retry decisions and connection behavior. |
| 124 | The packaged [Caddy patch](../nixos/caddy-body-timeout.patch) preserves failed |
| 125 | read deadlines while the upstream transport closes the body. Otherwise, |
| 126 | net/http can block draining a small unfinished upload before error handling. |
| 127 | Compare `bytes_read` with `content_length` to identify incomplete fixed-length |
| 128 | uploads. A missing length can mean a chunked request; it does not prove completion. |
| 129 | A 200 followed by a truncated stream needs the edge byte count and the hub's |
| 130 | stream outcome. A successful probe does not establish that every generation |
| 131 | stream completed. |
| 132 | |
| 133 | `snowglobe-substation-watch.service` follows Caddy with a persisted journal |
| 134 | cursor and samples TCP counters once per second for the hub and known API |
| 135 | clients. It survives service restarts and reboots. The latest two minutes are |
| 136 | retained in `/var/lib/snowglobe/substation-watch/recent.json`; each 499 or 5xx |
| 137 | retains its request metadata and surrounding TCP progress in a private file |
| 138 | named by request ID. The directory keeps the latest 100 incidents. No packet |
| 139 | payloads, request bodies, cookies, or credentials are collected. |
| 140 | |
| 141 | Watcher heartbeat and incident summaries appear in its journal and Victoria |
| 142 | Logs. `upload_complete: false` proves an incomplete fixed-length upload; |
| 143 | the saved receive/send queues, byte counters, receive windows, and retransmits |
| 144 | distinguish stalled sending from server backpressure and network loss. |
| 145 | Incident files also retain matching TCP sequence numbers, acknowledgements, |
| 146 | SACK ranges, windows and flags. These establish packet loss and which side |
| 147 | sent FIN/RST first. Only parsed transport fields are retained. |
| 148 | |
| 149 | Worktop's `net.paperclover.substation-watch` LaunchAgent records matching |
| 150 | Substation sockets under `~/.local/state/substation-watch`. Minute files retain |
| 151 | 12 hours of process IDs, CPU time/state, TCP queues, byte counters, |
| 152 | retransmissions and windows. |
| 153 | Paired captures showed Worktop leave a SACK-reported hole unrepaired for 75–109 |
| 154 | seconds. `net.paperclover.tcp-rack-off` disables RACK before login while retaining |
| 155 | SACK recovery; it is a system-wide transport trade, not a Substation setting. |
| 156 | The root-owned LaunchDaemon is installed from the exported |
| 157 | `tools/net.paperclover.tcp-rack-off.plist` and should exit zero after each boot. |
| 158 | The installed script is an exported revision of `tools/substation-client-watch.py`; |
| 159 | the LaunchAgent follows its `~/.local/share/substation-watch/current.py` link. |
| 160 | Match UTC timestamps to the server incident before comparing TCP progress. |
| 161 | An upload with queued bytes and no acknowledgement progress for 10 seconds |
| 162 | retains the preceding 120 socket samples in a private `stall-<timestamp>.json` file. |
| 163 | The latest 100 stalls survive rolling-history expiry and watcher restarts. |
| 164 | Each stalled socket has one archive, updated after the stall and after packet |
| 165 | ingestion catches up, retaining later retransmissions and closure. |
| 166 | When an administrator starts a TCP summary capture into `tcp-headers.log`, the |
| 167 | client watcher retains up to two minutes of matching sequence/ACK/SACK metadata |
| 168 | with each stall, bounded to 100,000 packets. |
| 169 | It consumes and truncates that append-only log at 64 MiB without restarting |
| 170 | tcpdump. Archives mark each truncation; exclude that instant from packet-loss |
| 171 | conclusions because concurrent writes can be lost during truncation. |
| 172 | Packet ingestion runs independently of socket sampling. Samples record unread |
| 173 | capture bytes, active ingestion and errors. Infer missing sender packets only |
| 174 | after ingestion finishes and the backlog clears, with no capture error or |
| 175 | truncation gap in that interval. The LaunchAgent uses `ProcessType: Standard`. |