1# Production data restore
2
3Every production promotion and code rollback backs up the active release
4before switching it. One ZFS snapshot operation captures mounted managed
5service datasets and PostgreSQL data together. A temporary isolated PostgreSQL
6clone produces service-owned database dumps and is destroyed before the
7backup is published. A failed deployment can leave a completed backup without
8a successful release-history entry.
9
10Services can set `snapshots = false` in Pkl for disposable managed data, as Local
11AI does for its context cache. Pool provisioning writes the policy to the
12dataset's `snowglobe:snapshots` property. Database dumps and their shared
13PostgreSQL snapshot remain included.
14
15`python3 tools/deploy.py backups` lists backup IDs and services. Each backup's
16private manifest records the service snapshots, database dumps and checksums,
17and a private `nomad.snap`. The Nomad snapshot includes jobs, ACLs, and secret
18variables. It is for explicit cluster recovery; service restore does not apply
19it. Earlier backups may lack this snapshot. Manifests and dumps remain on the
20OS disk; see [hardware and recovery](snowglobe-hardware.md) for off-server
21recovery requirements.
22
23To restore one service after production data restore is explicitly authorized:
24
25```sh
26python3 tools/deploy.py backups
27python3 tools/deploy.py rollback <old-release-id>
28python3 tools/deploy.py data-restore <service-id> --backup <backup-id> --discard-writes
29```
30
31The backup's `fromRelease` must match the active release. The command verifies
32backup snapshots and dumps before stopping the service, saves the current
33dataset and database as a safety copy, restores the selected data, then
34deploys and checks the service. **Writes after the chosen backup are discarded
35from the live service.** The safety snapshot and dump remain on the host.
36
37Restore PostgreSQL owner services individually; the PostgreSQL service itself
38is shared. Clover, Media, and external archive mounts are outside managed
39service restore. Restoring an entire shared dataset discards unrelated writes.
40For Shale, keep the complete service dataset together: repository visibility,
41repository objects, issue database, attachment files, and their manifest must
42agree. For Copyparty, also preserve its Nomad API-account hashes as described
43in [administration](copyparty.md). For Knife, preserve the external archive
44store and identity as described in [operations](knife.md).
45
46Verify the real application, authentication, persistence, provider connections,
47allocation health and logs after recovery. Code rollback only changes code;
48it does not restore data or undo a folder move.