| 1 | # Production data restore |
| 2 | |
| 3 | Every production promotion and code rollback backs up the active release |
| 4 | before switching it. One ZFS snapshot operation captures mounted managed |
| 5 | service datasets and PostgreSQL data together. A temporary isolated PostgreSQL |
| 6 | clone produces service-owned database dumps and is destroyed before the |
| 7 | backup is published. A failed deployment can leave a completed backup without |
| 8 | a successful release-history entry. |
| 9 | |
| 10 | Services can set `snapshots = false` in Pkl for disposable managed data, as Local |
| 11 | AI does for its context cache. Pool provisioning writes the policy to the |
| 12 | dataset's `snowglobe:snapshots` property. Database dumps and their shared |
| 13 | PostgreSQL snapshot remain included. |
| 14 | |
| 15 | `python3 tools/deploy.py backups` lists backup IDs and services. Each backup's |
| 16 | private manifest records the service snapshots, database dumps and checksums, |
| 17 | and a private `nomad.snap`. The Nomad snapshot includes jobs, ACLs, and secret |
| 18 | variables. It is for explicit cluster recovery; service restore does not apply |
| 19 | it. Earlier backups may lack this snapshot. Manifests and dumps remain on the |
| 20 | OS disk; see [hardware and recovery](snowglobe-hardware.md) for off-server |
| 21 | recovery requirements. |
| 22 | |
| 23 | To restore one service after production data restore is explicitly authorized: |
| 24 | |
| 25 | ```sh |
| 26 | python3 tools/deploy.py backups |
| 27 | python3 tools/deploy.py rollback <old-release-id> |
| 28 | python3 tools/deploy.py data-restore <service-id> --backup <backup-id> --discard-writes |
| 29 | ``` |
| 30 | |
| 31 | The backup's `fromRelease` must match the active release. The command verifies |
| 32 | backup snapshots and dumps before stopping the service, saves the current |
| 33 | dataset and database as a safety copy, restores the selected data, then |
| 34 | deploys and checks the service. **Writes after the chosen backup are discarded |
| 35 | from the live service.** The safety snapshot and dump remain on the host. |
| 36 | |
| 37 | Restore PostgreSQL owner services individually; the PostgreSQL service itself |
| 38 | is shared. Clover, Media, and external archive mounts are outside managed |
| 39 | service restore. Restoring an entire shared dataset discards unrelated writes. |
| 40 | For Shale, keep the complete service dataset together: repository visibility, |
| 41 | repository objects, issue database, attachment files, and their manifest must |
| 42 | agree. For Copyparty, also preserve its Nomad API-account hashes as described |
| 43 | in [administration](copyparty.md). For Knife, preserve the external archive |
| 44 | store and identity as described in [operations](knife.md). |
| 45 | |
| 46 | Verify the real application, authentication, persistence, provider connections, |
| 47 | allocation health and logs after recovery. Code rollback only changes code; |
| 48 | it does not restore data or undo a folder move. |