diff --git a/AGENTS.md b/AGENTS.md index 6af8a81b108f652b948b67eb8e8e81a3c429d6ca..f70bf30f32217a56f6e2acb2c2e4036e128facba 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,182 +1,120 @@ -# Snow Globe infrastructure - -This repo runs Clover's home server: her creative file store, shared apps, and -virtual machines. NixOS provides the host; Nomad schedules Podman containers; -Pkl describes services; the Rust/Solid dashboard manages the system. Treat -availability and existing data as part of correctness. - -## Access and authorization - -- Local edits and staging previews on Zenith are always implied by a task. - Carry them through verification without asking for permission again. Use - disposable preview data for writes and keep active production services intact. -- Production changes require a request to take the work to prod. A request to - fix, stage, test, or commit something does not authorize production deployment. - Once prod is requested, complete the normal deployment workflow. -- Read-only infrastructure inspection over SSH is allowed for the task. Confirm - the target host before a remote mutation. Set both deployment host and port - explicitly; the CLI's legacy defaults still point at the rehearsal VM. -- Private file contents require authorization per source. Infrastructure access, - a mounted share, or permission to list a tree does not authorize opening its - contents. Keep private material and derived model output inside its store. -- Never print credentials or encryption keys into chat, command arguments, - ordinary logs, or tracked files. Use existing private files or stdin for - imports; keep secret files owner-readable only. Nomad variables own service - secrets; configuration declares names and references, not values. -- Before destructive storage or OS work, require verified recovery: tested key - recovery and app/database restores, with at least two restricted copies stored - separately off the server. Local snapshots and a key archive alone do not - establish data recovery. Follow the linked cutover runbooks. -- Agent/model trials use disposable data and credentials, separated from the - personal archive and host credentials. A service preview may clone production - data and secrets; staging does not itself grant permission to inspect them. - -## Machines and paths - -| Surface | Connection / role | Configuration | -| --- | --- | --- | -| Mac checkout | `/Users/clo/dev/infra-2`; edit and upload here | This working copy | -| Optional rehearsal NixOS VM | `root@127.0.0.1`, SSH port `2222` | `nixos/vm.nix`, `studio.test`, demo pool | -| Zenith production | `root@10.0.0.1`, SSH port `22` | `nixos/zenith.nix`, `paperclover.net`, physical pool | - -`tools/deploy.py` reads `STUDIO_DEPLOY_HOST` and `STUDIO_DEPLOY_PORT`; defaults -are the rehearsal connection above. Set both explicitly for Zenith staging -and production. Prefer staging previews on Zenith for service and dashboard -verification; use the VM when host or guest isolation is required. -Host settings and dataset names have their canonical home in `nixos/zenith.nix` -and `config/site.pkl`; inspect them instead of copying values into new tooling. - -| Local / logical location | Remote location | Meaning | -| --- | --- | --- | -| Checkout release sources | `/opt/studio/releases/` | Uploaded, immutable release trees | -| Published main revision | `/opt/studio/main` | Release exported from described `main` | -| Active deployment | `/opt/studio/current` | Active release link; not an editable checkout | -| Zenith host installation | `/etc/nixos` | Installed host configuration; release activation also manages NixOS | -| Mac SMB `/Volumes/clover` | Zenith `/srv/clover` | Live personal files; not a disposable test tree | -| `/Volumes/clover/Media` | Zenith `/srv/clover/Media` | Media child dataset | -| Mac `/Volumes/media` (launcher default) | VM `/mnt/media-source` → `/srv/clover/Media` | Read-only 9P export and bindfs view for rehearsal | -| Managed service storage | `/srv/prod/` | Persistent, per-service ZFS datasets | -| Preview storage | `/srv/staging/` | Disposable service clones, with separate external clones where needed | -| VM storage | `/srv/vm` | Guest disks, access files, and installation library | -| Deployment/control state | `/var/lib/studio` | UID registry, stages, history, backups, dashboard state, run logs | -| Nomad state | `services.nomad.settings.data_dir` in `nixos/configuration.nix` | Separate from service data; includes ACLs and secrets | - -The rehearsal VM has its own Clover dataset. Its media view references the Mac's -mounted share read-only; those bytes are not synthetic fixtures. See -`tools/local-vm.sh` for launch/cache overrides and `tools/mac-domains.py` for -Mac rehearsal routing. Do not hand-edit DNS, trust, or host files to bypass them. - -## Where changes belong - -- `config/Service.pkl` owns the service schema; `config/site.pkl` owns site - defaults and overrides. `config/excluded-services.json` filters deployment - sources. Services are discovered from `service/*/*.pkl`. -- `service//service.pkl` owns a service's containers, routes, mounts, - requirements, secret declarations, health checks, and resource limits. - Sibling Pkl files define additional services, e.g. Postgres's pgAdmin. -- `tools/studio.py` owns rendering, provisioning, staging, and service setup. - `tools/deploy.py` uploads local sources or `main`; `tools/release.py` owns - release activation and code rollback; `tools/data.py` owns backup/restore. -- `tools/router.py` generates Caddy routes from service discovery. Change the - declaration or generator, not generated routes or Nomad jobs. -- `nixos/` owns host units, packages, isolation, mounts, and hardware settings. - `dashboard/src/` owns the HTTP server and auth; `dashboard/web/` owns the UI; - `tools/dashboard-host.py` and `tools/vms.py` own privileged host operations. -- `guest/` owns guest capture/provisioning and its shared protocol. Inspect - `guest/DESIGN.md` before changing the guest/host boundary. -- Runtime identities live in `/var/lib/studio/identities.json`, initially seeded - from `config/identities.json`. Preserve assigned numeric owners; do not - renumber services or recursively change ownership as a shortcut. - -## Adding or changing a service - -1. Read the closest existing service and the schema. Declare managed storage, - explicit external mounts, health checks, auth, and requirements. Use - `prepare` for repeatable work before startup and `setup` for repeatable - configuration after health. Declare providers instead of hardcoding their - current addresses or credentials. -2. Check side effects before staging: outbound mail, federation, downloads, - automation, fixed host ports, host networking, and exclusive GPU use. Use - preview-aware configuration or `stageIsolation = "fresh"` when a cloned app - would act as production. Fresh stages reject writable external mounts. -3. Run the relevant local checks, then stage the working files: - - ```sh - STUDIO_DEPLOY_HOST=root@10.0.0.1 STUDIO_DEPLOY_PORT=22 python3 tools/deploy.py stage - ``` - - The command prints `stage`, `release`, `host`, and `dataset`. Use that host; - do not invent a staging hostname. Repeating `stage ` updates its - existing preview and keeps preview data. Storage/isolation or incompatible - requirement changes can require destroying and recreating that preview. -4. Verify the actual app path, authentication, persistence, provider connections, - and healthy allocations. Check logs/restarts and sampled unaffected services. - An HTTP health response alone does not establish that the app works. - Writable external ZFS mounts get clones; read-only external mounts can still - reference the source. Review the actual bindings for host-networked services. -5. Clean up previews when finished using `python3 tools/deploy.py destroy - ` against the same target. Destruction deletes preview state; retain - anything the task needs first. The physical RTX 3090 fits one model instance; - stop its AI preview before a production AI deployment. -6. Commit reviewed changes. Production exports a described, conflict-free - **main**, not the working copy and not the preview's edited database. Only - after production is requested and the intended changes are on `main`: - - ```sh - STUDIO_DEPLOY_HOST=root@10.0.0.1 STUDIO_DEPLOY_PORT=22 python3 tools/deploy.py prod - ``` - - Check deployment history and real production paths. The release workflow - takes a pre-switch backup and performs health checks; use it instead of an - ad hoc container replacement or edits inside `/opt/studio/current`. - -Code rollback uses `deploy.py rollback [release-id]`. It does **not** restore -data. Data restore explicitly discards later writes and is a separate operation; -read `tools/data-restore.md` before proposing it. - -## Verification and change hygiene - -- Run checks that cover the change. Inspect a test's entry point first: some - `tools/*-test.py` scripts mutate host services or require Linux/real guests. - Run those against isolated staging fixtures. Tests that alter host services, - storage, or guests require a disposable VM or explicit production scope. -- Dashboard: `cd dashboard && npm run check && npm run build`; run Rust tests - with `cargo test --manifest-path dashboard/Cargo.toml`. Host tests include - `python3 tools/dashboard-host-test.py` and `python3 tools/dashboard-vms-test.py`. -- Deployment: `python3 tools/deploy-test.py`. On the target, the uploaded - `tools/studio.py validate` checks generated Nomad jobs; `check` checks routes. - Even render loads/allocates identities, so it is not a pure local formatter. -- Local AI: use the unit checks and live client probes in `tools/local-ai.md`. - Rehearsal has no production GPU/model instance; do not download model weights - or enable GPU inference there. -- UI changes need real browser verification, keyboard checks, and screenshots - or recordings for Clover to judge. Test VM consoles in the normal browser - panel. Metrics/traces must represent observed app telemetry. -- Prefer fewer names, fields, branches, casts, and public promises. Derive state - instead of storing it. Delete dead code and comments describing removed - behavior. One artifact has one home; link additional consumers to it. -- When a trade depends on how likely something is to change, report the trade - and ask Clover. Do not decide it by asserting that the thing is stable. +# Snow Globe + +Snow Globe is Clover's physical home computer: NixOS, Nomad/Podman services, +ZFS storage, and virtual machines. Pkl describes services; the Rust/Solid +web app manages the system. Availability and existing data are part of correctness. + +## Default workflow + +Finish a requested change through verification without another permission round: + +1. Inspect the current code, Jujutsu state, and relevant service/runbook. +2. Implement the change and run the checks that cover it. +3. Automatically deploy the working changes to staging on Snow Globe: + `python3 tools/deploy.py stage ` or `stage dashboard`. +4. Test the real preview: app behavior, authentication, persistence, dependencies, + logs/restarts, and effects on other services. For UI, use the normal browser + panel, check the keyboard, and capture screenshots or recordings. +5. Present the working preview URL and evidence. Keep it available for review. + Wait for Clover to request production promotion. + +A request to change, fix, test, stage, or commit does not authorize production. +Once Clover requests production, finish the normal deployment without asking +again: put the reviewed changes on a described, conflict-free **main**, run +`python3 tools/deploy.py prod`, and verify the actual production paths and history. +Production exports **main**; it never promotes a preview database or the working +copy. Do not move main or push merely because a local commit was requested. + +Two surfaces use the real machine during iteration: + +- **Local AI:** deploy and test changes directly in production without waiting + for a separate promotion request. The physical GPU is required; do not start a + competing preview model. Use disposable prompts, credentials, and workspaces. +- **Virtual machines:** iterate on Snow Globe using disposable guests and disks. + Test actual guest/host integration and consoles. Preserve active personal + guests; a VM task does not authorize unrelated guest or storage changes. + +Other service/dashboard changes follow staging first. Host/OS changes that cannot +be exercised in a service preview need isolated host testing; see +[host rehearsal](tools/rehearsal.md). Do not substitute an isolated test for +production verification after an authorized promotion. + +## Target and state + +The deployment CLI defaults to Snow Globe, `root@10.0.0.1`, SSH port `22`. +`SNOWGLOBE_DEPLOY_HOST` and `SNOWGLOBE_DEPLOY_PORT` override it; set both when targeting +another machine. Confirm the connected host before remote mutations. +The configured hostname is `snowglobe`, with pretty name `snow globe`. + +| Path | Purpose | +| --- | --- | +| `nixos/snowglobe.nix`, `config/site.pkl` | Host and site configuration | +| `/opt/snowglobe/releases/` | Immutable uploaded releases | +| `/opt/snowglobe/main`, `/opt/snowglobe/current` | Published main and active release links | +| `/etc/nixos` | Installed host configuration, managed by release activation | +| `/srv/prod/` | Persistent service ZFS datasets | +| `/srv/staging/` | Disposable preview data and external mount clones | +| `/srv/clover` (`/Volumes/clover` on Mac) | Live personal files | +| `/srv/clover/Media` | Separate media dataset | +| `/srv/vm` | Guest disks, access files, installation library | +| `/var/lib/snowglobe` | Identities, previews, history, backups, dashboard state, logs | + +Preserve persistent state, numeric owners, and existing consumers through +namespace changes. Read current configuration and host state before changing +operational assumptions. + +## Data and credentials + +- Infrastructure access and directory listings do not authorize reading private + contents. Get authorization per source; keep private material and derived model + output inside its store. Preview clones do not broaden that authorization. +- Keep credentials and encryption keys out of chat, command arguments, logs, and + tracked files. Use private files/stdin. Nomad variables own service secrets; + configuration declares references. Restrict private files to their consumers. +- Preserve assigned numeric owners in `/var/lib/snowglobe/identities.json`, seeded + from `config/identities.json`. Never renumber services or recursively chown data + as a shortcut. +- Before destructive storage or OS work, verify key recovery and application/ + database restores with at least two restricted copies off the server, stored + separately. Snapshots or a key archive alone do not establish recovery. +- Use the release workflow for activation and rollback, not ad hoc replacement + containers or edits inside `/opt/snowglobe/current`. Code rollback does not restore + data. Read [data restore](tools/data-restore.md) before proposing data loss. + +## Implementation and checks + +- `config/Service.pkl` owns the schema; `config/site.pkl` owns site settings. + Services are discovered from `service/*/*.pkl`. Declare routes, mounts, health, + authentication, resources, and provider requirements in the service definition. +- `tools/snowglobe.py` owns rendering/provisioning; `tools/deploy.py` uploads; + `tools/release.py` activates; `tools/data.py` backs up/restores; + `tools/router.py` derives Caddy routes. Change sources, not generated jobs/routes. +- `nixos/` owns the host. `dashboard/src/` and `dashboard/web/` own the app; + `tools/dashboard-host.py` and `tools/vms.py` own privileged operations. + Read `guest/DESIGN.md` before changing the guest/host boundary. +- Review outbound mail, federation, downloads, automation, fixed ports, host + networking, and GPU use before staging. Use preview-aware settings or fresh + isolation. Fresh previews reject writable external mounts. Read-only mounts + can still expose their source; inspect actual bindings. +- Repeating `stage ` updates its preview and retains preview data. Use + the printed hostname. Clean up superseded previews with + `python3 tools/deploy.py destroy `; retain evidence first. +- Dashboard: `cd dashboard && npm run check && npm run build` and + `cargo test --manifest-path dashboard/Cargo.toml`. Deployment: + `python3 tools/deploy-test.py`; target-side `tools/snowglobe.py validate` checks + jobs and `check` checks routes. Rendering can allocate identities. +- Read a test's entry point before running it. Host/service/storage/guest mutation + tests need disposable fixtures or the explicit production scope above. A health + response alone does not prove the app works. Metrics/traces must be observed. +- Prefer fewer moving parts. Derive state, remove dead code and false comments, + and keep one home for each artifact. When a trade depends on how likely a + default or interface is to change, explain the trade and ask Clover. - Use Jujutsu, never standalone `git`. Inspect `jj st`, the current description, - and `jj diff --git` before editing or committing; preserve unrelated work. - Split independent changes when committing. Do not push or move `main` merely - because a task requests a local commit. Use `Assisted-by: ` if adding - model attribution, never `Co-authored-by`. + and `jj diff --git`; preserve unrelated work and split independent changes. + Attribution uses `Assisted-by: `, never `Co-authored-by`. -## Operational references +## References -- Storage layout: [readme.md](readme.md). -- Physical host and recovery context: [tools/zenith-hardware.md](tools/zenith-hardware.md). -- Storage and permissions: [tools/media-cutover.md](tools/media-cutover.md), - [tools/storage-acl-cutover.md](tools/storage-acl-cutover.md). -- Legacy recovery and app imports: [tools/legacy-handoff.md](tools/legacy-handoff.md), - [tools/personal-cutover.md](tools/personal-cutover.md), and the service's - `tools/*-migration.md` runbook. -- Provider behavior: [tools/service-dependencies.md](tools/service-dependencies.md). -- Copyparty administration and folder moves: [tools/copyparty.md](tools/copyparty.md). -- Local inference: [tools/local-ai.md](tools/local-ai.md). - -Runbooks contain dated rehearsal evidence and migration history. Check current -configuration and read-only host state before treating an old observation as -today's topology. Keep deployment history and long investigations in their -runbooks; AGENTS.md carries the rules and links. +[Storage layout](readme.md) · [Physical host/recovery](tools/snowglobe-hardware.md) · +[Service dependencies](tools/service-dependencies.md) · [Files](tools/copyparty.md) · +[AI steering](service/local-ai/steering/README.md) · [AI cache](tools/local-ai-cache.md) diff --git a/config/OpenID.pkl b/config/OpenID.pkl index 9cdb9f2908fc3957c354286f142062a8cfc1e7ec..666120be172836b34f9c9ae25104eed0e0e36c4b 100644 --- a/config/OpenID.pkl +++ b/config/OpenID.pkl @@ -3,13 +3,14 @@ module OpenID import "Service.pkl" as service import "site.pkl" as site -hostname: String = "snowglobe.\(site.domain)" +const hostname: String = "snowglobe.\(site.domain)" issuer: String = "https://\(hostname)" class Client extends service.Requirement { alias: String = "oidc" fixed provider = "snowglobe" fixed kind = "client" + fixed issuerHostname = module.hostname clientId: String(isNotEmpty) name: String redirectUris: Listing diff --git a/config/Service.pkl b/config/Service.pkl index 6ab4e64926e60343647234ae20c5d1b27805e04f..7f84e850fff723d66ea6a545e3b12e6de46dc0f6 100644 --- a/config/Service.pkl +++ b/config/Service.pkl @@ -4,15 +4,6 @@ import "site.pkl" as site const nomadHostPort = "{{ if regexMatch \":\" .Address }}[{{ .Address }}]{{ else }}{{ .Address }}{{ end }}:{{ .Port }}" -/// Name shown in the management UI and service listings. -class Metadata { - name: String - tagline: String = "" - launcher: Boolean = true - /// Discovery group; absent uses the public route's auth role. - access: String? -} - /// One HTTP listener and its public route. class Http { containerPort: UInt16 @@ -107,7 +98,12 @@ class Container { id: String = read?("prop:serviceId") /// Stable numeric owner assigned by the deployment tool. uid: Int = read("prop:uid").toInt() -meta: Metadata +/// Name shown in the management UI and service listings. +name: String +tagline: String = "" +launcher: Boolean = true +/// Discovery group; absent uses the public route's auth role. +access: String? /// Allows site-specific services to stay out of deployments where they would cause side effects. enabled: Boolean = true /// Simple replaces one allocation; overlapped runs a canary alongside it. @@ -156,16 +152,16 @@ local deploymentContainers: Mapping = else throw("Declare a container or an input") -local vmStartupGrace: String? = if (read?("env:STUDIO_VM_ACCEL") == "qemu") "60m" else null +local vmStartupGrace: String? = if (read?("env:SNOWGLOBE_VM_ACCEL") == "qemu") "60m" else null output { renderer = new JsonRenderer {} value = new { id = module.id - name = module.meta.name - tagline = module.meta.tagline - launcher = module.meta.launcher - access = module.meta.access + name = module.name + tagline = module.tagline + launcher = module.launcher + access = module.access rollout = module.rollout stageIsolation = module.stageIsolation healthyDeadline = vmStartupGrace ?? module.healthyDeadline diff --git a/config/excluded-services.json b/config/excluded-services.json deleted file mode 100644 index b963165c9910b6805e86b32481f5b9882cd5e7ba..0000000000000000000000000000000000000000 --- a/config/excluded-services.json +++ /dev/null @@ -1,5 +0,0 @@ -[ - "evil-forgejo", - "evil-hedgedoc", - "keycloak" -] diff --git a/config/site.pkl b/config/site.pkl index f69227748696e0419051e11b1da376a091f61067..441da0a628e4d35567d194e6ab0ac2e914e01f07 100644 --- a/config/site.pkl +++ b/config/site.pkl @@ -1,18 +1,18 @@ module Site /// The path to the root ZFS store -root: String = read?("prop:root") ?? read?("env:STUDIO_ROOT") ?? "/srv" -pool: String = read?("prop:pool") ?? read?("env:STUDIO_POOL") ?? "studio-demo" -nodeName: String = read?("prop:nodeName") ?? read?("env:STUDIO_NODE_NAME") ?? "clover-demo" +root: String = read?("prop:root") ?? read?("env:SNOWGLOBE_ROOT") ?? "/srv" +pool: String = read?("prop:pool") ?? read?("env:SNOWGLOBE_POOL") ?? "snowglobe-demo" +nodeName: String = read?("prop:nodeName") ?? read?("env:SNOWGLOBE_NODE_NAME") ?? "snowglobe-rehearsal" /// The base domain that the infrastructure runs on -domain: String = read?("prop:domain") ?? read?("env:STUDIO_DOMAIN") ?? "studio.test" -ownerEmail: String = read?("prop:ownerEmail") ?? read?("env:STUDIO_OWNER_EMAIL") ?? "account@paperclover.net" -cloverRoot: String = read?("prop:cloverRoot") ?? read?("env:STUDIO_CLOVER_ROOT") ?? "\(root)/clover" -cloverUid: Int = (read?("prop:cloverUid") ?? read?("env:STUDIO_CLOVER_UID") ?? "3000").toInt() -cloverGid: Int = (read?("prop:cloverGid") ?? read?("env:STUDIO_CLOVER_GID") ?? "3000").toInt() -cloverReadOnly: Boolean = (read?("prop:cloverReadOnly") ?? read?("env:STUDIO_CLOVER_READ_ONLY") ?? "false") == "true" -mediaRoot: String = read?("prop:mediaRoot") ?? read?("env:STUDIO_MEDIA_ROOT") ?? "\(cloverRoot)/Media" -localAiSsdCache: String = read?("prop:localAiSsdCache") ?? read?("env:STUDIO_LOCAL_AI_SSD_CACHE") ?? "/var/cache/studio/local-ai" +domain: String = read?("prop:domain") ?? read?("env:SNOWGLOBE_DOMAIN") ?? "snowglobe.test" +ownerEmail: String = read?("prop:ownerEmail") ?? read?("env:SNOWGLOBE_OWNER_EMAIL") ?? "account@paperclover.net" +cloverRoot: String = read?("prop:cloverRoot") ?? read?("env:SNOWGLOBE_CLOVER_ROOT") ?? "\(root)/clover" +cloverUid: Int = (read?("prop:cloverUid") ?? read?("env:SNOWGLOBE_CLOVER_UID") ?? "3000").toInt() +cloverGid: Int = (read?("prop:cloverGid") ?? read?("env:SNOWGLOBE_CLOVER_GID") ?? "3000").toInt() +cloverReadOnly: Boolean = (read?("prop:cloverReadOnly") ?? read?("env:SNOWGLOBE_CLOVER_READ_ONLY") ?? "false") == "true" +mediaRoot: String = read?("prop:mediaRoot") ?? read?("env:SNOWGLOBE_MEDIA_ROOT") ?? "\(cloverRoot)/Media" +localAiSsdCache: String = read?("prop:localAiSsdCache") ?? read?("env:SNOWGLOBE_LOCAL_AI_SSD_CACHE") ?? "/var/cache/snowglobe/local-ai" jellyfinFolders: Mapping = new { ["Anime"] = "Anime" ["Movies"] = "Movies" @@ -21,6 +21,6 @@ jellyfinFolders: Mapping = new { ["Independent"] = "Videos/Independent" ["Paper Clover"] = "Videos/Paper Clover" } -mediaReadOnly: Boolean = (read?("prop:mediaReadOnly") ?? read?("env:STUDIO_MEDIA_READ_ONLY") ?? "false") == "true" -tlsInternal: Boolean = (read?("prop:tlsInternal") ?? read?("env:STUDIO_TLS_INTERNAL") ?? "false") == "true" || domain.endsWith(".test") +mediaReadOnly: Boolean = (read?("prop:mediaReadOnly") ?? read?("env:SNOWGLOBE_MEDIA_READ_ONLY") ?? "false") == "true" +tlsInternal: Boolean = (read?("prop:tlsInternal") ?? read?("env:SNOWGLOBE_TLS_INTERNAL") ?? "false") == "true" || domain.endsWith(".test") preview: Boolean = (read?("prop:preview") ?? "false") == "true" diff --git a/dashboard/dev.ts b/dashboard/dev.ts deleted file mode 100644 index 745ef565255688fc3efc45d1876f2d2ef55f4122..0000000000000000000000000000000000000000 --- a/dashboard/dev.ts +++ /dev/null @@ -1,37 +0,0 @@ -import { execFile, spawn } from "node:child_process"; -import { watch } from "node:fs"; -import { promisify } from "node:util"; - -const host = process.env.STUDIO_DEPLOY_HOST ?? "root@127.0.0.1"; -const ssh = ["-p", process.env.STUDIO_DEPLOY_PORT ?? "2222", "-o", "BatchMode=yes", "-o", "ConnectTimeout=8"]; -const dir = "/root/studio-dashboard-dev"; -const run = promisify(execFile); -const quote = (value: string) => "'" + value.replaceAll("'", "'\\''") + "'"; -const restart = `import os, shlex, subprocess -variables = dict(item.split("=", 1) for item in shlex.split(subprocess.check_output(["systemctl", "show", "-P", "Environment", "studio-dashboard"], text=True))) -for key in ["STUDIO_INDEX_POOL", "STUDIO_YT_STATE", "STUDIO_FILES_WRITABLE"]: variables.pop(key, None) -variables.update(PORT="7070", STUDIO_METRICS="follow", STUDIO_DATA_DIR="${dir}/data") -subprocess.run(["systemctl", "stop", "studio-dashboard-dev"], check=False) -subprocess.run(["systemd-run", "--unit=studio-dashboard-dev", "--collect", "--property=WorkingDirectory=${dir}/result/lib/home-dashboard", *["--setenv="+key+"="+value for key,value in variables.items()], "${dir}/result/bin/home-dashboard"], check=True)`; -let pending = false; -let rebuilding = false; -async function rebuild() { - pending = true; - if (rebuilding) return; - rebuilding = true; - try { - while (pending) { - pending = false; - await run("rsync", ["-a", "--delete", ...[".jj", ".git", "node_modules", "target", "dist", ".cache", "data", "result"].map(name => "--exclude=" + name), "-e", ["ssh", ...ssh].join(" "), "../", `${host}:${dir}/`]); - await run("ssh", [...ssh, host, `cd ${dir} && nix --extra-experimental-features 'nix-command flakes' build path:.#dashboard --cores 2 --max-jobs 1 -o result && python3 -c ${quote(restart)}`], { maxBuffer: 16 * 1024 * 1024 }); - console.log("Rust dashboard ready"); - } - } finally { rebuilding = false; } -} -await rebuild(); -watch("src", { recursive: true }, () => rebuild().catch(console.error)); -const tunnel = spawn("ssh", [...ssh, "-N", "-o", "ExitOnForwardFailure=yes", "-L", "7070:127.0.0.1:7070", host], { stdio: "inherit" }); -tunnel.on("exit", code => process.exit(code ?? 1)); -for (const signal of ["SIGINT", "SIGTERM"] as const) process.on(signal, () => { - run("ssh", [...ssh, host, "systemctl stop studio-dashboard-dev"]).finally(() => tunnel.kill()); -}); diff --git a/dashboard/package.json b/dashboard/package.json index e023c713b83c6c8f6ec552f4d0be1df5b67721c8..919d3cf51542cf6215065f50a2bbe56045f6a31f 100644 --- a/dashboard/package.json +++ b/dashboard/package.json @@ -3,7 +3,7 @@ "private": true, "type": "module", "scripts": { - "dev": "concurrently -k -n server,web \"node dev.ts\" \"vite\"", + "dev": "python3 ../tools/deploy.py stage dashboard", "build": "vite build", "check": "tsc --noEmit", "start": "cargo run --release" @@ -16,7 +16,6 @@ "@novnc/novnc": "^1.7.0", "@solidjs/router": "^1.0.0", "@types/node": "^26.6.2", - "concurrently": "^10.0.5", "esbuild": "0.28.2", "lucide-solid": "^1.48.0", "solid-js": "^1.9.15", diff --git a/dashboard/pnpm-lock.yaml b/dashboard/pnpm-lock.yaml index 0038601f2260dd83a3da50cbd6ec75dd4c04b649..13b306c9cd4b221db71af30def1dadcd6ee5ad29 100644 --- a/dashboard/pnpm-lock.yaml +++ b/dashboard/pnpm-lock.yaml @@ -24,9 +24,6 @@ importers: '@types/node': specifier: ^26.6.2 version: 26.6.2 - concurrently: - specifier: ^10.0.5 - version: 10.0.5 esbuild: specifier: 0.28.2 version: 0.28.2 @@ -555,14 +552,6 @@ packages: cpu: [x64] os: [win32] - ansi-regex@6.3.0: - resolution: {integrity: sha512-WpDfL7NO6j7tH88IDBNVdUJxDh9nmCteAVW9dsep846XdwF4naCBK+/tGLX3KJgcpgMRXCFlTM2hKGoK9FsdrQ==} - engines: {node: '>=12'} - - ansi-styles@6.2.3: - resolution: {integrity: sha512-4Dj6M28JB+oAH8kFkTLUo+a2jwOFkuqb3yucU0CANcRRUbxS0cP0nZYCGjcc3BNXwRIsUVmDGgzawme7zvJHvg==} - engines: {node: '>=12'} - babel-plugin-jsx-dom-expressions@0.40.10: resolution: {integrity: sha512-lxve6Y02YiZTldB7efKpnbf1BH00XCFZNYYW235jSGsYaJNFtHrYlKV6/O+miHbjqpIr9FTe5+0no4hofAMbfA==} peerDependencies: @@ -590,19 +579,6 @@ packages: caniuse-lite@1.0.30001812: resolution: {integrity: sha512-qN+QNNBr93TCmFrmte0bBCjSDMuRvt78VlHT99qIGPszm4QsqCX8lnyWUFkHi8B7ZBAPq8SH+UqyXNy/odMdng==} - chalk@5.6.2: - resolution: {integrity: sha512-7NzBL0rN6fMUW+f7A6Io4h40qQlG+xGmtMxfbnH/K7TAtt8JQWVQK+6g0UXKMeVJoyV5EkkNsErQ8pVD3bLHbA==} - engines: {node: ^12.17.0 || ^14.13 || >=16.0.0} - - cliui@9.0.1: - resolution: {integrity: sha512-k7ndgKhwoQveBL+/1tqGJYNz097I7WOvwbmmU2AR5+magtbjPWQTS1C5vzGkBC8Ym8UWRzfKUzUUqFLypY4Q+w==} - engines: {node: '>=20'} - - concurrently@10.0.5: - resolution: {integrity: sha512-JaP/CoftUrCcAFW/g//RbgEGwlelnEae6cfBLgH6ZdO6s8jPkn6p9SB9u6pdVxYXoiSnFqseOlHfrEfF82TVOg==} - engines: {node: '>=22'} - hasBin: true - convert-source-map@2.0.0: resolution: {integrity: sha512-Kvp459HrV2FEJ1CAsi1Ku+MY3kasH19TFykTz2xWmMeq6bk2NU3XXvfJ+Q61m0xktWwt+1HSYf3JZsTms3aRJg==} @@ -625,9 +601,6 @@ packages: electron-to-chromium@1.5.439: resolution: {integrity: sha512-qu6QIPXhsb+CRcAiTMNjR4A1y/7tCYKkKjr5CZXRVih6qkDf79peZ2BpEU3qoDBNCRrcy3Mra3X9nG5oruuA7Q==} - emoji-regex@10.6.0: - resolution: {integrity: sha512-toUI84YS5YmxW219erniWD0CIVOo46xGKColeNQRgOzDorgBi1v4D71/OFzgD9GO2UGKIv1C3Sp8DAn0+j5w7A==} - entities@6.0.1: resolution: {integrity: sha512-aN97NXWF6AWBTahfVOIrB/NShkzi5H7F9r1s9mD3cDj4Ko5f2qhhVoYMibXF7GlLveb/D2ioWay8lxI97Ven3g==} engines: {node: '>=0.12'} @@ -659,14 +632,6 @@ packages: resolution: {integrity: sha512-3hN7NaskYvMDLQY55gnW3NQ+mesEAepTqlg+VEbj7zzqEMBVNhzcGYYeqFo/TlYz6eQiFcp1HcsCZO+nGgS8zg==} engines: {node: '>=6.9.0'} - get-caller-file@2.0.5: - resolution: {integrity: sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg==} - engines: {node: 6.* || 8.* || >= 10.*} - - get-east-asian-width@1.7.0: - resolution: {integrity: sha512-XjH1AECxf0giL2V1aU8vKyRR2ppRUb5c0EvT7zuJTokQ74bNo52zOtghqdWIqrhUD79fo3x0WfKZdOqxF6LG1Q==} - engines: {node: '>=18'} - hono@4.13.9: resolution: {integrity: sha512-7dMkQmZoC4E6F7AtaQSPhlWAdnBti+j7rreMZl8QB4jFiEhP9TWbGWUMi8WYzBCgmgulxuvLQupKqo+Co6Omyg==} engines: {node: '>=16.9.0'} @@ -804,9 +769,6 @@ packages: engines: {node: ^20.19.0 || >=22.12.0} hasBin: true - rxjs@7.8.2: - resolution: {integrity: sha512-dhKf903U/PQZY6boNNtAGdWbG85WAbjT/1xYoZIC7FAY0yWapOBQVsVrDl58W86//e1VpMNBtRV4MaXfdMySFA==} - semver@6.3.1: resolution: {integrity: sha512-BR7VvDCVHO+q2xBEWskxS6DJE1qRnb7DxzUrogb71CWoSficBxYsiAGd+Kl0mmq/MprG9yArRkyrQxTO6XjMzA==} hasBin: true @@ -821,10 +783,6 @@ packages: resolution: {integrity: sha512-rVQVWjjSvlINzaQPZH5JFqsqEsIWdTxY3iJZCnTL/5gQbXIRooVZKI60tVCkOVfzcRPejboxO2t0P89dg5mQaA==} engines: {node: '>=10'} - shell-quote@1.9.0: - resolution: {integrity: sha512-Iov+JwFv/2HcTpcwNMKd8+IWNb8tboQJNQTkAY/LLVK7gGH9jy+LGkVqPxfekHl+yMmiqXszdGWXgkfml7hjqA==} - engines: {node: '>= 0.4'} - solid-js@1.9.15: resolution: {integrity: sha512-EeiY2xfpZJqPLjXspVEKjAII4yv8NyG//NxZ3IpOFHdUNnnTyL0uJOeS9LWGvA7cFCz5y94cjFwYlmw5Luncsg==} @@ -837,29 +795,10 @@ packages: resolution: {integrity: sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==} engines: {node: '>=0.10.0'} - string-width@7.2.0: - resolution: {integrity: sha512-tsaTIkKW9b4N+AEj+SVA+WhJzV7/zMhcSu78mLKWSk7cXMOSHsBKFWUs0fWwq8QyK3MgJBQRX6Gbi4kYbdvGkQ==} - engines: {node: '>=18'} - - strip-ansi@7.2.0: - resolution: {integrity: sha512-yDPMNjp4WyfYBkHnjIRLfca1i6KMyGCtsVgoKe/z1+6vukgaENdgGBZt+ZmKPc4gavvEZ5OgHfHdrazhgNyG7w==} - engines: {node: '>=12'} - - supports-color@10.2.2: - resolution: {integrity: sha512-SS+jx45GF1QjgEXQx4NJZV9ImqmO2NPz5FNsIHrsDjh2YsHnawpan7SNQ1o8NuhrbHZy9AZhIoCUiCeaW/C80g==} - engines: {node: '>=18'} - tinyglobby@0.2.17: resolution: {integrity: sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g==} engines: {node: '>=12.0.0'} - tree-kill@1.2.2: - resolution: {integrity: sha512-L0Orpi8qGpRG//Nd+H90vFB+3iHnue1zSSGmNOOCh1GLJ7rUKVwV2HvijphGQS2UmhUZewS9VgvxYIdgr+fG1A==} - hasBin: true - - tslib@2.8.1: - resolution: {integrity: sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==} - tsx@4.23.15: resolution: {integrity: sha512-Yiex1Ovn8z2xPpOWckIiysV1SSyRMY9BkLF++q0yKiDxCqRhosKfMg3janKkiLBwZ5c/YryloKwGZcrEmtwxKw==} engines: {node: '>=18.0.0'} @@ -943,25 +882,9 @@ packages: vite: optional: true - wrap-ansi@9.0.2: - resolution: {integrity: sha512-42AtmgqjV+X1VpdOfyTGOYRi0/zsoLqtXQckTmqTeybT+BDIbM/Guxo7x3pE2vtpr1ok6xRqM9OpBe+Jyoqyww==} - engines: {node: '>=18'} - - y18n@5.0.8: - resolution: {integrity: sha512-0pfFzegeDWJHJIAmTLRP2DwHjdF5s7jo9tuztdQxAhINCdvS+3nGINqPd00AphqJR/0LhANUS6/+7SCb98YOfA==} - engines: {node: '>=10'} - yallist@3.1.1: resolution: {integrity: sha512-a4UGQaWPH59mOXUYnAG2ewncQS4i4F43Tv3JoAM+s2VDAmS9NsK8GpDMLrCHPksFT7h3K6TOoUNn2pb7RoXx4g==} - yargs-parser@22.0.0: - resolution: {integrity: sha512-rwu/ClNdSMpkSrUb+d6BRsSkLUq1fmfsY6TOpYzTwvwkg1/NRG85KBy3kq++A8LKQwX6lsu+aWad+2khvuXrqw==} - engines: {node: ^20.19.0 || ^22.12.0 || >=23} - - yargs@18.0.0: - resolution: {integrity: sha512-4UEqdc2RYGHZc7Doyqkrqiln3p9X2DZVxaGbwhn2pi7MrRagKaOcIKe8L3OxYcbhXLgLFUS3zAYuQjKBQgmuNg==} - engines: {node: ^20.19.0 || ^22.12.0 || >=23} - snapshots: '@babel/code-frame@7.29.7': @@ -1316,10 +1239,6 @@ snapshots: '@typescript/typescript-win32-x64@7.0.2': optional: true - ansi-regex@6.3.0: {} - - ansi-styles@6.2.3: {} - babel-plugin-jsx-dom-expressions@0.40.10(@babel/core@7.29.7): dependencies: '@babel/core': 7.29.7 @@ -1348,23 +1267,6 @@ snapshots: caniuse-lite@1.0.30001812: {} - chalk@5.6.2: {} - - cliui@9.0.1: - dependencies: - string-width: 7.2.0 - strip-ansi: 7.2.0 - wrap-ansi: 9.0.2 - - concurrently@10.0.5: - dependencies: - chalk: 5.6.2 - rxjs: 7.8.2 - shell-quote: 1.9.0 - supports-color: 10.2.2 - tree-kill: 1.2.2 - yargs: 18.0.0 - convert-source-map@2.0.0: {} csstype@3.2.3: {} @@ -1377,8 +1279,6 @@ snapshots: electron-to-chromium@1.5.439: {} - emoji-regex@10.6.0: {} - entities@6.0.1: {} esbuild@0.28.2: @@ -1421,10 +1321,6 @@ snapshots: gensync@1.0.0-beta.2: {} - get-caller-file@2.0.5: {} - - get-east-asian-width@1.7.0: {} - hono@4.13.9: {} html-entities@2.3.3: {} @@ -1539,10 +1435,6 @@ snapshots: '@rolldown/binding-win32-arm64-msvc': 1.2.11 '@rolldown/binding-win32-x64-msvc': 1.2.11 - rxjs@7.8.2: - dependencies: - tslib: 2.8.1 - semver@6.3.1: {} seroval-plugins@1.5.6(seroval@1.5.6): @@ -1551,8 +1443,6 @@ snapshots: seroval@1.5.6: {} - shell-quote@1.9.0: {} - solid-js@1.9.15: dependencies: csstype: 3.2.3 @@ -1570,27 +1460,11 @@ snapshots: source-map-js@1.2.1: {} - string-width@7.2.0: - dependencies: - emoji-regex: 10.6.0 - get-east-asian-width: 1.7.0 - strip-ansi: 7.2.0 - - strip-ansi@7.2.0: - dependencies: - ansi-regex: 6.3.0 - - supports-color@10.2.2: {} - tinyglobby@0.2.17: dependencies: fdir: 6.5.0(picomatch@4.0.7) picomatch: 4.0.7 - tree-kill@1.2.2: {} - - tslib@2.8.1: {} - tsx@4.23.15: dependencies: esbuild: 0.28.2 @@ -1661,23 +1535,4 @@ snapshots: optionalDependencies: vite: 8.3.1(@types/node@26.6.2)(esbuild@0.28.2)(tsx@4.23.15) - wrap-ansi@9.0.2: - dependencies: - ansi-styles: 6.2.3 - string-width: 7.2.0 - strip-ansi: 7.2.0 - - y18n@5.0.8: {} - yallist@3.1.1: {} - - yargs-parser@22.0.0: {} - - yargs@18.0.0: - dependencies: - cliui: 9.0.1 - escalade: 3.2.0 - get-caller-file: 2.0.5 - string-width: 7.2.0 - y18n: 5.0.8 - yargs-parser: 22.0.0 diff --git a/dashboard/server/youtube-worker.py b/dashboard/server/youtube-worker.py index edaf45de929efa026d12359f2b69c152cc36d2f7..bcaf43955608213a85c939ff2e5c723d0649ce90 100644 --- a/dashboard/server/youtube-worker.py +++ b/dashboard/server/youtube-worker.py @@ -19,20 +19,20 @@ import yaml os.umask(0o007) -YT_CONFIG_DIR = os.environ.get("STUDIO_YT_CONFIG", "/srv/clover/Documents/Config/Youtube Downloader") +YT_CONFIG_DIR = os.environ.get("SNOWGLOBE_YT_CONFIG", "/srv/clover/Documents/Config/Youtube Downloader") FEED_CONFIG = os.path.join(YT_CONFIG_DIR, "feed.yaml") -STATE_DIR = os.environ.get("STUDIO_YT_STATE", "/srv/prod/ytdl/data") +STATE_DIR = os.environ.get("SNOWGLOBE_YT_STATE", "/srv/prod/ytdl/data") INTERVAL = int(os.environ.get("CHECK_INTERVAL", "1800")) SMTP_HOST = os.environ.get("SMTP_HOST", "") SMTP_PORT = int(os.environ.get("SMTP_PORT", "465")) SMTP_USER = os.environ.get("SMTP_USER", "") SMTP_PASS = os.environ.get("SMTP_PASS", "") -MAIL_FROM = os.environ.get("MAIL_FROM", f"yt-feed@{os.environ.get('STUDIO_DOMAIN', 'studio.test')}") -MAIL_TO = os.environ.get("MAIL_TO", os.environ.get("STUDIO_OWNER_EMAIL", "account@paperclover.net")) -BASE_URL = os.environ.get("BASE_URL", f"https://snowglobe.{os.environ.get('STUDIO_DOMAIN', 'studio.test')}/youtube").rstrip("/") -READ_ONLY = os.environ.get("STUDIO_MEDIA_READ_ONLY") == "true" +MAIL_FROM = os.environ.get("MAIL_FROM", f"yt-feed@{os.environ.get('SNOWGLOBE_DOMAIN', 'snowglobe.test')}") +MAIL_TO = os.environ.get("MAIL_TO", os.environ.get("SNOWGLOBE_OWNER_EMAIL", "account@paperclover.net")) +BASE_URL = os.environ.get("BASE_URL", f"https://snowglobe.{os.environ.get('SNOWGLOBE_DOMAIN', 'snowglobe.test')}/youtube").rstrip("/") +READ_ONLY = os.environ.get("SNOWGLOBE_MEDIA_READ_ONLY") == "true" -MEDIA_ROOT = os.environ.get("STUDIO_YT_MEDIA", "/srv/clover/Media") +MEDIA_ROOT = os.environ.get("SNOWGLOBE_YT_MEDIA", "/srv/clover/Media") INDIE_DIR = os.path.join(MEDIA_ROOT, "Indie Shows") INDEP_DIR = os.path.join(MEDIA_ROOT, "Videos/Independent") MUSIC_DIR = os.path.join(MEDIA_ROOT, "Intake - Music") diff --git a/dashboard/src/ai.rs b/dashboard/src/ai.rs index 2cc91690be394497b30902e47885a065a6e4ac58..a6954300d5661f0d1230bd100d2ce88920567ef7 100644 --- a/dashboard/src/ai.rs +++ b/dashboard/src/ai.rs @@ -29,7 +29,7 @@ pub async fn route( { return Err(Error::new( 403, - "Open Local AI in Snowglobe to copy its key.", + "Open Local AI in Snow Globe to copy its key.", )); } let jobs = core::nomad(&app, "/v1/jobs").await?; @@ -45,7 +45,7 @@ pub async fn route( ) })?; let job = core::nomad(&app, &format!("/v1/job/{}", encoded(id))).await?; - let hostname = string(&job["Meta"]["studio_hostname"]); + let hostname = string(&job["Meta"]["snowglobe_hostname"]); if hostname.is_empty() || !hostname .bytes() diff --git a/dashboard/src/apps.rs b/dashboard/src/apps.rs index fda98af268eb69c874cf1ca90080f8157546dcf3..a9d0fa10154a357f3abfb8b6f31e536262674f31 100644 --- a/dashboard/src/apps.rs +++ b/dashboard/src/apps.rs @@ -1,7 +1,7 @@ use crate::*; pub fn file_link(path: &str, zip: bool) -> Value { - let root = env("STUDIO_STORE_ROOT", "/srv"); + let root = env("SNOWGLOBE_STORE_ROOT", "/srv"); let Some(relative) = std::path::Path::new(path).strip_prefix(&root).ok() else { return Value::Null; }; @@ -10,7 +10,7 @@ pub fn file_link(path: &str, zip: bool) -> Value { .map(|c| encoded(&c.as_os_str().to_string_lossy())) .collect::>() .join("/"); - let Ok(files) = std::env::var("STUDIO_FILES_URL") else { + let Ok(files) = std::env::var("SNOWGLOBE_FILES_URL") else { return Value::Null; }; json!(format!("{files}/{path}{}", if zip { "?zip" } else { "" })) @@ -126,7 +126,7 @@ fn validate_vm(mut spec: Value) -> Result { } async fn paper(app: Arc, path: &str, body: Option) -> Result { let jobs = core::scan(app.clone()).await?; - let host = jobs.value["clover-source-of-truth"]["job"]["Meta"]["studio_hostname"] + let host = jobs.value["clover-source-of-truth"]["job"]["Meta"]["snowglobe_hostname"] .as_str() .ok_or_else(|| Error::new(502, "Paper Clover is unavailable."))?; let key = host::call( @@ -141,7 +141,7 @@ async fn paper(app: Arc, path: &str, body: Option) -> Result if path == "scan" { path.into() } else { - format!("studio/{path}") + format!("{}/{path}", namespace::PREVIOUS.as_str()) } ); let request = if let Some(body) = body { @@ -422,7 +422,7 @@ pub async fn route( } ["paper-clover"] if method == Method::GET => { let mut value = paper(app, "stats", None).await?; - value["browse"] = std::env::var("STUDIO_PUBLISHED_ROOT") + value["browse"] = std::env::var("SNOWGLOBE_PUBLISHED_ROOT") .ok() .map(|p| file_link(&p, false)) .unwrap_or(Value::Null); diff --git a/dashboard/src/auth.rs b/dashboard/src/auth.rs index 5f853215fdb6a896fe5ce255b9ec4f492fad20da..938817733489d4269d8891e51e6a3b6ba50cf9f4 100644 --- a/dashboard/src/auth.rs +++ b/dashboard/src/auth.rs @@ -170,7 +170,7 @@ impl Store { { return Err(Error::new( 500, - "Set separate HTTPS origins for Snowglobe and Files.", + "Set separate HTTPS origins for Snow Globe and Files.", )); } let rp_origin = url::Url::parse(&format!("https://{rp}"))?; @@ -419,7 +419,7 @@ impl Store { } db.execute("DELETE FROM sessions WHERE expires<=?", [now() as i64])?; let ip = headers - .get("X-Studio-Client-IP") + .get("X-Snowglobe-Client-IP") .and_then(|v| v.to_str().ok()) .unwrap_or("unknown"); db.execute( @@ -439,7 +439,7 @@ impl Store { } pub(crate) fn limit(&self, headers: &HeaderMap, name: &str) -> Result<()> { let ip = headers - .get("X-Studio-Client-IP") + .get("X-Snowglobe-Client-IP") .and_then(|v| v.to_str().ok()) .unwrap_or("unknown"); let db = self.db.lock().unwrap(); @@ -479,7 +479,7 @@ impl Store { if headers.get("origin").and_then(|v| v.to_str().ok()) != Some(self.origin.origin().ascii_serialization().as_str()) { - return Err(Error::new(403, "Open sign-in on Snowglobe and try again.")); + return Err(Error::new(403, "Open sign-in on Snow Globe and try again.")); } let cookie = cookie(headers, CSRF_COOKIE).unwrap_or_default(); if cookie.is_empty() diff --git a/dashboard/src/core.rs b/dashboard/src/core.rs index aaa08ef173da16edf8761b9b5f142de3baae01ac..827bab13abf52929ba57fcc406dbbd697ec723a4 100644 --- a/dashboard/src/core.rs +++ b/dashboard/src/core.rs @@ -5,16 +5,18 @@ use std::path::Path; pub async fn nomad(app: &App, path: &str) -> Result { let value = nomad_raw(app, path).await?; - Ok(value + let mut value = value .map(|bytes| serde_json::from_slice(&bytes)) .transpose()? - .unwrap_or(Value::Null)) + .unwrap_or(Value::Null); + namespace::nomad(&mut value); + Ok(value) } pub async fn nomad_raw(app: &App, path: &str) -> Result> { let request = async { let token = tokio::fs::read_to_string(env( - "STUDIO_NOMAD_TOKEN_FILE", - "/var/lib/studio/dashboard.token", + "SNOWGLOBE_NOMAD_TOKEN_FILE", + "/var/lib/snowglobe/dashboard.token", )) .await .map_err(|e| Error::new(501, format!("Nomad isn't connected: {e}")))?; @@ -222,25 +224,12 @@ pub async fn service_file(app: &App, id: &str) -> Result { if !valid_id(id) { return Err(Error::new(400, "Invalid service ID.")); } - let excluded = excluded_services(&app.repo).await?; - if excluded.iter().any(|service| service == id) { - return Err(Error::new( - 404, - format!("No service is named {id}. Pick one from the sidebar."), - )); - } let direct = app.repo.join("service").join(id).join("service.pkl"); if tokio::fs::try_exists(&direct).await? { return Ok(direct); } let mut dirs = tokio::fs::read_dir(app.repo.join("service")).await?; while let Some(dir) = dirs.next_entry().await? { - if excluded - .iter() - .any(|service| dir.file_name() == service.as_str()) - { - continue; - } let file = dir.path().join(format!("{id}.pkl")); if tokio::fs::try_exists(&file).await? { return Ok(file); @@ -264,7 +253,7 @@ async fn icons(app: Arc, id: String) -> Result> { let mut found = serde_json::Map::new(); for file in ["icon.svg","icon-light.svg","icon-dark.svg"] { let candidate = dir.join(file); - let fallback = PathBuf::from(env("STUDIO_ICON_DIR","server/icons")).join(&id).join(file); + let fallback = PathBuf::from(env("SNOWGLOBE_ICON_DIR","server/icons")).join(&id).join(file); let path = if tokio::fs::try_exists(&candidate).await? { candidate } else { fallback }; if let Ok(bytes) = tokio::fs::read(&path).await { found.insert(file.to_owned(),json!({"path":path,"hash":format!("{:x}",Sha1::digest(&bytes))[..12].to_owned()})); @@ -350,7 +339,7 @@ async fn summarize(app: Arc, id: &str, found: &Value) -> Value { } }) .sum(); - json!({"id":id,"name":meta["studio_name"].as_str().unwrap_or(id),"health":found["health"].as_str().unwrap_or("down"),"url":meta["studio_hostname"].as_str().filter(|h| !h.is_empty() && meta["studio_launcher"] != "false").map(|h| format!("https://{h}")),"icon":icon_of(app.clone(),id).await,"access":meta["studio_auth_role"],"tagline":meta["studio_tagline"],"cpu":usage["cpu"],"cpuLimit":if mhz > 0.0 { cpu/mhz } else {0.0},"memory":usage["memory"],"memoryLimit":memory*1048576.0}) + json!({"id":id,"name":meta["snowglobe_name"].as_str().unwrap_or(id),"health":found["health"].as_str().unwrap_or("down"),"url":meta["snowglobe_hostname"].as_str().filter(|h| !h.is_empty() && meta["snowglobe_launcher"] != "false").map(|h| format!("https://{h}")),"icon":icon_of(app.clone(),id).await,"access":meta["snowglobe_auth_role"],"tagline":meta["snowglobe_tagline"],"cpu":usage["cpu"],"cpuLimit":if mhz > 0.0 { cpu/mhz } else {0.0},"memory":usage["memory"],"memoryLimit":memory*1048576.0}) } async fn summaries(app: Arc) -> Result> { let state = app.clone(); @@ -444,19 +433,11 @@ async fn issues(app: Arc) -> Result { Ok(json!(issues)) } -async fn excluded_services(repo: &Path) -> Result> { - Ok(serde_json::from_slice( - &tokio::fs::read(repo.join("config/excluded-services.json")).await?, - )?) -} - async fn launcher_module(repo: &Path) -> Result { - let excluded = excluded_services(repo).await?; let mut files = Vec::new(); let mut directories = tokio::fs::read_dir(repo.join("service")).await?; while let Some(directory) = directories.next_entry().await? { - let directory_name = directory.file_name().to_string_lossy().into_owned(); - if excluded.contains(&directory_name) || !directory.file_type().await?.is_dir() { + if !directory.file_type().await?.is_dir() { continue; } let mut entries = tokio::fs::read_dir(directory.path()).await?; @@ -465,14 +446,7 @@ async fn launcher_module(repo: &Path) -> Result { if !path.extension().is_some_and(|extension| extension == "pkl") { continue; } - let name = if entry.file_name() == "service.pkl" { - directory_name.clone() - } else { - path.file_stem().unwrap().to_string_lossy().into_owned() - }; - if !excluded.contains(&name) { - files.push(path); - } + files.push(path); } } files.sort(); @@ -485,11 +459,11 @@ async fn launcher_module(repo: &Path) -> Result { } module.push_str(&format!("local services = Map({})\n", services.join(", "))); module.push_str(r#"output { renderer = new JsonRenderer { omitNullProperties = false } -value = services.filter((_, s) -> s.enabled && s.meta.launcher).mapValues((_, s) -> +value = services.filter((_, s) -> s.enabled && s.launcher).mapValues((_, s) -> let (tasks = s.containers?.toMap() ?? Map("app", s.container)) let (routes = tasks.filter((_, task) -> task?.http?.hostname != null).mapValues((_, task) -> task.http)) let (route = routes.values.firstOrNull) -new Dynamic { name = s.meta.name; tagline = s.meta.tagline; hostname = route?.hostname; access = s.meta.access ?? route?.authRole; urls = routes.mapValues((_, http) -> "https://\(http.hostname)") }) }"#); +new Dynamic { name = s.name; tagline = s.tagline; hostname = route?.hostname; access = s.access ?? route?.authRole; urls = routes.mapValues((_, http) -> "https://\(http.hostname)") }) }"#); Ok(module) } @@ -534,7 +508,7 @@ async fn service(app: Arc, id: &str) -> Result> { .map(|s| json!({"id":other,"name":s["name"],"kind":kind,"health":s["health"]})) }; let needs = |other: &str| { - serde_json::from_str::(string(&jobs.value[other]["job"]["Meta"]["studio_requires"])) + serde_json::from_str::(string(&jobs.value[other]["job"]["Meta"]["snowglobe_requires"])) .ok() }; let requirements = needs(id).map(|n| { @@ -557,7 +531,7 @@ async fn service(app: Arc, id: &str) -> Result> { }; let mut summary = summarize(app.clone(), id, found).await; let meta = &found["job"]["Meta"]; - let application_traces = if let Some(name) = meta["studio_trace_service"] + let application_traces = if let Some(name) = meta["snowglobe_trace_service"] .as_str() .filter(|s| !s.is_empty()) { @@ -568,7 +542,7 @@ async fn service(app: Arc, id: &str) -> Result> { false }; let metrics = - if array(&found["job"]["TaskGroups"]).iter().flat_map(|g| array(&g["Services"])).any(|s| array(&s["Tags"]).iter().any(|tag| string(tag).starts_with("studio-metrics-path="))) || meta["studio_metrics_pushed"] == "true" { + if array(&found["job"]["TaskGroups"]).iter().flat_map(|g| array(&g["Services"])).any(|s| array(&s["Tags"]).iter().any(|tag| string(tag).starts_with("snowglobe-metrics-path="))) || meta["snowglobe_metrics_pushed"] == "true" { telemetry::metric_names(app.clone(), id) .await .unwrap_or(json!([])) @@ -576,7 +550,7 @@ async fn service(app: Arc, id: &str) -> Result> { json!([]) }; let release = host::call(json!({"operation":"deploy.current"})).await?; - let secrets = serde_json::from_str::(string(&meta["studio_secrets"])) + let secrets = serde_json::from_str::(string(&meta["snowglobe_secrets"])) .ok() .map(|v| { array(&v) @@ -851,34 +825,23 @@ async fn definition(app: Arc, id: &str) -> Result { mod tests { use super::*; #[tokio::test] - async fn launcher_excludes_directories_and_grouped_definitions_before_import() { - let root = std::env::temp_dir().join(format!("studio-launcher-{}", uuid::Uuid::new_v4())); - tokio::fs::create_dir_all(root.join("config")) + async fn launcher_discovers_direct_and_grouped_definitions() { + let root = std::env::temp_dir().join(format!("snowglobe-launcher-{}", uuid::Uuid::new_v4())); + tokio::fs::create_dir_all(root.join("service/personal")) .await .unwrap(); - for directory in ["retired", "personal"] { - tokio::fs::create_dir_all(root.join("service").join(directory)) - .await - .unwrap(); - } - tokio::fs::write(root.join("config/excluded-services.json"), r#"["retired"]"#) + tokio::fs::write(root.join("service/README.md"), "not a service") .await .unwrap(); - for file in [ - "retired/service.pkl", - "personal/retired.pkl", - "personal/service.pkl", - "personal/fixture.pkl", - ] { - tokio::fs::write(root.join("service").join(file), "unread fixture") + for file in ["service.pkl", "fixture.pkl", "icon.svg"] { + tokio::fs::write(root.join("service/personal").join(file), "unread fixture") .await .unwrap(); } let module = launcher_module(&root).await.unwrap(); - assert!(!module.contains("retired")); - assert!(!module.contains("import*")); assert!(module.contains("/personal/service.pkl")); assert!(module.contains("/personal/fixture.pkl")); + assert!(!module.contains("icon.svg")); assert_eq!( module .lines() diff --git a/dashboard/src/deploys.rs b/dashboard/src/deploys.rs index 7dc51da14bd02472132459a3c2612030b221b042..bdc1dfcc42de5207cf117957f1368d48ab1ef6f6 100644 --- a/dashboard/src/deploys.rs +++ b/dashboard/src/deploys.rs @@ -15,7 +15,7 @@ async fn stages(app: Arc) -> Result { for stage in values.as_array_mut().unwrap() { let job = jobs.as_ref().map(|jobs| &jobs.value[string(&stage["id"])]); stage["hostname"] = json!( - job.and_then(|job| job["job"]["Meta"]["studio_hostname"].as_str()) + job.and_then(|job| job["job"]["Meta"]["snowglobe_hostname"].as_str()) .filter(|hostname| !hostname.is_empty()) ); stage["health"] = json!(job.map(|job| job["health"].as_str().unwrap_or("down"))); diff --git a/dashboard/src/files.rs b/dashboard/src/files.rs index 8401bc201fc76a7c99532e7decaa8a567893d48a..643ed1f154ac6a83b99071bcde9974c7c99e8ba8 100644 --- a/dashboard/src/files.rs +++ b/dashboard/src/files.rs @@ -52,7 +52,7 @@ struct Explorer { } impl Explorer { async fn new(root: PathBuf) -> Result { - let store = PathBuf::from(env("STUDIO_STORE_ROOT", "/srv")); + let store = PathBuf::from(env("SNOWGLOBE_STORE_ROOT", "/srv")); let scope = root .strip_prefix(&store) .map_err(|_| Error::new(500, "Library root is outside the store."))? @@ -577,7 +577,7 @@ async fn prune(ops: &Value) -> Result<()> { let mounts = mounts().await?; let datasets: HashSet<_> = array(&mounts) .iter() - .filter(|m| Path::new(string(&m["target"])).starts_with(env("STUDIO_STORE_ROOT", "/srv"))) + .filter(|m| Path::new(string(&m["target"])).starts_with(env("SNOWGLOBE_STORE_ROOT", "/srv"))) .map(|m| string(&m["source"])) .collect(); if datasets.is_empty() { @@ -605,11 +605,11 @@ pub async fn route( ) -> Result { let root = if media { env( - "STUDIO_MEDIA_ROOT", - &format!("{}/clover/Media", env("STUDIO_STORE_ROOT", "/srv")), + "SNOWGLOBE_MEDIA_ROOT", + &format!("{}/clover/Media", env("SNOWGLOBE_STORE_ROOT", "/srv")), ) } else { - env("STUDIO_STORE_ROOT", "/srv") + env("SNOWGLOBE_STORE_ROOT", "/srv") }; let explorer = Explorer::new(root.into()).await?; if parts == ["refresh"] && media && method == Method::POST { @@ -719,7 +719,7 @@ async fn change( parts: &[&str], body: Value, ) -> Result { - if env("STUDIO_FILES_WRITABLE", "") != "1" { + if env("SNOWGLOBE_FILES_WRITABLE", "") != "1" { return Err(Error::new( 501, "The file browser can't change files yet. Use copyparty instead.", @@ -1064,7 +1064,7 @@ async fn revert(explorer: &Explorer, op: &Value, mounts: &Value) -> Result<()> { Ok(()) } async fn refresh(app: Arc) -> Result { - let base = std::env::var("STUDIO_JELLYFIN_URL") + let base = std::env::var("SNOWGLOBE_JELLYFIN_URL") .map_err(|_| Error::new(501, "Jellyfin isn't connected to the dashboard."))?; let password = host::call( json!({"operation":"deploy.secret.get","service":"jellyfin","key":"admin_password"}), @@ -1078,7 +1078,7 @@ async fn refresh(app: Arc) -> Result { .post(format!("{base}/Users/AuthenticateByName")) .header( "Authorization", - "MediaBrowser Client=\"home server\", Device=\"server\", DeviceId=\"studio\", Version=\"1\"", + "MediaBrowser Client=\"home server\", Device=\"server\", DeviceId=\"snowglobe\", Version=\"1\"", ) .json(&json!({"Username":"snow","Pw":password})) .send() diff --git a/dashboard/src/guest.rs b/dashboard/src/guest.rs index a48fd86b970bfa86fce9fefba89f5155fe1bb61f..385973c67b2581f466cf7fdc4b650e99bf6b78f1 100644 --- a/dashboard/src/guest.rs +++ b/dashboard/src/guest.rs @@ -92,7 +92,7 @@ pub fn providers(auth: &auth::Store, next: &str) -> Result { fn registration(auth: &auth::Store, provider: &str) -> Result<(String, String)> { auth.db.lock().unwrap().query_row("SELECT client_id,secret FROM guest_providers WHERE id=?", [provider], |r|Ok((r.get(0)?,r.get(1)?))) - .optional()?.ok_or_else(|| Error::new(503, "This sign-in provider isn't available. Use your Snowglobe account or try again later.")) + .optional()?.ok_or_else(|| Error::new(503, "This sign-in provider isn't available. Use your Snow Globe account or try again later.")) } async fn response_bytes(mut response: reqwest::Response) -> Result> { @@ -528,7 +528,7 @@ async fn handle(app: &App, request: Request) -> Result { if !current.is_null() && !is_guest(¤t) { return Err(Error::new( 403, - "You're signed into Snowglobe. Open Shale to use your account.", + "You're signed into Snow Globe. Open Shale to use your account.", )); } let callback = format!("{}auth/guest/callback/{provider}", auth.origin); @@ -616,7 +616,7 @@ async fn handle(app: &App, request: Request) -> Result { let http = reqwest::Client::builder() .timeout(Duration::from_secs(10)) .redirect(reqwest::redirect::Policy::none()) - .user_agent("Snowglobe guest sign-in") + .user_agent("Snow Globe guest sign-in") .default_headers({ let mut headers = HeaderMap::new(); headers.insert("accept", "application/json".parse().unwrap()); diff --git a/dashboard/src/host.rs b/dashboard/src/host.rs index 1e4b976c16dbd276eecc6766140114ddfca98564..614bba347bdba754ca7729abaa75488d6b41f95c 100644 --- a/dashboard/src/host.rs +++ b/dashboard/src/host.rs @@ -41,8 +41,8 @@ pub async fn open(request: Value) -> Result<(Value, tokio::net::UnixStream)> { }; tokio::time::timeout(Duration::from_secs(timeout), async { let mut socket = tokio::net::UnixStream::connect(env( - "STUDIO_HOST_SOCKET", - "/run/studio-host/host.sock", + "SNOWGLOBE_HOST_SOCKET", + "/run/snowglobe-host/host.sock", )) .await?; #[cfg(target_os = "linux")] diff --git a/dashboard/src/index.rs b/dashboard/src/index.rs index d0d6b35602ee1c80a7e807f9158ed474de52a964..15182f2ed1dd1d06c9aec1680b9427a042413a69 100644 --- a/dashboard/src/index.rs +++ b/dashboard/src/index.rs @@ -820,7 +820,7 @@ pub async fn route(app: Arc, kind: &str, query: &HashMap) - })? .clone(); let _slot = app.heavy.acquire().await?; - let root = env("STUDIO_STORE_ROOT", "/srv"); + let root = env("SNOWGLOBE_STORE_ROOT", "/srv"); let rel = crate::files::relative( query.get("path").map(String::as_str).unwrap_or_default(), false, @@ -849,7 +849,7 @@ mod tests { #[test] fn reads_recover_interrupted_updates() { - if let Ok(file) = std::env::var("STUDIO_INDEX_CRASH_TEST_DB") { + if let Ok(file) = std::env::var("SNOWGLOBE_INDEX_CRASH_TEST_DB") { let db = Connection::open(file).unwrap(); db.execute_batch("PRAGMA cache_size=1; BEGIN IMMEDIATE; UPDATE file SET size=99;") .unwrap(); @@ -869,7 +869,7 @@ mod tests { assert!( std::process::Command::new(std::env::current_exe().unwrap()) .args(["--exact", thread.name().unwrap()]) - .env("STUDIO_INDEX_CRASH_TEST_DB", &file) + .env("SNOWGLOBE_INDEX_CRASH_TEST_DB", &file) .status() .unwrap() .success() diff --git a/dashboard/src/main.rs b/dashboard/src/main.rs index 4484c661cd89f0ee77be1190da2dd85c135467d7..e44fa733ef8533916cb79be5c9cbd7b8059e0851 100644 --- a/dashboard/src/main.rs +++ b/dashboard/src/main.rs @@ -10,6 +10,7 @@ mod guest; mod host; mod index; mod mcp; +mod namespace; mod observability; mod oidc; mod relay; @@ -113,6 +114,9 @@ struct App { } impl App { fn request(&self, method: Method, address: &str) -> Result { + if env("SNOWGLOBE_PREVIEW", "") == "1" && !matches!(method, Method::GET | Method::HEAD) { + return Err(Error::new(403, "This preview cannot change other services. Use production for app controls.")); + } let url = url::Url::parse(address)?; let client = match &self.internal { Some((base, client)) if url.origin() == base.origin() => client, @@ -264,6 +268,11 @@ async fn api(State(app): State>, request: Request) -> Result return users::picture(&app, &parts).await; } let me = user(request.headers())?; + if env("SNOWGLOBE_PREVIEW", "") == "1" + && matches!(parts[0], "seedbox" | "youtube" | "paper-clover") + { + return Err(Error::new(403, "This app uses production data. Open it in production to continue.")); + } let headers = request.headers().clone(); let section = match parts.first().copied().unwrap_or_default() { "host" | "metrics" | "live" => Some("metrics"), @@ -344,8 +353,8 @@ async fn api(State(app): State>, request: Request) -> Result #[tokio::main(worker_threads = 4)] async fn main() -> std::result::Result<(), Box> { - let address: std::net::IpAddr = env("STUDIO_LISTEN_ADDRESS", "127.0.0.1").parse()?; - let proof = match std::env::var_os("STUDIO_PROXY_TOKEN_FILE") { + let address: std::net::IpAddr = env("SNOWGLOBE_LISTEN_ADDRESS", "127.0.0.1").parse()?; + let proof = match std::env::var_os("SNOWGLOBE_PROXY_TOKEN_FILE").filter(|file| !file.is_empty()) { Some(file) => { let token = std::fs::read_to_string(file)?.trim().to_owned(); if token.len() != 64 || !token.bytes().all(|b| b.is_ascii_hexdigit()) { @@ -359,13 +368,13 @@ async fn main() -> std::result::Result<(), Box> { None if address.is_loopback() => None, None => { return Err(std::io::Error::other( - "Set STUDIO_PROXY_TOKEN_FILE before listening on a non-loopback address.", + "Set SNOWGLOBE_PROXY_TOKEN_FILE before listening on a non-loopback address.", ) .into()); } }; let certificates = if let Ok(path) = - std::env::var("STUDIO_CA_BUNDLE").or_else(|_| std::env::var("NODE_EXTRA_CA_CERTS")) + std::env::var("SNOWGLOBE_CA_BUNDLE").or_else(|_| std::env::var("NODE_EXTRA_CA_CERTS")) { reqwest::Certificate::from_pem_bundle(&std::fs::read(path)?)? } else { @@ -378,7 +387,7 @@ async fn main() -> std::result::Result<(), Box> { } http }; - let internal = std::env::var("STUDIO_INTERNAL_URL") + let internal = std::env::var("SNOWGLOBE_INTERNAL_URL") .ok() .map( |address| -> std::result::Result<_, Box> { @@ -392,17 +401,17 @@ async fn main() -> std::result::Result<(), Box> { || base.fragment().is_some() { return Err(std::io::Error::other( - "STUDIO_INTERNAL_URL must be an HTTPS origin.", + "SNOWGLOBE_INTERNAL_URL must be an HTTPS origin.", ) .into()); } let token = proof.as_ref().ok_or_else(|| { - std::io::Error::other("STUDIO_INTERNAL_URL requires STUDIO_PROXY_TOKEN_FILE.") + std::io::Error::other("SNOWGLOBE_INTERNAL_URL requires SNOWGLOBE_PROXY_TOKEN_FILE.") })?; let mut value = axum::http::HeaderValue::from_str(token)?; value.set_sensitive(true); let mut headers = HeaderMap::new(); - headers.insert("Studio-Proxy-Token", value); + headers.insert("Snowglobe-Proxy-Token", value); Ok(( base, client() @@ -425,35 +434,55 @@ async fn main() -> std::result::Result<(), Box> { let mut token = axum::http::HeaderValue::from_str(proof.as_ref().unwrap())?; token.set_sensitive(true); let mut headers = HeaderMap::new(); - headers.insert("Studio-Proxy-Token", token); + headers.insert("Snowglobe-Proxy-Token", token); shale_http = shale_http.default_headers(headers); } let shale_http = shale_http.build()?; let (live, _) = watch::channel(Bytes::new()); - let index = std::env::var("STUDIO_INDEX_POOL").ok().map(|pool| { + let index = std::env::var("SNOWGLOBE_INDEX_POOL").ok().map(|pool| { Arc::new(index::Index::new( pool, - env("STUDIO_INDEX_DIR", ".cache/index").into(), + env("SNOWGLOBE_INDEX_DIR", ".cache/index").into(), )) }); let origin = env( - "STUDIO_PUBLIC_ORIGIN", - &format!("https://snowglobe.{}", env("STUDIO_DOMAIN", "studio.test")), + "SNOWGLOBE_PUBLIC_ORIGIN", + &format!("https://snowglobe.{}", env("SNOWGLOBE_DOMAIN", "snowglobe.test")), ); let auth = auth::Store::new( - &PathBuf::from(env("STUDIO_DATA_DIR", "data")), + &PathBuf::from(env("SNOWGLOBE_DATA_DIR", "data")), &origin, &env( - "STUDIO_FILE_ORIGIN", - &format!("https://file.{}", env("STUDIO_DOMAIN", "studio.test")), + "SNOWGLOBE_FILE_ORIGIN", + &format!("https://file.{}", env("SNOWGLOBE_DOMAIN", "snowglobe.test")), ), &env( - "STUDIO_AUTH_RP_ID", - &format!("auth.{}", env("STUDIO_DOMAIN", "studio.test")), + "SNOWGLOBE_AUTH_RP_ID", + &format!("auth.{}", env("SNOWGLOBE_DOMAIN", "snowglobe.test")), ), ) .map_err(|error| std::io::Error::other(error.message))?; if let Some(path) = std::env::args().skip(1).next() { + if path == "--preview-account" { + if auth.ready() { + return Err(std::io::Error::other("Preview accounts require an empty store.").into()); + } + let id = uuid::Uuid::new_v4().to_string(); + let role = uuid::Uuid::new_v4().to_string(); + let password = mcp::secret(); + auth.import(json!({"rpId":env("SNOWGLOBE_AUTH_RP_ID", "auth.snowglobe.test"), + "roles":[{"id":role,"name":"infra-admin"}], + "users":[{"id":id,"username":"preview","enabled":true, + "firstName":"Preview","lastName":"Account","email":"preview@example.invalid", + "emailVerified":false,"requiredActions":[],"roles":[role],"credentials":[]}]})) + .map_err(|error| std::io::Error::other(error.message))?; + let hash = auth::password_hash(&password) + .map_err(|error| std::io::Error::other(error.message))?; + auth::set_password(&auth.db.lock().unwrap(), &id, &hash) + .map_err(|error| std::io::Error::other(error.message))?; + println!("{}", json!({"username":"preview","password":password})); + return Ok(()); + } if path == "--import-commit-identity" { let input = serde_json::from_reader(std::io::stdin())?; let output = identity::import(&auth, &input).map_err(|error| std::io::Error::other(error.message))?; @@ -486,13 +515,13 @@ async fn main() -> std::result::Result<(), Box> { println!("{result}"); return Ok(()); } - let import = PathBuf::from(env("STUDIO_DATA_DIR", "data")).join("accounts-import.json"); + let import = PathBuf::from(env("SNOWGLOBE_DATA_DIR", "data")).join("accounts-import.json"); if import.exists() { auth.import(serde_json::from_slice(&std::fs::read(&import)?)?) .map_err(|error| std::io::Error::other(error.message))?; std::fs::remove_file(&import)?; } - if env("STUDIO_AUTH_REQUIRED", "0") == "1" && !auth.ready() { + if env("SNOWGLOBE_AUTH_REQUIRED", "0") == "1" && !auth.ready() { return Err(std::io::Error::other( "Import accounts before starting native authentication.", ) @@ -500,13 +529,13 @@ async fn main() -> std::result::Result<(), Box> { } let app = Arc::new(App { auth, - mcp: mcp::Store::new(&PathBuf::from(env("STUDIO_DATA_DIR", "data")), &origin) + mcp: mcp::Store::new(&PathBuf::from(env("SNOWGLOBE_DATA_DIR", "data")), &origin) .map_err(|error| std::io::Error::other(error.message))?, relay: relay::Broker::default(), shale: shale::Backend::new( &env( - "STUDIO_SHALE_URL", - &format!("https://shale.{}", env("STUDIO_DOMAIN", "studio.test")), + "SNOWGLOBE_SHALE_URL", + &format!("https://shale.{}", env("SNOWGLOBE_DOMAIN", "snowglobe.test")), ), client(), ) @@ -517,8 +546,8 @@ async fn main() -> std::result::Result<(), Box> { cache: cache::Cache::default(), nomad_slots: Semaphore::new(4), specs: Mutex::new(HashMap::new()), - repo: env("STUDIO_REPO", "..").into(), - data: env("STUDIO_DATA_DIR", "data").into(), + repo: env("SNOWGLOBE_REPO", "..").into(), + data: env("SNOWGLOBE_DATA_DIR", "data").into(), live, usage: Mutex::new(HashMap::new()), vm_usage: Mutex::new(HashMap::new()), @@ -534,7 +563,7 @@ async fn main() -> std::result::Result<(), Box> { index.start(state).await; }); } - if std::env::var("STUDIO_YT_STATE").is_ok() { + if std::env::var("SNOWGLOBE_YT_STATE").is_ok() { for parts in [&[][..], &["library"][..]] { let state = app.clone(); tokio::spawn(async move { @@ -546,7 +575,7 @@ async fn main() -> std::result::Result<(), Box> { } core::start(app.clone()); telemetry::start(app.clone()); - let dist = env("STUDIO_WEB_DIR", "dist"); + let dist = env("SNOWGLOBE_WEB_DIR", "dist"); let router = Router::new() .route("/api/{*path}", any(api)) .route("/auth/{*path}", any(auth::route)) @@ -568,20 +597,20 @@ async fn main() -> std::result::Result<(), Box> { async move { request.headers_mut().remove("User-Id"); if mcp::public(request.uri().path()) { - request.headers_mut().remove("Studio-Proxy-Token"); + request.headers_mut().remove("Snowglobe-Proxy-Token"); request.headers_mut().remove("User-Name"); request.headers_mut().remove("User-Groups"); } else if let Some(proof) = proof { let supplied = request .headers() - .get("Studio-Proxy-Token") + .get("Snowglobe-Proxy-Token") .map(|value| value.as_bytes()) .unwrap_or_default(); if supplied.ct_eq(proof.as_bytes()).unwrap_u8() == 0 { return Error::new(403, "Open the dashboard through its sign-in page.") .into_response(); } - request.headers_mut().remove("Studio-Proxy-Token"); + request.headers_mut().remove("Snowglobe-Proxy-Token"); } if request.uri().path() == "/auth/shale/page" { return next.run(request).await; @@ -605,7 +634,7 @@ async fn main() -> std::result::Result<(), Box> { }; if account.is_null() { return if path.starts_with("/api/") { - Error::new(401, "Sign in to Snowglobe.").into_response() + Error::new(401, "Sign in to Snow Globe.").into_response() } else { ( StatusCode::FOUND, @@ -634,7 +663,7 @@ async fn main() -> std::result::Result<(), Box> { "location", format!( "https://shale.{}/", - env("STUDIO_DOMAIN", "studio.test") + env("SNOWGLOBE_DOMAIN", "snowglobe.test") ), )], ) @@ -650,7 +679,7 @@ async fn main() -> std::result::Result<(), Box> { .and_then(|v| v.to_str().ok()) != Some(app.auth.origin.origin().ascii_serialization().as_str()) { - return Error::new(403, "Open Snowglobe and try again.") + return Error::new(403, "Open Snow Globe and try again.") .into_response(); } if !array(&account["requiredActions"]).is_empty() diff --git a/dashboard/src/mcp.rs b/dashboard/src/mcp.rs index c812de698507efcbc3546e7e7daa41a9713c729f..ef3c9a08bebfcbc5eea3bfe1e5d9c85598b7a2fe 100644 --- a/dashboard/src/mcp.rs +++ b/dashboard/src/mcp.rs @@ -969,8 +969,8 @@ mod tests { let path = std::env::temp_dir() .canonicalize() .unwrap() - .join(format!("studio-mcp-test-{}", uuid::Uuid::new_v4())); - let store = Arc::new(Store::new(&path, "https://globe.studio.test").unwrap()); + .join(format!("snowglobe-mcp-test-{}", uuid::Uuid::new_v4())); + let store = Arc::new(Store::new(&path, "https://globe.snowglobe.test").unwrap()); Self { store, path } } fn code(&self, client: &str, code: &str) -> HashMap { diff --git a/dashboard/src/namespace-transition.json b/dashboard/src/namespace-transition.json new file mode 100644 index 0000000000000000000000000000000000000000..29f654d6dfc06fcc3a03dbb40098fc8bb727e62c --- /dev/null +++ b/dashboard/src/namespace-transition.json @@ -0,0 +1,4 @@ +{ + "previous": "studio", + "previousHost": "zenith" +} diff --git a/dashboard/src/namespace.rs b/dashboard/src/namespace.rs new file mode 100644 index 0000000000000000000000000000000000000000..03457f2467b745bfd5d9995b7b9afe0acb4b16be --- /dev/null +++ b/dashboard/src/namespace.rs @@ -0,0 +1,87 @@ +use crate::*; +use serde_json::Map; + +pub static PREVIOUS: std::sync::LazyLock = std::sync::LazyLock::new(|| { + serde_json::from_str::(include_str!("namespace-transition.json")).unwrap()["previous"] + .as_str().unwrap().to_owned() +}); + +fn keys(values: &mut Map, separator: &str) { + let prefix = format!("{}{separator}", PREVIOUS.as_str()); + let renamed: Vec<_> = values.keys().filter_map(|key| { + key.strip_prefix(&prefix).map(|suffix| (key.clone(), format!("snowglobe{separator}{suffix}"))) + }).collect(); + for (old, new) in renamed { + let value = values.remove(&old).unwrap(); + values.entry(new).or_insert(value); + } +} + +pub fn nomad(value: &mut Value) { + match value { + Value::Object(values) => { + if let Some(meta) = values.get_mut("Meta").and_then(Value::as_object_mut) { + keys(meta, "_"); + } + if let Some(tags) = values.get_mut("Tags").and_then(Value::as_array_mut) { + let prefix = format!("{}-metrics-path=", PREVIOUS.as_str()); + for tag in tags { + if let Some(path) = tag.as_str().and_then(|tag| tag.strip_prefix(&prefix)) { + *tag = json!(format!("snowglobe-metrics-path={path}")); + } + } + } + for value in values.values_mut() { + nomad(value); + } + } + Value::Array(values) => values.iter_mut().for_each(nomad), + _ => {} + } +} + +pub fn attributes(values: &mut Map) { + keys(values, "."); +} + +pub fn metric_selector(metric: &str, labels: &str) -> String { + let previous = metric.replacen("snowglobe_", &format!("{}_", PREVIOUS.as_str()), 1); + format!("sum without (__name__) ({metric}{labels}) or sum without (__name__) ({previous}{labels})") +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn persisted_metadata_is_read_once_with_current_values_winning() { + let old = PREVIOUS.as_str(); + let mut value = json!({"Meta": { + format!("{old}_name"): "old name", "snowglobe_name": "Snow Globe", + format!("{old}_hostname"): "knife.example.test" + }, "TaskGroups": [{"Services": [{"Tags": [format!("{old}-metrics-path=/metrics"), "caddy-host=knife.example.test"]}]}], + "Config": {"Env": {format!("{}_DOMAIN", old.to_uppercase()): "untouched"}}}); + nomad(&mut value); + assert_eq!(value["Meta"]["snowglobe_name"], "Snow Globe"); + assert_eq!(value["Meta"]["snowglobe_hostname"], "knife.example.test"); + assert!(value["Meta"].get(format!("{old}_name")).is_none()); + assert_eq!(value["TaskGroups"][0]["Services"][0]["Tags"][0], "snowglobe-metrics-path=/metrics"); + assert_eq!(value["Config"]["Env"][format!("{}_DOMAIN", old.to_uppercase())], "untouched"); + let normalized = value.clone(); + nomad(&mut value); + assert_eq!(value, normalized); + } + + #[test] + fn historical_trace_ownership_remains_explicit() { + let mut values = Map::from_iter([ + (format!("{}.service", PREVIOUS.as_str()), json!("private")), + (format!("{}.kind", PREVIOUS.as_str()), json!("edge")), + ("snowglobe.service".into(), json!("current")), + ]); + attributes(&mut values); + assert_eq!(values["snowglobe.service"], "current"); + assert_eq!(values["snowglobe.kind"], "edge"); + assert_eq!(values.len(), 2); + } +} diff --git a/dashboard/src/observability.rs b/dashboard/src/observability.rs index 0350ded0e7afeecdc968874d9544b79fc52de44b..1bf86dac2b40ce151eaa019570fee33d2d0f7ddc 100644 --- a/dashboard/src/observability.rs +++ b/dashboard/src/observability.rs @@ -129,14 +129,14 @@ impl ServerHandler for Observability { } } fn visible_span(span: &Value, jobs: &Value, resources: &Value) -> bool { - if let Some(id) = span["attributes"]["studio.service"].as_str() { + if let Some(id) = span["attributes"]["snowglobe.service"].as_str() { return array(resources).iter().any(|r| r == id); } let owners: Vec<_> = jobs .as_object() .into_iter() .flat_map(|jobs| jobs.iter()) - .filter(|(_, job)| job["job"]["Meta"]["studio_trace_service"] == span["service"]) + .filter(|(_, job)| job["job"]["Meta"]["snowglobe_trace_service"] == span["service"]) .map(|(id, _)| id) .collect(); owners.len() == 1 && array(resources).iter().any(|r| r == owners[0]) @@ -153,7 +153,7 @@ mod tests { use super::*; #[test] fn trace_grants_reject_foreign_and_ambiguous_spans() { - let jobs = json!({"allowed":{"job":{"Meta":{"studio_trace_service":"backend"}}},"private":{"job":{"Meta":{"studio_trace_service":"private-api"}}}}); + let jobs = json!({"allowed":{"job":{"Meta":{"snowglobe_trace_service":"backend"}}},"private":{"job":{"Meta":{"snowglobe_trace_service":"private-api"}}}}); let resources = json!(["allowed"]); assert!(visible_span( &json!({"service":"backend","attributes":{}}), @@ -166,17 +166,17 @@ mod tests { &resources )); assert!(!visible_span( - &json!({"service":"backend","attributes":{"studio.service":"private"}}), + &json!({"service":"backend","attributes":{"snowglobe.service":"private"}}), &jobs, &resources )); assert!(visible_span( - &json!({"service":"allowed","attributes":{"studio.service":"allowed"}}), + &json!({"service":"allowed","attributes":{"snowglobe.service":"allowed"}}), &jobs, &resources )); let mut ambiguous = jobs.clone(); - ambiguous["private"]["job"]["Meta"]["studio_trace_service"] = json!("backend"); + ambiguous["private"]["job"]["Meta"]["snowglobe_trace_service"] = json!("backend"); assert!(!visible_span( &json!({"service":"backend","attributes":{}}), &ambiguous, diff --git a/dashboard/src/relay.rs b/dashboard/src/relay.rs index 1e69e578e3e73839a2fa53415ab7677dd8097b63..7df6640dc68eb705c46ccd6f4aaf29cdefd87dd4 100644 --- a/dashboard/src/relay.rs +++ b/dashboard/src/relay.rs @@ -801,7 +801,7 @@ pub fn router(app: Arc) -> Router { let state = app.clone(); let expected_host = app.mcp.origin[url::Position::BeforeHost..url::Position::AfterPort].to_owned(); - let agent = env("STUDIO_AGENT_DIR", "agent"); + let agent = env("SNOWGLOBE_AGENT_DIR", "agent"); Router::new() .route("/pairing", any(pairing)) .route("/agent/connect", any(connect)) @@ -848,8 +848,8 @@ mod tests { let path = std::env::temp_dir() .canonicalize() .unwrap() - .join(format!("studio-relay-test-{}", uuid::Uuid::new_v4())); - let store = Arc::new(mcp::Store::new(&path, "https://globe.studio.test").unwrap()); + .join(format!("snowglobe-relay-test-{}", uuid::Uuid::new_v4())); + let store = Arc::new(mcp::Store::new(&path, "https://globe.snowglobe.test").unwrap()); let machine = uuid::Uuid::new_v4().to_string(); let grant = uuid::Uuid::new_v4().to_string(); { diff --git a/dashboard/src/shale.rs b/dashboard/src/shale.rs index d227dc7cafad2e1d8ca8d76c425ff36243f63153..3de986d0d5587fdbad631527265f74314b554e15 100644 --- a/dashboard/src/shale.rs +++ b/dashboard/src/shale.rs @@ -586,7 +586,7 @@ pub async fn manage( 600, )?; tx.commit()?; - Ok(axum::Json(json!({"redirect":app.shale.origin.join(&format!("/-/studio-mcp/{nonce}"))?.as_str()})).into_response()) + Ok(axum::Json(json!({"redirect":app.shale.origin.join(&format!("/-/snowglobe-mcp/{nonce}"))?.as_str()})).into_response()) } Method::DELETE => { let session = { @@ -639,7 +639,7 @@ pub async fn oauth(app: Arc, request: Request) -> Response { tx.commit()?; return Ok((StatusCode::FOUND, [ ("location", string(&link["authorization"]).to_owned()), - ("set-cookie", format!("studio_mcp_shale_link={nonce}; Path=/-/callback; Secure; HttpOnly; SameSite=Lax; Max-Age=600")), + ("set-cookie", format!("snowglobe_mcp_shale_link={nonce}; Path=/-/callback; Secure; HttpOnly; SameSite=Lax; Max-Age=600")), ]).into_response()); } if !callback { @@ -647,7 +647,7 @@ pub async fn oauth(app: Arc, request: Request) -> Response { } let cookies: Vec<_> = request.headers().get_all("cookie").iter() .filter_map(|v| v.to_str().ok()).flat_map(|v| v.split(';')) - .filter_map(|part| part.trim().strip_prefix("studio_mcp_shale_link=")).collect(); + .filter_map(|part| part.trim().strip_prefix("snowglobe_mcp_shale_link=")).collect(); if cookies.len() != 1 || cookies[0].len() != 43 { return Err(Error::new(400, "This Shale link cookie is missing. Start linking again.")); } @@ -724,7 +724,7 @@ pub async fn oauth(app: Arc, request: Request) -> Response { if callback { response.headers_mut().append( "set-cookie", - "studio_mcp_shale_link=; Path=/-/callback; Secure; HttpOnly; SameSite=Lax; Max-Age=0" + "snowglobe_mcp_shale_link=; Path=/-/callback; Secure; HttpOnly; SameSite=Lax; Max-Age=0" .parse() .unwrap(), ); @@ -754,7 +754,7 @@ mod tests { } #[test] fn repository_names_cannot_change_origin_or_path_segments() { - let origin = url::Url::parse("https://shale.studio.test").unwrap(); + let origin = url::Url::parse("https://shale.snowglobe.test").unwrap(); for name in [ "", ".", diff --git a/dashboard/src/shale_page.rs b/dashboard/src/shale_page.rs index a84143ec1426325c873384b8fc9b63c37fe5063f..eaf07b9d98287a6b0053e1e17531aa269a75735d 100644 --- a/dashboard/src/shale_page.rs +++ b/dashboard/src/shale_page.rs @@ -350,7 +350,7 @@ pub async fn proxy(app: Arc, request: Request) -> Result { .get("host") .and_then(|v| v.to_str().ok()) .unwrap_or_default(); - let domain = env("STUDIO_DOMAIN", "studio.test"); + let domain = env("SNOWGLOBE_DOMAIN", "snowglobe.test"); let service = host.strip_suffix(&format!(".{domain}")).unwrap_or_default().to_owned(); if service != "shale" && !(service @@ -361,13 +361,13 @@ pub async fn proxy(app: Arc, request: Request) -> Result { } let upstream: std::net::SocketAddr = parts .headers - .get("studio-shale-upstream") + .get("snowglobe-shale-upstream") .and_then(|v| v.to_str().ok()) .ok_or_else(|| Error::new(403, "Open Shale to continue."))? .parse()?; let uri: axum::http::Uri = parts .headers - .get("studio-shale-uri") + .get("snowglobe-shale-uri") .and_then(|v| v.to_str().ok()) .ok_or_else(|| Error::new(403, "Open Shale to continue."))? .parse()?; @@ -398,9 +398,9 @@ pub async fn proxy(app: Arc, request: Request) -> Result { } strip_hop_headers(&mut parts.headers); for name in [ - "studio-shale-upstream", - "studio-shale-uri", - "studio-proxy-token", + "snowglobe-shale-upstream", + "snowglobe-shale-uri", + "snowglobe-proxy-token", "user-name", "user-groups", "user-id", diff --git a/dashboard/src/shares.rs b/dashboard/src/shares.rs index 583683481ae992a66bd0900d2d30f2f19b0bd2f0..bc7c0e3e8cebfdaa03d6925bfe15e115aade004d 100644 --- a/dashboard/src/shares.rs +++ b/dashboard/src/shares.rs @@ -68,8 +68,8 @@ pub async fn route( .await?; if value.is_object() { value["filesUrl"] = json!(env( - "STUDIO_FILES_URL", - &format!("https://file.{}", env("STUDIO_DOMAIN", "studio.test")), + "SNOWGLOBE_FILES_URL", + &format!("https://file.{}", env("SNOWGLOBE_DOMAIN", "snowglobe.test")), )); } Ok(Document::new(value).response()) diff --git a/dashboard/src/storage.rs b/dashboard/src/storage.rs index 0a0d8f06c9c0c3bdca3d82b5b899e1020d6d41c5..a7d833a605d0e040a982a7a1afb69c8b62cff40d 100644 --- a/dashboard/src/storage.rs +++ b/dashboard/src/storage.rs @@ -147,7 +147,7 @@ pub async fn route( [] if method == Method::GET => { let found=app.cache.get("storage".into(),Duration::from_secs(30),move || async move { let (pool,mut datasets)=tokio::try_join!(pool(),datasets())?;let root=array(&datasets).iter().find(|d| d["name"]==pool["name"]).ok_or_else(|| Error::new(502,"The storage pool has no root dataset."))?;let held:f64=array(&datasets).iter().map(|d| number(&d["usedbysnapshots"])).sum();let space=json!({"live":number(&root["used"])-held,"held":held,"free":root["available"]}); - let media=env("STUDIO_MEDIA_ROOT",&format!("{}/clover/Media",env("STUDIO_STORE_ROOT","/srv")));for dataset in datasets.as_array_mut().unwrap() {let mount=dataset["mountpoint"].as_str().map(str::to_owned);dataset["link"]=mount.as_ref().map(|p| apps::file_link(p,false)).unwrap_or(Value::Null);dataset["media"]=mount.and_then(|p| std::path::Path::new(&p).strip_prefix(&media).ok().map(|p| p.to_string_lossy().into_owned())).map(|p| json!(p)).unwrap_or(Value::Null);}Ok(json!({"pool":pool,"datasets":datasets,"space":space})) + let media=env("SNOWGLOBE_MEDIA_ROOT",&format!("{}/clover/Media",env("SNOWGLOBE_STORE_ROOT","/srv")));for dataset in datasets.as_array_mut().unwrap() {let mount=dataset["mountpoint"].as_str().map(str::to_owned);dataset["link"]=mount.as_ref().map(|p| apps::file_link(p,false)).unwrap_or(Value::Null);dataset["media"]=mount.and_then(|p| std::path::Path::new(&p).strip_prefix(&media).ok().map(|p| p.to_string_lossy().into_owned())).map(|p| json!(p)).unwrap_or(Value::Null);}Ok(json!({"pool":pool,"datasets":datasets,"space":space})) }).await?; return Ok(found.response()); } diff --git a/dashboard/src/telemetry.rs b/dashboard/src/telemetry.rs index 2bf2d94de43d0a72bafc78fa2dec6995e12f9939..01a23fd0e7d9e11d8cb7a9d75bfe0a1006f225fd 100644 --- a/dashboard/src/telemetry.rs +++ b/dashboard/src/telemetry.rs @@ -47,7 +47,11 @@ pub async fn read( app.cache .get(key, Duration::from_secs(2), move || async move { let request = if let Some(form) = &form { - state.request(Method::POST, &url)?.form(form) + if env("SNOWGLOBE_PREVIEW", "") == "1" { + state.request(Method::GET, &url)?.query(form) + } else { + state.request(Method::POST, &url)?.form(form) + } } else { state.request(Method::GET, &url)? }; @@ -72,16 +76,16 @@ pub async fn read( } const METRICS: &[(&str, &str)] = &[ - ("host.cpu", "studio_host_cpu_percent"), - ("host.memory", "studio_host_memory_bytes"), - ("host.temperature", "studio_host_temperature_celsius"), - ("host.power", "studio_host_power_watts"), - ("host.gpu", "studio_host_gpu_percent"), - ("host.network", "studio_host_network_bytes_per_second"), - ("service.cpu", "studio_service_cpu_cores"), - ("service.memory", "studio_service_memory_bytes"), - ("vm.cpu", "studio_vm_cpu_percent"), - ("vm.memory", "studio_vm_memory_bytes"), + ("host.cpu", "snowglobe_host_cpu_percent"), + ("host.memory", "snowglobe_host_memory_bytes"), + ("host.temperature", "snowglobe_host_temperature_celsius"), + ("host.power", "snowglobe_host_power_watts"), + ("host.gpu", "snowglobe_host_gpu_percent"), + ("host.network", "snowglobe_host_network_bytes_per_second"), + ("service.cpu", "snowglobe_service_cpu_cores"), + ("service.memory", "snowglobe_service_memory_bytes"), + ("vm.cpu", "snowglobe_vm_cpu_percent"), + ("vm.memory", "snowglobe_vm_memory_bytes"), ]; fn series(result: &Value, application: bool) -> Value { @@ -173,9 +177,7 @@ pub async fn metrics( .find(|(key, _)| *key == name) .ok_or_else(|| Error::new(400, "Invalid metric."))? .1; - format!( - "{metric}{}", - service + let labels = service .map(|id| format!( "{{{}={}}}", if name.starts_with("vm.") { @@ -185,8 +187,8 @@ pub async fn metrics( }, quote(id) )) - .unwrap_or_default() - ) + .unwrap_or_default(); + namespace::metric_selector(metric, &labels) }; let key = if window.is_some() { format!("series:{selector}:fixed:{from}:{to}") @@ -225,8 +227,8 @@ pub async fn metric_names(app: Arc, id: &str) -> Result { ( "match[]", format!( - "{{{label}={},__name__!~\"studio_.*\"}}", - serde_json::to_string(id).unwrap() + "{{{label}={},__name__!~\"(snowglobe|{})_.*\"}}", + serde_json::to_string(id).unwrap(), namespace::PREVIOUS.as_str() ), ) })); @@ -366,6 +368,7 @@ fn span(row: &Value) -> Value { .map(|key| (key.into(), value.clone())) }) .collect(); + namespace::attributes(&mut attributes); for (from, to) in [ ("http.request.method", "http.request.method"), ("http.response.status_code", "http.response.status_code"), @@ -383,7 +386,7 @@ fn span(row: &Value) -> Value { .cloned() .unwrap_or(Value::Null); let service = attributes - .get("studio.service") + .get("snowglobe.service") .cloned() .unwrap_or_else(|| { row["resource_attr:service.name"] @@ -405,7 +408,7 @@ fn span(row: &Value) -> Value { } else { Value::Null }; - let name = if attributes.get("studio.kind").is_some_and(|v| v == "edge") { + let name = if attributes.get("snowglobe.kind").is_some_and(|v| v == "edge") { format!("edge: {}", row["name"].as_str().unwrap_or("request")) } else { string(&row["name"]).into() @@ -500,7 +503,7 @@ pub async fn traces( visible: impl Fn(&Value) -> bool, ) -> Result { let jobs = core::scan(app.clone()).await?; - let Some(service) = jobs.value[id]["job"]["Meta"]["studio_trace_service"] + let Some(service) = jobs.value[id]["job"]["Meta"]["snowglobe_trace_service"] .as_str() .filter(|s| !s.is_empty()) else { @@ -611,7 +614,7 @@ pub async fn traces( let value = |trace: &Value| { array(&trace["spans"]) .iter() - .find(|s| s["service"] == service && s["attributes"]["studio.kind"] != "edge") + .find(|s| s["service"] == service && s["attributes"]["snowglobe.kind"] != "edge") .map(|s| number(&s[if key == "_time" { "start" } else { "duration" }])) .unwrap_or(0.0) }; @@ -695,21 +698,21 @@ impl Recorder { }; let mut lines = vec![ format!( - "studio_host_cpu_percent {} {timestamp}", + "snowglobe_host_cpu_percent {} {timestamp}", number(&live["host"]["cpu"]) ), format!( - "studio_host_memory_bytes {} {timestamp}", + "snowglobe_host_memory_bytes {} {timestamp}", number(&live["host"]["memory"]) ), ]; for (id, (cpu, memory)) in usage { let id = serde_json::to_string(&id)?; lines.push(format!( - "studio_service_cpu_cores{{service={id}}} {cpu} {timestamp}" + "snowglobe_service_cpu_cores{{service={id}}} {cpu} {timestamp}" )); lines.push(format!( - "studio_service_memory_bytes{{service={id}}} {memory} {timestamp}" + "snowglobe_service_memory_bytes{{service={id}}} {memory} {timestamp}" )); } let sample = host::sample(app.clone()).await?; @@ -719,22 +722,22 @@ impl Recorder { let network_at = number(&sample["at"]); if let Some((old_rx, old_tx, t)) = self.network.filter(|(_, _, t)| network_at > *t) { lines.push(format!( - "studio_host_network_bytes_per_second{{direction=\"down\"}} {} {timestamp}", + "snowglobe_host_network_bytes_per_second{{direction=\"down\"}} {} {timestamp}", ((rx - old_rx) / (network_at - t)).max(0.0) )); lines.push(format!( - "studio_host_network_bytes_per_second{{direction=\"up\"}} {} {timestamp}", + "snowglobe_host_network_bytes_per_second{{direction=\"up\"}} {} {timestamp}", ((tx - old_tx) / (network_at - t)).max(0.0) )); } self.network = Some((rx, tx, network_at)); if let Some(value) = live["host"]["temperature"].as_f64() { lines.push(format!( - "studio_host_temperature_celsius {value} {timestamp}" + "snowglobe_host_temperature_celsius {value} {timestamp}" )); } if let Some(value) = sample["gpu"].as_f64() { - lines.push(format!("studio_host_gpu_percent {value} {timestamp}")); + lines.push(format!("snowglobe_host_gpu_percent {value} {timestamp}")); } if let Ok(Ok(stats)) = tokio::time::timeout( Duration::from_secs(8), @@ -757,11 +760,11 @@ impl Recorder { usage.insert(id.clone(), json!({"cpu":cpu,"memory":stat["memory"]})); next.insert(id.clone(), (number(&stat["cpu"]), at)); lines.push(format!( - "studio_vm_cpu_percent{{vm={}}} {cpu} {timestamp}", + "snowglobe_vm_cpu_percent{{vm={}}} {cpu} {timestamp}", serde_json::to_string(id)? )); lines.push(format!( - "studio_vm_memory_bytes{{vm={}}} {} {timestamp}", + "snowglobe_vm_memory_bytes{{vm={}}} {} {timestamp}", serde_json::to_string(id)?, number(&stat["memory"]) )); @@ -778,7 +781,7 @@ pub fn start(app: Arc) { loop { let result = async { let base = endpoint(app.clone(), "victoria-metrics").await?; - if env("STUDIO_METRICS", "") == "follow" { + if env("SNOWGLOBE_METRICS", "") == "follow" { let names: Vec<_> = METRICS .iter() .filter(|(k, _)| k.starts_with("service.") || k.starts_with("vm.")) @@ -828,7 +831,7 @@ pub fn start(app: Arc) { .flat_map(|group| array(&group["Services"])) .flat_map(move |service| { array(&service["Tags"]).iter().filter_map(move |tag| { - string(tag).strip_prefix("studio-metrics-path=").map( + string(tag).strip_prefix("snowglobe-metrics-path=").map( |path| { ( id.clone(), diff --git a/dashboard/src/vms.rs b/dashboard/src/vms.rs index a1c311e6cad72aea7683183be64379dd278e3274..f290e53c13ea9cda6fdc06874ba3e384856ef7d2 100644 --- a/dashboard/src/vms.rs +++ b/dashboard/src/vms.rs @@ -45,8 +45,8 @@ pub async fn console(request: Request, name: &str, serial: bool) -> Resultmaybe later -

Finish your profile and change any temporary password to open Snowglobe and Files.

+

Finish your profile and change any temporary password to open Snow Globe and Files.

@@ -212,7 +212,7 @@ function SignIn(props: { user: Self }) { }); return

sign-in

-

These sign-in methods work with Snowglobe and Files.

+

These sign-in methods work with Snow Globe and Files.

password{password() ? "set" : "none"} diff --git a/dashboard/web/pages/MCPConsent.tsx b/dashboard/web/pages/MCPConsent.tsx index 49f017c38fa99dde69d47b52f7e57f966a02f8a7..95797afa92cc36c79c9bf1a3a6fcf0f28d7ef23a 100644 --- a/dashboard/web/pages/MCPConsent.tsx +++ b/dashboard/web/pages/MCPConsent.tsx @@ -48,7 +48,7 @@ export function MCPConsent() { }; return
-
Snowglobe · MCP connection
+
Snow Globe · MCP connection

{consent.state === "errored" ? "Connection unavailable" : "Loading connection…"}

}> diff --git a/dashboard/web/pages/Users.tsx b/dashboard/web/pages/Users.tsx index 18c9dde32a1bb46fa1edcd78586beea964bcaab8..70c6141ab90e4e86057f5716d4183ff976fce7ca 100644 --- a/dashboard/web/pages/Users.tsx +++ b/dashboard/web/pages/Users.tsx @@ -78,7 +78,7 @@ function groupTip(group: string, d: Data) { const appName = (clientId: string, services: ServiceSummary[]) => - clientId === "dashboard" ? "Snowglobe" : clientId === "file" ? "Files" : services.find((service) => service.id === clientId)?.name ?? clientId; + clientId === "dashboard" ? "Snow Globe" : clientId === "file" ? "Files" : services.find((service) => service.id === clientId)?.name ?? clientId; /** "active 3h ago in Jellyfin, Shale", from their open sessions. */ function seen(user: User, services: ServiceSummary[]) { @@ -531,7 +531,7 @@ function Profile(props: { user: User; data: Data; refetch: () => unknown }) { const name = props.user.username; showTextDialog({ title: `Delete ${name}?`, - description: `${name} is signed out and loses access to Snowglobe and Files. This can't be undone.`, + description: `${name} is signed out and loses access to Snow Globe and Files. This can't be undone.`, label: `type ${name} to confirm`, validateInput: (value) => value === name, confirmLabel: "delete user", diff --git a/flake.nix b/flake.nix index 30220e04120f5e931eb4ed49ed103c62827811ee..7ee41c2b59a24f7fb4f675bf9bad35eacae1f526 100644 --- a/flake.nix +++ b/flake.nix @@ -1,5 +1,5 @@ { - description = "Clover infrastructure prototype"; + description = "Snow Globe infrastructure"; inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05"; @@ -14,7 +14,7 @@ system = "x86_64-linux"; modules = [ "${nixpkgs}/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix" - ./nixos/zenith-network.nix + ./nixos/snowglobe-network.nix ({ lib, pkgs, ... }: { networking.hostName = "infra-2-installer"; networking.networkmanager.enable = lib.mkForce false; @@ -40,9 +40,9 @@ modules = [ ./nixos/configuration.nix ./nixos/vm.nix ]; }; - nixosConfigurations.zenith = nixpkgs.lib.nixosSystem { + nixosConfigurations.snowglobe = nixpkgs.lib.nixosSystem { system = "x86_64-linux"; - modules = [ ./nixos/configuration.nix ./nixos/zenith.nix ./nixos/hardware-configuration.nix ]; + modules = [ ./nixos/configuration.nix ./nixos/snowglobe.nix ./nixos/hardware-configuration.nix ]; }; }; } diff --git a/nixos/configuration.nix b/nixos/configuration.nix index 46019971f6b0a28638cc43d7cd68c4d459ce1de5..03e3711d3d128e5622c84879faad2380fe7f343f 100644 --- a/nixos/configuration.nix +++ b/nixos/configuration.nix @@ -3,10 +3,10 @@ let dashboard = pkgs.callPackage ./dashboard.nix { }; dashboardPort = "7072"; internalPort = 8448; - proxyToken = "/var/lib/studio/dashboard-proxy.token"; + proxyToken = "/var/lib/snowglobe/dashboard-proxy.token"; hostTools = lib.fileset.toSource { root = ../.; - fileset = lib.fileset.unions [ ../tools/dashboard-host.py ../service/copyparty/manage.py ../service/copyparty/copyparty.conf ../tools/shale-labels.py ../config/shale.json ../dashboard/src/host-protocol.json ../tools/vms.py ../tools/vm-ssh.py ../tools/dashboard-run.py ../tools/release.py ../guest/protocol.json ]; + fileset = lib.fileset.unions [ ../tools/dashboard-host.py ../service/copyparty/manage.py ../service/copyparty/copyparty.conf ../tools/shale-labels.py ../config/shale.json ../dashboard/src/host-protocol.json ../tools/vms.py ../tools/vm-ssh.py ../tools/dashboard-run.py ../tools/release.py ../tools/namespace-transition.py ../dashboard/src/namespace-transition.json ../guest/protocol.json ]; }; nativePkl = pkgs.callPackage ./pkl.nix { }; screenPython = pkgs.python3.withPackages (packages: [ packages.pygobject3 packages.gst-python ]); @@ -70,7 +70,7 @@ in }; }; systemd.services.caddy.environment = { - OTEL_SERVICE_NAME = "studio-caddy"; + OTEL_SERVICE_NAME = "snowglobe-caddy"; OTEL_EXPORTER_OTLP_TRACES_ENDPOINT = "http://127.0.0.1:10428/insert/opentelemetry/v1/traces"; OTEL_TRACES_SAMPLER = "parentbased_traceidratio"; OTEL_TRACES_SAMPLER_ARG = "0.25"; @@ -109,14 +109,14 @@ in }; }; - systemd.services.studio-log-shipper = { + systemd.services.snowglobe-log-shipper = { wantedBy = [ "multi-user.target" ]; after = [ "nomad.service" ]; wants = [ "nomad.service" ]; - unitConfig.ConditionPathExists = "/opt/studio/current/tools/log-shipper.py"; + unitConfig.ConditionPathExists = "/opt/snowglobe/current/tools/log-shipper.py"; path = [ pkgs.systemd ]; serviceConfig = { - ExecStart = "${pkgs.python3}/bin/python3 /opt/studio/current/tools/log-shipper.py"; + ExecStart = "${pkgs.python3}/bin/python3 /opt/snowglobe/current/tools/log-shipper.py"; Restart = "always"; RestartSec = 5; }; @@ -127,16 +127,16 @@ in requires = [ "podman.socket" ]; }; - systemd.services.studio-router = { + systemd.services.snowglobe-router = { wantedBy = [ "multi-user.target" ]; wants = [ "nomad.service" "caddy.service" ]; after = [ "nomad.service" "caddy.service" ]; - unitConfig.ConditionPathExists = "/var/lib/studio/router.token"; + unitConfig.ConditionPathExists = "/var/lib/snowglobe/router.token"; environment = { - STUDIO_DOMAIN = config.environment.variables.STUDIO_DOMAIN; - STUDIO_DASHBOARD_PORT = dashboardPort; - STUDIO_INTERNAL_PORT = toString internalPort; - STUDIO_PROXY_TOKEN_FILE = proxyToken; + SNOWGLOBE_DOMAIN = config.environment.variables.SNOWGLOBE_DOMAIN; + SNOWGLOBE_DASHBOARD_PORT = dashboardPort; + SNOWGLOBE_INTERNAL_PORT = toString internalPort; + SNOWGLOBE_PROXY_TOKEN_FILE = proxyToken; }; serviceConfig = { ExecStart = "${pkgs.python3}/bin/python3 ${../tools/router.py}"; @@ -145,26 +145,27 @@ in }; }; - users.groups.studio-dashboard = { }; - users.users.studio-dashboard = { + users.groups.snowglobe-dashboard.gid = 994; + users.users.snowglobe-dashboard = { + uid = 997; isSystemUser = true; - group = "studio-dashboard"; + group = "snowglobe-dashboard"; }; - systemd.services.studio-vm-screen = { + systemd.services.snowglobe-vm-screen = { wantedBy = [ "multi-user.target" ]; - wants = [ "studio-host.service" ]; - after = [ "studio-host.service" ]; + wants = [ "snowglobe-host.service" ]; + after = [ "snowglobe-host.service" ]; environment = { GST_PLUGIN_SYSTEM_PATH_1_0 = lib.makeSearchPath "lib/gstreamer-1.0" screenPlugins; GI_TYPELIB_PATH = lib.makeSearchPath "lib/girepository-1.0" (map lib.getLib ([ pkgs.gst_all_1.gstreamer ] ++ screenPlugins)); LD_LIBRARY_PATH = "/run/opengl-driver/lib"; - STUDIO_VM_GUEST_PROTOCOL = toString ../guest/protocol.json; + SNOWGLOBE_VM_GUEST_PROTOCOL = toString ../guest/protocol.json; }; serviceConfig = { ExecStart = "${screenPython}/bin/python3 ${../tools/vm-screen.py}"; - User = "studio-dashboard"; - Group = "studio-dashboard"; - RuntimeDirectory = "studio-vm-screen"; + User = "snowglobe-dashboard"; + Group = "snowglobe-dashboard"; + RuntimeDirectory = "snowglobe-vm-screen"; RuntimeDirectoryMode = "0700"; RuntimeDirectoryPreserve = "yes"; UMask = "0077"; @@ -182,29 +183,29 @@ in RestartSec = 2; }; }; - systemd.services.studio-host = { + systemd.services.snowglobe-host = { wantedBy = [ "multi-user.target" ]; wants = [ "podman.socket" ]; after = [ "podman.socket" ]; path = [ pkgs.zfs pkgs.util-linux pkgs.python3 pkgs.libvirt pkgs.qemu pkgs.gptfdisk pkgs.cdrkit pkgs.openssl pkgs.openssh pkgs.podman pkgs.nomad pkgs.systemd pkgs.rsync pkgs.nixos-rebuild nativePkl ] ++ lib.optional (builtins.elem "nvidia" config.services.xserver.videoDrivers) (lib.getBin config.hardware.nvidia.package); - environment = (lib.filterAttrs (name: _: lib.hasPrefix "STUDIO_" name) config.environment.variables) // { - STUDIO_POOL = lib.attrByPath [ "STUDIO_POOL" ] "studio-demo" config.environment.variables; - STUDIO_DASHBOARD_USER = "studio-dashboard"; - STUDIO_VM_ACCEL = lib.attrByPath [ "STUDIO_VM_ACCEL" ] "kvm" config.environment.variables; + environment = (lib.filterAttrs (name: _: lib.hasPrefix "SNOWGLOBE_" name) config.environment.variables) // { + SNOWGLOBE_POOL = lib.attrByPath [ "SNOWGLOBE_POOL" ] "snowglobe-demo" config.environment.variables; + SNOWGLOBE_DASHBOARD_USER = "snowglobe-dashboard"; + SNOWGLOBE_VM_ACCEL = lib.attrByPath [ "SNOWGLOBE_VM_ACCEL" ] "kvm" config.environment.variables; }; serviceConfig = { ExecStart = "${pkgs.python3}/bin/python3 ${hostTools}/tools/dashboard-host.py"; Type = "notify"; - Group = "studio-dashboard"; - StateDirectory = "studio/host"; + Group = "snowglobe-dashboard"; + StateDirectory = "snowglobe/host"; StateDirectoryMode = "0700"; - RuntimeDirectory = "studio-host"; + RuntimeDirectory = "snowglobe-host"; RuntimeDirectoryMode = "0750"; RuntimeDirectoryPreserve = "yes"; UMask = "0077"; ProtectSystem = "strict"; - ReadWritePaths = [ "/srv/vm" "-/srv/prod/shale/data" "-/srv/prod/copyparty/cfg" "/srv/staging" "/srv/clover" "/var/lib/studio/dashboard" ] ++ lib.optionals (config.networking.hostName != "clover-demo") [ "/srv/clover/Media/vm" ]; + ReadWritePaths = [ "/srv/vm" "-/srv/prod/shale/data" "-/srv/prod/copyparty/cfg" "/srv/staging" "/srv/clover" "/var/lib/snowglobe/dashboard" ] ++ lib.optionals (config.networking.hostName != "snowglobe-rehearsal") [ "/srv/clover/Media/vm" ]; ProtectHome = true; PrivateTmp = true; NoNewPrivileges = true; @@ -220,49 +221,49 @@ in RestartSec = 2; }; }; - systemd.services.studio-dashboard = let - environment = (lib.filterAttrs (name: _: lib.hasPrefix "STUDIO_" name) config.environment.variables) // { - STUDIO_DATA_DIR = "/data"; - STUDIO_PROXY_TOKEN_FILE = "/run/secrets/dashboard-proxy.token"; - STUDIO_NOMAD_TOKEN_FILE = "/run/secrets/dashboard-nomad.token"; - STUDIO_CA_BUNDLE = "/run/secrets/ca-bundle.crt"; - STUDIO_INDEX_DIR = "/data/index"; - STUDIO_INTERNAL_URL = "https://dashboard.internal.${config.environment.variables.STUDIO_DOMAIN}:${toString internalPort}"; - STUDIO_FILES_URL = "https://file.${config.environment.variables.STUDIO_DOMAIN}"; - STUDIO_AUTH_REQUIRED = "1"; - STUDIO_AUTH_RP_ID = "auth.${config.environment.variables.STUDIO_DOMAIN}"; - STUDIO_FILE_ORIGIN = "https://file.${config.environment.variables.STUDIO_DOMAIN}"; - STUDIO_PUBLIC_ORIGIN = "https://snowglobe.${config.environment.variables.STUDIO_DOMAIN}"; - STUDIO_JELLYFIN_URL = "https://jelly.${config.environment.variables.STUDIO_DOMAIN}"; - STUDIO_SHALE_URL = "https://shale.${config.environment.variables.STUDIO_DOMAIN}"; - STUDIO_PUBLISHED_ROOT = "/srv/clover/Published"; - STUDIO_DOMAIN = config.environment.variables.STUDIO_DOMAIN; - STUDIO_POOL = lib.attrByPath [ "STUDIO_POOL" ] "studio-demo" config.environment.variables; - STUDIO_NODE_NAME = config.networking.hostName; - STUDIO_MEDIA_READ_ONLY = lib.attrByPath [ "STUDIO_MEDIA_READ_ONLY" ] "false" config.environment.variables; - STUDIO_CLOVER_READ_ONLY = lib.attrByPath [ "STUDIO_CLOVER_READ_ONLY" ] "false" config.environment.variables; + systemd.services.snowglobe-dashboard = let + environment = (lib.filterAttrs (name: _: lib.hasPrefix "SNOWGLOBE_" name) config.environment.variables) // { + SNOWGLOBE_DATA_DIR = "/data"; + SNOWGLOBE_PROXY_TOKEN_FILE = "/run/secrets/dashboard-proxy.token"; + SNOWGLOBE_NOMAD_TOKEN_FILE = "/run/secrets/dashboard-nomad.token"; + SNOWGLOBE_CA_BUNDLE = "/run/secrets/ca-bundle.crt"; + SNOWGLOBE_INDEX_DIR = "/data/index"; + SNOWGLOBE_INTERNAL_URL = "https://dashboard.internal.${config.environment.variables.SNOWGLOBE_DOMAIN}:${toString internalPort}"; + SNOWGLOBE_FILES_URL = "https://file.${config.environment.variables.SNOWGLOBE_DOMAIN}"; + SNOWGLOBE_AUTH_REQUIRED = "1"; + SNOWGLOBE_AUTH_RP_ID = "auth.${config.environment.variables.SNOWGLOBE_DOMAIN}"; + SNOWGLOBE_FILE_ORIGIN = "https://file.${config.environment.variables.SNOWGLOBE_DOMAIN}"; + SNOWGLOBE_PUBLIC_ORIGIN = "https://snowglobe.${config.environment.variables.SNOWGLOBE_DOMAIN}"; + SNOWGLOBE_JELLYFIN_URL = "https://jelly.${config.environment.variables.SNOWGLOBE_DOMAIN}"; + SNOWGLOBE_SHALE_URL = "https://shale.${config.environment.variables.SNOWGLOBE_DOMAIN}"; + SNOWGLOBE_PUBLISHED_ROOT = "/srv/clover/Published"; + SNOWGLOBE_DOMAIN = config.environment.variables.SNOWGLOBE_DOMAIN; + SNOWGLOBE_POOL = lib.attrByPath [ "SNOWGLOBE_POOL" ] "snowglobe-demo" config.environment.variables; + SNOWGLOBE_NODE_NAME = config.networking.hostName; + SNOWGLOBE_MEDIA_READ_ONLY = lib.attrByPath [ "SNOWGLOBE_MEDIA_READ_ONLY" ] "false" config.environment.variables; + SNOWGLOBE_CLOVER_READ_ONLY = lib.attrByPath [ "SNOWGLOBE_CLOVER_READ_ONLY" ] "false" config.environment.variables; }; in { inherit environment; wantedBy = [ "multi-user.target" ]; - wants = [ "nomad.service" "studio-router.service" "studio-host.service" "studio-vm-screen.service" ]; - after = [ "nomad.service" "studio-host.service" "studio-vm-screen.service" ]; - unitConfig.ConditionPathExists = [ "/var/lib/studio/dashboard.token" "/var/lib/studio/ca-bundle.crt" "/opt/studio/current" ]; + wants = [ "nomad.service" "snowglobe-router.service" "snowglobe-host.service" "snowglobe-vm-screen.service" ]; + after = [ "nomad.service" "snowglobe-host.service" "snowglobe-vm-screen.service" ]; + unitConfig.ConditionPathExists = [ "/var/lib/snowglobe/dashboard.token" "/var/lib/snowglobe/ca-bundle.crt" "/opt/snowglobe/current" ]; path = [ pkgs.podman pkgs.coreutils pkgs.curl pkgs.openssl pkgs.systemd ]; serviceConfig = { Type = "notify"; NotifyAccess = "all"; - ExecStart = pkgs.writeShellScript "studio-dashboard-start" '' - uid=$(id -u studio-dashboard) - gid=$(id -g studio-dashboard) + ExecStart = pkgs.writeShellScript "snowglobe-dashboard-start" '' + uid=$(id -u snowglobe-dashboard) + gid=$(id -g snowglobe-dashboard) optional=() if test -d /srv/prod/ytdl/data && test -d '/srv/clover/Documents/Config/Youtube Downloader'; then optional+=(--volume=/srv/prod/ytdl/data:/srv/prod/ytdl/data:rw - --env=STUDIO_YT_STATE=/srv/prod/ytdl/data - '--env=STUDIO_YT_CONFIG=/srv/clover/Documents/Config/Youtube Downloader' - --env=STUDIO_YT_MEDIA=/srv/clover/Media) + --env=SNOWGLOBE_YT_STATE=/srv/prod/ytdl/data + '--env=SNOWGLOBE_YT_CONFIG=/srv/clover/Documents/Config/Youtube Downloader' + --env=SNOWGLOBE_YT_MEDIA=/srv/clover/Media) fi - exec podman run --rm --replace --name=studio-dashboard --pull=never \ + exec podman run --rm --replace --name=snowglobe-dashboard --pull=never \ --sdnotify=conmon --cgroups=no-conmon --log-driver=journald \ --user="$uid:$gid" \ --group-add=${toString config.users.groups.chloe.gid} --network=podman \ @@ -272,20 +273,20 @@ in --tmpfs=/tmp:rw,noexec,nosuid,nodev,size=256m,mode=1777 \ --tmpfs=/run:rw,noexec,nosuid,nodev,size=16m,mode=755 \ --env=HOME=/data --env=XDG_CACHE_HOME=/data/cache \ - --volume=/var/lib/studio/dashboard:/data:rw \ - --volume=/run/studio-host:/run/studio-host:ro \ - --volume=/run/studio-vm-screen:/run/studio-vm-screen:ro \ + --volume=/var/lib/snowglobe/dashboard:/data:rw \ + --volume=/run/snowglobe-host:/run/snowglobe-host:ro \ + --volume=/run/snowglobe-vm-screen:/run/snowglobe-vm-screen:ro \ --volume=${proxyToken}:/run/secrets/dashboard-proxy.token:ro \ - --volume=/var/lib/studio/dashboard.token:/run/secrets/dashboard-nomad.token:ro \ - --volume=/var/lib/studio/ca-bundle.crt:/run/secrets/ca-bundle.crt:ro \ - --mount=type=bind,src=/srv/clover,dst=/srv/clover,bind-nonrecursive,bind-propagation=rslave,ro="$STUDIO_CLOVER_READ_ONLY" \ - --mount=type=bind,src=/srv/clover/Media,dst=/srv/clover/Media,bind-nonrecursive,bind-propagation=rslave,ro="$STUDIO_MEDIA_READ_ONLY" \ + --volume=/var/lib/snowglobe/dashboard.token:/run/secrets/dashboard-nomad.token:ro \ + --volume=/var/lib/snowglobe/ca-bundle.crt:/run/secrets/ca-bundle.crt:ro \ + --mount=type=bind,src=/srv/clover,dst=/srv/clover,bind-nonrecursive,bind-propagation=rslave,ro="$SNOWGLOBE_CLOVER_READ_ONLY" \ + --mount=type=bind,src=/srv/clover/Media,dst=/srv/clover/Media,bind-nonrecursive,bind-propagation=rslave,ro="$SNOWGLOBE_MEDIA_READ_ONLY" \ ${lib.concatMapStringsSep " " (name: lib.escapeShellArg "--env=${name}") (builtins.attrNames environment)} \ - ${lib.concatMapStringsSep " " (name: lib.escapeShellArg "--add-host=${name}.${config.environment.variables.STUDIO_DOMAIN}:host-gateway") [ "dashboard.internal" "auth" "jelly" "db" "shale" ]} \ + ${lib.concatMapStringsSep " " (name: lib.escapeShellArg "--add-host=${name}.${config.environment.variables.SNOWGLOBE_DOMAIN}:host-gateway") [ "dashboard.internal" "auth" "jelly" "db" "shale" ]} \ "''${optional[@]}" ${lib.escapeShellArg "${dashboard.image.imageName}:${dashboard.image.imageTag}"} ''; - ExecStop = "${pkgs.podman}/bin/podman stop --ignore --time=8 studio-dashboard"; - ExecStopPost = "${pkgs.podman}/bin/podman rm --ignore --force studio-dashboard"; + ExecStop = "${pkgs.podman}/bin/podman stop --ignore --time=8 snowglobe-dashboard"; + ExecStopPost = "${pkgs.podman}/bin/podman rm --ignore --force snowglobe-dashboard"; CPUWeight = 1000; KillMode = "mixed"; TimeoutStartSec = 180; @@ -299,9 +300,9 @@ in openssl rand -hex 32 > ${proxyToken}.pending mv ${proxyToken}.pending ${proxyToken} fi - chown root:studio-dashboard ${proxyToken} /var/lib/studio/dashboard.token - chmod 0640 ${proxyToken} /var/lib/studio/dashboard.token - chown -R studio-dashboard:studio-dashboard /var/lib/studio/dashboard + chown root:snowglobe-dashboard ${proxyToken} /var/lib/snowglobe/dashboard.token + chmod 0640 ${proxyToken} /var/lib/snowglobe/dashboard.token + chown -R snowglobe-dashboard:snowglobe-dashboard /var/lib/snowglobe/dashboard systemd-tmpfiles --create --prefix=/srv/prod/ytdl/data if ! podman image exists ${lib.escapeShellArg "${dashboard.image.imageName}:${dashboard.image.imageTag}"}; then podman load --quiet --input ${dashboard.image} @@ -309,7 +310,7 @@ in ''; postStart = '' for attempt in $(seq 1 30); do - if printf 'header = "Studio-Proxy-Token: %s"\n' "$(cat ${proxyToken})" | curl --config - --fail --silent http://127.0.0.1:${dashboardPort}/ >/dev/null; then + if printf 'header = "Snowglobe-Proxy-Token: %s"\n' "$(cat ${proxyToken})" | curl --config - --fail --silent http://127.0.0.1:${dashboardPort}/ >/dev/null; then exit 0 fi sleep 1 @@ -325,20 +326,20 @@ in "d /srv/vm 0755 root root -" "d /srv/logs 0770 clo chloe -" "A+ /srv/prod/ytdl/data - - - - g:chloe:rwX,m::rwX,d:g:chloe:rwx,d:m::rwx" - "d /var/lib/studio 0700 root root -" - "d /var/lib/studio/dashboard 0700 studio-dashboard studio-dashboard -" - "d /var/lib/studio/routes 0700 root root -" - "d /var/lib/caddy/studio 0755 caddy caddy -" + "d /var/lib/snowglobe 0700 root root -" + "d /var/lib/snowglobe/dashboard 0700 snowglobe-dashboard snowglobe-dashboard -" + "d /var/lib/snowglobe/routes 0700 root root -" + "d /var/lib/caddy/snowglobe 0755 caddy caddy -" "f /var/lib/caddy/routes.caddy 0640 caddy caddy -" "d /var/lib/swtpm-localca 0750 tss tss -" ]; - environment.variables.STUDIO_NODE_NAME = config.networking.hostName; - environment.variables.STUDIO_DOMAIN = lib.mkDefault "studio.test"; - environment.variables.STUDIO_VM_SECURE_OVMF_CODE = secureOvmf.firmware; - environment.variables.STUDIO_VM_SECURE_OVMF_VARS = secureOvmf.variablesMs; - environment.variables.STUDIO_VM_SECURE_AAVMF_CODE = "${secureAavmf}/code.fd"; - environment.variables.STUDIO_VM_SECURE_AAVMF_VARS = "${secureAavmf}/vars.fd"; - environment.variables.STUDIO_VM_WINDOWS_ARM_EMULATOR = "${windowsArmQemu}/bin/qemu-system-aarch64"; + environment.variables.SNOWGLOBE_NODE_NAME = config.networking.hostName; + environment.variables.SNOWGLOBE_DOMAIN = lib.mkDefault "snowglobe.test"; + environment.variables.SNOWGLOBE_VM_SECURE_OVMF_CODE = secureOvmf.firmware; + environment.variables.SNOWGLOBE_VM_SECURE_OVMF_VARS = secureOvmf.variablesMs; + environment.variables.SNOWGLOBE_VM_SECURE_AAVMF_CODE = "${secureAavmf}/code.fd"; + environment.variables.SNOWGLOBE_VM_SECURE_AAVMF_VARS = "${secureAavmf}/vars.fd"; + environment.variables.SNOWGLOBE_VM_WINDOWS_ARM_EMULATOR = "${windowsArmQemu}/bin/qemu-system-aarch64"; environment.systemPackages = with pkgs; [ acl curl jq openssl python3 rsync zfs qemu ] ++ [ nativePkl ]; } diff --git a/nixos/dashboard.nix b/nixos/dashboard.nix index 1c7f7f5b244a261406c00f7acb6354a0351bce04..e9f66b3a307867ede64b3ab9c1eb68bb09a9af62 100644 --- a/nixos/dashboard.nix +++ b/nixos/dashboard.nix @@ -6,9 +6,7 @@ let root = ../.; fileset = lib.fileset.unions [ ../config - (lib.fileset.difference ../service (lib.fileset.unions - (builtins.filter builtins.pathExists (map (name: ../service + "/${name}") - (builtins.fromJSON (builtins.readFile ../config/excluded-services.json)))))) + ../service ]; }; web = stdenv.mkDerivation (finalAttrs: { @@ -49,17 +47,17 @@ let }; dashboard = runCommand "home-dashboard-0.1.0" { passthru.image = dockerTools.buildLayeredImage { - name = "studio-dashboard"; + name = "snowglobe-dashboard"; contents = [ dashboard coreutils nativePkl youtubePython yt-dlp ffmpeg dockerTools.caCertificates ]; config = { User = "65534:65534"; Cmd = [ "${dashboard}/bin/home-dashboard" ]; WorkingDir = "${dashboard}/lib/home-dashboard"; Env = [ - "STUDIO_LISTEN_ADDRESS=0.0.0.0" "PORT=7072" - "STUDIO_PROXY_TOKEN_FILE=/run/secrets/dashboard-proxy.token" - "STUDIO_REPO=${source}" - "STUDIO_YT_PYTHON=${youtubePython}/bin/python3" + "SNOWGLOBE_LISTEN_ADDRESS=0.0.0.0" "PORT=7072" + "SNOWGLOBE_PROXY_TOKEN_FILE=/run/secrets/dashboard-proxy.token" + "SNOWGLOBE_REPO=${source}" + "SNOWGLOBE_YT_PYTHON=${youtubePython}/bin/python3" "PATH=${lib.makeBinPath [ coreutils nativePkl youtubePython yt-dlp ffmpeg ]}" ]; }; diff --git a/nixos/snowglobe-network.nix b/nixos/snowglobe-network.nix new file mode 100644 index 0000000000000000000000000000000000000000..34ac0759d7065889e8058d1bdcfdef3acffbb4d4 --- /dev/null +++ b/nixos/snowglobe-network.nix @@ -0,0 +1,10 @@ +{ + networking.useDHCP = false; + networking.interfaces.enp4s0.useDHCP = false; + networking.interfaces.enp4s0.ipv4.addresses = [ + { address = "10.0.0.1"; prefixLength = 24; } + { address = "192.168.0.1"; prefixLength = 24; } + ]; + networking.defaultGateway = { address = "10.0.0.2"; interface = "enp4s0"; }; + networking.nameservers = [ "1.1.1.1" "1.0.0.1" ]; +} diff --git a/nixos/snowglobe.nix b/nixos/snowglobe.nix new file mode 100644 index 0000000000000000000000000000000000000000..679d6671b16000ed64f9ba8306ee7bec0768622c --- /dev/null +++ b/nixos/snowglobe.nix @@ -0,0 +1,102 @@ +{ config, lib, pkgs, ... }: +let + adminKey = lib.fileContents ../config/admin.pub; + pool = "globe"; + xmppNetwork = builtins.fromJSON (builtins.readFile ../service/xmpp/network.json); + xmppTCPPorts = map (name: xmppNetwork.${name}) [ "c2s" "c2s_tls" "s2s" "turn" "turn_tls" ]; + xmppUDPPorts = [ xmppNetwork.turn xmppNetwork.turn_tls ]; + encryptionRoots = { + apps = "apps"; + clover = "clover"; + media = "clover/Media"; + logs = "logs"; + prod = "prod"; + staging = "staging"; + }; +in +{ + imports = [ ./snowglobe-network.nix ]; + + networking.hostName = "snowglobe"; + environment.etc."machine-info".text = '' + PRETTY_HOSTNAME="snow globe" + ''; + networking.hostId = "4fa19ccb"; + services.nomad.settings.name = config.networking.hostName; + networking.firewall.allowedTCPPorts = [ 80 443 ] ++ xmppTCPPorts; + networking.firewall.allowedUDPPorts = xmppUDPPorts; + networking.firewall.allowedUDPPortRanges = [{ from = xmppNetwork.relay_first; to = xmppNetwork.relay_last; }]; + networking.firewall.interfaces.enp4s0.allowedTCPPorts = [ 445 ]; + + systemd.services.xmpp-nat = { + wantedBy = [ "multi-user.target" ]; + wants = [ "network-online.target" ]; + after = [ "network-online.target" ]; + startAt = "*:0/5"; + serviceConfig = { Type = "oneshot"; DynamicUser = true; }; + script = '' + ${pkgs.miniupnpc}/bin/upnpc -u http://${config.networking.defaultGateway.address}:1900/whaoc/rootDesc.xml -e "Snow Globe XMPP" -r \ + ${lib.concatMapStringsSep " " (port: "${toString port} TCP") xmppTCPPorts} \ + ${lib.concatMapStringsSep " " (port: "${toString port} UDP") (xmppUDPPorts ++ lib.range xmppNetwork.relay_first xmppNetwork.relay_last)} + ''; + }; + + boot.loader.systemd-boot.enable = true; + boot.loader.efi.canTouchEfiVariables = true; + boot.zfs.forceImportRoot = false; + boot.zfs.extraPools = [ pool ]; + boot.zfs.requestEncryptionCredentials = false; + security.tpm2.enable = true; + + systemd.services.snowglobe-zfs-unlock = { + requires = [ "zfs-import-${pool}.service" "dev-tpmrm0.device" ]; + after = [ "zfs-import-${pool}.service" "dev-tpmrm0.device" ]; + before = [ "zfs-mount.service" ]; + requiredBy = [ "zfs-mount.service" ]; + unitConfig.DefaultDependencies = false; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + LoadCredentialEncrypted = lib.mapAttrsToList + (name: _: "zfs-${name}:/etc/credstore.encrypted/zfs-${name}") encryptionRoots; + }; + script = lib.concatStrings (lib.mapAttrsToList (name: dataset: '' + if test "$(${pkgs.zfs}/bin/zfs get -H -o value keystatus ${pool}/${dataset})" = unavailable; then + ${pkgs.zfs}/bin/zfs load-key -L "file://$CREDENTIALS_DIRECTORY/zfs-${name}" ${pool}/${dataset} + fi + '') encryptionRoots); + }; + + services.xserver.videoDrivers = [ "nvidia" ]; + hardware.graphics.enable = true; + hardware.nvidia.open = false; + hardware.nvidia.nvidiaPersistenced = true; + hardware.nvidia.nvidiaSettings = false; + + systemd.services.nomad = { + requires = [ "zfs-import-${pool}.service" ]; + after = [ "zfs-import-${pool}.service" "zfs-mount.service" ]; + preStart = '' + test "$(${pkgs.util-linux}/bin/findmnt -n -o SOURCE --mountpoint /srv)" = ${pool} + test "$(${pkgs.util-linux}/bin/findmnt -n -o SOURCE --mountpoint /srv/clover)" = ${pool}/clover + test "$(${pkgs.util-linux}/bin/findmnt -n -o SOURCE --mountpoint /srv/clover/Media)" = ${pool}/clover/Media + test "$(${pkgs.util-linux}/bin/findmnt -n -o SOURCE --mountpoint /srv/prod)" = ${pool}/prod + test "$(${pkgs.util-linux}/bin/findmnt -n -o SOURCE --mountpoint /srv/staging)" = ${pool}/staging + ''; + }; + + users.groups.chloe.gid = 3000; + users.users.root.openssh.authorizedKeys.keys = [ adminKey ]; + users.users.clo = { + isNormalUser = true; + uid = 3000; + group = "chloe"; + extraGroups = [ "wheel" ]; + openssh.authorizedKeys.keys = [ adminKey ]; + }; + environment.variables.SNOWGLOBE_POOL = pool; + environment.variables.SNOWGLOBE_DOMAIN = "paperclover.net"; + environment.variables.SNOWGLOBE_LOCAL_AI = "true"; + + system.stateVersion = "26.05"; +} diff --git a/nixos/vm.nix b/nixos/vm.nix index 981071106e29434daddd95331db30c253e8bf8d8..281c1e176995704208c6221f0d5dcc92a10f91d1 100644 --- a/nixos/vm.nix +++ b/nixos/vm.nix @@ -1,16 +1,16 @@ { lib, pkgs, ... }: { - networking.hostName = "clover-demo"; + networking.hostName = "snowglobe-rehearsal"; networking.hostId = "d3c10e01"; networking.useDHCP = true; networking.dhcpcd.denyInterfaces = [ "veth*" "podman*" ]; - environment.variables.STUDIO_JAVA_OPTIONS = "-XX:TieredStopAtLevel=1"; - environment.variables.STUDIO_INDEX_POOL = "studio-demo"; - environment.variables.STUDIO_API_TIMEOUT = "600"; - environment.variables.STUDIO_MEDIA_READ_ONLY = "true"; - environment.variables.STUDIO_VM_ACCEL = "qemu"; - environment.variables.STUDIO_VM_PRESETS_ROOT = "/srv/vm/.library/presets"; - environment.variables.STUDIO_VM_UPLOADS_ROOT = "/srv/vm/.library/iso"; + environment.variables.SNOWGLOBE_JAVA_OPTIONS = "-XX:TieredStopAtLevel=1"; + environment.variables.SNOWGLOBE_INDEX_POOL = "snowglobe-demo"; + environment.variables.SNOWGLOBE_API_TIMEOUT = "600"; + environment.variables.SNOWGLOBE_MEDIA_READ_ONLY = "true"; + environment.variables.SNOWGLOBE_VM_ACCEL = "qemu"; + environment.variables.SNOWGLOBE_VM_PRESETS_ROOT = "/srv/vm/.library/presets"; + environment.variables.SNOWGLOBE_VM_UPLOADS_ROOT = "/srv/vm/.library/iso"; services.nomad.settings.client.cpu_total_compute = 16000; services.nomad.settings.client.preferred_address_family = "ipv4"; services.nomad.settings.plugin.nomad-driver-podman.config.client_http_timeout = "10m"; @@ -30,10 +30,10 @@ settings = { interface = "tailscale0"; bind-dynamic = true; - address = "/studio.test/100.95.50.31"; + address = "/snowglobe.test/100.95.50.31"; }; }; - services.caddy.virtualHosts."*.studio.test".extraConfig = '' + services.caddy.virtualHosts."*.snowglobe.test".extraConfig = '' tls internal respond 404 ''; @@ -54,35 +54,35 @@ }; services.getty.autologinUser = "root"; users.groups.chloe.gid = 3000; - systemd.services.studio-demo-pool = { + systemd.services.snowglobe-demo-pool = { wantedBy = [ "multi-user.target" ]; - before = [ "studio-media-view.service" "nomad.service" ]; + before = [ "snowglobe-media-view.service" "nomad.service" ]; serviceConfig.Type = "oneshot"; script = '' - ${pkgs.coreutils}/bin/mkdir -p /var/lib/studio /srv/clover - if ! ${pkgs.zfs}/bin/zpool list studio-demo >/dev/null 2>&1; then - if test -e /var/lib/studio/zpool.img; then - ${pkgs.zfs}/bin/zpool import -d /var/lib/studio studio-demo + ${pkgs.coreutils}/bin/mkdir -p /var/lib/snowglobe /srv/clover + if ! ${pkgs.zfs}/bin/zpool list snowglobe-demo >/dev/null 2>&1; then + if test -e /var/lib/snowglobe/zpool.img; then + ${pkgs.zfs}/bin/zpool import -d /var/lib/snowglobe snowglobe-demo else - ${pkgs.coreutils}/bin/truncate -s 16G /var/lib/studio/zpool.img - ${pkgs.zfs}/bin/zpool create -f -o ashift=12 -O mountpoint=none studio-demo /var/lib/studio/zpool.img + ${pkgs.coreutils}/bin/truncate -s 16G /var/lib/snowglobe/zpool.img + ${pkgs.zfs}/bin/zpool create -f -o ashift=12 -O mountpoint=none snowglobe-demo /var/lib/snowglobe/zpool.img fi fi - if test "$(${pkgs.coreutils}/bin/stat -c %s /var/lib/studio/zpool.img)" -lt 17179869184; then - ${pkgs.coreutils}/bin/truncate -s 16G /var/lib/studio/zpool.img - ${pkgs.zfs}/bin/zpool online -e studio-demo /var/lib/studio/zpool.img + if test "$(${pkgs.coreutils}/bin/stat -c %s /var/lib/snowglobe/zpool.img)" -lt 17179869184; then + ${pkgs.coreutils}/bin/truncate -s 16G /var/lib/snowglobe/zpool.img + ${pkgs.zfs}/bin/zpool online -e snowglobe-demo /var/lib/snowglobe/zpool.img fi - if ! ${pkgs.zfs}/bin/zfs list studio-demo/clover >/dev/null 2>&1; then - ${pkgs.zfs}/bin/zfs create -o mountpoint=/srv/clover -o acltype=posixacl studio-demo/clover + if ! ${pkgs.zfs}/bin/zfs list snowglobe-demo/clover >/dev/null 2>&1; then + ${pkgs.zfs}/bin/zfs create -o mountpoint=/srv/clover -o acltype=posixacl snowglobe-demo/clover fi - test "$(${pkgs.zfs}/bin/zfs get -H -o value mountpoint studio-demo/clover)" = /srv/clover - ${pkgs.zfs}/bin/zfs mount studio-demo/clover 2>/dev/null || - ${pkgs.util-linux}/bin/findmnt -n -o SOURCE --mountpoint /srv/clover | ${pkgs.gnugrep}/bin/grep -qx studio-demo/clover + test "$(${pkgs.zfs}/bin/zfs get -H -o value mountpoint snowglobe-demo/clover)" = /srv/clover + ${pkgs.zfs}/bin/zfs mount snowglobe-demo/clover 2>/dev/null || + ${pkgs.util-linux}/bin/findmnt -n -o SOURCE --mountpoint /srv/clover | ${pkgs.gnugrep}/bin/grep -qx snowglobe-demo/clover ${pkgs.coreutils}/bin/chown 3000:3000 /srv/clover ${pkgs.coreutils}/bin/chmod 2770 /srv/clover ''; }; - systemd.services.studio-media-source = { + systemd.services.snowglobe-media-source = { serviceConfig = { Type = "oneshot"; RemainAfterExit = true; @@ -103,9 +103,9 @@ exit 1 ''; }; - systemd.services.studio-media-view = { - after = [ "studio-demo-pool.service" "studio-media-source.service" ]; - requires = [ "studio-demo-pool.service" "studio-media-source.service" ]; + systemd.services.snowglobe-media-view = { + after = [ "snowglobe-demo-pool.service" "snowglobe-media-source.service" ]; + requires = [ "snowglobe-demo-pool.service" "snowglobe-media-source.service" ]; serviceConfig = { Type = "oneshot"; RemainAfterExit = true; @@ -116,9 +116,9 @@ }; }; systemd.services.nomad = { - wants = [ "studio-demo-pool.service" ]; - requires = [ "studio-media-view.service" ]; - after = [ "studio-demo-pool.service" "studio-media-view.service" ]; + wants = [ "snowglobe-demo-pool.service" ]; + requires = [ "snowglobe-media-view.service" ]; + after = [ "snowglobe-demo-pool.service" "snowglobe-media-view.service" ]; }; system.stateVersion = "26.05"; diff --git a/nixos/zenith-network.nix b/nixos/zenith-network.nix deleted file mode 100644 index 34ac0759d7065889e8058d1bdcfdef3acffbb4d4..0000000000000000000000000000000000000000 --- a/nixos/zenith-network.nix +++ /dev/null @@ -1,10 +0,0 @@ -{ - networking.useDHCP = false; - networking.interfaces.enp4s0.useDHCP = false; - networking.interfaces.enp4s0.ipv4.addresses = [ - { address = "10.0.0.1"; prefixLength = 24; } - { address = "192.168.0.1"; prefixLength = 24; } - ]; - networking.defaultGateway = { address = "10.0.0.2"; interface = "enp4s0"; }; - networking.nameservers = [ "1.1.1.1" "1.0.0.1" ]; -} diff --git a/nixos/zenith.nix b/nixos/zenith.nix deleted file mode 100644 index 1a56e52db9a7aacbafc78190f3a390fe23dfd8c6..0000000000000000000000000000000000000000 --- a/nixos/zenith.nix +++ /dev/null @@ -1,98 +0,0 @@ -{ config, lib, pkgs, ... }: -let - adminKey = lib.fileContents ../config/admin.pub; - pool = "globe"; - xmppNetwork = builtins.fromJSON (builtins.readFile ../service/xmpp/network.json); - xmppTCPPorts = map (name: xmppNetwork.${name}) [ "c2s" "c2s_tls" "s2s" "turn" "turn_tls" ]; - xmppUDPPorts = [ xmppNetwork.turn xmppNetwork.turn_tls ]; - encryptionRoots = { - apps = "apps"; - clover = "clover"; - media = "clover/Media"; - logs = "logs"; - prod = "prod"; - staging = "staging"; - }; -in -{ - imports = [ ./zenith-network.nix ]; - - networking.hostName = "zenith"; - networking.hostId = "4fa19ccb"; - networking.firewall.allowedTCPPorts = [ 80 443 ] ++ xmppTCPPorts; - networking.firewall.allowedUDPPorts = xmppUDPPorts; - networking.firewall.allowedUDPPortRanges = [{ from = xmppNetwork.relay_first; to = xmppNetwork.relay_last; }]; - networking.firewall.interfaces.enp4s0.allowedTCPPorts = [ 445 ]; - - systemd.services.xmpp-nat = { - wantedBy = [ "multi-user.target" ]; - wants = [ "network-online.target" ]; - after = [ "network-online.target" ]; - startAt = "*:0/5"; - serviceConfig = { Type = "oneshot"; DynamicUser = true; }; - script = '' - ${pkgs.miniupnpc}/bin/upnpc -u http://${config.networking.defaultGateway.address}:1900/whaoc/rootDesc.xml -e "Snow Globe XMPP" -r \ - ${lib.concatMapStringsSep " " (port: "${toString port} TCP") xmppTCPPorts} \ - ${lib.concatMapStringsSep " " (port: "${toString port} UDP") (xmppUDPPorts ++ lib.range xmppNetwork.relay_first xmppNetwork.relay_last)} - ''; - }; - - boot.loader.systemd-boot.enable = true; - boot.loader.efi.canTouchEfiVariables = true; - boot.zfs.forceImportRoot = false; - boot.zfs.extraPools = [ pool ]; - boot.zfs.requestEncryptionCredentials = false; - security.tpm2.enable = true; - - systemd.services.studio-zfs-unlock = { - requires = [ "zfs-import-${pool}.service" "dev-tpmrm0.device" ]; - after = [ "zfs-import-${pool}.service" "dev-tpmrm0.device" ]; - before = [ "zfs-mount.service" ]; - requiredBy = [ "zfs-mount.service" ]; - unitConfig.DefaultDependencies = false; - serviceConfig = { - Type = "oneshot"; - RemainAfterExit = true; - LoadCredentialEncrypted = lib.mapAttrsToList - (name: _: "zfs-${name}:/etc/credstore.encrypted/zfs-${name}") encryptionRoots; - }; - script = lib.concatStrings (lib.mapAttrsToList (name: dataset: '' - if test "$(${pkgs.zfs}/bin/zfs get -H -o value keystatus ${pool}/${dataset})" = unavailable; then - ${pkgs.zfs}/bin/zfs load-key -L "file://$CREDENTIALS_DIRECTORY/zfs-${name}" ${pool}/${dataset} - fi - '') encryptionRoots); - }; - - services.xserver.videoDrivers = [ "nvidia" ]; - hardware.graphics.enable = true; - hardware.nvidia.open = false; - hardware.nvidia.nvidiaPersistenced = true; - hardware.nvidia.nvidiaSettings = false; - - systemd.services.nomad = { - requires = [ "zfs-import-${pool}.service" ]; - after = [ "zfs-import-${pool}.service" "zfs-mount.service" ]; - preStart = '' - test "$(${pkgs.util-linux}/bin/findmnt -n -o SOURCE --mountpoint /srv)" = ${pool} - test "$(${pkgs.util-linux}/bin/findmnt -n -o SOURCE --mountpoint /srv/clover)" = ${pool}/clover - test "$(${pkgs.util-linux}/bin/findmnt -n -o SOURCE --mountpoint /srv/clover/Media)" = ${pool}/clover/Media - test "$(${pkgs.util-linux}/bin/findmnt -n -o SOURCE --mountpoint /srv/prod)" = ${pool}/prod - test "$(${pkgs.util-linux}/bin/findmnt -n -o SOURCE --mountpoint /srv/staging)" = ${pool}/staging - ''; - }; - - users.groups.chloe.gid = 3000; - users.users.root.openssh.authorizedKeys.keys = [ adminKey ]; - users.users.clo = { - isNormalUser = true; - uid = 3000; - group = "chloe"; - extraGroups = [ "wheel" ]; - openssh.authorizedKeys.keys = [ adminKey ]; - }; - environment.variables.STUDIO_POOL = pool; - environment.variables.STUDIO_DOMAIN = "paperclover.net"; - environment.variables.STUDIO_LOCAL_AI = "true"; - - system.stateVersion = "26.05"; -} diff --git a/readme.md b/readme.md index 84a00ad4fa7e548579c99382fe93d6e585fec63a..0c0d67161c12909e94f52e4b00d973d25cb56050 100644 --- a/readme.md +++ b/readme.md @@ -8,16 +8,36 @@ from Jellyfin and Navidrome for media consumption to git hosting and render farms. The system is built out of a NixOS base (just to install the system itself), -using managed docker containers (managed via Nomad) and virtual machines to run -all the real services. There is a lot of custom tooling to spawn interactive -testing environments. For example, without even pushing any code, I can edit the -configuration of a service or upgrade it, then spin up an isolated domain like -`shale-4c84f9da.staging.paperclover.net` to verify changes. Then, pushing to -production ensures that the new deployment is healthy before routing traffic to -it -- It's kind of like "easy kubernetes." +using Podman containers scheduled by Nomad and virtual machines to run the +services. Changes automatically get a staging deployment on Snow Globe for +testing and review. Production promotion is a separate, explicit step. [paper clover]: https://paperclover.net +## change → preview → review + +The deployment tool defaults to Snow Globe at `root@10.0.0.1:22`. + +```sh +python3 tools/deploy.py stage shale +python3 tools/deploy.py stage dashboard +``` + +These upload working files and print the preview URL. Service previews use +separate ZFS storage; the dashboard preview uses disposable accounts and files, +with read-only observations of the real host. Test the app and keep its preview +available for review. Repeating the command updates that preview. + +After approval, commit the reviewed changes on `main` and run +`python3 tools/deploy.py prod`. The release workflow backs up state, activates +the release, and checks production health. Single-writer services can have a +brief interruption. Code rollback and data restore are separate operations. + +GPU AI changes are tested directly in production. VM integration uses disposable +guests on the real machine. Most other work follows the preview workflow above; +isolated host testing is documented in [tools/rehearsal.md](tools/rehearsal.md). +The complete agent workflow is in [AGENTS.md](AGENTS.md). + ## filesystem layout The computer mounts the ZFS root dataset under `/srv`, meaning "server," loosely @@ -29,7 +49,7 @@ srv/ +--- Archive/ personal archive. one folder per year, scrambled within. +--- Asset/ resources, sample packs, audio plugins, stock videos +--- Blender/ - +--- Font/ (moved from zenith Documents/Font, will be stable index) + +--- Font/ +--- Samples/ +--- Texture/ +--- Video/ @@ -52,7 +72,8 @@ srv/ +--- clover-sandbox/ [dataset] ``` -Media is it's own dataset so it can be snapshotted independently of my personal -data (less frequent, lower retention), and all my personal files are on the same -dataset to allow fast move/copying between top level folders. Services use their -own datasets to implement copy-on-write forks. +Media is its own dataset so it can be snapshotted independently of personal +data. Personal files share a dataset to allow fast moves between top-level +folders; services use separate datasets for copy-on-write previews. No automatic +snapshot retention policy is configured. Hardware, encryption, permissions, and +recovery boundaries are documented in [tools/snowglobe-hardware.md](tools/snowglobe-hardware.md). diff --git a/service/clover-source-of-truth/service.pkl b/service/clover-source-of-truth/service.pkl index f194c11ee625a0f060d9fa7591ac8b69c0953d3e..c3d757f541f61e2130d0976cdf11815cfda9852a 100644 --- a/service/clover-source-of-truth/service.pkl +++ b/service/clover-source-of-truth/service.pkl @@ -2,7 +2,8 @@ amends "../../config/Service.pkl" import "../../config/site.pkl" as site -meta { name = "Clover DB"; launcher = false } +name = "Clover DB" +launcher = false local isTest = site.domain.endsWith(".test") @@ -31,6 +32,6 @@ container { ["CLOVER_FILE_DERIVED"] = "/data/derived" ["CLOVER_SOT_KEY"] = "${secret.own.key}" ["CLOVER_INDEX_CORES"] = "4" - ["CLOVER_INDEXER"] = if (site.pool == "studio-demo") "0" else "1" + ["CLOVER_INDEXER"] = if (site.pool == "snowglobe-demo") "0" else "1" } } diff --git a/service/copyparty/copyparty.conf b/service/copyparty/copyparty.conf index 92fae5fa12a0f642b37312304b18e3be31502405..15dcdbe7cf23b4159eada845d91fc83d77a1b9b2 100644 --- a/service/copyparty/copyparty.conf +++ b/service/copyparty/copyparty.conf @@ -3,7 +3,7 @@ p: 3923 shr: /shr hist: /cfg/hists - df: STUDIO_MIN_FREE_GB + df: SNOWGLOBE_MIN_FREE_GB ver theme: 2 name: clover's nas @@ -19,7 +19,7 @@ auth-ord: pw,idp,ipu idp-h-usr: user-name idp-h-grp: user-groups - idp-h-key: STUDIO_IDP_HEADER + idp-h-key: SNOWGLOBE_IDP_HEADER idp-login: /auth/file/sign-in?rd={dst} idp-logout: /auth/file/sign-out idp-login-t: with snow globe diff --git a/service/copyparty/manage.py b/service/copyparty/manage.py index cfb6bc741bc2c9eb093114328e5793d25d8b7f37..72a0f2fe80b431a1a31f9831bc6cae974e2b728c 100644 --- a/service/copyparty/manage.py +++ b/service/copyparty/manage.py @@ -165,7 +165,7 @@ def configuration(cfg, state, items, source=None): raise Error(503, 'Copyparty sign-in configuration is unavailable. Redeploy the service.') capacity = os.statvfs(cfg) reserve = max(1, min(16, capacity.f_blocks * capacity.f_frsize // (4 * 10**9))) - base = base.replace('STUDIO_IDP_HEADER', 'X-Studio-Idp-' + proof).replace('STUDIO_MIN_FREE_GB', str(reserve)) + base = base.replace('SNOWGLOBE_IDP_HEADER', 'X-Snowglobe-Idp-' + proof).replace('SNOWGLOBE_MIN_FREE_GB', str(reserve)) return render(base, state, items).encode() @@ -196,11 +196,11 @@ def rename(source, destination): class Manager: def __init__(self, users): - root = Path(os.environ.get('STUDIO_ROOT', '/srv')) - self.root = Path(os.environ.get('STUDIO_CLOVER_ROOT', str(root / 'clover'))).resolve() - self.cfg = Path(os.environ.get('STUDIO_COPYPARTY_CFG', str(root / 'prod/copyparty/cfg'))) + root = Path(os.environ.get('SNOWGLOBE_ROOT', '/srv')) + self.root = Path(os.environ.get('SNOWGLOBE_CLOVER_ROOT', str(root / 'clover'))).resolve() + self.cfg = Path(os.environ.get('SNOWGLOBE_COPYPARTY_CFG', str(root / 'prod/copyparty/cfg'))) self.native = self.cfg / 'copyparty' - self.service = os.environ.get('STUDIO_COPYPARTY_SERVICE', 'copyparty') + self.service = os.environ.get('SNOWGLOBE_COPYPARTY_SERVICE', 'copyparty') self.users = users self.path = self.cfg / 'manager.json' self.changed = set() @@ -280,7 +280,7 @@ class Manager: raise Error(503, 'Copyparty address is unavailable. Redeploy its service.') address = '[' + address + ']' if ':' in address else address request = urllib.request.Request('http://' + address + ':' + str(port) + '/?reload=cfg', headers={ - 'User-Name': 'snow', 'User-Groups': '', 'X-Studio-Idp-' + items['idp_header']: '1', + 'User-Name': 'snow', 'User-Groups': '', 'X-Snowglobe-Idp-' + items['idp_header']: '1', }) try: with urllib.request.urlopen(request, timeout=remaining(deadline, 30)) as response: @@ -340,7 +340,7 @@ class Manager: if virtual == 'clover' or virtual.startswith('clover/'): return self.root / virtual[len('clover'):].lstrip('/') if virtual == 'media' or virtual.startswith('media/'): - media = Path(os.environ.get('STUDIO_MEDIA_ROOT', str(self.root / 'Media'))) + media = Path(os.environ.get('SNOWGLOBE_MEDIA_ROOT', str(self.root / 'Media'))) return media / virtual[len('media'):].lstrip('/') return None @@ -573,9 +573,9 @@ class Manager: def apply(self, action, payload): self.data = json.loads(runs.read(self.path, missing='{"accounts":[],"grants":[],"names":{}}')) - origin = Path(self.data.get('root', os.environ.get('STUDIO_SOURCE_CLOVER_ROOT', '/srv/clover'))) + origin = Path(self.data.get('root', os.environ.get('SNOWGLOBE_SOURCE_CLOVER_ROOT', '/srv/clover'))) if 'root' not in self.data and self.data.get('grants'): - origin = Path(os.environ.get('STUDIO_SOURCE_CLOVER_ROOT', str(self.root))) + origin = Path(os.environ.get('SNOWGLOBE_SOURCE_CLOVER_ROOT', str(self.root))) if origin != self.root: for grant in self.data['grants']: grant['path'] = str(self.root / Path(grant['path']).relative_to(origin)) diff --git a/service/copyparty/prepare.py b/service/copyparty/prepare.py index 7b32fcd84147c94adf934cdf0c5650c0abd36ee4..8ca419eea55204de75d7c3714dd2db07a58bc5c2 100644 --- a/service/copyparty/prepare.py +++ b/service/copyparty/prepare.py @@ -20,7 +20,7 @@ variable = json.loads(subprocess.check_output( items = variable['Items'] state_path = cfg / 'manager.json' state = json.loads(state_path.read_text()) if state_path.exists() else {'accounts': [], 'grants': [], 'names': {}} -canonical_root = Path(os.environ.get('STUDIO_CLOVER_ROOT', str(Path(os.environ.get('STUDIO_ROOT', '/srv')) / 'clover'))) +canonical_root = Path(os.environ.get('SNOWGLOBE_CLOVER_ROOT', str(Path(os.environ.get('SNOWGLOBE_ROOT', '/srv')) / 'clover'))) origin = Path(state.get('root', str(canonical_root))) if origin != canonical_root: for grant in state['grants']: diff --git a/service/copyparty/service.pkl b/service/copyparty/service.pkl index f175865200af4f065287a3bd0e7d206bc038901f..069dcdb85bb96585a52677deabdd60f203ddd559 100644 --- a/service/copyparty/service.pkl +++ b/service/copyparty/service.pkl @@ -2,8 +2,7 @@ amends "../../config/Service.pkl" import "../../config/site.pkl" as site -meta { name = "Copyparty" } -rollout = "simple" +name = "Copyparty" prepare = "prepare.py" secrets { @@ -12,8 +11,6 @@ secrets { container { image = "docker.io/copyparty/ac@sha256:9658d60c7fec0bffb968cacc677712be3c8a6e73bb388b6b47b7f2b178a754a6" - cpu = 200 - memory = 512 http { containerPort = 3923 diff --git a/service/dawarich/redis-insight.pkl b/service/dawarich/redis-insight.pkl index 60487a4648c7dd06d182e705b72d6ea20ab88241..1d5f3b124c43a1d5cac71727c76f611f13e12b36 100644 --- a/service/dawarich/redis-insight.pkl +++ b/service/dawarich/redis-insight.pkl @@ -1,6 +1,6 @@ amends "../../config/Service.pkl" -meta { name = "Redis Insight" } +name = "Redis Insight" dependsOn { "dawarich" } secrets { @@ -10,7 +10,6 @@ secrets { container { image = "docker.io/redis/redisinsight@sha256:0131987fd5fefe3828ed33666f9742bd071b729f5a95cf923c593c14a8995c2c" cpu = 150 - memory = 512 http { containerPort = 5540 diff --git a/service/dawarich/service.pkl b/service/dawarich/service.pkl index bf1b3bdf9096ad365047bb04f2633772608ba8d3..c2d9fdf54d4a0bcffcf10c8b0221ababfe265d06 100644 --- a/service/dawarich/service.pkl +++ b/service/dawarich/service.pkl @@ -32,7 +32,7 @@ local sharedVolumes: Mapping = new { ["/var/app/storage"] {} ["/var/app/tmp/imports/watched"] {} ["/etc/ssl/certs/ca-certificates.crt"] { - src = "/var/lib/studio/ca-bundle.crt" + src = "/var/lib/snowglobe/ca-bundle.crt" readOnly = true } } @@ -43,14 +43,15 @@ local databaseEnv = """ {{ range nomadService 1 (env "NOMAD_ALLOC_ID") "\(module.id)-redis" }}REDIS_URL=redis://\(module.nomadHostPort){{ end }} """ -meta { name = "Dawarich"; access = "infra-admin" } +name = "Dawarich" +access = "infra-admin" healthyDeadline = "20m" requirements { database new sso.Client { clientId = module.id - name = module.meta.name + name = module.name redirectUris { "https://\(module.containers["web"].http.hostname)/users/auth/openid_connect/callback" } } } @@ -66,7 +67,6 @@ containers { args { "ruby"; "-e"; "exit 0" } imageUser = true lifecycle = "prestart" - cpu = 200 memory = 1024 volumes = sharedVolumes env = commonEnv @@ -75,7 +75,6 @@ containers { ["web"] { image = dawarichImage - extraHosts { "\(sso.hostname):host-gateway" } entrypoint = "web-entrypoint.sh" args { "bin/rails"; "server"; "-p"; "3000"; "-b"; "::" } imageUser = true @@ -107,7 +106,6 @@ containers { entrypoint = "sidekiq-entrypoint.sh" args { "sidekiq" } imageUser = true - cpu = 200 memory = 1024 volumes = sharedVolumes env = (commonEnv) { diff --git a/service/ddns-updater/service.pkl b/service/ddns-updater/service.pkl index d87e5abeb670e37a7059fb7e355c57c819e846e3..66d40524c7ef822b580a341a7b9cf269ddf591e0 100644 --- a/service/ddns-updater/service.pkl +++ b/service/ddns-updater/service.pkl @@ -16,7 +16,7 @@ class DnsCredentials extends service.Requirement { srv: Listing = new {} } -meta { name = "DDNS" } +name = "DDNS" provide = "provide.py" enabled = !site.preview && !site.domain.endsWith(".test") diff --git a/service/evil-forgejo/api.py b/service/evil-forgejo/api.py deleted file mode 100644 index d2cb132a9f8f0ab951b6f984ae2f9e2c7e7f6d8e..0000000000000000000000000000000000000000 --- a/service/evil-forgejo/api.py +++ /dev/null @@ -1,33 +0,0 @@ -import json -import os -import subprocess -import urllib.request - - -def nomad(path): - request = urllib.request.Request( - "http://127.0.0.1:4646/v1/" + path, - headers={"X-Nomad-Token": os.environ["NOMAD_TOKEN"]}, - ) - with urllib.request.urlopen(request, timeout=5) as response: - return json.load(response) - - -def instance(job="evil-forgejo"): - running = [item["ID"] for item in nomad(f"job/{job}/allocations") - if item["ClientStatus"] == "running" and item["DesiredStatus"] == "run"] - if len(running) != 1: - raise ValueError(f"{job} needs one running allocation") - allocation = running[0] - services = [item for item in nomad(f"service/{job}-forgejo") if item["AllocID"] == allocation] - if len(services) != 1: - raise ValueError(f"{job} has no Forgejo listener") - result = subprocess.run( - ["podman", "--url", "unix:///run/podman/podman.sock", "ps", "--format", "{{.ID}} {{.Names}}"], - check=True, capture_output=True, text=True, - ) - containers = [parts[0] for line in result.stdout.splitlines() - if len(parts := line.split()) == 2 and parts[1] == "forgejo-" + allocation] - if len(containers) != 1: - raise ValueError(f"{job} Forgejo task is unavailable") - return containers[0], f"http://{services[0]['Address']}:{services[0]['Port']}" diff --git a/service/evil-forgejo/app.ini b/service/evil-forgejo/app.ini deleted file mode 100755 index ddb47dfbef93f127e67d82eefa0310b7dab69ced..0000000000000000000000000000000000000000 --- a/service/evil-forgejo/app.ini +++ /dev/null @@ -1,115 +0,0 @@ -APP_NAME="evil inc" -APP_SLOGAL="be evil. take control back." -RUN_MODE=prod -RUN_USER=git -WORK_PATH=/data/work - -[repository] -MAX_CREATION_LIMIT=20000 -ROOT=/data/git/repositories -DEFAULT_CLOSE_ISSUES_VIA_COMMITS_IN_ANY_BRANCH=true -DEFAULT_REPO_UNITS=repo.code,repo.issues,repo.pulls -DISABLE_STARS=true -DEFAULT_BRANCH=master - -[repository.local] -LOCAL_COPY_PATH=/data/work/tmp/local-repo - -[repository.upload] -TEMP_PATH=/data/work/uploads - -[ui] -EXPLORE_PAGING_NUM=100 -ISSUE_PAGING_NUM=500 -MEMBERS_PAGING_NUM=100 -THEMES=evil,evil-dark,evil-light -DEFAULT_THEME=evil - -[server] -APP_DATA_PATH=/data/work -DOMAIN=$GIT_HOST -SSH_DOMAIN=$GIT_HOST -HTTP_ADDR=0.0.0.0 -HTTP_PORT=$FORGEJO_HTTP_PORT -ROOT_URL=https://$GIT_HOST/ -LFS_JWT_SECRET=$FORGEJO_SERVER_LFS_JWT_SECRET -LFS_START_SERVER=true -OFFLINE_MODE=true -LANDING_PAGE=/evil -START_SSH_SERVER=false -SSH_CREATE_AUTHORIZED_KEYS_FILE=false - -[database] -DB_TYPE=postgres -HOST=$DB_HOST -NAME=$DB_NAME -USER=$DB_USER -PASSWD=$DB_PASSWORD -LOG_SQL=false -SCHEMA= -SSL_MODE=disable - -[indexer] -ISSUE_INDEXER_PATH=/data/work/indexers/issues.bleve - -[session] -PROVIDER_CONFIG=/data/work/sessions -PROVIDER=file - -[picture] -AVATAR_UPLOAD_PATH=/data/work/avatars -REPOSITORY_AVATAR_UPLOAD_PATH=/data/work/repo-avatars - -[admin] -DISABLE_REGULAR_ORG_CREATION=true - -[attachment] -PATH=/data/work/attachments - -[log] -MODE=console -LEVEL=info -ROOT_PATH=/data/work/log - -[security] -INSTALL_LOCK=true -REVERSE_PROXY_LIMIT=1 -REVERSE_PROXY_TRUSTED_PROXIES=* -PASSWORD_HASH_ALGO=pbkdf2_hi -SECRET_KEY=$FORGEJO_SECURITY_SECRET_KEY -INTERNAL_TOKEN=$FORGEJO_SECURITY_INTERNAL_TOKEN - -[service] -ENABLE_INTERNAL_SIGNIN=true -DISABLE_REGISTRATION=false -ALLOW_ONLY_EXTERNAL_REGISTRATION=true -SHOW_REGISTRATION_BUTTON=false -REQUIRE_SIGNIN_VIEW=false -REGISTER_EMAIL_CONFIRM=true -ENABLE_NOTIFY_MAIL=true -DEFAULT_KEEP_EMAIL_PRIVATE=false -DEFAULT_ALLOW_CREATE_ORGANIZATION=true -DEFAULT_ENABLE_TIMETRACKING=false - -[lfs] -PATH = /data/git/lfs - -[mailer] -ENABLED=$MAILER_ENABLED -; TODO: get evil mailing credentials. we likely will not need this -; for a while and can just use clover's email. its not that deep -FROM=noreply.evil-git@paperclover.net -PROTOCOL=smtps -SMTP_PORT=465 -SMTP_ADDR=$MAILER_ADDRESS -USER=$MAILER_USERNAME -PASSWD=$MAILER_PASSWORD - -[repository.pull-request] -DEFAULT_MERGE_STYLE=squash - -[repository.signing] -DEFAULT_TRUST_MODEL=committer - -[oauth2] -JWT_SECRET=$FORGEJO_OAUTH2_JWT_SECRET diff --git a/service/evil-forgejo/icon.svg b/service/evil-forgejo/icon.svg deleted file mode 100644 index c07b7c088f0f5d15f1a9762e1d54bbe6fd647548..0000000000000000000000000000000000000000 --- a/service/evil-forgejo/icon.svg +++ /dev/null @@ -1 +0,0 @@ -Forgejo diff --git a/service/evil-forgejo/provide.py b/service/evil-forgejo/provide.py deleted file mode 100644 index e09338d87db068f24b9fe57b7343af2b07c9adda..0000000000000000000000000000000000000000 --- a/service/evil-forgejo/provide.py +++ /dev/null @@ -1,67 +0,0 @@ -#!/usr/bin/env python3 -import json -import sys -import urllib.error -import urllib.request - -from api import instance - - -data = json.load(sys.stdin) -request = data["request"] -if request["kind"] != "oauth-client": - raise ValueError("unsupported Forgejo input") -stage = data.get("stageId") -existing = data.get("existing") or {} -if data.get("operation") == "delete" and not stage: - raise ValueError("refusing to delete a production OAuth client") -_, address = instance() -authorization = "token " + data["providerSecrets"]["automation_token"] - - -def api(path, method="GET", body=None, missing_ok=False): - query = urllib.request.Request( - address + "/api/v1" + path, - data=json.dumps(body).encode() if body is not None else None, - method=method, - headers={"Authorization": authorization, "Content-Type": "application/json"}, - ) - try: - with urllib.request.urlopen(query, timeout=10) as response: - return json.load(response) if response.status != 204 else None - except urllib.error.HTTPError as error: - if error.code == 404 and (method == "DELETE" or missing_ok): - return None - raise RuntimeError(f"Forgejo OAuth API returned HTTP {error.code}") from error - - -application_id = existing.get("applicationId") -if data.get("operation") == "delete": - if application_id: - api("/user/applications/oauth2/" + application_id, "DELETE") - sys.exit(0) - -suffix = request["name"] + (":" + stage if stage else "") -name = "Snow Globe: " + suffix -desired = {"name": name, "redirect_uris": request["redirectUris"], "confidential_client": True} -if application_id: - current = api("/user/applications/oauth2/" + application_id, missing_ok=True) - if current and current["client_id"] == existing.get("clientId") and existing.get("clientSecret"): - if any(current[key] != value for key, value in desired.items()): - api("/user/applications/oauth2/" + application_id, "PATCH", desired) - client_id = existing["clientId"] - client_secret = existing["clientSecret"] - else: - application_id = None -if not application_id: - applications = api("/user/applications/oauth2?limit=100") - if any(item["name"] in (name, "studio:" + suffix) for item in applications): - raise ValueError("Forgejo OAuth client exists without its stored secret") - created = api("/user/applications/oauth2", "POST", desired) - application_id = str(created["id"]) - client_id = created["client_id"] - client_secret = created["client_secret"] -if not client_id or not client_secret: - raise ValueError("Forgejo OAuth client has no credentials") -print(json.dumps({"applicationId": application_id, "clientId": client_id, - "clientSecret": client_secret, "providerUrl": "https://" + data["host"]})) diff --git a/service/evil-forgejo/public/assets/css/theme-evil-dark.css b/service/evil-forgejo/public/assets/css/theme-evil-dark.css deleted file mode 100644 index 6fe13ae4b74134363318cf84bab8daf1f62d52a4..0000000000000000000000000000000000000000 --- a/service/evil-forgejo/public/assets/css/theme-evil-dark.css +++ /dev/null @@ -1,118 +0,0 @@ -@import "theme-forgejo-dark.css"; - -:root { - --f: 0.75; - --l: 4; - --steel-900: lch(from #10161d l calc(c * var(--f)) 320); - --steel-850: lch(from #131a21 l calc(c * var(--f)) 320); - --steel-800: lch(from #171e26 l calc(c * var(--f)) 320); - --steel-750: lch(from #1d262f l calc(c * var(--f)) 320); - --steel-700: lch(from #242d38 l calc(c * var(--f)) 320); - --steel-650: lch(from #2b3642 l calc(c * var(--f)) 320); - --steel-600: lch(from #374351 l calc(c * var(--f)) 320); - --steel-550: lch(from #445161 l calc(c * var(--f)) 320); - --steel-500: lch(from #515f70 l calc(c * var(--f)) 320); - --steel-450: lch(from #5f6e80 l calc(c * var(--f)) 320); - --steel-400: lch(from #6d7d8f l calc(c * var(--f)) 320); - --steel-350: lch(from #7c8c9f l calc(c * var(--f)) 320); - --steel-300: lch(from #8c9caf l calc(c * var(--f)) 320); - --steel-250: lch(from #9dadc0 l calc(c * var(--f)) 320); - --steel-200: lch(from #aebed0 l calc(c * var(--f)) 320); - --steel-150: lch(from #c0cfe0 l calc(c * var(--f)) 320); - --steel-100: lch(from #d2e0f0 l calc(c * var(--f)) 320); - --zinc-50: lch(from #fafafa l calc(c * var(--f)) 320); - --zinc-100: lch(from #f4f4f5 l calc(c * var(--f)) 320); - --zinc-150: lch(from #ececee l calc(c * var(--f)) 320); - --zinc-200: lch(from #e4e4e7 l calc(c * var(--f)) 320); - --zinc-250: lch(from #dcdce0 l calc(c * var(--f)) 320); - --zinc-300: lch(from #d4d4d8 l calc(c * var(--f)) 320); - --zinc-350: lch(from #babac1 l calc(c * var(--f)) 320); - --zinc-400: lch(from #a1a1aa l calc(c * var(--f)) 320); - --zinc-450: lch(from #898992 l calc(c * var(--f)) 320); - --zinc-500: lch(from #71717a l calc(c * var(--f)) 320); - --zinc-550: lch(from #61616a l calc(c * var(--f)) 320); - --zinc-600: lch(from #52525b l calc(c * var(--f)) 320); - --zinc-650: lch(from #484850 l calc(c * var(--f)) 320); - --zinc-700: lch(from #3f3f46 l calc(c * var(--f)) 320); - --zinc-750: lch(from #333338 l calc(c * var(--f)) 320); - --zinc-800: lch(from #27272a l calc(c * var(--f)) 320); - --zinc-850: lch(from #1f1f23 l calc(c * var(--f)) 320); - --zinc-900: lch(from #18181b l calc(c * var(--f)) 320); - --color-primary: #e286f3; - --color-primary-contrast: lch(from var(--color-primary) 95% 0 h); - --color-primary-dark-1: lch(from var(--color-primary) l c h); - --color-primary-dark-2: lch(from var(--color-primary) calc(l * 0.85) c h); - --color-primary-dark-3: lch(from var(--color-primary) calc(l * 0.85) c h); - --color-primary-dark-4: lch(from var(--color-primary) calc(l * 0.72) c h); - --color-primary-dark-5: lch(from var(--color-primary) calc(l * 0.72) c h); - --color-primary-dark-6: lch(from var(--color-primary) calc(l * 0.72) c h); - --color-primary-dark-7: lch(from var(--color-primary) calc(l * 0.72) c h); - --color-primary-light-1: lch(from var(--color-primary) calc(l * 1.15) c h); - --color-primary-light-2: lch(from var(--color-primary) calc(l * 1.28) c h); - --color-primary-light-3: lch(from var(--color-primary) calc(l * 1.48) c h); - --color-primary-light-4: lch(from var(--color-primary) calc(l * 1.68) c h); - --color-primary-light-5: lch( - from - var(--color-primary) - calc(l * 1.82) - calc(c * 0.6) - h - ); - --color-primary-light-6: lch( - from - var(--color-primary) - calc(l * 1.92) - calc(c * 0.3) - h - ); - --color-primary-light-7: lch( - from - var(--color-primary) - calc(l * 1.98) - calc(c * 0.15) - h - ); - --color-primary-alpha-10: lch(from var(--color-primary) l c h / 0.1); - --color-primary-alpha-20: lch(from var(--color-primary) l c h / 0.2); - --color-primary-alpha-30: lch(from var(--color-primary) l c h / 0.3); - --color-primary-alpha-40: lch(from var(--color-primary) l c h / 0.4); - --color-primary-alpha-50: lch(from var(--color-primary) l c h / 0.5); - --color-primary-alpha-60: lch(from var(--color-primary) l c h / 0.6); - --color-primary-alpha-70: lch(from var(--color-primary) l c h / 0.7); - --color-primary-alpha-80: lch(from var(--color-primary) l c h / 0.8); - --color-primary-alpha-90: lch(from var(--color-primary) l c h / 0.9); - --color-active: var(--steel-600); - --color-secondary-alpha-10: lch(from var(--color-secondary) l c h / 0.1); - --color-secondary-alpha-20: lch(from var(--color-secondary) l c h / 0.2); - --color-secondary-alpha-30: lch(from var(--color-secondary) l c h / 0.3); - --color-secondary-alpha-40: lch(from var(--color-secondary) l c h / 0.4); - --color-secondary-alpha-50: lch(from var(--color-secondary) l c h / 0.5); - --color-secondary-alpha-60: lch(from var(--color-secondary) l c h / 0.6); - --color-secondary-alpha-70: lch(from var(--color-secondary) l c h / 0.7); - --color-secondary-alpha-80: lch(from var(--color-secondary) l c h / 0.8); - --color-secondary-alpha-90: lch(from var(--color-secondary) l c h / 0.9); -} - -.organization.profile:has(.overflow-menu-items > a[href="/evil"]) { - .org-avatar, .org-title { - clip: rect(1px, 1px, 1px, 1px); - clip-path: inset(50%); - height: 1px; - width: 1px; - margin: -1px; - overflow: hidden; - padding: 0; - position: absolute; - } - #org-info .render-content p { - display: none; - } - .org-header::before { - content: " "; - height: 60px; - width: 215px; - margin-left: 10px; - margin-bottom: 10px; - background: url(data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIGZpbGw9Im5vbmUiIHZpZXdCb3g9IjAgMCAxNDUzIDQwNiI+PHBhdGggZD0iTTQ2IDQ2YzIgMTctNyAxMzUtMiAxNDBzMTI1LTQgMTQyLTJjMjYgNSAyNiA0MS0xIDQ1LTE3IDItMTM2LTgtMTQxLTMtNCA0IDUgMTA1IDMgMTE5IDE0LTQgMjktNiA0My03IDIzLTEgODgtNCAxMDggMCAyMiA0IDI0IDM5LTEgNDQtMzAgNi03NCAwLTEwNyAzLTI3IDItODYgMjYtODgtMTgtMi00MiAxNC05OSA2LTE0MiAwLTQtNi03LTctMTItNC0xOSA1LTE3IDctMjYgOC0zOC0xMS0xNDQgMi0xNzFDMTQgOSAyMCA1IDI4IDRjMTktMyAxMjQtNSAxNDEtMSAyMyA1IDE4IDQzLTMgNDNINDZabTEwMzYgMTc2Yy0xNS0xMC0zMyAzNC0zNyA0NC01IDE0LTEwIDMxLTEzIDQ2cy02IDU3LTExIDY3Yy0xMCAyMC00MSAxNC00My0xMi0xLTE3IDAtMTQxIDQtMTUxIDQtMTMgMjItMTUgMzMtOGw3IDljMSAwIDEyLTE5IDE2LTIyIDQ2LTUxIDc4IDMgMTAzIDQwIDE5IDI4IDc4IDExNCA4OCAxNDAgNyAyMS0xMCAzOC0yOSAyOS05LTUtMzctNjAtNDUtNzNsLTczLTEwOVptMzI2LTEwMWMyNS0xIDMyIDMwIDEwIDM5LTM0IDE0LTU4IDgtOTEgMzgtNjggNjAtMTIgMTIzIDU0IDE0NiAxOCA2IDY3IDkgNzEgMjcgMTAgNDUtODUgMTItMTAzIDUtMTUyLTY0LTExNS0yMDkgMjktMjQ4IDctMiAyMy03IDMwLTdaTTYzNiA1YzcgNyA1IDQyIDQgNTMgMCA3OC04IDE1Ni02IDIzNCAwIDI0IDkgNjYgNSA4Ni0zIDE5LTMxIDIxLTM0LTEtNC0yNiA1LTcwIDUtMTAwIDEtNzgtMTQtMTg0LTUtMjU4IDItMTYgMTktMjYgMzEtMTRaTTMxMyAzMTBjMjEtMzIgMzQtNzAgNTktOTkgMTAtMTEgMzItMzYgNDgtMjQgMjQgMTktMTMgMzgtMjMgNDgtMjAgMjMtMzkgNTctNTAgODYtNSAxMS0xNyA1Ni0yMyA2MS0xMyAxMC0yNiAzLTMyLTExLTktMTctNDQtMTIwLTQ3LTEzNy0zLTI1IDIzLTM2IDM2LTE5czIwIDc0IDMyIDk1Wm0xNzAtMTEzYzEyLTEyIDMyLTQgMzQgMTIgNSAyOC0xIDg0LTEgMTE3IDAgMTQgNyA1Mi00IDYyLTggNy0yMiA3LTI5LTItMTEtMTEtMy02OC0zLTg1IDAtMjctNi03MC0xLTk1IDAtMyAyLTYgNC05Wm0xNS01MmEyMSAyMSAwIDEgMCAwLTQyIDIxIDIxIDAgMCAwIDAgNDJaIiBmaWxsPSIjZTZkYmU3Ii8+PHBhdGggZmlsbD0iI0UxODZGNCIgZD0iTTk1MSA0MGMyMyAyMS0xIDUzLTE2IDcwLTEzIDE1LTQ1IDUyLTY0IDUxLTYgMC00Ni0yMi01My0yNy0zOS0yNy0zNS0xMDIgMjctODkgOCAyIDI4IDE1IDMwIDE1IDMtMSAxMi0xNSAxNy0xOCAxNC0xMSA0NS0xNSA1OS0yWiIvPjxwYXRoIGQ9Ik04NjAgMzg3Yy0zLTQtNS05LTYtMTQtMy0yMi0zLTE1NSAwLTE3NiA0LTI0IDM3LTI1IDQwIDIgMyAyMCAzIDE1NiAwIDE3NS0yIDE3LTIyIDI0LTM0IDEzWiIgZmlsbD0iI2U2ZGJlNyIvPjwvc3ZnPg); - } -} diff --git a/service/evil-forgejo/public/assets/css/theme-evil-light.css b/service/evil-forgejo/public/assets/css/theme-evil-light.css deleted file mode 100644 index 28d31a947d52701a1e4df9c495ce3a9a22dbc381..0000000000000000000000000000000000000000 --- a/service/evil-forgejo/public/assets/css/theme-evil-light.css +++ /dev/null @@ -1,175 +0,0 @@ -@import "theme-forgejo-light.css"; - -:root { - --f: 5; - --l: 4; - --steel-900: lch(from #10161d l calc(c * var(--f)) 320); - --steel-850: lch(from #131a21 l calc(c * var(--f)) 320); - --steel-800: lch(from #171e26 l calc(c * var(--f)) 320); - --steel-750: lch(from #1d262f l calc(c * var(--f)) 320); - --steel-700: lch(from #242d38 l calc(c * var(--f)) 320); - --steel-650: lch(from #2b3642 l calc(c * var(--f)) 320); - --steel-600: lch(from #374351 l calc(c * var(--f)) 320); - --steel-550: lch(from #445161 l calc(c * var(--f)) 320); - --steel-500: lch(from #515f70 l calc(c * var(--f)) 320); - --steel-450: lch(from #5f6e80 l calc(c * var(--f)) 320); - --steel-400: lch(from #6d7d8f l calc(c * var(--f)) 320); - --steel-350: lch(from #7c8c9f l calc(c * var(--f)) 320); - --steel-300: lch(from #8c9caf l calc(c * var(--f)) 320); - --steel-250: lch(from #9dadc0 l calc(c * var(--f)) 320); - --steel-200: lch(from #aebed0 l calc(c * var(--f)) 320); - --steel-150: lch(from #c0cfe0 l calc(c * var(--f)) 320); - --steel-100: lch(from #d2e0f0 l calc(c * var(--f)) 320); - --zinc-50: lch(from #fafafa l calc(c * var(--f)) 320); - --zinc-100: lch(from #f4f4f5 l calc(c * var(--f)) 320); - --zinc-150: lch(from #ececee l calc(c * var(--f)) 320); - --zinc-200: lch(from #e4e4e7 l calc(c * var(--f)) 320); - --zinc-250: lch(from #dcdce0 l calc(c * var(--f)) 320); - --zinc-300: lch(from #d4d4d8 l calc(c * var(--f)) 320); - --zinc-350: lch(from #babac1 l calc(c * var(--f)) 320); - --zinc-400: lch(from #a1a1aa l calc(c * var(--f)) 320); - --zinc-450: lch(from #898992 l calc(c * var(--f)) 320); - --zinc-500: lch(from #71717a l calc(c * var(--f)) 320); - --zinc-550: lch(from #61616a l calc(c * var(--f)) 320); - --zinc-600: lch(from #52525b l calc(c * var(--f)) 320); - --zinc-650: lch(from #484850 l calc(c * var(--f)) 320); - --zinc-700: lch(from #3f3f46 l calc(c * var(--f)) 320); - --zinc-750: lch(from #333338 l calc(c * var(--f)) 320); - --zinc-800: lch(from #27272a l calc(c * var(--f)) 320); - --zinc-850: lch(from #1f1f23 l calc(c * var(--f)) 320); - --zinc-900: lch(from #18181b l calc(c * var(--f)) 320); - --color-primary: #b734cf; - --color-primary-contrast: lch(from var(--color-primary) 95% 0 h); - --color-primary-dark-1: lch(from var(--color-primary) l c h); - --color-primary-dark-2: lch(from var(--color-primary) calc(l * 0.85) c h); - --color-primary-dark-3: lch(from var(--color-primary) calc(l * 0.85) c h); - --color-primary-dark-4: lch(from var(--color-primary) calc(l * 0.72) c h); - --color-primary-dark-5: lch(from var(--color-primary) calc(l * 0.72) c h); - --color-primary-dark-6: lch(from var(--color-primary) calc(l * 0.72) c h); - --color-primary-dark-7: lch(from var(--color-primary) calc(l * 0.72) c h); - --color-primary-light-1: lch(from var(--color-primary) calc(l * 1.15) c h); - --color-primary-light-2: lch(from var(--color-primary) calc(l * 1.28) c h); - --color-primary-light-3: lch(from var(--color-primary) calc(l * 1.48) c h); - --color-primary-light-4: lch(from var(--color-primary) calc(l * 1.68) c h); - --color-primary-light-5: lch( - from - var(--color-primary) - calc(l * 1.82) - calc(c * 0.6) - h - ); - --color-primary-light-6: lch( - from - var(--color-primary) - calc(l * 1.92) - calc(c * 0.3) - h - ); - --color-primary-light-7: lch( - from - var(--color-primary) - calc(l * 1.98) - calc(c * 0.15) - h - ); - --color-primary-alpha-10: lch(from var(--color-primary) l c h / 0.1); - --color-primary-alpha-20: lch(from var(--color-primary) l c h / 0.2); - --color-primary-alpha-30: lch(from var(--color-primary) l c h / 0.3); - --color-primary-alpha-40: lch(from var(--color-primary) l c h / 0.4); - --color-primary-alpha-50: lch(from var(--color-primary) l c h / 0.5); - --color-primary-alpha-60: lch(from var(--color-primary) l c h / 0.6); - --color-primary-alpha-70: lch(from var(--color-primary) l c h / 0.7); - --color-primary-alpha-80: lch(from var(--color-primary) l c h / 0.8); - --color-primary-alpha-90: lch(from var(--color-primary) l c h / 0.9); - --color-active: var(--steel-600); - --color-secondary-alpha-10: lch(from var(--color-secondary) l c h / 0.1); - --color-secondary-alpha-20: lch(from var(--color-secondary) l c h / 0.2); - --color-secondary-alpha-30: lch(from var(--color-secondary) l c h / 0.3); - --color-secondary-alpha-40: lch(from var(--color-secondary) l c h / 0.4); - --color-secondary-alpha-50: lch(from var(--color-secondary) l c h / 0.5); - --color-secondary-alpha-60: lch(from var(--color-secondary) l c h / 0.6); - --color-secondary-alpha-70: lch(from var(--color-secondary) l c h / 0.7); - --color-secondary-alpha-80: lch(from var(--color-secondary) l c h / 0.8); - --color-secondary-alpha-90: lch(from var(--color-secondary) l c h / 0.9); - --color-body: lch(from #fff l calc(c * var(--f) + var(--l)) h); - --color-text-dark: lch(from #000 l calc(c * var(--f) + var(--l)) h); - --color-input-background: lch(from #fff l calc(c * var(--f) + var(--l)) h); - --color-input-toggle-background: lch( - from - #fff - l - calc(c * var(--f) + var(--l)) - h - ); - --color-header-wrapper-transparent: lch( - from - #d2e0f000 - l - calc(c * var(--f) + var(--l)) - h - ); - --color-light-border: lch(from #0000001d l calc(c * var(--f) + var(--l)) h); - --color-hover: lch(from #e4e4e4aa l calc(c * var(--f) + var(--l)) h); - --color-active: lch(from #e2e2e5 l calc(c * var(--f) + var(--l)) 320); - --color-markup-table-row: lch( - from - #ffffff06 - l - calc(c * var(--f) + var(--l)) - h - ); - --color-shadow: lch(from #00000060 l calc(c * var(--f) + var(--l)) h); - --color-text-focus: lch(from #fff l calc(c * var(--f) + var(--l)) h); - --color-reaction-bg: lch(from #0000000a l calc(c * var(--f) + var(--l)) h); - --color-tooltip-text: lch(from #ffffff l calc(c * var(--f) + var(--l)) h); - --color-tooltip-bg: lch(from #000000f0 l calc(c * var(--f) + var(--l)) h); - --color-label-bg: lch(from #cacaca7b l calc(c * var(--f) + var(--l)) h); - --color-label-hover-bg: lch( - from - #cacacaa0 - l - calc(c * var(--f) + var(--l)) - h - ); - --color-label-active-bg: lch( - from - #cacacaff - l - calc(c * var(--f) + var(--l)) - h - ); - --color-label-bg-alt: lch(from #cacacaff l calc(c * var(--f) + var(--l)) h); - --color-overlay-backdrop: lch( - from - #080808c0 - l - calc(c * var(--f) + var(--l)) - h - ); - --checkerboard-color-1: lch(from #ffffff l calc(c * var(--f) + var(--l)) h); - --checkerboard-color-2: lch(from #e5e5e5 l calc(c * var(--f) + var(--l)) h); -} - -.organization.profile:has(.overflow-menu-items > a[href="/evil"]) { - .org-avatar, .org-title { - clip: rect(1px, 1px, 1px, 1px); - clip-path: inset(50%); - height: 1px; - width: 1px; - margin: -1px; - overflow: hidden; - padding: 0; - position: absolute; - } - #org-info .render-content p { - display: none; - } - .org-header::before { - content: " "; - height: 60px; - width: 215px; - margin-left: 10px; - margin-bottom: 10px; - background: url(data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIGZpbGw9Im5vbmUiIHZpZXdCb3g9IjAgMCAxNDUzIDQwNiI+PHBhdGggZmlsbD0iIzIwMTQyMiIgZD0iTTQ2IDQ2YzIgMTctNyAxMzUtMiAxNDBzMTI1LTQgMTQyLTJjMjYgNSAyNiA0MS0xIDQ1LTE3IDItMTM2LTgtMTQxLTMtNCA0IDUgMTA1IDMgMTE5IDE0LTQgMjktNiA0My03IDIzLTEgODgtNCAxMDggMCAyMiA0IDI0IDM5LTEgNDQtMzAgNi03NCAwLTEwNyAzLTI3IDItODYgMjYtODgtMTgtMi00MiAxNC05OSA2LTE0MiAwLTQtNi03LTctMTItNC0xOSA1LTE3IDctMjYgOC0zOC0xMS0xNDQgMi0xNzFDMTQgOSAyMCA1IDI4IDRjMTktMyAxMjQtNSAxNDEtMSAyMyA1IDE4IDQzLTMgNDNINDZabTEwMzYgMTc2Yy0xNS0xMC0zMyAzNC0zNyA0NC01IDE0LTEwIDMxLTEzIDQ2cy02IDU3LTExIDY3Yy0xMCAyMC00MSAxNC00My0xMi0xLTE3IDAtMTQxIDQtMTUxIDQtMTMgMjItMTUgMzMtOGw3IDljMSAwIDEyLTE5IDE2LTIyIDQ2LTUxIDc4IDMgMTAzIDQwIDE5IDI4IDc4IDExNCA4OCAxNDAgNyAyMS0xMCAzOC0yOSAyOS05LTUtMzctNjAtNDUtNzNsLTczLTEwOVptMzI2LTEwMWMyNS0xIDMyIDMwIDEwIDM5LTM0IDE0LTU4IDgtOTEgMzgtNjggNjAtMTIgMTIzIDU0IDE0NiAxOCA2IDY3IDkgNzEgMjcgMTAgNDUtODUgMTItMTAzIDUtMTUyLTY0LTExNS0yMDkgMjktMjQ4IDctMiAyMy03IDMwLTdaTTYzNiA1YzcgNyA1IDQyIDQgNTMgMCA3OC04IDE1Ni02IDIzNCAwIDI0IDkgNjYgNSA4Ni0zIDE5LTMxIDIxLTM0LTEtNC0yNiA1LTcwIDUtMTAwIDEtNzgtMTQtMTg0LTUtMjU4IDItMTYgMTktMjYgMzEtMTRaTTMxMyAzMTBjMjEtMzIgMzQtNzAgNTktOTkgMTAtMTEgMzItMzYgNDgtMjQgMjQgMTktMTMgMzgtMjMgNDgtMjAgMjMtMzkgNTctNTAgODYtNSAxMS0xNyA1Ni0yMyA2MS0xMyAxMC0yNiAzLTMyLTExLTktMTctNDQtMTIwLTQ3LTEzNy0zLTI1IDIzLTM2IDM2LTE5czIwIDc0IDMyIDk1Wm0xNzAtMTEzYzEyLTEyIDMyLTQgMzQgMTIgNSAyOC0xIDg0LTEgMTE3IDAgMTQgNyA1Mi00IDYyLTggNy0yMiA3LTI5LTItMTEtMTEtMy02OC0zLTg1IDAtMjctNi03MC0xLTk1IDAtMyAyLTYgNC05Wm0xNS01MmEyMSAyMSAwIDEgMCAwLTQyIDIxIDIxIDAgMCAwIDAgNDJaIi8+PHBhdGggZmlsbD0iI0UxODZGNCIgZD0iTTk1MSA0MGMyMyAyMS0xIDUzLTE2IDcwLTEzIDE1LTQ1IDUyLTY0IDUxLTYgMC00Ni0yMi01My0yNy0zOS0yNy0zNS0xMDIgMjctODkgOCAyIDI4IDE1IDMwIDE1IDMtMSAxMi0xNSAxNy0xOCAxNC0xMSA0NS0xNSA1OS0yWiIvPjxwYXRoIGZpbGw9IiMyMDE0MjIiIGQ9Ik04NjAgMzg3Yy0zLTQtNS05LTYtMTQtMy0yMi0zLTE1NSAwLTE3NiA0LTI0IDM3LTI1IDQwIDIgMyAyMCAzIDE1NiAwIDE3NS0yIDE3LTIyIDI0LTM0IDEzWiIvPjwvc3ZnPg); - } -} diff --git a/service/evil-forgejo/public/assets/css/theme-evil.css b/service/evil-forgejo/public/assets/css/theme-evil.css deleted file mode 100755 index e7e204b32caae527d9f1cc844a0cf63fd8087ed7..0000000000000000000000000000000000000000 --- a/service/evil-forgejo/public/assets/css/theme-evil.css +++ /dev/null @@ -1,189 +0,0 @@ -@import "theme-forgejo-light.css"; -@import "theme-forgejo-dark.css" (prefers-color-scheme: dark); - -:root { - --f: 5; - --l: 4; - --steel-900: lch(from #10161d l calc(c * var(--f)) 320); - --steel-850: lch(from #131a21 l calc(c * var(--f)) 320); - --steel-800: lch(from #171e26 l calc(c * var(--f)) 320); - --steel-750: lch(from #1d262f l calc(c * var(--f)) 320); - --steel-700: lch(from #242d38 l calc(c * var(--f)) 320); - --steel-650: lch(from #2b3642 l calc(c * var(--f)) 320); - --steel-600: lch(from #374351 l calc(c * var(--f)) 320); - --steel-550: lch(from #445161 l calc(c * var(--f)) 320); - --steel-500: lch(from #515f70 l calc(c * var(--f)) 320); - --steel-450: lch(from #5f6e80 l calc(c * var(--f)) 320); - --steel-400: lch(from #6d7d8f l calc(c * var(--f)) 320); - --steel-350: lch(from #7c8c9f l calc(c * var(--f)) 320); - --steel-300: lch(from #8c9caf l calc(c * var(--f)) 320); - --steel-250: lch(from #9dadc0 l calc(c * var(--f)) 320); - --steel-200: lch(from #aebed0 l calc(c * var(--f)) 320); - --steel-150: lch(from #c0cfe0 l calc(c * var(--f)) 320); - --steel-100: lch(from #d2e0f0 l calc(c * var(--f)) 320); - --zinc-50: lch(from #fafafa l calc(c * var(--f)) 320); - --zinc-100: lch(from #f4f4f5 l calc(c * var(--f)) 320); - --zinc-150: lch(from #ececee l calc(c * var(--f)) 320); - --zinc-200: lch(from #e4e4e7 l calc(c * var(--f)) 320); - --zinc-250: lch(from #dcdce0 l calc(c * var(--f)) 320); - --zinc-300: lch(from #d4d4d8 l calc(c * var(--f)) 320); - --zinc-350: lch(from #babac1 l calc(c * var(--f)) 320); - --zinc-400: lch(from #a1a1aa l calc(c * var(--f)) 320); - --zinc-450: lch(from #898992 l calc(c * var(--f)) 320); - --zinc-500: lch(from #71717a l calc(c * var(--f)) 320); - --zinc-550: lch(from #61616a l calc(c * var(--f)) 320); - --zinc-600: lch(from #52525b l calc(c * var(--f)) 320); - --zinc-650: lch(from #484850 l calc(c * var(--f)) 320); - --zinc-700: lch(from #3f3f46 l calc(c * var(--f)) 320); - --zinc-750: lch(from #333338 l calc(c * var(--f)) 320); - --zinc-800: lch(from #27272a l calc(c * var(--f)) 320); - --zinc-850: lch(from #1f1f23 l calc(c * var(--f)) 320); - --zinc-900: lch(from #18181b l calc(c * var(--f)) 320); - --color-primary: #b734cf; - --color-primary-contrast: lch(from var(--color-primary) 95% 0 h); - --color-primary-dark-1: lch(from var(--color-primary) l c h); - --color-primary-dark-2: lch(from var(--color-primary) calc(l * 0.85) c h); - --color-primary-dark-3: lch(from var(--color-primary) calc(l * 0.85) c h); - --color-primary-dark-4: lch(from var(--color-primary) calc(l * 0.72) c h); - --color-primary-dark-5: lch(from var(--color-primary) calc(l * 0.72) c h); - --color-primary-dark-6: lch(from var(--color-primary) calc(l * 0.72) c h); - --color-primary-dark-7: lch(from var(--color-primary) calc(l * 0.72) c h); - --color-primary-light-1: lch(from var(--color-primary) calc(l * 1.15) c h); - --color-primary-light-2: lch(from var(--color-primary) calc(l * 1.28) c h); - --color-primary-light-3: lch(from var(--color-primary) calc(l * 1.48) c h); - --color-primary-light-4: lch(from var(--color-primary) calc(l * 1.68) c h); - --color-primary-light-5: lch( - from - var(--color-primary) - calc(l * 1.82) - calc(c * 0.6) - h - ); - --color-primary-light-6: lch( - from - var(--color-primary) - calc(l * 1.92) - calc(c * 0.3) - h - ); - --color-primary-light-7: lch( - from - var(--color-primary) - calc(l * 1.98) - calc(c * 0.15) - h - ); - --color-primary-alpha-10: lch(from var(--color-primary) l c h / 0.1); - --color-primary-alpha-20: lch(from var(--color-primary) l c h / 0.2); - --color-primary-alpha-30: lch(from var(--color-primary) l c h / 0.3); - --color-primary-alpha-40: lch(from var(--color-primary) l c h / 0.4); - --color-primary-alpha-50: lch(from var(--color-primary) l c h / 0.5); - --color-primary-alpha-60: lch(from var(--color-primary) l c h / 0.6); - --color-primary-alpha-70: lch(from var(--color-primary) l c h / 0.7); - --color-primary-alpha-80: lch(from var(--color-primary) l c h / 0.8); - --color-primary-alpha-90: lch(from var(--color-primary) l c h / 0.9); - --color-active: var(--steel-600); - --color-secondary-alpha-10: lch(from var(--color-secondary) l c h / 0.1); - --color-secondary-alpha-20: lch(from var(--color-secondary) l c h / 0.2); - --color-secondary-alpha-30: lch(from var(--color-secondary) l c h / 0.3); - --color-secondary-alpha-40: lch(from var(--color-secondary) l c h / 0.4); - --color-secondary-alpha-50: lch(from var(--color-secondary) l c h / 0.5); - --color-secondary-alpha-60: lch(from var(--color-secondary) l c h / 0.6); - --color-secondary-alpha-70: lch(from var(--color-secondary) l c h / 0.7); - --color-secondary-alpha-80: lch(from var(--color-secondary) l c h / 0.8); - --color-secondary-alpha-90: lch(from var(--color-secondary) l c h / 0.9); -} -@media (not (prefers-color-scheme: dark)) { - :root { - --color-body: lch(from #fff l calc(c * var(--f) + var(--l)) h); - --color-text-dark: lch(from #000 l calc(c * var(--f) + var(--l)) h); - --color-input-background: lch(from #fff l calc(c * var(--f) + var(--l)) h); - --color-input-toggle-background: lch( - from - #fff - l - calc(c * var(--f) + var(--l)) - h - ); - --color-header-wrapper-transparent: lch( - from - #d2e0f000 - l - calc(c * var(--f) + var(--l)) - h - ); - --color-light-border: lch(from #0000001d l calc(c * var(--f) + var(--l)) h); - --color-hover: lch(from #e4e4e4aa l calc(c * var(--f) + var(--l)) h); - --color-active: lch(from #e2e2e5 l calc(c * var(--f) + var(--l)) 320); - --color-markup-table-row: lch( - from - #ffffff06 - l - calc(c * var(--f) + var(--l)) - h - ); - --color-shadow: lch(from #00000060 l calc(c * var(--f) + var(--l)) h); - --color-text-focus: lch(from #fff l calc(c * var(--f) + var(--l)) h); - --color-reaction-bg: lch(from #0000000a l calc(c * var(--f) + var(--l)) h); - --color-tooltip-text: lch(from #ffffff l calc(c * var(--f) + var(--l)) h); - --color-tooltip-bg: lch(from #000000f0 l calc(c * var(--f) + var(--l)) h); - --color-label-bg: lch(from #cacaca7b l calc(c * var(--f) + var(--l)) h); - --color-label-hover-bg: lch( - from - #cacacaa0 - l - calc(c * var(--f) + var(--l)) - h - ); - --color-label-active-bg: lch( - from - #cacacaff - l - calc(c * var(--f) + var(--l)) - h - ); - --color-label-bg-alt: lch(from #cacacaff l calc(c * var(--f) + var(--l)) h); - --color-overlay-backdrop: lch( - from - #080808c0 - l - calc(c * var(--f) + var(--l)) - h - ); - --checkerboard-color-1: lch(from #ffffff l calc(c * var(--f) + var(--l)) h); - --checkerboard-color-2: lch(from #e5e5e5 l calc(c * var(--f) + var(--l)) h); - } -} -@media (prefers-color-scheme: dark) { - :root { - --f: 0.75; - --color-primary: #e286f3; - } -} - -.organization.profile:has(.overflow-menu-items > a[href="/evil"]) { - .org-avatar, .org-title { - clip: rect(1px, 1px, 1px, 1px); - clip-path: inset(50%); - height: 1px; - width: 1px; - margin: -1px; - overflow: hidden; - padding: 0; - position: absolute; - } - #org-info .render-content p { - display: none; - } - .org-header::before { - content: " "; - height: 60px; - width: 215px; - margin-left: 10px; - margin-bottom: 10px; - background: url(data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIGZpbGw9Im5vbmUiIHZpZXdCb3g9IjAgMCAxNDUzIDQwNiI+PHBhdGggZmlsbD0iIzIwMTQyMiIgZD0iTTQ2IDQ2YzIgMTctNyAxMzUtMiAxNDBzMTI1LTQgMTQyLTJjMjYgNSAyNiA0MS0xIDQ1LTE3IDItMTM2LTgtMTQxLTMtNCA0IDUgMTA1IDMgMTE5IDE0LTQgMjktNiA0My03IDIzLTEgODgtNCAxMDggMCAyMiA0IDI0IDM5LTEgNDQtMzAgNi03NCAwLTEwNyAzLTI3IDItODYgMjYtODgtMTgtMi00MiAxNC05OSA2LTE0MiAwLTQtNi03LTctMTItNC0xOSA1LTE3IDctMjYgOC0zOC0xMS0xNDQgMi0xNzFDMTQgOSAyMCA1IDI4IDRjMTktMyAxMjQtNSAxNDEtMSAyMyA1IDE4IDQzLTMgNDNINDZabTEwMzYgMTc2Yy0xNS0xMC0zMyAzNC0zNyA0NC01IDE0LTEwIDMxLTEzIDQ2cy02IDU3LTExIDY3Yy0xMCAyMC00MSAxNC00My0xMi0xLTE3IDAtMTQxIDQtMTUxIDQtMTMgMjItMTUgMzMtOGw3IDljMSAwIDEyLTE5IDE2LTIyIDQ2LTUxIDc4IDMgMTAzIDQwIDE5IDI4IDc4IDExNCA4OCAxNDAgNyAyMS0xMCAzOC0yOSAyOS05LTUtMzctNjAtNDUtNzNsLTczLTEwOVptMzI2LTEwMWMyNS0xIDMyIDMwIDEwIDM5LTM0IDE0LTU4IDgtOTEgMzgtNjggNjAtMTIgMTIzIDU0IDE0NiAxOCA2IDY3IDkgNzEgMjcgMTAgNDUtODUgMTItMTAzIDUtMTUyLTY0LTExNS0yMDkgMjktMjQ4IDctMiAyMy03IDMwLTdaTTYzNiA1YzcgNyA1IDQyIDQgNTMgMCA3OC04IDE1Ni02IDIzNCAwIDI0IDkgNjYgNSA4Ni0zIDE5LTMxIDIxLTM0LTEtNC0yNiA1LTcwIDUtMTAwIDEtNzgtMTQtMTg0LTUtMjU4IDItMTYgMTktMjYgMzEtMTRaTTMxMyAzMTBjMjEtMzIgMzQtNzAgNTktOTkgMTAtMTEgMzItMzYgNDgtMjQgMjQgMTktMTMgMzgtMjMgNDgtMjAgMjMtMzkgNTctNTAgODYtNSAxMS0xNyA1Ni0yMyA2MS0xMyAxMC0yNiAzLTMyLTExLTktMTctNDQtMTIwLTQ3LTEzNy0zLTI1IDIzLTM2IDM2LTE5czIwIDc0IDMyIDk1Wm0xNzAtMTEzYzEyLTEyIDMyLTQgMzQgMTIgNSAyOC0xIDg0LTEgMTE3IDAgMTQgNyA1Mi00IDYyLTggNy0yMiA3LTI5LTItMTEtMTEtMy02OC0zLTg1IDAtMjctNi03MC0xLTk1IDAtMyAyLTYgNC05Wm0xNS01MmEyMSAyMSAwIDEgMCAwLTQyIDIxIDIxIDAgMCAwIDAgNDJaIi8+PHBhdGggZmlsbD0iI0UxODZGNCIgZD0iTTk1MSA0MGMyMyAyMS0xIDUzLTE2IDcwLTEzIDE1LTQ1IDUyLTY0IDUxLTYgMC00Ni0yMi01My0yNy0zOS0yNy0zNS0xMDIgMjctODkgOCAyIDI4IDE1IDMwIDE1IDMtMSAxMi0xNSAxNy0xOCAxNC0xMSA0NS0xNSA1OS0yWiIvPjxwYXRoIGZpbGw9IiMyMDE0MjIiIGQ9Ik04NjAgMzg3Yy0zLTQtNS05LTYtMTQtMy0yMi0zLTE1NSAwLTE3NiA0LTI0IDM3LTI1IDQwIDIgMyAyMCAzIDE1NiAwIDE3NS0yIDE3LTIyIDI0LTM0IDEzWiIvPjwvc3ZnPg); - @media (prefers-color-scheme: dark) { - background: url(data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIGZpbGw9Im5vbmUiIHZpZXdCb3g9IjAgMCAxNDUzIDQwNiI+PHBhdGggZD0iTTQ2IDQ2YzIgMTctNyAxMzUtMiAxNDBzMTI1LTQgMTQyLTJjMjYgNSAyNiA0MS0xIDQ1LTE3IDItMTM2LTgtMTQxLTMtNCA0IDUgMTA1IDMgMTE5IDE0LTQgMjktNiA0My03IDIzLTEgODgtNCAxMDggMCAyMiA0IDI0IDM5LTEgNDQtMzAgNi03NCAwLTEwNyAzLTI3IDItODYgMjYtODgtMTgtMi00MiAxNC05OSA2LTE0MiAwLTQtNi03LTctMTItNC0xOSA1LTE3IDctMjYgOC0zOC0xMS0xNDQgMi0xNzFDMTQgOSAyMCA1IDI4IDRjMTktMyAxMjQtNSAxNDEtMSAyMyA1IDE4IDQzLTMgNDNINDZabTEwMzYgMTc2Yy0xNS0xMC0zMyAzNC0zNyA0NC01IDE0LTEwIDMxLTEzIDQ2cy02IDU3LTExIDY3Yy0xMCAyMC00MSAxNC00My0xMi0xLTE3IDAtMTQxIDQtMTUxIDQtMTMgMjItMTUgMzMtOGw3IDljMSAwIDEyLTE5IDE2LTIyIDQ2LTUxIDc4IDMgMTAzIDQwIDE5IDI4IDc4IDExNCA4OCAxNDAgNyAyMS0xMCAzOC0yOSAyOS05LTUtMzctNjAtNDUtNzNsLTczLTEwOVptMzI2LTEwMWMyNS0xIDMyIDMwIDEwIDM5LTM0IDE0LTU4IDgtOTEgMzgtNjggNjAtMTIgMTIzIDU0IDE0NiAxOCA2IDY3IDkgNzEgMjcgMTAgNDUtODUgMTItMTAzIDUtMTUyLTY0LTExNS0yMDkgMjktMjQ4IDctMiAyMy03IDMwLTdaTTYzNiA1YzcgNyA1IDQyIDQgNTMgMCA3OC04IDE1Ni02IDIzNCAwIDI0IDkgNjYgNSA4Ni0zIDE5LTMxIDIxLTM0LTEtNC0yNiA1LTcwIDUtMTAwIDEtNzgtMTQtMTg0LTUtMjU4IDItMTYgMTktMjYgMzEtMTRaTTMxMyAzMTBjMjEtMzIgMzQtNzAgNTktOTkgMTAtMTEgMzItMzYgNDgtMjQgMjQgMTktMTMgMzgtMjMgNDgtMjAgMjMtMzkgNTctNTAgODYtNSAxMS0xNyA1Ni0yMyA2MS0xMyAxMC0yNiAzLTMyLTExLTktMTctNDQtMTIwLTQ3LTEzNy0zLTI1IDIzLTM2IDM2LTE5czIwIDc0IDMyIDk1Wm0xNzAtMTEzYzEyLTEyIDMyLTQgMzQgMTIgNSAyOC0xIDg0LTEgMTE3IDAgMTQgNyA1Mi00IDYyLTggNy0yMiA3LTI5LTItMTEtMTEtMy02OC0zLTg1IDAtMjctNi03MC0xLTk1IDAtMyAyLTYgNC05Wm0xNS01MmEyMSAyMSAwIDEgMCAwLTQyIDIxIDIxIDAgMCAwIDAgNDJaIiBmaWxsPSIjZTZkYmU3Ii8+PHBhdGggZmlsbD0iI0UxODZGNCIgZD0iTTk1MSA0MGMyMyAyMS0xIDUzLTE2IDcwLTEzIDE1LTQ1IDUyLTY0IDUxLTYgMC00Ni0yMi01My0yNy0zOS0yNy0zNS0xMDIgMjctODkgOCAyIDI4IDE1IDMwIDE1IDMtMSAxMi0xNSAxNy0xOCAxNC0xMSA0NS0xNSA1OS0yWiIvPjxwYXRoIGQ9Ik04NjAgMzg3Yy0zLTQtNS05LTYtMTQtMy0yMi0zLTE1NSAwLTE3NiA0LTI0IDM3LTI1IDQwIDIgMyAyMCAzIDE1NiAwIDE3NS0yIDE3LTIyIDI0LTM0IDEzWiIgZmlsbD0iI2U2ZGJlNyIvPjwvc3ZnPg); - } - } -} diff --git a/service/evil-forgejo/public/assets/img/apple-touch-icon.png b/service/evil-forgejo/public/assets/img/apple-touch-icon.png deleted file mode 100644 index e36c65b9ddde8a87650e731cbfb9be7658750de9..0000000000000000000000000000000000000000 Binary files a/service/evil-forgejo/public/assets/img/apple-touch-icon.png and /dev/null differ diff --git a/service/evil-forgejo/public/assets/img/avatar_default.png b/service/evil-forgejo/public/assets/img/avatar_default.png deleted file mode 100644 index e5826b20abcdd5f50018f00781b5af612e420793..0000000000000000000000000000000000000000 Binary files a/service/evil-forgejo/public/assets/img/avatar_default.png and /dev/null differ diff --git a/service/evil-forgejo/public/assets/img/favicon.png b/service/evil-forgejo/public/assets/img/favicon.png deleted file mode 100644 index 0d6b8cb661a5b12954f3336bfc39cda87b8b9f38..0000000000000000000000000000000000000000 Binary files a/service/evil-forgejo/public/assets/img/favicon.png and /dev/null differ diff --git a/service/evil-forgejo/public/assets/img/favicon.svg b/service/evil-forgejo/public/assets/img/favicon.svg deleted file mode 100644 index fbab2c55fa0646f4f56b29aaf0de4f15def9a16a..0000000000000000000000000000000000000000 --- a/service/evil-forgejo/public/assets/img/favicon.svg +++ /dev/null @@ -1,14 +0,0 @@ - - - - - - - - - - - - - - diff --git a/service/evil-forgejo/public/assets/img/logo.png b/service/evil-forgejo/public/assets/img/logo.png deleted file mode 100644 index c60882e000aff0359156e3d7d655437aaf445052..0000000000000000000000000000000000000000 Binary files a/service/evil-forgejo/public/assets/img/logo.png and /dev/null differ diff --git a/service/evil-forgejo/public/assets/img/logo.svg b/service/evil-forgejo/public/assets/img/logo.svg deleted file mode 100644 index fbab2c55fa0646f4f56b29aaf0de4f15def9a16a..0000000000000000000000000000000000000000 --- a/service/evil-forgejo/public/assets/img/logo.svg +++ /dev/null @@ -1,14 +0,0 @@ - - - - - - - - - - - - - - diff --git a/service/evil-forgejo/service.pkl b/service/evil-forgejo/service.pkl deleted file mode 100644 index 5e9f987c1015d90a7e2d85aa0b29e3a45019328a..0000000000000000000000000000000000000000 --- a/service/evil-forgejo/service.pkl +++ /dev/null @@ -1,114 +0,0 @@ -extends "../../config/Service.pkl" - -import "../../config/Service.pkl" as service -import "../../config/site.pkl" as site -import "../postgres/service.pkl" as postgres - -class OAuthClient extends service.Requirement { - alias: String = "oauth" - fixed provider = "evil-forgejo" - fixed kind = "oauth-client" - name: String(isNotEmpty) - redirectUris: Listing -} - -local gitHost = if (site.domain.endsWith(".test")) "git.evil.\(site.domain)" else "git.evil.inc" - -meta { name = "evil.inc Git" } -healthyDeadline = "20m" -healthRestartGrace = "2m" -setup = "setup.py" -provide = "provide.py" - -requirements { new postgres.Database { name = "evil_forgejo" } } - -secrets { - ["lfs_jwt"] {} - ["oauth_jwt"] {} - ["security_key"] {} - ["internal_token"] {} - ["anubis_key"] {} - ["automation_password"] {} -} - -requiredSecrets = if (site.domain.endsWith(".test")) new Listing {} else new Listing { - "mailer_address" - "mailer_username" - "mailer_password" -} - -containers { - ["forgejo"] { - image = "code.forgejo.org/forgejo/forgejo@sha256:3d0ba60633e5ce50a4ac8afce06797eda5728411db047373e7d7d68cecb5bfe8" - entrypoint = "/bin/sh" - args { "/studio/start.sh" } - hostNetwork = true - imageUser = true - cpu = 300 - memory = 1024 - - http { - containerPort = 3000 - checkPath = "/api/healthz" - } - - volumes { - ["/data"] {} - ["/studio/start.sh"] { config = "start.sh" } - ["/studio/app.ini"] { config = "app.ini" } - ["/custom/public"] { config = "public" } - } - - env { - ["FORGEJO_CUSTOM"] = "/custom" - ["USER_UID"] = "\(module.uid)" - ["USER_GID"] = "\(module.uid)" - ["GIT_HOST"] = gitHost - ["DB_NAME"] = "${secret.database.name}" - ["DB_USER"] = "${secret.database.username}" - ["DB_PASSWORD"] = "${secret.database.password}" - ["FORGEJO_SERVER_LFS_JWT_SECRET"] = "${secret.own.lfs_jwt}" - ["FORGEJO_OAUTH2_JWT_SECRET"] = "${secret.own.oauth_jwt}" - ["FORGEJO_SECURITY_SECRET_KEY"] = "${secret.own.security_key}" - ["FORGEJO_SECURITY_INTERNAL_TOKEN"] = "${secret.own.internal_token}" - ["MAILER_ENABLED"] = if (site.domain.endsWith(".test")) "false" else "true" - ["MAILER_ADDRESS"] = if (site.domain.endsWith(".test")) "127.0.0.1" else "${secret.own.mailer_address}" - ["MAILER_USERNAME"] = if (site.domain.endsWith(".test")) "" else "${secret.own.mailer_username}" - ["MAILER_PASSWORD"] = if (site.domain.endsWith(".test")) "" else "${secret.own.mailer_password}" - } - - envTemplate = """ - FORGEJO_HTTP_PORT={{ env "NOMAD_PORT_forgejo_http" }} - {{ range nomadService 1 (env "NOMAD_ALLOC_ID") "postgres" }}DB_HOST=\(module.nomadHostPort){{ end }} - """ - } - - ["anubis"] { - image = "ghcr.io/techarohq/anubis@sha256:2babd956e4eb6daa06a41a9b3cbde394a0b2dae1872395f02a380436637c1bfa" - hostNetwork = true - imageUser = true - cpu = 100 - memory = 256 - - http { - containerPort = 80 - hostname = gitHost - checkPath = "/api/healthz" - checkHeaders { ["X-Real-Ip"] = "127.0.0.1" } - forwardRealIp = true - } - - env { - ["COOKIE_DOMAIN"] = gitHost - ["REDIRECT_DOMAINS"] = gitHost - ["ED25519_PRIVATE_KEY_HEX"] = "${secret.own.anubis_key}" - ["COOKIE_PREFIX"] = "anubis_v1" - ["DIFFICULTY"] = "5" - } - - envTemplate = """ - BIND=:{{ env "NOMAD_PORT_anubis_http" }} - TARGET=http://127.0.0.1:{{ env "NOMAD_PORT_forgejo_http" }} - """ - } -} diff --git a/service/evil-forgejo/setup.py b/service/evil-forgejo/setup.py deleted file mode 100644 index 9621dad73a0b9ed982ccad29645c60cdb4be3d9b..0000000000000000000000000000000000000000 --- a/service/evil-forgejo/setup.py +++ /dev/null @@ -1,70 +0,0 @@ -#!/usr/bin/env python3 -import json -import os -import secrets -import subprocess -import sys -import tempfile -import urllib.error -import urllib.request - -from api import instance - - -data = json.load(sys.stdin) -container, address = instance(data["serviceId"]) -script = """ -IFS= read -r password -cd /app/gitea -su-exec git /usr/local/bin/gitea admin user create \ - --username studio-automation --email studio-automation@users.invalid \ - --password "$password" --must-change-password=false >/dev/null 2>&1 || true -su-exec git /usr/local/bin/gitea admin user change-password \ - --username studio-automation --password "$password" >/dev/null -su-exec git /usr/local/bin/gitea admin user must-change-password \ - --unset studio-automation >/dev/null -""" -result = subprocess.run( - ["podman", "--url", "unix:///run/podman/podman.sock", "exec", "-i", container, - "/bin/sh", "-ec", script], - input=data["automation_password"] + "\n", text=True, capture_output=True, -) -if result.returncode: - raise RuntimeError("could not prepare Forgejo automation account") - -path = "nomad/jobs/" + data["serviceId"] -request = urllib.request.Request( - "http://127.0.0.1:4646/v1/var/" + path, - headers={"X-Nomad-Token": os.environ["NOMAD_TOKEN"]}, -) -with urllib.request.urlopen(request, timeout=5) as response: - variable = json.load(response) -current = variable["Items"].get("automation_token") -if current: - check = urllib.request.Request(address + "/api/v1/user", headers={"Authorization": "token " + current}) - try: - with urllib.request.urlopen(check, timeout=5) as response: - response.read() - except urllib.error.HTTPError as error: - if error.code not in (401, 403): - raise - current = None -if not current: - generated = subprocess.run( - ["podman", "--url", "unix:///run/podman/podman.sock", "exec", container, - "/bin/sh", "-ec", "cd /app/gitea; su-exec git /usr/local/bin/gitea admin user generate-access-token --username studio-automation --token-name studio-" + secrets.token_hex(8) + " --raw"], - check=True, capture_output=True, text=True, - ).stdout.strip() - if len(generated) < 20: - raise ValueError("Forgejo did not generate an automation token") - with tempfile.NamedTemporaryFile("w", suffix=".nv.hcl", delete=False) as file: - file.write("items {\n" + "".join( - f" {key} = {json.dumps(value)}\n" - for key, value in {**variable["Items"], "automation_token": generated}.items() - ) + "}\n") - filename = file.name - try: - subprocess.run(["nomad", "var", "put", "-in=hcl", "-out=none", - f"-check-index={variable['ModifyIndex']}", path, "@" + filename], check=True) - finally: - os.unlink(filename) diff --git a/service/evil-forgejo/start.sh b/service/evil-forgejo/start.sh deleted file mode 100644 index c09e16491182b7c1138cf131028c3dc55b22a559..0000000000000000000000000000000000000000 --- a/service/evil-forgejo/start.sh +++ /dev/null @@ -1,10 +0,0 @@ -#!/bin/sh -set -eu - -mkdir -p /custom/conf -envsubst < /studio/app.ini > /custom/conf/app.ini -chown "$USER_UID:$USER_GID" /custom/conf /custom/conf/app.ini -chmod 750 /custom/conf -chmod 600 /custom/conf/app.ini -# The image's s6 bundle starts an SSH daemon on the host's port 22. -exec /usr/bin/entrypoint /bin/bash -c 'source /etc/s6/gitea/setup; cd /app/gitea; exec su-exec git /usr/local/bin/gitea web' diff --git a/service/evil-hedgedoc/icon.svg b/service/evil-hedgedoc/icon.svg deleted file mode 100644 index 7bf57345a95b8f4af6b66008e672e153963bc5c8..0000000000000000000000000000000000000000 --- a/service/evil-hedgedoc/icon.svg +++ /dev/null @@ -1 +0,0 @@ -HedgeDoc diff --git a/service/evil-hedgedoc/service.pkl b/service/evil-hedgedoc/service.pkl deleted file mode 100644 index b485325ca9dc33be80990abbb889aea16bcf52c8..0000000000000000000000000000000000000000 --- a/service/evil-hedgedoc/service.pkl +++ /dev/null @@ -1,66 +0,0 @@ -amends "../../config/Service.pkl" - -import "../../config/site.pkl" as site -import "../evil-forgejo/service.pkl" as forgejo -import "../postgres/service.pkl" as postgres - -local host = if (site.preview) "\(module.id).evil.\(site.domain)" else if (site.domain.endsWith(".test")) "md.evil.\(site.domain)" else "md.evil.inc" - -meta { name = "evil.inc Notes" } -healthyDeadline = "15m" -healthRestartGrace = "5m" - -requirements { - new postgres.Database { name = "evil_hedgedoc" } - new forgejo.OAuthClient { - name = "HedgeDoc" - redirectUris { "https://\(host)/auth/oauth2/callback" } - } -} - -secrets { ["session_secret"] {} } - -container { - image = "quay.io/hedgedoc/hedgedoc@sha256:7b3f79667ad58c6419758547f10940638bcdc85ebee2a4e650318a704095975b" - cpu = 200 - memory = 768 - - http { - containerPort = 3000 - hostname = host - checkPath = "/_health" - metricsPath = "/metrics" - } - - volumes { ["/hedgedoc/public/uploads"] {} } - - env { - ["CMD_DB_DIALECT"] = "postgres" - ["CMD_DB_DATABASE"] = "${secret.database.name}" - ["CMD_DB_USERNAME"] = "${secret.database.username}" - ["CMD_DB_PASSWORD"] = "${secret.database.password}" - ["CMD_DOMAIN"] = host - ["CMD_PROTOCOL_USESSL"] = "true" - ["CMD_URL_ADDPORT"] = "false" - ["CMD_EMAIL"] = "false" - ["CMD_ALLOW_ANONYMOUS"] = "false" - ["CMD_ALLOW_ANONYMOUS_EDITS"] = "false" - ["CMD_SESSION_SECRET"] = "${secret.own.session_secret}" - ["CMD_OAUTH2_PROVIDERNAME"] = "git.evil.inc" - ["CMD_OAUTH2_CLIENT_ID"] = "${secret.oauth.clientId}" - ["CMD_OAUTH2_CLIENT_SECRET"] = "${secret.oauth.clientSecret}" - ["CMD_OAUTH2_BASEURL"] = "${secret.oauth.providerUrl}" - ["CMD_OAUTH2_AUTHORIZATION_URL"] = "${secret.oauth.providerUrl}/login/oauth/authorize" - ["CMD_OAUTH2_USER_PROFILE_ID_ATTR"] = "id" - ["CMD_OAUTH2_USER_PROFILE_USERNAME_ATTR"] = "username" - ["CMD_OAUTH2_USER_PROFILE_DISPLAY_NAME_ATTR"] = "full_name" - ["CMD_OAUTH2_USER_PROFILE_EMAIL_ATTR"] = "email" - } - - envTemplate = """ - {{ range nomadService 1 (env "NOMAD_ALLOC_ID") "postgres" }}CMD_DB_HOST={{ .Address }} - CMD_DB_PORT={{ .Port }}{{ end }} - {{ range nomadService 1 (env "NOMAD_ALLOC_ID") "evil-forgejo-forgejo" }}CMD_OAUTH2_TOKEN_URL=http://\(module.nomadHostPort)/login/oauth/access_token - CMD_OAUTH2_USER_PROFILE_URL=http://\(module.nomadHostPort)/api/v1/user{{ end }} - """ -} diff --git a/service/flaresolverr/service.pkl b/service/flaresolverr/service.pkl index cea31d6230cbd71df9017292e44e471ac46e3be9..542917da76fb1afd287211edbc0181ac3c9b84e9 100644 --- a/service/flaresolverr/service.pkl +++ b/service/flaresolverr/service.pkl @@ -1,6 +1,6 @@ amends "../../config/Service.pkl" -meta { name = "FlareSolverr" } +name = "FlareSolverr" rollout = "overlapped" container { diff --git a/service/forward-auth/service.pkl b/service/forward-auth/service.pkl index 7d003455ee7984c493a24b017006d99a0b7abf86..be5a42fd94d7edd45a13eade8712ff8a0401d7ef 100644 --- a/service/forward-auth/service.pkl +++ b/service/forward-auth/service.pkl @@ -5,13 +5,13 @@ import "../../config/OpenID.pkl" as sso local isPreview = read?("prop:preview") == "true" -meta { name = "Forward Auth" } +name = "Forward Auth" rollout = "overlapped" requirements { new sso.Client { clientId = module.id - name = module.meta.name + name = module.name redirectUris { when (isPreview) { "https://\(module.id).\(site.domain)/snow.oauth2/callback" } when (!isPreview) { for (host in new Listing { "music"; "seedbox"; "ddns"; "redis"; "pg"; "snr"; "rdr"; "logs"; "metrics"; "traces"; "jkt"; "openclaw" }) { @@ -29,7 +29,6 @@ container { image = "quay.io/oauth2-proxy/oauth2-proxy@sha256:56e3daedf765c7a1eea6e366fbe684be7d3084830ade14b6174570d3c7960954" cpu = 100 memory = 256 - extraHosts { "\(sso.hostname):host-gateway" } http { containerPort = 4180 @@ -39,7 +38,7 @@ container { volumes { ["/etc/ssl/certs/ca-certificates.crt"] { - src = "/var/lib/studio/ca-bundle.crt" + src = "/var/lib/snowglobe/ca-bundle.crt" readOnly = true } } diff --git a/service/intake/service.pkl b/service/intake/service.pkl index 6eccfe99bda203db3887a32f7c05994573a0e72c..b4a689aebc5ea877a8c45b79a10a451c923a0b46 100644 --- a/service/intake/service.pkl +++ b/service/intake/service.pkl @@ -2,7 +2,8 @@ amends "../../config/Service.pkl" import "../../config/site.pkl" as site -meta { name = "JSON Intake"; launcher = false } +name = "JSON Intake" +launcher = false secrets = if (site.domain.endsWith(".test")) new { ["key"] {} } else new {} requiredSecrets = if (site.domain.endsWith(".test")) new {} else new { "key" } diff --git a/service/jackett/service.pkl b/service/jackett/service.pkl index 7fb12a443c4b2a37597dc140ceb4141b396f1a02..3b7a454e18bf59e5f5e66b07f03f448647b81544 100644 --- a/service/jackett/service.pkl +++ b/service/jackett/service.pkl @@ -1,11 +1,9 @@ amends "../../config/Service.pkl" -meta { name = "Jackett" } +name = "Jackett" container { image = "lscr.io/linuxserver/jackett@sha256:139a0819074ef16556399a66ee3c492f0d202ec20467cf9300d7126b8a31f2b5" - cpu = 200 - memory = 512 http { containerPort = 9117 diff --git a/service/jellyfin/configure.py b/service/jellyfin/configure.py index 6e0973ba670cdcfefbc6a8d894513a241e99ed27..a38488d8b89d258d90a8c7a1c0357ca3c0d53ad6 100644 --- a/service/jellyfin/configure.py +++ b/service/jellyfin/configure.py @@ -10,13 +10,13 @@ import urllib.request config = json.load(sys.stdin) base = f"https://{config['host']}" -context = ssl.create_default_context(cafile="/var/lib/studio/ca-bundle.crt") +context = ssl.create_default_context(cafile="/var/lib/snowglobe/ca-bundle.crt") def request(path, data=None, token=None): headers = { "Content-Type": "application/json", - "Authorization": 'MediaBrowser Client="home server", Device="server", DeviceId="studio", Version="1"', + "Authorization": 'MediaBrowser Client="home server", Device="server", DeviceId="snowglobe", Version="1"', } if token: headers["X-Emby-Token"] = token diff --git a/service/jellyfin/service.pkl b/service/jellyfin/service.pkl index fd6b9ccc6cc71e9d4969fccdd509f1c3ca76b39d..35887232132c6faf328fc3efb37210d081ee51f5 100644 --- a/service/jellyfin/service.pkl +++ b/service/jellyfin/service.pkl @@ -3,9 +3,7 @@ amends "../../config/Service.pkl" import "../../config/site.pkl" as site import "../../config/OpenID.pkl" as sso -meta { name = "Jellyfin" } -// SQLite under /config requires one writer during rollout. -rollout = "simple" +name = "Jellyfin" healthyDeadline = "20m" healthRestartGrace = "20m" prepare = "prepare.py" @@ -16,7 +14,7 @@ secrets { ["admin_password"] {} } requirements { new sso.Client { clientId = module.id - name = module.meta.name + name = module.name redirectUris { "https://\(module.container.http.hostname)/sso/OID/r/snow" "https://\(module.container.http.hostname)/sso/OID/redirect/snow" @@ -28,7 +26,6 @@ container { image = "docker.io/jellyfin/jellyfin@sha256:aefb67e6a7ff1debdd154a78a7bbb780fd0c873d8639210a7f6a2016ad2b35db" cpu = 500 memory = 3072 - extraHosts { "\(sso.hostname):host-gateway" } http { containerPort = 8096 @@ -53,7 +50,7 @@ container { } ["/config/config/branding.xml"] { config = "branding.xml" } ["/etc/ssl/certs/ca-certificates.crt"] { - src = "/var/lib/studio/ca-bundle.crt" + src = "/var/lib/snowglobe/ca-bundle.crt" readOnly = true } } diff --git a/service/keycloak/api.py b/service/keycloak/api.py deleted file mode 100644 index 6cd2533e3cdb1b75890c7bbfc9dbcdada009fab1..0000000000000000000000000000000000000000 --- a/service/keycloak/api.py +++ /dev/null @@ -1,83 +0,0 @@ -import http.client -import io -import json -import os -import socket -import ssl -import time -import urllib.error -import urllib.parse - - -class LoopbackHTTPS(http.client.HTTPSConnection): - def connect(self): - sock = socket.create_connection(("127.0.0.1", self.port), self.timeout) - self.sock = self._context.wrap_socket(sock, server_hostname=self.host) - - -class Keycloak: - def configure_profile(self): - profile = self.request("/admin/realms/master/users/profile") - if not any(attribute["name"] == "picture" for attribute in profile["attributes"]): - profile["attributes"].append({"name": "picture", "displayName": "Profile picture", - "permissions": {"view": ["admin", "user"], "edit": ["admin", "user"]}, - "validations": {"length": {"max": 8192}}}) - self.request("/admin/realms/master/users/profile", "PUT", profile) - - def __init__(self, host, password, attempts=300, cafile=None): - self.host = host - self.context = ssl.create_default_context( - cafile=cafile or ("/var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt" - if host.endswith(".test") else None) - ) - self.token = None - for attempt in range(attempts): - try: - body = urllib.parse.urlencode({ - "client_id": "admin-cli", - "grant_type": "password", - "username": "admin", - "password": password, - }).encode() - self.token = self.request("/realms/master/protocol/openid-connect/token", "POST", body)["access_token"] - break - except urllib.error.HTTPError as error: - if error.code not in (502, 503, 504) or attempt == attempts - 1: - raise - time.sleep(2) - except (ConnectionError, TimeoutError, http.client.HTTPException, ssl.SSLError): - if attempt == attempts - 1: - raise - time.sleep(2) - - def request(self, path, method="GET", body=None, full=False): - headers = {} - if self.token: - headers["Authorization"] = "Bearer " + self.token - if isinstance(body, (dict, list)): - body = json.dumps(body).encode() - headers["Content-Type"] = "application/json" - elif body is not None: - headers["Content-Type"] = "application/x-www-form-urlencoded" - connection = LoopbackHTTPS( - self.host, - timeout=int(os.getenv("STUDIO_API_TIMEOUT", "10")), - context=self.context, - ) - try: - connection.request(method, path, body=body, headers=headers) - response = connection.getresponse() - content = response.read(16 * 1024 * 1024 + 1) - if len(content) > 16 * 1024 * 1024: - raise ValueError("Keycloak response exceeds 16 MiB") - if response.status >= 400: - raise urllib.error.HTTPError( - "https://" + self.host + path, response.status, response.reason, - response.headers, io.BytesIO(content), - ) - value = json.loads(content) if content else None - if full: - return {"body": value, "id": response.getheader("Location", "").rsplit("/", 1)[-1] or None} - return value - finally: - connection.close() diff --git a/service/keycloak/configure.py b/service/keycloak/configure.py deleted file mode 100644 index 7a6cafa54cf3e5299b6c251915685997f935de03..0000000000000000000000000000000000000000 --- a/service/keycloak/configure.py +++ /dev/null @@ -1,67 +0,0 @@ -#!/usr/bin/env python3 -import json -import sys - -from api import Keycloak - - -config = json.load(sys.stdin) -keycloak = Keycloak(config["host"], config["password"]) -keycloak.configure_profile() -keycloak.request( - "/admin/realms/master", - "PUT", - { - "displayName": "snow sign on", - "displayNameHtml": '
snow sign on
', - "editUsernameAllowed": True, - "resetPasswordAllowed": True, - "rememberMe": True, - "loginWithEmailAllowed": True, - "registrationAllowed": False, - "webAuthnPolicyRpEntityName": "snow sign on", - "webAuthnPolicyPasswordlessPasskeysEnabled": True, - "loginTheme": "snow", - }, -) -roles_path = "/admin/realms/master/roles" -roles = {role["name"]: role for role in keycloak.request(roles_path)} -for name in ("admin", "infra-admin", "media", "media-manage"): - if name not in roles: - keycloak.request(roles_path, "POST", {"name": name}) - roles[name] = keycloak.request(f"{roles_path}/{name}") - -users = keycloak.request("/admin/realms/master/users?username=admin&exact=true") -if len(users) != 1: - raise ValueError("expected one bootstrap admin user") -if users[0].get("email") or users[0].get("emailVerified"): - keycloak.request( - f"/admin/realms/master/users/{users[0]['id']}", "PUT", - {**users[0], "email": None, "emailVerified": False}, - ) -mapping_path = f"/admin/realms/master/users/{users[0]['id']}/role-mappings/realm" -assigned = {role["name"] for role in keycloak.request(mapping_path)} -missing = [roles[name] for name in ("admin", "media", "media-manage") if name not in assigned] -if missing: - keycloak.request(mapping_path, "POST", missing) - -snow = keycloak.request("/admin/realms/master/users?username=snow&exact=true") -if not snow: - keycloak.request("/admin/realms/master/users", "POST", { - "username": "snow", - "enabled": True, - "credentials": [{"type": "password", "value": config["snow_password"], "temporary": False}], - }) - snow = keycloak.request("/admin/realms/master/users?username=snow&exact=true") -if len(snow) != 1: - raise ValueError("expected one snow user") -if snow[0].get("email") != config["ownerEmail"] or not snow[0].get("emailVerified"): - keycloak.request( - f"/admin/realms/master/users/{snow[0]['id']}", "PUT", - {**snow[0], "email": config["ownerEmail"], "emailVerified": True}, - ) -mapping_path = f"/admin/realms/master/users/{snow[0]['id']}/role-mappings/realm" -assigned = {role["name"] for role in keycloak.request(mapping_path)} -missing = [roles[name] for name in ("infra-admin", "media", "media-manage") if name not in assigned] -if missing: - keycloak.request(mapping_path, "POST", missing) diff --git a/service/keycloak/icon.svg b/service/keycloak/icon.svg deleted file mode 100644 index 3e8115efc160742cda84171b4bd4414ec59f6af5..0000000000000000000000000000000000000000 --- a/service/keycloak/icon.svg +++ /dev/null @@ -1 +0,0 @@ - \ No newline at end of file diff --git a/service/keycloak/provide.py b/service/keycloak/provide.py deleted file mode 100644 index bc31a2cbe699ba539ed8de0f83e70d48f5a3d1f9..0000000000000000000000000000000000000000 --- a/service/keycloak/provide.py +++ /dev/null @@ -1,100 +0,0 @@ -#!/usr/bin/env python3 -import json -import sys -import urllib.parse - -from api import Keycloak - - -data = json.load(sys.stdin) -request = data["request"] -if request["kind"] != "client": - raise ValueError("unsupported Keycloak input") -stage_id = data.get("stageId") -client_id = request["clientId"] -if stage_id and client_id != stage_id: - client_id += "-" + stage_id[-8:] -if data.get("operation") == "delete" and not stage_id: - raise ValueError("refusing to delete a production client") -existing = data.get("existing") or {} -if existing.get("clientId") and existing["clientId"] != client_id: - raise ValueError("client ID changed; migrate the existing client before deployment") -keycloak = Keycloak(data["host"], data["providerSecrets"]["password"]) -desired = { - "protocol": "openid-connect", - "clientId": client_id, - "name": request["name"], - "publicClient": False, - "authorizationServicesEnabled": False, - "serviceAccountsEnabled": False, - "implicitFlowEnabled": False, - "directAccessGrantsEnabled": False, - "standardFlowEnabled": True, - "frontchannelLogout": True, - "redirectUris": request["redirectUris"], - "attributes": {"post.logout.redirect.uris": "*"}, -} -path = "/admin/realms/master/clients" -query = "?" + urllib.parse.urlencode({"clientId": client_id}) -found = keycloak.request(path + query) -matches = [client for client in found if client["clientId"] == client_id] -if len(matches) > 1: - raise ValueError(f"duplicate Keycloak client: {client_id}") -if data.get("operation") == "delete": - if matches: - keycloak.request(f"{path}/{matches[0]['id']}", "DELETE") - sys.exit(0) -if not matches: - keycloak.request(path, "POST", desired) - found = keycloak.request(path + query) - matches = [client for client in found if client["clientId"] == client_id] - if len(matches) != 1: - raise ValueError(f"Keycloak client was not created: {client_id}") -uuid = matches[0]["id"] -current = keycloak.request(f"{path}/{uuid}") -attributes = {**(current.get("attributes") or {}), **desired["attributes"]} -if any(current.get(key) != value for key, value in desired.items() if key != "attributes") or current.get("attributes", {}) != attributes: - keycloak.request(f"{path}/{uuid}", "PUT", {**current, **desired, "attributes": attributes}) -aliases = request.get("usernameAliases", {}) -if aliases: - if len(set(aliases.values())) != len(aliases): - raise ValueError("Shale username aliases must be unique") - roles_path = f"{path}/{uuid}/roles" - roles = {role["name"]: role for role in keycloak.request(roles_path)} - if set(roles) - set(aliases.values()): - raise ValueError("username alias clients cannot also carry permission roles") - for username, alias in aliases.items(): - users = keycloak.request("/admin/realms/master/users?" + urllib.parse.urlencode({"username": username, "exact": "true"})) - if len(users) != 1: - raise ValueError(f"expected one alias account: {username}") - if alias not in roles: - keycloak.request(roles_path, "POST", {"name": alias, "description": "Shale username alias"}) - roles[alias] = keycloak.request(roles_path + "/" + urllib.parse.quote(alias, safe="")) - owners = keycloak.request(roles_path + "/" + urllib.parse.quote(alias, safe="") + "/users") - if any(owner["id"] != users[0]["id"] for owner in owners): - raise ValueError(f"username alias already assigned: {alias}") - assigned = keycloak.request(f"/admin/realms/master/users/{users[0]['id']}/role-mappings/clients/{uuid}") - if any(role["name"] != alias for role in assigned): - raise ValueError(f"multiple username aliases for {username}") - if not assigned: - keycloak.request(f"/admin/realms/master/users/{users[0]['id']}/role-mappings/clients/{uuid}", "POST", [roles[alias]]) - mapper = { - "name": "Shale username alias", "protocol": "openid-connect", - "protocolMapper": "oidc-usermodel-client-role-mapper", - "config": {"usermodel.clientRoleMapping.clientId": client_id, - "claim.name": "preferred_username", "jsonType.label": "String", - "multivalued": "false", "access.token.claim": "true", - "id.token.claim": "true", "userinfo.token.claim": "true"}, - } - mappers_path = f"{path}/{uuid}/protocol-mappers/models" - matches = [item for item in keycloak.request(mappers_path) if item["name"] == mapper["name"]] - if len(matches) > 1: - raise ValueError("duplicate Shale username mapper") - if not matches: - keycloak.request(mappers_path, "POST", mapper) - elif any(matches[0].get(key) != value for key, value in mapper.items()): - keycloak.request(f"{mappers_path}/{matches[0]['id']}", "PUT", {**mapper, "id": matches[0]["id"]}) -secret = keycloak.request(f"{path}/{uuid}/client-secret")["value"] -if not secret: - raise ValueError(f"Keycloak client has no secret: {client_id}") -print(json.dumps({"clientId": client_id, "clientSecret": secret, "issuerUrl": f"https://{data['host']}/realms/master"})) diff --git a/service/keycloak/service.pkl b/service/keycloak/service.pkl deleted file mode 100644 index e4b5ceba87a7e4822d39616028726aa1778dfa8c..0000000000000000000000000000000000000000 --- a/service/keycloak/service.pkl +++ /dev/null @@ -1,73 +0,0 @@ -extends "../../config/Service.pkl" - -import "../../config/Service.pkl" as service -import "../../config/site.pkl" as site -import "../postgres/service.pkl" as postgres - -class OpenIDClient extends service.Requirement { - alias: String = "oidc" - fixed provider = "keycloak" - fixed kind = "client" - clientId: String(isNotEmpty) - name: String - redirectUris: Listing = new { "*" } - usernameAliases: Mapping = new {} -} - -local database = new postgres.Database { name = "keycloak_next" } - -meta { name = "Keycloak"; launcher = false } -traceServiceName = module.id -dependsOn { "victoria" } -healthyDeadline = "20m" -setup = "configure.py" -provide = "provide.py" - -requirements { database } - -container { - image = "quay.io/keycloak/keycloak@sha256:82a77884f3af238beab1e7afd63b5f530e1b5c0590bd7aa60b40a40463e29b2c" - rootGroup = true - cpu = 500 - memory = 1200 - args { "start" } - - http { - containerPort = 8080 - subdomain = "auth" - plainHostnames { "keycloak.\(site.domain)" } - checkPath = "/realms/master" - } - - volumes { - ["/opt/keycloak/data"] {} - ["/opt/keycloak/themes/snow"] { config = "theme" } - } - - env { - ["JAVA_TOOL_OPTIONS"] = read?("env:STUDIO_JAVA_OPTIONS") ?? "" - ["KC_DB"] = "postgres" - ["KC_DB_USERNAME"] = "${secret.database.username}" - ["KC_DB_PASSWORD"] = "${secret.database.password}" - ["KC_BOOTSTRAP_ADMIN_USERNAME"] = "admin" - ["KC_BOOTSTRAP_ADMIN_PASSWORD"] = "${secret.own.password}" - ["KC_HTTP_ENABLED"] = "true" - ["KC_PROXY_HEADERS"] = "xforwarded" - ["KC_HOSTNAME"] = "https://\(module.container.http.hostname)" - ["KC_TRACING_ENABLED"] = "true" - ["KC_TRACING_PROTOCOL"] = "http/protobuf" - ["KC_TELEMETRY_SERVICE_NAME"] = module.id - ["KC_TRACING_SAMPLER_TYPE"] = "parentbased_traceidratio" - ["KC_TRACING_SAMPLER_RATIO"] = "0.25" - } - - envTemplate = """ - {{ range nomadService 1 (env "NOMAD_ALLOC_ID") "\(database.provider)" }}KC_DB_URL=jdbc:postgresql://\(module.nomadHostPort)/{{ with nomadVar "nomad/jobs/\(module.id)/inputs/database" }}{{ .name }}{{ end }}{{ end }} - {{ range nomadService 1 (env "NOMAD_ALLOC_ID") "victoria-traces" }}KC_TRACING_ENDPOINT=http://\(module.nomadHostPort)/insert/opentelemetry{{ end }} - """ -} - -secrets { - ["password"] {} - ["snow_password"] {} -} diff --git a/service/keycloak/theme/login/resources/css/styles.css b/service/keycloak/theme/login/resources/css/styles.css deleted file mode 120000 index d770636a4ef025e125c1a98b577ed77d2eeef25c..0000000000000000000000000000000000000000 --- a/service/keycloak/theme/login/resources/css/styles.css +++ /dev/null @@ -1 +0,0 @@ -../../../../../../dashboard/web/sso/css/styles.css \ No newline at end of file diff --git a/service/keycloak/theme/login/resources/font/.gitignore b/service/keycloak/theme/login/resources/font/.gitignore deleted file mode 100644 index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..0000000000000000000000000000000000000000 diff --git a/service/keycloak/theme/login/resources/img/license.txt b/service/keycloak/theme/login/resources/img/license.txt deleted file mode 120000 index a8df3a0a49d5efc3e25eb1eff8aa0a544eb386fe..0000000000000000000000000000000000000000 --- a/service/keycloak/theme/login/resources/img/license.txt +++ /dev/null @@ -1 +0,0 @@ -../../../../../../dashboard/web/sso/img/license.txt \ No newline at end of file diff --git a/service/keycloak/theme/login/resources/img/miku-dark.png b/service/keycloak/theme/login/resources/img/miku-dark.png deleted file mode 120000 index b847b152a6470d4a21fbe441d040bb9a08e98cf6..0000000000000000000000000000000000000000 --- a/service/keycloak/theme/login/resources/img/miku-dark.png +++ /dev/null @@ -1 +0,0 @@ -../../../../../../dashboard/web/sso/img/miku-dark.png \ No newline at end of file diff --git a/service/keycloak/theme/login/resources/img/miku-light.png b/service/keycloak/theme/login/resources/img/miku-light.png deleted file mode 120000 index ff83a9d81a3d43936b42b58f9a2f478ac46fdde8..0000000000000000000000000000000000000000 --- a/service/keycloak/theme/login/resources/img/miku-light.png +++ /dev/null @@ -1 +0,0 @@ -../../../../../../dashboard/web/sso/img/miku-light.png \ No newline at end of file diff --git a/service/keycloak/theme/login/resources/js/stars.js b/service/keycloak/theme/login/resources/js/stars.js deleted file mode 100644 index 20327f1eba4fe0d989631e63dc3c3aef94e0d35d..0000000000000000000000000000000000000000 --- a/service/keycloak/theme/login/resources/js/stars.js +++ /dev/null @@ -1,105 +0,0 @@ -const shader = ` - // Heavily modified off of https://www.shadertoy.com/view/3sKGWw - // big thanks to these resources as well: - // https://youtu.be/3CycKKJiwis - // https://www.shadertoy.com/view/4djSRW - // https://www.shadertoy.com/view/3s3GDn - - precision mediump float; - uniform vec2 u_resolution; - uniform float u_time; - uniform bool u_mask; - - float random(vec2 p){ - vec3 p3 = fract(vec3(p.xyx) * .1031); - p3 += dot(p3, p3.yzx + 33.33); - return fract((p3.x + p3.y) * p3.z); - } - - vec2 random2(vec2 p){ - vec3 p3 = fract(vec3(p.xyx) * vec3(.1031, .1030, .0973)); - p3 += dot(p3, p3.yzx+33.33); - return fract((p3.xx+p3.yz)*p3.zy); - } - - void main() { - float glow; - vec2 uv; - vec2 fl; - vec2 local_uv; - vec2 seed; - vec2 centre; - vec2 cell; - float t = u_time * 0.00015; - float blinkTime = u_time * 0.0068; - float rotationAngle = -t * 21.0; - mat2 rotation = mat2(cos(rotationAngle), -sin(rotationAngle), - sin(rotationAngle), cos(rotationAngle)); - vec3 col = vec3(0); - - centre = vec2(cos(t), sin(t)) * 5.0 + 0.5; - - // Get uv from the fragment coordinates, rotation and depth - uv = (centre - (gl_FragCoord.xy - u_resolution / 2.0) / (1600.0 / 25.0)) * rotation; - fl = floor(uv); - // The local cell coordinates. uv-fl == frac(uv) - local_uv = uv - fl - 0.5; - - // For a 3x3 group of cells around the fragment, find the distance from the points - // of each to the current fragment and draw an accumulative glow accordingly - // The local cell is (0, 0) - for(float j = -1.0; j <= 1.0; j++){ - for(float k = -1.0; k <= 1.0; k++){ - cell = vec2(j, k); - seed = 128.0 + (fl + cell); - glow = - ((0.6 + 0.5 * sin((128.0 * random(seed)) * blinkTime))) * 25.0 - * pow(0.005 / length(local_uv - (cell + 0.9 * (random2(seed) - 0.5))), 2.0); - col += vec3(0.2 * glow) + 0.1 * glow; - } - } - - col = 1.0 - exp(-col); - col = pow(col, vec3(0.45)); - - if (u_mask) { - float mask = smoothstep(100.0, 300.0, length(vec2(gl_FragCoord.x - u_resolution.x/2.0, gl_FragCoord.y - 200.0))); - col *= mask; - } - - gl_FragColor = vec4(col * (u_mask ? 0.6 : 0.25), 0); - } -`; - -function createShader(gl, type, source) { - const shader = gl.createShader(type); - gl.shaderSource(shader, source); - gl.compileShader(shader); - const success = gl.getShaderParameter(shader, gl.COMPILE_STATUS); - if (success) { - return shader; - } - - // eslint-disable-next-line no-console - console.error(gl.getShaderInfoLog(shader)); - gl.deleteShader(shader); - throw new Error("Failed to compile shader"); -} - -function createProgram(gl, vertexShader, fragmentShader) { - const program = gl.createProgram(); - - gl.attachShader(program, vertexShader); - gl.attachShader(program, fragmentShader); - gl.linkProgram(program); - - const success = gl.getProgramParameter(program, gl.LINK_STATUS); - if (success) { - return program; - } - - // eslint-disable-next-line no-console - console.error(gl.getProgramInfoLog(program)); - gl.deleteProgram(program); - throw new Error("Failed to link program"); -} diff --git a/service/keycloak/theme/login/resources/js/webauthnRegister.js b/service/keycloak/theme/login/resources/js/webauthnRegister.js deleted file mode 100644 index eb98092ae45cd3931051e1bab579762534b847cd..0000000000000000000000000000000000000000 --- a/service/keycloak/theme/login/resources/js/webauthnRegister.js +++ /dev/null @@ -1,159 +0,0 @@ -import { base64url } from "rfc4648"; - -export async function registerByWebAuthn(input) { - // Check if WebAuthn is supported by this browser - if (!window.PublicKeyCredential) { - returnFailure(input.errmsg); - return; - } - - const publicKey = { - challenge: base64url.parse(input.challenge, { loose: true }), - rp: { id: input.rpId, name: input.rpEntityName }, - user: { - id: base64url.parse(input.userid, { loose: true }), - name: input.username, - displayName: input.username, - }, - pubKeyCredParams: getPubKeyCredParams(input.signatureAlgorithms), - }; - - if (input.attestationConveyancePreference !== "not specified") { - publicKey.attestation = input.attestationConveyancePreference; - } - - const authenticatorSelection = {}; - let isAuthenticatorSelectionSpecified = false; - - if (input.authenticatorAttachment !== "not specified") { - authenticatorSelection.authenticatorAttachment = - input.authenticatorAttachment; - isAuthenticatorSelectionSpecified = true; - } - - if (input.requireResidentKey !== "not specified") { - if (input.requireResidentKey === "Yes") { - authenticatorSelection.requireResidentKey = true; - } else { - authenticatorSelection.requireResidentKey = false; - } - isAuthenticatorSelectionSpecified = true; - } - - if (input.userVerificationRequirement !== "not specified") { - authenticatorSelection.userVerification = input.userVerificationRequirement; - isAuthenticatorSelectionSpecified = true; - } - - if (isAuthenticatorSelectionSpecified) { - publicKey.authenticatorSelection = authenticatorSelection; - } - - if (input.createTimeout !== 0) { - publicKey.timeout = input.createTimeout * 1000; - } - - const excludeCredentials = getExcludeCredentials(input.excludeCredentialIds); - if (excludeCredentials.length > 0) { - publicKey.excludeCredentials = excludeCredentials; - } - - try { - const result = await doRegister(publicKey); - returnSuccess(result, input.initLabel, input.initLabelPrompt); - } catch (error) { - returnFailure(error); - } -} - -function doRegister(publicKey) { - return navigator.credentials.create({ publicKey }); -} - -function getPubKeyCredParams(signatureAlgorithmsList) { - const pubKeyCredParams = []; - if (signatureAlgorithmsList.length === 0) { - pubKeyCredParams.push({ type: "public-key", alg: -7 }); - return pubKeyCredParams; - } - - for (const entry of signatureAlgorithmsList) { - pubKeyCredParams.push({ - type: "public-key", - alg: entry, - }); - } - - return pubKeyCredParams; -} - -function getExcludeCredentials(excludeCredentialIds) { - const excludeCredentials = []; - if (excludeCredentialIds === "") { - return excludeCredentials; - } - - for (const entry of excludeCredentialIds.split(",")) { - excludeCredentials.push({ - type: "public-key", - id: base64url.parse(entry, { loose: true }), - }); - } - - return excludeCredentials; -} - -function getTransportsAsString(transportsList) { - if (!Array.isArray(transportsList)) { - return ""; - } - - return transportsList.join(); -} - -function returnSuccess(result, initLabel, initLabelPrompt) { - document.getElementById("clientDataJSON").value = base64url.stringify( - new Uint8Array(result.response.clientDataJSON), - { pad: false }, - ); - document.getElementById("attestationObject").value = base64url.stringify( - new Uint8Array(result.response.attestationObject), - { pad: false }, - ); - document.getElementById("publicKeyCredentialId").value = base64url.stringify( - new Uint8Array(result.rawId), - { pad: false }, - ); - - if (typeof result.response.getTransports === "function") { - const transports = result.response.getTransports(); - if (transports) { - document.getElementById("transports").value = getTransportsAsString( - transports, - ); - } - } else { - console.log( - "Your browser is not able to recognize supported transport media for the authenticator.", - ); - } - - // let labelResult = window.prompt(initLabelPrompt, initLabel); - // if (labelResult === null) { - // labelResult = initLabel; - // } - document.getElementById("authenticatorLabel").value = "Passkey " + - new Date().toString(); - - document.getElementById("register").requestSubmit(); -} - -function returnFailure(err) { - if (err.name === "NotAllowedError") { - document.getElementById("error").value = - "the operation was cancelled / browser does not support passkey"; - } else { - document.getElementById("error").value = err; - } - document.getElementById("register").requestSubmit(); -} diff --git a/service/keycloak/theme/login/template.ftl b/service/keycloak/theme/login/template.ftl deleted file mode 100644 index 712a4f97ea88328c90c22b552a9774e5637a5fa6..0000000000000000000000000000000000000000 --- a/service/keycloak/theme/login/template.ftl +++ /dev/null @@ -1,209 +0,0 @@ -<#import "footer.ftl" as loginFooter> -<#macro registrationLayout bodyClass="" displayInfo=false displayMessage=true displayRequiredFields=false> - - dir="${(locale.rtl)?then('rtl','ltr')}"> - - - - - - - <#if properties.meta?has_content> - <#list properties.meta?split(' ') as meta> - - - - sso (snow sign on) - - <#if properties.stylesCommon?has_content> - <#list properties.stylesCommon?split(' ') as style> - - - - <#if properties.styles?has_content> - <#list properties.styles?split(' ') as style> - - - - <#if properties.scripts?has_content> - <#list properties.scripts?split(' ') as script> - - - - - - <#if scripts??> - <#list scripts as script> - - - - - - <#if authenticationSession??> - - - - - -
-
-
${kcSanitize(msg("loginTitleHtml",(realm.displayNameHtml!'')))?no_esc}
-
-
-
- <#if realm.internationalizationEnabled && locale.supported?size gt 1> -
-
- -
-
- - <#if !(auth?has_content && auth.showUsername() && !auth.showResetCredentials())> - <#if displayRequiredFields> -
-
- * ${msg("requiredFields")} -
-
-

<#nested "header">

-
-
- <#else> -

<#nested "header">

- - <#else> - <#if displayRequiredFields> -
-
- * ${msg("requiredFields")} -
-
- <#nested "show-username"> -
- - - - -
-
-
- <#else> - <#nested "show-username"> -
- - - - -
- - -
-
-
- - <#-- App-initiated actions should not see warning messages about the need to complete the action --> - <#-- during login. --> - <#if displayMessage && message?has_content && (message.type != 'warning' || !isAppInitiatedAction??)> -
-
- <#if message.type = 'success'> - <#if message.type = 'warning'> - <#if message.type = 'error'> - <#if message.type = 'info'> -
- ${kcSanitize(message.summary)?no_esc} -
- - - <#nested "form"> - - <#if auth?has_content && auth.showTryAnotherWayLink()> -
- -
- - - <#nested "socialProviders"> - - <#if displayInfo> -
-
- <#nested "info"> -
-
- -
-
- - - <@loginFooter.content/> -
-
- - - - diff --git a/service/keycloak/theme/login/theme.properties b/service/keycloak/theme/login/theme.properties deleted file mode 100644 index 4782b6590d5e7fb87a10c56a54cc7aa7b9cdb0f1..0000000000000000000000000000000000000000 --- a/service/keycloak/theme/login/theme.properties +++ /dev/null @@ -1,128 +0,0 @@ -parent=base -import=common/keycloak - -styles=css/styles.css -stylesCommon= - -darkMode=false - -kcFormGroupClass=pf-v5-c-form__group -kcFormGroupLabelClass=pf-v5-c-form__group-label pf-v5-u-pb-xs -kcFormLabelClass=pf-v5-c-form__label -kcFormLabelTextClass=pf-v5-c-form__label-text - -kcLabelClass=pf-v5-c-form__label -kcInputClass=pf-v5-c-form-control -kcInputGroup=pf-v5-c-input-group -kcFormHelperTextClass=pf-v5-c-form__helper-text -kcInputHelperTextClass=pf-v5-c-helper-text pf-v5-u-display-flex pf-v5-u-justify-content-space-between -kcInputHelperTextItemClass=pf-v5-c-helper-text__item -kcInputHelperTextItemTextClass=pf-v5-c-helper-text__item-text -kcInputGroupItemClass=pf-v5-c-input-group__item -kcFill=pf-m-fill -kcError=pf-m-error -kcLoginFooterBand=pf-v5-c-login__main-footer-band -kcLoginFooterBandItem=pf-v5-c-login__main-footer-band-item - -kcCheckboxClass=pf-v5-c-check -kcCheckboxInputClass=pf-v5-c-check__input -kcCheckboxLabelClass=pf-v5-c-check__label -kcCheckboxLabelRequiredClass=pf-v5-c-check__label-required - -kcInputRequiredClass=pf-v5-c-form__label-required -kcInputErrorMessageClass=pf-v5-c-helper-text__item-text pf-m-error kc-feedback-text -kcFormControlUtilClass=pf-v5-c-form-control__utilities -kcInputErrorIconStatusClass=pf-v5-c-form-control__icon pf-m-status -kcInputErrorIconClass=fas fa-exclamation-circle -kcAlertClass=pf-v5-c-alert pf-m-inline pf-v5-u-mb-md -kcAlertIconClass=pf-v5-c-alert__icon -kcAlertTitleClass=pf-v5-c-alert__title -kcAlertDescriptionClass=pf-v5-c-alert__description -kcFormPasswordVisibilityButtonClass=pf-v5-c-button pf-m-control -kcFormControlToggleIcon=pf-v5-c-form-control__toggle-icon -kcFormActionGroupClass=pf-v5-c-form__actions pf-v5-u-pt-xs -kcFormReadOnlyClass=pf-m-readonly - -kcPanelClass=pf-v5-c-panel pf-m-raised -kcPanelMainClass=pf-v5-c-panel__main -kcPanelMainBodyClass=pf-v5-c-panel__main-body -kcListClass=pf-v5-c-list - -kcButtonClass=pf-v5-c-button -kcButtonPrimaryClass=pf-v5-c-button pf-m-primary -kcButtonSecondaryClass=pf-v5-c-button pf-m-secondary -kcButtonBlockClass=pf-m-block -kcButtonLinkClass=pf-v5-c-button pf-m-link -kcCommonLogoIdP=pf-v5-c-login__main-footer-links-item -kcFormSocialAccountListClass=pf-v5-c-login__main-body pf-v5-u-pl-0 pf-v5-u-pr-0 -kcFormSocialAccountListItemClass=pf-v5-u-pb-sm -kcFormSocialAccountNameClass=pf-v5-u-m-auto -kcFormSocialAccountListButtonClass=pf-v5-c-button pf-m-secondary pf-m-block pf-v5-u-display-flex pf-v5-u-align-items-center pf-v5-u-justify-content-space-between -kcFormSocialAccountListButtonDisabledClass=pf-m-aria-disabled -kcFormSocialAccountListGridClass=pf-v5-l-grid pf-m-gutter pf-m-all-6-col-on-xl pf-m-all-6-col-on-sm -kcFormSocialAccountGridItem=pf-v5-l-grid__item - -kcLoginClass=pf-v5-c-login__main -kcFormClass=pf-v5-c-form pf-v5-u-w-100 -kcFormCardClass=card-pf - -kcResetFlowIcon=pf-icon fas fa-share-square - -kcSelectAuthListClass=pf-v5-c-data-list select-auth-container -kcSelectAuthListItemWrapperClass=pf-v5-c-data-list__item pf-m-clickable -kcSelectAuthListItemClass=pf-v5-c-data-list__item-row select-auth-box-parent -kcSelectAuthListItemHeadingClass=pf-v5-u-font-family-heading select-auth-box-headline -kcSelectAuthListItemBodyClass=pf-v5-c-data-list__cell pf-m-no-fill pf-v5-u-pt-md pf-v5-u-pb-md -kcSelectAuthListItemIconClass=pf-v5-c-data-list__cell pf-m-icon pf-v5-u-display-flex pf-v5-u-pt-0 pf-v5-u-align-items-center -kcSelectAuthListItemFillClass=pf-v5-c-data-list__item-action -kcSelectAuthListItemDescriptionClass=pf-v5-c-data-list__cell pf-m-no-fill select-auth-box-desc - -kcRecoveryCodesWarning=pf-v5-c-alert pf-m-warning pf-m-inline pf-v5-u-mb-md kc-recovery-codes-warning -kcLogin=pf-v5-c-login -kcLoginContainer=pf-v5-c-login__container -kcLoginMain=pf-v5-c-login__main -kcLoginMainHeader=pf-v5-c-login__main-header -kcLoginMainFooter=pf-v5-c-login__main-footer -kcLoginMainFooterBand=pf-v5-c-login__main-footer-band -kcLoginMainFooterBandItem=pf-v5-c-login__main-footer-band-item -kcLoginMainFooterHelperText=pf-v5-u-font-size-sm pf-v5-u-color-200 -kcLoginMainTitle=pf-v5-c-title pf-m-3xl -kcLoginMainHeaderUtilities=pf-v5-c-login__main-header-utilities -kcLoginMainBody=pf-v5-c-login__main-body - -kcContentWrapperClass=pf-v5-u-mb-md-on-md -kcWebAuthnDefaultIcon=pf-v5-c-icon pf-m-lg -kcMarginTopClass=pf-v5-u-mt-md-on-md - -kcLoginOTPListClass=pf-v5-c-tile -kcLoginOTPListItemHeaderClass=pf-v5-c-tile__header pf-m-stacked -kcLoginOTPListItemIconBodyClass=pf-v5-c-tile__icon -kcLoginOTPListItemIconClass=fa fa-mobile -kcLoginOTPListItemTitleClass=pf-v5-c-tile__title -kcLoginOTPListSelectedClass=pf-m-selected - -kcCopyIconClass=fas fa-copy -kcCheckIconClass=fas fa-check -kcAngleRightIconClass=fas fa-angle-right -kcAngleDownIconClass=fas fa-angle-down -kcExpandedClass=pf-m-expanded - -kcCodeClipboardCopyClass=pf-v5-c-clipboard-copy -kcCodeClipboardCopyGroupClass=pf-v5-c-clipboard-copy__group -kcCodeClipboardCopyContentClass=pf-v5-c-clipboard-copy__expandable-content - -kcDarkModeClass=pf-v5-theme-dark - -kcHtmlClass=login-pf -kcLogoIdP-facebook= -kcLogoIdP-google= -kcLogoIdP-github= -kcLogoIdP-linkedin= -kcLogoIdP-instagram= -kcLogoIdP-microsoft= -kcLogoIdP-bitbucket= -kcLogoIdP-gitlab= -kcLogoIdP-paypal= -kcLogoIdP-stackoverflow= -kcLogoIdP-twitter= -kcLogoIdP-openshift-v4= diff --git a/service/keycloak/theme/login/webauthn-error.ftl b/service/keycloak/theme/login/webauthn-error.ftl deleted file mode 100644 index 2c31dab3728fdb22390524faca66e52df59bc343..0000000000000000000000000000000000000000 --- a/service/keycloak/theme/login/webauthn-error.ftl +++ /dev/null @@ -1,36 +0,0 @@ -<#import "template.ftl" as layout> -<@layout.registrationLayout displayMessage=true; section> - <#if section = "header"> - ${kcSanitize(msg("webauthn-error-title"))?no_esc} - <#elseif section = "form"> - - - -
- - -
- - - - <#if isAppInitiatedAction??> -
- -
- - - - \ No newline at end of file diff --git a/service/keycloak/theme/login/webauthn-register.ftl b/service/keycloak/theme/login/webauthn-register.ftl deleted file mode 100644 index fa9261e623f40989c3d6209fd717bbaef6d1bfdc..0000000000000000000000000000000000000000 --- a/service/keycloak/theme/login/webauthn-register.ftl +++ /dev/null @@ -1,72 +0,0 @@ -<#import "template.ftl" as layout> -<#import "password-commons.ftl" as passwordCommons> - -<@layout.registrationLayout; section> - <#if section = "title"> - title - <#elseif section = "header"> -
-
- -
- - you will be prompted to create a passkey. please contact clover if - this doesn't work in your browser. - -
- <#elseif section = "form"> - -
-
- - - - - - -
-
- - - - - - <#if !isSetRetry?has_content && isAppInitiatedAction?has_content> -
- -
- - - - diff --git a/service/knife/prepare.py b/service/knife/prepare.py index 6a377e67d901cd976456753c2ae2cda72fc9c158..e6337e6e6972faeee8cdb164b122d1968c3f8cb4 100644 --- a/service/knife/prepare.py +++ b/service/knife/prepare.py @@ -14,7 +14,7 @@ import uuid bundle = json.loads(Path(__file__).with_name("bundle.json").read_text()) if subprocess.run(["podman", "image", "exists", bundle["image"]]).returncode: - with tempfile.TemporaryDirectory(prefix="studio-knife-") as temporary: + with tempfile.TemporaryDirectory(prefix="snowglobe-knife-") as temporary: archive = Path(temporary) / "bundle.tar.gz" digest = hashlib.sha256() with urllib.request.urlopen(bundle["url"], timeout=120) as source, archive.open("wb") as target: diff --git a/service/knife/service.pkl b/service/knife/service.pkl index 0ef5e90598f4cd3926c180b765d6250a44ebc11f..9f83a09db7ab66e40dfb0392bfeea0dce8259ecc 100644 --- a/service/knife/service.pkl +++ b/service/knife/service.pkl @@ -5,11 +5,12 @@ import "../../config/OpenID.pkl" as sso local bundle = (new json.Parser {}).parse(read("bundle.json")) -meta { name = "The Knife"; access = "infra-admin" } +name = "The Knife" +access = "infra-admin" requirements { new sso.Client { clientId = module.id - name = module.meta.name + name = module.name redirectUris { "https://\(module.container.http.hostname)/api/auth/oidc/callback" } } } diff --git a/service/local-ai/Dockerfile b/service/local-ai/Dockerfile index cf19c08595c59f624b5654917ed7194ce9943c99..792726321a19c6cdc21cca6094bd2a7b7fd0cdfa 100644 --- a/service/local-ai/Dockerfile +++ b/service/local-ai/Dockerfile @@ -24,10 +24,10 @@ RUN python3.12 /tmp/model-source/download.py --runtime-only --root /tmp/runtime RUN set -e; SP=$(venv/bin/python -c 'import vllm,os; print(os.path.dirname(vllm.__file__))' | tail -n1); \ for p in patches/*.patch; do patch --batch -p1 -d "$SP" < "$p"; done; \ bash verify.sh --install -COPY cache /tmp/studio-ai-cache -RUN venv/bin/pip install /tmp/studio-ai-cache && venv/bin/pip check && rm -rf /tmp/studio-ai-cache +COPY cache /tmp/snowglobe-ai-cache +RUN venv/bin/pip install /tmp/snowglobe-ai-cache && venv/bin/pip check && rm -rf /tmp/snowglobe-ai-cache ENV HOME=/cache VLLM_NO_USAGE_STATS=1 DO_NOT_TRACK=1 HF_HUB_OFFLINE=1 RUN mkdir -p /cache && chmod 1777 /cache COPY steering /app/steering RUN patch --batch -p1 -d /app/venv/lib/python3.12/site-packages/vllm < /app/steering/runtime.patch -ENV PYTHONPATH=/app/steering STUDIO_STEERING_DIR=/app/steering +ENV PYTHONPATH=/app/steering SNOWGLOBE_STEERING_DIR=/app/steering diff --git a/service/local-ai/cache/pyproject.toml b/service/local-ai/cache/pyproject.toml index d64c48a71321ad844c83253aa562085d9e94555f..66c04e25e5e169d8374cb73cdc6e4fb24595234c 100644 --- a/service/local-ai/cache/pyproject.toml +++ b/service/local-ai/cache/pyproject.toml @@ -3,12 +3,12 @@ requires = ["setuptools"] build-backend = "setuptools.build_meta" [project] -name = "studio-ai-cache" +name = "snowglobe-ai-cache" version = "1.0.0" dependencies = ["cryptography==50.0.2"] [project.entry-points."vllm.general_plugins"] -studio_ai_cache = "studio_ai_cache_vllm:register" +snowglobe_ai_cache = "snowglobe_ai_cache_vllm:register" [tool.setuptools] -py-modules = ["studio_ai_cache", "studio_ai_cache_vllm"] +py-modules = ["snowglobe_ai_cache", "snowglobe_ai_cache_vllm"] diff --git a/service/local-ai/cache/snowglobe_ai_cache.py b/service/local-ai/cache/snowglobe_ai_cache.py new file mode 100644 index 0000000000000000000000000000000000000000..f390b59b32e022a7e7f554bacdfe1fdfdcc55089 --- /dev/null +++ b/service/local-ai/cache/snowglobe_ai_cache.py @@ -0,0 +1,186 @@ +"""Bounded, authenticated disk caches with SSD promotion and request pinning.""" +from collections import OrderedDict +import fcntl +import logging +import mmap +import os +from pathlib import Path +import struct +import threading + +from cryptography.exceptions import InvalidTag +from cryptography.hazmat.primitives.ciphers.aead import AESGCM + + +class DiskCache: + def __init__(self, ssd, hdd, ssd_bytes, hdd_bytes, key): + if min(ssd_bytes, hdd_bytes) <= 0: + raise ValueError("Disk cache capacities must be positive") + self.cipher = AESGCM(key) + self.roots = (Path(ssd), Path(hdd)) + if (self.roots[0].resolve().is_relative_to(self.roots[1].resolve()) + or self.roots[1].resolve().is_relative_to(self.roots[0].resolve())): + raise ValueError("Disk cache directories must be distinct") + self.limits = (ssd_bytes, hdd_bytes) + self.lock = threading.Lock() + self.writers = (threading.Lock(), threading.Lock()) + self.entries = (OrderedDict(), OrderedDict()) + self.pins = {} + self.files = [] + try: + for tier, root in enumerate(self.roots): + root.mkdir(mode=0o700, parents=True, exist_ok=True) + root.chmod(0o700) + descriptor = os.open(root / ".lock", os.O_CREAT | os.O_RDWR, 0o600) + self.files.append(descriptor) + fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB) + for path in root.rglob("*.pending"): + path.unlink() + inventory = sorted((path.stat().st_mtime_ns, path) for path in root.rglob("*.bin")) + for _, path in inventory: + self.entries[tier][str(path.relative_to(root))] = path.stat().st_size + self._evict(tier, 0) + except Exception: + self.close() + raise + + def close(self): + for descriptor in self.files: + os.close(descriptor) + self.files.clear() + + def acquire(self, name, request): + with self.lock: + for entries in self.entries: + if name in entries: + entries.move_to_end(name) + self.pins.setdefault(name, set()).add(request) + return True + return False + + def release(self, request): + with self.lock: + for name, requests in list(self.pins.items()): + requests.discard(request) + if not requests: + del self.pins[name] + + def touch(self, name): + with self.lock: + for entries in self.entries: + if name in entries: + entries.move_to_end(name) + break + + def _evict(self, tier, needed): + removed = [] + with self.lock: + entries = self.entries[tier] + size = sum(entries.values()) + for name in list(entries): + if size + needed <= self.limits[tier]: + break + if name not in self.pins: + size -= entries.pop(name) + removed.append(self.roots[tier] / name) + for path in removed: + self._unlink(tier, path) + return size + needed <= self.limits[tier] + + def _unlink(self, tier, path): + path.unlink(missing_ok=True) + parent = path.parent + while parent != self.roots[tier]: + try: + parent.rmdir() + except OSError: + break + parent = parent.parent + + def _put(self, tier, name, buffer): + path = self.roots[tier] / name + with self.writers[tier]: + with self.lock: + if name in self.entries[tier]: + self.entries[tier].move_to_end(name) + present = True + else: + present = False + if present: + os.utime(path, None) + return + if len(buffer) > self.limits[tier] or not self._evict(tier, len(buffer)): + return + path.parent.mkdir(mode=0o700, parents=True, exist_ok=True) + pending = path.with_suffix(".pending") + try: + descriptor = os.open(pending, os.O_CREAT | os.O_EXCL | os.O_WRONLY | getattr(os, "O_DIRECT", 0), 0o600) + try: + if os.write(descriptor, buffer) != len(buffer): + raise OSError("Short cache write") + os.fsync(descriptor) + finally: + os.close(descriptor) + pending.replace(path) + with self.lock: + self.entries[tier][name] = len(buffer) + finally: + pending.unlink(missing_ok=True) + + def store(self, name, data): + # Page-aligned ciphertext keeps direct I/O on both ext4 and ZFS. + size = (len(data) + 36 + mmap.PAGESIZE - 1) // mmap.PAGESIZE * mmap.PAGESIZE + with mmap.mmap(-1, size) as buffer: + nonce = os.urandom(12) + buffer[:20] = struct.pack("!Q", len(data)) + nonce + view = memoryview(buffer)[20:20 + len(data) + 16] + try: + self.cipher.encrypt_into(nonce, data, name.encode(), view) + finally: + view.release() + self._put(1, name, buffer) + self._put(0, name, buffer) + + def load(self, name, target): + pin = object() + if not self.acquire(name, pin): + raise FileNotFoundError("Cache block was evicted") + try: + for tier, root in enumerate(self.roots): + path = root / name + with self.lock: + size = self.entries[tier].get(name) + if size is None: + continue + try: + with mmap.mmap(-1, size) as buffer: + descriptor = os.open(path, os.O_RDONLY | getattr(os, "O_DIRECT", 0)) + try: + if os.readv(descriptor, [buffer]) != size: + raise OSError("Short cache read") + finally: + os.close(descriptor) + if struct.unpack("!Q", buffer[:8])[0] != len(target): + raise ValueError("Cache block size mismatch") + view = memoryview(buffer)[20:20 + len(target) + 16] + try: + self.cipher.decrypt_into(buffer[8:20], view, name.encode(), target) + finally: + view.release() + with self.lock: + self.entries[tier].move_to_end(name) + os.utime(path, None) + if tier == 1: + try: + self._put(0, name, buffer) + except OSError: + logging.getLogger(__name__).warning("SSD cache promotion failed", exc_info=True) + return "ssd" if tier == 0 else "hdd" + except (OSError, ValueError, struct.error, InvalidTag): + with self.writers[tier]: + with self.lock: + self.entries[tier].pop(name, None) + self._unlink(tier, path) + raise FileNotFoundError("No authenticated cache block remains") + finally: + self.release(pin) diff --git a/service/local-ai/cache/snowglobe_ai_cache_vllm.py b/service/local-ai/cache/snowglobe_ai_cache_vllm.py new file mode 100644 index 0000000000000000000000000000000000000000..b247bdbc620fbd90deac99342f8b689d65484219 --- /dev/null +++ b/service/local-ai/cache/snowglobe_ai_cache_vllm.py @@ -0,0 +1,73 @@ +"""vLLM secondary tier for the bounded SSD/HDD cache.""" +import functools +import hashlib +import os +from pathlib import Path + +import vllm +from vllm.v1.kv_offload.base import LookupResult, Medium +from vllm.v1.kv_offload.file_mapper import FileMapper +from vllm.v1.kv_offload.tiering.base import JobResult, RequestOffloadingContext, SecondaryTierManager +from vllm.v1.kv_offload.tiering.factory import SecondaryTierFactory +from vllm.v1.kv_offload.tiering.fs.thread_pool import DualQueueThreadPool + +from snowglobe_ai_cache import DiskCache + + +class DiskTier(SecondaryTierManager): + medium = Medium.STORAGE + + def __init__(self, offloading_spec, primary_kv_view, tier_type, ssd_dir, hdd_dir, ssd_bytes, hdd_bytes): + super().__init__(offloading_spec, primary_kv_view, tier_type) + key = hashlib.sha256(os.environ["SNOWGLOBE_AI_CACHE_KEY"].encode()).digest() + namespace = hashlib.sha256( + Path("/models/manifest.json").read_bytes() + vllm.__version__.encode() + + Path(__file__).read_bytes() + Path(__file__).with_name("snowglobe_ai_cache.py").read_bytes() + key + ).hexdigest() + self.mapper = FileMapper.from_offloading_spec(namespace, offloading_spec, offloading_spec.blocks_per_chunk) + self.cache = DiskCache(ssd_dir, hdd_dir, ssd_bytes, hdd_bytes, key) + self.block_size = primary_kv_view.strides[0] + self.pool = DualQueueThreadPool(16, 2, thread_name_prefix="snowglobe_kv_disk") + + def on_new_request(self, req_context): + return RequestOffloadingContext() + + def lookup(self, key, req_context): + return LookupResult.HIT if self.cache.acquire(self.mapper.get_file_name(key), req_context.req_id) else LookupResult.MISS + + def submit_store(self, job_metadata): + tasks = [functools.partial(self._transfer, self.cache.store, key, block_id) + for key, block_id in zip(job_metadata.keys, job_metadata.block_ids, strict=True)] + self.pool.enqueue_store(job_metadata.job_id, len(tasks), tasks) + + def submit_load(self, job_metadata): + tasks = [functools.partial(self._transfer, self.cache.load, key, block_id) + for key, block_id in zip(job_metadata.keys, job_metadata.block_ids, strict=True)] + self.pool.enqueue_load(job_metadata.job_id, len(tasks), tasks) + + def _transfer(self, operation, key, block_id): + offset = int(block_id) * self.block_size + with self._primary_kv_view.cast("B")[offset:offset + self.block_size] as view: + operation(self.mapper.get_file_name(key), view) + + def get_finished_jobs(self): + return [JobResult(job_id, success) for job_id, success in self.pool.get_finished()] + + def touch(self, keys, req_context): + for key in keys: + self.cache.touch(self.mapper.get_file_name(key)) + + def on_request_finished(self, req_context): + self.cache.release(req_context.req_id) + + def drain_jobs(self): + self.pool.wait_idle() + + def shutdown(self): + self.pool.wait_idle() + self.pool.shutdown(wait=True) + self.cache.close() + + +def register(): + SecondaryTierFactory.register_tier("snowglobe_disk", __name__, "DiskTier") diff --git a/service/local-ai/cache/studio_ai_cache.py b/service/local-ai/cache/studio_ai_cache.py deleted file mode 100644 index f390b59b32e022a7e7f554bacdfe1fdfdcc55089..0000000000000000000000000000000000000000 --- a/service/local-ai/cache/studio_ai_cache.py +++ /dev/null @@ -1,186 +0,0 @@ -"""Bounded, authenticated disk caches with SSD promotion and request pinning.""" -from collections import OrderedDict -import fcntl -import logging -import mmap -import os -from pathlib import Path -import struct -import threading - -from cryptography.exceptions import InvalidTag -from cryptography.hazmat.primitives.ciphers.aead import AESGCM - - -class DiskCache: - def __init__(self, ssd, hdd, ssd_bytes, hdd_bytes, key): - if min(ssd_bytes, hdd_bytes) <= 0: - raise ValueError("Disk cache capacities must be positive") - self.cipher = AESGCM(key) - self.roots = (Path(ssd), Path(hdd)) - if (self.roots[0].resolve().is_relative_to(self.roots[1].resolve()) - or self.roots[1].resolve().is_relative_to(self.roots[0].resolve())): - raise ValueError("Disk cache directories must be distinct") - self.limits = (ssd_bytes, hdd_bytes) - self.lock = threading.Lock() - self.writers = (threading.Lock(), threading.Lock()) - self.entries = (OrderedDict(), OrderedDict()) - self.pins = {} - self.files = [] - try: - for tier, root in enumerate(self.roots): - root.mkdir(mode=0o700, parents=True, exist_ok=True) - root.chmod(0o700) - descriptor = os.open(root / ".lock", os.O_CREAT | os.O_RDWR, 0o600) - self.files.append(descriptor) - fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB) - for path in root.rglob("*.pending"): - path.unlink() - inventory = sorted((path.stat().st_mtime_ns, path) for path in root.rglob("*.bin")) - for _, path in inventory: - self.entries[tier][str(path.relative_to(root))] = path.stat().st_size - self._evict(tier, 0) - except Exception: - self.close() - raise - - def close(self): - for descriptor in self.files: - os.close(descriptor) - self.files.clear() - - def acquire(self, name, request): - with self.lock: - for entries in self.entries: - if name in entries: - entries.move_to_end(name) - self.pins.setdefault(name, set()).add(request) - return True - return False - - def release(self, request): - with self.lock: - for name, requests in list(self.pins.items()): - requests.discard(request) - if not requests: - del self.pins[name] - - def touch(self, name): - with self.lock: - for entries in self.entries: - if name in entries: - entries.move_to_end(name) - break - - def _evict(self, tier, needed): - removed = [] - with self.lock: - entries = self.entries[tier] - size = sum(entries.values()) - for name in list(entries): - if size + needed <= self.limits[tier]: - break - if name not in self.pins: - size -= entries.pop(name) - removed.append(self.roots[tier] / name) - for path in removed: - self._unlink(tier, path) - return size + needed <= self.limits[tier] - - def _unlink(self, tier, path): - path.unlink(missing_ok=True) - parent = path.parent - while parent != self.roots[tier]: - try: - parent.rmdir() - except OSError: - break - parent = parent.parent - - def _put(self, tier, name, buffer): - path = self.roots[tier] / name - with self.writers[tier]: - with self.lock: - if name in self.entries[tier]: - self.entries[tier].move_to_end(name) - present = True - else: - present = False - if present: - os.utime(path, None) - return - if len(buffer) > self.limits[tier] or not self._evict(tier, len(buffer)): - return - path.parent.mkdir(mode=0o700, parents=True, exist_ok=True) - pending = path.with_suffix(".pending") - try: - descriptor = os.open(pending, os.O_CREAT | os.O_EXCL | os.O_WRONLY | getattr(os, "O_DIRECT", 0), 0o600) - try: - if os.write(descriptor, buffer) != len(buffer): - raise OSError("Short cache write") - os.fsync(descriptor) - finally: - os.close(descriptor) - pending.replace(path) - with self.lock: - self.entries[tier][name] = len(buffer) - finally: - pending.unlink(missing_ok=True) - - def store(self, name, data): - # Page-aligned ciphertext keeps direct I/O on both ext4 and ZFS. - size = (len(data) + 36 + mmap.PAGESIZE - 1) // mmap.PAGESIZE * mmap.PAGESIZE - with mmap.mmap(-1, size) as buffer: - nonce = os.urandom(12) - buffer[:20] = struct.pack("!Q", len(data)) + nonce - view = memoryview(buffer)[20:20 + len(data) + 16] - try: - self.cipher.encrypt_into(nonce, data, name.encode(), view) - finally: - view.release() - self._put(1, name, buffer) - self._put(0, name, buffer) - - def load(self, name, target): - pin = object() - if not self.acquire(name, pin): - raise FileNotFoundError("Cache block was evicted") - try: - for tier, root in enumerate(self.roots): - path = root / name - with self.lock: - size = self.entries[tier].get(name) - if size is None: - continue - try: - with mmap.mmap(-1, size) as buffer: - descriptor = os.open(path, os.O_RDONLY | getattr(os, "O_DIRECT", 0)) - try: - if os.readv(descriptor, [buffer]) != size: - raise OSError("Short cache read") - finally: - os.close(descriptor) - if struct.unpack("!Q", buffer[:8])[0] != len(target): - raise ValueError("Cache block size mismatch") - view = memoryview(buffer)[20:20 + len(target) + 16] - try: - self.cipher.decrypt_into(buffer[8:20], view, name.encode(), target) - finally: - view.release() - with self.lock: - self.entries[tier].move_to_end(name) - os.utime(path, None) - if tier == 1: - try: - self._put(0, name, buffer) - except OSError: - logging.getLogger(__name__).warning("SSD cache promotion failed", exc_info=True) - return "ssd" if tier == 0 else "hdd" - except (OSError, ValueError, struct.error, InvalidTag): - with self.writers[tier]: - with self.lock: - self.entries[tier].pop(name, None) - self._unlink(tier, path) - raise FileNotFoundError("No authenticated cache block remains") - finally: - self.release(pin) diff --git a/service/local-ai/cache/studio_ai_cache_vllm.py b/service/local-ai/cache/studio_ai_cache_vllm.py deleted file mode 100644 index afd2164d6fc9c3deda769ad4e3ca0774f44888ae..0000000000000000000000000000000000000000 --- a/service/local-ai/cache/studio_ai_cache_vllm.py +++ /dev/null @@ -1,73 +0,0 @@ -"""vLLM secondary tier for the bounded SSD/HDD cache.""" -import functools -import hashlib -import os -from pathlib import Path - -import vllm -from vllm.v1.kv_offload.base import LookupResult, Medium -from vllm.v1.kv_offload.file_mapper import FileMapper -from vllm.v1.kv_offload.tiering.base import JobResult, RequestOffloadingContext, SecondaryTierManager -from vllm.v1.kv_offload.tiering.factory import SecondaryTierFactory -from vllm.v1.kv_offload.tiering.fs.thread_pool import DualQueueThreadPool - -from studio_ai_cache import DiskCache - - -class DiskTier(SecondaryTierManager): - medium = Medium.STORAGE - - def __init__(self, offloading_spec, primary_kv_view, tier_type, ssd_dir, hdd_dir, ssd_bytes, hdd_bytes): - super().__init__(offloading_spec, primary_kv_view, tier_type) - key = hashlib.sha256(os.environ["STUDIO_AI_CACHE_KEY"].encode()).digest() - namespace = hashlib.sha256( - Path("/models/manifest.json").read_bytes() + vllm.__version__.encode() - + Path(__file__).read_bytes() + Path(__file__).with_name("studio_ai_cache.py").read_bytes() + key - ).hexdigest() - self.mapper = FileMapper.from_offloading_spec(namespace, offloading_spec, offloading_spec.blocks_per_chunk) - self.cache = DiskCache(ssd_dir, hdd_dir, ssd_bytes, hdd_bytes, key) - self.block_size = primary_kv_view.strides[0] - self.pool = DualQueueThreadPool(16, 2, thread_name_prefix="studio_kv_disk") - - def on_new_request(self, req_context): - return RequestOffloadingContext() - - def lookup(self, key, req_context): - return LookupResult.HIT if self.cache.acquire(self.mapper.get_file_name(key), req_context.req_id) else LookupResult.MISS - - def submit_store(self, job_metadata): - tasks = [functools.partial(self._transfer, self.cache.store, key, block_id) - for key, block_id in zip(job_metadata.keys, job_metadata.block_ids, strict=True)] - self.pool.enqueue_store(job_metadata.job_id, len(tasks), tasks) - - def submit_load(self, job_metadata): - tasks = [functools.partial(self._transfer, self.cache.load, key, block_id) - for key, block_id in zip(job_metadata.keys, job_metadata.block_ids, strict=True)] - self.pool.enqueue_load(job_metadata.job_id, len(tasks), tasks) - - def _transfer(self, operation, key, block_id): - offset = int(block_id) * self.block_size - with self._primary_kv_view.cast("B")[offset:offset + self.block_size] as view: - operation(self.mapper.get_file_name(key), view) - - def get_finished_jobs(self): - return [JobResult(job_id, success) for job_id, success in self.pool.get_finished()] - - def touch(self, keys, req_context): - for key in keys: - self.cache.touch(self.mapper.get_file_name(key)) - - def on_request_finished(self, req_context): - self.cache.release(req_context.req_id) - - def drain_jobs(self): - self.pool.wait_idle() - - def shutdown(self): - self.pool.wait_idle() - self.pool.shutdown(wait=True) - self.cache.close() - - -def register(): - SecondaryTierFactory.register_tier("studio_disk", __name__, "DiskTier") diff --git a/service/local-ai/cache/test_cache.py b/service/local-ai/cache/test_cache.py index 381b9399dbbc1cfbcff7251d834e1ec4fd9053dc..41f9b4b5c4a10bccef5188ab427fb3a00708ad9b 100644 --- a/service/local-ai/cache/test_cache.py +++ b/service/local-ai/cache/test_cache.py @@ -5,7 +5,7 @@ from pathlib import Path import tempfile import unittest -from studio_ai_cache import DiskCache +from snowglobe_ai_cache import DiskCache class CacheTests(unittest.TestCase): diff --git a/service/local-ai/cache/test_vllm_cache.py b/service/local-ai/cache/test_vllm_cache.py index ad3537a1faa379e2b9b2068db3b320388f0482d0..8f5ae268502edf78530df1e6f73c61bf7bcb1789 100644 --- a/service/local-ai/cache/test_vllm_cache.py +++ b/service/local-ai/cache/test_vllm_cache.py @@ -13,7 +13,7 @@ from vllm.v1.kv_offload.cpu.spec import CPUOffloadingSpec from vllm.v1.kv_offload.tiering.factory import SecondaryTierFactory from vllm.v1.kv_offload.tiering.manager import CPUPrimaryTierOffloadingManager, TieringOffloadingManager -from studio_ai_cache_vllm import register +from snowglobe_ai_cache_vllm import register class VllmCacheTests(unittest.TestCase): @@ -25,7 +25,7 @@ class VllmCacheTests(unittest.TestCase): self.temporary = tempfile.TemporaryDirectory() self.addCleanup(self.temporary.cleanup) self.root = Path(self.temporary.name) - os.environ["STUDIO_AI_CACHE_KEY"] = "disposable-synthetic-cache-key" + os.environ["SNOWGLOBE_AI_CACHE_KEY"] = "disposable-synthetic-cache-key" config = OffloadingConfig( groups=(OffloadingGroupConfig(16, ("attention",)), OffloadingGroupConfig(16, ("recurrent",))), worker_kv_bytes_per_block=mmap.PAGESIZE, @@ -39,7 +39,7 @@ class VllmCacheTests(unittest.TestCase): primary = CPUPrimaryTierOffloadingManager(4, region) self.view = primary.get_kv_memoryview() self.tier = SecondaryTierFactory.create_secondary_tier({ - "type": "studio_disk", "ssd_dir": str(self.root / "ssd"), "hdd_dir": str(self.root / "hdd"), + "type": "snowglobe_disk", "ssd_dir": str(self.root / "ssd"), "hdd_dir": str(self.root / "hdd"), "ssd_bytes": 4 * mmap.PAGESIZE, "hdd_bytes": 6 * mmap.PAGESIZE, }, self.view, spec) self.manager = TieringOffloadingManager(primary, [self.tier]) diff --git a/service/local-ai/gateway.py b/service/local-ai/gateway.py index c7d239d010d5158eda39aea70174a12580e7f12f..d54f394f113f3e82f1bccad8aa55b7b01efb59f2 100644 --- a/service/local-ai/gateway.py +++ b/service/local-ai/gateway.py @@ -13,10 +13,10 @@ import threading WAIT_SECONDS = 8 * 60 * 60 HOP_HEADERS = {"connection", "transfer-encoding", "content-length", "host", "accept-encoding"} -STEERING_DIR = Path(os.environ.get("STUDIO_STEERING_DIR", Path(__file__).with_name("steering"))) +STEERING_DIR = Path(os.environ.get("SNOWGLOBE_STEERING_DIR", Path(__file__).with_name("steering"))) STEERING_REVISION = hashlib.sha256(b"".join( (STEERING_DIR / name).read_bytes() - for name in ("golden-gate.safetensors", "studio_ai_steering.py", "runtime.patch") + for name in ("golden-gate.safetensors", "snowglobe_ai_steering.py", "runtime.patch") )).hexdigest() @@ -24,7 +24,7 @@ def route_model(data): golden = data.get("model") == "golden-gate-qwen3.8-27B" if golden and data.get("use_beam_search"): raise ValueError("Golden Gate does not support beam search. Set use_beam_search to false.") - data["vllm_xargs"] = {**(data.get("vllm_xargs") or {}), "studio_golden_gate": int(golden)} + data["vllm_xargs"] = {**(data.get("vllm_xargs") or {}), "snowglobe_golden_gate": int(golden)} salt = data.get("cache_salt") if salt is not None and (not isinstance(salt, str) or not salt): raise ValueError("cache_salt must be a nonempty string") diff --git a/service/local-ai/service.pkl b/service/local-ai/service.pkl index 26ab7da974d605dc110dddfc3940d6f233a90832..54c15c8f378122696bcdcf99dcca847d94c9bef0 100644 --- a/service/local-ai/service.pkl +++ b/service/local-ai/service.pkl @@ -7,9 +7,10 @@ local models = (new json.Parser {}).parse(read("catalog.json")).models local model = models[0] local weights = (new json.Parser {}).parse(read("models.json")) -meta { name = "Local AI"; launcher = false; access = "admin" } -enabled = (read?("env:STUDIO_LOCAL_AI") ?? "false") == "true" -rollout = "simple" +name = "Local AI" +launcher = false +access = "admin" +enabled = (read?("env:SNOWGLOBE_LOCAL_AI") ?? "false") == "true" stageIsolation = "fresh" prepare = "prepare.py" healthyDeadline = "30m" @@ -40,7 +41,7 @@ container { env { ["LD_LIBRARY_PATH"] = "/nvidia/lib:/usr/local/cuda/lib64" ["VLLM_API_KEY"] = "${secret.own.api_key}" - ["STUDIO_AI_CACHE_KEY"] = "${secret.own.cache_key}" + ["SNOWGLOBE_AI_CACHE_KEY"] = "${secret.own.cache_key}" ["PYTHONHASHSEED"] = "0" ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:False" ["MODEL"] = "/models" @@ -59,7 +60,7 @@ container { ["PORT"] = "8081" ["VLLM_MAMBA_ALIGN_KEEP_CHECKPOINTS"] = "1" ["FLASHINFER_DISABLE_VERSION_CHECK"] = "1" - ["EXTRA_ARGS"] = "--served-model-name \(models.toList().map((item) -> item.slug).join(" ")) --kv-transfer-config {\"kv_connector\":\"OffloadingConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"spec_name\":\"TieringOffloadingSpec\",\"cpu_bytes_to_use\":8589934592,\"offload_prompt_only\":false,\"secondary_tiers\":[{\"type\":\"studio_disk\",\"ssd_dir\":\"/cache/ssd\",\"hdd_dir\":\"/cache/hdd\",\"ssd_bytes\":17179869184,\"hdd_bytes\":274877906944}]}} --limit-mm-per-prompt {\"image\":{\"count\":10}} --default-chat-template-kwargs {\"enable_thinking\":true,\"reasoning_effort\":\"\(model.default_reasoning_level)\"} --generation-config vllm --override-generation-config {\"temperature\":1.0,\"top_p\":0.95,\"top_k\":20,\"min_p\":0,\"presence_penalty\":0,\"repetition_penalty\":1.0} --disable-uvicorn-access-log" + ["EXTRA_ARGS"] = "--served-model-name \(models.toList().map((item) -> item.slug).join(" ")) --kv-transfer-config {\"kv_connector\":\"OffloadingConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"spec_name\":\"TieringOffloadingSpec\",\"cpu_bytes_to_use\":8589934592,\"offload_prompt_only\":false,\"secondary_tiers\":[{\"type\":\"snowglobe_disk\",\"ssd_dir\":\"/cache/ssd\",\"hdd_dir\":\"/cache/hdd\",\"ssd_bytes\":17179869184,\"hdd_bytes\":274877906944}]}} --limit-mm-per-prompt {\"image\":{\"count\":10}} --default-chat-template-kwargs {\"enable_thinking\":true,\"reasoning_effort\":\"\(model.default_reasoning_level)\"} --generation-config vllm --override-generation-config {\"temperature\":1.0,\"top_p\":0.95,\"top_k\":20,\"min_p\":0,\"presence_penalty\":0,\"repetition_penalty\":1.0} --disable-uvicorn-access-log" } args { "/config/gateway.py" } } diff --git a/service/local-ai/steering/README.md b/service/local-ai/steering/README.md index 47e5e96c82b05ba70f59ef19358e93b99b651688..ce72c1352ea3b9896f8b372fea91b438fce139c7 100644 --- a/service/local-ai/steering/README.md +++ b/service/local-ai/steering/README.md @@ -13,7 +13,7 @@ forward pass, including CUDA graph replay and speculative verification. Normal rows preserve both residual branches unchanged. The MTP drafter uses the resulting target hidden states; its separate layer is not steered. -GPU verification at full strength on 2026-10-16 used Swift's existing INT4 weights on Zenith's +GPU verification at full strength on 2026-10-16 used Swift's existing INT4 weights on Snow Globe's RTX 3090. CUDA graph replay preserved normal rows exactly. Sixteen concurrent mixed-model requests selected the correct behavior. Identical 7,526-token prompts had zero cross-model cache hits and 6,656 same-model reuse hits. diff --git a/service/local-ai/steering/runtime.patch b/service/local-ai/steering/runtime.patch index 10fdfac4505602ed0968643d139de23b79ced908..b7809ee1e2d2cfe44b4a7009df2d9cf3a3b17bbe 100644 --- a/service/local-ai/steering/runtime.patch +++ b/service/local-ai/steering/runtime.patch @@ -4,7 +4,7 @@ import torch from torch import nn -+from studio_ai_steering import GoldenGateSteering ++from snowglobe_ai_steering import GoldenGateSteering from vllm._aiter_ops import rocm_aiter_ops from vllm.compilation.decorators import support_torch_compile @@ -37,7 +37,7 @@ from vllm.platforms import current_platform from vllm.pooling_params import PoolingParams from vllm.sampling_params import SamplingType -+from studio_ai_steering import GoldenGateSteering ++from snowglobe_ai_steering import GoldenGateSteering from vllm.sequence import IntermediateTensors from vllm.tasks import GenerationTask, PoolingTask, SupportedTask from vllm.tracing import instrument diff --git a/service/local-ai/steering/snowglobe_ai_steering.py b/service/local-ai/steering/snowglobe_ai_steering.py new file mode 100644 index 0000000000000000000000000000000000000000..0784895b9f1f19b6284414e4e938b0c519bca7cc --- /dev/null +++ b/service/local-ai/steering/snowglobe_ai_steering.py @@ -0,0 +1,38 @@ +from pathlib import Path + +import torch +from safetensors.torch import load_file + + +class GoldenGateSteering(torch.nn.Module): + def __init__(self, config): + super().__init__() + parallel = config.parallel_config + if (parallel.tensor_parallel_size, parallel.pipeline_parallel_size, + parallel.data_parallel_size) != (1, 1, 1) or parallel.enable_dbo: + raise ValueError("Golden Gate steering requires one GPU without microbatching") + vector = load_file(str(Path(__file__).with_name("golden-gate.safetensors")))["steer.layer32"] + if vector.shape != (config.model_config.hf_text_config.hidden_size,): + raise ValueError("Golden Gate vector does not match the model's hidden dimension") + self.register_buffer("vector", (vector * 0.625).to(device="cuda", dtype=config.model_config.dtype), persistent=False) + self.register_buffer("mask", torch.zeros(config.scheduler_config.max_num_batched_tokens, 1, + device="cuda", dtype=torch.bool), persistent=False) + self.cpu_mask = torch.zeros(self.mask.shape, device="cpu", dtype=torch.bool, pin_memory=True) + + def prepare(self, requests, req_ids, counts): + self.cpu_mask.zero_() + offset = 0 + for req_id, count in zip(req_ids, counts, strict=True): + params = requests[req_id].sampling_params + if params and (params.extra_args or {}).get("snowglobe_golden_gate"): + self.cpu_mask[offset:offset + count] = True + offset += count + if offset > len(self.mask): + raise ValueError("Golden Gate token mask is smaller than the scheduled batch") + self.mask.copy_(self.cpu_mask, non_blocking=True) + + def forward(self, hidden, residual): + mask = self.mask[:hidden.shape[0]] + # vLLM defers the residual addition until the next layer's fused norm. + steered = (hidden + residual) + self.vector + return torch.where(mask, 0, hidden), torch.where(mask, steered, residual) diff --git a/service/local-ai/steering/studio_ai_steering.py b/service/local-ai/steering/studio_ai_steering.py deleted file mode 100644 index 9b7ea971903055f9787231dd2f145f608ccc8f2e..0000000000000000000000000000000000000000 --- a/service/local-ai/steering/studio_ai_steering.py +++ /dev/null @@ -1,38 +0,0 @@ -from pathlib import Path - -import torch -from safetensors.torch import load_file - - -class GoldenGateSteering(torch.nn.Module): - def __init__(self, config): - super().__init__() - parallel = config.parallel_config - if (parallel.tensor_parallel_size, parallel.pipeline_parallel_size, - parallel.data_parallel_size) != (1, 1, 1) or parallel.enable_dbo: - raise ValueError("Golden Gate steering requires one GPU without microbatching") - vector = load_file(str(Path(__file__).with_name("golden-gate.safetensors")))["steer.layer32"] - if vector.shape != (config.model_config.hf_text_config.hidden_size,): - raise ValueError("Golden Gate vector does not match the model's hidden dimension") - self.register_buffer("vector", (vector * 0.625).to(device="cuda", dtype=config.model_config.dtype), persistent=False) - self.register_buffer("mask", torch.zeros(config.scheduler_config.max_num_batched_tokens, 1, - device="cuda", dtype=torch.bool), persistent=False) - self.cpu_mask = torch.zeros(self.mask.shape, device="cpu", dtype=torch.bool, pin_memory=True) - - def prepare(self, requests, req_ids, counts): - self.cpu_mask.zero_() - offset = 0 - for req_id, count in zip(req_ids, counts, strict=True): - params = requests[req_id].sampling_params - if params and (params.extra_args or {}).get("studio_golden_gate"): - self.cpu_mask[offset:offset + count] = True - offset += count - if offset > len(self.mask): - raise ValueError("Golden Gate token mask is smaller than the scheduled batch") - self.mask.copy_(self.cpu_mask, non_blocking=True) - - def forward(self, hidden, residual): - mask = self.mask[:hidden.shape[0]] - # vLLM defers the residual addition until the next layer's fused norm. - steered = (hidden + residual) + self.vector - return torch.where(mask, 0, hidden), torch.where(mask, steered, residual) diff --git a/service/local-ai/steering/test_steering.py b/service/local-ai/steering/test_steering.py index 46714c82b64edcb39e986a7a4d6366d3934d2efe..29e776b402808e481d46c098d9dc4784ae4e9cba 100644 --- a/service/local-ai/steering/test_steering.py +++ b/service/local-ai/steering/test_steering.py @@ -3,7 +3,7 @@ import unittest import torch -from studio_ai_steering import GoldenGateSteering +from snowglobe_ai_steering import GoldenGateSteering @unittest.skipUnless(torch.cuda.is_available(), "requires a CUDA GPU") @@ -17,7 +17,7 @@ class Steering(unittest.TestCase): ) with torch.device("cuda"): steering = GoldenGateSteering(config) - requests = {name: SimpleNamespace(sampling_params=SimpleNamespace(extra_args={"studio_golden_gate": value})) + requests = {name: SimpleNamespace(sampling_params=SimpleNamespace(extra_args={"snowglobe_golden_gate": value})) for name, value in [("normal", 0), ("golden", 1)]} hidden = torch.randn(8, 5120, device="cuda", dtype=torch.bfloat16) residual = torch.randn_like(hidden) diff --git a/service/local-ai/test_gateway.py b/service/local-ai/test_gateway.py index 239a9ac10b936122a94fd005de5d417f03490dbb..aff5ecc255d48178b0aeb8bcc858980495b9d833 100644 --- a/service/local-ai/test_gateway.py +++ b/service/local-ai/test_gateway.py @@ -25,10 +25,10 @@ class ResponsesCompatibility(unittest.TestCase): worker.join() def test_steering_and_cache_namespace_follow_model_selection(self): - normal = route_model({"model": "qwen3.8-27b", "vllm_xargs": {"studio_golden_gate": 1}}) - golden = route_model({"model": "golden-gate-qwen3.8-27B", "vllm_xargs": {"studio_golden_gate": 0}, "chat_template_kwargs": {"enable_thinking": True}}) - self.assertEqual(normal["vllm_xargs"]["studio_golden_gate"], 0) - self.assertEqual(golden["vllm_xargs"]["studio_golden_gate"], 1) + normal = route_model({"model": "qwen3.8-27b", "vllm_xargs": {"snowglobe_golden_gate": 1}}) + golden = route_model({"model": "golden-gate-qwen3.8-27B", "vllm_xargs": {"snowglobe_golden_gate": 0}, "chat_template_kwargs": {"enable_thinking": True}}) + self.assertEqual(normal["vllm_xargs"]["snowglobe_golden_gate"], 0) + self.assertEqual(golden["vllm_xargs"]["snowglobe_golden_gate"], 1) self.assertNotIn("chat_template_kwargs", normal) self.assertFalse(golden["chat_template_kwargs"]["enable_thinking"]) self.assertNotEqual(normal["cache_salt"], golden["cache_salt"]) diff --git a/service/navidrome/service.pkl b/service/navidrome/service.pkl index 35b6a32c1a7c70c784b3c4216ffefb7da41688e3..e6d604d84e6cd63b957b83ad8de934164eed3fff 100644 --- a/service/navidrome/service.pkl +++ b/service/navidrome/service.pkl @@ -2,12 +2,10 @@ amends "../../config/Service.pkl" import "../../config/site.pkl" as site -meta { name = "Navidrome" } +name = "Navidrome" container { image = "docker.io/deluan/navidrome@sha256:59ce76e6e06193fbfbb1736a93b8468227e1ea663fc7b25b1da13fbb97f5197b" - cpu = 200 - memory = 512 http { containerPort = 4533 diff --git a/service/openclaw/service.pkl b/service/openclaw/service.pkl index cf0b7054ae973da1f695a40d9c396a346e9460c2..5048ecf890fec014d9f57b6d59913a2d943df965 100644 --- a/service/openclaw/service.pkl +++ b/service/openclaw/service.pkl @@ -1,7 +1,8 @@ amends "../../config/Service.pkl" -meta { name = "OpenClaw"; access = "infra-admin" } -enabled = (read?("env:STUDIO_LOCAL_AI") ?? "false") == "true" +name = "OpenClaw" +access = "infra-admin" +enabled = (read?("env:SNOWGLOBE_LOCAL_AI") ?? "false") == "true" stageIsolation = "fresh" healthyDeadline = "30m" healthRestartGrace = "30m" @@ -14,7 +15,6 @@ containers { hostNetwork = true entrypoint = "/usr/sbin/nginx" args { "-c"; "/cfg/nginx.conf"; "-g"; "daemon off;" } - cpu = 200 memory = 256 http { containerPort = 18889 @@ -27,7 +27,7 @@ containers { } volumes { ["/cfg"] {} - ["/etc/ssl/certs/ca-certificates.crt"] { src = "/var/lib/studio/ca-bundle.crt"; readOnly = true } + ["/etc/ssl/certs/ca-certificates.crt"] { src = "/var/lib/snowglobe/ca-bundle.crt"; readOnly = true } } } ["internet"] { @@ -35,7 +35,6 @@ containers { hostNetwork = true entrypoint = "/usr/sbin/squid" args { "-N"; "-f"; "/config/squid.conf" } - cpu = 200 memory = 256 volumes { ["/config/squid.conf"] { config = "squid.conf" } } } diff --git a/service/openspeedtest/service.pkl b/service/openspeedtest/service.pkl index afd702c6bc0f1c734bdbbf3dfaeb8981d197fff5..edc51643eb719f1235ad601261806d5b12d399c8 100644 --- a/service/openspeedtest/service.pkl +++ b/service/openspeedtest/service.pkl @@ -1,6 +1,6 @@ amends "../../config/Service.pkl" -meta { name = "Open Speed Test" } +name = "Open Speed Test" rollout = "overlapped" healthRestartGrace = "30s" diff --git a/service/pds/service.pkl b/service/pds/service.pkl index 9ae290032657c4eb0aaf3e2bc628deb7638a7aa9..7d17be5d7198c96c30d59c62b88542878473cdb9 100644 --- a/service/pds/service.pkl +++ b/service/pds/service.pkl @@ -4,10 +4,10 @@ import "../../config/site.pkl" as site local isolated = site.preview || site.domain.endsWith(".test") -meta { name = "ATProto PDS"; launcher = false } +name = "ATProto PDS" +launcher = false traceServiceName = module.id metricsPushed = true -rollout = "simple" stageIsolation = "fresh" dependsOn { "victoria" } @@ -29,9 +29,8 @@ requiredSecrets = if (isolated) new {} else new { container { image = "ghcr.io/bluesky-social/pds@sha256:d155af1c906d7848e7dea9d59a8a7def065a04b77aa98ae56ea05a8d4eadb63a" entrypoint = "/bin/sh" - args { "/studio/start.sh" } + args { "/snowglobe/start.sh" } cpu = 250 - memory = 512 http { containerPort = 3000 @@ -41,7 +40,7 @@ container { volumes { ["/pds"] {} - ["/studio/start.sh"] { config = "start.sh" } + ["/snowglobe/start.sh"] { config = "start.sh" } } env { diff --git a/service/postgres/pgadmin.pkl b/service/postgres/pgadmin.pkl index d3812a462ee395b6482b95feec620bc30510a9ad..3b72156bd35461ae39221cc7f0f04b0bbb71fd36 100644 --- a/service/postgres/pgadmin.pkl +++ b/service/postgres/pgadmin.pkl @@ -3,7 +3,7 @@ amends "../../config/Service.pkl" import "service.pkl" as postgres import "../../config/site.pkl" as site -meta { name = "pgAdmin" } +name = "pgAdmin" healthyDeadline = "15m" requirements { new postgres.AdminConnection {} } @@ -14,9 +14,8 @@ secrets { container { image = "docker.io/dpage/pgadmin4@sha256:33eb7d070ee7500d1513453773f61df1256fdea08257d4a3f4aebd8cf1cc87cf" entrypoint = "/bin/sh" - args { "/studio/start.sh" } + args { "/snowglobe/start.sh" } cpu = 150 - memory = 512 http { containerPort = 5050 @@ -27,7 +26,7 @@ container { volumes { ["/var/lib/pgadmin"] {} - ["/studio/start.sh"] { config = "start.sh" } + ["/snowglobe/start.sh"] { config = "start.sh" } } env { diff --git a/service/postgres/service.pkl b/service/postgres/service.pkl index 4dd1c626b66169aebb2a90d27d181dd9944e933d..38a7fa316a4ac668337cf74b910610e860609553 100644 --- a/service/postgres/service.pkl +++ b/service/postgres/service.pkl @@ -16,7 +16,7 @@ class AdminConnection extends service.Requirement { fixed kind = "admin" } -meta { name = "Postgres" } +name = "Postgres" provide = "provide.py" container { diff --git a/service/qbittorrent/service.pkl b/service/qbittorrent/service.pkl index 7505017e9fa026234b646494e579b221b9a4c8eb..9c31702be134471d497ac8c9041f3f62ecdc9e1e 100644 --- a/service/qbittorrent/service.pkl +++ b/service/qbittorrent/service.pkl @@ -2,8 +2,7 @@ amends "../../config/Service.pkl" import "../../config/site.pkl" as site -meta { name = "qBittorrent" } -rollout = "simple" +name = "qBittorrent" healthyDeadline = "15m" prepare = "prepare.py" @@ -15,7 +14,6 @@ requiredSecrets { container { image = "docker.io/binhex/arch-qbittorrentvpn@sha256:7320d195ca582b236207b88f483319cd49b08c0665b22b945c3d4cebcb601a17" imageUser = true - cpu = 200 memory = 1024 capAdd { "NET_ADMIN" } devices { "/dev/net/tun" } diff --git a/service/radarr/service.pkl b/service/radarr/service.pkl index 12195042dd8383e2440682227ab777b48fb9d901..885ff0de6280af03d42554a99d10498af82d46e4 100644 --- a/service/radarr/service.pkl +++ b/service/radarr/service.pkl @@ -2,7 +2,7 @@ amends "../../config/Service.pkl" import "../../config/site.pkl" as site -meta { name = "Radarr" } +name = "Radarr" setup = "tools/configure-arr.py" dependsOn { "qbittorrent" diff --git a/service/samba/service.pkl b/service/samba/service.pkl index db52ea66aaa7cfd26824a4f29ed4b2b440f5cacc..4069c53e23d1104122846d69d5ed2f1e136b63fe 100644 --- a/service/samba/service.pkl +++ b/service/samba/service.pkl @@ -2,7 +2,7 @@ amends "../../config/Service.pkl" import "../../config/site.pkl" as site -meta { name = "Samba" } +name = "Samba" container { image = "ghcr.io/servercontainers/samba@sha256:31b90ea7fe3258d30fccd971c85743b92694605f4b43dc8a4df23a202fced06a" diff --git a/service/shale/prepare.py b/service/shale/prepare.py index 29f817a05e536bf679dd13eab10f1b111c69860f..6afa2038cfcb5a63d0da74b8f853085f9e0e60bb 100644 --- a/service/shale/prepare.py +++ b/service/shale/prepare.py @@ -17,7 +17,7 @@ for config in (root / "repositories_owned").glob("*/config"): file.write("\n[http]\n\treceivepack = true\n") path = root / "data/astheno.shale.db" if path.exists(): - domain = os.environ["STUDIO_DOMAIN"] + domain = os.environ["SNOWGLOBE_DOMAIN"] old, new = "auth." + domain + "/realms/master", "snowglobe." + domain db = sqlite3.connect(path, timeout=30) db.execute("BEGIN IMMEDIATE") diff --git a/service/shale/service.pkl b/service/shale/service.pkl index 08c97a18ca22170c6eacb32b0c1d4c1cdb4b728f..556de423d541818c4d70775eb97bfd274856dd95 100644 --- a/service/shale/service.pkl +++ b/service/shale/service.pkl @@ -3,13 +3,11 @@ amends "../../config/Service.pkl" import "../../config/OpenID.pkl" as sso import "../../config/site.pkl" as site -meta { - name = "Shale" -} +name = "Shale" requirements { new sso.Client { clientId = module.id - name = module.meta.name + name = module.name redirectUris { "https://\(module.container.http.hostname)/-/callback" } usernameAliases { ["snow"] = "clover" } allowGuests = true @@ -21,7 +19,6 @@ prepare = "prepare.py" container { memory = 4096 image = "docker.io/astheno/shale@sha256:b55cc42c4eb2c1e21733405bec795e49ddd7c71fb1755d2aa2ee4361f750cebd" - extraHosts { "\(sso.hostname):host-gateway" } http { containerPort = 8000 @@ -29,11 +26,11 @@ container { overrideFiles { ["/-/theme.css"] = "theme.css" ["/-/repo-icons/"] = "icons" - ["/-/studio-readme/"] = "readme" + ["/-/snowglobe-readme/"] = "readme" } headHtml { ["/*"] = """ - + """ } } @@ -45,7 +42,7 @@ container { ["/etc/crontabs"] {} ["/etc/shale/theme.css"] { config = "theme.css" } ["/etc/ssl/certs/ca-certificates.crt"] { - src = "/var/lib/studio/ca-bundle.crt" + src = "/var/lib/snowglobe/ca-bundle.crt" readOnly = true } } diff --git a/service/sonarr/service.pkl b/service/sonarr/service.pkl index 3bd486efc6bb7c0a7790c7c6728ea671a288707e..dec1278d4d963aeb91bff4c1243386dcc3133711 100644 --- a/service/sonarr/service.pkl +++ b/service/sonarr/service.pkl @@ -2,7 +2,7 @@ amends "../../config/Service.pkl" import "../../config/site.pkl" as site -meta { name = "Sonarr" } +name = "Sonarr" setup = "tools/configure-arr.py" dependsOn { "qbittorrent" diff --git a/service/tailscale/service.pkl b/service/tailscale/service.pkl index 0deb752b39960e997ca52b844357b6da32f431ab..42c7f3f660dabbccdf59a1d0a2649c805fabde52 100644 --- a/service/tailscale/service.pkl +++ b/service/tailscale/service.pkl @@ -2,7 +2,7 @@ amends "../../config/Service.pkl" import "../../config/site.pkl" as site -meta { name = "Tailscale" } +name = "Tailscale" container { image = "docker.io/tailscale/tailscale@sha256:2667499ed87ae29218f292556ba062918402dd5e92e93637af14867e4df12dd3" diff --git a/service/victoria/service.pkl b/service/victoria/service.pkl index 019449b6d64973361f470d924be2a2de80fd3833..783ada6df75137b36d30d6527820b9c29acefa60 100644 --- a/service/victoria/service.pkl +++ b/service/victoria/service.pkl @@ -1,12 +1,12 @@ amends "../../config/Service.pkl" -meta { name = "Victoria"; tagline = "Metrics, logs, and traces" } +name = "Victoria" +tagline = "Metrics, logs, and traces" containers { ["logs"] { image = "docker.io/victoriametrics/victoria-logs:v1.52.0" cpu = 300 - memory = 512 args { "-storageDataPath=/logs"; "-retentionPeriod=30d" } http { @@ -22,7 +22,6 @@ containers { ["metrics"] { image = "docker.io/victoriametrics/victoria-metrics:v1.152.0" cpu = 300 - memory = 512 args { "-storageDataPath=/metrics"; "-retentionPeriod=30d"; "-usePromCompatibleNaming" } http { @@ -39,7 +38,6 @@ containers { ["traces"] { image = "docker.io/victoriametrics/victoria-traces:v0.11.1" cpu = 300 - memory = 512 args { "-storageDataPath=/traces"; "-retentionPeriod=30d" } http { diff --git a/service/xmpp/prosody.cfg.lua b/service/xmpp/prosody.cfg.lua index 5bd2726f0d22616bc5096321e8fffaf1636bd042..6a501a348c8e0cda11d8361185c58f674273317c 100644 --- a/service/xmpp/prosody.cfg.lua +++ b/service/xmpp/prosody.cfg.lua @@ -4,7 +4,7 @@ local network = Lua.require("prosody.util.json").decode(Lua.io.open("/etc/xmpp-n data_path = "/var/lib/prosody" pidfile = "/var/lib/prosody/prosody.pid" -plugin_paths = { "/opt/prosody-modules", "/studio" } +plugin_paths = { "/opt/prosody-modules", "/snowglobe" } admins = { "clover@" .. domain } bootstrap_password = ENV_XMPP_PASSWORD authentication = "internal_hashed" diff --git a/service/xmpp/service.pkl b/service/xmpp/service.pkl index a23de704f9cd7296a2c00571f4020405bf28d098..f7e7910fd0b6959d4fe82b57fb1ae0605025e7da 100644 --- a/service/xmpp/service.pkl +++ b/service/xmpp/service.pkl @@ -8,7 +8,8 @@ local network = new json.Parser {}.parse(read("network.json").text) local host = read?("prop:hostname") ?? "xmpp.\(site.domain)" local accountDomain = if (site.preview) host else site.domain -meta { name = "XMPP"; launcher = false } +name = "XMPP" +launcher = false stageIsolation = "fresh" healthyDeadline = "10m" setup = "setup.py" @@ -28,12 +29,12 @@ containers { ["certificates"] { build = "." entrypoint = "/usr/bin/python3" - args { "/studio/run.py"; "certificates" } + args { "/snowglobe/run.py"; "certificates" } hostNetwork = true lifecycle = "prestartSidecar" cpu = 100 memory = 256 - volumes { ["/tls"] {}; ["/studio"] { config = "runtime" } } + volumes { ["/tls"] {}; ["/snowglobe"] { config = "runtime" } } env { ["XMPP_DOMAIN"] = accountDomain ["XMPP_HOST"] = host @@ -44,9 +45,8 @@ containers { ["prosody"] { build = "." entrypoint = "/usr/bin/python3" - args { "/studio/run.py"; "prosody" } + args { "/snowglobe/run.py"; "prosody" } hostNetwork = true - memory = 512 http { containerPort = 5280; hostname = host; loopback = true; checkPath = "/health"; forwardRealIp = true } tcp { new { name = "c2s"; containerPort = network.c2s; hostPort = containerPort; loopback = false } @@ -56,7 +56,7 @@ containers { volumes { ["/var/lib/prosody"] {} ["/tls"] { readOnly = true } - ["/studio"] { config = "runtime" } + ["/snowglobe"] { config = "runtime" } ["/etc/prosody/prosody.cfg.lua"] { config = "prosody.cfg.lua" } } env { @@ -72,7 +72,7 @@ containers { ["turn"] { build = "." entrypoint = "/usr/bin/python3" - args { "/studio/run.py"; "turn" } + args { "/snowglobe/run.py"; "turn" } hostNetwork = true memory = 256 tcp { @@ -82,7 +82,7 @@ containers { new { name = "relay-\(port)"; containerPort = port; hostPort = port; loopback = false } } } - volumes { ["/tls"] { readOnly = true }; ["/studio"] { config = "runtime" } } + volumes { ["/tls"] { readOnly = true }; ["/snowglobe"] { config = "runtime" } } env { ["XMPP_HOST"] = host; ["TURN_SECRET"] = "${secret.own.turn}" } } } diff --git a/service/xmpp/setup.py b/service/xmpp/setup.py index 6933841a5d959e8bdecdda2b86500ceb8876541b..a014e3df5efb59e9987faad384d053fbeec3ba6f 100644 --- a/service/xmpp/setup.py +++ b/service/xmpp/setup.py @@ -22,7 +22,7 @@ if len(sys.argv) == 1: sys.exit() stage = sys.argv[1] -metadata = Path("/var/lib/studio/stages") / (stage + ".json") +metadata = Path("/var/lib/snowglobe/stages") / (stage + ".json") network = json.loads(Path(__file__).with_name("network.json").read_text()) rules = [ ["-p", "tcp", "-m", "multiport", "--dports", ",".join(str(network[key]) for key in ( diff --git a/service/ytdl/service.pkl b/service/ytdl/service.pkl index d68da5991d2c66c4319bd7557eb3878a4cfe68a2..79bc2d910e8e916b390c2633a97e314bf7ec6fe0 100644 --- a/service/ytdl/service.pkl +++ b/service/ytdl/service.pkl @@ -4,7 +4,7 @@ import "../../config/site.pkl" as site local passive = site.preview || site.mediaReadOnly -meta { name = "YTDL" } +name = "YTDL" requiredSecrets { "smtp_host" @@ -43,7 +43,6 @@ containers { else "sh /config-yt/archive-loop.sh" } - cpu = 200 memory = if (passive) 256 else 512 http = if (passive) new { diff --git a/tools/check-legacy-handoff.sh b/tools/check-legacy-handoff.sh deleted file mode 100644 index 5a3cf5d67e8fedbf5c0731f5ccbf15cc85a97588..0000000000000000000000000000000000000000 --- a/tools/check-legacy-handoff.sh +++ /dev/null @@ -1,24 +0,0 @@ -#!/bin/sh -set -eu - -handoff=${1:?Expected a legacy handoff directory} -target=${2:?Expected a production target} -port=${3:?Expected a target SSH port} -printf '%s\n' "$handoff" | grep -Eq '^/mnt/storage1/apps/studio-handoff/[0-9]{8}T[0-9]{6}Z-[0-9a-f]{6}$' || { - echo 'Invalid legacy handoff directory' >&2 - exit 1 -} -ssh -p "$port" "$target" " - set -eu - test -f '$handoff/manifest.json' - pool=\$(findmnt -n -o SOURCE --mountpoint /srv) - test \"\$(findmnt -n -o FSTYPE --mountpoint /srv)\" = zfs - zpool list \"\$pool\" >/dev/null - apps=\$(findmnt -n -o SOURCE --mountpoint /mnt/storage1/apps) - test \"\$apps\" = \"\$pool/apps\" - test \"\$(findmnt -n -o FSTYPE --mountpoint /mnt/storage1/apps)\" = zfs - test \"\$(zfs get -H -o value encryption \"\$apps\")\" = aes-256-gcm -" || { - echo 'Legacy handoff or encrypted apps dataset is unavailable' >&2 - exit 1 -} diff --git a/tools/configure-arr.py b/tools/configure-arr.py index 00a24e3de0246b03959f3fbc769f71ab3fca3ead..80c0d519e91d793122779f916551fa304f9365a2 100644 --- a/tools/configure-arr.py +++ b/tools/configure-arr.py @@ -6,7 +6,7 @@ from urllib.parse import urlsplit, urlunsplit import urllib.request import xml.etree.ElementTree as ET -from studio import load +from snowglobe import load data = json.load(sys.stdin) diff --git a/tools/copyparty-cutover-test.py b/tools/copyparty-cutover-test.py deleted file mode 100644 index 39defc1dd6ece739b7b150404147363e21180471..0000000000000000000000000000000000000000 --- a/tools/copyparty-cutover-test.py +++ /dev/null @@ -1,299 +0,0 @@ -#!/usr/bin/env python3 -import importlib.util -import json -import os -from pathlib import Path -import sqlite3 -import sys -import tempfile -import unittest -from unittest.mock import patch - -spec = importlib.util.spec_from_file_location('cutover', Path(__file__).with_name('copyparty-cutover.py')) -cutover = importlib.util.module_from_spec(spec) -spec.loader.exec_module(cutover) -copyparty = cutover.copyparty -public_statuses = cutover.public_statuses - - -class Cutover(unittest.TestCase): - def setUp(self): - self.temporary = tempfile.TemporaryDirectory(prefix='copyparty-cutover-') - self.addCleanup(self.temporary.cleanup) - self.base = Path(self.temporary.name) - self.root = self.base / 'clover' - self.cfg = self.base / 'cfg' - (self.root / 'Share').mkdir(parents=True) - (self.cfg / 'copyparty').mkdir(parents=True) - self.environment = patch.dict(os.environ, {'STUDIO_CLOVER_ROOT': str(self.root), 'STUDIO_COPYPARTY_CFG': str(self.cfg)}) - self.environment.start() - self.addCleanup(self.environment.stop) - self.manager = copyparty.Manager([]) - self.manager.job = {'ID': 'copyparty', 'Stop': False, 'Meta': {'studio_hostname': 'copyparty.example.invalid'}} - self.events = [] - edge = patch.object(cutover, 'public_statuses', side_effect=lambda host, keys, deadline: {key: 200 for key in keys}) - edge.start() - self.addCleanup(edge.stop) - - def stop(): - self.events.append('stop') - if self.manager.offline_at is None: - self.manager.offline_at = cutover.time.monotonic() - self.manager.stopped = True - - def start(): - self.events.append('start') - self.manager.stopped = False - - self.manager.stop = stop - self.manager.start = start - for source, _ in cutover.MOVES: - path = self.root / source - (path / 'nested').mkdir(parents=True) - (path / 'selected.txt').write_text('Disposable fixture') - (path / 'nested' / 'child.txt').write_text('Disposable child') - (self.cfg / 'copyparty.conf').write_text('Fixture configuration') - self.database = self.cfg / 'copyparty/shares.db' - with sqlite3.connect(self.database) as db: - db.executescript("CREATE TABLE kv(k text,v int); INSERT INTO kv VALUES('sver',2); CREATE TABLE sh(k text,pw text,vp text,pr text,st int,un text,t0 int,t1 int); CREATE TABLE sf(k text,vp text);") - db.executemany('INSERT INTO sh VALUES(?,?,?,?,?,?,?,?)', [ - ('folder-key', 'opaque-hash', 'clover/Documents/Public/Snowbound', 'rw', 0, 'snow', 10, 20), - ('nested-key', '', 'clover/Documents/Public/Snowbound/nested', 'r', 0, 'snow', 30, 0), - ('selected-key', 'another-hash', 'clover/Documents/EvilTakeout', 'r', 1, 'snow', 40, 50), - ('unaffected-key', '', 'clover/Other', 'r', 0, 'snow', 60, 0), - ]) - db.execute('INSERT INTO sf VALUES(?,?)', ('selected-key', 'selected.txt')) - with sqlite3.connect(self.database) as db: - self.original = cutover.rows(db) - self.before_content = {str(path.relative_to(self.root)): path.read_bytes() for path in self.root.rglob('*.txt')} - - def test_move_preserves_keys_selections_passwords_inodes_and_content(self): - expected = cutover.plan(self.manager) - result = cutover.apply(self.manager, expected, self.base / 'backups') - self.assertEqual(result['result'], 'moved') - self.assertEqual(self.events, ['stop', 'start']) - with sqlite3.connect(self.database) as db: - actual = cutover.rows(db) - self.assertEqual(actual, cutover.remapped(self.manager, self.original, cutover.mappings(self.manager))) - self.assertEqual(actual['sf'], self.original['sf']) - for detail in expected['moves']: - source, destination = Path(detail['source']), Path(detail['destination']) - self.assertFalse(source.exists()) - self.assertEqual(destination.stat().st_ino, detail['inode']) - for old_path, contents in self.before_content.items(): - old = self.root / old_path - if old.is_relative_to(source): - self.assertEqual((destination / old.relative_to(source)).read_bytes(), contents) - backup = Path(result['backup']) - for child in backup.iterdir(): - self.assertEqual(child.stat().st_mode & 0o777, 0o600) - self.assertEqual(backup.stat().st_mode & 0o777, 0o700) - - def test_public_probe_sends_bounded_verified_head_and_preserves_redirect_status(self): - response = cutover.subprocess.CompletedProcess([], 0, stdout='302', stderr='') - with patch.object(cutover.subprocess, 'run', return_value=response) as run: - actual = public_statuses('copyparty.example.invalid', ['selected file'], cutover.time.monotonic() + 1) - self.assertEqual(actual, {'selected file': 302}) - arguments = run.call_args.args[0] - self.assertEqual(arguments[:2], ['curl', '--disable']) - self.assertIn('--head', arguments) - self.assertEqual(arguments[arguments.index('--output') + 1], '/dev/null') - self.assertEqual(arguments[arguments.index('--write-out') + 1], '%{http_code}') - self.assertEqual(arguments[arguments.index('--noproxy') + 1], '*') - self.assertEqual(arguments[-2:], ['--', 'https://copyparty.example.invalid/shr/selected%20file/']) - self.assertLessEqual(run.call_args.kwargs['timeout'], 1) - self.assertEqual(float(arguments[arguments.index('--max-time') + 1]), run.call_args.kwargs['timeout']) - for flag in ['--insecure', '-k', '--location', '-L', '--include', '-i']: - self.assertNotIn(flag, arguments) - - def test_public_probe_treats_network_tls_and_process_timeout_as_unavailable(self): - for response in [ - cutover.subprocess.CompletedProcess([], 35, stdout='000', stderr='TLS problem'), - cutover.subprocess.CompletedProcess([], 28, stdout='200', stderr='Timeout'), - cutover.subprocess.CompletedProcess([], 0, stdout='000', stderr=''), - cutover.subprocess.CompletedProcess([], 0, stdout='200 headers', stderr=''), - ]: - with patch.object(cutover.subprocess, 'run', return_value=response): - self.assertEqual(public_statuses('host', ['key'], cutover.time.monotonic() + 1), {'key': None}) - with patch.object(cutover.subprocess, 'run', side_effect=cutover.subprocess.TimeoutExpired('curl', 0.1)): - self.assertEqual(public_statuses('host', ['key'], cutover.time.monotonic() + 0.1), {'key': None}) - with patch.object(cutover.subprocess, 'run') as run: - self.assertEqual(public_statuses('host', ['key'], cutover.time.monotonic() - 1), {'key': None}) - run.assert_not_called() - - def test_stalled_probe_children_cannot_hold_the_executor_past_its_budget(self): - run = cutover.subprocess.run - def stalled(arguments, **options): - return run([sys.executable, '-c', 'import time; time.sleep(10)'], **options) - started = cutover.time.monotonic() - with patch.object(cutover.subprocess, 'run', side_effect=stalled): - actual = public_statuses('host', ['one', 'two'], started + 0.15) - self.assertEqual(actual, {'one': None, 'two': None}) - self.assertLess(cutover.time.monotonic() - started, 1.5) - - def test_public_poll_recovers_and_timeout_obeys_its_budget(self): - clock = [0.0] - def sleep(delay): - clock[0] += delay - with patch.object(cutover.time, 'monotonic', side_effect=lambda: clock[0]), patch.object(cutover.time, 'sleep', side_effect=sleep): - with patch.object(cutover, 'public_statuses', side_effect=[{'key': 503}, {'key': 200}]): - cutover.wait_public('host', {'key': 200}, 1) - self.assertEqual(clock[0], 0.5) - with patch.object(cutover, 'public_statuses', return_value={'key': None}), self.assertRaises(TimeoutError): - cutover.wait_public('host', {'key': 200}, 1) - self.assertEqual(clock[0], 1) - - def test_edge_timeout_rolls_back_and_records_recovery_evidence(self): - expected = cutover.plan(self.manager) - with patch.object(cutover, 'wait_public', side_effect=[TimeoutError('injected edge timeout'), None]): - with self.assertRaisesRegex(TimeoutError, 'edge timeout'): - cutover.apply(self.manager, expected, self.base / 'backups') - self.assertEqual(cutover.plan(self.manager), expected) - self.assertEqual(self.events, ['stop', 'start', 'stop', 'start']) - evidence = json.loads(next((self.base / 'backups').glob('*/result.json')).read_text()) - self.assertEqual(evidence['result'], 'rolled-back') - self.assertEqual(evidence['publicStatuses'], {key: 200 for key in expected['affectedKeys']}) - - def test_existing_grants_follow_moves_and_configuration_is_rerendered(self): - source, destination = cutover.mappings(self.manager)[0] - state = {'accounts': [], 'grants': [{'id': 'fixture', 'subject': 'user:fixture', 'path': str(source), 'permissions': 'r'}], 'names': {}} - (self.cfg / 'manager.json').write_text(json.dumps(state)) - expected = cutover.plan(self.manager) - with patch.object(copyparty.runs, 'nomad', return_value={'Items': {}}), patch.object(copyparty, 'configuration', return_value=b'Updated configuration') as render: - cutover.apply(self.manager, expected, self.base / 'backups') - saved = json.loads((self.cfg / 'manager.json').read_text()) - self.assertEqual(saved['grants'][0]['path'], str(destination)) - self.assertEqual(render.call_args.args[1]['grants'], saved['grants']) - self.assertEqual((self.cfg / 'copyparty.conf').read_bytes(), b'Updated configuration') - - def test_failed_restart_restores_files_and_database_before_restart(self): - def start(): - self.events.append('start') - if self.events.count('start') == 1: - raise RuntimeError('injected startup failure') - self.manager.stopped = False - self.manager.start = start - expected = cutover.plan(self.manager) - with self.assertRaisesRegex(RuntimeError, 'injected startup failure'): - cutover.apply(self.manager, expected, self.base / 'backups') - self.assertEqual(self.events, ['stop', 'start', 'stop', 'start']) - self.assertEqual(cutover.plan(self.manager), expected) - with sqlite3.connect(self.database) as db: - self.assertEqual(cutover.rows(db), self.original) - for path, content in self.before_content.items(): - self.assertEqual((self.root / path).read_bytes(), content) - - def test_rollback_uses_actual_helper_stop_and_start_with_frozen_nomad_job(self): - self.manager.stop = copyparty.Manager.stop.__get__(self.manager) - self.manager.start = copyparty.Manager.start.__get__(self.manager) - self.manager.job = None - requests = [] - def nomad(path, method='GET', body=None, **_): - requests.append((path, method, body)) - if path == 'job/copyparty' and method == 'GET': - return {'ID': 'copyparty', 'Stop': False, 'Meta': {'immutable': 'fixture', 'studio_hostname': 'copyparty.example.invalid'}} - if path.endswith('/allocations'): - return [{'ClientStatus': 'complete'}] - if path.startswith('var/'): - return {'Items': {'idp_header': 'fixture'}} - return {} - expected = cutover.plan(self.manager) - with patch.object(copyparty.runs, 'nomad', side_effect=nomad), patch.object(self.manager, 'reload', side_effect=[copyparty.Error(400, 'injected reload rejection'), None]): - with self.assertRaisesRegex(copyparty.Error, 'injected reload rejection'): - cutover.apply(self.manager, expected, self.base / 'backups') - self.assertEqual(cutover.plan(self.manager), expected) - registrations = [body for path, method, body in requests if path == 'jobs' and method == 'POST'] - self.assertEqual(registrations, [{'Job': self.manager.job}, {'Job': self.manager.job}]) - self.assertEqual(len([path for path, method, _ in requests if method == 'DELETE']), 2) - self.assertFalse(self.manager.stopped) - - def test_partial_rename_failure_rolls_back_before_start(self): - native_rename = copyparty.rename - calls = 0 - def rename(source, destination): - nonlocal calls - calls += 1 - if calls == 3: - raise RuntimeError('injected third rename failure') - self.assertTrue(self.manager.stopped) - native_rename(source, destination) - expected = cutover.plan(self.manager) - with patch.object(copyparty, 'rename', side_effect=rename), self.assertRaisesRegex(RuntimeError, 'third rename'): - cutover.apply(self.manager, expected, self.base / 'backups') - self.assertEqual(cutover.plan(self.manager), expected) - self.assertEqual(self.events, ['stop', 'start']) - - def test_stop_timeout_still_restarts_the_frozen_job(self): - def stop(): - self.manager.offline_at = cutover.time.monotonic() - self.manager.stopped = True - raise RuntimeError('injected stop timeout') - self.manager.stop = stop - expected = cutover.plan(self.manager) - with self.assertRaisesRegex(RuntimeError, 'stop timeout'): - cutover.apply(self.manager, expected, self.base / 'backups') - self.assertEqual(self.events, ['start']) - self.assertEqual(cutover.plan(self.manager), expected) - - def test_ambiguous_stop_response_reregisters_even_without_stopped_flag(self): - def stop(): - self.manager.offline_at = cutover.time.monotonic() - self.manager.stopped = False - raise TimeoutError('lost Nomad stop response') - self.manager.stop = stop - expected = cutover.plan(self.manager) - with self.assertRaisesRegex(TimeoutError, 'lost Nomad stop response'): - cutover.apply(self.manager, expected, self.base / 'backups') - self.assertEqual(self.events, ['start']) - self.assertEqual(cutover.plan(self.manager), expected) - - def test_verification_budget_reserves_time_for_restart(self): - expected = cutover.plan(self.manager) - original_tree = cutover.tree - def timed_tree(path, deadline=None): - if deadline is not None: - raise TimeoutError('injected verification deadline') - return original_tree(path) - with patch.object(cutover, 'tree', side_effect=timed_tree), self.assertRaisesRegex(TimeoutError, 'verification deadline'): - cutover.apply(self.manager, expected, self.base / 'backups') - self.assertEqual(self.events, ['stop', 'start']) - self.assertEqual(cutover.plan(self.manager), expected) - - def test_source_and_database_drift_stop_before_mutation(self): - expected = cutover.plan(self.manager) - (self.root / cutover.MOVES[0][0] / 'changed.txt').write_text('Concurrent disposable write') - with self.assertRaisesRegex(ValueError, 'plan changed'): - cutover.apply(self.manager, expected, self.base / 'backups') - self.assertEqual(self.events, []) - - def test_parent_shares_and_parent_selections_require_explicit_url_preservation(self): - for selected in [0, 1]: - with sqlite3.connect(self.database) as db: - db.execute('INSERT OR REPLACE INTO sh VALUES(?,?,?,?,?,?,?,?)', ('parent', '', 'clover/Documents/Public', 'r', selected, 'snow', 1, 0)) - if selected: - db.execute('INSERT INTO sf VALUES(?,?)', ('parent', 'Snowbound')) - with self.assertRaisesRegex(ValueError, 'parent of a moved folder'): - cutover.plan(self.manager) - with sqlite3.connect(self.database) as db: - db.execute("DELETE FROM sh WHERE k='parent'") - db.execute("DELETE FROM sf WHERE k='parent'") - - def test_symlinks_mounts_and_existing_destinations_reject_before_downtime(self): - source, destination = cutover.mappings(self.manager)[0] - destination.mkdir() - with self.assertRaises(ValueError): - cutover.plan(self.manager) - destination.rmdir() - link = source / 'link' - link.symlink_to('selected.txt') - with self.assertRaisesRegex(ValueError, 'symbolic link'): - cutover.plan(self.manager) - link.unlink() - with patch.object(cutover.os.path, 'ismount', side_effect=lambda path: Path(path) == source): - with self.assertRaises(ValueError): - cutover.plan(self.manager) - self.assertEqual(self.events, []) - - -if __name__ == '__main__': - unittest.main() diff --git a/tools/copyparty-cutover.py b/tools/copyparty-cutover.py deleted file mode 100644 index 8f4dbbd4bde2a32d8ff87249263e6053a264e104..0000000000000000000000000000000000000000 --- a/tools/copyparty-cutover.py +++ /dev/null @@ -1,279 +0,0 @@ -#!/usr/bin/env python3 -import argparse -import concurrent.futures -import copy -import hashlib -import importlib.util -import json -import os -import re -from pathlib import Path -import sqlite3 -import socket -import subprocess -import stat -import time -import uuid -from urllib.parse import quote - - -spec = importlib.util.spec_from_file_location("copyparty", Path(__file__).resolve().parent.parent / "service/copyparty/manage.py") -copyparty = importlib.util.module_from_spec(spec) -spec.loader.exec_module(copyparty) - - -MOVES = ( - ('Documents/Public/Snowbound', 'Clover - Snowbound'), - ('Documents/Public/Meadow', 'Clover - Meadow Chat'), - ('Documents/Public/Bill Wurtz Music', 'Xavier - Bill Wurtz Music'), - ('Documents/HOTEWIG Reanimated', 'Collab - HOTEWIG Reanimated'), - ('Documents/EvilTakeout', 'Lino - Evil Inc Data Takeout'), - ('Documents/Windows7', 'Clover - Windows 7'), -) - - -def digest(value): - return hashlib.sha256(json.dumps(value, ensure_ascii=False, separators=(',', ':')).encode()).hexdigest() - - -def rows(database): - if database.execute("SELECT v FROM kv WHERE k='sver'").fetchone() != (2,): - raise ValueError('Unsupported Copyparty share database version.') - return { - 'sh': database.execute('SELECT * FROM sh ORDER BY k').fetchall(), - 'sf': database.execute('SELECT * FROM sf ORDER BY k,vp').fetchall(), - } - - -def mappings(manager): - return [(manager.root / source, manager.root / 'Share' / name) for source, name in MOVES] - - -def remapped(manager, before, moves): - after = copy.deepcopy(before) - for index, row in enumerate(before['sh']): - source_path = manager.real(row[2].strip('/')) - for source, destination in moves: - if source_path is not None and source_path.is_relative_to(source): - changed = list(row) - changed[2] = manager.virtual(destination / source_path.relative_to(source)) - after['sh'][index] = tuple(changed) - break - if (source_path is not None and source.is_relative_to(source_path)) or row[2].strip('/') == '': - raise ValueError('A share includes a parent of a moved folder. Preserve its URL paths before cutover: ' + row[0]) - return after - - -def tree(path, deadline=None): - def unreadable(error): - raise error - - result = [] - for directory, dirs, files in os.walk(path, followlinks=False, onerror=unreadable): - if deadline is not None and time.monotonic() >= deadline: - raise TimeoutError("Folder verification exceeded the cutover budget.") - base = Path(directory) - if base != path and os.path.ismount(base): - raise ValueError('A source contains a mounted filesystem: ' + str(base)) - for entry in [base] + [base / name for name in files] + [base / name for name in dirs if (base / name).is_symlink()]: - if deadline is not None and time.monotonic() >= deadline: - raise TimeoutError("Folder verification exceeded the cutover budget.") - info = entry.lstat() - if stat.S_ISLNK(info.st_mode): - raise ValueError('Review this source symbolic link before cutover: ' + str(entry)) - result.append((str(entry.relative_to(path)), info.st_dev, info.st_ino, info.st_mode, info.st_size if not stat.S_ISDIR(info.st_mode) else 0, info.st_mtime_ns)) - dirs[:] = [name for name in dirs if not (base / name).is_symlink()] - return sorted(result) - - -def plan(manager, deadline=None): - moves = mappings(manager) - details = [] - for source, destination in moves: - manager.path_for(str(source)) - manager.path_for(str(destination), missing=True) - if source.is_symlink() or os.path.ismount(source) or os.path.lexists(destination): - raise ValueError('A source is mounted or a destination already exists: ' + str(source)) - source_info, parent_info = source.stat(), destination.parent.stat() - if source_info.st_dev != parent_info.st_dev: - raise ValueError('Source and destination must be on the same filesystem: ' + str(source)) - manifest = tree(source, deadline) - details.append({'source': str(source), 'destination': str(destination), 'device': source_info.st_dev, - 'inode': source_info.st_ino, 'entries': len(manifest), 'tree': digest(manifest)}) - with sqlite3.connect((manager.native / 'shares.db').as_uri() + '?mode=ro', uri=True) as database: - before = rows(database) - after = remapped(manager, before, moves) - return {'host': socket.gethostname(), 'service': manager.service, 'configRoot': str(manager.cfg), 'moves': details, 'shares': len(before['sh']), 'selections': len(before['sf']), - 'affectedKeys': [old[0] for old, new in zip(before['sh'], after['sh']) if old != new], - 'sharesBefore': digest(before), 'sharesAfter': digest(after), - 'config': digest((manager.cfg / 'copyparty.conf').read_text()), - 'manager': digest((manager.cfg / 'manager.json').read_text() if (manager.cfg / 'manager.json').exists() else None)} - - -def write_database(database, data): - current = rows(database) - if current['sf'] != data['sf'] or len(current['sh']) != len(data['sh']) or any( - old[:2] + old[3:] != new[:2] + new[3:] - for old, new in zip(current['sh'], data['sh']) - ): - raise ValueError('Share keys, selections, passwords, or permissions changed during cutover.') - with database: - database.executemany('UPDATE sh SET vp=? WHERE k=?', [(row[2], row[0]) for row in data['sh']]) - if rows(database) != data: - raise ValueError('Copyparty share paths changed while the database was being written.') - - -def public_statuses(host, keys, deadline): - def head(key): - timeout = min(3, deadline - time.monotonic()) - if timeout <= 0: - return key, None - try: - response = subprocess.run( - ['curl', '--disable', '--noproxy', '*', '--silent', '--show-error', - '--head', '--output', '/dev/null', '--write-out', '%{http_code}', - '--max-time', str(timeout), '--', 'https://' + host + '/shr/' + quote(key, safe='') + '/'], - capture_output=True, text=True, timeout=timeout, - ) - status = response.stdout.strip() - return key, int(status) if response.returncode == 0 and re.fullmatch(r'[1-5][0-9]{2}', status) else None - except (OSError, subprocess.TimeoutExpired): - return key, None - - with concurrent.futures.ThreadPoolExecutor(max_workers=4) as workers: - return dict(workers.map(head, keys)) - - -def wait_public(host, expected, deadline): - while time.monotonic() < deadline: - if public_statuses(host, expected, deadline) == expected: - return - time.sleep(min(0.5, max(0, deadline - time.monotonic()))) - raise TimeoutError('Public share links did not recover before the cutover deadline.') - - -def apply(manager, expected, backup_root): - if plan(manager) != expected: - raise ValueError('The cutover plan changed. Review a new plan before applying.') - if manager.job is None: - manager.job = copyparty.runs.nomad('job/' + manager.service) - host = (manager.job or {}).get('Meta', {}).get('studio_hostname', '') - if not isinstance(host, str) or not re.fullmatch(r'[a-zA-Z0-9.-]+', host): - raise ValueError('Copyparty has no public hostname. Check its deployed service route.') - manager.job['Stop'] = False - baseline = public_statuses(host, expected['affectedKeys'], time.monotonic() + 15) - if any(status is None for status in baseline.values()): - raise ValueError('Public shares are unavailable. Restore their current routes before cutover.') - backup_root.mkdir(parents=True, exist_ok=True, mode=0o700) - metadata = backup_root.lstat() - if not stat.S_ISDIR(metadata.st_mode) or metadata.st_uid != os.geteuid() or metadata.st_mode & 0o077: - raise ValueError('Cutover backups require an owner-only directory.') - backup = backup_root / (time.strftime('%Y%m%dT%H%M%SZ', time.gmtime()) + '-' + uuid.uuid4().hex[:8]) - backup.mkdir(mode=0o700) - database_path = manager.native / 'shares.db' - saved_files = {} - for path in (manager.cfg / 'copyparty.conf', manager.cfg / 'manager.json'): - saved_files[path] = path.read_bytes() if path.exists() else None - if saved_files[path] is not None: - copyparty.atomic(backup / path.name, saved_files[path]) - copyparty.atomic(backup / 'plan.json', json.dumps(expected, ensure_ascii=False).encode()) - moved = [] - before = None - database = None - manager.deadline = time.monotonic() + 285 - mutation_deadline = manager.deadline - 205 - restart_attempted = False - try: - manager.stop() - copyparty.atomic(backup / 'nomad-job.json', json.dumps(manager.job).encode()) - if plan(manager, mutation_deadline) != expected: - raise ValueError('The source files or share rows changed before Copyparty stopped.') - database = sqlite3.connect(database_path) - before = rows(database) - with sqlite3.connect(backup / 'shares.db') as target: - database.backup(target) - (backup / 'shares.db').chmod(0o600) - after = remapped(manager, before, mappings(manager)) - data_path = manager.cfg / 'manager.json' - data = json.loads(saved_files[data_path]) if saved_files[data_path] is not None else None - for source, destination in mappings(manager): - copyparty.rename(source, destination) - moved.append((source, destination)) - write_database(database, after) - if data is not None: - for grant in data.get('grants', []): - original = Path(grant['path']) - for source, destination in mappings(manager): - if original.is_relative_to(source): - grant['path'] = str(destination / original.relative_to(source)) - break - copyparty.atomic(data_path, json.dumps(data, ensure_ascii=False).encode()) - variable = copyparty.runs.nomad('var/nomad/jobs/' + manager.service) - rendered = copyparty.configuration(manager.cfg, {**data, 'root': str(manager.root)}, variable['Items']) - info = (manager.cfg / 'copyparty.conf').stat() - copyparty.atomic(manager.cfg / 'copyparty.conf', rendered, (info.st_uid, info.st_gid)) - for detail in expected['moves']: - destination = Path(detail['destination']) - if digest(tree(destination, mutation_deadline)) != detail['tree']: - raise ValueError('Moved folder metadata changed: ' + str(destination)) - if digest(rows(database)) != expected['sharesAfter']: - raise ValueError('Share keys, selections, passwords, or permissions changed during cutover.') - if time.monotonic() >= mutation_deadline: - raise TimeoutError("Folder updates exceeded the cutover budget.") - restart_attempted = True - manager.start() - wait_public(host, baseline, manager.offline_at + 150) - result = {'result': 'moved', 'moves': len(moved), 'affectedKeys': expected['affectedKeys'], 'backup': str(backup), - 'downtimeSeconds': time.monotonic() - manager.offline_at, 'publicStatuses': baseline} - copyparty.atomic(backup / 'result.json', json.dumps(result).encode()) - return result - except BaseException as failure: - if moved or before is not None: - if restart_attempted: - manager.stop() - if database is None: - database = sqlite3.connect(database_path) - if before is not None: - write_database(database, before) - for source, destination in reversed(moved): - copyparty.rename(destination, source) - for path, content in saved_files.items(): - if content is None: - path.unlink(missing_ok=True) - else: - owner = path.stat() - copyparty.atomic(path, content, (owner.st_uid, owner.st_gid)) - if manager.job is not None: - manager.start() - wait_public(host, baseline, manager.deadline) - evidence = {'result': 'rolled-back', 'reason': str(failure), 'downtimeSeconds': time.monotonic() - manager.offline_at, - 'publicStatuses': baseline} - copyparty.atomic(backup / 'result.json', json.dumps(evidence).encode()) - raise - finally: - if database is not None: - database.close() - - -def main(): - parser = argparse.ArgumentParser(description='Review or apply the six Clover share-folder moves.') - parser.add_argument('--apply', action='store_true') - parser.add_argument('--plan', type=Path, help='Save the reviewed plan, or read it when applying.') - args = parser.parse_args() - manager = copyparty.Manager([]) - if args.apply: - if args.plan is None: - parser.error('--apply requires --plan from the reviewed preflight') - lock = os.open(manager.cfg / '.manager.lock', os.O_WRONLY | os.O_CREAT | os.O_NOFOLLOW, 0o600) - with os.fdopen(lock, 'w') as locked: - copyparty.fcntl.flock(locked, copyparty.fcntl.LOCK_EX) - print(json.dumps(apply(manager, json.loads(args.plan.read_text()), Path('/var/lib/studio/copyparty-cutover')))) - else: - value = plan(manager) - if args.plan is not None: - copyparty.atomic(args.plan, json.dumps(value, ensure_ascii=False, indent=2).encode()) - print(json.dumps(value, ensure_ascii=False)) - - -if __name__ == '__main__': - main() diff --git a/tools/copyparty-test.py b/tools/copyparty-test.py index bb4f8600a21c64400057dc520f782833ad83e412..0266e36583509296400a377d2bf72972cd6a5767 100644 --- a/tools/copyparty-test.py +++ b/tools/copyparty-test.py @@ -32,7 +32,7 @@ class Shares(unittest.TestCase): self.subject = str(uuid.uuid4()) self.users = [{'id': self.subject, 'username': 'test-user', 'enabled': True, 'kind': None, 'requiredActions': []}] self.variable = {'ModifyIndex': 1, 'Items': {'idp_header': 'a' * 32, 'unrelated': 'retained'}} - self.environment = patch.dict(os.environ, {'STUDIO_CLOVER_ROOT': str(self.root), 'STUDIO_COPYPARTY_CFG': str(self.cfg)}) + self.environment = patch.dict(os.environ, {'SNOWGLOBE_CLOVER_ROOT': str(self.root), 'SNOWGLOBE_COPYPARTY_CFG': str(self.cfg)}) self.environment.start() self.addCleanup(self.environment.stop) self.nomad = patch.object(shares.runs, 'nomad', side_effect=self.request) @@ -312,7 +312,7 @@ class Shares(unittest.TestCase): clone = self.root.parent / 'clover-clone' clone.mkdir() (clone / 'work').mkdir() - with patch.dict(os.environ, {'STUDIO_CLOVER_ROOT': str(clone)}): + with patch.dict(os.environ, {'SNOWGLOBE_CLOVER_ROOT': str(clone)}): state = self.call('state') self.assertEqual(state['grants'][0]['path'], str(clone / 'work')) self.assertEqual(state['links'][0]['path'], str(clone / 'source')) diff --git a/tools/copyparty.md b/tools/copyparty.md index bf4fc5b5f48fdf09f50b6e873339a5391dbb6848..63fac1babe58fa0de15018956d3909b10b36b0b9 100644 --- a/tools/copyparty.md +++ b/tools/copyparty.md @@ -1,10 +1,9 @@ -# Copyparty administration and folder cutover +# Copyparty administration -Snowglobe's admin Shares page manages native public links and separate API +Snow Globe's Shares page manages native public links and separate API accounts. A person's page assigns folder permissions. Existing Copyparty permissions remain additive; changing a folder is an explicit retarget or move action. New shared folders default to `Share/Name - Description` under Clover. -The legacy `clover/Public` mount is retired; use managed links and grants. Copyparty owns its native `cfg/copyparty/shares.db` and selected-file records. `cfg/manager.json` stores API account labels, UUID-based folder grants, and link @@ -13,89 +12,14 @@ labels. API password hashes live in the service's Nomad variable under config file cannot recover managed accounts. `prepare.py` regenerates config from the canonical service policy and managed state on every deployment. -## Six-folder production move - -`copyparty-cutover.py` contains the requested six source/destination pairs. -It requires a reviewed plan, absent destinations, and same-filesystem renames. -The plan binds the host, service, roots, source metadata, and complete native -share projection. No file contents are read. Public HEAD responses are captured -before stopping the service and verified after restart; existing keys, -passwords, permissions, expirations, and selected-file records must agree. - -Run the script from the reviewed immutable release on the confirmed target: - -```sh -umask 077 -python3 tools/copyparty-cutover.py --plan /root/copyparty-cutover-plan.json -python3 tools/copyparty-cutover.py --apply --plan /root/copyparty-cutover-plan.json -``` - -Copyparty remains stopped through all folder and database changes. A mismatch -restores the old paths and records before restarting the frozen original job. -Mutation verification has an 80-second cutoff; public readiness has a -150-second forward deadline, leaving recovery time within the 285-second -overall budget. Restricted backups and result evidence live under -`/var/lib/studio/copyparty-cutover`. Code rollback does not undo this file move. - -## 2026-10-16 staging evidence - -Zenith preview `copyparty-preview-9909de81` verified API account scope, -permission changes, password/session revocation, signed UUID identity grants, -symlink confinement, selected-file retention, and nested public links. A -disposable offline move completed in 6.94 seconds. The six requested clone -folders were then renamed with unchanged inode/metadata manifests: all 20 -share records and four selected-file rows matched the expected path-only -projection, and all nine affected public URLs returned their original 200 -responses within a 78.2-second upper bound. Production source metadata and -share records remained unchanged during rehearsal. - -## 2026-10-16 production evidence - -Release `765dec72842b4d08` deployed described main revision -`e0cc66a00eb2757846148a241961225d1f9b1253`. All production routes passed -the release workflow's checks. The canonical host unit now serves share -operations; obsolete Win10/ARM host overrides were archived with the release -backup after their effective VM code matched main byte for byte. - -The six requested folders moved to `Share` with their requested names. -All 2,184 metadata entries retained their device, inode, mode, size, and mtime. -All 20 native shares and four selected-file rows matched the path-only -projection, and all nine affected public URLs returned their original 200 -responses. Meadow had no native public link; its folder moved without creating -one. Copyparty remained stopped throughout the folder and database changes. - -The move's stop-to-public-readiness interval was 13.44 seconds. Independent -one-second HTTPS HEAD sampling bounded total public unavailability across -release activation and the move to 55.12 seconds over three intervals. -Restricted plan, database/config backups, frozen job, and result evidence are -in `/var/lib/studio/copyparty-cutover/20261016T082910Z-424b5410`. - -## 2026-10-16 account creation timeout - -The production host unit allowed only loopback traffic, but Podman's loopback -port forwarding translates requests onto its bridge. A read-only health probe -under the same IP restrictions timed out after 3.003 seconds; allowing the -existing Podman subnet returned HTTP 200 in 0.0013 seconds. Account creation -waited for config reload and its recovery reload before returning an error. -The host allowlist now derives the bridge range from the declared Podman default -subnet, preserving the existing network. - -The hardened preview created an account in 0.54 seconds and rotated its -password in 0.60 seconds. Double-clicked submits sent one request each. -Shares-list checks also caught absolute copy-status text expanding the document -outside its scroller. Anchoring that hidden text and bounding the list during -refresh errors kept the document at 720 × 320 in a 720 × 320 viewport, with -only the table scrolling. Its headers and last-row keyboard actions stayed -reachable; the hidden live status remained in the accessibility tree. - -Production account creation completed in 0.60 seconds, rotation in 0.47 -seconds, and deletion in 0.15 seconds under the effective host restrictions. -The disposable account was removed; shares and grants remained unchanged. - -The first rollout, `a8982234e99ab23c`, also exported newer sibling Clover DB -sources. That image failed before database startup because its nonroot user -could not create its launcher cache. Clover DB was restored through the -previous immutable release's service deployment. Its newer source also carries -database migrations outside this change's scope. The build manifest now pins -sitegen revision `13f64d71038f5e72ab34dc1f55805e2fee05f368`; all 183 exported -files, file types, and executable bits match the previous healthy context. +A folder move must update the native share paths and selected-file records +while Copyparty is stopped, preserving keys, passwords, permissions, expirations, +and selected files. Code rollback does not undo filesystem moves. Rehearse on +clones, retain restricted database/config backups, and verify the original +public links after restart. The completed six-folder move's recovery evidence +remains at `/var/lib/snowglobe/copyparty-cutover/20261016T082910Z-424b5410`. + +The privileged host unit reaches Copyparty through Podman's declared bridge +subnet. A loopback-only network allowlist prevents configuration reloads even +when its destination is a loopback-forwarded container port. Keep the derived +bridge access when changing host restrictions. diff --git a/tools/dashboard-auth-test.py b/tools/dashboard-auth-test.py index 9863d8e70fafebdf5180a507b56bd535629dfaf8..2d6a1878897175bbce1f5a9563c12fd7904184b0 100644 --- a/tools/dashboard-auth-test.py +++ b/tools/dashboard-auth-test.py @@ -62,9 +62,9 @@ def main(): proof = uuid.uuid4().hex + uuid.uuid4().hex (data / 'proof').write_text(proof) (data / 'source.json').write_text(json.dumps(export)) - environment = {**os.environ, 'STUDIO_DOMAIN': 'paperclover.net', 'STUDIO_DATA_DIR': str(data), - 'STUDIO_PUBLIC_ORIGIN': origin, 'STUDIO_AUTH_RP_ID': rp, 'STUDIO_FILE_ORIGIN': file, - 'STUDIO_WEB_DIR': str(Path('dashboard/dist').resolve()), 'STUDIO_PROXY_TOKEN_FILE': str(data / 'proof'), 'STUDIO_AUTH_REQUIRED': '1'} + environment = {**os.environ, 'SNOWGLOBE_DOMAIN': 'paperclover.net', 'SNOWGLOBE_DATA_DIR': str(data), + 'SNOWGLOBE_PUBLIC_ORIGIN': origin, 'SNOWGLOBE_AUTH_RP_ID': rp, 'SNOWGLOBE_FILE_ORIGIN': file, + 'SNOWGLOBE_WEB_DIR': str(Path('dashboard/dist').resolve()), 'SNOWGLOBE_PROXY_TOKEN_FILE': str(data / 'proof'), 'SNOWGLOBE_AUTH_REQUIRED': '1'} binary = str(args.binary.resolve()) imported = subprocess.run([binary, '--import-accounts', str(data / 'source.json')], env=environment, capture_output=True, text=True) assert imported.returncode == 0, imported.stderr @@ -97,7 +97,7 @@ def main(): server.terminate(); server.wait(timeout=10) def request(path, method='GET', body=None, cookies=None, status=200, extra=None, host=origin, include_headers=False): - headers = {'Studio-Proxy-Token': proof, 'Host': host.split('://')[1], 'X-Studio-Client-IP': '127.0.0.1'} + headers = {'Snowglobe-Proxy-Token': proof, 'Host': host.split('://')[1], 'X-Snowglobe-Client-IP': '127.0.0.1'} if body is not None: headers.update({'Origin': origin, 'Content-Type': 'application/json'}) if cookies: headers['Cookie'] = '; '.join(f'{k}={v}' for k, v in cookies.items()) headers.update(extra or {}) @@ -120,7 +120,7 @@ def main(): request('/api/account/pictures/missing', status=404) request('/api/me', status=401, extra={'User-Name': name, 'User-Groups': 'infra-admin'}) request('/api/shares', status=401, extra={'User-Name': name, 'User-Groups': 'infra-admin'}) - request('/auth/status', status=403, extra={'Studio-Proxy-Token': 'wrong'}) + request('/auth/status', status=403, extra={'Snowglobe-Proxy-Token': 'wrong'}) csrf = request('/auth/status', cookies=cookies)['csrf'] csrf_headers = request('/auth/status', cookies=cookies, include_headers=True) assert csrf_headers['set-cookie'].startswith('__Host-snow-csrf=') diff --git a/tools/dashboard-backup-test.py b/tools/dashboard-backup-test.py index 3130548cc3ca79b63443b56577ffe0ff136d8f5b..42f267a86d9f847bb424063b0869f8b184b8f70b 100644 --- a/tools/dashboard-backup-test.py +++ b/tools/dashboard-backup-test.py @@ -33,11 +33,14 @@ def main(): assert db.execute('SELECT value FROM durable').fetchall() == [(name,)] revision = 'a' * 16 (directory / 'manifest.json').write_text(json.dumps({'id': backup_id, 'fromRelease': revision, 'services': {'dashboard': {'files': files}}})) - data.current_release = lambda: Path('/releases') / revision + active = data.STATE / "releases" / revision + (active / "tools").mkdir(parents=True) + (active / "tools/snowglobe.py").touch() + data.current_release = lambda: active lifecycle = [] def run(*args, **kwargs): lifecycle.append(args) - if args == ('systemctl', 'stop', 'studio-dashboard'): + if args == ('systemctl', 'stop', 'snowglobe-dashboard'): for db in connections: db.close() data.run = run @@ -47,7 +50,7 @@ def main(): db.commit() (live / 'pictures' / 'avatar').write_bytes(b'later picture') data.restore(backup_id, 'dashboard') - assert lifecycle == [('systemctl', 'stop', 'studio-dashboard'), ('systemctl', 'start', 'studio-dashboard'), ('systemctl', 'is-active', '--quiet', 'studio-dashboard')] + assert lifecycle == [('systemctl', 'stop', 'snowglobe-dashboard'), ('systemctl', 'start', 'snowglobe-dashboard'), ('systemctl', 'is-active', '--quiet', 'snowglobe-dashboard')] assert (live / 'pictures' / 'avatar').read_bytes() == b'original picture' for name in ['accounts.sqlite', 'connections.sqlite']: with sqlite3.connect(live / name) as db: diff --git a/tools/dashboard-bench.py b/tools/dashboard-bench.py index d1296a7648ba4f641fe69b9df5de3736c9593c35..1a4c9a4d2de757a374431ae68c61c3b1dd841681 100644 --- a/tools/dashboard-bench.py +++ b/tools/dashboard-bench.py @@ -12,7 +12,7 @@ parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--clients", type=int, default=24) parser.add_argument("--rounds", type=int, default=20) parser.add_argument("--port", type=int, default=7072) -parser.add_argument("--unit", default="studio-dashboard.service") +parser.add_argument("--unit", default="snowglobe-dashboard.service") parser.add_argument("--include-files", action="store_true") args = parser.parse_args() if args.clients < 1 or args.rounds < 1: diff --git a/tools/dashboard-browser-test.py b/tools/dashboard-browser-test.py index 8cba87bcbdf9ac6b74301bccbe4bda869e896928..4d4f8fba4cf904f641faaed5898ad61ceb5878ee 100644 --- a/tools/dashboard-browser-test.py +++ b/tools/dashboard-browser-test.py @@ -23,7 +23,7 @@ def main(): original = routes.read_bytes() metadata = routes.stat() candidate = router.render(Path(router.TOKEN).read_text().strip()) - start = "globe." + os.environ["STUDIO_DOMAIN"] + " {" + start = "globe." + os.environ["SNOWGLOBE_DOMAIN"] + " {" begin = original.index(start.encode()) depth = 0 end = None @@ -33,9 +33,9 @@ def main(): end = position + 1 break assert end is not None - replacement = candidate[candidate.index(start):candidate.index("dashboard.internal." + os.environ["STUDIO_DOMAIN"] + ":")].strip().encode() + replacement = candidate[candidate.index(start):candidate.index("dashboard.internal." + os.environ["SNOWGLOBE_DOMAIN"] + ":")].strip().encode() content = original[:begin] + replacement + original[end:] - active = subprocess.run(["systemctl", "is-active", "studio-router"], capture_output=True).returncode == 0 + active = subprocess.run(["systemctl", "is-active", "snowglobe-router"], capture_output=True).returncode == 0 pending = routes.with_name("routes.browser-test.pending") def install(value): @@ -47,9 +47,9 @@ def main(): try: if active: - subprocess.run(["systemctl", "stop", "studio-router"], check=True, capture_output=True, timeout=30) + subprocess.run(["systemctl", "stop", "snowglobe-router"], check=True, capture_output=True, timeout=30) install(content) - args.ready_file.write_text(json.dumps({"origin": "https://globe." + os.environ["STUDIO_DOMAIN"], "ready": True}) + "\n") + args.ready_file.write_text(json.dumps({"origin": "https://globe." + os.environ["SNOWGLOBE_DOMAIN"], "ready": True}) + "\n") args.ready_file.chmod(0o600) deadline = time.monotonic() + args.timeout while not stop.exists() and time.monotonic() < deadline: @@ -60,7 +60,7 @@ def main(): assert hashlib.sha256(routes.read_bytes()).digest() == hashlib.sha256(original).digest() finally: if active: - subprocess.run(["systemctl", "start", "studio-router"], check=True, capture_output=True, timeout=30) + subprocess.run(["systemctl", "start", "snowglobe-router"], check=True, capture_output=True, timeout=30) pending.unlink(missing_ok=True) args.ready_file.unlink(missing_ok=True) stop.unlink(missing_ok=True) diff --git a/tools/dashboard-deploy-test.py b/tools/dashboard-deploy-test.py index 677d6ad20d6f9392ebad13dd45eb538720706eb7..bd710ce71a75ab5d15cdc140abdf67846ad7bf3c 100644 --- a/tools/dashboard-deploy-test.py +++ b/tools/dashboard-deploy-test.py @@ -18,7 +18,7 @@ host = importlib.import_module("dashboard-host") class DeploymentBoundary(unittest.TestCase): def setUp(self): - self.temporary = tempfile.TemporaryDirectory(prefix="studio-deploy-boundary-") + self.temporary = tempfile.TemporaryDirectory(prefix="snowglobe-deploy-boundary-") self.addCleanup(self.temporary.cleanup) self.root = Path(self.temporary.name).resolve() self.state = self.root / "state" @@ -31,7 +31,7 @@ class DeploymentBoundary(unittest.TestCase): context = patch.object(release, name, value) context.start() self.addCleanup(context.stop) - self.host = host.Host("studio-demo") + self.host = host.Host("snowglobe-demo") state_context = patch.object(runs, "HOST_STATE", self.state / "host") state_context.start() self.addCleanup(state_context.stop) @@ -44,8 +44,8 @@ class DeploymentBoundary(unittest.TestCase): path.write_text("fixture") else: path.mkdir() - (self.snapshot / "tools/studio.py").write_text("print('fixture')") - (self.snapshot / ".studio-release.json").write_text(json.dumps({"id": self.version, "digest": release.tree_digest(self.snapshot), "version": 2, "main": {"commit": "b" * 40, "description": "Fixture main"}})) + (self.snapshot / "tools/snowglobe.py").write_text("print('fixture')") + (self.snapshot / ".snowglobe-release.json").write_text(json.dumps({"id": self.version, "digest": release.tree_digest(self.snapshot), "version": 2, "main": {"commit": "b" * 40, "description": "Fixture main"}})) (self.state / "stages").mkdir() self.stage = "fixture-preview" self.stage_file = self.state / "stages" / (self.stage + ".json") @@ -76,13 +76,13 @@ class DeploymentBoundary(unittest.TestCase): executed.assert_not_called() def test_commands_come_from_verified_releases_and_history(self): - self.assertEqual(runs.command("destroy", self.stage)[1:], [str(self.snapshot / "tools/studio.py"), "destroy", self.stage]) + self.assertEqual(runs.command("destroy", self.stage)[1:], [str(self.snapshot / "tools/snowglobe.py"), "destroy", self.stage]) self.assertEqual(runs.command("deploy", self.version)[1:], [str(Path(runs.__file__).with_name("release.py")), "deploy", self.version]) self.assertEqual(runs.command("rollback", "1")[1:], [str(Path(runs.__file__).with_name("release.py")), "rollback", self.version]) - self.assertEqual(runs.command("start", "fixture")[1:], [str(self.snapshot / "tools/studio.py"), "deploy", "fixture"]) + self.assertEqual(runs.command("start", "fixture")[1:], [str(self.snapshot / "tools/snowglobe.py"), "deploy", "fixture"]) self.assertEqual(runs.command("stop", "fixture"), ["nomad", "job", "stop", "-yes", "fixture"]) self.assertEqual(runs.command("restart", "fixture"), ["nomad", "job", "restart", "-yes", "fixture"]) - (self.snapshot / "tools/studio.py").write_text("changed") + (self.snapshot / "tools/snowglobe.py").write_text("changed") for action, target in [("destroy", self.stage), ("deploy", self.version), ("rollback", "1"), ("start", "fixture")]: with self.subTest(action=action), self.assertRaisesRegex(ValueError, "contents changed"): runs.command(action, target) @@ -91,11 +91,11 @@ class DeploymentBoundary(unittest.TestCase): with patch.object(release, "SOURCES", tuple(name for name in release.SOURCES if name != "guest")): digest = release.tree_digest(self.snapshot) shutil.rmtree(self.snapshot / "guest") - manifest = self.snapshot / ".studio-release.json" + manifest = self.snapshot / ".snowglobe-release.json" details = json.loads(manifest.read_text()) details["digest"] = digest manifest.write_text(json.dumps(details)) - self.assertEqual(runs.command("destroy", self.stage)[1:], [str(self.snapshot / "tools/studio.py"), "destroy", self.stage]) + self.assertEqual(runs.command("destroy", self.stage)[1:], [str(self.snapshot / "tools/snowglobe.py"), "destroy", self.stage]) shutil.rmtree(self.snapshot / "tools") with self.assertRaisesRegex(ValueError, "contents changed"): runs.command("destroy", self.stage) @@ -137,7 +137,7 @@ class DeploymentBoundary(unittest.TestCase): with self.assertRaisesRegex(ValueError, "main changed"): release.main() activate.assert_not_called() - manifest = self.snapshot / ".studio-release.json" + manifest = self.snapshot / ".snowglobe-release.json" original = json.loads(manifest.read_text()) for revision in [None, {"commit": "b" * 40, "description": " "}, {"commit": "escape", "description": "main"}]: manifest.write_text(json.dumps({**original, "main": revision})) @@ -169,7 +169,7 @@ class DeploymentBoundary(unittest.TestCase): output = 'job "fixture" {\n' if argv[-1] == "render" else '"leader"' return subprocess.CompletedProcess(argv, 0, stdout=output, stderr="") - with patch.object(release.subprocess, "run", side_effect=execute): + with patch.object(release.subprocess, "run", side_effect=execute), patch.object(release, "host_configuration", return_value="snowglobe"), patch.object(release.transition, "activate"): self.assertIsNone(release.activate(self.version, initial=True)) self.assertEqual(release.current_release(), self.version) @@ -205,7 +205,7 @@ class DeploymentBoundary(unittest.TestCase): root = runs.HOST_STATE (root / "runs").mkdir(exist_ok=True) (root / "runs" / (started["id"] + ".log")).write_text("fixture\n") - restored = host.Host("studio-demo") + restored = host.Host("snowglobe-demo") active = subprocess.CompletedProcess([], 0, stdout="ActiveState=active\nExecMainStatus=0\nLoadState=loaded\n") with patch.object(runs.subprocess, "run", return_value=active): self.assertEqual(restored.handle({"operation": "deploy.last"}), {**started, "code": None}) @@ -249,7 +249,7 @@ class DeploymentBoundary(unittest.TestCase): def test_secret_scope_and_compare_and_set_preserve_siblings(self): (self.state / "nomad.token").write_text("private-fixture-token") - job = {"Meta": {"studio_secrets": json.dumps([{"name": "token", "generated": True, "bytes": 16}])}} + job = {"Meta": {"snowglobe_secrets": json.dumps([{"name": "token", "generated": True, "bytes": 16}])}} existing = {"ModifyIndex": 7, "Items": {"token": "old", "sibling": "keep", "undeclared": "private"}} with patch.object(runs, "nomad", side_effect=[job, existing]) as requested: self.assertEqual(runs.secret("get", "fixture", "token"), "old") @@ -266,11 +266,11 @@ class DeploymentBoundary(unittest.TestCase): def test_rotations_use_the_recorded_size_and_refuse_external_secrets(self): (self.state / "nomad.token").write_text("private-fixture-token") for generated, count in [(False, 16), (True, True), (True, 4097), (True, 0)]: - job = {"Meta": {"studio_secrets": json.dumps([{"name": "token", "generated": generated, "bytes": count}])}} + job = {"Meta": {"snowglobe_secrets": json.dumps([{"name": "token", "generated": generated, "bytes": count}])}} with patch.object(runs, "nomad", side_effect=[job, None]) as requested, self.assertRaises(runs.Error): runs.secret("rotate", "fixture", "token") self.assertEqual(requested.call_count, 2) - job = {"Meta": {"studio_secrets": '[{"name":"token","generated":true,"bytes":16}]'}} + job = {"Meta": {"snowglobe_secrets": '[{"name":"token","generated":true,"bytes":16}]'}} with patch.object(runs, "nomad", side_effect=[job, None, {}]) as requested, patch.object(runs.subprocess, "run", return_value=subprocess.CompletedProcess([], 0)): runs.secret("rotate", "fixture", "token") value = requested.call_args.args[2]["Items"]["token"] diff --git a/tools/dashboard-deploy-vm-test.py b/tools/dashboard-deploy-vm-test.py index e052d21447834b560067a4208fe21a50f9a33254..e1508330425d4701342ca3cc1fb91f8dc9c9c553 100644 --- a/tools/dashboard-deploy-vm-test.py +++ b/tools/dashboard-deploy-vm-test.py @@ -29,7 +29,7 @@ def main(): service_id = "boundary-service-" + suffix managed_file = release.STATE / "managed-jobs.json" managed_bytes = managed_file.read_bytes() - container = "studio-dashboard-deploy-test-" + suffix + container = "snowglobe-dashboard-deploy-test-" + suffix current = release.current_release() history_bytes = release.HISTORY.read_bytes() stages_before = {path.name for path in (release.STATE / "stages").glob("*.json")} @@ -55,7 +55,7 @@ def main(): def http(path, body=None, status=200, section="deploys", groups="infra-admin", method=None): request = urllib.request.Request("http://127.0.0.1:7076/api/" + section + path, data=json.dumps(body).encode() if body is not None else None, method=method, - headers={"User-Name": "fixture", "User-Groups": groups, "Studio-Proxy-Token": proof, "Content-Type": "application/json"}) + headers={"User-Name": "fixture", "User-Groups": groups, "Snowglobe-Proxy-Token": proof, "Content-Type": "application/json"}) try: response = urllib.request.urlopen(request, timeout=40) except urllib.error.HTTPError as error: @@ -95,7 +95,7 @@ def main(): time.sleep(.1) raise AssertionError("container deployment view did not start") - with tempfile.TemporaryDirectory(prefix="studio-deploy-boundary-", dir="/run") as temporary: + with tempfile.TemporaryDirectory(prefix="snowglobe-deploy-boundary-", dir="/run") as temporary: data = Path(temporary) / "data" data.mkdir() os.chown(data, 65534, 65534) @@ -139,24 +139,24 @@ def main(): path.write_text("fixture") else: path.mkdir() - script = fixture_release / "tools/studio.py" + script = fixture_release / "tools/snowglobe.py" script.write_text("import time\ntime.sleep(3)\nprint('approved fixture worker')\n") digest = release.tree_digest(fixture_release) destination = release.RELEASES / digest[:16] assert not destination.exists() fixture_release.rename(destination) fixture_release = destination - (fixture_release / ".studio-release.json").write_text(json.dumps({"id": digest[:16], "digest": digest, "version": 2})) + (fixture_release / ".snowglobe-release.json").write_text(json.dumps({"id": digest[:16], "digest": digest, "version": 2})) value = json.loads(stage_file.read_text()) stage_file.write_text(json.dumps({**value, "release": digest[:16]})) active = call("deploy.start", action="destroy", target=stage_id) owned_runs.append(active["id"]) call("deploy.start", action="deploy", target=current, status=409) - shell("systemctl", "restart", "studio-host-boundary-test.service") + shell("systemctl", "restart", "snowglobe-host-boundary-test.service") assert call("deploy.last")["id"] == active["id"] result = done(active["id"]) assert result["code"] == 0 and "approved fixture worker" in result["lines"], result - (fixture_release / "tools/studio.py").write_text("print('unapproved changed worker')") + (fixture_release / "tools/snowglobe.py").write_text("print('unapproved changed worker')") call("deploy.start", action="destroy", target=stage_id, status=502) assert call("deploy.last")["id"] == active["id"] stage_file.write_text(json.dumps(value)) @@ -166,9 +166,9 @@ def main(): shell("podman", "run", "-d", "--name=" + container, "--pull=never", "--user=65534:65534", "--read-only", "--cap-drop=all", "--security-opt=no-new-privileges", "--pids-limit=128", "--memory=512m", "--cpus=2", "--tmpfs=/tmp:rw,noexec,nosuid,nodev,size=64m", - "--publish=127.0.0.1:7076:7072", "--volume=" + str(args.socket.parent) + ":/run/studio-host:ro", + "--publish=127.0.0.1:7076:7072", "--volume=" + str(args.socket.parent) + ":/run/snowglobe-host:ro", "--volume=" + str(proxy_token) + ":/run/secrets/dashboard-proxy.token:ro", - "--volume=" + str(data) + ":/data:rw,nosuid,nodev", "--env=STUDIO_DATA_DIR=/data", args.image) + "--volume=" + str(data) + ":/data:rw,nosuid,nodev", "--env=SNOWGLOBE_DATA_DIR=/data", args.image) overview = ready() assert overview["current"] == current and overview["recorded"] assert len(overview["history"]) == len(json.loads(history_bytes)) @@ -187,7 +187,7 @@ def main(): managed_file.write_text(json.dumps([*json.loads(managed_bytes), service_id])) service_created = True nomad("jobs", {"Job": {"ID": service_id, "Name": service_id, "Namespace": "default", "Datacenters": ["*"], "Type": "service", - "Meta": {"studio_secrets": json.dumps([{"name": "fixture", "generated": True, "bytes": 16}, {"name": "outside", "generated": False}])}, + "Meta": {"snowglobe_secrets": json.dumps([{"name": "fixture", "generated": True, "bytes": 16}, {"name": "outside", "generated": False}])}, "TaskGroups": [{"Name": "fixture", "Count": 1, "Tasks": [{"Name": "idle", "Driver": "podman", "Config": {"image": "docker.io/library/alpine:3.22", "command": "/bin/sleep", "args": ["600"]}, "Resources": {"CPU": 25, "MemoryMB": 32}}]}]}}) @@ -219,7 +219,7 @@ def main(): assert secret_value not in "\n".join(call("deploy.run", id=saved["id"])["lines"]) assert http(prefix + "fixture", section="services")["value"] == secret_value assert nomad("var/nomad/jobs/" + service_id)["Items"]["outside"] == "keep" - assert not list(Path("/var/lib/studio/host-boundary-test/runs").glob("*.input")) + assert not list(Path("/var/lib/snowglobe/host-boundary-test/runs").glob("*.input")) http(prefix + "fixture/rotate", {}, status=204, section="services") saved = call("deploy.last") owned_runs.append(saved["id"]) @@ -284,7 +284,7 @@ def main(): finally: subprocess.run(["podman", "rm", "--force", container], capture_output=True) for identity in owned_runs: - subprocess.run(["systemctl", "stop", "studio-run-" + identity], capture_output=True, timeout=15) + subprocess.run(["systemctl", "stop", "snowglobe-run-" + identity], capture_output=True, timeout=15) if service_created: nomad("job/" + service_id + "?purge=true", method="DELETE") nomad("var/nomad/jobs/" + service_id, method="DELETE") diff --git a/tools/dashboard-file-boundary-test.py b/tools/dashboard-file-boundary-test.py index 0ffd92776210700abdddfbe936456f9cb263bc5b..f59c7405425f470caa17d9412daf9ab3110151df 100644 --- a/tools/dashboard-file-boundary-test.py +++ b/tools/dashboard-file-boundary-test.py @@ -18,7 +18,7 @@ import uuid def main(): parser = argparse.ArgumentParser() parser.add_argument("socket", type=Path) - parser.add_argument("--pool", default="studio-demo") + parser.add_argument("--pool", default="snowglobe-demo") parser.add_argument("--image", required=True) parser.add_argument("--output", type=Path) args = parser.parse_args() @@ -27,7 +27,7 @@ def main(): dataset = args.pool + "/files-boundary-test-" + suffix mount = Path("/srv/.files-boundary-test-" + suffix) private = Path("/srv/.files-private-test-" + suffix) - container = "studio-dashboard-files-test-" + suffix + container = "snowglobe-dashboard-files-test-" + suffix def run(*argv): return subprocess.run(argv, check=True, capture_output=True, text=True).stdout @@ -35,7 +35,7 @@ def main(): def call(path, body=None, status=200): req = urllib.request.Request("http://127.0.0.1:7075" + path, data=json.dumps(body).encode() if body is not None else None, - headers={"User-Name": "fixture", "User-Groups": "infra-admin", "Studio-Proxy-Token": proof, "Content-Type": "application/json"}) + headers={"User-Name": "fixture", "User-Groups": "infra-admin", "Snowglobe-Proxy-Token": proof, "Content-Type": "application/json"}) try: response = urllib.request.urlopen(req, timeout=30) except urllib.error.HTTPError as error: @@ -75,7 +75,7 @@ def main(): time.sleep(.1) raise AssertionError((tree, meta)) - with tempfile.TemporaryDirectory(prefix="studio-file-boundary-", dir="/run") as temporary: + with tempfile.TemporaryDirectory(prefix="snowglobe-file-boundary-", dir="/run") as temporary: os.chown(temporary, 65534, 65534) data = Path(temporary) / "data" data.mkdir() @@ -98,17 +98,17 @@ def main(): launch = ["podman", "run", "-d", "--name=" + container, "--pull=never", "--user=65534:65534", "--read-only", "--cap-drop=all", "--security-opt=no-new-privileges", "--pids-limit=128", "--memory=512m", "--cpus=2", "--tmpfs=/tmp:rw,noexec,nosuid,nodev,size=64m", - "--publish=127.0.0.1:7075:7072", "--volume=" + str(args.socket.parent) + ":/run/studio-host:ro", + "--publish=127.0.0.1:7075:7072", "--volume=" + str(args.socket.parent) + ":/run/snowglobe-host:ro", "--volume=" + str(proxy_token) + ":/run/secrets/dashboard-proxy.token:ro", "--volume=" + str(data) + ":/data:rw,nosuid,nodev", - "--env=STUDIO_DATA_DIR=/data", "--env=STUDIO_FILES_WRITABLE=1"] + "--env=SNOWGLOBE_DATA_DIR=/data", "--env=SNOWGLOBE_FILES_WRITABLE=1"] run(*launch, "--mount=type=bind,src=" + str(mount) + ",dst=" + str(mount) + ",bind-nonrecursive,bind-propagation=rslave", "--mount=type=bind,src=" + str(mount / "media") + ",dst=" + str(mount / "media") + ",bind-nonrecursive,bind-propagation=rslave", "--mount=type=bind,src=" + str(mount / "docs") + ",dst=" + str(mount / "docs") + ",bind-nonrecursive,bind-propagation=rslave", - "--env=STUDIO_STORE_ROOT=" + str(mount), - "--env=STUDIO_MEDIA_ROOT=" + str(mount / "media"), - "--env=STUDIO_INDEX_POOL=" + dataset, "--env=STUDIO_INDEX_DIR=/data/index", args.image) + "--env=SNOWGLOBE_STORE_ROOT=" + str(mount), + "--env=SNOWGLOBE_MEDIA_ROOT=" + str(mount / "media"), + "--env=SNOWGLOBE_INDEX_POOL=" + dataset, "--env=SNOWGLOBE_INDEX_DIR=/data/index", args.image) ready() baseline_size = len("media fixture") + len("document fixture") baseline = indexed(baseline_size, 2) @@ -242,7 +242,7 @@ def main(): (partial / "fixture").write_text("partial mount fixture") os.chown(partial / "fixture", 65534, 65534) run(*launch, "--volume=" + str(partial) + ":" + str(partial) + ":rw,nosuid,nodev", - "--env=STUDIO_STORE_ROOT=" + str(private), args.image) + "--env=SNOWGLOBE_STORE_ROOT=" + str(private), args.image) ready() call("/api/storage/files/delete", {"paths": ["data/fixture"]}, 409) assert (partial / "fixture").read_text() == "partial mount fixture" diff --git a/tools/dashboard-files-test.py b/tools/dashboard-files-test.py index feea6c4bf057fe23bbe364eb2f7b8c2e8821dea4..2413025849c459c7fc0c7b4447b4435a18a9f64e 100644 --- a/tools/dashboard-files-test.py +++ b/tools/dashboard-files-test.py @@ -4,17 +4,17 @@ import argparse,json,os,shlex,subprocess,tempfile,time,urllib.request,urllib.err from pathlib import Path parser=argparse.ArgumentParser(description=__doc__) parser.add_argument('binary') -parser.add_argument('--pool',default='studio-demo') +parser.add_argument('--pool',default='snowglobe-demo') args=parser.parse_args() run=lambda *argv:subprocess.run(argv,check=True,capture_output=True,text=True) dataset=args.pool+'/dashboard-test-'+uuid.uuid4().hex[:8] -unit='studio-dashboard-files-test' -with tempfile.TemporaryDirectory(prefix='studio-files-test-') as temporary: +unit='snowglobe-dashboard-files-test' +with tempfile.TemporaryDirectory(prefix='snowglobe-files-test-') as temporary: root=Path(temporary)/'store';media=root/'media';docs=root/'work/docs' - variables=dict(v.split('=',1) for v in shlex.split(run('systemctl','show','-P','Environment','studio-dashboard').stdout)) - for key in ['STUDIO_INDEX_POOL','STUDIO_YT_STATE']:variables.pop(key,None) - variables.update(PORT='7074',STUDIO_METRICS='follow',STUDIO_STORE_ROOT=str(root),STUDIO_MEDIA_ROOT=str(media),STUDIO_DATA_DIR=temporary+'/state',STUDIO_FILES_WRITABLE='1',STUDIO_INDEX_POOL=dataset,STUDIO_INDEX_DIR=temporary+'/index') - proof={'Studio-Proxy-Token':Path(variables['STUDIO_PROXY_TOKEN_FILE']).read_text().strip()} if 'STUDIO_PROXY_TOKEN_FILE' in variables else {} + variables=dict(v.split('=',1) for v in shlex.split(run('systemctl','show','-P','Environment','snowglobe-dashboard').stdout)) + for key in ['SNOWGLOBE_INDEX_POOL','SNOWGLOBE_YT_STATE']:variables.pop(key,None) + variables.update(PORT='7074',SNOWGLOBE_METRICS='follow',SNOWGLOBE_STORE_ROOT=str(root),SNOWGLOBE_MEDIA_ROOT=str(media),SNOWGLOBE_DATA_DIR=temporary+'/state',SNOWGLOBE_FILES_WRITABLE='1',SNOWGLOBE_INDEX_POOL=dataset,SNOWGLOBE_INDEX_DIR=temporary+'/index') + proof={'Snowglobe-Proxy-Token':Path(variables['SNOWGLOBE_PROXY_TOKEN_FILE']).read_text().strip()} if 'SNOWGLOBE_PROXY_TOKEN_FILE' in variables else {} def start(): run('systemd-run','--unit='+unit,'--collect','--property=CPUWeight=1000','--property=CPUQuota=100%','--property=PrivateMounts=yes',*['--setenv='+k+'='+v for k,v in variables.items()],str(Path(args.binary).resolve())) deadline=time.monotonic()+120 @@ -75,7 +75,7 @@ with tempfile.TemporaryDirectory(prefix='studio-files-test-') as temporary: call('/api/media/ops/'+op['id']+'/undo',{},409);assert (media/'new/extra').exists() (media/'new/extra').unlink();undo(op) assert not any(entry['name']=='new' for entry in call('/api/media/list')['entries']) - run('systemctl','stop',unit);variables.pop('STUDIO_FILES_WRITABLE');start() + run('systemctl','stop',unit);variables.pop('SNOWGLOBE_FILES_WRITABLE');start() call('/api/media/folder',{'path':'','name':'denied'},501);assert not (media/'denied').exists() print('PASS: tree totals, text preview, path confinement, bulk rename, atomic clash refusal, collapsed move, dataset protection, cross-dataset refusal, multi-dataset delete/undo, persistent journal, index wake after mutation, occupied-folder refusal, read-only mode') finally: diff --git a/tools/dashboard-host-test.py b/tools/dashboard-host-test.py index b643273627b0db20e573fa0eefd5b6b628a31867..3d9fd67e2c3a08d0ef1b396f55398288d157c2bf 100644 --- a/tools/dashboard-host-test.py +++ b/tools/dashboard-host-test.py @@ -19,7 +19,7 @@ spec.loader.exec_module(host_module) class Boundary(unittest.TestCase): def setUp(self): - self.host = host_module.Host("studio-demo") + self.host = host_module.Host("snowglobe-demo") @unittest.skipUnless(sys.platform == "linux", "Linux Unix peer credentials") def test_large_json_header_preserves_stream_bytes_and_enforces_limit(self): @@ -62,15 +62,15 @@ class Boundary(unittest.TestCase): {"operation": "storage.datasets", "env": {"PATH": "/tmp"}}, {"operation": "host.sample", "path": "/root/private"}, {"operation": "host.usage", "refresh": True, "pid": 1}, - {"operation": "deploy.managed", "path": "/var/lib/studio/nomad.token"}, + {"operation": "deploy.managed", "path": "/var/lib/snowglobe/nomad.token"}, {"operation": "storage.snapshots"}] for refresh in [None, [], {}, 0, 1, "true"]: requests.append({"operation": "host.usage", "refresh": refresh}) - for dataset in [None, [], "other", "studio-demo-other", "studio-demo/../other", - "studio-demo//child", "studio-demo/child\nother", "-r", "studio-demo/a@snap"]: + for dataset in [None, [], "other", "snowglobe-demo-other", "snowglobe-demo/../other", + "snowglobe-demo//child", "snowglobe-demo/child\nother", "-r", "snowglobe-demo/a@snap"]: requests.append({"operation": "storage.snapshots", "dataset": dataset}) for snapshot in [None, [], "a%b", "a,b", "a@b", "a\nb", "../other", "a;id"]: - requests.append({"operation": "storage.reclaim", "dataset": "studio-demo/a", + requests.append({"operation": "storage.reclaim", "dataset": "snowglobe-demo/a", "from": snapshot, "to": "safe"}) with patch.object(host_module, "command") as command: for request in requests: @@ -94,32 +94,32 @@ class Boundary(unittest.TestCase): def test_reclaim_is_always_a_dry_run(self): with patch.object(host_module, "command", return_value="reclaim\t1024") as command: - self.host.handle({"operation": "storage.reclaim", "dataset": "studio-demo/a", + self.host.handle({"operation": "storage.reclaim", "dataset": "snowglobe-demo/a", "from": "snapshot one", "to": "snapshot two"}) command.assert_called_once_with("zfs", "destroy", "-nvp", - "studio-demo/a@snapshot one%snapshot two") + "snowglobe-demo/a@snapshot one%snapshot two") def test_mounts_exclude_other_pools(self): mounts = {"filesystems": [{"source": source, "target": "/srv"} - for source in ["studio-demo", "studio-demo/a", "studio-demo-other/a", "other/a", "studio-demo/a@dash-123"]]} + for source in ["snowglobe-demo", "snowglobe-demo/a", "snowglobe-demo-other/a", "other/a", "snowglobe-demo/a@dash-123"]]} with patch.object(host_module, "command", return_value=json.dumps(mounts)): self.assertEqual([m["source"] for m in self.host.handle({"operation": "storage.mounts"})["filesystems"]], - ["studio-demo", "studio-demo/a"]) + ["snowglobe-demo", "snowglobe-demo/a"]) def test_snapshot_arguments_cannot_be_options(self): with patch.object(host_module, "command", return_value="") as command: - self.host.handle({"operation": "storage.removed", "dataset": "studio-demo/a", "snapshot": "snapshot -r"}) - command.assert_called_once_with("zfs", "diff", "-H", "studio-demo/a@snapshot -r", "studio-demo/a") + self.host.handle({"operation": "storage.removed", "dataset": "snowglobe-demo/a", "snapshot": "snapshot -r"}) + command.assert_called_once_with("zfs", "diff", "-H", "snowglobe-demo/a@snapshot -r", "snowglobe-demo/a") def test_snapshot_mutations_are_scoped(self): requests = [] - for datasets in [None, {}, [], [None], ["other/private"], ["studio-demo/../escape"], ["-r"]]: + for datasets in [None, {}, [], [None], ["other/private"], ["snowglobe-demo/../escape"], ["-r"]]: requests.append({"operation": "files.snapshot", "datasets": datasets}) for snapshot in ["before", "dash-1%after", "dash-1,after", "dash-../other", "dash--r"]: - requests.append({"operation": "files.discard", "dataset": "studio-demo/a", "snapshot": snapshot}) + requests.append({"operation": "files.discard", "dataset": "snowglobe-demo/a", "snapshot": snapshot}) for value in ["before%after", "-r", "before,after", "before@after"]: - requests.append({"operation": "storage.destroy", "dataset": "studio-demo/a", "from": value, "to": "after"}) - requests.append({"operation": "files.snapshot", "datasets": ["studio-demo/a"], "recursive": True}) + requests.append({"operation": "storage.destroy", "dataset": "snowglobe-demo/a", "from": value, "to": "after"}) + requests.append({"operation": "files.snapshot", "datasets": ["snowglobe-demo/a"], "recursive": True}) with patch.object(host_module, "command") as command: for request in requests: with self.subTest(request=request), self.assertRaises(host_module.Rejected): @@ -128,14 +128,14 @@ class Boundary(unittest.TestCase): def test_file_snapshots_use_only_the_undo_namespace(self): with patch.object(host_module.time, "time_ns", return_value=123), patch.object(host_module, "command", return_value="") as command: - name = self.host.handle({"operation": "files.snapshot", "datasets": ["studio-demo/b", "studio-demo/a", "studio-demo/a"]}) + name = self.host.handle({"operation": "files.snapshot", "datasets": ["snowglobe-demo/b", "snowglobe-demo/a", "snowglobe-demo/a"]}) self.assertEqual(name, "dash-123") - command.assert_called_once_with("zfs", "snapshot", "studio-demo/a@dash-123", "studio-demo/b@dash-123") - with patch.object(host_module, "command", return_value="studio-demo/a@before\nstudio-demo/a@dash-123\nstudio-demo/a@dash-manual\n"): - self.assertEqual(self.host.handle({"operation": "files.snapshots", "datasets": ["studio-demo/a"]}), ["studio-demo/a@dash-123"]) + command.assert_called_once_with("zfs", "snapshot", "snowglobe-demo/a@dash-123", "snowglobe-demo/b@dash-123") + with patch.object(host_module, "command", return_value="snowglobe-demo/a@before\nsnowglobe-demo/a@dash-123\nsnowglobe-demo/a@dash-manual\n"): + self.assertEqual(self.host.handle({"operation": "files.snapshots", "datasets": ["snowglobe-demo/a"]}), ["snowglobe-demo/a@dash-123"]) with patch.object(host_module, "command", return_value="") as command: - self.host.handle({"operation": "files.discard", "dataset": "studio-demo/a", "snapshot": "dash-123"}) - command.assert_called_once_with("zfs", "destroy", "studio-demo/a@dash-123") + self.host.handle({"operation": "files.discard", "dataset": "snowglobe-demo/a", "snapshot": "dash-123"}) + command.assert_called_once_with("zfs", "destroy", "snowglobe-demo/a@dash-123") def test_child_output_is_bounded(self): with patch.object(host_module, "MAX_RESPONSE", 4096): @@ -143,7 +143,7 @@ class Boundary(unittest.TestCase): host_module.command(sys.executable, "-c", "import os; os.write(2, b'x' * 1000000)") def test_index_operations_preserve_other_snapshots(self): - dataset = "studio-demo/a" + dataset = "snowglobe-demo/a" with patch.object(host_module, "command") as command: for name in ["manual", "dash-123", "index-fixture", "index-123%manual", "index-../escape"]: for request in [{"operation": "index.discard", "dataset": dataset, "snapshot": name}, diff --git a/tools/dashboard-host-vm-test.py b/tools/dashboard-host-vm-test.py index a064aa840f8ab8870b0fb410e0184ee3e25baae7..ba978950ce6526e3353a8f9b84fe88fc470131a4 100644 --- a/tools/dashboard-host-vm-test.py +++ b/tools/dashboard-host-vm-test.py @@ -35,7 +35,7 @@ def request(socket_path, payload): def main(): parser = argparse.ArgumentParser() parser.add_argument("socket") - parser.add_argument("--pool", default="studio-demo") + parser.add_argument("--pool", default="snowglobe-demo") parser.add_argument("--user", default="nobody") parser.add_argument("--client", action="store_true") args = parser.parse_args() diff --git a/tools/dashboard-host.py b/tools/dashboard-host.py index ee91e1c4b988199afe3a846489ba698d25e60300..adaa9dad30877d9aa24dbaacd0be97f7b3ce1c7e 100644 --- a/tools/dashboard-host.py +++ b/tools/dashboard-host.py @@ -430,7 +430,7 @@ def stream_upload(connection, request): Path(temporary).unlink(missing_ok=True) -def serve_connection(connection, host, allowed_uid): +def serve_connection(connection, host, allowed_uid, allowed_operations=None): with connection: connection.settimeout(65) _, uid, _ = struct.unpack("3i", connection.getsockopt(socket.SOL_SOCKET, socket.SO_PEERCRED, 12)) @@ -451,6 +451,11 @@ def serve_connection(connection, host, allowed_uid): if len(line) > MAX_REQUEST or not line.endswith(b"\n"): raise Rejected("The host request is too large or incomplete.") request = json.loads(line) + if allowed_operations is not None and ( + not isinstance(request, dict) or not isinstance(request.get("operation"), str) + or request["operation"] not in allowed_operations + ): + raise dashboard_runs.Error(403, "This preview cannot change Snow Globe. Use production for host controls.") if isinstance(request, dict) and request.get("operation") in {"vm.console", "vm.serial", "vm.guest", "vm.upload"}: streams = GUEST_SLOTS if request["operation"] == "vm.guest" else STREAM_SLOTS if not streams.acquire(blocking=False): @@ -484,9 +489,9 @@ def serve_connection(connection, host, allowed_uid): def main(): - host = Host(os.environ["STUDIO_POOL"]) - allowed_uid = pwd.getpwnam(os.environ["STUDIO_DASHBOARD_USER"]).pw_uid - socket_path = Path(os.environ.get("STUDIO_HOST_SOCKET", "/run/studio-host/host.sock")) + host = Host(os.environ["SNOWGLOBE_POOL"]) + allowed_uid = pwd.getpwnam(os.environ["SNOWGLOBE_DASHBOARD_USER"]).pw_uid + socket_path = Path(os.environ.get("SNOWGLOBE_HOST_SOCKET", "/run/snowglobe-host/host.sock")) socket_path.unlink(missing_ok=True) with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as listener: listener.bind(str(socket_path)) diff --git a/tools/dashboard-iam-test.py b/tools/dashboard-iam-test.py deleted file mode 100644 index d482b449fb9676a269d8c073f972df3b07cefc3e..0000000000000000000000000000000000000000 --- a/tools/dashboard-iam-test.py +++ /dev/null @@ -1,156 +0,0 @@ -#!/usr/bin/env python3 -import argparse -import importlib -import json -from pathlib import Path -import subprocess -import sys -import time -import urllib.error -import urllib.parse -import urllib.request -import uuid - - -def main(): - parser = argparse.ArgumentParser() - parser.add_argument("--url", required=True) - parser.add_argument("--socket", required=True) - parser.add_argument("--proof-file", type=Path, required=True) - parser.add_argument("--user", default="studio-dashboard") - parser.add_argument("--output", type=Path) - args = parser.parse_args() - if args.output: - args.output.unlink(missing_ok=True) - sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "service/keycloak")) - from api import Keycloak - keycloak = Keycloak("keycloak.studio.test", importlib.import_module("dashboard-run").secret("get", "keycloak", "password"), attempts=1) - keycloak.configure_profile() - name = "iam-fixture-" + uuid.uuid4().hex - password = uuid.uuid4().hex + "A1!" - identity = None - proof = args.proof_file.read_text().strip() - admin = keycloak.request("/admin/realms/master/users?username=admin&exact=true")[0] - admin = keycloak.request("/admin/realms/master/users/" + admin["id"]) - admin_roles = keycloak.request("/admin/realms/master/users/" + admin["id"] + "/role-mappings/realm") - server_role = next(role for role in admin_roles if role["name"] == "admin") - - def http(path, method="GET", body=None, status=200, actor="fixture-operator", groups="infra-admin"): - request = urllib.request.Request(args.url + "/api/" + path, method=method, - data=json.dumps(body).encode() if body is not None else None, - headers={"User-Name": actor, "User-Groups": groups, "Studio-Proxy-Token": proof, "Content-Type": "application/json"}) - try: - response = urllib.request.urlopen(request, timeout=70) - except urllib.error.HTTPError as error: - response = error - with response: - content = response.read() - assert response.status == status, (path, response.status, content[:200]) - return json.loads(content) if content and response.headers.get("Content-Type", "").startswith("application/json") else None - - def broker(path=None, method="GET", body=None, status=200, **fields): - payload = {"operation": "iam.request", "path": path, "method": method, "body": body} if path is not None else fields - process = subprocess.run([sys.executable, str(Path(__file__).with_name("dashboard-host-vm-test.py")), args.socket, - "--client", "--user", args.user], input=json.dumps(payload).encode() + b"\n", capture_output=True, timeout=75) - assert process.returncode == 0, "broker client failed" - result = json.loads(process.stdout) - if status != 200: - assert result.get("status") == status, (path, result) - return None - assert "value" in result, (path, result) - return result["value"] - - def sign_in(secret): - body = urllib.parse.urlencode({"client_id": "admin-cli", "grant_type": "password", "username": name, "password": secret}).encode() - return keycloak.request("/realms/master/protocol/openid-connect/token", "POST", body) - - try: - started = time.monotonic() - directory = http("users") - directory_seconds = time.monotonic() - started - assert all(user["username"] != "admin" for user in directory["users"]) - roles = {role["name"]: role for role in directory["groups"]} - assert set(roles) == {"infra-admin", "media", "media-manage"} - http("users", status=403, groups="media") - created = http("users", "POST", {"profile": {"username": name, "email": name + "@fixture.invalid", "firstName": "Native", "lastName": "Fixture"}, - "groups": [roles["media"]["id"]], "setup": {"kind": "password", "password": password}}, status=201) - identity = created["id"] - path = "users/" + identity - assert keycloak.request("/admin/realms/master/" + path)["username"] == name - http(path, "PATCH", {"requiredActions": [], "emailVerified": True, "firstName": "Verified"}, status=204) - http(path + "/password", "PUT", {"password": password, "temporary": False}, status=204) - tokens = sign_in(password) - assert tokens["access_token"] - http(path + "/credentials") - http(path + "/groups/" + roles["infra-admin"]["id"], "PUT", status=204) - http(path + "/groups/" + roles["infra-admin"]["id"], "DELETE", status=400, actor=name) - http(path + "/groups/" + roles["infra-admin"]["id"], "DELETE", status=204) - mapped = {role["name"] for role in keycloak.request("/admin/realms/master/" + path + "/role-mappings/realm")} - assert mapped & {"infra-admin", "media", "media-manage", "admin"} == {"media"} - http("account", "PATCH", {"firstName": "Account"}, status=204, actor=name, groups="media") - assert http("account", actor=name, groups="media")["firstName"] == "Account" - broker("/" + path, "PUT", {"attributes": {"picture": ["https://fixture.invalid/picture"]}}) - profile = keycloak.request("/admin/realms/master/" + path) - assert profile["attributes"]["picture"] == ["https://fixture.invalid/picture"] - assert profile["firstName"] == "Account" and profile["email"] == name + "@fixture.invalid" - broker("/" + path, "PUT", {"attributes": {"picture": None}}) - profile = keycloak.request("/admin/realms/master/" + path) - assert "picture" not in profile.get("attributes", {}) - assert profile["firstName"] == "Account" and profile["email"] == name + "@fixture.invalid" - http(path + "/logout", "POST", status=204) - try: - keycloak.request("/realms/master/protocol/openid-connect/token", "POST", urllib.parse.urlencode({ - "client_id": "admin-cli", "grant_type": "refresh_token", "refresh_token": tokens["refresh_token"]}).encode()) - except urllib.error.HTTPError as error: - assert error.code == 400 - else: - raise AssertionError("logged-out refresh token remained usable") - changed = uuid.uuid4().hex + "A1!" - http(path + "/password", "PUT", {"password": changed, "temporary": False}, status=204) - assert sign_in(changed)["access_token"] - http(path, "PATCH", {"enabled": False}, status=204) - try: - sign_in(changed) - except urllib.error.HTTPError as error: - assert error.code == 400 - else: - raise AssertionError("disabled fixture signed in") - http(path, "PATCH", {"enabled": True}, status=204) - for operation in ["get", "set", "rotate"]: - fields = {"operation": "deploy.secret." + operation, "service": "keycloak", "key": "password"} - if operation == "set": - fields["value"] = "fixture-not-applied" - broker(status=403, **fields) - for forbidden in ["/clients", "/realm-settings", "/users/" + identity + "/../../clients", "/users?max=1000&first=0"]: - broker(forbidden, status=400) - broker("/users/" + admin["id"] + "/reset-password", "PUT", {"type": "password", "value": "fixture-not-applied", "temporary": False}, status=403) - broker("/users/" + admin["id"], "DELETE", status=403) - broker("/" + path + "/role-mappings/realm", "POST", [server_role], status=403) - broker("/" + path + "/role-mappings/realm", "POST", [{"id": server_role["id"], "name": "infra-admin"}], status=403) - assert keycloak.request("/admin/realms/master/users/" + admin["id"]) == admin - assert keycloak.request("/admin/realms/master/users/" + admin["id"] + "/role-mappings/realm") == admin_roles - Keycloak("keycloak.studio.test", importlib.import_module("dashboard-run").secret("get", "keycloak", "password"), attempts=1) - http(path, "DELETE", status=204) - identity = None - result = {"native_user_crud": True, "password_sign_in_and_reset": True, "disable_rejects_sign_in": True, - "dashboard_role_grants": True, "self_lockout_refused": True, "account_profile": True, - "logout_revokes_refresh": True, "non_admin_refused": True, "bootstrap_user_hidden_and_protected": True, - "credential_export_and_change_refused": True, "admin_paths_and_forged_roles_refused": True, - "existing_realm_preserved": True, "directory_seconds": round(directory_seconds, 3)} - result["picture_persists_and_preserves_profile"] = True - finally: - if identity is not None: - keycloak.request("/admin/realms/master/users/" + identity, "DELETE") - remaining = keycloak.request("/admin/realms/master/users?username=" + name + "&exact=true") - for user in remaining: - keycloak.request("/admin/realms/master/users/" + user["id"], "DELETE") - assert not keycloak.request("/admin/realms/master/users?username=" + name + "&exact=true") - result["owned_fixture_removed"] = True - if args.output: - args.output.parent.mkdir(parents=True, exist_ok=True) - args.output.write_text(json.dumps(result, indent=2) + "\n") - print(json.dumps(result)) - - -if __name__ == "__main__": - main() diff --git a/tools/dashboard-mcp-test.py b/tools/dashboard-mcp-test.py deleted file mode 100644 index bbb2cefe937c86f12fe403c78ceea6f4345d4036..0000000000000000000000000000000000000000 --- a/tools/dashboard-mcp-test.py +++ /dev/null @@ -1,237 +0,0 @@ -#!/usr/bin/env python3 -import argparse -import base64 -import hashlib -import importlib -import json -from pathlib import Path -import subprocess -import sys -import time -import urllib.error -import urllib.parse -import urllib.request -import uuid - - -class NoRedirect(urllib.request.HTTPRedirectHandler): - def redirect_request(self, request, fp, code, message, headers, newurl): - return None - - -def main(): - parser = argparse.ArgumentParser() - parser.add_argument("--url", required=True) - parser.add_argument("--proof-file", type=Path, required=True) - parser.add_argument("--restart-unit") - parser.add_argument("--output", type=Path) - args = parser.parse_args() - if args.output: - args.output.unlink(missing_ok=True) - origin = "https://globe.studio.test" - resource = origin + "/mcp/observability" - opener = urllib.request.build_opener(NoRedirect) - proof = args.proof_file.read_text().strip() - sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "service/keycloak")) - from api import Keycloak - keycloak = Keycloak("keycloak.studio.test", importlib.import_module("dashboard-run").secret("get", "keycloak", "password"), attempts=1) - marker = "mcp-fixture-" + uuid.uuid4().hex - names = [marker + "-one", marker + "-two"] - identities = {} - - def http(path, method="GET", body=None, status=200, actor=None, groups="infra-admin", token=None, form=False, headers=None): - fields = {"Host": "globe.studio.test", "Content-Type": "application/x-www-form-urlencoded" if form else "application/json"} - if actor is not None: - fields.update({"Studio-Proxy-Token": proof, "User-Name": actor, "User-Groups": groups, "Origin": origin}) - if token is not None: - fields.update({"Authorization": "Bearer " + token, "Accept": "application/json, text/event-stream", "MCP-Protocol-Version": "2025-11-25"}) - fields.update(headers or {}) - encoded = urllib.parse.urlencode(body).encode() if form else json.dumps(body).encode() if body is not None else None - request = urllib.request.Request(args.url + path, method=method, data=encoded, headers=fields) - try: - response = opener.open(request, timeout=70) - except urllib.error.HTTPError as error: - response = error - with response: - content = response.read() - assert response.status == status, (path, response.status, content[:300]) - if content and response.headers.get("Content-Type", "").startswith("application/json"): - value = json.loads(content) - else: - value = content.decode() - return value, response.headers - - def consent(client, actor, chosen, scope="observability:read offline_access"): - verifier = uuid.uuid4().hex + uuid.uuid4().hex - challenge = base64.urlsafe_b64encode(hashlib.sha256(verifier.encode()).digest()).decode().rstrip("=") - request = {"response_type": "code", "client_id": client["client_id"], "redirect_uri": client["redirect_uris"][0], - "code_challenge_method": "S256", "code_challenge": challenge, "resource": resource, "scope": scope, "state": marker} - _, headers = http("/oauth/authorize?" + urllib.parse.urlencode(request), status=302) - pending = urllib.parse.urlsplit(headers["Location"]).path.removeprefix("/connect/") - path = "/api/mcp/consent/" + pending - details, _ = http(path, actor=actor) - assert details["client"] == client["client_name"] - http(path, actor=names[1] if actor == names[0] else names[0], status=403) - http(path, "POST", {"resources": ["outside-grant"]}, actor=actor, status=403) - http(path, "POST", {"resources": chosen}, actor=actor, headers={"Origin": "https://other.invalid"}, status=403) - result, _ = http(path, "POST", {"resources": chosen}, actor=actor) - query = urllib.parse.parse_qs(urllib.parse.urlsplit(result["redirect"]).query) - assert query["state"] == [marker] and query["iss"] == [origin + "/"] - http(path, actor=actor, status=404) - return {"grant_type": "authorization_code", "client_id": client["client_id"], "redirect_uri": client["redirect_uris"][0], - "code_verifier": verifier, "code": query["code"][0], "resource": resource} - - def rpc(token, method, params=None): - result, _ = http("/mcp/observability", "POST", {"jsonrpc": "2.0", "id": 1, "method": method, - **({"params": params} if params is not None else {})}, token=token) - assert "error" not in result, (method, result) - return result["result"] - - def ingest(service, path, body, content_type="application/json"): - readonly = Path("/var/lib/studio/dashboard.token").read_text().strip() - request = urllib.request.Request("http://127.0.0.1:4646/v1/service/" + service, headers={"X-Nomad-Token": readonly}) - with urllib.request.urlopen(request, timeout=10) as response: - upstream = json.load(response)[0] - request = urllib.request.Request(f"http://{upstream['Address']}:{upstream['Port']}" + path, - data=body, headers={"Content-Type": content_type}) - with urllib.request.urlopen(request, timeout=10) as response: - assert response.status == 200 - - try: - role = keycloak.request("/admin/realms/master/roles/infra-admin") - for name in names: - result = keycloak.request("/admin/realms/master/users", "POST", {"username": name, "enabled": True}, full=True) - identities[name] = result["id"] - keycloak.request("/admin/realms/master/users/" + result["id"] + "/role-mappings/realm", "POST", [role]) - metadata, _ = http("/.well-known/oauth-protected-resource/mcp/observability") - assert metadata["resource"] == resource and metadata["authorization_servers"] == [origin + "/"] - metadata, _ = http("/.well-known/oauth-authorization-server") - assert metadata["issuer"] == origin + "/" and metadata["code_challenge_methods_supported"] == ["S256"] - _, headers = http("/mcp/observability", status=401, headers={"User-Name": names[0], "User-Groups": "infra-admin"}) - assert headers["WWW-Authenticate"].startswith("Bearer resource_metadata=") - http("/api/mcp", status=403, headers={"User-Name": names[0], "User-Groups": "infra-admin"}) - overview, _ = http("/api/mcp", actor=names[0]) - assert not overview["connections"] and overview["catalogs"][0]["endpoint"] == resource - readonly = Path("/var/lib/studio/dashboard.token").read_text().strip() - request = urllib.request.Request("http://127.0.0.1:4646/v1/jobs", headers={"X-Nomad-Token": readonly}) - with urllib.request.urlopen(request, timeout=10) as response: - jobs = json.load(response) - aliases = {} - for job in jobs: - request = urllib.request.Request("http://127.0.0.1:4646/v1/job/" + urllib.parse.quote(job["ID"], safe=""), headers={"X-Nomad-Token": readonly}) - with urllib.request.urlopen(request, timeout=10) as response: - aliases[job["ID"]] = json.load(response).get("Meta", {}).get("studio_trace_service") - services = [job for job, alias in aliases.items() if alias and list(aliases.values()).count(alias) == 1] - assert len(services) >= 2, "need two trace-enabled services for scoped native fixture" - allowed, denied = services[:2] - client, _ = http("/oauth/register", "POST", {"client_name": marker, "redirect_uris": ["http://127.0.0.1:29999/callback"], - "token_endpoint_auth_method": "none"}, status=201) - form = consent(client, names[0], [allowed]) - wrong = {**form, "code_verifier": "wrong"} - http("/oauth/token", "POST", wrong, form=True, status=400) - tokens, _ = http("/oauth/token", "POST", form, form=True) - http("/oauth/token", "POST", form, form=True, status=400) - token = tokens["access_token"] - assert rpc(token, "initialize", {"protocolVersion": "2025-11-25", "capabilities": {}, "clientInfo": {"name": marker, "version": "1"}})["capabilities"]["tools"] == {} - listed = rpc(token, "tools/list") - assert listed["ttlMs"] == 0 and listed["cacheScope"] == "private", listed - tools = listed["tools"] - assert {tool["name"] for tool in tools} == {"get_logs", "get_traces", "get_trace"} - assert all(tool["annotations"]["readOnlyHint"] for tool in tools) - refused = rpc(token, "tools/call", {"name": "get_logs", "arguments": {"service": denied}}) - assert refused["isError"] - http("/mcp/observability", "POST", {"jsonrpc": "2.0", "id": 1, "method": "tools/list"}, token=token, headers={"Origin": "https://other.invalid"}, status=403) - at = time.time() - 60 - row = {"_time": str(at), "_msg": marker, "source": "nomad", "job": allowed, "task": "fixture", "stream": "stdout"} - ingest("victoria-logs", "/insert/jsonline", (json.dumps(row) + "\n").encode(), "application/stream+json") - trace_id = uuid.uuid4().hex - spans = [] - for service, span_id, parent in [(denied, "1234567890abcdef", ""), (allowed, "abcdef1234567890", "1234567890abcdef")]: - spans.append({"resource": {"attributes": [{"key": "service.name", "value": {"stringValue": aliases[service]}}]}, - "scopeSpans": [{"spans": [{"traceId": trace_id, "spanId": span_id, "parentSpanId": parent, - "name": marker + ("-foreign-secret" if service == denied else "-allowed"), "kind": 1, - "startTimeUnixNano": str(int(at * 1e9)), "endTimeUnixNano": str(int((at + .01) * 1e9)), - "status": {"code": 2 if service == denied else 1}, - "attributes": [{"key": "studio.service", "value": {"stringValue": service}}]}]}]}) - ingest("victoria-traces", "/insert/opentelemetry/v1/traces", json.dumps({"resourceSpans": spans}).encode()) - deadline = time.monotonic() + 30 - while True: - try: - logs = rpc(token, "tools/call", {"name": "get_logs", "arguments": {"service": allowed, "q": marker}}) - assert not logs.get("isError") and any(row["text"] == marker for row in logs["structuredContent"]["logs"]), logs - trace = rpc(token, "tools/call", {"name": "get_trace", "arguments": {"service": allowed, "trace_id": trace_id}}) - assert not trace.get("isError") and len(trace["structuredContent"]["trace"]["spans"]) == 1, trace - traces = rpc(token, "tools/call", {"name": "get_traces", "arguments": {"service": allowed, "q": marker}}) - assert not traces.get("isError") and any(row["id"] == trace_id and row["spans"] == 1 for row in traces["structuredContent"]["traces"]), traces - assert "foreign-secret" not in json.dumps(trace) + json.dumps(traces) - break - except (AssertionError, urllib.error.HTTPError): - if time.monotonic() > deadline: - raise - time.sleep(1) - overview, _ = http("/api/mcp", actor=names[0]) - grant_id = overview["connections"][0]["id"] - other, _ = http("/api/mcp", actor=names[1]) - assert not other["connections"] - http("/api/mcp/connections/" + grant_id, "DELETE", actor=names[1], status=404) - if args.restart_unit: - subprocess.run(["systemctl", "restart", args.restart_unit], check=True, capture_output=True, timeout=60) - rpc(token, "tools/list") - refresh = {"grant_type": "refresh_token", "client_id": client["client_id"], "refresh_token": tokens["refresh_token"], "resource": resource} - rotated, _ = http("/oauth/token", "POST", refresh, form=True) - assert rotated["refresh_token"] != tokens["refresh_token"] - http("/oauth/token", "POST", refresh, form=True, status=400) - http("/mcp/observability", token=token, status=401) - http("/mcp/observability", token=rotated["access_token"], status=401) - second = consent(client, names[0], [allowed], "observability:read") - tokens, _ = http("/oauth/token", "POST", second, form=True) - assert "refresh_token" not in tokens - overview, _ = http("/api/mcp", actor=names[0]) - http("/api/mcp/connections/" + overview["connections"][0]["id"], "DELETE", actor=names[0], status=204) - http("/mcp/observability", token=tokens["access_token"], status=401) - for change in ["disable", "remove-role", "delete-user"]: - exchange = consent(client, names[0], [allowed]) - tokens, _ = http("/oauth/token", "POST", exchange, form=True) - path = "/admin/realms/master/users/" + identities[names[0]] - if change == "disable": - keycloak.request(path, "PUT", {"enabled": False}) - elif change == "remove-role": - keycloak.request(path + "/role-mappings/realm", "DELETE", [role]) - else: - keycloak.request(path, "DELETE") - refresh = {"grant_type": "refresh_token", "client_id": client["client_id"], "refresh_token": tokens["refresh_token"]} - rejected, _ = http("/oauth/token", "POST", refresh, form=True, status=400) - assert rejected["error"] == "invalid_grant" - http("/mcp/observability", token=tokens["access_token"], status=401) - if change == "disable": - keycloak.request(path, "PUT", {"enabled": True}) - elif change == "remove-role": - keycloak.request(path + "/role-mappings/realm", "POST", [role]) - result = {"resource_metadata": True, "pkce_code_single_use": True, "shared_realm_identity": True, - "consent_owner_and_origin": True, "sdk_initialization_and_tools": True, "explicit_service_grants": True, - "native_log_retrieval": True, "native_trace_retrieval": True, "foreign_spans_and_summary_filtered": True, - "cross_user_connections_refused": True, "refresh_rotation_and_replay_revocation": True, - "dashboard_revocation_immediate": True, "refresh_requires_consent": True, - "refresh_honors_account_disable_role_removal_and_deletion": True, - "restart_preserves_tokens": bool(args.restart_unit)} - finally: - for name in names: - try: - overview, _ = http("/api/mcp", actor=name) - for connection in overview["connections"]: - http("/api/mcp/connections/" + connection["id"], "DELETE", actor=name, status=204) - except (AssertionError, urllib.error.URLError): - pass - for name in names: - for identity in keycloak.request("/admin/realms/master/users?username=" + name + "&exact=true"): - keycloak.request("/admin/realms/master/users/" + identity["id"], "DELETE") - assert not keycloak.request("/admin/realms/master/users?username=" + name + "&exact=true") - result["owned_users_removed"] = True - if args.output: - args.output.parent.mkdir(parents=True, exist_ok=True) - args.output.write_text(json.dumps(result, indent=2) + "\n") - print(json.dumps(result)) - - -if __name__ == "__main__": - main() diff --git a/tools/dashboard-metrics-test.py b/tools/dashboard-metrics-test.py index f8f89a940b29deed8c399e8674631776743bc0a2..50551d83d0042b3eb82c8f723143a2f86694c9dc 100644 --- a/tools/dashboard-metrics-test.py +++ b/tools/dashboard-metrics-test.py @@ -50,15 +50,15 @@ nomad = ThreadingHTTPServer(("127.0.0.1", 0), Nomad) for server in [metrics, nomad]: threading.Thread(target=server.serve_forever, daemon=True).start() -unit = "studio-dashboard-metrics-test" -with tempfile.TemporaryDirectory(prefix="studio-metrics-test-") as temporary: - environment = subprocess.check_output(["systemctl", "show", "-P", "Environment", "studio-dashboard"], text=True) +unit = "snowglobe-dashboard-metrics-test" +with tempfile.TemporaryDirectory(prefix="snowglobe-metrics-test-") as temporary: + environment = subprocess.check_output(["systemctl", "show", "-P", "Environment", "snowglobe-dashboard"], text=True) variables = dict(item.split("=", 1) for item in shlex.split(environment)) - for key in ["STUDIO_INDEX_POOL", "STUDIO_YT_STATE", "STUDIO_FILES_WRITABLE"]: + for key in ["SNOWGLOBE_INDEX_POOL", "SNOWGLOBE_YT_STATE", "SNOWGLOBE_FILES_WRITABLE"]: variables.pop(key, None) - variables.update(PORT="7074", STUDIO_METRICS="follow", STUDIO_DATA_DIR=temporary, + variables.update(PORT="7074", SNOWGLOBE_METRICS="follow", SNOWGLOBE_DATA_DIR=temporary, NOMAD_ADDR=f"http://127.0.0.1:{nomad.server_port}") - proof = {"Studio-Proxy-Token": Path(variables["STUDIO_PROXY_TOKEN_FILE"]).read_text().strip()} if "STUDIO_PROXY_TOKEN_FILE" in variables else {} + proof = {"Snowglobe-Proxy-Token": Path(variables["SNOWGLOBE_PROXY_TOKEN_FILE"]).read_text().strip()} if "SNOWGLOBE_PROXY_TOKEN_FILE" in variables else {} def request(range=3600): started = time.monotonic() req = urllib.request.Request(f"http://127.0.0.1:7074/api/metrics/host.cpu?range={range}", diff --git a/tools/dashboard-oidc-test.py b/tools/dashboard-oidc-test.py index 0017bd7b12bb4867d146fcf6fe05aa4f104763dc..6d72ab1b61717a223dd021af83188f987da10478 100644 --- a/tools/dashboard-oidc-test.py +++ b/tools/dashboard-oidc-test.py @@ -43,9 +43,9 @@ def main(): data = Path(temporary).resolve() (data / 'proof').write_text(proof) (data / 'source.json').write_text(json.dumps(export)) - environment = {**os.environ, 'STUDIO_DOMAIN': 'paperclover.net', 'STUDIO_DATA_DIR': str(data), - 'STUDIO_PUBLIC_ORIGIN': origin, 'STUDIO_AUTH_RP_ID': 'auth.paperclover.net', 'STUDIO_FILE_ORIGIN': 'https://file.paperclover.net', - 'STUDIO_WEB_DIR': str(Path('dashboard/dist').resolve()), 'STUDIO_PROXY_TOKEN_FILE': str(data / 'proof'), 'STUDIO_AUTH_REQUIRED': '1'} + environment = {**os.environ, 'SNOWGLOBE_DOMAIN': 'paperclover.net', 'SNOWGLOBE_DATA_DIR': str(data), + 'SNOWGLOBE_PUBLIC_ORIGIN': origin, 'SNOWGLOBE_AUTH_RP_ID': 'auth.paperclover.net', 'SNOWGLOBE_FILE_ORIGIN': 'https://file.paperclover.net', + 'SNOWGLOBE_WEB_DIR': str(Path('dashboard/dist').resolve()), 'SNOWGLOBE_PROXY_TOKEN_FILE': str(data / 'proof'), 'SNOWGLOBE_AUTH_REQUIRED': '1'} binary = str(args.binary.resolve()) result = subprocess.run([binary, '--import-accounts', str(data / 'source.json')], env=environment, capture_output=True, text=True) assert result.returncode == 0, result.stderr @@ -80,7 +80,7 @@ def main(): server.terminate(); server.wait(timeout=10) cookies = {} def request(path, method='GET', body=None, status=200, extra=None, session=True, form=False): - headers = {'Studio-Proxy-Token': proof, 'Host': 'snowglobe.paperclover.net', 'X-Studio-Client-IP': '127.0.0.1'} + headers = {'Snowglobe-Proxy-Token': proof, 'Host': 'snowglobe.paperclover.net', 'X-Snowglobe-Client-IP': '127.0.0.1'} if body is not None: headers.update({'Origin': origin, 'Content-Type': 'application/x-www-form-urlencoded' if form else 'application/json'}) if session: headers['Cookie'] = '; '.join(f'{k}={v}' for k,v in cookies.items()) headers.update(extra or {}) @@ -97,7 +97,7 @@ def main(): csrf = request('/auth/status')['csrf'] login = {'csrf': csrf, 'username': 'oidc-test', 'password': password, 'next': '/'} request('/auth/password', 'POST', login) - metadata = request('/.well-known/openid-configuration', extra={'Studio-Proxy-Token': 'wrong'}) + metadata = request('/.well-known/openid-configuration', extra={'Snowglobe-Proxy-Token': 'wrong'}) assert metadata['issuer'] == origin and metadata['id_token_signing_alg_values_supported'] == ['RS256'] jwk = request('/auth/oidc/jwks')['keys'][0] decode = lambda s: base64.urlsafe_b64decode(s + '=' * (-len(s) % 4)) diff --git a/tools/dashboard-package-test.py b/tools/dashboard-package-test.py index 4ea96eaccd7fae6dc2905866ebaccced8f36e9fd..4ea686cec6cad94aed61aec3d1563aebec6b55ce 100644 --- a/tools/dashboard-package-test.py +++ b/tools/dashboard-package-test.py @@ -22,13 +22,13 @@ def main(): parser.add_argument("--image", required=True) parser.add_argument("--output", type=Path) args = parser.parse_args() - container = "studio-dashboard-package-test-" + uuid.uuid4().hex[:12] + container = "snowglobe-dashboard-package-test-" + uuid.uuid4().hex[:12] proof = uuid.uuid4().hex + uuid.uuid4().hex def shell(*argv): return subprocess.run(argv, capture_output=True, text=True, check=True, timeout=60).stdout - with tempfile.TemporaryDirectory(prefix="studio-dashboard-package-", dir="/run") as temporary: + with tempfile.TemporaryDirectory(prefix="snowglobe-dashboard-package-", dir="/run") as temporary: data = Path(temporary) / "data" data.mkdir() os.chown(data, 65534, 65534) @@ -45,10 +45,10 @@ def main(): "--security-opt=no-new-privileges", "--memory=512m", "--cpus=2", "--pids-limit=128", "--publish=127.0.0.1::7072", "--volume=" + str(data) + ":/data:rw", "--volume=" + str(token) + ":/run/secrets/dashboard-proxy.token:ro", - "--tmpfs=/tmp:rw,noexec,nosuid,size=64m", "--env=STUDIO_DATA_DIR=/data", - "--env=STUDIO_DOMAIN=studio.test", "--env=STUDIO_MEDIA_READ_ONLY=true", - "--env=STUDIO_YT_STATE=/data/yt", "--env=STUDIO_YT_CONFIG=/data/config", - "--env=STUDIO_YT_MEDIA=/data/media", args.image) + "--tmpfs=/tmp:rw,noexec,nosuid,size=64m", "--env=SNOWGLOBE_DATA_DIR=/data", + "--env=SNOWGLOBE_DOMAIN=snowglobe.test", "--env=SNOWGLOBE_MEDIA_READ_ONLY=true", + "--env=SNOWGLOBE_YT_STATE=/data/yt", "--env=SNOWGLOBE_YT_CONFIG=/data/config", + "--env=SNOWGLOBE_YT_MEDIA=/data/media", args.image) info = json.loads(shell("podman", "inspect", container))[0] port = info["NetworkSettings"]["Ports"]["7072/tcp"][0]["HostPort"] base = "http://127.0.0.1:" + port @@ -58,7 +58,7 @@ def main(): "User-Name": "fixture", "User-Groups": "infra-admin,media,media-manage", } if supplied is not None: - headers["Studio-Proxy-Token"] = supplied + headers["Snowglobe-Proxy-Token"] = supplied request = urllib.request.Request(base + path, headers=headers) with urllib.request.urlopen(request, timeout=30) as response: return response.read() @@ -80,11 +80,11 @@ def main(): except urllib.error.HTTPError as error: assert error.code == 403, (path, error.code) assert json.loads(get("/api/me"))["name"] == "fixture" - python = next(value.split("=", 1)[1] for value in info["Config"]["Env"] if value.startswith("STUDIO_YT_PYTHON=")) + python = next(value.split("=", 1)[1] for value in info["Config"]["Env"] if value.startswith("SNOWGLOBE_YT_PYTHON=")) address = next(network["IPAddress"] for network in info["NetworkSettings"]["Networks"].values() if network["IPAddress"]) neighbor = f"""import urllib.request, urllib.error request = urllib.request.Request('http://{address}:7072/api/me', headers={{ - 'User-Name': 'snow', 'User-Groups': 'infra-admin', 'Studio-Proxy-Token': '0' * 64, + 'User-Name': 'snow', 'User-Groups': 'infra-admin', 'Snowglobe-Proxy-Token': '0' * 64, }}) try: urllib.request.urlopen(request, timeout=5) @@ -96,9 +96,9 @@ print('refused') assert shell("podman", "run", "--rm", "--read-only", "--cap-drop=ALL", "--security-opt=no-new-privileges", "--entrypoint=" + python, args.image, "-c", neighbor).strip() == "refused" unsecured = subprocess.run(["podman", "run", "--rm", "--read-only", "--network=none", "--cap-drop=ALL", - "--security-opt=no-new-privileges", "--unsetenv=STUDIO_PROXY_TOKEN_FILE", args.image], + "--security-opt=no-new-privileges", "--unsetenv=SNOWGLOBE_PROXY_TOKEN_FILE", args.image], capture_output=True, text=True, timeout=10) - assert unsecured.returncode != 0 and "Set STUDIO_PROXY_TOKEN_FILE" in unsecured.stderr, unsecured.stderr + assert unsecured.returncode != 0 and "Set SNOWGLOBE_PROXY_TOKEN_FILE" in unsecured.stderr, unsecured.stderr class Auth(BaseHTTPRequestHandler): def do_GET(self): @@ -117,13 +117,13 @@ print('refused') "/v1/service/forward-auth": [{"ServiceName": "forward-auth", "AllocID": "fixture", "Address": "127.0.0.1", "Port": auth.server_port, "Tags": []}], "/v1/allocation/fixture/checks": {"ready": {"Status": "success"}}, }[path] - os.environ.update(STUDIO_DOMAIN="studio.test", STUDIO_DASHBOARD_PORT=port, STUDIO_PROXY_TOKEN_FILE=str(token)) + os.environ.update(SNOWGLOBE_DOMAIN="snowglobe.test", SNOWGLOBE_DASHBOARD_PORT=port, SNOWGLOBE_PROXY_TOKEN_FILE=str(token)) with socket.socket() as reservation: reservation.bind(("127.0.0.1", 0)) proxy_port = reservation.getsockname()[1] config = Path(temporary) / "Caddyfile" config.write_text("{\n admin off\n auto_https off\n}\n" + router.render("fixture").replace( - "globe.studio.test {", "http://127.0.0.1:" + str(proxy_port) + " {" + "globe.snowglobe.test {", "http://127.0.0.1:" + str(proxy_port) + " {" ).replace(" tls internal\n", "")) config.chmod(0o600) log = Path(temporary) / "caddy.log" @@ -141,7 +141,7 @@ print('refused') if time.monotonic() >= deadline or caddy.poll() is not None: raise AssertionError(log.read_text().replace(proof, "")) time.sleep(.1) - forged = {"User-Name": "snow", "User-Groups": "infra-admin", "Studio-Proxy-Token": "0" * 64} + forged = {"User-Name": "snow", "User-Groups": "infra-admin", "Snowglobe-Proxy-Token": "0" * 64} request = urllib.request.Request("http://127.0.0.1:" + str(proxy_port) + "/api/me", headers={**forged, "Cookie": "fixture=1"}) with urllib.request.urlopen(request, timeout=10) as response: @@ -171,7 +171,7 @@ print('refused') auth.shutdown() apps = json.loads(get("/api/launcher")) assert any(app["id"] == "shale" for app in apps), apps - assert all(app["url"].endswith(".studio.test") for app in apps), apps + assert all(app["url"].endswith(".snowglobe.test") for app in apps), apps icon = next(url for app in apps for url in (app["icon"] or {}).values() if url) assert b"= 1 - a1.send({"id": frame["id"], "result": {"late": True}}) - a1.mode = "reply" - assert command(control["key"], first["id"])["result"]["params"] == {"limit": 30} - a1.frames.get(timeout=2) - a1.mode = "drop" - started = time.monotonic() - outcome = command(control["key"], first["id"], "send_message", write_params, status=409) - assert "unknown" in outcome["error"] and time.monotonic() - started < 5 - a1.frames.get(timeout=2) - a1.close() - a1 = connect(token1) - time.sleep(.2) - assert a1.frames.empty() - assert command(control["key"], first["id"])["result"]["method"] == "list_threads" - a1.frames.get(timeout=2) - metadata, _ = http("/.well-known/oauth-protected-resource/mcp/agents") - assert metadata["resource"] == resource and "sessions:write" in metadata["scopes_supported"] - client, _ = http("/oauth/register", "POST", {"client_name": marker, "redirect_uris": ["http://127.0.0.1:20001/callback"]}, status=201) - verifier = uuid.uuid4().hex + uuid.uuid4().hex - fields = {"response_type": "code", "client_id": client["client_id"], "redirect_uri": client["redirect_uris"][0], "resource": resource, "scope": "sessions:read sessions:write offline_access", - "code_challenge_method": "S256", "code_challenge": base64.urlsafe_b64encode(hashlib.sha256(verifier.encode()).digest()).decode().rstrip("=")} - _, headers = http("/oauth/authorize?" + urllib.parse.urlencode(fields), status=302) - request_id = urllib.parse.urlsplit(headers["Location"]).path.removeprefix("/connect/") - consent_path = "/api/mcp/consent/" + request_id - details, _ = http(consent_path, actor=names[0]) - assert {r["id"] for r in details["resources"]} == {first["id"], second["id"]} - http(consent_path, actor=names[1], status=403) - result, _ = http(consent_path, "POST", {"resources": [first["id"]]}, actor=names[0]) - code = urllib.parse.parse_qs(urllib.parse.urlsplit(result["redirect"]).query)["code"][0] - tokens, _ = http("/oauth/token", "POST", {"grant_type": "authorization_code", "client_id": client["client_id"], "redirect_uri": fields["redirect_uri"], "resource": resource, "code_verifier": verifier, "code": code}, form=True) - credentials.extend([tokens["access_token"], tokens["refresh_token"]]) - assert len(call(tokens["access_token"], "list_machines")["structuredContent"]["machines"]) == 1 - http("/mcp/observability", "POST", {}, token=tokens["access_token"], status=401) - http("/oauth/token", "POST", {"grant_type": "refresh_token", "client_id": client["client_id"], "refresh_token": tokens["refresh_token"], "resource": origin + "/mcp/observability"}, status=400, form=True) - rotated, _ = http("/oauth/token", "POST", {"grant_type": "refresh_token", "client_id": client["client_id"], "refresh_token": tokens["refresh_token"], "resource": resource}, form=True) - credentials.extend([rotated["access_token"], rotated["refresh_token"]]) - with sqlite3.connect(args.data_dir / "connections.sqlite") as db: - rows = db.execute("SELECT key,value FROM records").fetchall() - assert all(not any(secret in key + value for secret in credentials) for key, value in rows) - expired, _ = http("/pairing", "POST", {"name": marker, "platform": "test"}, status=201) - code_hash = hashlib.sha256(expired["code"].replace("-", "").encode()).hexdigest() - db.execute("UPDATE records SET expires=1 WHERE key=?", ("pair:" + code_hash,)) - http("/pairing", token=expired["token"], status=410) - subprocess.run(["systemctl", "restart", args.restart_unit], check=True, timeout=180, capture_output=True) - for agent in agents: - assert agent.closed.wait(5) - a1, a2 = connect(token1), connect(token2) - assert command(read["key"], first["id"])["result"]["method"] == "list_threads" - a1.frames.get(timeout=2) - assert len(call(rotated["access_token"], "list_machines")["structuredContent"]["machines"]) == 1 - http("/api/mcp/connections/" + control["id"], "DELETE", actor=names[0], status=204) - http("/api/v1/machines", token=control["key"], status=401) - assert a1.frames.empty() - http("/api/mcp/relay/machines/" + first["id"], "DELETE", actor=names[0], status=204) - assert a1.closed.wait(5) - Agent(args.url, token1, status=401) - http("/pairing", token=token1, status=410) - remaining = call(read["key"], "list_machines")["structuredContent"]["machines"] - assert len(remaining) == 1 and remaining[0]["id"] == second["id"] and remaining[0]["selected"] - http("/api/v1/targets", "PUT", {"machine_ids": [first["id"]]}, token=read["key"], status=403) - a2.send({"id": str(uuid.uuid4()), "result": {}, "error": "malformed"}) - assert a2.closed.wait(5) - original_agent_checks = None - if args.agent_dir: - node = next(Path("/nix/store").glob("*nodejs-24*/bin/node")) - with tempfile.TemporaryDirectory(prefix="studio-relay-cli-") as temporary: - local = Path(temporary) - home = local / "home" - home.mkdir() - (home / "codex").mkdir() - (home / "claude/projects").mkdir(parents=True) - with sqlite3.connect(home / "codex/state_5.sqlite") as catalog: - catalog.execute("CREATE TABLE threads(id TEXT,title TEXT,cwd TEXT,updated_at INTEGER,rollout_path TEXT,archived INTEGER)") - allowed = local / "workspace" - allowed.mkdir() - resolver = local / "lookup.cjs" - resolver.write_text("const dns=require('node:dns'); const original=dns.lookup; dns.lookup=function(host,opts,callback){if(host==='globe.studio.test'){if(typeof opts==='function'){callback=opts;opts={};} process.nextTick(()=>opts?.all?callback(null,[{address:'127.0.0.1',family:4}]):callback(null,'127.0.0.1',4));}else{return original.apply(this,arguments);}};\n") - executable = local / "model-fixture" - executable.write_text("#!/bin/sh\nexec " + str(node) + " " + str(args.agent_dir / "fake-cli.mjs") + ' "$@"\n') - executable.chmod(0o700) - log = local / "agent.log" - environment = {**os.environ, "HOME": str(home), "CODEX_HOME": str(home / "codex"), "CLAUDE_CONFIG_DIR": str(home / "claude"), - "PATH": str(node.parent) + ":" + os.environ.get("PATH", ""), "NODE_OPTIONS": "--require " + str(resolver), "NODE_EXTRA_CA_CERTS": str(args.agent_ca)} - output = queue.Queue() - with log.open("w") as stderr: - process = subprocess.Popen([str(node), str(args.agent_dir / "dist/agent.js"), "run", "--server", args.agent_origin, - "--name", marker, "--data-dir", str(local / "identity"), "--allow-root", str(allowed), - "--codex-bin", str(executable), "--claude-bin", str(executable)], env=environment, stdout=subprocess.PIPE, stderr=stderr, text=True) - reader = threading.Thread(target=lambda: [output.put(line) for line in process.stdout], daemon=True) - reader.start() - try: - deadline = time.monotonic() + 15 - code = None - while not code and time.monotonic() < deadline: - line = output.get(timeout=15) - match = re.search(r"code ([A-F0-9]{5}-[A-F0-9]{5})", line) - if match: - code = match[1] - assert code, "Original agent did not begin pairing" - machine, _ = http("/api/mcp/relay/pair", "POST", {"code": code}, actor=names[0]) - deadline = time.monotonic() + 15 - while time.monotonic() < deadline: - overview, _ = http("/api/mcp", actor=names[0]) - if any(m["id"] == machine["id"] and m["online"] for m in overview["machines"]): - break - time.sleep(.2) - else: - raise AssertionError("Original agent did not connect") - credential_file = local / "identity/agent.json" - assert credential_file.stat().st_mode & 0o777 == 0o600 - native_key = key(names[0], [machine["id"]], write=True) - assert command(native_key["key"], machine["id"])["result"] == {"threads": [], "errors": []} - for provider in ["codex", "claude"]: - started = command(native_key["key"], machine["id"], "start_thread", {"provider": provider, "cwd": str(allowed), "message": "owned fixture"})["result"] - thread = started["thread_id"] - time.sleep(.05) - read_fields = {"provider": provider, "thread_id": thread} - transcript = command(native_key["key"], machine["id"], "read_thread", read_fields)["result"] - assert "fixture reply" in json.dumps(transcript) - command(native_key["key"], machine["id"], "send_message", {**read_fields, "message": "slow"}) - if provider == "claude": - live = command(native_key["key"], machine["id"], "read_thread", read_fields)["result"] - command(native_key["key"], machine["id"], "interrupt_thread", {**read_fields, "expected_turn_id": live["active_turn_id"]}) - command(native_key["key"], machine["id"], "start_thread", {"provider": "codex", "cwd": str(home), "message": "outside allowed directory"}, status=400) - http("/api/mcp/relay/machines/" + machine["id"], "DELETE", actor=names[0], status=204) - assert process.wait(timeout=10) != 0 - original_agent_checks = {"tls_pairing_and_outbound_wss": True, "credential_mode_0600": True, "codex_and_claude_owned_session_roundtrip": True, - "session_control_and_expected_turn": True, "local_directory_allowlist": True, "unlink_rejects_reconnect": True} - finally: - if process.poll() is None: - process.terminate() - process.wait(timeout=10) - process.stdout.close() - reader.join(timeout=5) - result = {"native_pairing_single_use_and_expiry": True, "existing_agents_wire_protocol": True, "ordinary_realm_user_consent": True, - "machine_and_cross_catalog_isolation": True, "explicit_control_scope": True, "single_machine_writes": True, - "per_connection_target_selection": True, "native_mcp_tools": True, "owned_machine_stream": True, "eight_pending_limit": True, "unicode_command_frame_budget": True, - "heartbeat_and_timeout_unknown_outcome": True, "disconnect_unknown_outcome_no_replay": True, - "hash_only_credentials": True, "restart_preserves_pairing_keys_and_oauth": True, - "revoke_and_unlink_immediate": True, "origin_host_and_malformed_frames_refused": True, - "timeout_seconds": round(elapsed, 2)} - if original_agent_checks is not None: - result["original_node_agent"] = original_agent_checks - finally: - for agent in agents: - agent.close() - keycloak = Keycloak("keycloak.studio.test", importlib.import_module("dashboard-run").secret("get", "keycloak", "password"), attempts=1) - for name in names: - try: - overview, _ = http("/api/mcp", actor=name) - for connection in overview["connections"]: - http("/api/mcp/connections/" + connection["id"], "DELETE", actor=name, status=204) - for machine in overview["machines"]: - http("/api/mcp/relay/machines/" + machine["id"], "DELETE", actor=name, status=204) - finally: - for identity in keycloak.request("/admin/realms/master/users?username=" + name + "&exact=true"): - keycloak.request("/admin/realms/master/users/" + identity["id"], "DELETE") - assert not keycloak.request("/admin/realms/master/users?username=" + name + "&exact=true") - result["owned_users_and_machines_removed"] = True - if args.output: - args.output.write_text(json.dumps(result, indent=2) + "\n") - print(json.dumps(result)) - - -if __name__ == "__main__": - main() diff --git a/tools/dashboard-routing-test.py b/tools/dashboard-routing-test.py index 5c35b61f0bbfa0270cf203b18f8c22318336e796..a50b6ebe4ff412745c194eaa3b840a98a3da112e 100644 --- a/tools/dashboard-routing-test.py +++ b/tools/dashboard-routing-test.py @@ -22,8 +22,8 @@ def main(): parser.add_argument("--output", type=Path) args = parser.parse_args() proof = uuid.uuid4().hex + uuid.uuid4().hex - container = "studio-dashboard-routing-test-" + uuid.uuid4().hex[:12] - fixture_id = "studio-routing-fixture-" + uuid.uuid4().hex[:12] + container = "snowglobe-dashboard-routing-test-" + uuid.uuid4().hex[:12] + fixture_id = "snowglobe-routing-fixture-" + uuid.uuid4().hex[:12] trace_id = uuid.uuid4().hex redirected = [] @@ -46,19 +46,19 @@ def main(): self.send_header("Content-Type", "application/json") self.end_headers() self.wfile.write(json.dumps({"path": self.path, "host": self.headers.get("Host"), - "proof": self.headers.get("Studio-Proxy-Token"), + "proof": self.headers.get("Snowglobe-Proxy-Token"), "user": self.headers.get("User-Name")}).encode()) def log_message(self, *args): pass - with tempfile.TemporaryDirectory(prefix="studio-dashboard-routing-", dir="/run") as temporary, HTTPServer(("127.0.0.1", 0), Fixture) as fixture: + with tempfile.TemporaryDirectory(prefix="snowglobe-dashboard-routing-", dir="/run") as temporary, HTTPServer(("127.0.0.1", 0), Fixture) as fixture: root = Path(temporary) threading.Thread(target=fixture.serve_forever, daemon=True).start() token = root / "proxy.token" token.write_text(proof) readonly = root / "nomad.token" - readonly.write_bytes(Path("/var/lib/studio/dashboard.token").read_bytes()) + readonly.write_bytes(Path("/var/lib/snowglobe/dashboard.token").read_bytes()) for file in [token, readonly]: os.chown(file, 0, 65534) file.chmod(0o440) @@ -71,14 +71,14 @@ def main(): continue else: raise AssertionError("no fixture listener available") - os.environ.update(STUDIO_DOMAIN="studio.test", STUDIO_INTERNAL_PORT=str(port), - STUDIO_DASHBOARD_PORT="7072", STUDIO_PROXY_TOKEN_FILE=str(token)) + os.environ.update(SNOWGLOBE_DOMAIN="snowglobe.test", SNOWGLOBE_INTERNAL_PORT=str(port), + SNOWGLOBE_DASHBOARD_PORT="7072", SNOWGLOBE_PROXY_TOKEN_FILE=str(token)) real_nomad = router.nomad def discovery(path, supplied): if path in ["/v1/service/routing-fixture", "/v1/service/qbittorrent"]: return [{"ServiceName": path.rsplit("/", 1)[1], "AllocID": "routing-fixture", "Address": "127.0.0.1", - "Port": fixture.server_port, "Tags": ["caddy-host=routing-fixture.studio.test"]}] + "Port": fixture.server_port, "Tags": ["caddy-host=routing-fixture.snowglobe.test"]}] if path == "/v1/allocation/routing-fixture/checks": return {"ready": {"Status": "failure"}} value = real_nomad(path, supplied) @@ -87,7 +87,7 @@ def main(): return value router.nomad = discovery - host = "dashboard.internal.studio.test" + host = "dashboard.internal.snowglobe.test" content = router.render(Path(router.TOKEN).read_text().strip()) gateway = content[content.index(host + ":" + str(port) + " {"):] config = root / "Caddyfile" @@ -107,7 +107,7 @@ def main(): time.sleep(.1) ca.chmod(0o444) image = json.loads(shell("podman", "image", "inspect", args.image))[0] - python = next(value.split("=", 1)[1] for value in image["Config"]["Env"] if value.startswith("STUDIO_YT_PYTHON=")) + python = next(value.split("=", 1)[1] for value in image["Config"]["Env"] if value.startswith("SNOWGLOBE_YT_PYTHON=")) client = f"""import json, ssl, time, urllib.request, urllib.error base = 'https://{host}:{port}' proof = open('/proxy.token').read().strip() @@ -116,7 +116,7 @@ context = ssl.create_default_context(cafile='/ca.crt') def request(path, supplied=proof, method='GET', body=None, content_type='application/json'): headers = {{'User-Name': 'forged', 'X-Nomad-Token': nomad, 'Content-Type': content_type}} if supplied is not None: - headers['Studio-Proxy-Token'] = supplied + headers['Snowglobe-Proxy-Token'] = supplied query = urllib.request.Request(base + path, data=body, headers=headers, method=method) try: with urllib.request.urlopen(query, context=context, timeout=15) as response: @@ -127,8 +127,8 @@ for supplied in [None, '0' * 64, proof[:-1], proof + '0']: assert request('/nomad/v1/services', supplied)[0] == 403 status, body = request('/nomad/v1/services') assert status == 200 and isinstance(json.loads(body), list), (status, body) -assert request('/nomad/v1/var/studio-routing-fixture')[0] == 403 -status, body = request('/nomad/v1/jobs', method='POST', body=json.dumps({{'Job': {{'ID': 'studio-routing-fixture', 'Name': 'studio-routing-fixture', 'Type': 'service', 'Datacenters': ['clover'], 'TaskGroups': [{{'Name': 'fixture', 'Count': 0, 'Tasks': [{{'Name': 'probe', 'Driver': 'podman', 'Config': {{'image': 'alpine:3.22'}}, 'Resources': {{'CPU': 100, 'MemoryMB': 32}}}}]}}]}}}}).encode()) +assert request('/nomad/v1/var/snowglobe-routing-fixture')[0] == 403 +status, body = request('/nomad/v1/jobs', method='POST', body=json.dumps({{'Job': {{'ID': 'snowglobe-routing-fixture', 'Name': 'snowglobe-routing-fixture', 'Type': 'service', 'Datacenters': ['clover'], 'TaskGroups': [{{'Name': 'fixture', 'Count': 0, 'Tasks': [{{'Name': 'probe', 'Driver': 'podman', 'Config': {{'image': 'alpine:3.22'}}, 'Resources': {{'CPU': 100, 'MemoryMB': 32}}}}]}}]}}}}).encode()) assert status == 403, (status, body) assert request('/services/unknown-fixture/health')[0] == 404 for service in ['victoria-metrics', 'victoria-logs', 'victoria-traces']: @@ -139,7 +139,7 @@ result = json.loads(body) assert status == 200 and result == {{'path': '/probe?fixture=1', 'host': '127.0.0.1:{fixture.server_port}', 'proof': None, 'user': None}}, result # Keep fixture samples outside VictoriaMetrics' query latency window. at = time.time() - 60 -metric = 'studio_service_cpu_cores{{service="{fixture_id}"}} 0.125 ' + str(int(at * 1000)) + '\\n' +metric = 'snowglobe_service_cpu_cores{{service="{fixture_id}"}} 0.125 ' + str(int(at * 1000)) + '\\n' status, body = request('/services/victoria-metrics/api/v1/import/prometheus', method='POST', body=metric.encode(), content_type='text/plain') assert status == 204, (status, body) row = {{'_time': str(at), '_msg': 'Routing fixture', 'source': 'nomad', 'job': '{fixture_id}', 'task': 'probe', 'stream': 'stdout'}} @@ -172,15 +172,15 @@ print(json.dumps({{'private_bridge': True, 'nonroot': __import__('os').getuid() "--add-host=" + host + ":host-gateway", "--publish=127.0.0.1::7072", "--volume=" + str(ca) + ":/ca.crt:ro", "--volume=" + str(token) + ":/proxy.token:ro", "--volume=" + str(readonly) + ":/nomad.token:ro", "--volume=" + str(data) + ":/data:rw", - "--env=STUDIO_PROXY_TOKEN_FILE=/proxy.token", "--env=STUDIO_NOMAD_TOKEN_FILE=/nomad.token", - "--env=STUDIO_CA_BUNDLE=/ca.crt", f"--env=STUDIO_INTERNAL_URL=https://{host}:{port}", - "--env=STUDIO_DATA_DIR=/data", "--env=STUDIO_DOMAIN=studio.test", args.image) + "--env=SNOWGLOBE_PROXY_TOKEN_FILE=/proxy.token", "--env=SNOWGLOBE_NOMAD_TOKEN_FILE=/nomad.token", + "--env=SNOWGLOBE_CA_BUNDLE=/ca.crt", f"--env=SNOWGLOBE_INTERNAL_URL=https://{host}:{port}", + "--env=SNOWGLOBE_DATA_DIR=/data", "--env=SNOWGLOBE_DOMAIN=snowglobe.test", args.image) info = json.loads(shell("podman", "inspect", container))[0] app_port = info["NetworkSettings"]["Ports"]["7072/tcp"][0]["HostPort"] def dashboard(path): request = urllib.request.Request("http://127.0.0.1:" + app_port + path, headers={ - "Studio-Proxy-Token": proof, "User-Name": "fixture", "User-Groups": "infra-admin", + "Snowglobe-Proxy-Token": proof, "User-Name": "fixture", "User-Groups": "infra-admin", }) with urllib.request.urlopen(request, timeout=20) as response: return json.load(response) diff --git a/tools/dashboard-run.py b/tools/dashboard-run.py index 6739267fac253b5e5ebaf557afa1ca74a31dc598..36ab84c9ac42f1a3645fbc318b9681626e0b16a0 100644 --- a/tools/dashboard-run.py +++ b/tools/dashboard-run.py @@ -28,7 +28,7 @@ FIELDS = { ACTIONS = {"deploy", "destroy", "rollback", "start", "stop", "restart", "secret-set", "secret-rotate"} MAX_LOG = 1024 * 1024 MAX_METADATA = 65536 -HOST_STATE = Path(os.environ.get("STUDIO_HOST_STATE_ROOT", "/var/lib/studio/host")) +HOST_STATE = Path(os.environ.get("SNOWGLOBE_HOST_STATE_ROOT", "/var/lib/snowglobe/host")) class Error(Exception): @@ -98,7 +98,7 @@ def nomad(path, method="GET", body=None, timeout=10): def secret(action, target, key, value=None): service(target, key) job = nomad("job/" + target) - specs = json.loads((job or {}).get("Meta", {}).get("studio_secrets", "[]")) + specs = json.loads((job or {}).get("Meta", {}).get("snowglobe_secrets", "[]")) if not isinstance(specs, list): raise Error(502, "The service's secret names couldn't be read. Deploy its current release again.") matches = [spec for spec in specs if isinstance(spec, dict) and spec.get("name") == key] @@ -160,7 +160,7 @@ def command(action, target, key=None): version = release.current_release() if version is None: raise Error(409, "No release is running. Deploy a release before starting this service.") - return [sys.executable, str(release.check_release(version) / "tools/studio.py"), "deploy", target] + return [sys.executable, str(release.transition.layout(release.check_release(version))["script"]), "deploy", target] if action == "rollback": saved = entry(target) version = saved["release"] @@ -177,7 +177,7 @@ def command(action, target, key=None): if not isinstance(version, str): raise Error(409, "This stage has no release. Stage it again.") root = release.check_release(version) - return [sys.executable, str(root / "tools/studio.py"), "destroy", target] + return [sys.executable, str(release.transition.layout(root)["script"]), "destroy", target] def last(): @@ -201,7 +201,7 @@ def run(identity): raise Error(404, "That run is no longer available.") code = read(root / (identity + ".exit"), 64) if code is None: - shown = subprocess.run(["systemctl", "show", "--property=ActiveState,ExecMainStatus,LoadState", "studio-run-" + identity], + shown = subprocess.run(["systemctl", "show", "--property=ActiveState,ExecMainStatus,LoadState", "snowglobe-run-" + identity], capture_output=True, text=True, timeout=5) state = dict(line.split("=", 1) for line in shown.stdout.splitlines() if "=" in line) if shown.returncode and state.get("LoadState") != "not-found": @@ -237,10 +237,10 @@ def start(action, target, key=None, value=None): pending = HOST_STATE / "last-run.pending" pending.write_text(json.dumps(metadata) + "\n") pending.replace(HOST_STATE / "last-run.json") - args = ["systemd-run", "--unit=studio-run-" + identity, "--collect", "--quiet", + args = ["systemd-run", "--unit=snowglobe-run-" + identity, "--collect", "--quiet", "--property=RuntimeMaxSec=3600", "--property=TimeoutStopSec=10", "--property=MemoryMax=4G", "--property=TasksMax=1024", "--property=CPUWeight=10"] - args.extend("--setenv=" + key + "=" + value for key, value in os.environ.items() if key == "PATH" or key.startswith("STUDIO_")) + args.extend("--setenv=" + key + "=" + value for key, value in os.environ.items() if key == "PATH" or key.startswith("SNOWGLOBE_")) subprocess.run([*args, "--", sys.executable, str(Path(__file__)), identity, action, target, *([key] if key is not None else [])], check=True, capture_output=True, timeout=10) except BaseException: @@ -256,7 +256,7 @@ def handle(request): raise Error(400, "Choose a secret from this service's list.") service(request["service"], request["key"]) process = subprocess.run(["systemd-run", "--pipe", "--wait", "--collect", "--quiet", - "--unit=studio-secret-read-" + str(uuid.uuid4()), "--property=RuntimeMaxSec=25", + "--unit=snowglobe-secret-read-" + str(uuid.uuid4()), "--property=RuntimeMaxSec=25", "--property=MemoryMax=128M", "--property=TasksMax=8", "--property=ProtectSystem=strict", "--property=ProtectHome=yes", "--property=NoNewPrivileges=yes", "--property=CapabilityBoundingSet=", "--property=RestrictAddressFamilies=AF_INET AF_UNIX", "--property=IPAddressDeny=any", @@ -372,7 +372,7 @@ def worker(identity, action, target, key=None): try: argv = command(action, target, key) output.write(("$ " + " ".join(argv) + "\n").encode()) - environment = None + environment = release.transition.environment(Path(argv[1]).parent.parent) if len(argv) > 1 and argv[1].endswith(".py") else None if action in {"stop", "restart"}: environment = {**os.environ, "NOMAD_TOKEN": read(release.STATE / "nomad.token", missing="").strip()} if not environment["NOMAD_TOKEN"]: diff --git a/tools/dashboard-shale-attribution-test.py b/tools/dashboard-shale-attribution-test.py index e957069f9d547ee63f0cdd04740aba380716844c..50d38647b8988b7804aca5b0a01602ebad9f6942 100644 --- a/tools/dashboard-shale-attribution-test.py +++ b/tools/dashboard-shale-attribution-test.py @@ -127,14 +127,14 @@ def main(): host_thread = threading.Thread(target=host_requests, daemon=True) host_thread.start() - origin = 'https://globe.studio.test' - environment = {**os.environ, 'PORT': str(port), 'STUDIO_LISTEN_ADDRESS': '127.0.0.1', - 'STUDIO_DATA_DIR': str(root / 'data'), 'STUDIO_PUBLIC_ORIGIN': origin, - 'STUDIO_AUTH_RP_ID': 'studio.test', 'STUDIO_FILE_ORIGIN': 'https://file.studio.test', - 'STUDIO_SHALE_URL': 'https://localhost:' + str(backend.server_port), - 'STUDIO_CA_BUNDLE': str(cert), 'STUDIO_REPO': str(Path(__file__).resolve().parent.parent), - 'STUDIO_AUTH_REQUIRED': '0', 'STUDIO_HOST_SOCKET': str(root / 'host.sock')} - for name in ('STUDIO_PROXY_TOKEN_FILE', 'STUDIO_INTERNAL_URL', 'STUDIO_YT_STATE', 'STUDIO_INDEX_POOL'): + origin = 'https://globe.snowglobe.test' + environment = {**os.environ, 'PORT': str(port), 'SNOWGLOBE_LISTEN_ADDRESS': '127.0.0.1', + 'SNOWGLOBE_DATA_DIR': str(root / 'data'), 'SNOWGLOBE_PUBLIC_ORIGIN': origin, + 'SNOWGLOBE_AUTH_RP_ID': 'snowglobe.test', 'SNOWGLOBE_FILE_ORIGIN': 'https://file.snowglobe.test', + 'SNOWGLOBE_SHALE_URL': 'https://localhost:' + str(backend.server_port), + 'SNOWGLOBE_CA_BUNDLE': str(cert), 'SNOWGLOBE_REPO': str(Path(__file__).resolve().parent.parent), + 'SNOWGLOBE_AUTH_REQUIRED': '0', 'SNOWGLOBE_HOST_SOCKET': str(root / 'host.sock')} + for name in ('SNOWGLOBE_PROXY_TOKEN_FILE', 'SNOWGLOBE_INTERNAL_URL', 'SNOWGLOBE_YT_STATE', 'SNOWGLOBE_INDEX_POOL'): environment.pop(name, None) with (root / 'server.log').open('wb') as log: server = subprocess.Popen([str(args.binary.resolve())], env=environment, stdout=log, stderr=log) @@ -162,7 +162,7 @@ def main(): def call(name, fields, error=False): connection = http.client.HTTPConnection('127.0.0.1', port, timeout=15) connection.request('POST', '/mcp/shale', json.dumps({'jsonrpc': '2.0', 'id': 1, 'method': 'tools/call', 'params': {'name': name, 'arguments': fields}}), - {'Host': 'globe.studio.test', 'Authorization': 'Bearer fixture', 'Accept': 'application/json, text/event-stream', 'Content-Type': 'application/json'}) + {'Host': 'globe.snowglobe.test', 'Authorization': 'Bearer fixture', 'Accept': 'application/json, text/event-stream', 'Content-Type': 'application/json'}) response = connection.getresponse() body = response.read() connection.close() diff --git a/tools/dashboard-shale-link-test.py b/tools/dashboard-shale-link-test.py deleted file mode 100644 index 719ccac8afc8ab6b466acb9768542af3d06343cb..0000000000000000000000000000000000000000 --- a/tools/dashboard-shale-link-test.py +++ /dev/null @@ -1,463 +0,0 @@ -#!/usr/bin/env python3 -import argparse -import base64 -from concurrent.futures import ThreadPoolExecutor -import copy -import http.cookiejar -import hashlib -import importlib -import json -import math -from pathlib import Path -import sqlite3 -import ssl -import subprocess -import sys -import time -import urllib.error -import urllib.parse -import urllib.request -import uuid - - -class NoRedirect(urllib.request.HTTPRedirectHandler): - def redirect_request(self, request, fp, code, message, headers, newurl): - return None - - -def main(): - parser = argparse.ArgumentParser() - parser.add_argument('--url', required=True) - parser.add_argument('--proof-file', type=Path, required=True) - parser.add_argument('--data-dir', type=Path, required=True) - parser.add_argument('--restart-unit', required=True) - parser.add_argument('--shale-origin', required=True) - parser.add_argument('--shale-database', type=Path, required=True) - parser.add_argument('--shale-container', required=True) - parser.add_argument('--output', type=Path) - args = parser.parse_args() - if args.output: - args.output.unlink(missing_ok=True) - repo = Path(__file__).resolve().parent.parent - sys.path.insert(0, str(repo / 'service/keycloak')) - from api import Keycloak, LoopbackHTTPS - Page = importlib.import_module('dashboard-shale-test').Page - keycloak = Keycloak('keycloak.studio.test', importlib.import_module('dashboard-run').secret('get', 'keycloak', 'password'), attempts=1) - context = ssl.create_default_context(cafile='/var/lib/studio/ca-bundle.crt') - origin = 'https://globe.studio.test' - proof = args.proof_file.read_text().strip() - marker = 'shale-link-' + uuid.uuid4().hex - accounts = [(marker + '-one', uuid.uuid4().hex + 'A1!'), (marker + '-two', uuid.uuid4().hex + 'A1!'), - ('ai', uuid.uuid4().hex + 'A1!')] - ids = [] - sessions = {name: uuid.uuid4().hex + uuid.uuid4().hex for name, _ in accounts} - - class TLS(urllib.request.HTTPSHandler): - def https_open(self, request): - parsed = urllib.parse.urlsplit(request.full_url) - assert parsed.netloc in ('keycloak.studio.test', urllib.parse.urlsplit(args.shale_origin).netloc) - return self.do_open(LoopbackHTTPS, request, context=context) - - def browser(): - cookies = http.cookiejar.CookieJar() - return urllib.request.build_opener(TLS(), NoRedirect(), urllib.request.HTTPCookieProcessor(cookies)), cookies - - def request(client, url, method='GET', body=None, headers=None, status=None): - try: - response = client.open(urllib.request.Request(url, method=method, data=body, headers=headers or {}), timeout=45) - except urllib.error.HTTPError as error: - response = error - with response: - value = response.read(4 * 1024 * 1024 + 1) - assert len(value) <= 4 * 1024 * 1024 - if status is not None: - assert response.status == status, (urllib.parse.urlsplit(url).path, response.status, value[:300]) - return response.status, response.headers, value.decode() - - def api(actor, method='GET', path='/api/mcp', status=200, body=None): - _, _, body = request(urllib.request.build_opener(NoRedirect()), args.url + path, method, - body=json.dumps(body).encode() if body is not None else None, - headers={'Host': 'globe.studio.test', 'Studio-Proxy-Token': proof, 'User-Name': actor, - 'Cookie': '__Host-snow-session=' + sessions[actor], - 'User-Groups': '', 'Origin': origin, 'Content-Type': 'application/json'}, status=status) - return json.loads(body) if body and status < 400 else body or None - - def records(prefix): - path = args.data_dir / 'connections.sqlite' - with sqlite3.connect(path.as_uri() + '?mode=ro', uri=True) as db: - return {key: json.loads(value) for key, value in db.execute('SELECT key,value FROM records WHERE substr(key,1,?)=?', (len(prefix), prefix))} - - def session(index): - return records('shale-session:').get('shale-session:' + ids[index]) - - def session_count(index): - with sqlite3.connect(args.shale_database.as_uri() + '?mode=ro', uri=True) as db: - return db.execute('SELECT count(*) FROM sessions s JOIN users u ON s.user=u.id WHERE u.snowflake=?', (ids[index],)).fetchone()[0] - - def start(index, pending=None): - target = api(accounts[index][0], 'POST', '/api/mcp/shale', body={'request': pending} if pending else {})['redirect'] - assert target.startswith(args.shale_origin + '/-/studio-mcp/') - client, cookies = browser() - _, headers, _ = request(client, target, status=302) - link_cookies = [cookie for cookie in cookies if cookie.name == 'studio_mcp_shale_link'] - assert len(link_cookies) == 1 - cookie = link_cookies[0] - assert not cookie.domain_specified and cookie.secure and cookie.path == '/-/callback' - assert cookie.has_nonstandard_attr('HttpOnly') and cookie.get_nonstandard_attr('SameSite') == 'Lax' - assert headers.get('Referrer-Policy') == 'no-referrer' - assert headers.get('Cache-Control') == 'no-store' - request(browser()[0], target, status=410) - return client, cookies, headers['Location'], target - - def authorize(client, authorization, index): - status, headers, body = request(client, authorization, status=200) - form = next(form for form in Page(body).forms if any(field.get('name') == 'password' for field in form['fields'])) - target = urllib.parse.urljoin(authorization, form['action']) - assert urllib.parse.urlsplit(target).hostname == 'keycloak.studio.test' - fields = {field['name']: field.get('value', '') for field in form['fields'] if field.get('name')} - fields.update(username=accounts[index][0], password=accounts[index][1]) - status, headers, body = request(client, target, 'POST', urllib.parse.urlencode(fields).encode(), - {'Content-Type': 'application/x-www-form-urlencoded', 'Origin': 'https://keycloak.studio.test'}) - for _ in range(6): - assert status in (302, 303), status - target = urllib.parse.urljoin(target, headers['Location']) - parts = urllib.parse.urlsplit(target) - if parts.netloc == urllib.parse.urlsplit(args.shale_origin).netloc: - assert parts.path == '/-/callback' and 'code' in dict(urllib.parse.parse_qsl(parts.query)) - return target - assert parts.netloc == 'keycloak.studio.test' - status, headers, body = request(client, target) - raise AssertionError('too many sign-in redirects') - - def finish(client, target, status=303, pending=None): - _, headers, _ = request(client, target, status=status) - assert headers.get('Cache-Control') == 'no-store' - assert all(not cookie.startswith('SessionID=') for cookie in headers.get_all('Set-Cookie', [])) - assert any('studio_mcp_shale_link=;' in cookie and 'Max-Age=0' in cookie for cookie in headers.get_all('Set-Cookie', [])) - if status == 303: - assert headers['Location'] == origin + ('/connect/' + pending if pending else '/mcp/settings/shale') - - def backend_session(value, status): - return request(browser()[0], args.shale_origin + '/-/settings', headers={'Cookie': 'SessionID=' + value['session']}, status=status) - - def link(index, pending=None): - client, cookies, authorization, _ = start(index, pending) - target = authorize(client, authorization, index) - captured = http.cookiejar.CookieJar() - for cookie in cookies: - captured.set_cookie(copy.copy(cookie)) - finish(client, target, pending=pending) - replay = urllib.request.build_opener(TLS(), NoRedirect(), urllib.request.HTTPCookieProcessor(captured)) - finish(replay, target, 410) - result = session(index) - assert result and result['origin'] == args.shale_origin + '/' - assert result['linkedAt'] > 0 and len(result['session']) > 20 - overview = api(accounts[index][0])['shale'] - assert overview is not None and set(overview) == {'linkedAt'}, overview - assert math.isclose(overview['linkedAt'], result['linkedAt'], rel_tol=0, abs_tol=1e-6), overview - assert session_count(index) == 1 - backend_session(result, 200) - return result - - def public(path, method='GET', body=None, status=200, token=None, form=False): - headers = {'Host': 'globe.studio.test', 'Content-Type': 'application/x-www-form-urlencoded' if form else 'application/json'} - if token: - headers.update({'Authorization': 'Bearer ' + token, 'Accept': 'application/json, text/event-stream', - 'MCP-Protocol-Version': '2025-11-25'}) - encoded = urllib.parse.urlencode(body).encode() if form else json.dumps(body).encode() if body is not None else None - _, headers, body = request(urllib.request.build_opener(NoRedirect()), args.url + path, method, encoded, headers, status) - return (json.loads(body) if body and headers.get('Content-Type', '').startswith('application/json') else body or None), headers - - def pending_request(client, index, scope): - verifier = uuid.uuid4().hex + uuid.uuid4().hex - challenge = base64.urlsafe_b64encode(hashlib.sha256(verifier.encode()).digest()).decode().rstrip('=') - _, headers = public('/oauth/authorize?' + urllib.parse.urlencode({'response_type': 'code', - 'client_id': client['client_id'], 'redirect_uri': client['redirect_uris'][0], 'code_challenge_method': 'S256', - 'code_challenge': challenge, 'resource': origin + '/mcp/shale', 'scope': scope, 'state': marker}), status=302) - pending = urllib.parse.urlsplit(headers['Location']).path.removeprefix('/connect/') - details = api(accounts[index][0], path='/api/mcp/consent/' + pending) - assert details['client'] == marker - api(accounts[1 if index == 0 else 0][0], path='/api/mcp/consent/' + pending, status=403) - return pending, verifier, details - - def consent(client, index, repository, scope): - pending, verifier, details = pending_request(client, index, scope) - available = {r['id'] for r in details['resources']} - assert details['linked'] and (repository == 'all' or repository in available), details - if index == 0: - assert {'alpha', 'beta'} <= available, details - path = '/api/mcp/consent/' + pending - api(accounts[index][0], 'POST', path, 403, {'resources': ['outside-grant']}) - api(accounts[index][0], 'POST', path, 403 if len(available) >= 2 else 400, {'resources': [repository, repository]}) - result = api(accounts[index][0], 'POST', path, body={'resources': 'all' if repository == 'all' else [repository]}) - query = urllib.parse.parse_qs(urllib.parse.urlsplit(result['redirect']).query) - assert query['state'] == [marker] - tokens, _ = public('/oauth/token', 'POST', {'grant_type': 'authorization_code', 'client_id': client['client_id'], - 'redirect_uri': client['redirect_uris'][0], 'code_verifier': verifier, 'code': query['code'][0], - 'resource': origin + '/mcp/shale'}, form=True) - return tokens - - def rpc(token, method, params=None, error=False): - value, _ = public('/mcp/shale', 'POST', {'jsonrpc': '2.0', 'id': 1, 'method': method, - **({'params': params} if params is not None else {})}, token=token) - assert 'error' not in value, value - value = value['result'] - assert bool(value.get('isError')) == error, value - return value if error or 'structuredContent' not in value else value['structuredContent'] - - def call(token, tool, fields=None, error=False): - return rpc(token, 'tools/call', {'name': tool, 'arguments': fields or {}}, error=error) - - def backend(index, suffix, fields=None): - headers = {'Cookie': 'SessionID=' + session(index)['session']} - if fields is not None: - headers.update({'Origin': args.shale_origin, 'Referer': args.shale_origin + suffix, - 'Content-Type': 'application/x-www-form-urlencoded'}) - return request(browser()[0], args.shale_origin + suffix, 'POST' if fields is not None else 'GET', - urllib.parse.urlencode(fields).encode() if fields is not None else None, headers) - - def submit_backend(index, path, changes): - status, _, body = backend(index, path) - assert status == 200, (path, status, body[:700]) - forms = [form for form in Page(body).forms if changes.keys() <= {field.get('name') for field in form['fields']} - and ('t' not in changes or any(field.get('name') == 't' and field.get('value') == changes['t'] for field in form['fields']))] - assert len(forms) == 1, path - form = forms[0] - assert urllib.parse.urljoin(args.shale_origin + path, form.get('action', '')) == args.shale_origin + path - fields = {field['name']: field.get('value', '') for field in form['fields'] - if field.get('name') and field.get('type') == 'hidden'} - fields.update(timezone='UTC', tzoffset='+00:00', **changes) - return backend(index, path, fields) - - def repository(index, name): - status, headers, _ = submit_backend(1, '/-/new', {'name': name, 'description': 'Owned Shale MCP fixture', 'access': 'private'}) - assert status == 303, status - target = urllib.parse.urlsplit(urllib.parse.urljoin(args.shale_origin, headers['Location'])) - assert target.netloc == urllib.parse.urlsplit(args.shale_origin).netloc - assert target.path.rstrip('/') == '/' + name, target.path - if index == 0: - with sqlite3.connect(args.shale_database) as db: - identity = db.execute('SELECT id FROM users WHERE snowflake=?', (ids[index],)).fetchone()[0] - assert db.execute('UPDATE repositories SET owner=? WHERE name=?', (identity, name)).rowcount == 1 - - for name, _ in accounts: - assert not keycloak.request('/admin/realms/master/users?username=' + name + '&exact=true'), 'Fixture account already exists: ' + name - try: - for name, credential in accounts: - keycloak.request('/admin/realms/master/users', 'POST', {'username': name, 'firstName': name, - 'lastName': 'Fixture', 'email': name + '@fixture.invalid', 'emailVerified': True, 'enabled': True, - 'credentials': [{'type': 'password', 'value': credential, 'temporary': False}]}) - found = keycloak.request('/admin/realms/master/users?username=' + name + '&exact=true') - assert len(found) == 1 - ids.append(found[0]['id']) - with sqlite3.connect(args.data_dir / 'accounts.sqlite') as db: - profile = {key: found[0].get(key) for key in ['username', 'firstName', 'lastName', 'email', 'emailVerified', 'enabled']} - profile['requiredActions'] = [] - db.execute('INSERT INTO users(id,profile) VALUES (?,?)', (ids[-1], json.dumps(profile))) - stamp = int(time.time()) - db.execute('INSERT INTO sessions VALUES (?,?,?,?,?,?,?,?)', - (hashlib.sha256(sessions[name].encode()).hexdigest(), ids[-1], 'dashboard', stamp + 3600, '127.0.0.1', stamp, stamp, stamp)) - assert all(api(name)['shale'] is None for name, _ in accounts) - api(accounts[0][0], 'POST', '/api/mcp/shale', status=200) - old_target = api(accounts[0][0], 'POST', '/api/mcp/shale')['redirect'] - latest_target = api(accounts[0][0], 'POST', '/api/mcp/shale')['redirect'] - request(browser()[0], old_target, status=410) - with sqlite3.connect(args.data_dir / 'connections.sqlite') as db: - db.execute("UPDATE records SET expires=1 WHERE substr(key,1,11)='shale-link:'") - request(browser()[0], latest_target, status=410) - client, cookies, authorization, _ = start(0) - request(client, args.shale_origin + '/-/callback?state=other&code=fixture', status=403) - request(browser()[0], args.url + '/oauth/shale/link/' + 'a' * 43, headers={'Host': 'globe.studio.test'}, status=403) - client, cookies, authorization, _ = start(0) - finish(client, authorize(client, authorization, 1), 403) - assert session(0) is None and session(1) is None and session_count(1) == 0 - oauth_client, _ = public('/oauth/register', 'POST', {'client_name': marker, - 'redirect_uris': ['http://127.0.0.1:29999/shale-callback'], 'token_endpoint_auth_method': 'none'}, status=201) - pending, _, details = pending_request(oauth_client, 0, 'shale:read') - assert not details['linked'] and not details['resources'] - api(accounts[1][0], 'POST', '/api/mcp/shale', 403, {'request': pending}) - first = link(0, pending) - api(accounts[0][0], 'POST', '/api/mcp/consent/' + pending, body={'deny': True}) - assert api(accounts[1][0])['shale'] is None - second = link(1) - assert first['session'] != second['session'] - new_first = link(0) - assert new_first['session'] != first['session'] - backend_session(first, 303) - backend_session(second, 200) - assert session_count(0) == 1 - for index, name in [(0, 'alpha'), (0, 'beta'), (1, 'foreign')]: - repository(index, name) - readonly = consent(oauth_client, 0, 'alpha', 'shale:read offline_access') - writing = consent(oauth_client, 0, 'alpha', 'shale:read shale:write offline_access') - other = consent(oauth_client, 1, 'foreign', 'shale:read shale:write') - read = readonly['access_token'] - write = writing['access_token'] - initialized = rpc(read, 'initialize', {'protocolVersion': '2025-11-25', 'capabilities': {}, - 'clientInfo': {'name': marker, 'version': '1'}}) - assert initialized['capabilities']['tools'] == {} - assert 'Assisted-by:' in initialized['instructions'] and 'model' in initialized['instructions'] - listed = rpc(read, 'tools/list') - assert listed["ttlMs"] == 0 and listed["cacheScope"] == "private", listed - tools = listed["tools"] - assert {tool['name'] for tool in tools} == {'list_repositories', 'list_issues', 'get_issue', 'create_issue', - 'comment_issue', 'set_issue_status', 'set_issue_title', 'list_labels', 'create_label', 'update_label', 'delete_label', 'set_issue_labels'} - assert all('model' in tool['inputSchema']['required'] for tool in tools - if not tool['annotations']['readOnlyHint']) - create_schema = next(tool['inputSchema'] for tool in tools if tool['name'] == 'create_issue') - assert create_schema['properties']['status']['enum'] == ['todo', 'triage'] - assert all(tool['annotations']['readOnlyHint'] == tool['name'].startswith(('list_', 'get_')) for tool in tools) - assert {repo['id'] for repo in call(read, 'list_repositories')['repositories']} == {'alpha'} - grant = next(value for value in records('grant:').values() if value['scopes'] == ['shale:read', 'offline_access'] and value['user'] == ids[0]) - endpoint = '/api/mcp/connections/' + grant['id'] - details = api(accounts[0][0], path=endpoint) - assert details['linked'] and details['selected'] == ['alpha'] - assert {'alpha', 'beta'} <= {resource['id'] for resource in details['resources']} - api(accounts[1][0], path=endpoint, status=404) - api(accounts[1][0], 'POST', endpoint, 404, {'resources': ['foreign']}) - api(accounts[0][0], 'POST', endpoint, 403, {'resources': ['foreign']}) - api(accounts[0][0], 'POST', endpoint, 403, {'resources': ['alpha', 'alpha']}) - api(accounts[0][0], 'POST', endpoint, 400, {'resources': []}) - api(accounts[0][0], 'POST', endpoint, 204, {'resources': ['beta']}) - assert {repo['id'] for repo in call(read, 'list_repositories')['repositories']} == {'beta'} - call(read, 'list_issues', {'repository': 'alpha'}, error=True) - call(read, 'create_issue', {'repository': 'beta', 'title': 'REFUSED'}, error=True) - api(accounts[0][0], 'POST', endpoint, 204, {'resources': 'all'}) - assert api(accounts[0][0], path=endpoint)['selected'] == 'all' - assert {repo['id'] for repo in call(read, 'list_repositories')['repositories']} == {'alpha', 'beta'} - call(read, 'list_issues', {'repository': 'foreign'}, error=True) - call(read, 'create_issue', {'repository': 'beta', 'title': 'REFUSED'}, error=True) - nested = 'userscripts/nested-fixture' - repository(0, nested) - assert {repo['id'] for repo in call(read, 'list_repositories')['repositories']} == {'alpha', 'beta', nested} - assert not call(read, 'list_issues', {'repository': nested})['issues'] - api(accounts[0][0], 'POST', endpoint, 204, {'resources': [nested]}) - assert {repo['id'] for repo in call(read, 'list_repositories')['repositories']} == {nested} - call(read, 'list_issues', {'repository': 'beta'}, error=True) - api(accounts[0][0], 'POST', endpoint, 204, {'resources': ['alpha']}) - - assert {repo['id'] for repo in call(other['access_token'], 'list_repositories')['repositories']} == {'foreign'} - all_repositories = consent(oauth_client, 0, 'all', 'shale:read') - assert {repo['id'] for repo in call(all_repositories['access_token'], 'list_repositories')['repositories']} == {'alpha', 'beta', nested} - assert not call(read, 'list_issues', {'repository': 'alpha'})['issues'] - for token, name in [(read, 'beta'), (read, 'foreign'), (other['access_token'], 'alpha'), - (write, 'alpha/../foreign'), (write, 'https://other.invalid')]: - call(token, 'list_issues', {'repository': name}, error=True) - call(read, 'create_issue', {'repository': 'alpha', 'title': 'REFUSED'}, error=True) - call(write, 'create_issue', {'repository': 'alpha', 'title': 'REFUSED', 'url': 'https://other.invalid'}, error=True) - public('/mcp/observability', 'POST', {'jsonrpc': '2.0', 'id': 1, 'method': 'tools/list'}, token=read, status=401) - call(write, 'create_issue', {'repository': 'alpha', 'title': 'REFUSED'}, error=True) - title = 'MCP & Unicode ☃' - status, _, _ = submit_backend(0, '/alpha/issues/new', {'title': title, - 'description': 'Owned body & Unicode ☃'}) - assert status == 303 - created = call(read, 'get_issue', {'repository': 'alpha', 'id': 1})['issue'] - assert created['title'] == title and created['id'] == 1 - assert len(created['comments']) == 1 and 'Unicode ☃' in created['comments'][0]['text'] - assert '' in created['comments'][0]['text'] - listed = call(read, 'list_issues', {'repository': 'alpha', 'q': 'is:open'})['issues'][0] - assert listed['title'] == title and listed['status'] == created['status'] - rejected = call(read, 'list_issues', {'repository': 'alpha', 'q': 'Unicode'}, error=True) - assert 'filter syntax' in rejected['content'][0]['text'], rejected - issue_fields = {'repository': 'alpha', 'id': created['id']} - status, _, _ = submit_backend(0, '/alpha/issues/labels', {'name': 'Owned ☃', 'description': 'Fixture label', 'color': '#ff40ff'}) - assert status == 303 - status, _, _ = submit_backend(0, '/alpha/issues/1', {'t': 'labels', 'labels': '1'}) - assert status == 303 - changed = call(read, 'get_issue', issue_fields)['issue'] - assert changed['labels'] == ['Owned ☃'], changed - with sqlite3.connect(args.shale_database) as db: - rows = db.execute('SELECT id FROM users WHERE snowflake=?', (ids[1],)).fetchall() - assert len(rows) == 1 - original_owner = db.execute("SELECT owner FROM repositories WHERE name='alpha'").fetchone()[0] - db.execute("UPDATE repositories SET owner=?,access_git_webui='private',access_issues='private' WHERE name='alpha'", (rows[0][0],)) - try: - call(read, 'get_issue', issue_fields, error=True) - call(write, 'create_issue', {'repository': 'alpha', 'title': 'REFUSED'}, error=True) - finally: - with sqlite3.connect(args.shale_database) as db: - db.execute("UPDATE repositories SET owner=?,access_git_webui='public',access_issues='public' WHERE name='alpha'", (original_owner,)) - with sqlite3.connect(args.shale_database) as db: - assert db.execute("SELECT count(*) FROM issues WHERE title='REFUSED'").fetchone()[0] == 0 - subprocess.run(['systemctl', 'restart', args.restart_unit], check=True, capture_output=True, timeout=30) - assert session(0) == new_first and session(1) == second - assert math.isclose(api(accounts[0][0])['shale']['linkedAt'], new_first['linkedAt'], rel_tol=0, abs_tol=1e-6) - backend_session(new_first, 200) - assert call(read, 'get_issue', issue_fields)['issue'] == changed - backend(0, '/-/logout') - call(read, 'get_issue', issue_fields, error=True) - new_first = link(0) - assert call(read, 'get_issue', issue_fields)['issue'] == changed - client, cookies, authorization, _ = start(0) - target = authorize(client, authorization, 0) - subprocess.run(['podman', 'pause', args.shale_container], check=True, capture_output=True) - try: - with ThreadPoolExecutor(max_workers=2) as pool: - callback = pool.submit(finish, client, target, 410) - deadline = time.monotonic() + 10 - while not any(record.get('phase') == 'processing' for record in records('shale-link:').values()): - assert time.monotonic() < deadline, 'callback did not reach backend' - time.sleep(.05) - unlink = pool.submit(api, accounts[0][0], 'DELETE', '/api/mcp/shale', 204) - deadline = time.monotonic() + 10 - while session(0) is not None or records('shale-link:'): - assert time.monotonic() < deadline, 'unlink did not cancel the callback' - time.sleep(.05) - subprocess.run(['podman', 'unpause', args.shale_container], check=True, capture_output=True) - callback.result(timeout=30) - unlink.result(timeout=30) - finally: - subprocess.run(['podman', 'unpause', args.shale_container], capture_output=True) - assert session(0) is None and session_count(0) == 0 - assert api(accounts[0][0])['shale'] is None and session(1) == second - backend_session(second, 200) - public('/mcp/shale', token=read, status=401) - public('/mcp/shale', token=write, status=401) - public('/oauth/token', 'POST', {'grant_type': 'refresh_token', 'client_id': oauth_client['client_id'], - 'refresh_token': readonly['refresh_token'], 'resource': origin + '/mcp/shale'}, form=True, status=400) - api(accounts[1][0], 'DELETE', '/api/mcp/shale', 204) - assert session(1) is None and session_count(1) == 0 - backend_session(second, 303) - api(accounts[2][0], 'DELETE', '/api/mcp/shale', 204) - assert session(2) is None - assert not records('shale-link:') and not records('shale-session:') - result = {'two_user_oidc_linking': True, 'same_realm_ordinary_users': True, 'host_only_callback_cookie': True, - 'single_use_state_and_expiry': True, 'account_mismatch_refused_and_session_removed': True, - 'cross_user_session_isolation': True, 'backend_session_not_exposed_to_browser': True, - 'relink_revokes_previous_backend_session': True, 'restart_preserves_credentials': True, - 'unlink_cancels_inflight_callback': True, 'unlink_revokes_backend_session': True, - 'oauth_link_returns_to_consent': True, 'sdk_catalog_and_repository_grants': True, - 'cross_repository_and_cross_user_tools_refused': True, 'read_only_and_audience_enforced': True, - 'native_issue_read': True, 'unicode_and_rendered_text': True, - 'native_issue_labels_read': True, - 'backend_permission_changes_enforced': True, 'expired_backend_session_refused': True, - 'all_repository_approval_and_dynamic_access': True, 'nested_repository_paths': True, 'existing_token_resource_edits': True, - 'restart_preserves_mcp_access': True, 'unlink_revokes_mcp_access_and_refresh': True} - finally: - keycloak = Keycloak('keycloak.studio.test', importlib.import_module('dashboard-run').secret('get', 'keycloak', 'password'), attempts=1) - cleanup_errors = [] - for (name, _), identity in zip(accounts, ids): - try: - api(name, 'DELETE', '/api/mcp/shale', 204) - except Exception as error: - cleanup_errors.append(error) - try: - keycloak.request('/admin/realms/master/users/' + identity, 'DELETE') - except Exception as error: - cleanup_errors.append(error) - with sqlite3.connect(args.data_dir / 'accounts.sqlite') as db: - db.execute('PRAGMA foreign_keys=ON') - for identity in ids: - for table in ['sessions', 'credentials', 'memberships']: - db.execute(f'DELETE FROM {table} WHERE user_id=?', (identity,)) - db.execute('DELETE FROM users WHERE id=?', (identity,)) - if cleanup_errors: - raise cleanup_errors[0] - result['owned_users_and_credentials_removed'] = True - if args.output: - args.output.write_text(json.dumps(result, indent=2) + '\n') - print(json.dumps(result), flush=True) - - -if __name__ == '__main__': - main() diff --git a/tools/dashboard-shale-page-test.py b/tools/dashboard-shale-page-test.py index 2fa1bd85acf48806b50b0194aabbefe364df49a1..cff199376d419dcc6801bebf9e371396a4cd4485 100644 --- a/tools/dashboard-shale-page-test.py +++ b/tools/dashboard-shale-page-test.py @@ -61,7 +61,7 @@ def main(): return self.respond(b' ' * (9 * 1024 * 1024), 'text/html') if self.path == '/-/login': callback = 'https://' + self.headers['Host'] + '/-/callback' - return self.respond(b'', 'text/plain', 302, [('Location', 'https://snowglobe.studio.test/auth/oidc/authorize?redirect_uri=' + urllib.parse.quote(callback, safe=''))]) + return self.respond(b'', 'text/plain', 302, [('Location', 'https://snowglobe.snowglobe.test/auth/oidc/authorize?redirect_uri=' + urllib.parse.quote(callback, safe=''))]) if self.path == '/redirect': return self.respond(b'', 'text/plain', 302, [('Location', '/snowbound/issues/26'), ('Set-Cookie', 'SessionID=new; Path=/; HttpOnly'), ('Set-Cookie', 'other=kept; Path=/')]) if self.path.startswith('/-/') and args.page: @@ -108,8 +108,8 @@ def main(): subprocess.run(['openssl', 'req', '-x509', '-newkey', 'rsa:2048', '-nodes', '-days', '1', '-subj', '/CN=localhost', '-addext', 'subjectAltName=DNS:localhost', '-addext', 'basicConstraints=CA:FALSE', '-keyout', str(gateway_key), '-out', str(certificate)], check=True, capture_output=True) data = root / 'data' data.mkdir() - environment = {**os.environ, 'PORT': str(dashboard_port), 'STUDIO_DOMAIN': 'studio.test', 'STUDIO_DATA_DIR': str(data), 'STUDIO_PROXY_TOKEN_FILE': str(token), 'STUDIO_INTERNAL_URL': f'https://localhost:{internal_port}', 'STUDIO_CA_BUNDLE': str(certificate), 'STUDIO_REPO': str(repo), 'STUDIO_WEB_DIR': str(repo / 'dashboard/dist')} - for key in ['STUDIO_INDEX_POOL', 'STUDIO_AUTH_REQUIRED', 'STUDIO_YT_STATE']: + environment = {**os.environ, 'PORT': str(dashboard_port), 'SNOWGLOBE_DOMAIN': 'snowglobe.test', 'SNOWGLOBE_DATA_DIR': str(data), 'SNOWGLOBE_PROXY_TOKEN_FILE': str(token), 'SNOWGLOBE_INTERNAL_URL': f'https://localhost:{internal_port}', 'SNOWGLOBE_CA_BUNDLE': str(certificate), 'SNOWGLOBE_REPO': str(repo), 'SNOWGLOBE_WEB_DIR': str(repo / 'dashboard/dist')} + for key in ['SNOWGLOBE_INDEX_POOL', 'SNOWGLOBE_AUTH_REQUIRED', 'SNOWGLOBE_YT_STATE']: environment.pop(key, None) log = stack.enter_context((root / 'dashboard.log').open('w')) dashboard = subprocess.Popen([str(args.binary)], env=environment, stdout=log, stderr=log) @@ -130,17 +130,17 @@ def main(): profile = {'kind': 'guest', 'enabled': True, 'username': username, 'guestProvider': provider, 'firstName': name} db.execute('INSERT INTO users(id,profile) VALUES (?,?)', [username, json.dumps(profile)]) db.execute('INSERT INTO external_identities VALUES (?,?,?)', [provider, subject, username]) - os.environ.update(STUDIO_DOMAIN='studio.test', STUDIO_DASHBOARD_PORT=str(dashboard_port), STUDIO_PROXY_TOKEN_FILE=str(token), STUDIO_INTERNAL_PORT=str(internal_port)) + os.environ.update(SNOWGLOBE_DOMAIN='snowglobe.test', SNOWGLOBE_DASHBOARD_PORT=str(dashboard_port), SNOWGLOBE_PROXY_TOKEN_FILE=str(token), SNOWGLOBE_INTERNAL_PORT=str(internal_port)) router.ROUTE_DIR = str(root / 'routes') Path(router.ROUTE_DIR).mkdir() - (Path(router.ROUTE_DIR) / 'shale.json').write_text(json.dumps({'headHtml': {'shale.studio.test': {'/snowbound/*': ''}}})) + (Path(router.ROUTE_DIR) / 'shale.json').write_text(json.dumps({'headHtml': {'shale.snowglobe.test': {'/snowbound/*': ''}}})) def nomad(path, token): if path == '/v1/services': return [{'Namespace': 'default', 'Services': [{'ServiceName': 'shale'}, {'ServiceName': 'shale-preview-12345678'}]}] if path.startswith('/v1/service/'): service = path.rsplit('/', 1)[1] - host = 'shale.studio.test' if service == 'shale' else service + '.studio.test' + host = 'shale.snowglobe.test' if service == 'shale' else service + '.snowglobe.test' return [{'Address': '127.0.0.1', 'Port': fixture.server_port, 'ServiceName': service, 'AllocID': service, 'JobID': service, 'Tags': ['caddy-host=' + host]}] return {'ready': {'Status': 'success'}} @@ -156,8 +156,8 @@ def main(): return rendered[start:end].replace(host + ' {', f'http://127.0.0.1:{listener} {{', 1).replace(' tls internal\n', '').replace(' route {\n', f' route {{\n request_header Host {host}\n', 1) preview_port = port() - config = '{\n admin off\n auto_https off\n}\n' + local_site('shale.studio.test', gateway_port) + '\n' + local_site('shale-preview-12345678.studio.test', preview_port) - internal_host = f'dashboard.internal.studio.test:{internal_port}' + config = '{\n admin off\n auto_https off\n}\n' + local_site('shale.snowglobe.test', gateway_port) + '\n' + local_site('shale-preview-12345678.snowglobe.test', preview_port) + internal_host = f'dashboard.internal.snowglobe.test:{internal_port}' start = rendered.index(internal_host + ' {') config += '\n' + rendered[start:].replace(internal_host + ' {', f'https://localhost:{internal_port} {{', 1).replace(' tls internal\n', f' tls {certificate} {gateway_key}\n', 1) config_path = root / 'Caddyfile' @@ -170,7 +170,7 @@ def main(): origin = f'http://127.0.0.1:{gateway_port}' def request(path, *, headers=None, body=None, direct=False): - supplied = {'Cookie': 'SessionID=preserved', 'Origin': 'https://shale.studio.test', **(headers or {})} + supplied = {'Cookie': 'SessionID=preserved', 'Origin': 'https://shale.snowglobe.test', **(headers or {})} url = (f'http://127.0.0.1:{dashboard_port}' if direct else origin) + path try: response = opener.open(urllib.request.Request(url, headers=supplied, data=body), timeout=15) @@ -199,25 +199,25 @@ def main(): assert b'rewrite-test' in body assert headers['Cache-Control'] == 'no-store' and headers.get('ETag') is None and headers.get('Last-Modified') is None assert observations[-1][1] == '/snowbound/issues/26?query=kept', observations[-1] - assert observations[-1][2]['host'] == 'shale.studio.test', observations[-1] + assert observations[-1][2]['host'] == 'shale.snowglobe.test', observations[-1] assert observations[-1][2]['cookie'] == 'SessionID=preserved', observations[-1] status, headers, body = request('/json', headers={'Accept': 'application/json'}) assert status == 200 and headers['Content-Type'].startswith('application/json') assert body == b'{"issue":{"title":" & Unicode"}}' assert observations[-1][2]['accept'] == 'application/json' - assert not any(key.lower().startswith('studio-') for key in observations[-1][2]), observations[-1] + assert not any(key.lower().startswith('snowglobe-') for key in observations[-1][2]), observations[-1] status, headers, body = request('/~guest-github-24465214') assert status == 302 and headers['Location'] == 'https://github.com/paperclover' and headers['Referrer-Policy'] == 'no-referrer' assert request('/~guest-astheno-abc/')[0] == 200 assert observations[-1][1] == '/~guest-astheno-abc/' assert request('/~guest-astheno-unknown')[0] == 200 status, headers, body = request('/-/login') - assert status == 302 and urllib.parse.parse_qs(urllib.parse.urlparse(headers['Location']).query)['redirect_uri'] == ['https://shale.studio.test/-/callback'] + assert status == 302 and urllib.parse.parse_qs(urllib.parse.urlparse(headers['Location']).query)['redirect_uri'] == ['https://shale.snowglobe.test/-/callback'] status, headers, body = request('/redirect') assert status == 302 and headers['Location'] == '/snowbound/issues/26' and len(headers.get_all('Set-Cookie')) == 2 with opener.open(f'http://127.0.0.1:{preview_port}/snowbound/issues/26') as response: assert response.status == 200 - assert observations[-1][2]['host'] == 'shale-preview-12345678.studio.test' + assert observations[-1][2]['host'] == 'shale-preview-12345678.snowglobe.test' assert any(link.get('href') == 'https://github.com/paperclover' for link in Links(response.read().decode()).links) assert b'/~guest-github-24465214' in request('/repo/info/refs')[2] assert request('/binary')[2] == blob @@ -226,7 +226,7 @@ def main(): assert status == 201 and body == b'field=unchanged' and observations[-1][1] == '/submit?keep=yes' assert request('/submit', body=b'blocked', headers={'Origin': 'https://other.example'})[0] == 403 assert request('/auth/shale/page', direct=True)[0] == 403 - assert request('/auth/shale/page', direct=True, headers={'Studio-Proxy-Token': proof, 'Studio-Shale-Upstream': f'127.0.0.1:{fixture.server_port}', 'Studio-Shale-Uri': '/', 'Host': 'snowglobe.studio.test'})[0] == 403 + assert request('/auth/shale/page', direct=True, headers={'Snowglobe-Proxy-Token': proof, 'Snowglobe-Shale-Upstream': f'127.0.0.1:{fixture.server_port}', 'Snowglobe-Shale-Uri': '/', 'Host': 'snowglobe.snowglobe.test'})[0] == 403 print(json.dumps({'html_links': 'passed', 'provider_redirects': 'passed', 'cookies_forms_and_binary': 'passed', 'proxy_boundary': 'passed', 'head_injection': 'passed', 'origin': origin}), flush=True) if args.browser_ready_file: args.browser_ready_file.write_text(json.dumps({'origin': origin})) diff --git a/tools/dashboard-shale-test.py b/tools/dashboard-shale-test.py deleted file mode 100644 index b343cc0e04897cb23e2e44891002b71fbdfbde74..0000000000000000000000000000000000000000 --- a/tools/dashboard-shale-test.py +++ /dev/null @@ -1,468 +0,0 @@ -#!/usr/bin/env python3 -import argparse -import base64 -from contextlib import closing, contextmanager -import copy -from html.parser import HTMLParser -import http.client -import http.cookiejar -import http.server -import importlib -import json -import os -from pathlib import Path -import re -import sqlite3 -import ssl -import socket -import subprocess -import sys -import tempfile -import threading -import time -import urllib.error -import urllib.parse -import urllib.request -import uuid - - -class Page(HTMLParser): - def __init__(self, text): - super().__init__() - self.forms = [] - self.links = [] - self.codes = [] - self.form = None - self.code = None - self.select = None - self.feed(text) - - def handle_starttag(self, tag, attrs): - attrs = dict(attrs) - if tag == "form": - self.form = {**attrs, "fields": []} - self.forms.append(self.form) - if tag in {"input", "select", "textarea", "button"} and self.form is not None: - self.form["fields"].append(attrs) - if tag == "select": - self.select = attrs - attrs["options"] = [] - if tag == "option" and self.select is not None: - self.select["options"].append(attrs) - if tag == "a": - self.links.append(attrs.get("href", "")) - if tag in {"code", "pre", "textarea"}: - self.code = "" - - def handle_data(self, data): - if self.code is not None: - self.code += data - - def handle_endtag(self, tag): - if tag == "select": - self.select = None - if tag == "form": - self.form = None - if tag in {"code", "pre", "textarea"} and self.code is not None: - self.codes.append(self.code) - self.code = None - - -@contextmanager -def instance(keycloak, marker, image, traces_endpoint, traces_env, *, host="localhost", dashboard_port=None): - container = "studio-shale-backend-test-" + marker - client_name = "shale-fixture-" + marker - client_secret = uuid.uuid4().hex + uuid.uuid4().hex - client_id = None - caddy = None - proxy = None - - def run(*argv): - try: - return subprocess.check_output(argv, text=True, stderr=subprocess.PIPE).strip() - except subprocess.CalledProcessError as error: - raise AssertionError(f"{argv[0]} failed ({error.returncode})") from None - - with tempfile.TemporaryDirectory(prefix="studio-shale-backend-", dir="/run") as temporary: - root = Path(temporary) - with socket.socket() as reservation: - reservation.bind(("127.0.0.1", 0)) - backend_port = reservation.getsockname()[1] - with socket.socket() as reservation: - for port in range(20000, 32001): - try: - reservation.bind(("127.0.0.1", port)) - break - except OSError: - continue - else: - raise AssertionError("no fixture HTTPS port available") - origin = "https://" + host + ":" + str(port) - try: - keycloak.request("/admin/realms/master/clients", "POST", { - "clientId": client_name, "protocol": "openid-connect", "publicClient": False, - "standardFlowEnabled": True, "secret": client_secret, - "redirectUris": [origin + "/-/callback"], "webOrigins": [origin], - }) - clients = keycloak.request("/admin/realms/master/clients?clientId=" + client_name) - assert len(clients) == 1 - client_id = clients[0]["id"] - authority = Path("/var/lib/caddy/.local/share/caddy/pki/authorities/local") - run("openssl", "req", "-new", "-newkey", "rsa:2048", "-nodes", "-keyout", str(root / "key.pem"), - "-out", str(root / "request.pem"), "-subj", "/CN=" + host, - "-addext", "subjectAltName=DNS:" + host + ",IP:127.0.0.1") - run("openssl", "x509", "-req", "-in", str(root / "request.pem"), "-CA", str(authority / "root.crt"), - "-CAkey", str(authority / "root.key"), "-set_serial", "0x" + marker, - "-out", str(root / "cert.pem"), "-days", "1", "-copy_extensions", "copyall") - config = root / "Caddyfile" - bridge = importlib.import_module("router").shale_mcp_routes(dashboard_port) if dashboard_port else [] - upstream_port = backend_port - if dashboard_port: - drop = root / "drop-next-write" - - class Proxy(http.server.BaseHTTPRequestHandler): - def log_message(self, *args): - pass - - def do_GET(self): - with closing(http.client.HTTPConnection("127.0.0.1", backend_port, timeout=20)) as upstream: - upstream.request(self.command, self.path, - self.rfile.read(int(self.headers.get("Content-Length", 0))), dict(self.headers)) - response = upstream.getresponse() - body = response.read() - if self.command == "POST" and drop.exists() and drop.read_text() == self.path: - drop.unlink() - self.close_connection = True - self.connection.shutdown(socket.SHUT_RDWR) - return - self.send_response(response.status) - for name, value in response.getheaders(): - if name.lower() not in {"connection", "transfer-encoding", "content-length"}: - self.send_header(name, value) - self.send_header("Content-Length", str(len(body))) - self.end_headers() - self.wfile.write(body) - - do_POST = do_GET - - proxy = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Proxy) - proxy_thread = threading.Thread(target=proxy.serve_forever, daemon=True) - proxy_thread.start() - upstream_port = proxy.server_port - config.write_text("{\n admin off\n auto_https disable_redirects\n skip_install_trust\n}\n" + origin + - " {\n tls " + str(root / "cert.pem") + " " + str(root / "key.pem") + - "\n" + "\n".join(bridge) + "\n handle {\n reverse_proxy 127.0.0.1:" + str(upstream_port) + "\n }\n}\n") - binary = (Path("/proc") / run("systemctl", "show", "-P", "MainPID", "caddy") / "exe").resolve(strict=True) - with (root / "caddy.log").open("w") as output: - caddy = subprocess.Popen([str(binary), "run", "--config", str(config), "--adapter", "caddyfile"], - stdout=output, stderr=output, - env={**os.environ, "XDG_DATA_HOME": temporary, "XDG_CONFIG_HOME": temporary}) - volumes = [] - for directory, target in [("data", "/data"), ("owned", "/repositories_owned"), ("mirrors", "/repositories_mirrors"), ("cron", "/etc/crontabs")]: - path = root / directory - path.mkdir() - os.chown(path, 65534, 65534) - volumes.append("--volume=" + str(path) + ":" + target + ":rw") - run("podman", "run", "-d", "--name=" + container, "--pull=never", "--user=65534:65534", - "--read-only", "--cap-drop=all", "--security-opt=no-new-privileges", "--memory=512m", "--cpus=2", "--pids-limit=128", - "--tmpfs=/tmp:rw,noexec,nosuid,nodev,size=64m", "--publish=127.0.0.1:" + str(backend_port) + ":8000", - "--add-host=keycloak.studio.test:host-gateway", "--volume=/var/lib/studio/ca-bundle.crt:/etc/ssl/certs/ca-certificates.crt:ro", - "--env=DOMAIN=" + host + ":" + str(port), "--env=SERVER_TITLE=MCP fixture", - "--env=SESSION_SECRET=" + uuid.uuid4().hex + uuid.uuid4().hex, - "--env=OAUTH2_CLIENT=oidc,keycloak.studio.test/realms/master|" + client_name + "|" + client_secret, - "--env=NPROC=2", "--env=MIRRORS_DISABLE_CLONE=true", "--env=MIRRORS_DISABLE_PULL_POLL=true", - *(["--env=OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=" + traces_endpoint] if traces_env else []), - *volumes, image, *(["--otel-exporter-otlp-traces-endpoint", traces_endpoint] if traces_endpoint and not traces_env else [])) - deployed = json.loads(run("podman", "inspect", container))[0] - assert deployed["ImageName"] == image - assert deployed["Config"]["User"] == "65534:65534" - context = ssl.create_default_context(cafile="/var/lib/studio/ca-bundle.crt") - from api import LoopbackHTTPS - - class HTTPS(urllib.request.HTTPSHandler): - def https_open(self, request): - assert urllib.parse.urlsplit(request.full_url).netloc == urllib.parse.urlsplit(origin).netloc - return self.do_open(LoopbackHTTPS, request, context=context) - - readiness = urllib.request.build_opener(HTTPS()) - deadline = time.monotonic() + 15 - while True: - try: - with readiness.open(origin, timeout=2) as response: - assert response.status == 200 - break - except OSError: - if time.monotonic() >= deadline: - raise AssertionError((root / "caddy.log").read_text()) - time.sleep(.1) - yield origin, root / "data/astheno.shale.db", container - except Exception: - if subprocess.run(["podman", "container", "exists", container], capture_output=True).returncode == 0: - state = json.loads(run("podman", "inspect", container))[0]["State"] - print(json.dumps({"fixture_backend_running": state["Running"], "fixture_backend_exit_code": state["ExitCode"], - "fixture_backend_oom_killed": state["OOMKilled"]}), flush=True) - raise - finally: - try: - subprocess.run(["podman", "rm", "--ignore", "--force", container], check=True, capture_output=True) - finally: - try: - if caddy: - caddy.terminate() - try: - caddy.wait(timeout=10) - except subprocess.TimeoutExpired: - caddy.kill() - caddy.wait(timeout=10) - finally: - if proxy: - proxy.shutdown() - proxy.server_close() - proxy_thread.join(timeout=5) - from api import Keycloak - keycloak = Keycloak(keycloak.host, importlib.import_module("dashboard-run").secret("get", "keycloak", "password"), attempts=1) - if client_id is None: - clients = keycloak.request("/admin/realms/master/clients?clientId=" + client_name) - assert len(clients) <= 1 - client_id = clients[0]["id"] if clients else None - if client_id: - keycloak.request("/admin/realms/master/clients/" + client_id, "DELETE") - - - -def main(): - parser = argparse.ArgumentParser() - parser.add_argument("--output", type=Path) - parser.add_argument("--image") - parser.add_argument("--traces-endpoint") - parser.add_argument("--traces-env", action="store_true") - args = parser.parse_args() - if args.output: - args.output.unlink(missing_ok=True) - assert not args.traces_env or args.traces_endpoint - if args.traces_endpoint: - endpoint = urllib.parse.urlsplit(args.traces_endpoint) - assert endpoint.scheme in {"http", "https"} and endpoint.hostname - assert not endpoint.username and not endpoint.password - repo = Path(__file__).resolve().parent.parent - image = args.image or re.search(r'^\s*image = "([^"]+)"', (repo / "service/shale/service.pkl").read_text(), re.MULTILINE).group(1) - sys.path.insert(0, str(repo / "tools")) - sys.path.insert(0, str(repo / "service/keycloak")) - from api import Keycloak, LoopbackHTTPS - keycloak_host = "keycloak.studio.test" - shale_host = "localhost" - context = ssl.create_default_context(cafile="/var/lib/studio/ca-bundle.crt") - - class HTTPS(urllib.request.HTTPSHandler, urllib.request.HTTPHandler): - def http_open(self, request): - raise AssertionError("unencrypted fixture redirect") - - def https_open(self, request): - parts = urllib.parse.urlsplit(request.full_url) - assert parts.netloc in {keycloak_host, urllib.parse.urlsplit(shale).netloc} - return self.do_open(LoopbackHTTPS, request, context=context) - - class NoRedirect(urllib.request.HTTPRedirectHandler): - def redirect_request(self, req, fp, code, msg, headers, newurl): - return None - - key = uuid.uuid4().hex - name = "mcp-fixture-" + key - password = uuid.uuid4().hex + "A1!" - keycloak = Keycloak(keycloak_host, importlib.import_module("dashboard-run").secret("get", "keycloak", "password")) - cookies = http.cookiejar.CookieJar() - opener = urllib.request.build_opener(HTTPS(), urllib.request.HTTPCookieProcessor(cookies)) - - def request(url, fields=None, headers=None, client=opener): - headers = dict(headers or {}) - data = None - if fields is not None: - parts = urllib.parse.urlsplit(url) - headers.update(Origin=parts.scheme + "://" + parts.netloc, Referer=parts._replace(fragment="").geturl()) - data = urllib.parse.urlencode(fields).encode() - req = urllib.request.Request(url, headers=headers, data=data) - try: - response = client.open(req, timeout=15) - except urllib.error.HTTPError as error: - response = error - with response: - return response.status, response.url, response.headers, response.read().decode() - - def submit(url, form, changes, client=opener): - target = urllib.parse.urljoin(url, form.get("action", "")) - assert urllib.parse.urlsplit(target).netloc == urllib.parse.urlsplit(shale).netloc - fields = {a["name"]: a.get("value", "") for a in form["fields"] if a.get("name") and a.get("type") == "hidden"} - fields.update(timezone="UTC", tzoffset="+00:00", **changes) - return request(target, fields, client=client) - - accounts = [(name, password), ("mcp-stranger-" + key, uuid.uuid4().hex + "A1!")] - try: - for username, credential in accounts: - keycloak.request("/admin/realms/master/users", "POST", { - "username": username, "firstName": username, "lastName": "Fixture", "email": username + "@fixture.invalid", - "emailVerified": True, "enabled": True, - "credentials": [{"type": "password", "value": credential, "temporary": False}], - }) - users = keycloak.request("/admin/realms/master/users?username=" + username + "&exact=true") - assert len(users) == 1 - with instance(keycloak, key, image, args.traces_endpoint, args.traces_env) as (shale, database, container): - def login(client, username, credential): - status, url, _, body = request(shale + "/-/login", client=client) - assert status == 200, status - form = next(f for f in Page(body).forms if any(a.get("name") == "password" for a in f["fields"])) - fields = {a["name"]: a.get("value", "") for a in form["fields"] if a.get("name")} - fields.update(username=username, password=credential) - status, url, _, _ = request(urllib.parse.urljoin(url, form["action"]), fields, client=client) - assert status == 200 and urllib.parse.urlsplit(url).hostname == shale_host - - login(opener, *accounts[0]) - stranger = urllib.request.build_opener(HTTPS(), urllib.request.HTTPCookieProcessor(http.cookiejar.CookieJar())) - login(stranger, *accounts[1]) - status, url, _, body = request(shale + "/-/settings") - assert status == 200, status - form = next(f for f in Page(body).forms if any(a.get("name") == "label" for a in f["fields"])) - assert form.get("enctype", "application/x-www-form-urlencoded") == "application/x-www-form-urlencoded" - status, _, _, body = submit(url, form, {"label": name}) - if status != 200: - state = json.loads(subprocess.check_output(["podman", "inspect", container], text=True))[0]["State"] - with sqlite3.connect(database.as_uri() + "?mode=ro", uri=True) as db: - count = db.execute("SELECT count(*) FROM personal_tokens").fetchone()[0] - result = {"oidc_fixture_user": True, "mint_status": status, - "backend_exit_code": state["ExitCode"], "token_rows_after_response": count} - print(json.dumps(result), flush=True) - if args.output: - args.output.write_text(json.dumps(result, indent=2) + "\n") - raise AssertionError("token mint failed; write outcome recorded without replay") - with sqlite3.connect(database.as_uri() + "?mode=ro", uri=True) as db: - users = db.execute("SELECT id FROM users WHERE name=?", (name,)).fetchall() - assert len(users) == 1 - identity = users[0][0] - tokens = db.execute("SELECT uuid FROM personal_tokens WHERE agent=? AND label=?", (identity, name)).fetchall() - assert len(tokens) == 1 - token_id = tokens[0][0] - page = Page(body) - tokens = {text.strip() for text in [*page.codes, *[a.get("value", "") for f in page.forms for a in f["fields"]]] - if re.fullmatch(r"[A-Za-z0-9_:.=+-]{24,}", text.strip())} - assert len(tokens) == 1 - token = next(iter(tokens)) - anonymous = urllib.request.build_opener(HTTPS(), NoRedirect()) - for authorization in ["Bearer " + token, "token " + token, - "Basic " + base64.b64encode((name + ":" + token).encode()).decode()]: - status, _, _, body = request(shale + "/-/settings", headers={"Authorization": authorization}, client=anonymous) - assert status == 303 and name not in body - status, url, _, body = request(shale + "/-/new") - assert status == 200 - form = next(f for f in Page(body).forms if any(a.get("name") == "name" for a in f["fields"])) - status, url, _, body = submit(url, form, {"name": "fixture", "description": "Disposable MCP adapter test", "access": "private"}) - assert status == 200, status - with sqlite3.connect(database.as_uri() + "?mode=ro", uri=True) as db: - assert db.execute("SELECT count(*) FROM repositories WHERE name='fixture' AND owner=?", (identity,)).fetchone()[0] == 1 - repository_url = url - for client in (anonymous, stranger): - status, _, _, _ = request(repository_url, client=client) - assert status in (403, 404), status - issues = next(link for link in Page(body).links if "issues" in link) - status, url, _, body = request(urllib.parse.urljoin(repository_url, issues)) - assert status == 200 - new_issue = next(link for link in Page(body).links if link == "./new") - status, url, _, body = request(urllib.parse.urljoin(url, new_issue)) - assert status == 200 - title = "Fixture & Unicode ☃" - description = "Disposable body & Unicode ☃" - form = next(f for f in Page(body).forms if any(a.get("name") == "title" for a in f["fields"])) - status, url, _, body = submit(url, form, {"title": title, "description": description}) - assert status == 200, status - issue_url = url - with sqlite3.connect(database.as_uri() + "?mode=ro", uri=True) as db: - issues = db.execute("SELECT id,title FROM issues").fetchall() - assert len(issues) == 1 and issues[0][1] == title - assert db.execute("SELECT payload FROM issue_actions WHERE issue=?", (issues[0][0],)).fetchone()[0] == description - status, _, _, _ = request(issue_url, client=anonymous) - assert status in (403, 404), status - status, _, _, body = request(issue_url) - assert status == 200 - page = Page(body) - comment = "Fixture comment & Unicode ☃" - form = next(f for f in page.forms if any(a.get("name") == "t" and a.get("value") == "comment" for a in f["fields"])) - status, _, _, body = submit(issue_url, form, {"comment": comment}) - assert status == 200, status - with sqlite3.connect(database.as_uri() + "?mode=ro", uri=True) as db: - assert db.execute("SELECT count(*) FROM issue_actions WHERE payload=?", (comment,)).fetchone()[0] == 1 - form = next(f for f in Page(body).forms if any(a.get("name") == "t" and a.get("value") == "status" for a in f["fields"])) - status_field = next(a for a in form["fields"] if a.get("name") == "status") - assert any(option["value"] == "done" for option in status_field["options"]) - status, _, _, body = submit(issue_url, form, {"status": "done"}) - assert status == 200 - with sqlite3.connect(database.as_uri() + "?mode=ro", uri=True) as db: - assert db.execute("SELECT status FROM issues").fetchone()[0] == "done" - status, _, _, stranger_settings = request(shale + "/-/settings", client=stranger) - assert status == 200 - stranger_csrf = {a["name"]: a["value"] for f in Page(stranger_settings).forms for a in f["fields"] if a.get("name") == "csrf_token"} - for client, csrf in [(anonymous, {}), (stranger, stranger_csrf)]: - status, _, _, _ = request(issue_url, {"t": "comment", "comment": "UNAUTHORIZED", - "timezone": "UTC", "tzoffset": "+00:00", **csrf}, client=client) - assert status in ((400, 403, 404) if client is anonymous else (403, 404)), status - with sqlite3.connect(database.as_uri() + "?mode=ro", uri=True) as db: - assert db.execute("SELECT count(*) FROM issue_actions WHERE payload='UNAUTHORIZED'").fetchone()[0] == 0 - subprocess.run(["podman", "restart", container], check=True, capture_output=True) - deadline = time.monotonic() + 15 - while True: - status, _, _, body = request(issue_url) - if status == 200: - break - assert time.monotonic() < deadline, status - time.sleep(.1) - assert "Unicode ☃" in body - for client in (anonymous, stranger): - status, _, _, _ = request(issue_url, client=client) - assert status in (403, 404), status - status, url, _, body = request(shale + "/-/settings") - form = next(f for f in Page(body).forms if any(a.get("name") == "label" for a in f["fields"])) - status, _, _, body = submit(url, form, {"label": "second"}) - assert status == 200, status - form = next(f for f in Page(body).forms if any(a.get("name") == "t" and a.get("value") == "token_revoke" for a in f["fields"])) - status, _, _, _ = submit(url, form, {}) - assert status == 200 - with sqlite3.connect(database.as_uri() + "?mode=ro", uri=True) as db: - assert db.execute("SELECT count(*) FROM personal_tokens WHERE agent=?", (identity,)).fetchone()[0] == 1 - assert db.execute("SELECT count(*) FROM personal_tokens WHERE uuid=?", (token_id,)).fetchone()[0] == 0 - captured = http.cookiejar.CookieJar() - for cookie in cookies: - if cookie.domain.startswith("localhost"): - captured.set_cookie(copy.copy(cookie)) - assert len(captured) > 0 - replay = urllib.request.build_opener(HTTPS(), NoRedirect(), urllib.request.HTTPCookieProcessor(captured)) - status, _, _, _ = request(issue_url, client=replay) - assert status == 200 - status, _, _, _ = request(shale + "/-/logout") - assert status == 200 - status, _, _, _ = request(issue_url) - assert status in (403, 404), status - replay_status, _, _, _ = request(issue_url, client=replay) - assert replay_status in (200, 303, 403, 404), replay_status - with sqlite3.connect(database.as_uri() + "?mode=ro", uri=True) as db: - session_rows = db.execute("SELECT count(*) FROM sessions WHERE user=?", (identity,)).fetchone()[0] - result = {"backend_image": image, - "backend_image_id": json.loads(subprocess.check_output(["podman", "inspect", container], text=True))[0]["Image"], - "traces_endpoint": args.traces_endpoint, "traces_environment_variable": args.traces_env, - "logout_cookie_replay_status": replay_status, - "backend_session_rows_after_logout": session_rows, - "checks": ["two_user_oidc", "personal_token_mint", "pat_not_html_auth", "private_repository_create", - "anonymous_read_denied", "other_user_read_denied", "issue_create_unicode", "comment_create_unicode", - "issue_close", "anonymous_write_denied", "other_user_write_denied", "session_survives_restart", - "second_token_mint", "token_revoke", "logout_denies_private_read", - "cross_user_denied_after_restart", "captured_session_valid_before_logout", "logout_cookie_replay_probe"]} - finally: - for username, _ in accounts: - users = keycloak.request("/admin/realms/master/users?username=" + username + "&exact=true") - for user in users: - assert user["username"] == username - keycloak.request("/admin/realms/master/users/" + user["id"], "DELETE") - result["checks"].append("fixture_cleanup") - print(json.dumps(result), flush=True) - if args.output: - args.output.write_text(json.dumps(result, indent=2) + "\n") - - -if __name__ == "__main__": - main() diff --git a/tools/dashboard-stage-test.py b/tools/dashboard-stage-test.py new file mode 100644 index 0000000000000000000000000000000000000000..99f1cfb78d04c5e922f95832ffb6f6cf53f2b8f8 --- /dev/null +++ b/tools/dashboard-stage-test.py @@ -0,0 +1,113 @@ +#!/usr/bin/env python3 +from importlib import import_module +import json +import os +from pathlib import Path +import struct +import subprocess +import tempfile +import unittest +from unittest.mock import Mock, patch + + +stage = import_module("dashboard-stage") +boundary = import_module("dashboard-host") + + +class PreviewBoundary(unittest.TestCase): + def call(self, request): + payload = json.dumps(request).encode() + b"\n" + connection = Mock() + connection.__enter__ = Mock(return_value=connection) + connection.__exit__ = Mock(return_value=False) + connection.getsockopt.return_value = struct.pack("3i", 1, os.getuid(), 1) + connection.recv.side_effect = [payload, payload] + host = Mock() + host.handle.return_value = {"observed": True} + with patch.object(boundary.socket, "SO_PEERCRED", 17, create=True), patch.object(boundary, "stream_console") as stream, patch.object(boundary, "stream_upload") as upload: + boundary.serve_connection(connection, host, os.getuid(), stage.OPERATIONS) + stream.assert_not_called() + upload.assert_not_called() + sent = b"".join(call.args[0] for call in connection.sendall.call_args_list) + return json.loads(sent[4:]), host + + def test_denies_host_mutations_and_vm_streams_before_dispatch(self): + for operation in ["vm.create", "vm.start", "vm.console", "vm.serial", "vm.guest", "vm.upload", "storage.destroy", "deploy.start", "deploy.secret.get", "shale.write", "copyparty.manage", []]: + with self.subTest(operation=operation): + response, host = self.call({"operation": operation}) + self.assertEqual(response["status"], 403) + host.handle.assert_not_called() + + def test_read_uses_the_real_host_handler(self): + for operation in ["host.sample", "vm.node", "vm.domains", "vm.library"]: + with self.subTest(operation=operation): + request = {"operation": operation} + response, host = self.call(request) + self.assertEqual(response, {"value": {"observed": True}}) + host.handle.assert_called_once_with(request) + + def test_stage_identity_rejects_paths_and_production(self): + for identity in ["dashboard", "knife-preview-12345678", "dashboard-preview-../private", "dashboard-preview-12345678\n"]: + with self.assertRaisesRegex(ValueError, "invalid dashboard stage"): + stage.metadata(identity) + + def test_file_operations_cannot_name_production_or_account_dataset(self): + dataset = "globe/staging/dashboard-preview-12345678" + host = stage.PreviewHost({"clone": dataset}) + for operation in ["files.snapshot", "files.snapshots", "files.discard"]: + for target in ["globe/prod/knife", dataset, dataset + "/files/child", dataset + "/files/../data"]: + request = {"operation": operation, "dataset": target} if operation == "files.discard" else {"operation": operation, "datasets": [target]} + with self.subTest(request=request), patch.object(boundary.Host, "handle") as handler: + with self.assertRaises(boundary.dashboard_runs.Error) as error: + host.handle(request) + self.assertEqual(error.exception.status, 403) + handler.assert_not_called() + with patch.object(boundary.Host, "handle", return_value="real snapshot") as handler: + request = {"operation": "files.snapshot", "datasets": [dataset + "/files"]} + self.assertEqual(host.handle(request), "real snapshot") + handler.assert_called_once_with(request) + + def test_failed_child_creation_and_repeated_token_revoke_still_clean_parent(self): + identity = "dashboard-preview-12345678" + dataset = "globe/staging/" + identity + saved = {"clone": dataset, "mount": "/srv/staging/" + identity, "tokenAccessor": "already-revoked", + "external": {"files": {"clone": dataset + "/files"}}} + missing_token = subprocess.CompletedProcess([], 1, stderr="Error deleting token: Unexpected response code: 400 (Cannot delete nonexistent tokens: already-revoked)") + okay = subprocess.CompletedProcess([], 0, stderr="") + def process(argv, **kwargs): + return missing_token if argv[:4] == ["nomad", "acl", "token", "delete"] else okay + with patch.object(stage, "metadata", return_value=saved), patch.object(stage.snowglobe, "token"), patch.object(stage.subprocess, "run", side_effect=process), patch.object(stage.snowglobe, "command", return_value=subprocess.CompletedProcess([], 0, stdout=dataset + "\n")) as commands, patch.object(stage.snowglobe, "destroy_stage") as destroy, patch.object(stage.shutil, "rmtree"): + stage.destroy(identity) + self.assertEqual(saved["external"], {}) + self.assertEqual(saved["clone"], dataset) + destroy.assert_called_once_with(identity, saved, {}) + self.assertFalse(any(call.args[:2] == ("zfs", "destroy") for call in commands.call_args_list)) + + +class NativePreview(unittest.TestCase): + def test_empty_proof_requires_loopback_and_fresh_account_stays_private(self): + binary = Path(os.environ.get("DASHBOARD_BINARY", "dashboard/target/debug/home-dashboard")).resolve() + if not binary.exists(): + self.skipTest("build home-dashboard to exercise its native auth CLI") + with tempfile.TemporaryDirectory() as temporary: + env = {**os.environ, "SNOWGLOBE_LISTEN_ADDRESS": "0.0.0.0", "SNOWGLOBE_PROXY_TOKEN_FILE": "", "SNOWGLOBE_DATA_DIR": temporary, + "SNOWGLOBE_DOMAIN": "preview.test", "SNOWGLOBE_PUBLIC_ORIGIN": "https://dashboard.preview.test", "SNOWGLOBE_FILE_ORIGIN": "https://files.preview.test", + "SNOWGLOBE_AUTH_RP_ID": "dashboard.preview.test"} + rejected = subprocess.run([str(binary), "--preview-account"], env=env, capture_output=True, text=True) + self.assertNotEqual(rejected.returncode, 0) + self.assertIn("before listening on a non-loopback address", rejected.stderr) + self.assertFalse(Path(temporary, "accounts.sqlite").exists()) + env["SNOWGLOBE_LISTEN_ADDRESS"] = "127.0.0.1" + created = subprocess.run([str(binary), "--preview-account"], env=env, capture_output=True, text=True, check=True) + credentials = json.loads(created.stdout) + self.assertEqual(credentials["username"], "preview") + self.assertGreaterEqual(len(credentials["password"]), 32) + self.assertEqual(Path(temporary, "accounts.sqlite").stat().st_mode & 0o777, 0o600) + again = subprocess.run([str(binary), "--preview-account"], env=env, capture_output=True, text=True) + self.assertNotEqual(again.returncode, 0) + self.assertIn("empty store", again.stderr) + self.assertNotIn(credentials["password"], again.stdout + again.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/dashboard-stage.py b/tools/dashboard-stage.py new file mode 100644 index 0000000000000000000000000000000000000000..1c8e1e116faa9205d368bdf1d0052e3433feec00 --- /dev/null +++ b/tools/dashboard-stage.py @@ -0,0 +1,214 @@ +#!/usr/bin/env python3 +import concurrent.futures +from importlib import import_module +import json +import os +from pathlib import Path +import re +import secrets +import shlex +import shutil +import socket +import subprocess +import sys +import threading +import time + +import snowglobe + + +OPERATIONS = frozenset({ + "host.sample", "host.usage", "storage.mounts", "storage.datasets", "storage.pool", + "storage.snapshots", "vm.node", "vm.domains", "vm.library", "deploy.current", "deploy.main", "deploy.history", + "deploy.stages", "deploy.managed", "deploy.release", "deploy.last", + "files.snapshot", "files.snapshots", "files.discard", +}) +STAGE_ID = re.compile(r"dashboard-preview-[0-9a-f]{8}\Z") +boundary = import_module("dashboard-host") + + +class PreviewHost(boundary.Host): + def __init__(self, saved): + super().__init__(saved["clone"].split("/", 1)[0]) + self.stage = saved + + def handle(self, request): + operation = request.get("operation") if isinstance(request, dict) else None + if operation in {"files.snapshot", "files.snapshots", "files.discard"}: + datasets = request.get("datasets") if operation != "files.discard" else [request.get("dataset")] + if not isinstance(datasets, list) or datasets != [self.stage["clone"] + "/files"]: + raise boundary.dashboard_runs.Error(403, "File changes are limited to this preview's files.") + if operation == "storage.mounts" and set(request) == {"operation"}: + mounted = json.loads(snowglobe.command( + "findmnt", "--json", "-o", "SOURCE,TARGET,FSTYPE,OPTIONS", "--mountpoint", self.stage["mount"] + "/files", capture=True, + ).stdout) + mounted["filesystems"][0]["target"] = "/srv" + return mounted + return super().handle(request) + + +def metadata(stage): + if not STAGE_ID.fullmatch(stage): + raise ValueError("invalid dashboard stage ID") + saved = json.loads((snowglobe.STATE / "stages" / f"{stage}.json").read_text()) + if saved.get("sourceId") != "dashboard": + raise ValueError("not a dashboard stage") + return saved + + +def serve(stage): + saved = metadata(stage) + host = PreviewHost(saved) + runtime = Path("/run/snowglobe-preview") / stage + runtime.mkdir(parents=True, exist_ok=True) + os.chown(runtime, 0, saved["uid"]) + runtime.chmod(0o750) + path = runtime / "host.sock" + path.unlink(missing_ok=True) + with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as listener: + listener.bind(str(path)) + os.chown(path, 0, saved["uid"]) + path.chmod(0o660) + listener.listen(8) + slots = threading.BoundedSemaphore(8) + with concurrent.futures.ThreadPoolExecutor(max_workers=4) as workers: + while True: + connection, _ = listener.accept() + if not slots.acquire(blocking=False): + connection.close() + continue + work = workers.submit(boundary.serve_connection, connection, host, saved["uid"], OPERATIONS) + work.add_done_callback(lambda _: slots.release()) + + +def stage(): + snowglobe.token() + site = json.loads(snowglobe.command("pkl", "eval", "-f", "json", str(snowglobe.REPO / "config/site.pkl"), capture=True).stdout) + site["hostRoot"] = str(Path(site["root"]) / "prod/dashboard") + site["stagingRoot"] = str(Path(site["root"]) / "staging") + snowglobe.check_staging_pool(site) + directory = snowglobe.STATE / "stages" + previous = [path for path in directory.glob("*.json") if json.loads(path.read_text()).get("sourceId") == "dashboard"] + if len(previous) > 1: + raise ValueError("multiple dashboard stages exist; destroy one first") + created = not previous + identity = "dashboard-preview-" + secrets.token_hex(4) if created else previous[0].stem + hostname = identity + "." + site["domain"] + root = Path(site["stagingRoot"]) / identity + saved = {"sourceId": "dashboard", "tasks": ["app"], "inputs": {}, "mount": str(root), + "clone": site["pool"] + "/staging/" + identity, "uid": snowglobe.identity("dashboard"), + "stageIsolation": "fresh"} if created else metadata(identity) + nix = ["nix", "--extra-experimental-features", "nix-command flakes"] + package = "path:" + str(snowglobe.REPO) + "#dashboard-image" + image = subprocess.run([*nix, "eval", "--raw", package, "--apply", 'image: "${image.imageName}:${image.imageTag}"'], + stdout=subprocess.PIPE, text=True, check=True).stdout.strip() + if subprocess.run(["podman", "image", "exists", image], capture_output=True).returncode: + built = subprocess.run([*nix, "build", package, "--no-link", "--print-out-paths"], + stdout=subprocess.PIPE, text=True, check=True).stdout.strip() + snowglobe.command("podman", "load", "--quiet", "--input", built) + env = { + "SNOWGLOBE_DOMAIN": site["domain"], "SNOWGLOBE_PUBLIC_ORIGIN": "https://" + hostname, + "SNOWGLOBE_FILE_ORIGIN": "https://files-" + hostname, "SNOWGLOBE_AUTH_RP_ID": hostname, + "SNOWGLOBE_AUTH_REQUIRED": "1", "SNOWGLOBE_PROXY_TOKEN_FILE": "", "SNOWGLOBE_DATA_DIR": "/data", + "SNOWGLOBE_PREVIEW": "1", "SNOWGLOBE_METRICS": "follow", + "SNOWGLOBE_LISTEN_ADDRESS": "127.0.0.1", "PORT": "${NOMAD_PORT_http}", + "SNOWGLOBE_NOMAD_TOKEN_FILE": "/run/secrets/nomad.token", "NOMAD_ADDR": "http://127.0.0.1:4646", + "SNOWGLOBE_HOST_SOCKET": "/run/snowglobe-host/host.sock", "SNOWGLOBE_NODE_NAME": site["nodeName"], + "SNOWGLOBE_STORE_ROOT": "/srv", "SNOWGLOBE_MEDIA_ROOT": "/srv/Media", + "SNOWGLOBE_FILES_WRITABLE": "1", "SNOWGLOBE_CLOVER_READ_ONLY": "false", "SNOWGLOBE_MEDIA_READ_ONLY": "false", + "HOME": "/data", "XDG_CACHE_HOME": "/data/cache", + } + try: + if created: + snowglobe.command("zfs", "create", "-o", "mountpoint=" + str(root), saved["clone"]) + saved["external"] = {"files": {"clone": saved["clone"] + "/files"}} + directory.mkdir(parents=True, exist_ok=True) + snowglobe.write_private(directory / f"{identity}.json", json.dumps(saved)) + snowglobe.command("zfs", "create", "-o", "mountpoint=" + str(root / "files"), saved["clone"] + "/files") + for path in (root / "data", root / "files", root / "files/Media"): + path.mkdir(parents=True, exist_ok=True) + os.chown(path, saved["uid"], saved["uid"]) + path.chmod(0o750) + fixture = root / "files/preview.txt" + fixture.write_text("Disposable Snow Globe dashboard preview.\n") + fixture.chmod(0o644) + directory.mkdir(parents=True, exist_ok=True) + saved.update(release=snowglobe.REPO.name, ready=False) + snowglobe.write_private(directory / f"{identity}.json", json.dumps(saved)) + if created: + policy = 'namespace "default" { capabilities = ["list-jobs", "read-job"] }\nnode { policy = "read" }\n' + snowglobe.command("nomad", "acl", "policy", "apply", identity, "-", input=policy) + token = json.loads(snowglobe.command("nomad", "acl", "token", "create", "-policy=" + identity, + "-name=" + identity, "-json", capture=True).stdout) + saved["tokenAccessor"] = token["AccessorID"] + snowglobe.write_private(directory / f"{identity}.json", json.dumps(saved)) + snowglobe.write_private(root / "nomad.token", token["SecretID"] + "\n") + os.chown(root / "nomad.token", saved["uid"], saved["uid"]) + imported = snowglobe.command("podman", "run", "--rm", "--network=none", "--read-only", + "--user=" + str(saved["uid"]), "--cap-drop=ALL", "--security-opt=no-new-privileges", + "--entrypoint=/bin/home-dashboard", + "--volume=" + str(root / "data") + ":/data:rw", + *["--env=" + key + "=" + value for key, value in env.items()], image, "--preview-account", capture=True) + snowglobe.write_private(root / "access.json", imported.stdout) + runtime = "/run/snowglobe-preview/" + identity + host_environment = snowglobe.transition.host_environment() + host_path = next(value for value in shlex.split(host_environment) if value.startswith("PATH=")) + snowglobe.transition.stop_preview(identity) + snowglobe.command("systemd-run", "--collect", "--quiet", "--unit=snowglobe-preview-" + identity, + "--property=Restart=on-failure", "--property=MemoryMax=256M", "--property=TasksMax=32", + "--setenv=" + host_path, + sys.executable, str(Path(__file__).resolve()), "--serve", identity) + for _ in range(100): + if Path(runtime, "host.sock").exists(): + break + time.sleep(.1) + else: + raise RuntimeError("dashboard preview host socket did not start") + job = {"ID": identity, "Name": identity, "Type": "service", "Datacenters": ["clover"], + "Meta": {"snowglobe_service": "dashboard", "snowglobe_name": "Snow Globe preview", "snowglobe_hostname": hostname}, + "TaskGroups": [{"Name": "app", "Count": 1, "Networks": [{"Mode": "host", "DynamicPorts": [{"Label": "http", "HostNetwork": "loopback"}]}], + "Services": [{"Name": identity, "Provider": "nomad", "PortLabel": "http", "TaskName": "app", "Tags": ["caddy-host=" + hostname], + "Checks": [{"Name": "Dashboard sign-in", "Type": "http", "Path": "/sign-in", "Interval": 10000000000, "Timeout": 5000000000}]}], + "Tasks": [{"Name": "app", "Driver": "podman", "User": str(saved["uid"]) + ":" + str(saved["uid"]), "Env": env, + "Config": {"image": image, "network_mode": "host", "cap_drop": ["ALL"], "security_opt": ["no-new-privileges"], + "volumes": [str(root / "data") + ":/data:rw", str(root / "files") + ":/srv:rw,rslave", runtime + ":/run/snowglobe-host:ro", + str(root / "nomad.token") + ":/run/secrets/nomad.token:ro"]}, + "Resources": {"CPU": 1000, "MemoryMB": 2048}}]}]} + snowglobe.command("nomad", "job", "run", "-json", "-", input=json.dumps(job)) + snowglobe.check_http(hostname, "/sign-in", site["tlsInternal"]) + saved["ready"] = True + snowglobe.write_private(directory / f"{identity}.json", json.dumps(saved)) + except BaseException: + if created and (directory / f"{identity}.json").exists(): + destroy(identity) + raise + print(f"stage={identity}\nrelease={saved['release']}\nhost={hostname}\ndataset={saved['clone']}\naccess={root / 'access.json'}") + + +def destroy(identity): + snowglobe.token() + saved = metadata(identity) + snowglobe.transition.stop_preview(identity) + if saved.get("tokenAccessor"): + revoked = subprocess.run(["nomad", "acl", "token", "delete", saved["tokenAccessor"]], capture_output=True, text=True) + if revoked.returncode and "400 (Cannot delete nonexistent tokens:" not in revoked.stderr: + raise RuntimeError(revoked.stderr.strip()) + snowglobe.command("nomad", "acl", "policy", "delete", identity, capture=True) + datasets = set(snowglobe.command("zfs", "list", "-H", "-o", "name", capture=True).stdout.splitlines()) + saved["external"] = {name: item for name, item in saved.get("external", {}).items() if item["clone"] in datasets} + if saved.get("external", {}).get("files", {}).get("clone") == saved["clone"] + "/files": + snapshots = snowglobe.command("zfs", "list", "-H", "-t", "snapshot", "-r", "-o", "name", saved["clone"] + "/files", capture=True).stdout.splitlines() + for snapshot in snapshots: + if snapshot.startswith(saved["clone"] + "/files@"): + snowglobe.command("zfs", "destroy", snapshot) + if saved["clone"] not in datasets: + del saved["clone"] + snowglobe.destroy_stage(identity, saved, {}) + shutil.rmtree(Path("/run/snowglobe-preview") / identity, ignore_errors=True) + + +if __name__ == "__main__": + if len(sys.argv) == 3 and sys.argv[1] == "--serve": + serve(sys.argv[2]) + else: + raise SystemExit("Expected --serve STAGE") diff --git a/tools/dashboard-unit-test.py b/tools/dashboard-unit-test.py deleted file mode 100644 index e78a8340b338d8e6c07d76208ca03f9446d93935..0000000000000000000000000000000000000000 --- a/tools/dashboard-unit-test.py +++ /dev/null @@ -1,303 +0,0 @@ -#!/usr/bin/env python3 -import argparse -from contextlib import ExitStack -import importlib -import json -import os -from pathlib import Path -import pwd -import re -import shutil -import socket -import subprocess -import sys -import tempfile -import time -import urllib.request -import uuid - -import router - - -def main(): - parser = argparse.ArgumentParser() - parser.add_argument("--output", type=Path) - parser.add_argument("--browser-ready-file", type=Path) - parser.add_argument("--browser-timeout", type=int, default=180) - parser.add_argument("--relay-agent-dir", type=Path) - args = parser.parse_args() - assert 1 <= args.browser_timeout <= 600 - repo = Path(__file__).resolve().parent.parent - nix = ["nix", "--extra-experimental-features", "nix-command flakes"] - base = "path:" + str(repo) + "#nixosConfigurations.vm.config" - - def shell(*argv, timeout=60): - result = subprocess.run(argv, capture_output=True, text=True, timeout=timeout) - if result.returncode: - raise AssertionError(result.stderr) - return result.stdout - - units = {name: shell(*nix, "eval", base + '.systemd.units."' + name + '.service".text', - "--offline", "--raw", "--option", "eval-cache", "false") - for name in ["studio-dashboard", "studio-host"]} - permissions = [rule for rule in json.loads(shell(*nix, "eval", base + ".systemd.tmpfiles.rules", - "--offline", "--json", "--option", "eval-cache", "false")) - if rule.startswith("A+ /srv/prod/ytdl/data ")] - assert len(permissions) == 1, permissions - scripts = re.findall(r"^Exec(?:Start|StartPre|StartPost)=(/nix/store/\S+)$", units["studio-dashboard"], re.M) - assert len(scripts) == 3, scripts - shell(*nix, "build", *[base + '.systemd.units."' + name + '.service".unit' - for name in units], "--offline", "--no-link", timeout=1200) - try: - account = pwd.getpwnam("studio-dashboard") - except KeyError: - shell("useradd", "--system", "--user-group", "--no-create-home", "studio-dashboard") - account = pwd.getpwnam("studio-dashboard") - assert account.pw_uid != 0 and account.pw_gid != 0 - app_unit = "studio-dashboard-unit-test" - host_unit = "studio-host-unit-test" - container = "studio-dashboard-unit-test" - unit_files = [Path("/run/systemd/system/" + name + ".service") for name in [app_unit, host_unit]] - assert not any(file.exists() for file in unit_files), unit_files - assert not json.loads(shell("podman", "ps", "--all", "--filter=name=^" + container + "$", "--format=json")) - proof = uuid.uuid4().hex + uuid.uuid4().hex - - with tempfile.TemporaryDirectory(prefix="studio-dashboard-unit-", dir="/run") as temporary, ExitStack() as fixtures: - root = Path(temporary) - library = root / "clover" - season = library / "Media/jellyfin/Indie Shows/Fixture/Season 1" - config = library / "Documents/Config/Youtube Downloader" - config.mkdir(parents=True) - season.mkdir(parents=True) - (config / "feed.yaml").write_text("shows: []\n") - (season / "S01E01 - Fixture.mp4").write_bytes(b"fixture") - (season / "S01E01 - Fixture.nfo").write_text("Unit fixture") - for directory in [library, *[p for p in library.rglob("*") if p.is_dir()]]: - os.chown(directory, 3000, 3000) - directory.chmod(0o2770) - data = root / "data" - data.mkdir(mode=0o700) - (data / "existing.json").write_text('{"fixture":true}\n') - state = root / "yt-state" - (state / "nested").mkdir(parents=True) - (state / "existing.json").write_text('{"fixture":true}\n') - for path in [state, *state.rglob("*")]: - os.chown(path, 3118, 3118) - path.chmod(0o750 if path.is_dir() else 0o640) - token = root / "proxy.token" - token.write_text(proof) - readonly = root / "nomad.token" - readonly.write_bytes(Path("/var/lib/studio/dashboard.token").read_bytes()) - ca = root / "ca.crt" - ca.write_bytes(Path("/var/lib/studio/ca-bundle.crt").read_bytes()) - ca.chmod(0o444) - acl = root / "permissions.conf" - acl.write_text(permissions[0].replace("/srv/prod/ytdl/data", str(state)) + "\n") - with socket.socket() as reservation: - reservation.bind(("127.0.0.1", 0)) - port = reservation.getsockname()[1] - sys.path.insert(0, str(repo / "service/keycloak")) - from api import Keycloak - keycloak = Keycloak("keycloak.studio.test", importlib.import_module("dashboard-run").secret("get", "keycloak", "password"), attempts=1) - shale_image = re.search(r'^\s*image = "([^"]+)"', (repo / "service/shale/service.pkl").read_text(), re.M).group(1) - shale_origin, shale_database, shale_container = fixtures.enter_context(importlib.import_module("dashboard-shale-test").instance( - keycloak, uuid.uuid4().hex, shale_image, None, False, host="shale.studio.test", dashboard_port=port)) - with socket.socket() as reservation: - for gateway_port in range(20000, 32001): - try: - reservation.bind(("0.0.0.0", gateway_port)) - break - except OSError: - continue - else: - raise AssertionError("no fixture gateway port available") - os.environ.update(STUDIO_DOMAIN="studio.test", STUDIO_INTERNAL_PORT=str(gateway_port), - STUDIO_DASHBOARD_PORT=str(port), STUDIO_PROXY_TOKEN_FILE=str(token)) - rendered = router.render(Path(router.TOKEN).read_text().strip()) - gateway = rendered[rendered.index(f"dashboard.internal.studio.test:{gateway_port} {{"):] - gateway_config = root / "Caddyfile" - gateway_config.write_text("{\n admin off\n auto_https disable_redirects\n skip_install_trust\n}\n" + gateway + - f"\nglobe.studio.test:{gateway_port} {{\n tls internal\n reverse_proxy 127.0.0.1:{port} {{\n header_up Host globe.studio.test\n }}\n}}\n") - gateway_config.chmod(0o600) - caddy_binary = (Path("/proc") / shell("systemctl", "show", "-P", "MainPID", "caddy").strip() / "exe").resolve(strict=True) - caddy = None - replacements = { - "https://dashboard.internal.studio.test:8448": f"https://dashboard.internal.studio.test:{gateway_port}", - "--name=studio-dashboard ": "--name=" + container + " ", - "--volume=/run/studio-host:/run/studio-host:ro": "--volume=/run/" + host_unit + ":/run/studio-host:ro", - "/var/lib/studio/dashboard-proxy.token": str(token), - "/var/lib/studio/dashboard.token": str(readonly), - "/var/lib/studio/ca-bundle.crt": str(ca), - "/var/lib/studio/dashboard": str(data), - "/srv/prod/ytdl/data": str(state), - "/srv/clover": str(library), - "--prefix=" + str(state): "--prefix=" + str(state) + " " + str(acl), - "--publish=127.0.0.1:7072:7072": f"--publish=127.0.0.1:{port}:7072", - "http://127.0.0.1:7072/": f"http://127.0.0.1:{port}/", - } - text = units["studio-dashboard"] - for source in scripts: - content = Path(source).read_text() - # Only host mount sources move; container paths retain the generated unit's contract. - for original, replacement in replacements.items(): - if original in ["/srv/prod/ytdl/data", "/srv/clover"]: - content = content.replace("test -d " + original, "test -d " + replacement) - content = content.replace("test -d '" + original, "test -d '" + replacement) - content = content.replace("src=" + original, "src=" + replacement) - content = content.replace("--volume=" + original + ":", "--volume=" + replacement + ":") - content = content.replace("--prefix=" + original, "--prefix=" + replacement) - else: - content = content.replace(original, replacement) - destination = root / Path(source).name - destination.write_text(content) - destination.chmod(0o700) - text = text.replace(source, str(destination)) - text = text.replace("https://dashboard.internal.studio.test:8448", f"https://dashboard.internal.studio.test:{gateway_port}") - text = text.replace('Environment="STUDIO_SHALE_URL=https://shale.studio.test"', 'Environment="STUDIO_SHALE_URL=' + shale_origin + '"') - text = re.sub(r'^Environment="STUDIO_INDEX_POOL=.*"\n', "", text, flags=re.M) - text = text.replace("studio-host.service", host_unit + ".service").replace("studio-router.service", "") - text = text.replace("--time=8 studio-dashboard", "--time=8 " + container).replace("--force studio-dashboard", "--force " + container) - for original in ["/var/lib/studio/dashboard.token", "/var/lib/studio/ca-bundle.crt"]: - text = text.replace("ConditionPathExists=" + original, "ConditionPathExists=" + replacements[original]) - unit_files[0].write_text(text) - host_text = units["studio-host"].replace("RuntimeDirectory=studio-host", "RuntimeDirectory=" + host_unit).replace("StateDirectory=studio/host", "StateDirectory=studio/host-unit-test") - host_text = host_text.replace("[Service]\n", "[Service]\n" + f'Environment="STUDIO_HOST_SOCKET=/run/{host_unit}/host.sock"\nEnvironment="STUDIO_HOST_STATE_ROOT=/var/lib/studio/host-unit-test"\n') - unit_files[1].write_text(host_text) - try: - with (root / "caddy.log").open("w") as log: - caddy = subprocess.Popen([str(caddy_binary), "run", "--config", str(gateway_config), "--adapter", "caddyfile"], - stdout=log, stderr=log, env={**os.environ, "XDG_DATA_HOME": str(root), "XDG_CONFIG_HOME": str(root)}) - gateway_ca = root / "caddy/pki/authorities/local/root.crt" - deadline = time.monotonic() + 15 - while not gateway_ca.exists(): - if caddy.poll() is not None or time.monotonic() > deadline: - raise AssertionError("fixture gateway did not start") - time.sleep(.1) - with ca.open("ab") as bundle: - bundle.write(gateway_ca.read_bytes()) - shell("systemctl", "daemon-reload") - shell("systemctl", "start", app_unit, timeout=180) - info = json.loads(shell("podman", "inspect", container))[0] - assert info["Config"]["User"] == f"{account.pw_uid}:{account.pw_gid}", info["Config"]["User"] - host = info["HostConfig"] - assert host["ReadonlyRootfs"] and not host["Privileged"] and not host["Devices"] - assert host["NetworkMode"] != "host" and host["PidsLimit"] == 256 - assert host["Memory"] == 2 * 1024 ** 3 and host["NanoCpus"] == 4 * 10 ** 9 - assert "no-new-privileges" in host["SecurityOpt"] - mounts = {mount["Destination"]: mount for mount in info["Mounts"]} - sources = {"/data": data, "/run/studio-host": Path("/run/" + host_unit), - "/run/secrets/dashboard-proxy.token": token, "/run/secrets/dashboard-nomad.token": readonly, - "/run/secrets/ca-bundle.crt": ca, "/srv/clover": library, - "/srv/clover/Media": library / "Media", "/srv/prod/ytdl/data": state} - assert mounts.keys() == sources.keys(), mounts.keys() - for target, source in sources.items(): - assert mounts[target]["Source"] == str(source), mounts[target] - assert mounts[target]["RW"] == (target in ["/data", "/srv/clover", "/srv/prod/ytdl/data"]), mounts[target] - for namespace in ["net", "pid", "mnt"]: - assert Path(f"/proc/{info['State']['Pid']}/ns/{namespace}").stat().st_ino != Path(f"/proc/self/ns/{namespace}").stat().st_ino - status = shell("podman", "exec", container, "/bin/cat", "/proc/1/status") - assert f"Uid:\t{account.pw_uid}\t{account.pw_uid}\t{account.pw_uid}\t{account.pw_uid}" in status - assert all(field + ":\t0000000000000000" in status for field in ["CapEff", "CapPrm", "CapBnd", "CapAmb"]) - for denied in ["/var/lib/studio/nomad.token", "/run/podman/podman.sock", "/run/libvirt/libvirt-sock", "/dev/zfs", "/opt/studio/current", "/srv/vm", "/srv/prod/keycloak"]: - result = subprocess.run(["podman", "exec", container, "/bin/test", "-e", denied], capture_output=True) - assert result.returncode != 0, denied - assert data.stat().st_uid == account.pw_uid and (data / "existing.json").stat().st_uid == account.pw_uid - python = next(value.split("=", 1)[1] for value in info["Config"]["Env"] if value.startswith("STUDIO_YT_PYTHON=")) - shell("podman", "exec", container, python, "-c", "import ssl; assert ssl.create_default_context().cert_store_stats()['x509_ca'] > 0") - refused = "import socket; s=socket.socket(socket.AF_UNIX); s.connect('/run/studio-host/host.sock'); " + "\ntry:\n s.sendall(b'{\"operation\":\"host.sample\"}\\n'); assert s.recv(4) == b''\nexcept (BrokenPipeError, ConnectionResetError):\n pass\n" - shell("podman", "exec", f"--user=65534:{account.pw_gid}", container, python, "-c", refused) - shell("podman", "exec", container, python, "-c", "from pathlib import Path; p=Path('/srv/prod/ytdl/data'); (p/'existing.json').write_text('fixture'); (p/'nested/new.json').write_text('fixture')") - assert (state / "existing.json").stat().st_uid == 3118 and state.stat().st_uid == 3118 - assert (state / "nested/new.json").exists() - shell("podman", "exec", "--user=3118:3000", container, python, "-c", "from pathlib import Path; p=Path('/srv/prod/ytdl/data'); (p/'nested/new.json').open('a').write('service'); (p/'nested/service.json').write_text('service')") - shell("podman", "exec", container, python, "-c", "from pathlib import Path; p=Path('/srv/prod/ytdl/data/nested/service.json'); assert p.read_text() == 'service'; p.open('a').write('dashboard')") - assert subprocess.run(["podman", "exec", container, "/bin/touch", "/srv/clover/Media/escaped"], capture_output=True).returncode != 0 - - def get(path): - request = urllib.request.Request(f"http://127.0.0.1:{port}" + path, headers={ - "Studio-Proxy-Token": proof, "User-Name": "fixture", "User-Groups": "infra-admin", - }) - with urllib.request.urlopen(request, timeout=30) as response: - return json.load(response) - - machine = get("/api/host") - assert machine["cores"] > 0 and machine["memory"] > 2 * 1024 ** 3 - assert get("/api/media/list?path=jellyfin/Indie%20Shows/Fixture/Season%201")["entries"] - assert get("/api/youtube/library") - assert get("/api/youtube") - iam_output = root / "iam-checks.json" - shell(sys.executable, str(repo / "tools/dashboard-iam-test.py"), "--url", f"http://127.0.0.1:{port}", - "--socket", f"/run/{host_unit}/host.sock", "--proof-file", str(token), - "--output", str(iam_output), timeout=180) - iam_checks = json.loads(iam_output.read_text()) - mcp_output = root / "mcp-checks.json" - shell(sys.executable, str(repo / "tools/dashboard-mcp-test.py"), "--url", f"http://127.0.0.1:{port}", - "--proof-file", str(token), "--restart-unit", app_unit, - "--output", str(mcp_output), timeout=240) - mcp_checks = json.loads(mcp_output.read_text()) - relay_output = root / "relay-checks.json" - shell(sys.executable, str(repo / "tools/dashboard-relay-test.py"), "--url", f"http://127.0.0.1:{port}", - "--proof-file", str(token), "--data-dir", str(data), "--restart-unit", app_unit, - "--output", str(relay_output), *(["--agent-dir", str(args.relay_agent_dir), "--agent-origin", f"https://globe.studio.test:{gateway_port}", "--agent-ca", str(gateway_ca)] if args.relay_agent_dir else []), timeout=240) - relay_checks = json.loads(relay_output.read_text()) - shale_output = root / "shale-link-checks.json" - shell(sys.executable, str(repo / "tools/dashboard-shale-link-test.py"), "--url", f"http://127.0.0.1:{port}", - "--proof-file", str(token), "--data-dir", str(data), "--restart-unit", app_unit, - "--shale-origin", shale_origin, "--shale-database", str(shale_database), "--shale-container", shale_container, - "--output", str(shale_output), timeout=240) - shale_checks = json.loads(shale_output.read_text()) - worker = "from pathlib import Path; found=[]\nfor p in Path('/proc').iterdir():\n if p.name.isdigit():\n try:\n argv=(p/'cmdline').read_bytes().rstrip(b'\\0').split(b'\\0')\n if argv[1:] == [b'server/youtube-worker.py']:\n found.append((p/'status').read_text())\n except (FileNotFoundError, PermissionError):\n pass\nassert len(found) == 1, len(found)\n" + f"assert 'Uid:\\t{account.pw_uid}\\t{account.pw_uid}\\t{account.pw_uid}\\t{account.pw_uid}' in found[0]\nassert 'CapEff:\\t0000000000000000' in found[0] and 'NoNewPrivs:\\t1' in found[0]\n" - shell("podman", "exec", container, python, "-c", worker) - socket_inode = Path("/run/" + host_unit).stat().st_ino - shell("systemctl", "restart", host_unit) - assert Path("/run/" + host_unit).stat().st_ino == socket_inode - time.sleep(1.1) - assert get("/api/host")["memory"] == machine["memory"] - started = time.monotonic() - shell("systemctl", "restart", app_unit, timeout=180) - restart_seconds = time.monotonic() - started - assert restart_seconds < 15, restart_seconds - assert get("/api/me")["name"] == "fixture" and (data / "existing.json").exists() - browser_checks = None - if args.browser_ready_file: - browser_checks = json.loads(shell(sys.executable, str(repo / "tools/dashboard-browser-test.py"), - "--ready-file", str(args.browser_ready_file), "--timeout", str(args.browser_timeout), timeout=args.browser_timeout + 60)) - started = time.monotonic() - shell("systemctl", "stop", app_unit, timeout=30) - elapsed = time.monotonic() - started - assert elapsed < 15 and shell("systemctl", "show", "-P", "Result", app_unit).strip() == "success" - result = {"image": info["ImageName"], "generated_nixos_unit": True, "dedicated_uid": account.pw_uid, "readonly_rootfs": True, - "zero_capabilities": True, "quotas": True, "private_network": True, "bounded_mounts": True, - "host_broker_uid_auth": True, "other_uid_same_group_refused": True, - "state_ownership_migration": True, "shared_youtube_acl": True, - "youtube_owner_preserved": True, "readonly_media": True, "unprivileged_youtube_worker": True, - "python_public_tls_roots": True, - "iam": iam_checks, "mcp": mcp_checks, "relay": relay_checks, "shale_link": shale_checks, - "broker_restart_socket_preserved": True, "container_restart_state_preserved": True, - "cached_image_restart_seconds": round(restart_seconds, 2), - "systemd_stop_seconds": round(elapsed, 2)} - if browser_checks is not None: - result["browser_route_fixture"] = browser_checks - if args.output: - args.output.parent.mkdir(parents=True, exist_ok=True) - args.output.write_text(json.dumps(result, indent=2) + "\n") - print(json.dumps(result)) - finally: - subprocess.run(["systemctl", "stop", app_unit, host_unit], capture_output=True, timeout=60) - subprocess.run(["podman", "rm", "--ignore", "--force", container], capture_output=True, timeout=15) - if caddy is not None: - caddy.terminate() - try: - caddy.wait(timeout=10) - except subprocess.TimeoutExpired: - caddy.kill() - caddy.wait(timeout=5) - for file in unit_files: - file.unlink(missing_ok=True) - for directory in ["/var/lib/studio/host-unit-test", "/run/" + host_unit]: - shutil.rmtree(directory, ignore_errors=True) - shell("systemctl", "daemon-reload") - - -if __name__ == "__main__": - main() diff --git a/tools/dashboard-vm-boundary-test.py b/tools/dashboard-vm-boundary-test.py index b9596be5eef989e3c69d07d73919578d6eecfe54..e796141831acd73c4aca74c8607c53ca029e5fea 100644 --- a/tools/dashboard-vm-boundary-test.py +++ b/tools/dashboard-vm-boundary-test.py @@ -116,7 +116,7 @@ raise AssertionError('interrupted creation reported success') attempt = subprocess.run([sys.executable, "-c", interrupted, str(Path(__file__).parent), phase], input=json.dumps({**spec, "image": installer.name if phase == "copy" else "blank"}), capture_output=True, text=True, timeout=60, - env={**os.environ, "STUDIO_VM_IMAGES_ROOT": str(images)}) + env={**os.environ, "SNOWGLOBE_VM_IMAGES_ROOT": str(images)}) elapsed = time.monotonic() - started assert attempt.returncode == 1, attempt.stderr if phase == "copy": diff --git a/tools/data-restore.md b/tools/data-restore.md index a9252f48b486e9b92aae4878e55cc056dd46d6f2..5a14dc9b40d15f601db16e30a002e785dea78745 100644 --- a/tools/data-restore.md +++ b/tools/data-restore.md @@ -1,12 +1,21 @@ -# Production data rollback +# Production data restore -Every promotion and code rollback backs up the active release before switching it. One ZFS snapshot operation captures the mounted service datasets and PostgreSQL data at the same instant. A temporary clone of that PostgreSQL snapshot runs without a network and produces the service-owned database dumps, including services with multiple database inputs. The clone is destroyed before the backup is published. `python3 tools/deploy.py backups` lists the backup IDs and services. A failed promotion can leave a completed backup even when release history has no new entry. +Every production promotion and code rollback backs up the active release +before switching it. One ZFS snapshot operation captures mounted managed +service datasets and PostgreSQL data together. A temporary isolated PostgreSQL +clone produces service-owned database dumps and is destroyed before the +backup is published. A failed deployment can leave a completed backup without +a successful release-history entry. -The backup now also saves a private `nomad.snap` and its SHA-256 in the manifest. A disposable backup run against the VM's live Nomad server verified the file and digest, then removed it. This snapshot includes jobs, ACLs, and secret variables; keep the backup directory private. Service data restore does not apply the whole Nomad snapshot, which is reserved for an explicit cluster recovery. The backup directory is still on the OS disk until Snow Globe control state has durable storage. +`python3 tools/deploy.py backups` lists backup IDs and services. Each backup's +private manifest records the service snapshots, database dumps and checksums, +and a private `nomad.snap`. The Nomad snapshot includes jobs, ACLs, and secret +variables. It is for explicit cluster recovery; service restore does not apply +it. Earlier backups may lack this snapshot. Manifests and dumps remain on the +OS disk; see [hardware and recovery](snowglobe-hardware.md) for off-server +recovery requirements. -At inspection, the VM retained 25 release backups: their PostgreSQL dumps and manifests used `275 MB`, and their service snapshots numbered 550. No automatic retention policy deletes these backups yet; choose one before long-running production use. Earlier backup manifests do not contain `nomad.snap`. - -To restore one service, roll its code back first, then run: +To restore one service after production data restore is explicitly authorized: ```sh python3 tools/deploy.py backups @@ -14,14 +23,21 @@ python3 tools/deploy.py rollback python3 tools/deploy.py data-restore --backup --discard-writes ``` -The backup's `fromRelease` must match the active release. Data restore stops that service, saves its current dataset and database as a safety copy, restores the chosen ZFS snapshot and PostgreSQL dump, then deploys and checks the service. **Changes written after the chosen backup are discarded from the live service.** The safety snapshot and dump remain on the host. Postgres itself is shared; restore its owner services individually. Shared Clover and Media mounts are outside this service restore; restoring an entire shared dataset would discard unrelated users' writes. +The backup's `fromRelease` must match the active release. The command verifies +backup snapshots and dumps before stopping the service, saves the current +dataset and database as a safety copy, restores the selected data, then +deploys and checks the service. **Writes after the chosen backup are discarded +from the live service.** The safety snapshot and dump remain on the host. -The CLI checks that the backup snapshot and database dumps exist before stopping the service. A disposable missing-snapshot manifest on the VM was rejected while Redis Insight stayed running; the fixture was removed. +Restore PostgreSQL owner services individually; the PostgreSQL service itself +is shared. Clover, Media, and external archive mounts are outside managed +service restore. Restoring an entire shared dataset discards unrelated writes. +For Shale, keep the complete service dataset together: repository visibility, +repository objects, issue database, attachment files, and their manifest must +agree. For Copyparty, also preserve its Nomad API-account hashes as described +in [administration](copyparty.md). For Knife, preserve the external archive +store and identity as described in [operations](knife.md). -On the VM, backup `20260927T030206Z-67f38b` captured 22 service snapshots and four database dumps from a cloned PostgreSQL snapshot. All dump hashes and snapshots verified; the temporary container and dataset were removed while the production PostgreSQL allocation stayed running. An injected dump failure after clone startup left the ZFS dataset/snapshot list and backup directory list unchanged, with no probe container. Earlier rollback and restore tests are recorded in the release history. - -# VM restore rehearsal - -On 2026-09-27, backup `20260927T033539Z-123cab` captured the running x86 VM's service datasets and PostgreSQL databases. A throwaway file and table were then added to `evil-hedgedoc`. Running `data.py restore 20260927T033539Z-123cab evil-hedgedoc --discard-writes` stopped the job, saved the pre-restore safety snapshot, restored the dataset and database, and redeployed it. Both probes disappeared, `https://md.evil.studio.test/_health` returned 200 from the Mac, and all 19 VM routes passed their health checks. This proves the explicit data restore path on the VM; the production NAS cutover remains separate. - -The `20260927T055034Z-27d895` backup retained all 22 service snapshots and four matching PostgreSQL dumps. Its Dawarich dump restored into a new database owned by `svc_dawarich` in an isolated PostgreSQL ZFS clone, with `pgcrypto` and `postgis` created first and `pg_restore --role=svc_dawarich`. The restored `points` table contained 22,666 rows. The probe container and clone were removed, the live PostgreSQL allocation stayed running, and Dawarich still returned HTTP 200. +Verify the real application, authentication, persistence, provider connections, +allocation health and logs after recovery. Code rollback only changes code; +it does not restore data or undo a folder move. diff --git a/tools/data.py b/tools/data.py index 0f5fdcc6ef6f6fc017f8b145fa7f0f56e0b4409e..4016bcc137debbef9ee6e320b998838da88730d3 100644 --- a/tools/data.py +++ b/tools/data.py @@ -4,6 +4,7 @@ from contextlib import closing, contextmanager from datetime import datetime, timezone import hashlib import json +from importlib import import_module import os from pathlib import Path import re @@ -14,8 +15,10 @@ import subprocess import time -ROOT = Path("/opt/studio") -STATE = Path("/var/lib/studio") +transition = import_module("namespace-transition") + +ROOT = Path("/opt/snowglobe") +STATE = Path("/var/lib/snowglobe") BACKUPS = STATE / "backups" BACKUP_ID = re.compile(r"\d{8}T\d{6}Z-[0-9a-f]{6}\Z") SERVICE_ID = re.compile(r"[a-z][a-z0-9-]*\Z") @@ -32,7 +35,7 @@ def current_release(): if not current.is_symlink(): raise ValueError("No active home server release") release = current.resolve() - if release.parent != ROOT / "releases" or not RELEASE_ID.fullmatch(release.name): + if release.parent != (ROOT / "releases").resolve() or not RELEASE_ID.fullmatch(release.name): raise ValueError("Active home server release is invalid") return release @@ -118,7 +121,7 @@ def cloned_postgres(snapshot, clone, mountpoint): image = inspect["Image"] if not re.fullmatch(r"[0-9a-f]{64}", image): raise ValueError("Postgres image ID is invalid") - probe = "studio-backup-" + clone.rsplit("/", 1)[1] + probe = "snowglobe-backup-" + clone.rsplit("/", 1)[1] created = False try: run("zfs", "clone", "-o", f"mountpoint={mountpoint}", snapshot, clone) @@ -175,7 +178,7 @@ def backup(from_release, to_release): directory = BACKUPS / backup_id manifest = {"id": backup_id, "fromRelease": from_release, "toRelease": to_release, "time": int(time.time()), "services": {}} - snapshot = "studio-" + backup_id + snapshot = "snowglobe-" + backup_id datasets = set() allocations = {} for service in sorted(names): @@ -262,7 +265,8 @@ def restore(backup_id, service): path = source / name if not path.resolve().is_relative_to(source.resolve()) or not path.is_file() or checksum(path) != digest: raise ValueError("Dashboard backup is missing or changed") - run("systemctl", "stop", "studio-dashboard") + unit = transition.layout(current_release())["prefix"] + "-dashboard" + run("systemctl", "stop", unit) safety = BACKUPS / ("before-restore-" + datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ") + "-" + secrets.token_hex(3)) safety.mkdir(mode=0o700) files = backup_dashboard(safety / "dashboard") @@ -273,8 +277,8 @@ def restore(backup_id, service): if (root / "pictures").exists(): shutil.rmtree(root / "pictures") shutil.copytree(source, root, dirs_exist_ok=True) - run("systemctl", "start", "studio-dashboard") - run("systemctl", "is-active", "--quiet", "studio-dashboard") + run("systemctl", "start", unit) + run("systemctl", "is-active", "--quiet", unit) print(f"restored=dashboard backup={backup_id} safety={safety.name}") return dataset = entry.get("dataset") @@ -319,9 +323,9 @@ def restore(backup_id, service): with (directory / database["dump"]).open("rb") as input_file: run(*PODMAN, "exec", "-i", container, "pg_restore", "-U", "postgres", "-d", name, "--no-owner", "--no-acl", "--role=" + owner, stdin=input_file) - script = current_release() / "tools/studio.py" - run("python3", str(script), "deploy", service) - run("python3", str(script), "check", service) + script = transition.layout(current_release())["script"] + run("python3", str(script), "deploy", service, env=transition.environment(current_release())) + run("python3", str(script), "check", service, env=transition.environment(current_release())) print(f"restored={service} backup={backup_id} safety={safety}") diff --git a/tools/dawarich-migration.md b/tools/dawarich-migration.md deleted file mode 100644 index 3f814ab10c66d25db8d61ef7e20268d4f8a66f1a..0000000000000000000000000000000000000000 --- a/tools/dawarich-migration.md +++ /dev/null @@ -1,31 +0,0 @@ -# Dawarich migration - -The 2026-10-04 offline handoff includes Dawarich's PostgreSQL 18 database, with a verified restore and hashes for every table, sequence, and large object. Restore it before starting Dawarich. Keep the original Rails `SECRET_KEY_BASE` and copy `/var/app/storage` into the managed service dataset before the first application boot. - -The production database import on 2026-10-04 passed all 63 exported relation and large-object fingerprints before any Dawarich process started. Evidence is `/var/lib/studio/dawarich-import-7hw6v9w1/verified.json`. PostGIS retains its 8,500 built-in spatial references; PostgreSQL extension dumps contain only custom reference rows. Removing the built-ins changes geometry JSON serialization as well as breaking reference lookup. The importer preserves built-ins and replaces only explicitly dumped custom SRIDs. - -[import-personal-postgres.py](import-personal-postgres.py) reads the allocated production database from `nomad/jobs/dawarich/inputs/database`. The target database and `svc_dawarich` role must already exist. For initial production migration, keep both Nomad jobs stopped and use an isolated PostGIS initializer with `--network none`, mounting the final PostgreSQL data directory: - -```sh -python3 tools/import-personal-postgres.py dawarich \ - /mnt/storage1/apps/studio-handoff/20261004T230507Z-e75419 \ - --container PERSONAL_POSTGRES_INITIALIZER -``` - -The importer saves a private target backup under `/var/lib/studio/dawarich-import-*`, restores application objects as the allocated owner, preserves PostGIS reference data, and compares the result with the export's content hashes. It leaves Dawarich stopped. Stop and remove the initializer before deploying the production Postgres job. For a later import into an already running production Postgres allocation, omit `--container`; Dawarich must still be stopped. - -The historical preview results below concern the rehearsal VM. Zenith's separate Redis dump was 86,557 bytes and belonged to Dawarich. DB 0 contained caches and track-generation keys; DB 1 held 106 queued `Family::Invitations::CleanupJob` entries. The old image's nightly schedule sent them to `family`, while its worker listened to `families`; that queue reached 107 entries on 2026-09-27. The pinned image schedules this job on `families`, and all 14 of its scheduled queues appear in the worker's queue list. - -The rehearsal's source and target used PostgreSQL 18 with PostGIS. The tested transfer used a consistent custom-format dump made with `pg_dump -Fc --no-owner --no-acl --exclude-extension=postgis`, then restored it into a **separate stage database** after creating the PostGIS extension there. It did not write to Zenith or the VM's production database. - -The Rails `SECRET_KEY_BASE` must stay the same as the source value so encrypted records remain readable. The old `/var/app/storage` contained one file; its checksum matched the staged copy. The generated `/var/app/public` assets and `/var/app/tmp` cache were not imported. The stage used its own Keycloak client and database; its two imported user records remained intact. - -Before the new image ran, the source and staged database both had 22,666 points, 1,011 tracks, 2 users, and 74 visits. The current image rebuilds derived tracks and visits on startup; after a worker restart, the staged database had 22,666 points, 698 tracks, 2 users, 139 visits, and 74 newly generated track segments. Track count is therefore not a row-for-row import check. Point and user counts, storage checksums, HTTPS health, and the SSO login flow should be checked separately. - -The first staged boot exposed a startup race: Sidekiq loaded the old `TrackSegment` schema before the web entrypoint finished migrations, then logged `unknown attribute 'start_at'`. Restarting the worker after migrations restored segment creation. The service now runs Redis as a prestart sidecar and completes a migration task before starting web and worker. Its web entrypoint still repeats the upstream migration command, which is idempotent but extends cold startup on the emulated x86 VM. - -The corrected deployment is `dawarich-preview-1e4dac4a` at `https://dawarich-preview-1e4dac4a.studio.test` (release `554d23a23d940690`). Nomad marked it healthy; the HTTPS health endpoint returned 200 with certificate verification, and the new worker allocation logged no `start_at` errors. - -[import-dawarich.sh](import-dawarich.sh) repeats the preview restore from the latest read-only Zenith app snapshot and copies the small storage directory. It compares the source and target `SECRET_KEY_BASE` by hash, backs up the target ZFS dataset and database, restores the source PostgreSQL dump with PostGIS, and checks point/user counts before starting the preview. The 2026-09-26 run completed successfully: 22,666 points and two users survived migrations, HTTPS health returned 200 from this Mac with certificate verification, Nomad marked the job healthy, and the worker had no `start_at` error. - -The shell importer requires the old running source and serves VM previews. The Python importer uses the retained offline handoff for the same-machine production cutover. diff --git a/tools/ddns-migration.md b/tools/ddns-migration.md deleted file mode 100644 index e665bb4c0bf8849d62b660e081a6bcb5914a50f1..0000000000000000000000000000000000000000 --- a/tools/ddns-migration.md +++ /dev/null @@ -1,9 +0,0 @@ -# DDNS cutover - -The `.test` VM does not run DDNS. Evaluating `service/ddns-updater/service.pkl` on the VM with `domain=paperclover.net` produced valid JSON for the Cloudflare `*.paperclover.net` record and required `cloudflare_zone_id` and `cloudflare_api_token`. Zenith's legacy `.env` contains the corresponding `CLOUDFLARE_ZONE_ID` and `CLOUDFLARE_API_TOKEN` keys. No public DNS record was changed during this check. - -The current worktree's `config/`, `service/`, and `tools/` generated 27 production-domain jobs, all accepted by `nomad job validate` on the VM. This was validation only; it did not submit jobs or contact Cloudflare. - -Snow Globe replaces a service hostname's first label with its stage ID: Shale becomes `shale-preview-12345678.paperclover.net`, while evil.inc Forgejo becomes `evil-forgejo-preview-12345678.evil.paperclover.net`. Cloudflare's wildcard DNS records [cover multiple levels when no specific record takes precedence](https://developers.cloudflare.com/dns/manage-dns-records/reference/wildcard-dns-records/). Read-only DNS queries resolved both a sample Shale stage name and an otherwise nonexistent name under `evil.paperclover.net` to the current public address. The README's `.staging.paperclover.net` example describes a different hostname scheme from the current CLI. No public record changed; ACME issuance still needs a live cutover check. - -At production cutover, set `STUDIO_DEPLOY_HOST` and `STUDIO_DEPLOY_PORT` for the new host, then run `bash tools/import-legacy-secrets.sh ddns-updater CLOUDFLARE_ZONE_ID CLOUDFLARE_API_TOKEN`. The order matches the required secret fields in the service definition. Start the Snow Globe DDNS job only after the old updater is stopped; otherwise both updaters can write the wildcard record. Verify the provider status and the public record after the new job starts. diff --git a/tools/deploy-test.py b/tools/deploy-test.py index 5fd561735473ea8beb51f84c32c3d727c123b1bd..bb7c1f434c2f1143d73437a77fd5e65fffc1a51a 100644 --- a/tools/deploy-test.py +++ b/tools/deploy-test.py @@ -15,7 +15,7 @@ import release class MainExport(unittest.TestCase): def test_production_exports_main_while_staging_exports_working_files(self): - with tempfile.TemporaryDirectory(prefix="studio-main-export-") as temporary: + with tempfile.TemporaryDirectory(prefix="snowglobe-main-export-") as temporary: root = Path(temporary) repo = root / "repo" repo.mkdir() @@ -28,11 +28,12 @@ class MainExport(unittest.TestCase): else: path.mkdir() (path / "fixture").write_text("committed") - (repo / "config/excluded-services.json").write_text('["retired"]\n') - (repo / "service/retired").mkdir() - (repo / "service/retired/private").write_text("must not be exported") + (repo / "service/app").mkdir() + (repo / "service/app/service.pkl").write_text("service definition") + (repo / "service/app/worker.pkl").write_text("grouped definition") executable = repo / "tools/fixture" executable.chmod(0o755) + subprocess.run(["jj", "file", "track", "--include-ignored", "AGENTS.md"], cwd=repo, check=True, capture_output=True) for args in [["describe", "-m", "Test main"], ["bookmark", "set", "main"]]: subprocess.run(["jj", *args], cwd=repo, check=True, capture_output=True) commit = subprocess.run(["jj", "--ignore-working-copy", "log", "-r", "main", "--no-graph", "-T", "commit_id"], cwd=repo, check=True, capture_output=True, text=True).stdout @@ -42,8 +43,6 @@ class MainExport(unittest.TestCase): original = subprocess.run def transport(argv, **kwargs): - if argv[0] == "jj" and "show" in argv and argv[-1].startswith("service/retired/"): - raise AssertionError("read excluded source") if argv[0] == "ssh": return subprocess.CompletedProcess(argv, 1) if argv[0] == "rsync" and "-e" in argv: @@ -64,13 +63,13 @@ class MainExport(unittest.TestCase): self.assertEqual((main / "service/fixture").read_text(), "committed") self.assertFalse((main / "service/new").exists()) self.assertTrue(os.access(main / "tools/fixture", os.X_OK)) - self.assertEqual(json.loads((main / ".studio-release.json").read_text())["main"], {"commit": commit, "description": "Test main\n"}) + self.assertEqual(json.loads((main / ".snowglobe-release.json").read_text())["main"], {"commit": commit, "description": "Test main\n"}) self.assertEqual((stage / "service/fixture").read_text(), "unfinished") self.assertEqual((stage / "service/new").read_text(), "new working file") - self.assertNotIn("main", json.loads((stage / ".studio-release.json").read_text())) - self.assertFalse((main / "service/retired").exists()) - self.assertFalse((stage / "service/retired").exists()) - self.assertFalse(any("retired" in relative.parts for _, relative in release.files(repo))) + self.assertNotIn("main", json.loads((stage / ".snowglobe-release.json").read_text())) + for exported in (main, stage): + self.assertEqual((exported / "service/app/service.pkl").read_text(), "service definition") + self.assertEqual((exported / "service/app/worker.pkl").read_text(), "grouped definition") subprocess.run(["jj", "bookmark", "set", "main"], cwd=repo, check=True, capture_output=True) with self.assertRaisesRegex(ValueError, "commit description"): deploy.upload(main=True) @@ -80,7 +79,7 @@ class BuildSourceExport(unittest.TestCase): def setUp(self): stack = ExitStack() self.addCleanup(stack.close) - root = Path(stack.enter_context(tempfile.TemporaryDirectory(prefix="studio-build-source-"))).resolve() + root = Path(stack.enter_context(tempfile.TemporaryDirectory(prefix="snowglobe-build-source-"))).resolve() self.repo = root / "repo" self.source = root / "source" self.remote = root / "remote" @@ -93,7 +92,6 @@ class BuildSourceExport(unittest.TestCase): else: path.mkdir() (path / "fixture").write_text("fixture") - (self.repo / "config/excluded-services.json").write_text("[]\n") (self.source / "lib").mkdir() (self.source / "lib/committed").write_text("committed library") (self.source / "src").mkdir() @@ -113,6 +111,7 @@ class BuildSourceExport(unittest.TestCase): self.spec = {"source": "../source", "revision": self.commit, "dockerfile": "src/app.dockerfile", "include": ["run.sh", "lib"]} self.manifest.write_text(json.dumps(self.spec)) + subprocess.run(["jj", "file", "track", "--include-ignored", "AGENTS.md"], cwd=self.repo, check=True, capture_output=True) for args in [["describe", "-m", "Pinned build source"], ["bookmark", "set", "main"]]: subprocess.run(["jj", *args], cwd=self.repo, check=True, capture_output=True) original = subprocess.run diff --git a/tools/deploy.py b/tools/deploy.py index a9df7cea0675617fb905ad6d7d9ba01a763e7ee7..e2503f1631644297fa2f745039b3151818263c0f 100644 --- a/tools/deploy.py +++ b/tools/deploy.py @@ -12,13 +12,13 @@ import sys import tempfile import time -from release import SOURCES, excluded_services, tree_digest +from release import SOURCES, tree_digest REPO = Path(__file__).resolve().parent.parent -HOST = os.environ.get("STUDIO_DEPLOY_HOST", "root@127.0.0.1") -PORT = os.environ.get("STUDIO_DEPLOY_PORT", "2222") -REMOTE = Path("/opt/studio") +HOST = os.environ.get("SNOWGLOBE_DEPLOY_HOST", "root@10.0.0.1") +PORT = os.environ.get("SNOWGLOBE_DEPLOY_PORT", "22") +REMOTE = Path("/opt/snowglobe") NAME = re.compile(r"[a-z][a-z0-9-]*\Z") RELEASE_ID = re.compile(r"[0-9a-f]{16}\Z") @@ -32,18 +32,27 @@ def ssh(command, capture=False): def run_logged(mode, target, command): run = f"{int(time.time() * 1000)}-{mode}-{target or 'current'}" - path = f"/var/lib/studio/runs/{run}.log" - script = f"umask 077; mkdir -p /var/lib/studio/runs; set -o pipefail; {command} 2>&1 | tee {shlex.quote(path)}" + path = f"/var/lib/snowglobe/runs/{run}.log" + script = f"umask 077; mkdir -p /var/lib/snowglobe/runs; set -o pipefail; {command} 2>&1 | tee {shlex.quote(path)}" ssh("bash -c " + shlex.quote(script)) print(f"run={run}") +def prepare_remote(): + subprocess.run( + ["ssh", "-p", PORT, "-o", "BatchMode=yes", HOST, "python3 - prepare"], + input="NAMESPACE = " + repr(json.loads((REPO / "dashboard/src/namespace-transition.json").read_text())) + "\n" + + (REPO / "tools/namespace-transition.py").read_text(), text=True, check=True, + ) + + def sync_manager(release): source = REMOTE / "releases" / release / "tools" - ssh(f"cp {source}/release.py {source}/data.py {REMOTE}/") + ssh(f"cp {source}/release.py {source}/data.py {source}/namespace-transition.py {REMOTE}/") + ssh(f"cp {source}/../dashboard/src/namespace-transition.json {REMOTE}/namespace-transition.json") -def export_revision(source, commit, paths, destination, excluded=frozenset()): +def export_revision(source, commit, paths, destination): entries = subprocess.run( ["jj", "--ignore-working-copy", "file", "list", "-r", commit, "-T", '\"[\" ++ json(path) ++ \",\" ++ json(file_type) ++ \",\" ++ json(executable) ++ \"]\\n\"', @@ -53,11 +62,6 @@ def export_revision(source, commit, paths, destination, excluded=frozenset()): for entry in entries: name, kind, executable = json.loads(entry) relative = Path(name) - if relative.parts[:1] == ("service",) and ( - set(relative.parts[1:]) & excluded or - relative.suffix == ".pkl" and relative.stem in excluded - ): - continue if kind != "file" or relative.is_absolute() or ".." in relative.parts: raise ValueError(f"unsupported revision file: {name}") target = destination / relative @@ -70,7 +74,6 @@ def export_revision(source, commit, paths, destination, excluded=frozenset()): def upload(main=False): - excluded = set() if main else excluded_services(REPO) with tempfile.TemporaryDirectory() as temporary: snapshot = Path(temporary) revision = None @@ -84,23 +87,13 @@ def upload(main=False): if conflicted or not description.strip(): raise ValueError("main needs a commit description and no conflicts before deployment") revision = {"commit": commit, "description": description} - # Production must use main's exclusions too. A working staging - # change must not remove dependencies from the committed release. - excluded = set(json.loads(subprocess.run( - ["jj", "--ignore-working-copy", "file", "show", "-r", commit, - "config/excluded-services.json"], - cwd=REPO, check=True, capture_output=True, text=True, - ).stdout)) - export_revision(REPO, commit, SOURCES, snapshot, excluded) + export_revision(REPO, commit, SOURCES, snapshot) else: subprocess.run( ["rsync", "-a", "--exclude=.DS_Store", "--exclude=__pycache__", "--exclude=*.pyc", "--exclude=.identities.lock", "--exclude=identities.pending", "--exclude=._*", "--exclude=dashboard/node_modules", "--exclude=dashboard/dist", "--exclude=dashboard/.cache", "--exclude=dashboard/data", "--exclude=dashboard/target", - *(f"--exclude=/service/{name}/" for name in sorted(excluded)), - *(f"--exclude=/service/*/{name}.pkl" for name in sorted(excluded)), - *(f"--exclude=/service/*/{name}/" for name in sorted(excluded)), *(str(REPO / path) for path in SOURCES), str(snapshot) + "/"], check=True, ) @@ -156,21 +149,21 @@ def upload(main=False): details = {"id": release, "digest": digest, "version": 2} if revision: details["main"] = revision - (snapshot / ".studio-release.json").write_text(json.dumps(details) + "\n") + (snapshot / ".snowglobe-release.json").write_text(json.dumps(details) + "\n") if subprocess.run( ["ssh", "-p", PORT, "-o", "BatchMode=yes", HOST, f"test -d {remote_release}"], check=False, ).returncode == 0: ssh(f"python3 {remote_release}/tools/release.py verify {release}") if revision: - script = "import pathlib,sys; p=pathlib.Path(" + repr(str(remote_release / ".studio-release.json")) + "); t=p.with_suffix('.pending'); t.write_text(sys.stdin.read()); t.chmod(0o444); t.replace(p)" + script = "import pathlib,sys; p=pathlib.Path(" + repr(str(remote_release / ".snowglobe-release.json")) + "); t=p.with_suffix('.pending'); t.write_text(sys.stdin.read()); t.chmod(0o444); t.replace(p)" subprocess.run(["ssh", "-p", PORT, "-o", "BatchMode=yes", HOST, "python3 -c " + shlex.quote(script)], input=json.dumps(details) + "\n", text=True, check=True) return release ssh(f"mkdir -p {remote_release}") subprocess.run( ["rsync", "-a", "-e", f"ssh -p {PORT} -o BatchMode=yes", - *(str(snapshot / path) for path in SOURCES), str(snapshot / ".studio-release.json"), + *(str(snapshot / path) for path in SOURCES), str(snapshot / ".snowglobe-release.json"), f"{HOST}:{remote_release}/"], check=True, ) ssh(f"chmod -R a-w {remote_release}") @@ -215,29 +208,30 @@ def main(): values = args.file.read_text() if not values.strip(): raise ValueError("secret input is empty") + prepare_remote() manager = f"python3 {REMOTE}/release.py" if args.mode == "stage": release = upload() overrides = "".join(f" --env {shlex.quote(value)}" for value in args.env) - run_logged("stage", args.target, f"python3 {REMOTE}/releases/{release}/tools/studio.py stage {args.target}{overrides}") + run_logged("stage", args.target, f"python3 {REMOTE}/releases/{release}/tools/snowglobe.py stage {args.target}{overrides}") elif args.mode == "secrets": release = upload() keys = "".join(f" --key {shlex.quote(key)}" for key in args.key) subprocess.run( ["ssh", "-p", PORT, "-o", "BatchMode=yes", HOST, - f"python3 {REMOTE}/releases/{release}/tools/studio.py secrets {args.target}{keys}"], + f"python3 {REMOTE}/releases/{release}/tools/snowglobe.py secrets {args.target}{keys}"], input=values, text=True, check=True, ) elif args.mode == "bootstrap": release = upload(main=True) sync_manager(release) ssh(f"{manager} publish {release}") - ssh(f"python3 {REMOTE}/releases/{release}/tools/studio.py pool") - ssh(f"python3 {REMOTE}/releases/{release}/tools/studio.py bootstrap") + ssh(f"python3 {REMOTE}/releases/{release}/tools/snowglobe.py pool") + ssh(f"python3 {REMOTE}/releases/{release}/tools/snowglobe.py bootstrap") ssh(f"python3 {REMOTE}/releases/{release}/tools/release.py bootstrap {release}") elif args.mode == "allocate": release = upload() - ssh(f"python3 {REMOTE}/releases/{release}/tools/studio.py allocate {args.target}") + ssh(f"python3 {REMOTE}/releases/{release}/tools/snowglobe.py allocate {args.target}") elif args.mode in {"publish", "prod"}: release = upload(main=True) sync_manager(release) @@ -255,11 +249,8 @@ def main(): release = upload() run_logged("data-restore", args.target, f"python3 {REMOTE}/releases/{release}/tools/data.py restore {args.backup} {args.target} --discard-writes") else: - metadata = json.loads(ssh(f"cat /var/lib/studio/stages/{args.target}.json", capture=True).stdout) - release = metadata.get("release") - if not isinstance(release, str) or not RELEASE_ID.fullmatch(release): - raise ValueError("stage has no successful release; inspect it before removal") - run_logged("destroy", args.target, f"python3 {REMOTE}/releases/{release}/tools/studio.py destroy {args.target}") + release = upload() + run_logged("destroy", args.target, f"python3 {REMOTE}/releases/{release}/tools/release.py stage-destroy {args.target}") if __name__ == "__main__": diff --git a/tools/evil-forgejo-migration.md b/tools/evil-forgejo-migration.md deleted file mode 100644 index 80c31c8623519e5437fb7a9de7d2e70208aab184..0000000000000000000000000000000000000000 --- a/tools/evil-forgejo-migration.md +++ /dev/null @@ -1,27 +0,0 @@ -# evil.inc Forgejo cutover - -Zenith's Forgejo database is 21 MB; its app directory is about 14 GB. The VM Forgejo has the same usernames for all seven numeric user IDs referenced by the imported HedgeDoc OAuth profiles. This checks identity continuity without inspecting private repository contents. The app directory must move with the database because it holds Git objects, LFS data, attachments, avatars, and indexes. - -Snow Globe generates a new automation account password, but the imported Forgejo database needs the old signing and security keys. The named secret import preserves Snow Globe's automation password while replacing only the legacy values. With `STUDIO_DEPLOY_HOST` and `STUDIO_DEPLOY_PORT` pointing at the production host, run: - -```sh -bash tools/import-legacy-secrets.sh evil-forgejo \ - mailer_address=MAILER_ADDRESS \ - mailer_username=MAILER_USERNAME \ - mailer_password=MAILER_PASSWORD \ - lfs_jwt=EVIL_FORGEJO_SERVER_LFS_JWT_SECRET \ - oauth_jwt=EVIL_FORGEJO_OAUTH2_JWT_SECRET \ - security_key=EVIL_FORGEJO_SECURITY_SECRET_KEY \ - internal_token=EVIL_FORGEJO_SECURITY_INTERNAL_TOKEN \ - anubis_key=ANUBIS_PRIVATE_KEY -``` - -The source `.env` contains one unquoted value for each named key. The importer streams them through stdin and checks the destination field names against the service definition. Stop the old Forgejo and HedgeDoc before the final database and app-directory copy, and keep both stopped until the new Forgejo and HedgeDoc identities are checked. Do not switch the public route before the repository data and database have been restored together. - -For a same-machine OS replacement, use the [legacy PostgreSQL handoff](legacy-handoff.md) before reboot. After the encrypted apps dataset is mounted on NixOS, import the original Forgejo secrets with `STUDIO_LEGACY_HANDOFF` set, stop the Snow Globe Forgejo job, then run `bash tools/import-evil-forgejo.sh`. It checks the retained database dump and secret fingerprints, copies `/mnt/storage1/apps/evil-infra/forgejo` directly into the managed service dataset, compares a content digest without exposing file contents, restores the database, and checks user/repository counts. It keeps the new job stopped and prints the pre-import ZFS snapshot and database dump for recovery. The 14 GB production copy has not run on the VM. - -A read-only Zenith `pg_dump` restored into a disposable VM PostgreSQL database on 2026-09-26. The source and restored database both had 10 users and 15 repositories; the disposable database was removed afterward. No repository contents were opened or copied. This verifies the logical database restore separately from the 14 GB file transfer. - -A disposable VM fixture also passed the importer's `rsync -aH --numeric-ids --one-file-system --delete` copy and `tree-hash.py` comparison, including a hardlink, symlink, owner, group, and mode. The production 14 GB transfer remains untested. - -After a database import, Snow Globe's saved HedgeDoc OAuth application ID may point to a different legacy client. The provider now checks the client ID before updating an application and creates a fresh OAuth client when the saved one is missing or belongs to another client. A VM probe used a real ID collision: the unrelated client remained unchanged, the replacement client was created, and the temporary replacement was deleted after the check. diff --git a/tools/export-legacy-postgres.sh b/tools/export-legacy-postgres.sh deleted file mode 100644 index 58ffb0afe0983cfda8b5a114e4e7f24b2cabf86d..0000000000000000000000000000000000000000 --- a/tools/export-legacy-postgres.sh +++ /dev/null @@ -1,80 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -source_host=${STUDIO_MIGRATION_SOURCE:-zenith} -root=/mnt/storage1/apps/studio-handoff -id=$(date -u +%Y%m%dT%H%M%SZ)-$(python3 -c 'import secrets; print(secrets.token_hex(3))') -pending=$root/.pending-$id -final=$root/$id - -ssh "$source_host" 'test "$(findmnt -n -o SOURCE --mountpoint /mnt/storage1/apps)" = storage1/apps; test "$(findmnt -n -o FSTYPE --mountpoint /mnt/storage1/apps)" = zfs' -ssh "$source_host" 'sudo -n docker info >/dev/null' - -check_stopped() { - [[ $(ssh "$source_host" "sudo -n docker inspect -f '{{.State.Running}}' evil-forgejo") == false ]] || { - echo 'Stop Zenith evil-forgejo before exporting the final database' >&2 - exit 1 - } - [[ $(ssh "$source_host" "sudo -n docker ps -a --filter label=com.docker.compose.service=evil-hedgedoc --format '{{.State}}'") == exited ]] || { - echo 'Stop Zenith evil-hedgedoc before exporting the final databases' >&2 - exit 1 - } -} - -check_stopped -ssh "$source_host" 'sudo -n docker exec postgres pg_isready -U postgres >/dev/null' -ssh "$source_host" 'test -r /mnt/storage1/apps/home-infra/.env' -ssh "$source_host" "set -e; test ! -e '$final'; mkdir -m 700 -p '$root'; mkdir -m 700 '$pending'" -for database in evil-forgejo evil-hedgedoc; do - ssh "$source_host" "set -e; umask 077; sudo -n docker exec postgres pg_dump -U postgres -Fc --no-owner --no-acl '$database' > '$pending/$database.dump'; test -s '$pending/$database.dump'; sudo -n docker exec -i postgres pg_restore -l < '$pending/$database.dump' >/dev/null" -done -check_stopped -ssh "$source_host" python3 - "$pending" "$id" <<'PY' -import hashlib -import json -from pathlib import Path -import subprocess -import sys - -folder = Path(sys.argv[1]) -manifest = {"id": sys.argv[2], "databases": {}} -queries = { - "evil-forgejo": 'SELECT (SELECT count(*) FROM "user"), (SELECT count(*) FROM repository)', - "evil-hedgedoc": 'SELECT (SELECT count(*) FROM "Notes"), (SELECT count(*) FROM "Users"), (SELECT count(*) FROM "Revisions"), (SELECT count(*) FROM "Authors")', -} -for name in ("evil-forgejo", "evil-hedgedoc"): - source = folder / f"{name}.dump" - digest = hashlib.sha256() - with source.open("rb") as file: - for chunk in iter(lambda: file.read(1024 * 1024), b""): - digest.update(chunk) - result = subprocess.run(["sudo", "-n", "docker", "exec", "postgres", "psql", "-U", "postgres", "-d", name, - "-At", "-c", queries[name]], check=True, capture_output=True, text=True) - counts = [int(value) for value in result.stdout.strip().split("|")] - if len(counts) != (4 if name == "evil-hedgedoc" else 2): - raise ValueError(f"Unexpected count result for {name}") - manifest["databases"][name] = {"file": source.name, "bytes": source.stat().st_size, - "sha256": digest.hexdigest(), "counts": counts} -legacy_env = (Path("/mnt/storage1/apps/home-infra/.env")).read_text().splitlines() -forgejo_secrets = { - "lfs_jwt": "EVIL_FORGEJO_SERVER_LFS_JWT_SECRET", - "oauth_jwt": "EVIL_FORGEJO_OAUTH2_JWT_SECRET", - "security_key": "EVIL_FORGEJO_SECURITY_SECRET_KEY", - "internal_token": "EVIL_FORGEJO_SECURITY_INTERNAL_TOKEN", - "anubis_key": "ANUBIS_PRIVATE_KEY", - "mailer_address": "MAILER_ADDRESS", - "mailer_username": "MAILER_USERNAME", - "mailer_password": "MAILER_PASSWORD", -} -fingerprints = {} -for target, source in forgejo_secrets.items(): - values = [line.split("=", 1)[1] for line in legacy_env if line.startswith(source + "=")] - if len(values) != 1 or not values[0]: - raise ValueError(f"Legacy Forgejo secret is unavailable: {source}") - fingerprints[target] = hashlib.sha256(values[0].encode()).hexdigest() -manifest["databases"]["evil-forgejo"]["secretSha256"] = fingerprints -(folder / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n") -PY -ssh "$source_host" "set -e; chmod 600 '$pending/manifest.json'; mv '$pending' '$final'; sync" -check_stopped -echo "Legacy PostgreSQL handoff: $final" diff --git a/tools/export-offline-postgres.py b/tools/export-offline-postgres.py deleted file mode 100644 index 884dca427d861617bc7e1f8b9eb75a62a6d398aa..0000000000000000000000000000000000000000 --- a/tools/export-offline-postgres.py +++ /dev/null @@ -1,220 +0,0 @@ -#!/usr/bin/env python3 -import argparse -from contextlib import contextmanager -from datetime import datetime, timezone -import hashlib -import json -import os -from pathlib import Path -import secrets -import shutil -import subprocess -import tempfile - - -def run(command, *, user=None, output=None): - result = subprocess.run(command, stdout=output or subprocess.PIPE, - stderr=subprocess.PIPE, user=user, group=user, - extra_groups=[] if user is not None else None) - if result.returncode: - with (destination / 'diagnostics.log').open('ab') as log: - log.write(result.stderr) - raise RuntimeError(f'{Path(command[0]).name} failed; see private diagnostics.log') - return result.stdout - - -def sha256(path): - digest = hashlib.sha256() - with path.open('rb') as source: - for block in iter(lambda: source.read(1024 * 1024), b''): - digest.update(block) - return digest.hexdigest() - - -def tree_hash(path): - digest = hashlib.sha256() - for entry in sorted(path.rglob('*')): - digest.update(str(entry.relative_to(path)).encode() + b'\0') - if entry.is_symlink(): - raise ValueError(f'External cluster link requires separate backup: {entry}') - if entry.is_file(): - digest.update(bytes.fromhex(sha256(entry))) - return digest.hexdigest() - - -def identifier(value): - return '"' + value.replace('"', '""') + '"' - - -@contextmanager -def server(runtime, folder, *, source=None): - folder.mkdir(mode=0o700) - data = folder / 'data' - socket = folder / 'socket' - socket.mkdir(mode=0o700) - os.chown(folder, 65534, 65534) - os.chown(socket, 65534, 65534) - if source: - run(['rsync', '-aH', '--numeric-ids', str(source) + '/', str(data) + '/']) - (data / 'postmaster.pid').unlink(missing_ok=True) - for entry in [data, *data.rglob('*')]: - os.chown(entry, 65534, 65534) - else: - run([str(runtime / 'bin/initdb'), '-D', str(data), '-U', 'infra2_backup_verifier', - '--auth=trust', '--locale=C.UTF-8'], user=65534) - config = folder / 'postgresql.conf' - config.write_text("listen_addresses = ''\nssl = off\nlogging_collector = off\n" - "shared_preload_libraries = ''\n" - f"unix_socket_directories = '{socket}'\n" - f"hba_file = '{folder / 'pg_hba.conf'}'\n") - (folder / 'pg_hba.conf').write_text('local all all trust\n') - os.chown(config, 65534, 65534) - os.chown(folder / 'pg_hba.conf', 65534, 65534) - control = [str(runtime / 'bin/pg_ctl'), '-D', str(data)] - try: - run(control + ['-l', str(folder / 'server.log'), '-o', f'-c config_file={config}', - '-w', 'start'], user=65534) - yield socket - finally: - if (data / 'postmaster.pid').exists(): - run(control + ['-m', 'fast', '-w', 'stop'], user=65534) - - -def sql(runtime, socket, database, query, *, role='postgres'): - return run([str(runtime / 'bin/psql'), '-X', '-h', str(socket), '-U', role, - '-d', database, '-v', 'ON_ERROR_STOP=1', '-At', '-c', query]) - - -def fingerprint(runtime, socket, database): - relations = json.loads(sql(runtime, socket, database, """ - SELECT coalesce(json_agg(x ORDER BY n, c), '[]') FROM ( - SELECT n.nspname n, c.relname c, c.relkind k - FROM pg_class c JOIN pg_namespace n ON n.oid=c.relnamespace - WHERE n.nspname NOT LIKE 'pg_%' AND n.nspname <> 'information_schema' - AND c.relkind IN ('r','m','S') - ) x - """)) - result = {} - for relation in relations: - table = identifier(relation['n']) + '.' + identifier(relation['c']) - query = (f'SELECT last_value,is_called FROM {table}' if relation['k'] == 'S' else - f"COPY (SELECT row_to_json(t)::text FROM {table} t " - 'ORDER BY row_to_json(t)::text COLLATE "C") TO STDOUT') - content = sql(runtime, socket, database, query) - result[table] = {'rows': len(content.splitlines()), - 'sha256': hashlib.sha256(content).hexdigest()} - for name, query in { - 'large_objects': 'COPY (SELECT loid,pageno,encode(data,\'hex\') FROM pg_largeobject ORDER BY loid,pageno) TO STDOUT', - 'large_object_owners': 'COPY (SELECT oid,pg_get_userbyid(lomowner),lomacl::text FROM pg_largeobject_metadata ORDER BY oid) TO STDOUT', - }.items(): - content = sql(runtime, socket, database, query) - result[name] = {'rows': len(content.splitlines()), - 'sha256': hashlib.sha256(content).hexdigest()} - return result - - -parser = argparse.ArgumentParser(description='Export and restore-check stopped PostgreSQL clusters from private copies.') -parser.add_argument('--cluster', nargs=2, action='append', required=True, metavar=('SOURCE', 'RUNTIME')) -parser.add_argument('--locale-archive', type=Path, required=True) -args = parser.parse_args() -if os.geteuid() != 0: - parser.error('Run as root on the installer') -os.umask(0o077) -os.environ['LOCALE_ARCHIVE'] = str(args.locale_archive) -os.environ.pop('PGOPTIONS', None) -root = Path('/mnt/storage1/apps/studio-handoff') -mounted = subprocess.check_output(['findmnt', '-n', '-o', 'SOURCE,FSTYPE', '--mountpoint', str(root.parent)], text=True).split() -if len(mounted) != 2 or mounted[1] != 'zfs' or not mounted[0].endswith('/apps'): - parser.error('Expected retained apps dataset mounted at /mnt/storage1/apps') -if subprocess.run(['pgrep', '-x', 'postgres'], stdout=subprocess.DEVNULL).returncode == 0: - parser.error('Stop PostgreSQL before taking an offline copy') -handoff_id = datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%SZ-') + secrets.token_hex(3) -root.mkdir(mode=0o700, exist_ok=True) -destination = root / ('.pending-' + handoff_id) -destination.mkdir(mode=0o700) -manifest = {'id': handoff_id, 'clusters': {}, 'databases': {}} -work = Path(tempfile.mkdtemp(prefix='infra2-pg-export-', dir='/run')) -os.chown(work, 65534, 65534) -try: - for source_arg, runtime_arg in args.cluster: - source, runtime = Path(source_arg).resolve(), Path(runtime_arg).resolve() - if not source.is_relative_to(root.parent) or not (source / 'PG_VERSION').is_file(): - raise ValueError('Source must be a retained cluster inside the apps dataset') - major = (source / 'PG_VERSION').read_text().strip() - version = run([str(runtime / 'bin/postgres'), '--version']).decode().strip() - if version.split()[-1].split('.')[0] != major or major in manifest['clusters']: - raise ValueError('Runtime major must match its unique source cluster') - before = tree_hash(source) - output = destination / ('pg' + major) - output.mkdir(mode=0o700) - run(['tar', '-C', str(source), '-czf', str(output / 'cluster.tar.gz'), '.']) - cluster = {'source': str(source), 'runtime': version, 'sourceSha256': before, 'databases': {}} - with server(runtime, work / ('source' + major), source=source) as socket: - databases = json.loads(sql(runtime, socket, 'postgres', """ - SELECT json_agg(x ORDER BY name) FROM ( - SELECT datname name, datallowconn connect, datistemplate template, - pg_get_userbyid(datdba) owner FROM pg_database WHERE datname <> 'template0' - ) x - """)) - tablespaces = json.loads(sql(runtime, socket, 'postgres', "SELECT coalesce(json_agg(spcname), '[]') FROM pg_tablespace WHERE spcname NOT IN ('pg_default','pg_global')")) - if tablespaces: - raise ValueError('Custom tablespaces require an explicit restore layout') - with (output / 'globals.sql').open('wb') as file: - run([str(runtime / 'bin/pg_dumpall'), '-h', str(socket), '-U', 'postgres', '--globals-only'], output=file) - for index, database in enumerate(databases): - name = database['name'] - if not database['connect']: - sql(runtime, socket, 'postgres', f'ALTER DATABASE {identifier(name)} ALLOW_CONNECTIONS true') - dump = output / f'{index:03d}.dump' - with dump.open('wb') as file: - run([str(runtime / 'bin/pg_dump'), '-h', str(socket), '-U', 'postgres', '-d', name, - '-Fc', '--create'], output=file) - entry = {**database, 'file': str(dump.relative_to(destination)), 'bytes': dump.stat().st_size, - 'sha256': sha256(dump), 'contents': fingerprint(runtime, socket, name)} - cluster['databases'][name] = entry - if name in {'evil-forgejo', 'evil-hedgedoc'} and major == '18': - query = ('SELECT (SELECT count(*) FROM "user"), (SELECT count(*) FROM repository)' if name == 'evil-forgejo' else - 'SELECT (SELECT count(*) FROM "Notes"), (SELECT count(*) FROM "Users"), (SELECT count(*) FROM "Revisions"), (SELECT count(*) FROM "Authors")') - entry['counts'] = [int(n) for n in sql(runtime, socket, name, query).decode().strip().split('|')] - if tree_hash(source) != before: - raise RuntimeError('Original cluster changed during export; export is not publishable') - with server(runtime, work / ('restore' + major)) as socket: - bootstrap = 'infra2_backup_verifier' - sql(runtime, socket, 'postgres', f'CREATE DATABASE {bootstrap}', role=bootstrap) - run([str(runtime / 'bin/psql'), '-X', '-h', str(socket), '-U', bootstrap, '-d', bootstrap, - '-v', 'ON_ERROR_STOP=1', '-f', str(output / 'globals.sql')]) - sql(runtime, socket, bootstrap, 'ALTER DATABASE template1 IS_TEMPLATE false', role=bootstrap) - for name, entry in cluster['databases'].items(): - run([str(runtime / 'bin/pg_restore'), '-h', str(socket), '-U', bootstrap, '-d', bootstrap, - '--clean', '--if-exists', '--create', '--exit-on-error', str(destination / entry['file'])]) - if fingerprint(runtime, socket, name) != entry['contents']: - raise RuntimeError(f'Restored data differs: PostgreSQL {major} database {name}') - print(f'PostgreSQL {major}: restored and verified {name}', flush=True) - cluster['restoreVerified'] = True - manifest['clusters'][major] = cluster - for name in ('evil-forgejo', 'evil-hedgedoc'): - entry = manifest['clusters']['18']['databases'][name] - (destination / (name + '.dump')).symlink_to(entry['file']) - manifest['databases'][name] = {key: entry[key] for key in ('bytes', 'sha256', 'counts')} - manifest['databases'][name]['file'] = name + '.dump' - keys = {'lfs_jwt': 'EVIL_FORGEJO_SERVER_LFS_JWT_SECRET', 'oauth_jwt': 'EVIL_FORGEJO_OAUTH2_JWT_SECRET', - 'security_key': 'EVIL_FORGEJO_SECURITY_SECRET_KEY', 'internal_token': 'EVIL_FORGEJO_SECURITY_INTERNAL_TOKEN', - 'anubis_key': 'ANUBIS_PRIVATE_KEY', 'mailer_address': 'MAILER_ADDRESS', - 'mailer_username': 'MAILER_USERNAME', 'mailer_password': 'MAILER_PASSWORD'} - legacy_env = (root.parent / 'home-infra/.env').read_text().splitlines() - fingerprints = {} - for target, key in keys.items(): - values = [line.split('=', 1)[1] for line in legacy_env if line.startswith(key + '=')] - if len(values) != 1 or not values[0]: - raise ValueError(f'Legacy secret unavailable: {key}') - fingerprints[target] = hashlib.sha256(values[0].encode()).hexdigest() - manifest['databases']['evil-forgejo']['secretSha256'] = fingerprints - manifest['files'] = {str(path.relative_to(destination)): {'bytes': path.stat().st_size, 'sha256': sha256(path)} - for path in destination.rglob('*') if path.is_file() and not path.is_symlink()} - (destination / 'manifest.json').write_text(json.dumps(manifest, indent=2) + '\n') - final = root / handoff_id - destination.rename(final) - os.sync() - print(f'Verified PostgreSQL handoff: {final}', flush=True) -finally: - shutil.rmtree(work) diff --git a/tools/guest-channel-test.py b/tools/guest-channel-test.py index 2310a5fa3e540fc24c4ce939d103ab49b0c9a8fa..0dc640ab603638db9aa0bac9990d613430705f44 100644 --- a/tools/guest-channel-test.py +++ b/tools/guest-channel-test.py @@ -15,13 +15,13 @@ class ChannelTests(unittest.TestCase): def test_only_the_running_domains_owned_channel_is_opened(self): path = f"/run/libvirt/qemu/channel/9-fixture/{vms.GUEST_CHANNEL}" self.assertEqual(self.request(path), {"path": path}) - for source, state, channel in [("/run/studio-host/host.sock", "running", None), + for source, state, channel in [("/run/snowglobe-host/host.sock", "running", None), (path.replace("9-fixture", "8-other"), "running", None), (path, "paused", None), (path, "running", "org.qemu.guest_agent.0")]: with self.subTest(source=source, state=state, channel=channel), self.assertRaises(ValueError): self.request(source, state, channel) with self.assertRaises(ValueError): - vms.validate("guest", {"name": "fixture", "path": "/run/studio-host/host.sock"}) + vms.validate("guest", {"name": "fixture", "path": "/run/snowglobe-host/host.sock"}) with self.assertRaises(ValueError): vms.validate("guest", {"name": "../other"}) diff --git a/tools/guest-provider.py b/tools/guest-provider.py index c9211f1441b8f8e745e0f2ef8c050c8a1e0bc933..36df703f9e5eea6110117a92363e44170d7ecef7 100644 --- a/tools/guest-provider.py +++ b/tools/guest-provider.py @@ -22,8 +22,8 @@ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.R parser.add_argument('provider', choices=['github', 'astheno']) parser.add_argument('--client-id') parser.add_argument('--disable', action='store_true') -parser.add_argument('--host', default=os.environ.get('STUDIO_DEPLOY_HOST', 'root@zenith')) -parser.add_argument('--port', default=os.environ.get('STUDIO_DEPLOY_PORT', '22')) +parser.add_argument('--host', default=os.environ.get('SNOWGLOBE_DEPLOY_HOST', 'root@10.0.0.1')) +parser.add_argument('--port', default=os.environ.get('SNOWGLOBE_DEPLOY_PORT', '22')) args = parser.parse_args() if not args.disable and not args.client_id: parser.error('--client-id is required unless --disable is used') @@ -31,7 +31,7 @@ payload = {'provider': args.provider, 'enabled': not args.disable} if not args.disable: payload.update(clientId=args.client_id, clientSecret=getpass.getpass('Client secret: ')) result = subprocess.run(['ssh', '-p', args.port, args.host, - 'podman exec -i studio-dashboard /bin/home-dashboard --guest-provider'], + 'podman exec -i snowglobe-dashboard /bin/home-dashboard --guest-provider'], input=json.dumps(payload), text=True, capture_output=True) if result.returncode: raise SystemExit('Provider configuration did not complete. Check the dashboard and SSH connection.') diff --git a/tools/hedgedoc-migration.md b/tools/hedgedoc-migration.md deleted file mode 100644 index 43b3ef77d5a9755dc8d1a8fdcf706524c2553ae0..0000000000000000000000000000000000000000 --- a/tools/hedgedoc-migration.md +++ /dev/null @@ -1,9 +0,0 @@ -# HedgeDoc migration - -`bash tools/import-hedgedoc.sh evil-hedgedoc-preview-f6eb07a4` restored Zenith's HedgeDoc database into a disposable stage and copied its uploads. The importer checked the upload copy by checksum, backed up the stage database, compared the restored table counts with Zenith, and restarted the stage. On 2026-09-26 both sides had 39 notes, 8 users, 183 revisions, and 68 authors. The stage returned HTTPS 200; `/auth/oauth2` redirected to the staged Forgejo OAuth client at `git.evil.studio.test` with the stage callback URL. No note bodies or repository contents were inspected. - -The seven HedgeDoc OAuth profile IDs all resolve to the same Forgejo usernames on Zenith and the VM when matched by numeric user ID. That checks account continuity for the imported database; an interactive sign-in remains the final identity check. - -For production, keep the Forgejo user IDs stable. Stop Zenith's HedgeDoc and the Snow Globe HedgeDoc job, set `STUDIO_DEPLOY_HOST` and `STUDIO_DEPLOY_PORT` for the new host, then run `bash tools/import-hedgedoc.sh evil-hedgedoc`. The importer refuses a running source or destination, verifies the uploads and four table counts, leaves a pre-import destination database dump, and leaves Snow Globe stopped. Promote a tested preview to start the new service, then verify HTTPS and a Forgejo sign-in before switching the public route. The old HedgeDoc data stays in place until cutover is accepted. - -For a same-machine OS replacement, use the [legacy PostgreSQL handoff](legacy-handoff.md) and set `STUDIO_LEGACY_HANDOFF` when running the production importer. It verifies the retained dump and reads uploads from the mounted old apps dataset without contacting old Docker. diff --git a/tools/import-arr.sh b/tools/import-arr.sh deleted file mode 100644 index d782cedfd826ebab8783d70b7139241e1f2a6449..0000000000000000000000000000000000000000 --- a/tools/import-arr.sh +++ /dev/null @@ -1,127 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -service=${1:?usage: tools/import-arr.sh sonarr|radarr [preview-id]} -[[ $service == sonarr || $service == radarr ]] || { echo 'Expected sonarr or radarr' >&2; exit 1; } -instance=${2:-$service} -[[ $instance == "$service" || $instance =~ ^${service}-preview-[0-9a-f]{8}$ ]] || { echo 'Expected the service ID or its preview ID' >&2; exit 1; } - -source_host=${STUDIO_MIGRATION_SOURCE:-zenith} -target_host=${STUDIO_DEPLOY_HOST:-root@127.0.0.1} -target_port=${STUDIO_DEPLOY_PORT:-2222} -remote=(ssh -p "$target_port" "$target_host") -source_copy_host=$source_host -source_rsh=ssh -offline=false -if [[ -n ${STUDIO_LEGACY_HANDOFF:-} ]]; then - [[ $instance == "$service" && -n ${STUDIO_DEPLOY_HOST:-} ]] || { - echo 'Offline handoff requires a production ARR service and target' >&2; exit 1; - } - sh "$(dirname "$0")/check-legacy-handoff.sh" "$STUDIO_LEGACY_HANDOFF" "$target_host" "$target_port" - source_copy_host=$target_host - source_rsh="ssh -p $target_port" - offline=true -fi -source_ssh() { - if [[ $offline == true ]]; then - "${remote[@]}" "$@" - else - ssh "$source_host" "$@" - fi -} -if [[ $instance == "$service" ]]; then - root="/srv/prod/$service" - if [[ $offline == false ]]; then - source_running=$(ssh "$source_host" "sudo -n docker inspect -f '{{.State.Running}}' $service") - [[ $source_running == false ]] || { echo "Stop Zenith $service before importing production data" >&2; exit 1; } - fi - response=$("${remote[@]}" "curl -s -w '\n%{http_code}' -H \"X-Nomad-Token: \$(cat /var/lib/studio/nomad.token)\" http://127.0.0.1:4646/v1/job/$service") - status=${response##*$'\n'} - if [[ $status == 200 ]]; then - stopped=$(python3 -c 'import json,sys; print(str(json.load(sys.stdin)["Stop"]).lower())' <<<"${response%$'\n'*}") - [[ $stopped == true ]] || { echo "Stop production $service before importing" >&2; exit 1; } - elif [[ $status != 404 ]]; then - echo "Could not verify production $service job state: $status" >&2 - exit 1 - fi -else - root="/srv/staging/$instance" - source_id=$("${remote[@]}" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/stages/$instance.json\"))[\"sourceId\"])'") - [[ $source_id == "$service" ]] || { echo 'Preview belongs to another service' >&2; exit 1; } -fi -dataset=$("${remote[@]}" "findmnt -n -o SOURCE --mountpoint $root") -[[ $dataset == */prod/"$service" || $dataset == */staging/"$instance" ]] || { echo "Expected a mounted service dataset at $root" >&2; exit 1; } - -scratch=$(mktemp -d) -source_backup= -cleanup() { - rm -rf "$scratch" - if [[ -n $source_backup ]]; then - source_ssh "rm -f '$source_backup'" - fi -} -trap cleanup EXIT - -source_backup=$(source_ssh "python3 - '/mnt/storage1/apps/$service/$service.db'" <<'PY' -import os -import sqlite3 -import sys -import tempfile - -source = sqlite3.connect(f"file:{sys.argv[1]}?mode=ro", uri=True) -fd, name = tempfile.mkstemp(prefix="studio-arr-", suffix=".db") -os.close(fd) -try: - target = sqlite3.connect(name) - source.backup(target) - if target.execute("PRAGMA integrity_check").fetchone()[0] != "ok": - raise ValueError("ARR SQLite backup is invalid") - target.close() - source.close() -except BaseException: - os.unlink(name) - raise -print(name) -PY -) -rsync -a --exclude="/$service.db*" --exclude='/logs.db*' --exclude='/*.pid' --exclude='/logs/' \ - -e "$source_rsh" "$source_copy_host:/mnt/storage1/apps/$service/" "$scratch/config/" -if [[ $offline == true ]]; then - scp -q -P "$target_port" "$source_copy_host:$source_backup" "$scratch/config/$service.db" -else - scp -q "$source_copy_host:$source_backup" "$scratch/config/$service.db" -fi - -if [[ $instance != "$service" ]]; then - "${remote[@]}" "NOMAD_TOKEN=\$(cat /var/lib/studio/nomad.token) nomad job stop -yes $instance" -fi -for _ in {1..30}; do - running=$("${remote[@]}" "curl -s -H \"X-Nomad-Token: \$(cat /var/lib/studio/nomad.token)\" http://127.0.0.1:4646/v1/job/$instance/allocations" | - python3 -c 'import json,sys; print(sum(x["ClientStatus"] == "running" for x in json.load(sys.stdin)))') - [[ $running == 0 ]] && break - sleep 2 -done -[[ $running == 0 ]] || { echo "$instance still has a running allocation" >&2; exit 1; } - -snapshot="$dataset@before-arr-import-$(date +%s)-$$" -"${remote[@]}" "zfs snapshot $snapshot" -rsync -a --delete -e "ssh -p $target_port" "$scratch/config/" "$target_host:$root/config/" -uid=$("${remote[@]}" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/identities.json\"))[\"$service\"])'") -"${remote[@]}" "chown -R $uid:$uid $root/config; chmod 750 $root/config; python3 -c 'import sqlite3; c=sqlite3.connect(\"$root/config/$service.db\"); assert c.execute(\"PRAGMA integrity_check\").fetchone()[0] == \"ok\"'" -local_hash=$(shasum -a 256 "$scratch/config/$service.db" | cut -d ' ' -f 1) -remote_hash=$("${remote[@]}" "sha256sum $root/config/$service.db" | cut -d ' ' -f 1) -[[ $local_hash == "$remote_hash" ]] || { echo 'Copied database hash differs from source backup' >&2; exit 1; } - -if [[ $instance == "$service" ]]; then - if [[ $offline == false ]]; then - [[ $(ssh "$source_host" "sudo -n docker inspect -f '{{.State.Running}}' $service") == false ]] || { - echo "Zenith $service restarted during import; leave the home server stopped" >&2 - exit 1 - } - fi - echo "Imported production $service from Zenith. Previous dataset state: $snapshot" -else - python3 "$(dirname "$0")/deploy.py" stage "$service" - "${remote[@]}" "zfs destroy $snapshot" - echo "Imported and tested $instance" -fi diff --git a/tools/import-dashboard-auth.py b/tools/import-dashboard-auth.py deleted file mode 100644 index a3f0f296990bfc0c0c30559de4a42159046c9d83..0000000000000000000000000000000000000000 --- a/tools/import-dashboard-auth.py +++ /dev/null @@ -1,59 +0,0 @@ -#!/usr/bin/env python3 -import argparse -import hashlib -import json -import os -from pathlib import Path -import subprocess - - -parser = argparse.ArgumentParser(description="Copy Keycloak accounts into a private dashboard import") -parser.add_argument("--host", required=True) -parser.add_argument("--output", required=True, type=Path) -args = parser.parse_args() -if args.output.exists(): - parser.error("output already exists") - -remote = r''' -import json, subprocess, urllib.request -request = urllib.request.Request("http://127.0.0.1:4646/v1/job/postgres/allocations", headers={"X-Nomad-Token":open("/var/lib/studio/nomad.token").read().strip()}) -allocations = [a["ID"] for a in json.load(urllib.request.urlopen(request)) if a["ClientStatus"] == "running" and a["DesiredStatus"] == "run"] -assert len(allocations) == 1 -containers = [line.split()[0] for line in subprocess.check_output(["podman", "ps", "--format", "{{.ID}} {{.Names}}"], text=True).splitlines() if line.split()[1].endswith(allocations[0])] -assert len(containers) == 1 -sql = """ -BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ READ ONLY; -SELECT json_build_object( - 'issuer','https://auth.paperclover.net/realms/master', - 'rpId','auth.paperclover.net', - 'roles',(SELECT coalesce(json_agg(json_build_object('id',id,'name',name)), '[]') FROM keycloak_role WHERE realm_id=(SELECT id FROM realm WHERE name='master') AND client_role=false), - 'users',(SELECT coalesce(json_agg(json_build_object( - 'id',u.id,'username',u.username,'email',u.email,'firstName',u.first_name,'lastName',u.last_name, - 'enabled',u.enabled,'emailVerified',u.email_verified,'createdTimestamp',u.created_timestamp, - 'requiredActions',(SELECT coalesce(json_agg(required_action),'[]') FROM user_required_action WHERE user_id=u.id), - 'attributes',(SELECT coalesce(json_object_agg(name,vals),'{}') FROM (SELECT name,json_agg(value) AS vals FROM user_attribute WHERE user_id=u.id GROUP BY name)a), - 'roles',(SELECT coalesce(json_agg(role_id),'[]') FROM user_role_mapping WHERE user_id=u.id), - 'credentials',(SELECT coalesce(json_agg(json_build_object('id',id,'type',type,'userLabel',user_label,'createdDate',created_date,'credentialData',credential_data::json,'secretData',secret_data::json)),'[]') FROM credential WHERE user_id=u.id) - )),'[]') FROM user_entity u WHERE realm_id=(SELECT id FROM realm WHERE name='master')) -); -COMMIT; -""" -result = subprocess.run(["podman", "exec", "-i", containers[0], "psql", "-U", "postgres", "-d", "keycloak_next", "-tA", "-v", "ON_ERROR_STOP=1"], input=sql, text=True, capture_output=True, check=True) -print(next(line for line in result.stdout.splitlines() if line.startswith('{'))) -''' -result = subprocess.run(["ssh", "-o", "BatchMode=yes", args.host, "python3", "-"], input=remote, text=True, capture_output=True, check=True) -data = json.loads(result.stdout) -if not data["users"]: - raise ValueError("source realm has no accounts") -content = (json.dumps(data, separators=(",", ":")) + "\n").encode() -args.output.parent.mkdir(mode=0o700, parents=True, exist_ok=True) -with os.fdopen(os.open(args.output, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600), "wb") as output: - output.write(content) - output.flush() - os.fsync(output.fileno()) -counts = {} -for user in data["users"]: - for credential in user["credentials"]: - kind = credential["type"] - counts[kind] = counts.get(kind, 0) + 1 -print(json.dumps({"accounts": len(data["users"]), "credentials": counts, "sha256": hashlib.sha256(content).hexdigest()})) diff --git a/tools/import-dawarich.sh b/tools/import-dawarich.sh deleted file mode 100644 index 6bc6937680fa96bce514ebb9141f9d3a7ecfefa6..0000000000000000000000000000000000000000 --- a/tools/import-dawarich.sh +++ /dev/null @@ -1,73 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -instance=${1:?usage: tools/import-dawarich.sh dawarich-preview-XXXXXXXX} -[[ $instance =~ ^dawarich-preview-[0-9a-f]{8}$ ]] || { - echo 'Expected a Dawarich preview ID' >&2 - exit 1 -} - -source_host=${STUDIO_MIGRATION_SOURCE:-zenith} -target_host=${STUDIO_DEPLOY_HOST:-root@127.0.0.1} -target_port=${STUDIO_DEPLOY_PORT:-2222} -remote=(ssh -p "$target_port" "$target_host") - -root=/srv/staging/$instance -source_id=$("${remote[@]}" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/stages/$instance.json\"))[\"sourceId\"])'") -[[ $source_id == dawarich ]] || { echo 'Preview belongs to another service' >&2; exit 1; } -snapshot=$(ssh "$source_host" '/usr/sbin/zfs list -H -t snapshot -o name -s creation -r storage1/apps | tail -n 1') -[[ $snapshot == storage1/apps@* ]] || { echo 'No Zenith app snapshot is available' >&2; exit 1; } -source_root=/mnt/storage1/apps/.zfs/snapshot/${snapshot#*@}/dawarich_storage - -dataset=$("${remote[@]}" "findmnt -n -o SOURCE --mountpoint $root") -[[ $dataset == */staging/$instance ]] || { echo 'Preview dataset is not mounted' >&2; exit 1; } - -source_secret=$(ssh "$source_host" "sudo -n docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' dawarich-app" | - python3 -c 'import hashlib,sys; values=[line.split("=",1)[1] for line in sys.stdin.read().splitlines() if line.startswith("SECRET_KEY_BASE=")]; assert len(values)==1; print(hashlib.sha256(values[0].encode()).hexdigest())') -own=$("${remote[@]}" "NOMAD_TOKEN=\$(cat /var/lib/studio/nomad.token) nomad var get -out json nomad/jobs/$instance" | - python3 -c 'import hashlib,json,sys; print(hashlib.sha256(json.load(sys.stdin)["Items"]["secret_key_base"].encode()).hexdigest())') -[[ $source_secret == "$own" ]] || { echo 'Dawarich secret differs from Zenith; import the original before restoring data' >&2; exit 1; } - -db_json=$("${remote[@]}" "NOMAD_TOKEN=\$(cat /var/lib/studio/nomad.token) nomad var get -out json nomad/jobs/$instance/inputs/database") -read -r database owner < <(python3 -c 'import json,sys; d=json.load(sys.stdin)["Items"]; print(d["name"], d["username"])' <<<"$db_json") -[[ $database =~ ^dawarich_s_[0-9a-f]{8}$ && $owner == svc_$database ]] || { echo 'Unexpected preview database' >&2; exit 1; } - -allocation=$("${remote[@]}" 'NOMAD_TOKEN=$(cat /var/lib/studio/nomad.token) nomad job allocs -json postgres' | - python3 -c 'import json,sys; ids=[a["ID"] for a in json.load(sys.stdin) if a["ClientStatus"]=="running" and a["DesiredStatus"]=="run"]; assert len(ids)==1; print(ids[0])') -container=app-$allocation -podman='podman --url unix:///run/podman/podman.sock' -scratch=$(mktemp -d) -trap 'rm -rf "$scratch"' EXIT -rsync -a "$source_host:$source_root/" "$scratch/" -drift=$(rsync -rlnc --delete --out-format='%n' "$source_host:$source_root/" "$scratch/") -[[ -z $drift ]] || { echo 'Zenith storage changed during copy; retry the import' >&2; exit 1; } -source_counts=$(ssh "$source_host" 'sudo -n docker exec postgres psql -U postgres -d dawarich -At -c "SELECT (SELECT count(*) FROM points), (SELECT count(*) FROM users)"') - -"${remote[@]}" "NOMAD_TOKEN=\$(cat /var/lib/studio/nomad.token) nomad job stop -yes $instance" -for _ in {1..30}; do - running=$("${remote[@]}" "NOMAD_TOKEN=\$(cat /var/lib/studio/nomad.token) nomad job allocs -json $instance" | - python3 -c 'import json,sys; print(sum(a["ClientStatus"]=="running" for a in json.load(sys.stdin)))') - [[ $running == 0 ]] && break - sleep 2 -done -[[ $running == 0 ]] || { echo 'Dawarich allocation is still running' >&2; exit 1; } - -backup_snapshot=$dataset@before-dawarich-import-$(date +%s)-$$ -"${remote[@]}" "zfs snapshot $backup_snapshot" -rsync -a --delete -e "ssh -p $target_port" "$scratch/" "$target_host:$root/var/app/storage/" -uid=$("${remote[@]}" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/identities.json\"))[\"dawarich\"])'") -"${remote[@]}" "chown -R $uid:$uid $root/var/app/storage" -drift=$(rsync -rlnc --delete --out-format='%n' -e "ssh -p $target_port" "$scratch/" "$target_host:$root/var/app/storage/") -[[ -z $drift ]] || { echo "Storage copy differs from Zenith; restore $backup_snapshot" >&2; exit 1; } - -backup=/var/lib/studio/$instance-before-import.dump -"${remote[@]}" "umask 077; $podman exec $container pg_dump -U postgres -Fc $database > $backup; test -s $backup" -"${remote[@]}" "$podman exec $container psql -U postgres -d $database -v ON_ERROR_STOP=1 -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public AUTHORIZATION $owner; CREATE EXTENSION postgis;' >/dev/null" -ssh "$source_host" 'sudo -n docker exec postgres pg_dump -U postgres -Fc --no-owner --no-acl --exclude-extension=postgis dawarich' | - "${remote[@]}" "$podman exec -i $container pg_restore -U postgres -d $database --no-owner --no-acl --role=$owner" -target_counts=$("${remote[@]}" "$podman exec $container psql -U postgres -d $database -At -c 'SELECT (SELECT count(*) FROM points), (SELECT count(*) FROM users)'") -[[ $source_counts == "$target_counts" ]] || { echo "Database counts differ from Zenith; restore $backup" >&2; exit 1; } - -python3 "$(dirname "$0")/deploy.py" stage dawarich -"${remote[@]}" "zfs destroy $backup_snapshot" -echo "Imported $instance from $snapshot. Previous database: $backup" diff --git a/tools/import-evil-forgejo.sh b/tools/import-evil-forgejo.sh deleted file mode 100644 index c1b218e62159ccd533fc411c2a376f76fb8035fb..0000000000000000000000000000000000000000 --- a/tools/import-evil-forgejo.sh +++ /dev/null @@ -1,77 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -: "${STUDIO_DEPLOY_HOST:?Set STUDIO_DEPLOY_HOST to the production target}" -: "${STUDIO_LEGACY_HANDOFF:?Set STUDIO_LEGACY_HANDOFF to the offline PostgreSQL handoff}" - -target_port=${STUDIO_DEPLOY_PORT:-22} -remote=(ssh -p "$target_port" "$STUDIO_DEPLOY_HOST") -sh "$(dirname "$0")/check-legacy-handoff.sh" "$STUDIO_LEGACY_HANDOFF" "$STUDIO_DEPLOY_HOST" "$target_port" -root=/srv/prod/evil-forgejo -source_dir=/mnt/storage1/apps/evil-infra/forgejo -database=evil_forgejo -owner=svc_evil_forgejo -podman='podman --url unix:///run/podman/podman.sock' - -entry=$("${remote[@]}" "python3 - '$STUDIO_LEGACY_HANDOFF' evil-forgejo" < "$(dirname "$0")/verify-legacy-dump.py") -source_counts=$(python3 -c 'import json,sys; print("|".join(map(str,json.load(sys.stdin)["counts"])))' <<<"$entry") -"${remote[@]}" "test \$(findmnt -n -o FSTYPE --mountpoint $root) = zfs; test -d $source_dir" - -response=$("${remote[@]}" 'curl -s -w "\n%{http_code}" -H "X-Nomad-Token: $(cat /var/lib/studio/nomad.token)" http://127.0.0.1:4646/v1/job/evil-forgejo') -[[ ${response##*$'\n'} == 200 ]] || { echo 'evil.inc Forgejo is unavailable on the home server' >&2; exit 1; } -stopped=$(python3 -c 'import json,sys; print(str(json.load(sys.stdin)["Stop"]).lower())' <<<"${response%$'\n'*}") -[[ $stopped == true ]] || { echo 'Stop evil.inc Forgejo on the home server before importing' >&2; exit 1; } -for _ in {1..30}; do - running=$("${remote[@]}" 'NOMAD_TOKEN=$(cat /var/lib/studio/nomad.token) nomad job allocs -json evil-forgejo' | - python3 -c 'import json,sys; print(sum(a["ClientStatus"] == "running" for a in json.load(sys.stdin)))') - [[ $running == 0 ]] && break - sleep 2 -done -[[ $running == 0 ]] || { echo 'evil.inc Forgejo did not stop on the home server' >&2; exit 1; } - -db_json=$("${remote[@]}" 'NOMAD_TOKEN=$(cat /var/lib/studio/nomad.token) nomad var get -out json nomad/jobs/evil-forgejo/inputs/database') -read -r actual_database actual_owner < <(python3 -c 'import json,sys; d=json.load(sys.stdin)["Items"]; print(d["name"], d["username"])' <<<"$db_json") -[[ $actual_database == "$database" && $actual_owner == "$owner" ]] || { echo 'Unexpected evil.inc Forgejo database on the home server' >&2; exit 1; } - -"${remote[@]}" "python3 - '$STUDIO_LEGACY_HANDOFF'" <<'PY' -import hashlib -import json -import os -from pathlib import Path -import subprocess -import sys - -legacy = json.loads((Path(sys.argv[1]) / "manifest.json").read_text())["databases"]["evil-forgejo"]["secretSha256"] -environment = {**os.environ, "NOMAD_TOKEN": Path("/var/lib/studio/nomad.token").read_text().strip()} -result = subprocess.run(["nomad", "var", "get", "-out", "json", "nomad/jobs/evil-forgejo"], - check=True, capture_output=True, text=True, env=environment) -actual = json.loads(result.stdout)["Items"] -for name, fingerprint in legacy.items(): - if name not in actual or hashlib.sha256(actual[name].encode()).hexdigest() != fingerprint: - raise ValueError(f"Import the original evil.inc Forgejo secret before restoring data: {name}") -PY - -allocation=$("${remote[@]}" 'NOMAD_TOKEN=$(cat /var/lib/studio/nomad.token) nomad job allocs -json postgres' | - python3 -c 'import json,sys; ids=[a["ID"] for a in json.load(sys.stdin) if a["ClientStatus"] == "running" and a["DesiredStatus"] == "run"]; assert len(ids) == 1; print(ids[0])') -container=app-$allocation -source_manifest=$("${remote[@]}" "python3 - '$source_dir'" < "$(dirname "$0")/tree-hash.py") -source_bytes=$(python3 -c 'import json,sys; print(json.load(sys.stdin)["bytes"])' <<<"$source_manifest") -available=$("${remote[@]}" "df -B1 --output=avail $root | tail -n 1") -(( available > source_bytes )) || { echo 'Too little free space for the evil.inc Forgejo app directory' >&2; exit 1; } - -dataset=$("${remote[@]}" "findmnt -n -o SOURCE --mountpoint $root") -[[ $dataset == */prod/evil-forgejo ]] || { echo 'Unexpected production dataset' >&2; exit 1; } -snapshot=$dataset@before-evil-forgejo-import-$(date +%s)-$$ -"${remote[@]}" "set -e; zfs snapshot '$snapshot'; mkdir -p '$root/data'; rsync -aH --numeric-ids --one-file-system --delete '$source_dir/' '$root/data/'" -target_manifest=$("${remote[@]}" "python3 - '$root/data'" < "$(dirname "$0")/tree-hash.py") -[[ $source_manifest == "$target_manifest" ]] || { echo "Forgejo app-directory copy differs; restore $snapshot" >&2; exit 1; } -uid=$("${remote[@]}" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/identities.json\"))[\"evil-forgejo\"])'") -"${remote[@]}" "chown -R $uid:$uid '$root/data'" - -backup=$root/before-import-$(date +%s)-$$.dump -"${remote[@]}" "umask 077; $podman exec $container pg_dump -U postgres -Fc $database > '$backup'; test -s '$backup'" -"${remote[@]}" "$podman exec $container psql -U postgres -d $database -v ON_ERROR_STOP=1 -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public AUTHORIZATION $owner;' >/dev/null" -"${remote[@]}" "set -o pipefail; cat '$STUDIO_LEGACY_HANDOFF/evil-forgejo.dump' | $podman exec -i $container pg_restore -U postgres -d $database --no-owner --no-acl --role=$owner" -counts=$("${remote[@]}" "$podman exec $container psql -U postgres -d $database -At -c 'SELECT (SELECT count(*) FROM \"user\"), (SELECT count(*) FROM repository)'") -[[ $counts == "$source_counts" ]] || { echo "Forgejo database counts differ; restore $backup" >&2; exit 1; } -echo "Imported evil.inc Forgejo app files and database; verified user/repository counts: $counts. Previous dataset: $snapshot. Previous database: $backup" diff --git a/tools/import-forgejo-issues.py b/tools/import-forgejo-issues.py deleted file mode 100644 index ea01b6f991d307b51f2e1537eb91a86255837a19..0000000000000000000000000000000000000000 --- a/tools/import-forgejo-issues.py +++ /dev/null @@ -1,512 +0,0 @@ -#!/usr/bin/env python3 -"""Import a verified personal Forgejo export into stopped Shale, without replacing native issues.""" -import argparse -from collections import Counter, defaultdict -from datetime import datetime, timezone -import hashlib -import grp -import json -import os -from pathlib import Path -import re -import shutil -import sqlite3 -import sys -import tempfile -import urllib.request -from urllib.parse import quote - -ALPHABET = '0123456789ABCDEFGHJKMNPQRSTVWXYZ' -EPOCH = 1577836800 # Shale's ULIDs use 2020-01-01, rather than the Unix epoch. -CLOSED = ('done', 'not_planned', 'duplicate', 'invalid') -EVENTS = ['comment', 'reopened', 'closed', 'issue reference', 'commit reference', - 'comment reference', 'pull request reference', 'label changed', 'milestone changed', - 'assignee changed', 'title changed', 'branch deleted', 'time tracking started', - 'time tracking stopped', 'time added', 'time tracking canceled', 'deadline added', - 'deadline changed', 'deadline removed', 'dependency added', 'dependency removed', - 'code comment', 'review', 'locked', 'unlocked', 'target branch changed', - 'time deleted', 'review requested', 'merged', 'pull request updated', - 'project changed', 'project column changed', 'review dismissed', 'reference changed', - 'automatic merge scheduled', 'automatic merge canceled', 'pinned', 'unpinned'] - - -def timestamp(epoch): - return datetime.fromtimestamp(int(epoch), timezone.utc).isoformat(timespec='seconds') - - -def identifier(kind, key, epoch): - milliseconds = (int(epoch) - EPOCH) * 1000 - if not 0 <= milliseconds < 2 ** 48: - raise ValueError('Source creation time is outside Shale ULID range') - entropy = int.from_bytes(hashlib.sha256(f'personal-forgejo:{kind}:{key}'.encode()).digest()[:10], 'big') - value = (milliseconds << 80) | entropy - return ''.join(ALPHABET[(value >> (5 * i)) & 31] for i in range(25, -1, -1)) - - -def digest(path): - return hashlib.sha256(path.read_bytes()).hexdigest() - - -def stopped(job): - token = os.environ.get('NOMAD_TOKEN') or Path('/var/lib/studio/nomad.token').read_text().strip() - def read(path): - request = urllib.request.Request('http://127.0.0.1:4646/v1/' + path, - headers={'X-Nomad-Token': token}) - with urllib.request.urlopen(request, timeout=10) as response: - return json.load(response) - if not read('job/' + job).get('Stop') or any( - item['ClientStatus'] in ('pending', 'running') for item in read('job/' + job + '/allocations')): - raise ValueError('Stop the target Shale job and wait for its allocations before importing') - - -def migrate(database, source, attachments, target, evidence, before_commit=lambda: None): - db = sqlite3.connect(database) - db.row_factory = sqlite3.Row - db.execute('PRAGMA foreign_keys=ON') - if db.execute('PRAGMA integrity_check').fetchone()[0] != 'ok': - raise ValueError('Shale SQLite integrity check failed') - source_hash = digest(source) - data = json.loads(source.read_text()) - for field in ['repositories', 'issues', 'comments', 'labels', 'issue_labels', 'authors', - 'attachments', 'assignees', 'milestones', 'pull_requests', 'history']: - if not isinstance(data.get(field), list): - raise ValueError(f'Missing source table: {field}') - owners = db.execute("SELECT * FROM users WHERE name='clover'").fetchall() - if len(owners) != 1 or owners[0]['snowflake'] != '8c909fab-98b0-4472-9171-9606ccebe9d5': - raise ValueError('Clover original identity is missing') - owner = owners[0]['id'] - repos = {r['name']: dict(r) for r in db.execute('SELECT * FROM repositories')} - sources = {r['id']: r for r in data['repositories']} - issue_sources = {r['id']: r for r in data['issues']} - if len(issue_sources) != len(data['issues']): - raise ValueError('Duplicate source issue IDs') - for r in data['repositories']: - if r['name'] not in repos or repos[r['name']]['owner'] != owner: - raise ValueError(f'Expected imported Clover repository: {r["name"]}') - if any(c['issue_id'] not in issue_sources for c in data['comments']): - raise ValueError('Orphan source comment') - with sqlite3.connect(evidence / 'before.db') as backup: - db.backup(backup) - originals = {table: [dict(r) for r in db.execute(f'SELECT * FROM {table}')] - for table in ['issues', 'issue_actions', 'issue_labels', 'issues__labels', 'users']} - db.execute('BEGIN IMMEDIATE') - db.execute('CREATE TABLE IF NOT EXISTS studio_forgejo_records(' - 'kind TEXT NOT NULL, source_id TEXT NOT NULL, target_id INTEGER, ' - 'source_json TEXT NOT NULL, PRIMARY KEY(kind,source_id))') - previous = db.execute("SELECT source_json FROM studio_forgejo_records WHERE kind='export' AND source_id='personal'").fetchone() - if previous and json.loads(previous[0])['sha256'] != source_hash: - raise ValueError('A different export was already imported; review the source before merging') - - def saved(kind, key): - row = db.execute('SELECT target_id FROM studio_forgejo_records WHERE kind=? AND source_id=?', (kind, str(key))).fetchone() - return row[0] if row else None - - def record(kind, key, target_id, row): - db.execute('INSERT OR IGNORE INTO studio_forgejo_records VALUES(?,?,?,?)', - (kind, str(key), target_id, json.dumps(row, ensure_ascii=False, sort_keys=True))) - - users = {1: owner} - names = {r['name'] for r in db.execute('SELECT name FROM users')} - actors = {int(r.get('poster_id') or 0) for r in [*data['issues'], *data['comments'], *data['history']]} - actors.update(int(r['assignee_id']) for r in data['assignees']) - author_source = {r['id']: r for r in data['authors']} - for key in sorted(actors - {1}): - row = author_source.get(key, {'id': key, 'name': 'forgejo-system' if key == 0 else f'forgejo-user-{key}', - 'created_unix': min(i['created_unix'] for i in data['issues']), - 'updated_unix': max(i['updated_unix'] for i in data['issues'])}) - existing = saved('user', key) - if existing is None: - name = row['name'] - if name in names: - name = 'forgejo-' + name - if name in names: - raise ValueError('Historical author name collision') - cursor = db.execute('INSERT INTO users(uuid,provider,snowflake,name,joined_on,last_updated) VALUES(?,?,?,?,?,?)', - (identifier('user', key, row['created_unix']), 'personal-forgejo.invalid', - str(key), name, timestamp(row['created_unix']), timestamp(row['updated_unix']))) - existing = cursor.lastrowid - names.add(name) - record('user', key, existing, row) - users[key] = existing - record('user', 1, owner, author_source[1]) - - external = {'paperclover': owner} - external_rows = defaultdict(list) - for row in [*data['issues'], *data['comments']]: - if row.get('original_author'): - external_rows[row['original_author']].append(row) - for name, rows in external_rows.items(): - if name == 'paperclover': - continue - if not re.fullmatch(r'[A-Za-z0-9_.-]+', name): - raise ValueError('Unsupported external author name') - key = 'external:' + name - existing = saved('user', key) - if existing is None: - display = name if name not in names else 'github-' + name - if display in names: - raise ValueError('External author name collision') - created = min(row['created_unix'] for row in rows) - updated = max(row['updated_unix'] or row['created_unix'] for row in rows) - existing = db.execute('INSERT INTO users(uuid,provider,snowflake,name,joined_on,last_updated) VALUES(?,?,?,?,?,?)', - (identifier('user', key, created), 'personal-forgejo-external.invalid', name, - display, timestamp(created), timestamp(updated))).lastrowid - names.add(display) - record('user', key, existing, {'original_author': name}) - external[name] = existing - - def actor(row): - return external[row['original_author']] if row.get('original_author') else users[row['poster_id']] - - labels = {} - for row in sorted(data['labels'], key=lambda r: r['id']): - if row['repo_id'] not in sources: - raise ValueError('Label has no source repository') - repo = repos[sources[row['repo_id']]['name']] - existing = saved('label', row['id']) - if existing is None: - color = row['color'].lstrip('#') - if not re.fullmatch('[0-9a-fA-F]{6}', color): - raise ValueError('Invalid source label color') - epoch = row['created_unix'] or sources[row['repo_id']]['created_unix'] - existing = db.execute('INSERT INTO issue_labels(uuid,repo,name,description,color,last_updated) VALUES(?,?,?,?,?,?)', - (identifier('label', row['id'], epoch), repo['id'], row['name'], row['description'], - '#' + color, timestamp(row['updated_unix'] or epoch))).lastrowid - record('label', row['id'], existing, row) - labels[row['id']] = existing - - # Preserve incoming numbers when free; collisions go above BOTH namespaces. - occupied = defaultdict(set) - ceilings = defaultdict(int) - for row in db.execute('SELECT repo,number FROM issues'): - occupied[row['repo']].add(row['number']) - ceilings[row['repo']] = max(ceilings[row['repo']], row['number']) - for row in data['issues']: - repo = repos[sources[row['repo_id']]['name']]['id'] - ceilings[repo] = max(ceilings[repo], row['index']) - issue_map = {} - collisions = [] - assignees = defaultdict(list) - for row in data['assignees']: - assignees[row['issue_id']].append(users[row['assignee_id']]) - planned = [] - for row in sorted(data['issues'], key=lambda r: (r['repo_id'], r['index'])): - repo = repos[sources[row['repo_id']]['name']] - existing = saved('issue', row['id']) - number = row['index'] - if existing is None: - if number in occupied[repo['id']]: - ceilings[repo['id']] += 1 - number = ceilings[repo['id']] - occupied[repo['id']].add(number) - else: - number = db.execute('SELECT number FROM issues WHERE id=?', (existing,)).fetchone()[0] - planned.append((repo, number, existing, row)) - # Shale allocates the next number from the issue with the highest row ID. - for repo, number, existing, row in sorted(planned, key=lambda item: (item[0]['id'], item[1])): - if existing is None: - existing = db.execute('INSERT INTO issues(uuid,repo,number,author,last_updated,title,status,assignee) VALUES(?,?,?,?,?,?,?,?)', - (identifier('issue', row['id'], row['created_unix']), repo['id'], number, - actor(row), timestamp(row['updated_unix']), row['name'], - 'done' if row['is_closed'] else 'todo', - assignees[row['id']][0] if len(assignees[row['id']]) == 1 else None)).lastrowid - record('issue', row['id'], existing, row) - if number != row['index']: - collisions.append({'repo': repo['name'], 'forgejo': row['index'], 'shale': number}) - issue_map[row['id']] = {'id': existing, 'number': number, 'repo': repo['name'], - 'url': '/' + quote(repo['name'], safe='/') + '/issues/' + str(number)} - - # Preserve attachment authorization by checking the associated Shale issue - # before Caddy serves the copied file. No unauthenticated static directory. - attachment_map = {} - manifest = [] - for row in data['attachments']: - if row['issue_id'] not in issue_map: - raise ValueError('An attachment belongs to an unsupported release or missing issue') - if row['external_url']: - attachment_map[row['uuid']] = row['external_url'] - record('attachment', row['id'], None, row) - continue - key = row['uuid'] - if not re.fullmatch('[0-9a-f-]{36}', key): - raise ValueError('Unsafe attachment UUID') - source_file = attachments / key[0] / key[1] / key - if source_file.is_symlink() or not source_file.is_file() or source_file.stat().st_size != row['size']: - raise ValueError('Source attachment missing or size differs') - name = Path(row['name']).name - if name != row['name'] or not name or any(c in name for c in '\\"\r\n'): - raise ValueError('Unsafe attachment filename') - destination = target / 'forgejo-attachments' / key / name - destination.parent.mkdir(parents=True, exist_ok=True) - if destination.exists() and digest(destination) != digest(source_file): - raise ValueError('Attachment copy differs') - if not destination.exists(): - shutil.copyfile(source_file, destination) - checksum = digest(source_file) - if digest(destination) != checksum: - raise ValueError('Attachment checksum failed') - # Caddy serves these only after Shale authorizes the associated issue. - # Keep them unavailable to other local users. - caddy_group = grp.getgrnam('caddy').gr_gid - for directory in [target / 'forgejo-attachments', destination.parent]: - os.chown(directory, 0, caddy_group) - directory.chmod(0o750) - os.chown(destination, 0, caddy_group) - destination.chmod(0o640) - path = '/-/forgejo-attachments/' + key + '/' + quote(name, safe='') - attachment_map[key] = path - manifest.append({'path': path, 'file': key + '/' + name, - 'issue': issue_map[row['issue_id']]['url'], 'sha256': checksum}) - record('attachment', row['id'], None, row) - - source_urls = {} - for row in data['issues']: - repo = sources[row['repo_id']] - owner_name = repo['owner_name'] - for kind in ['issues', 'pulls']: - source_urls[f'/{owner_name}/{repo["name"]}/{kind}/{row["index"]}'] = issue_map[row['id']]['url'] - - def rewrite(text, repo_id): - def attachment(match): - key = match.group(1) - if key not in attachment_map: - raise ValueError('Issue references an attachment absent from the source export') - return attachment_map[key] - text = re.sub(r'(?:https?://(?:git|forgejo)\.paperclover\.net)?/attachments/([0-9a-f-]{36})', attachment, text) - def link(match): - return source_urls.get(match.group(1), match.group(0)) - text = re.sub(r'https?://(?:git|forgejo)\.paperclover\.net(/[^\s)<>]+/(?:issues|pulls)/[0-9]+)', link, text) - text = re.sub(r'(?<=\]\()(/[^\s)<>]+/(?:issues|pulls)/[0-9]+)(?=\))', link, text) - # Leave code, quoted text, and bare #references untouched; map explicit links only. - return text - - def action(kind, key, issue_id, actor, payload, added, updated=None, source_row=None): - existing = saved(kind, key) - if existing is not None: - return existing - created = db.execute('INSERT INTO issue_actions(uuid,issue,actor,kind,payload,added_on,last_updated) VALUES(?,?,?,?,?,?,?)', - (identifier(kind, key, added), issue_id, actor, kind if kind in ['comment', 'update_status', 'add_label', 'remove_label'] else 'comment', - payload, timestamp(added), timestamp(updated or added))).lastrowid - record(kind, key, created, source_row or {}) - return created - - milestone_source = {r['id']: r for r in data['milestones']} - pulls = {r['issue_id']: r for r in data['pull_requests']} - body_edits = defaultdict(int) - for revision in data['history']: - if not revision['comment_id'] and not revision['is_first_created']: - body_edits[revision['issue_id']] = max(body_edits[revision['issue_id']], revision['edited_unix']) - for row in data['issues']: - item = issue_map[row['id']] - body = rewrite(row['content'], row['repo_id']) - metadata = [] - if item['number'] != row['index']: - metadata.append(f'Original Forgejo issue #{row["index"]}.') - if row['is_pull']: - pr = pulls[row['id']] - metadata.append(f'Imported pull request: `{pr["head_branch"]}` → `{pr["base_branch"]}`.' + - (f' Merged as `{pr["merged_commit_id"]}`.' if pr['has_merged'] else '')) - if row['milestone_id']: - metadata.append('Milestone: ' + milestone_source[row['milestone_id']]['name'] + '.') - if row['is_locked']: - metadata.append('The original discussion was locked.') - if len(assignees[row['id']]) > 1: - original_assignees = [author_source[a['assignee_id']]['name'] for a in data['assignees'] if a['issue_id'] == row['id']] - metadata.append('Original assignees: ' + ', '.join(original_assignees) + '.') - if metadata: - body += '\n\n---\n\n' + '\n\n'.join(metadata) - action('issue-body', row['id'], item['id'], actor(row), body, - row['created_unix'], max(row['created_unix'], body_edits[row['id']]), row) - # Shale's status-change renderer requires an initial status action. - # Forgejo stores initial state on the issue, rather than as a comment. - action('update_status', 'initial-' + str(row['id']), item['id'], actor(row), - 'todo', row['created_unix']) - - for row in sorted(data['comments'], key=lambda r: (r['created_unix'], r['id'])): - item = issue_map[row['issue_id']] - kind = 'comment' - payload = rewrite(row['content'], issue_sources[row['issue_id']]['repo_id']) - if row['type'] in [1, 2]: - kind, payload = 'update_status', 'todo' if row['type'] == 1 else 'done' - elif row['type'] == 7 and row['label_id'] in labels: - kind = 'add_label' if row['content'] == '1' else 'remove_label' - payload = str(labels[row['label_id']]) - elif row['type'] != 0: - event = EVENTS[row['type']] if row['type'] < len(EVENTS) else f'event {row["type"]}' - detail = [] - for field in ['old_title', 'new_title', 'old_ref', 'new_ref', 'commit_sha', 'commit_id', 'tree_path', 'line']: - if row.get(field): - detail.append(f'{field}: {row[field]}') - for field in ['old_milestone_id', 'milestone_id']: - if row.get(field): - detail.append(field + ': ' + milestone_source.get(row[field], {}).get('name', str(row[field]))) - if row.get('dependent_issue_id'): - linked = issue_map.get(row['dependent_issue_id']) - detail.append('Dependency: ' + (f'[{linked["repo"]}#{linked["number"]}]({linked["url"]})' if linked else str(row['dependent_issue_id']))) - if row.get('ref_issue_id') and row['ref_issue_id'] in issue_map: - linked = issue_map[row['ref_issue_id']] - detail.append(f'[{linked["repo"]}#{linked["number"]}]({linked["url"]})') - payload = 'Forgejo: ' + event + '.' + ('\n\n' + '\n\n'.join(detail) if detail else '') + ('\n\n' + payload if payload else '') - existing = saved('forgejo-comment', row['id']) - if existing is None: - # Every original comment/event maps to exactly one Shale action. - action_id = action(kind, 'forgejo-' + str(row['id']), item['id'], actor(row), payload, - row['created_unix'], row['updated_unix'], row) - record('forgejo-comment', row['id'], action_id, row) - - for row in data['issue_labels']: - item = issue_map[row['issue_id']] - existing = saved('issue-label', row['id']) - if existing is None: - label = labels[row['label_id']] - epoch = issue_sources[row['issue_id']]['updated_unix'] - existing = db.execute('INSERT INTO issues__labels(uuid,issue,label) VALUES(?,?,?)', - (identifier('issue-label', row['id'], epoch), item['id'], label)).lastrowid - record('issue-label', row['id'], existing, row) - - for table in ['history', 'milestones', 'pull_requests', 'assignees']: - for row in data[table]: - record(table, row['id'], None, row) - for source_repo in data['repositories']: - repo = repos[source_repo['name']] - if any(i['repo_id'] == source_repo['id'] for i in data['issues']): - db.execute("UPDATE repositories SET access_issues=? WHERE id=? AND access_issues='off'", - ('private' if source_repo['is_private'] else 'public', repo['id'])) - for field in ['access_issues_submit', 'access_issues_comment']: - db.execute(f"UPDATE repositories SET {field}='private' WHERE id=? AND {field}='off'", (repo['id'],)) - slots = ','.join('?' for _ in CLOSED) - db.execute(f'UPDATE repositories SET open_issues=(SELECT count(*) FROM issues WHERE repo=repositories.id AND status NOT IN ({slots}))', CLOSED) - db.execute(f'UPDATE issue_labels SET open_issues=(SELECT count(*) FROM issues__labels il JOIN issues i ON i.id=il.issue WHERE il.label=issue_labels.id AND i.status NOT IN ({slots}))', CLOSED) - for table, rows in originals.items(): - for row in rows: - current = dict(db.execute(f'SELECT * FROM {table} WHERE id=?', (row['id'],)).fetchone()) - if table == 'issue_labels': - current.pop('open_issues');row = {k:v for k,v in row.items() if k != 'open_issues'} - if current != row: - raise ValueError(f'Existing Shale {table} row changed') - if db.execute('PRAGMA foreign_key_check').fetchall(): - raise ValueError('Imported data has invalid foreign keys') - if db.execute('SELECT 1 FROM issues i WHERE i.id=(SELECT max(id) FROM issues WHERE repo=i.repo) ' - 'AND i.number<>(SELECT max(number) FROM issues WHERE repo=i.repo)').fetchone(): - raise ValueError('Issue row order would reuse a number; repair it before importing') - for kind, rows in [('issue', data['issues']), ('forgejo-comment', data['comments']), ('label', data['labels']), - ('issue-label', data['issue_labels']), ('attachment', data['attachments']), ('history', data['history'])]: - count = db.execute('SELECT count(*) FROM studio_forgejo_records WHERE kind=?', (kind,)).fetchone()[0] - if count != len(rows): - raise ValueError(f'Incomplete import: {kind}') - record('export', 'personal', None, {'sha256': source_hash}) - before_commit() - db.commit() - db.close() - report = {'source_sha256': source_hash, 'issues': len(data['issues']), 'comments_and_events': len(data['comments']), - 'labels': len(labels), 'attachments': len(manifest), 'history_revisions': len(data['history']), - 'number_collisions': collisions, 'issue_mapping': issue_map} - (evidence / 'report.json').write_text(json.dumps(report, indent=2) + '\n') - (target / 'forgejo-attachments.json').write_text(json.dumps(manifest, indent=2) + '\n') - print(json.dumps({k:v for k,v in report.items() if k != 'issue_mapping'}, indent=2)) - - -def repair_issue_order(database, evidence, before_commit): - with sqlite3.connect(database) as db: - db.row_factory = sqlite3.Row - db.execute('PRAGMA foreign_keys=ON') - with sqlite3.connect(evidence / 'before.db') as backup: - db.backup(backup) - db.execute('BEGIN IMMEDIATE') - db.execute('PRAGMA defer_foreign_keys=ON') - if db.execute('PRAGMA integrity_check').fetchone()[0] != 'ok' or db.execute('PRAGMA foreign_key_check').fetchall(): - raise ValueError('Shale database integrity failed') - if db.execute('SELECT 1 FROM issues GROUP BY repo,number HAVING count(*)>1').fetchone(): - raise ValueError('Duplicate issue numbers require separate review') - tables = [r[0] for r in db.execute("SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%'")] - if any(not re.fullmatch('[a-z_]+', table) for table in tables): - raise ValueError('Unexpected table name') - references = {table: [r['from'] for r in db.execute(f'PRAGMA foreign_key_list({table})') if r['table'] == 'issues'] for table in tables} - for table in tables: - for column in db.execute(f'PRAGMA table_info({table})'): - if re.fullmatch('(.*_)?issue(_id)?', column['name']) and column['name'] not in references[table]: - raise ValueError('Undeclared issue reference requires review') - imported = {r[0] for r in db.execute("SELECT target_id FROM studio_forgejo_records WHERE kind='issue'")} - swaps = {} - for repo in db.execute('SELECT DISTINCT repo FROM issues'): - newest = db.execute('SELECT id,number FROM issues WHERE repo=? ORDER BY id DESC LIMIT 1', repo).fetchone() - highest = db.execute('SELECT id,number FROM issues WHERE repo=? ORDER BY number DESC LIMIT 1', repo).fetchone() - if newest['number'] != highest['number']: - if newest['id'] not in imported or highest['id'] not in imported: - raise ValueError('Repair would change a native issue ID') - swaps[newest['id']], swaps[highest['id']] = highest['id'], newest['id'] - - def contents(): - identities = dict(db.execute('SELECT id,uuid FROM issues')) - result = {} - for table in tables: - rows = [] - for record in db.execute(f'SELECT * FROM {table}'): - row = dict(record) - if table == 'issues': - row['id'] = identities[row['id']] if row['id'] in imported else row['id'] - for column in references[table]: - row[column] = identities[row[column]] - if table == 'studio_forgejo_records' and row['kind'] == 'issue': - row['target_id'] = identities[row['target_id']] - rows.append(tuple(row.items())) - result[table] = Counter(rows) - return result - - before = contents() - for old, new in [*((old, -old) for old in swaps), *((-old, new) for old, new in swaps.items())]: - db.execute('UPDATE issues SET id=? WHERE id=?', (new, old)) - for table, columns in references.items(): - for column in columns: - db.execute(f'UPDATE {table} SET {column}=? WHERE {column}=?', (new, old)) - db.execute("UPDATE studio_forgejo_records SET target_id=? WHERE kind='issue' AND target_id=?", (new, old)) - if before != contents() or db.execute('PRAGMA foreign_key_check').fetchall(): - raise ValueError('Repair changed issue content or references') - before_commit() - db.commit() - report = {'surrogate_id_swaps': swaps, 'all_table_contents_preserved': True} - (evidence / 'report.json').write_text(json.dumps(report, indent=2) + '\n') - print(json.dumps(report)) - - -def main(): - parser = argparse.ArgumentParser() - parser.add_argument('--target', type=Path, required=True) - parser.add_argument('--source', type=Path, required=True) - parser.add_argument('--proof', type=Path, help='Verified export proof; defaults to source-proof.json beside the source') - parser.add_argument('--attachments', type=Path, default=Path('/mnt/storage1/apps/forgejo/work/attachments')) - parser.add_argument('--rehearsal', action='store_true') - parser.add_argument('--repair-issue-order', action='store_true') - args = parser.parse_args() - if os.geteuid() != 0: - parser.error('Run as root on Zenith') - os.umask(0o077) - target = args.target.resolve() - if 'evil' in str(target).lower() or target.is_relative_to('/mnt/storage1/apps'): - parser.error('Use a managed Shale target or isolated copy') - if args.rehearsal: - if target.is_relative_to('/srv/prod'): - parser.error('Rehearsal must use an isolated target') - elif target != Path('/srv/prod/shale'): - parser.error('Production imports must target /srv/prod/shale') - guard = lambda: None - if target == Path('/srv/prod/shale'): - guard = lambda: stopped('shale') - elif target.parent == Path('/srv/staging'): - if not re.fullmatch(r'shale-preview-[0-9a-f]{8}', target.name): - parser.error('Expected a managed Shale stage') - guard = lambda: stopped(target.name) - proof = args.proof or args.source.parent / 'source-proof.json' - if not proof.is_file() or json.loads(proof.read_text()).get('sourceSha256') != digest(args.source): - parser.error('Verified export proof is missing or the source checksum differs') - guard() - evidence = Path(tempfile.mkdtemp(prefix='issue-import-', dir=str(args.source.parent))) - if args.repair_issue_order: - repair_issue_order(target / 'data/astheno.shale.db', evidence, guard) - else: - migrate(target / 'data/astheno.shale.db', args.source, args.attachments, target, evidence, guard) - print('Evidence:', evidence) - - -if __name__ == '__main__': - main() diff --git a/tools/import-forgejo-to-shale.py b/tools/import-forgejo-to-shale.py deleted file mode 100644 index d9e93fab91054b5c9d0827169793706d3ae112fb..0000000000000000000000000000000000000000 --- a/tools/import-forgejo-to-shale.py +++ /dev/null @@ -1,240 +0,0 @@ -#!/usr/bin/env python3 -import argparse -from datetime import datetime, timezone -import hashlib -import json -import os -from pathlib import Path -import re -import secrets -import shutil -import sqlite3 -import tempfile -import urllib.error -import urllib.request - - -SOURCE = Path('/mnt/storage1/apps/forgejo/git/repositories') -ACCESS = ('access_git_webui', 'access_git_proto', 'access_git_proto_push', 'access_issues', - 'access_issues_submit', 'access_issues_comment', 'access_readme') - - -def sha256(path): - digest = hashlib.sha256() - with path.open('rb') as source: - for block in iter(lambda: source.read(1024 * 1024), b''): - digest.update(block) - return digest.hexdigest() - - -def refs(repository): - result = {} - packed = repository / 'packed-refs' - if packed.exists(): - for line in packed.read_text().splitlines(): - if line and line[0] not in '#^': - value, name = line.split(' ') - result[name] = value - for path in (repository / 'refs').rglob('*'): - if path.is_symlink(): - raise ValueError('Repository refs contain a link. Resolve it before importing') - if path.is_file(): - result[str(path.relative_to(repository))] = path.read_text().strip() - for name, value in result.items(): - if not name.startswith('refs/') or any(part in ('', '.', '..') for part in name.split('/')): - raise ValueError('Repository has an unsafe ref name. Review its retained refs') - if not re.fullmatch('[0-9a-f]{40}', value): - raise ValueError('Repository has a symbolic or unsupported ref. Resolve it before importing') - return result - - -def stopped(): - token = os.environ.get('NOMAD_TOKEN') or Path('/var/lib/studio/nomad.token').read_text().strip() - def read(path): - request = urllib.request.Request('http://127.0.0.1:4646/v1/' + path, - headers={'X-Nomad-Token': token}) - with urllib.request.urlopen(request, timeout=10) as response: - return json.load(response) - try: - job = read('job/shale') - except urllib.error.HTTPError as error: - if error.code == 404: - return - raise - if not job.get('Stop') or any(item['ClientStatus'] in ('pending', 'running') - for item in read('job/shale/allocations')): - raise ValueError('Stop Shale and wait for its allocations before importing') - - -def main(): - parser = argparse.ArgumentParser(description='Import retained personal Forgejo histories into stopped Shale.') - parser.add_argument('--target', type=Path, required=True, help='Copied Shale service root') - parser.add_argument('--metadata', type=Path, required=True, help='Personal Forgejo repository metadata JSON') - args = parser.parse_args() - if os.geteuid() != 0: - parser.error('Run as root on Zenith') - os.umask(0o077) - target = args.target.resolve() - retained = Path('/mnt/storage1/apps') - if target.is_relative_to(retained) or retained.is_relative_to(target): - raise ValueError('Target overlaps retained app data. Use the managed Shale copy') - database = target / 'data/astheno.shale.db' - if not database.is_file(): - raise ValueError('Shale database is missing. Import its retained state first') - metadata = json.loads(args.metadata.read_text()) - required = {'owner_name', 'lower_name', 'name', 'is_private', 'default_branch', 'is_empty', 'is_mirror'} - if not isinstance(metadata, list) or any(not isinstance(row, dict) or not required.issubset(row) for row in metadata): - raise ValueError('Forgejo metadata is incomplete. Export the documented personal repository fields') - if any(not re.fullmatch(r'[A-Za-z0-9_.-]+', row[field]) or row[field] in ('.', '..') - for row in metadata for field in ('owner_name', 'lower_name', 'name')): - raise ValueError('Repository metadata has unsafe names. Review the personal Forgejo export') - indexed = {(row['owner_name'], row['lower_name']): row for row in metadata} - if len(indexed) != len(metadata) or any(type(row[field]) is not bool for row in metadata - for field in ('is_private', 'is_empty', 'is_mirror')): - raise ValueError('Forgejo metadata has duplicate repositories or missing visibility') - sources = sorted(SOURCE.glob('*/*.git')) - if not sources: - raise ValueError('No retained personal Forgejo repositories were found') - if any(path.is_file() for path in (SOURCE.parent / 'lfs').rglob('*')): - raise ValueError('Forgejo has LFS data. Preserve its serving path before migrating repositories') - for source in sources: - if source.is_symlink() or (source.parent.name, source.name[:-4]) not in indexed: - raise ValueError('A retained repository has no verified metadata. Export its personal Forgejo database entry') - if any((source / folder).is_symlink() for folder in ('objects', 'refs')): - raise ValueError('Repository storage contains a link. Preserve its contents before importing') - if (source / 'shallow').exists() or (source / 'objects/info/alternates').exists(): - raise ValueError('Repository history depends on external objects. Preserve them before importing') - if any((source / 'objects/pack').glob('*.promisor')): - raise ValueError('Repository has promised objects. Retrieve the complete history before importing') - expected_paths = {(row['owner_name'], row['lower_name']) for row in metadata if not row['is_empty']} - if not expected_paths.issubset({(source.parent.name, source.name[:-4]) for source in sources}): - raise ValueError('A nonempty Forgejo repository is missing from retained storage') - stopped() - evidence = Path(tempfile.mkdtemp(prefix='shale-forgejo-import-', dir='/var/lib/studio')) - db = sqlite3.connect(database) - db.row_factory = sqlite3.Row - if db.execute('PRAGMA integrity_check').fetchone()[0] != 'ok': - raise ValueError('Shale database failed integrity checking. Recover the retained copy') - owners = db.execute("SELECT id FROM users WHERE name='snow'").fetchall() - if len(owners) != 1: - raise ValueError('Shale needs one existing snow account. Preserve its original identity before importing') - owner = owners[0]['id'] - stat = database.stat() - with sqlite3.connect(evidence / 'before.db') as backup: - db.backup(backup) - existing = {row['name'].casefold(): dict(row) for row in db.execute( - 'SELECT id,uuid,owner,name,' + ','.join(ACCESS) + ' FROM repositories')} - records = [] - now = datetime.now(timezone.utc).isoformat(timespec='seconds') - try: - for key, row in indexed.items(): - name = row['name'] if key[0] == 'clo' else key[0] + '/' + row['name'] - entry = existing.get(name.casefold()) - if entry and entry['owner'] != owner: - raise ValueError('Repository belongs to another Shale account. Review its ownership before merging') - if entry and row['is_private']: - for field in ACCESS: - db.execute(f"UPDATE repositories SET {field}='private' WHERE id=? AND {field} IN ('public','unlisted')", (entry['id'],)) - db.commit() - db.execute('BEGIN IMMEDIATE') - for key, row in sorted(indexed.items()): - source = SOURCE / key[0] / (key[1] + '.git') - name = row['name'] if key[0] == 'clo' else key[0] + '/' + row['name'] - entry = existing.get(name.casefold()) - if entry: - directory = target / 'repositories_owned' / entry['uuid'] - if not directory.is_dir(): - raise ValueError('An existing Shale repository directory is missing. Recover it before merging') - else: - alphabet = '0123456789ABCDEFGHJKMNPQRSTVWXYZ' - number = secrets.randbits(128) - repo_id = ''.join(alphabet[(number >> (5 * position)) & 31] for position in range(25, -1, -1)) - directory = target / 'repositories_owned' / repo_id - directory.mkdir(mode=0o700) - os.chown(directory, stat.st_uid, stat.st_gid) - for folder in ('objects', 'refs'): - (directory / folder).mkdir(mode=0o700) - os.chown(directory / folder, stat.st_uid, stat.st_gid) - (directory / 'config').write_text('[core]\nrepositoryformatversion = 0\nbare = true\n[http]\nreceivepack = true\n') - access = 'private' if row['is_private'] else 'public' - cursor = db.execute( - 'INSERT INTO repositories(uuid,owner,created_on,name,description,' + ','.join(ACCESS) + ',last_updated) ' - 'VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?)', - (repo_id, owner, now, name, '', access, access, 'private', 'off', 'off', 'off', access, now)) - entry = {'id': cursor.lastrowid, 'uuid': repo_id, 'name': name} - existing[name.casefold()] = entry - previous = refs(directory) - original = refs(source) if source.exists() else {} - objects = {} - for path in sorted((source / 'objects').rglob('*')) if source.exists() else (): - if path.is_symlink(): - raise ValueError('Repository objects contain a link. Preserve its contents before importing') - relative = path.relative_to(source / 'objects') - if not path.is_file() or relative.parts[0] == 'info': - continue - checksum = sha256(path) - destination = directory / 'objects' / relative - destination.parent.mkdir(mode=0o700, parents=True, exist_ok=True) - if destination.exists(): - if sha256(destination) != checksum: - raise ValueError('Object files differ under the same name. Keep Shale stopped and review the copies') - else: - shutil.copyfile(path, destination) - os.chmod(destination, 0o600) - os.chown(destination, stat.st_uid, stat.st_gid) - if sha256(destination) != checksum: - raise RuntimeError('Copied objects failed verification. Keep Shale stopped') - objects[str(relative)] = checksum - mapped = {} - combined = dict(previous) - for reference, value in original.items(): - destination = reference - if destination in combined and combined[destination] != value: - category = 'heads' if reference.startswith('refs/heads/') else 'tags' if reference.startswith('refs/tags/') else None - destination = ('refs/' + category + '/forgejo/' + key[0] + '/' + reference.split('/', 2)[2] - if category else 'refs/forgejo/' + key[0] + '/' + reference[5:]) - if destination in combined and combined[destination] != value: - destination += '-'+value - path = directory / destination - path.parent.mkdir(mode=0o700, parents=True, exist_ok=True) - path.write_text(value + '\n') - os.chown(path, stat.st_uid, stat.st_gid) - combined[destination] = value - mapped[reference] = destination - if not (directory / 'HEAD').exists(): - desired = 'refs/heads/' + row['default_branch'] - if original and desired not in original: - desired = next((reference for reference in sorted(original) if reference.startswith('refs/heads/')), desired) - (directory / 'HEAD').write_text('ref: ' + desired + '\n') - os.chown(directory / 'HEAD', stat.st_uid, stat.st_gid) - os.chown(directory / 'config', stat.st_uid, stat.st_gid) - final = refs(directory) - if any(final.get(reference) != value for reference, value in previous.items()) or any( - final.get(mapped[reference]) != value for reference, value in original.items() - ): - raise RuntimeError('Ref verification failed. Keep Shale stopped and review the retained refs') - if source.exists() and refs(source) != original: - raise RuntimeError('Source refs changed during import. Keep Shale stopped') - for parent in (directory / 'objects', directory / 'refs'): - for path in parent.rglob('*'): - if path.is_dir(): - os.chown(path, stat.st_uid, stat.st_gid) - records.append({'source': '/'.join(key), 'target': entry['name'], 'uuid': entry['uuid'], - 'private': row['is_private'], 'sourceMirror': row['is_mirror'], 'objects': objects, - 'originalRefs': original, 'refMapping': mapped, 'previousRefs': previous, - 'head': (directory / 'HEAD').read_text().strip()}) - stopped() - if db.execute('PRAGMA foreign_key_check').fetchall(): - raise RuntimeError('Shale database references are invalid. Keep it stopped') - db.commit() - finally: - db.close() - (evidence / 'repositories.json').write_text(json.dumps(records, indent=2) + '\n') - print(f'Imported and verified {len(records)} personal repositories. Evidence and previous database: {evidence}') - - -if __name__ == '__main__': - try: - main() - except (ValueError, RuntimeError, OSError, sqlite3.Error, urllib.error.HTTPError) as error: - raise SystemExit(str(error)) from None diff --git a/tools/import-hedgedoc.sh b/tools/import-hedgedoc.sh deleted file mode 100644 index 5ccd1ac787b423af85539313a351a00c7ad8c9e2..0000000000000000000000000000000000000000 --- a/tools/import-hedgedoc.sh +++ /dev/null @@ -1,123 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -instance=${1:?usage: tools/import-hedgedoc.sh evil-hedgedoc[-preview-XXXXXXXX]} -[[ $instance == evil-hedgedoc || $instance =~ ^evil-hedgedoc-preview-[0-9a-f]{8}$ ]] || { echo 'Expected a HedgeDoc service or preview ID' >&2; exit 1; } -if [[ $instance == evil-hedgedoc ]]; then - root=/srv/prod/evil-hedgedoc -else - root="/srv/staging/$instance" -fi - -source_host=${STUDIO_MIGRATION_SOURCE:-zenith} -target_host=${STUDIO_DEPLOY_HOST:-root@127.0.0.1} -target_port=${STUDIO_DEPLOY_PORT:-2222} -remote=(ssh -p "$target_port" "$target_host") -path="nomad/jobs/$instance/inputs/database" -handoff=${STUDIO_LEGACY_HANDOFF:-} -if [[ -n $handoff ]]; then - [[ $instance == evil-hedgedoc && -n ${STUDIO_DEPLOY_HOST:-} ]] || { - echo 'Offline handoff requires production HedgeDoc and a target' >&2 - exit 1 - } - sh "$(dirname "$0")/check-legacy-handoff.sh" "$handoff" "$target_host" "$target_port" - entry=$("${remote[@]}" "python3 - '$handoff' evil-hedgedoc" < "$(dirname "$0")/verify-legacy-dump.py") - source_counts=$(python3 -c 'import json,sys; print("|".join(map(str,json.load(sys.stdin)["counts"])))' <<<"$entry") - upload_host=$target_host - upload_rsh="ssh -p $target_port" -else - upload_host=$source_host - upload_rsh=ssh -fi - -if [[ $instance != evil-hedgedoc ]]; then - "${remote[@]}" "test -f /var/lib/studio/stages/$instance.json" -else - [[ -n ${STUDIO_DEPLOY_HOST:-} ]] || { echo 'Set STUDIO_DEPLOY_HOST to the production target' >&2; exit 1; } - if [[ -z $handoff ]]; then - source_state=$(ssh "$source_host" "sudo -n docker ps -a --filter label=com.docker.compose.service=evil-hedgedoc --format '{{.State}}'") - [[ $source_state == exited ]] || { echo 'Stop Zenith evil-hedgedoc before importing production data' >&2; exit 1; } - fi - job_response=$("${remote[@]}" 'curl -s -w "\n%{http_code}" -H "X-Nomad-Token: $(cat /var/lib/studio/nomad.token)" http://127.0.0.1:4646/v1/job/evil-hedgedoc') - job_status=${job_response##*$'\n'} - if [[ $job_status == 200 ]]; then - stopped=$(python3 -c 'import json,sys; print(str(json.load(sys.stdin)["Stop"]).lower())' <<<"${job_response%$'\n'*}") - [[ $stopped == true ]] || { echo 'Stop production HedgeDoc before importing' >&2; exit 1; } - for _ in {1..30}; do - running=$("${remote[@]}" 'curl -s -H "X-Nomad-Token: $(cat /var/lib/studio/nomad.token)" http://127.0.0.1:4646/v1/job/evil-hedgedoc/allocations' | python3 -c 'import json,sys; print(sum(x["ClientStatus"] == "running" for x in json.load(sys.stdin)))') - [[ $running == 0 ]] && break - sleep 2 - done - [[ $running == 0 ]] || { echo 'Production HedgeDoc allocation did not stop' >&2; exit 1; } - elif [[ $job_status != 404 ]]; then - echo "Could not verify production job state: $job_status" >&2 - exit 1 - fi -fi -"${remote[@]}" "test \$(findmnt -n -o FSTYPE --mountpoint $root) = zfs" -db_json=$("${remote[@]}" "NOMAD_TOKEN=\$(cat /var/lib/studio/nomad.token) nomad var get -out json $path") -read -r database owner < <(python3 -c 'import json,sys; d=json.load(sys.stdin)["Items"]; print(d["name"], d["username"])' <<<"$db_json") -if [[ $instance == evil-hedgedoc ]]; then - [[ $database == evil_hedgedoc && $owner == svc_evil_hedgedoc ]] || { echo 'Unexpected production database' >&2; exit 1; } -else - [[ $database =~ ^evil_hedgedoc_s_[0-9a-f]{8}$ && $owner =~ ^svc_evil_hedgedoc_s_[0-9a-f]{8}$ ]] || { echo 'Unexpected preview database' >&2; exit 1; } -fi - -alloc_json=$("${remote[@]}" 'NOMAD_TOKEN=$(cat /var/lib/studio/nomad.token) nomad job allocs -json postgres') -allocation=$(python3 -c 'import json,sys; ids=[x["ID"] for x in json.load(sys.stdin) if x["ClientStatus"] == "running" and x["DesiredStatus"] == "run"]; assert len(ids) == 1; print(ids[0])' <<<"$alloc_json") -container="app-$allocation" -podman='podman --url unix:///run/podman/podman.sock' -scratch=$(mktemp -d) -trap 'rm -rf "$scratch"' EXIT -rsync -a -e "$upload_rsh" "$upload_host:/mnt/storage1/apps/evil-infra/hedgedoc/" "$scratch/uploads/" -drift=$(rsync -rnc --delete --out-format='%n' -e "$upload_rsh" "$upload_host:/mnt/storage1/apps/evil-infra/hedgedoc/" "$scratch/uploads/") -[[ -z $drift ]] || { echo 'Source uploads changed during copy' >&2; exit 1; } - -if [[ $instance != evil-hedgedoc ]]; then - "${remote[@]}" "NOMAD_TOKEN=\$(cat /var/lib/studio/nomad.token) nomad job stop -yes $instance" - for _ in {1..30}; do - running=$("${remote[@]}" "NOMAD_TOKEN=\$(cat /var/lib/studio/nomad.token) nomad job allocs -json $instance" | python3 -c 'import json,sys; print(sum(x["ClientStatus"] == "running" for x in json.load(sys.stdin)))') - [[ $running == 0 ]] && break - sleep 2 - done - [[ $running == 0 ]] || { echo 'Preview did not stop' >&2; exit 1; } -fi - -uploads="$root/hedgedoc/public/uploads" -rsync -a --delete -e "ssh -p $target_port" "$scratch/uploads/" "$target_host:$uploads/" -drift=$(rsync -rnc --delete --out-format='%n' -e "ssh -p $target_port" "$scratch/uploads/" "$target_host:$uploads/") -[[ -z $drift ]] || { echo 'Target uploads differ from source copy' >&2; exit 1; } -uid=$("${remote[@]}" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/identities.json\"))[\"evil-hedgedoc\"])'") -"${remote[@]}" "chown -R $uid:$uid $uploads; chmod 750 $uploads" - -backup="/var/lib/studio/$instance-before-import.dump" -"${remote[@]}" "umask 077; $podman exec $container pg_dump -U postgres -Fc $database > $backup; test -s $backup" -"${remote[@]}" "$podman exec $container psql -U postgres -d $database -v ON_ERROR_STOP=1 -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public AUTHORIZATION $owner;' >/dev/null" - -if [[ -n $handoff ]]; then - "${remote[@]}" "set -o pipefail; cat '$handoff/evil-hedgedoc.dump' | $podman exec -i $container pg_restore -U postgres -d $database --no-owner --no-acl --role=$owner" -else - ssh "$source_host" 'sudo -n docker exec postgres pg_dump -U postgres -Fc evil-hedgedoc' | - "${remote[@]}" "$podman exec -i $container pg_restore -U postgres -d $database --no-owner --no-acl --role=$owner" -fi - -counts_sql='SELECT (SELECT count(*) FROM "Notes"), (SELECT count(*) FROM "Users"), (SELECT count(*) FROM "Revisions"), (SELECT count(*) FROM "Authors");' -if [[ -z $handoff ]]; then - source_counts=$(printf '%s\n' "$counts_sql" | ssh "$source_host" sudo -n docker exec -i postgres psql -U postgres -d evil-hedgedoc -At) -fi -target_counts=$(printf '%s\n' "$counts_sql" | "${remote[@]}" "$podman exec -i $container psql -U postgres -d $database -At") -[[ $source_counts == "$target_counts" ]] || { echo 'Restored HedgeDoc record counts differ from Zenith' >&2; exit 1; } -echo "Verified HedgeDoc Notes, Users, Revisions, and Authors counts: $target_counts" - -if [[ $instance == evil-hedgedoc ]]; then - if [[ -z $handoff ]]; then - [[ $(ssh "$source_host" "sudo -n docker ps -a --filter label=com.docker.compose.service=evil-hedgedoc --format '{{.State}}'") == exited ]] || { - echo 'Zenith evil-hedgedoc restarted during import; leave the home server stopped' >&2 - exit 1 - } - fi - echo "Production data imported; promote a tested HedgeDoc preview. Previous database: $backup" -else - python3 "$(dirname "$0")/deploy.py" stage evil-hedgedoc - echo "Preview database imported; previous preview dump: $backup" -fi diff --git a/tools/import-jackett.sh b/tools/import-jackett.sh deleted file mode 100644 index c57c86f9778266faad70deed9a0e2f43b2eb8928..0000000000000000000000000000000000000000 --- a/tools/import-jackett.sh +++ /dev/null @@ -1,100 +0,0 @@ -#!/bin/sh -set -eu - -instance=${1:?usage: tools/import-jackett.sh jackett|jackett-preview-XXXXXXXX} -case $instance in - jackett) - test -n "${STUDIO_DEPLOY_HOST:-}" || { echo 'Set STUDIO_DEPLOY_HOST to the production target' >&2; exit 1; } - production=true - ;; - jackett-preview-*) - printf '%s\n' "$instance" | grep -Eq '^jackett-preview-[0-9a-f]{8}$' || { - echo 'Expected a Jackett preview ID' >&2; exit 1; - } - production=false - ;; - *) echo 'Expected jackett or its preview ID' >&2; exit 1 ;; -esac - -source_host=${STUDIO_MIGRATION_SOURCE:-zenith} -target_host=${STUDIO_DEPLOY_HOST:-root@127.0.0.1} -target_port=${STUDIO_DEPLOY_PORT:-2222} -offline=false -copy_host=$source_host -copy_rsh=ssh -if [ -n "${STUDIO_LEGACY_HANDOFF:-}" ]; then - test "$production" = true || { echo 'Offline handoff requires production Jackett' >&2; exit 1; } - sh "$(dirname "$0")/check-legacy-handoff.sh" "$STUDIO_LEGACY_HANDOFF" "$target_host" "$target_port" - offline=true - copy_host=$target_host - copy_rsh="ssh -p $target_port" -fi -if [ "$production" = true ]; then - root=/srv/prod/jackett/config/Jackett - mount=/srv/prod/jackett - source_root=/mnt/storage1/apps/jackett - if [ "$offline" = false ]; then - test "$(ssh "$source_host" "sudo -n docker inspect -f '{{.State.Running}}' jackett")" = false || { - echo 'Stop Zenith jackett before importing production data' >&2; exit 1; - } - fi - response=$(ssh -p "$target_port" "$target_host" 'curl -s -w "\n%{http_code}" -H "X-Nomad-Token: $(cat /var/lib/studio/nomad.token)" http://127.0.0.1:4646/v1/job/jackett') - status=$(printf '%s\n' "$response" | tail -n 1) - test "$status" = 200 || { echo "Could not verify jackett on the home server: $status" >&2; exit 1; } - stopped=$(printf '%s\n' "$response" | sed '$d' | python3 -c 'import json,sys; print(str(json.load(sys.stdin)["Stop"]).lower())') - test "$stopped" = true || { echo 'Stop jackett on the home server before importing production data' >&2; exit 1; } -else - root="/srv/staging/$instance/config/Jackett" - mount="/srv/staging/$instance" - source_id=$(ssh -p "$target_port" "$target_host" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/stages/$instance.json\"))[\"sourceId\"])'") - test "$source_id" = jackett || { echo 'Preview belongs to another service' >&2; exit 1; } - source_snapshot=$(ssh "$source_host" '/usr/sbin/zfs list -H -t snapshot -o name -s creation -r storage1/apps | tail -n 1') - case $source_snapshot in storage1/apps@*) ;; *) echo 'No apps ZFS snapshot is available' >&2; exit 1 ;; esac - source_root="/mnt/storage1/apps/.zfs/snapshot/${source_snapshot#*@}/jackett" -fi - -dataset=$(ssh -p "$target_port" "$target_host" "findmnt -n -o SOURCE --mountpoint $mount") -if [ "$production" = true ]; then - case $dataset in */prod/jackett) ;; *) echo 'Production dataset is not mounted' >&2; exit 1 ;; esac -else - case $dataset in */staging/"$instance") ;; *) echo 'Preview dataset is not mounted' >&2; exit 1 ;; esac -fi - -scratch=$(mktemp -d) -trap 'rm -rf "$scratch"' EXIT -rsync -a --exclude='/log.txt*' -e "$copy_rsh" "$copy_host:$source_root/" "$scratch/" -python3 - "$scratch" <<'PY' -import json -from pathlib import Path -import sys - -root = Path(sys.argv[1]) -config = json.loads((root / "ServerConfig.json").read_text()) -if not config.get("APIKey") or not list((root / "Indexers").glob("*.json")): - raise ValueError("Jackett configuration is incomplete") -PY - -if [ "$production" = false ]; then - ssh -p "$target_port" "$target_host" "NOMAD_TOKEN=\$(cat /var/lib/studio/nomad.token) nomad job stop -yes $instance" -fi -snapshot="$dataset@before-jackett-import-$(date +%s)-$$" -ssh -p "$target_port" "$target_host" "set -eu; for i in \$(seq 1 30); do running=\$(curl -fsS -H \"X-Nomad-Token: \$(cat /var/lib/studio/nomad.token)\" http://127.0.0.1:4646/v1/job/$instance/allocations | python3 -c 'import json,sys; print(sum(a[\"ClientStatus\"] == \"running\" for a in json.load(sys.stdin)))'); test \"\$running\" = 0 && break; sleep 2; done; test \"\$running\" = 0; zfs snapshot $snapshot" - -rsync -a --delete -e "ssh -p $target_port" "$scratch/" "$target_host:$root/" -owner=$(ssh -p "$target_port" "$target_host" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/identities.json\"))[\"jackett\"])'") -ssh -p "$target_port" "$target_host" "chown -R $owner:$owner $root" -drift=$(rsync -rlnc --delete --out-format='%n' -e "ssh -p $target_port" "$scratch/" "$target_host:$root/") -test -z "$drift" || { echo "Jackett copy differs from source; restore $snapshot" >&2; exit 1; } - -if [ "$production" = true ]; then - if [ "$offline" = false ]; then - test "$(ssh "$source_host" "sudo -n docker inspect -f '{{.State.Running}}' jackett")" = false || { - echo 'Zenith jackett restarted during import; leave the home server stopped' >&2; exit 1; - } - fi - echo "Imported production Jackett. Previous dataset state: $snapshot" -else - python3 "$(dirname "$0")/deploy.py" stage jackett - ssh -p "$target_port" "$target_host" "zfs destroy $snapshot" - echo "Imported and tested $instance from $source_snapshot" -fi diff --git a/tools/import-jellyfin.sh b/tools/import-jellyfin.sh deleted file mode 100644 index e08f394ec17a71f71074bf96107622367156cd64..0000000000000000000000000000000000000000 --- a/tools/import-jellyfin.sh +++ /dev/null @@ -1,103 +0,0 @@ -#!/bin/sh -set -eu - -instance=${1:?usage: tools/import-jellyfin.sh jellyfin|jellyfin-preview-XXXXXXXX} -case $instance in - jellyfin) - test -n "${STUDIO_DEPLOY_HOST:-}" || { echo 'Set STUDIO_DEPLOY_HOST to the production target' >&2; exit 1; } - production=true - ;; - jellyfin-preview-*) - printf '%s\n' "$instance" | grep -Eq '^jellyfin-preview-[0-9a-f]{8}$' || { - echo 'Expected a Jellyfin preview ID' >&2; exit 1; - } - production=false - ;; - *) echo 'Expected jellyfin or its preview ID' >&2; exit 1 ;; -esac - -source_host=${STUDIO_MIGRATION_SOURCE:-zenith} -target_host=${STUDIO_DEPLOY_HOST:-root@127.0.0.1} -target_port=${STUDIO_DEPLOY_PORT:-2222} -offline=false -if [ -n "${STUDIO_LEGACY_HANDOFF:-}" ]; then - test "$production" = true || { echo 'Offline handoff requires production Jellyfin' >&2; exit 1; } - sh "$(dirname "$0")/check-legacy-handoff.sh" "$STUDIO_LEGACY_HANDOFF" "$target_host" "$target_port" - offline=true -fi -if [ "$production" = true ]; then - root=/srv/prod/jellyfin/config - mount=/srv/prod/jellyfin - source_root=/mnt/storage1/apps/jellyfin - if [ "$offline" = false ]; then - test "$(ssh "$source_host" "sudo -n docker inspect -f '{{.State.Running}}' jellyfin")" = false || { - echo 'Stop Zenith jellyfin before importing production data' >&2; exit 1; - } - fi - response=$(ssh -p "$target_port" "$target_host" 'curl -s -w "\n%{http_code}" -H "X-Nomad-Token: $(cat /var/lib/studio/nomad.token)" http://127.0.0.1:4646/v1/job/jellyfin') - status=$(printf '%s\n' "$response" | tail -n 1) - test "$status" = 200 || { echo "Could not verify jellyfin on the home server: $status" >&2; exit 1; } - stopped=$(printf '%s\n' "$response" | sed '$d' | python3 -c 'import json,sys; print(str(json.load(sys.stdin)["Stop"]).lower())') - test "$stopped" = true || { echo 'Stop jellyfin on the home server before importing production data' >&2; exit 1; } - media_fs=$(ssh -p "$target_port" "$target_host" 'findmnt -n -o FSTYPE --mountpoint /srv/clover/Media') - test "$media_fs" = zfs || { echo 'Mount the real Media ZFS dataset before importing production data' >&2; exit 1; } - source_label=stopped-live-source - if [ "$offline" = true ]; then source_label=mounted-old-apps; fi -else - root="/srv/staging/$instance/config" - mount="/srv/staging/$instance" - source_id=$(ssh -p "$target_port" "$target_host" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/stages/$instance.json\"))[\"sourceId\"])'") - test "$source_id" = jellyfin || { echo 'Preview belongs to another service' >&2; exit 1; } - source_snapshot=$(ssh "$source_host" '/usr/sbin/zfs list -H -t snapshot -o name -s creation -r storage1/apps | tail -n 1') - case $source_snapshot in storage1/apps@*) ;; *) echo 'No apps ZFS snapshot is available' >&2; exit 1 ;; esac - source_label=${source_snapshot#*@} - source_root="/mnt/storage1/apps/.zfs/snapshot/$source_label/jellyfin" -fi - -dataset=$(ssh -p "$target_port" "$target_host" "findmnt -n -o SOURCE --mountpoint $mount") -if [ "$production" = true ]; then - case $dataset in */prod/jellyfin) ;; *) echo 'Production dataset is not mounted' >&2; exit 1 ;; esac -else - case $dataset in */staging/"$instance") ;; *) echo 'Preview dataset is not mounted' >&2; exit 1 ;; esac -fi - -if [ "$offline" = false ]; then - scratch=$(mktemp -d) - trap 'rm -rf "$scratch"' EXIT - rsync -a --exclude='/cache/' --exclude='/log/' --exclude='/transcodes/' --exclude='*-shm' \ - "$source_host:$source_root/" "$scratch/" -fi - -if [ "$production" = false ]; then - ssh -p "$target_port" "$target_host" "NOMAD_TOKEN=\$(cat /var/lib/studio/nomad.token) nomad job stop -yes $instance" -fi -snapshot="$dataset@before-jellyfin-import-$(date +%s)-$$" -ssh -p "$target_port" "$target_host" "set -eu; for i in \$(seq 1 30); do running=\$(curl -fsS -H \"X-Nomad-Token: \$(cat /var/lib/studio/nomad.token)\" http://127.0.0.1:4646/v1/job/$instance/allocations | python3 -c 'import json,sys; print(sum(a[\"ClientStatus\"] == \"running\" for a in json.load(sys.stdin)))'); test \"\$running\" = 0 && break; sleep 2; done; test \"\$running\" = 0; zfs snapshot $snapshot" - -if [ "$offline" = true ]; then - ssh -p "$target_port" "$target_host" "rsync -a --delete --delete-excluded --exclude='/cache/' --exclude='/log/' --exclude='/transcodes/' --exclude='*-shm' '$source_root/' '$root/'" -else - rsync -a --delete -e "ssh -p $target_port" "$scratch/" "$target_host:$root/" -fi -owner=$(ssh -p "$target_port" "$target_host" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/identities.json\"))[\"jellyfin\"])'") -ssh -p "$target_port" "$target_host" "chown -R $owner:$owner $root" -if [ "$offline" = true ]; then - drift=$(ssh -p "$target_port" "$target_host" "rsync -rlnc --delete --delete-excluded --exclude='/cache/' --exclude='/log/' --exclude='/transcodes/' --exclude='*-shm' --out-format='%n' '$source_root/' '$root/'") -else - drift=$(rsync -rlnc --delete --out-format='%n' -e "ssh -p $target_port" "$scratch/" "$target_host:$root/") -fi -test -z "$drift" || { echo "Jellyfin copy differs from source; restore $snapshot" >&2; exit 1; } -ssh -p "$target_port" "$target_host" "python3 -c 'import sqlite3; from pathlib import Path; root=Path(\"$root/data\"); assert all((root / name).is_file() and sqlite3.connect(\"file:\" + str(root / name) + \"?mode=ro\", uri=True).execute(\"PRAGMA integrity_check\").fetchone()[0] == \"ok\" for name in (\"jellyfin.db\", \"introskipper/introskipper.db\"))'" - -if [ "$production" = true ]; then - if [ "$offline" = false ]; then - test "$(ssh "$source_host" "sudo -n docker inspect -f '{{.State.Running}}' jellyfin")" = false || { - echo 'Zenith jellyfin restarted during import; leave the home server stopped' >&2; exit 1; - } - fi - echo "Imported production Jellyfin from $source_label. Previous dataset state: $snapshot" -else - python3 "$(dirname "$0")/deploy.py" stage jellyfin - ssh -p "$target_port" "$target_host" "zfs destroy $snapshot" - echo "Imported and tested $instance from $source_snapshot" -fi diff --git a/tools/import-legacy-secrets.sh b/tools/import-legacy-secrets.sh deleted file mode 100644 index 1151baa7ea2ea81daebc0557adb43a64eaae7dc4..0000000000000000000000000000000000000000 --- a/tools/import-legacy-secrets.sh +++ /dev/null @@ -1,54 +0,0 @@ -#!/bin/bash -set -euo pipefail - -service=${1:?usage: tools/import-legacy-secrets.sh SERVICE ENV_NAME...|KEY=ENV_NAME...} -shift -(( $# > 0 )) || { echo 'Expected at least one environment variable name' >&2; exit 1; } -[[ $service =~ ^[a-z][a-z0-9-]*$ ]] || { echo 'Invalid service name' >&2; exit 1; } -sources=() -keys=() -for item in "$@"; do - if [[ $item == *=* ]]; then - key=${item%%=*} - source=${item#*=} - [[ $key =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] || { echo 'Invalid secret key' >&2; exit 1; } - keys+=("$key") - else - source=$item - fi - [[ $source =~ ^[A-Z][A-Z0-9_]*$ ]] || { echo 'Invalid environment variable name' >&2; exit 1; } - sources+=("$source") -done -(( ${#keys[@]} == 0 || ${#keys[@]} == ${#sources[@]} )) || { echo 'Use a target key for every source name' >&2; exit 1; } -options=() -for key in "${keys[@]}"; do - options+=(--key "$key") -done -: "${STUDIO_DEPLOY_HOST:?Set STUDIO_DEPLOY_HOST to the production target}" - -source_host=${STUDIO_MIGRATION_SOURCE:-zenith} -repo=$(cd "$(dirname "$0")/.." && pwd) -names=$(printf ' %s' "${sources[@]}") -if [[ -n ${STUDIO_LEGACY_HANDOFF:-} ]]; then - sh "$repo/tools/check-legacy-handoff.sh" "$STUDIO_LEGACY_HANDOFF" "$STUDIO_DEPLOY_HOST" "${STUDIO_DEPLOY_PORT:-22}" - source_remote=(ssh -p "${STUDIO_DEPLOY_PORT:-22}" "$STUDIO_DEPLOY_HOST") -else - source_remote=(ssh "$source_host") -fi - -"${source_remote[@]}" "python3 -$names <<'PY' -from pathlib import Path -import sys - -lines = Path('/mnt/storage1/apps/home-infra/.env').read_text().splitlines() -if len(set(sys.argv[1:])) != len(sys.argv[1:]): - raise SystemExit('Duplicate source secret name') -secrets = [] -for name in sys.argv[1:]: - values = [line.split('=', 1)[1] for line in lines if line.startswith(name + '=')] - if len(values) != 1 or not values[0]: - raise SystemExit('Expected one nonempty source secret: ' + name) - secrets.append(values[0]) -for value in secrets: - print(value) -PY" | python3 "$repo/tools/deploy.py" secrets "$service" --file - "${options[@]}" diff --git a/tools/import-navidrome.sh b/tools/import-navidrome.sh deleted file mode 100644 index 19980dd6c21918041683cc4eb43ff6ea0100a23d..0000000000000000000000000000000000000000 --- a/tools/import-navidrome.sh +++ /dev/null @@ -1,102 +0,0 @@ -#!/bin/sh -set -eu - -instance=${1:?usage: tools/import-navidrome.sh navidrome|navidrome-preview-XXXXXXXX} -case $instance in - navidrome) - test -n "${STUDIO_DEPLOY_HOST:-}" || { echo 'Set STUDIO_DEPLOY_HOST to the production target' >&2; exit 1; } - production=true - ;; - navidrome-preview-*) - printf '%s\n' "$instance" | grep -Eq '^navidrome-preview-[0-9a-f]{8}$' || { - echo 'Expected a Navidrome preview ID' >&2; exit 1; - } - production=false - ;; - *) echo 'Expected navidrome or its preview ID' >&2; exit 1 ;; -esac - -source_host=${STUDIO_MIGRATION_SOURCE:-zenith} -target_host=${STUDIO_DEPLOY_HOST:-root@127.0.0.1} -target_port=${STUDIO_DEPLOY_PORT:-2222} -offline=false -if [ -n "${STUDIO_LEGACY_HANDOFF:-}" ]; then - test "$production" = true || { echo 'Offline handoff requires production Navidrome' >&2; exit 1; } - sh "$(dirname "$0")/check-legacy-handoff.sh" "$STUDIO_LEGACY_HANDOFF" "$target_host" "$target_port" - offline=true -fi -if [ "$production" = true ]; then - root=/srv/prod/navidrome/data - mount=/srv/prod/navidrome - source_root=/mnt/storage1/apps/navidrone - if [ "$offline" = false ]; then - test "$(ssh "$source_host" "sudo -n docker inspect -f '{{.State.Running}}' navidrome")" = false || { - echo 'Stop Zenith navidrome before importing production data' >&2; exit 1; - } - fi - response=$(ssh -p "$target_port" "$target_host" 'curl -s -w "\n%{http_code}" -H "X-Nomad-Token: $(cat /var/lib/studio/nomad.token)" http://127.0.0.1:4646/v1/job/navidrome') - status=$(printf '%s\n' "$response" | tail -n 1) - test "$status" = 200 || { echo "Could not verify navidrome on the home server: $status" >&2; exit 1; } - stopped=$(printf '%s\n' "$response" | sed '$d' | python3 -c 'import json,sys; print(str(json.load(sys.stdin)["Stop"]).lower())') - test "$stopped" = true || { echo 'Stop navidrome on the home server before importing production data' >&2; exit 1; } - media_fs=$(ssh -p "$target_port" "$target_host" 'findmnt -n -o FSTYPE --mountpoint /srv/clover/Media') - test "$media_fs" = zfs || { echo 'Mount the real Media ZFS dataset before importing production data' >&2; exit 1; } -else - root="/srv/staging/$instance/data" - mount="/srv/staging/$instance" - source_id=$(ssh -p "$target_port" "$target_host" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/stages/$instance.json\"))[\"sourceId\"])'") - test "$source_id" = navidrome || { echo 'Preview belongs to another service' >&2; exit 1; } - source_snapshot=$(ssh "$source_host" '/usr/sbin/zfs list -H -t snapshot -o name -s creation -r storage1/apps | tail -n 1') - case $source_snapshot in storage1/apps@*) ;; *) echo 'No apps ZFS snapshot is available' >&2; exit 1 ;; esac - source_root="/mnt/storage1/apps/.zfs/snapshot/${source_snapshot#*@}/navidrone" -fi - -dataset=$(ssh -p "$target_port" "$target_host" "findmnt -n -o SOURCE --mountpoint $mount") -if [ "$production" = true ]; then - case $dataset in */prod/navidrome) ;; *) echo 'Production dataset is not mounted' >&2; exit 1 ;; esac -else - case $dataset in */staging/"$instance") ;; *) echo 'Preview dataset is not mounted' >&2; exit 1 ;; esac -fi - -if [ "$offline" = true ]; then - ssh -p "$target_port" "$target_host" "test -s '$source_root/navidrome.db'" || { echo 'Navidrome database is missing' >&2; exit 1; } -else - scratch=$(mktemp -d) - trap 'rm -rf "$scratch"' EXIT - rsync -a --exclude='*-shm' "$source_host:$source_root/" "$scratch/" - test -s "$scratch/navidrome.db" || { echo 'Navidrome database is missing' >&2; exit 1; } -fi - -if [ "$production" = false ]; then - ssh -p "$target_port" "$target_host" "NOMAD_TOKEN=\$(cat /var/lib/studio/nomad.token) nomad job stop -yes $instance" -fi -snapshot="$dataset@before-navidrome-import-$(date +%s)-$$" -ssh -p "$target_port" "$target_host" "set -eu; for i in \$(seq 1 30); do running=\$(curl -fsS -H \"X-Nomad-Token: \$(cat /var/lib/studio/nomad.token)\" http://127.0.0.1:4646/v1/job/$instance/allocations | python3 -c 'import json,sys; print(sum(a[\"ClientStatus\"] == \"running\" for a in json.load(sys.stdin)))'); test \"\$running\" = 0 && break; sleep 2; done; test \"\$running\" = 0; zfs snapshot $snapshot" - -if [ "$offline" = true ]; then - ssh -p "$target_port" "$target_host" "rsync -a --delete --delete-excluded --exclude='*-shm' '$source_root/' '$root/'" -else - rsync -a --delete -e "ssh -p $target_port" "$scratch/" "$target_host:$root/" -fi -owner=$(ssh -p "$target_port" "$target_host" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/identities.json\"))[\"navidrome\"])'") -ssh -p "$target_port" "$target_host" "chown -R $owner:$owner $root" -if [ "$offline" = true ]; then - drift=$(ssh -p "$target_port" "$target_host" "rsync -rlnc --delete --delete-excluded --exclude='*-shm' --out-format='%n' '$source_root/' '$root/'") -else - drift=$(rsync -rlnc --delete --out-format='%n' -e "ssh -p $target_port" "$scratch/" "$target_host:$root/") -fi -test -z "$drift" || { echo "Navidrome copy differs from source; restore $snapshot" >&2; exit 1; } -ssh -p "$target_port" "$target_host" "python3 -c 'import sqlite3; db=sqlite3.connect(\"file:$root/navidrome.db?mode=ro\", uri=True); assert db.execute(\"PRAGMA integrity_check\").fetchone()[0] == \"ok\"; assert db.execute(\"select count(*) from user where user_name=\\\"snow\\\" and is_admin=1\").fetchone()[0] == 1'" - -if [ "$production" = true ]; then - if [ "$offline" = false ]; then - test "$(ssh "$source_host" "sudo -n docker inspect -f '{{.State.Running}}' navidrome")" = false || { - echo 'Zenith navidrome restarted during import; leave the home server stopped' >&2; exit 1; - } - fi - echo "Imported production Navidrome. Previous dataset state: $snapshot" -else - python3 "$(dirname "$0")/deploy.py" stage navidrome - ssh -p "$target_port" "$target_host" "zfs destroy $snapshot" - echo "Imported and tested $instance from $source_snapshot" -fi diff --git a/tools/import-pds-secrets.py b/tools/import-pds-secrets.py deleted file mode 100644 index ab0219eafd23978a4635cb244b6644156b72e92c..0000000000000000000000000000000000000000 --- a/tools/import-pds-secrets.py +++ /dev/null @@ -1,44 +0,0 @@ -#!/usr/bin/env python3 -import os -from pathlib import Path -import re -import subprocess - - -if not os.environ.get("STUDIO_DEPLOY_HOST"): - raise SystemExit("Set STUDIO_DEPLOY_HOST to the production target") - -if os.environ.get("STUDIO_LEGACY_HANDOFF"): - ssh = ["ssh", "-o", "BatchMode=yes", "-p", os.environ.get("STUDIO_DEPLOY_PORT", "22"), os.environ["STUDIO_DEPLOY_HOST"]] -else: - ssh = ["ssh", "-o", "BatchMode=yes", os.environ.get("STUDIO_MIGRATION_SOURCE", "zenith")] - -source = subprocess.run( - [*ssh, "cat", "/mnt/storage1/apps/home-infra/.env"], - check=True, capture_output=True, text=True, -).stdout -names = { - "PDS_JWT_SECRET": "jwt_secret", - "PDS_ADMIN_PASSWORD": "admin_password", - "PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX": "plc_rotation_key", - "MAILER_ADDRESS": "mailer_address", - "MAILER_USERNAME": "mailer_username", - "MAILER_PASSWORD": "mailer_password", -} -values = {} -for line in source.splitlines(): - key, separator, value = line.partition("=") - key = key.strip() - if separator and key in names: - if names[key] in values: - raise ValueError(f"duplicate source secret: {key}") - values[names[key]] = value.strip() -if len(values) != len(names) or not all(values.values()): - raise ValueError("PDS source secrets are incomplete") -if not re.fullmatch(r"[0-9a-fA-F]{64}", values["plc_rotation_key"]): - raise ValueError("invalid PDS rotation key") - -subprocess.run( - ["python3", str(Path(__file__).with_name("deploy.py")), "secrets", "pds", "--file", "-"], - input="".join(values[name] + "\n" for name in names.values()), text=True, check=True, -) diff --git a/tools/import-pds.sh b/tools/import-pds.sh deleted file mode 100644 index f425567344e3490d21068dbcd69e83dd11ac9226..0000000000000000000000000000000000000000 --- a/tools/import-pds.sh +++ /dev/null @@ -1,127 +0,0 @@ -#!/bin/sh -set -eu - -instance=${1:?usage: tools/import-pds.sh pds|pds-preview-XXXXXXXX} -case $instance in - pds) - test -n "${STUDIO_DEPLOY_HOST:-}" || { echo 'Set STUDIO_DEPLOY_HOST to the production target' >&2; exit 1; } - production=true - ;; - pds-preview-*) - printf '%s\n' "$instance" | grep -Eq '^pds-preview-[0-9a-f]{8}$' || { - echo 'Expected a PDS preview ID' >&2; exit 1; - } - production=false - ;; - *) echo 'Expected pds or its preview ID' >&2; exit 1 ;; -esac - -source_host=${STUDIO_MIGRATION_SOURCE:-zenith} -target_host=${STUDIO_DEPLOY_HOST:-root@127.0.0.1} -target_port=${STUDIO_DEPLOY_PORT:-2222} -offline=false -copy_host=$source_host -copy_rsh=ssh -if [ -n "${STUDIO_LEGACY_HANDOFF:-}" ]; then - test "$production" = true || { echo 'Offline handoff requires production PDS' >&2; exit 1; } - sh "$(dirname "$0")/check-legacy-handoff.sh" "$STUDIO_LEGACY_HANDOFF" "$target_host" "$target_port" - offline=true - copy_host=$target_host - copy_rsh="ssh -p $target_port" -fi -source_ssh() { - if [ "$offline" = true ]; then - ssh -p "$target_port" "$target_host" "$@" - else - ssh "$source_host" "$@" - fi -} -if [ "$production" = true ]; then - root=/srv/prod/pds/pds - mount=/srv/prod/pds - if [ "$offline" = false ]; then - source_running=$(ssh "$source_host" "sudo -n docker inspect -f '{{.State.Running}}' pds") - test "$source_running" = false || { echo 'Stop Zenith pds before importing production data' >&2; exit 1; } - fi - response=$(ssh -p "$target_port" "$target_host" 'curl -s -w "\n%{http_code}" -H "X-Nomad-Token: $(cat /var/lib/studio/nomad.token)" http://127.0.0.1:4646/v1/job/pds') - status=$(printf '%s\n' "$response" | tail -n 1) - test "$status" = 200 || { echo "Could not verify pds on the home server: $status" >&2; exit 1; } - stopped=$(printf '%s\n' "$response" | sed '$d' | python3 -c 'import json,sys; print(str(json.load(sys.stdin)["Stop"]).lower())') - test "$stopped" = true || { echo 'Stop pds on the home server before importing production data' >&2; exit 1; } -else - root="/srv/staging/$instance/pds" - mount="/srv/staging/$instance" - source_id=$(ssh -p "$target_port" "$target_host" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/stages/$instance.json\"))[\"sourceId\"])'") - test "$source_id" = pds || { echo 'Preview belongs to another service' >&2; exit 1; } -fi -scratch=$(mktemp -d) -source_backup= -cleanup() { - rm -rf "$scratch" - if [ -n "$source_backup" ]; then - source_ssh "rm -rf '$source_backup'" - fi -} -trap cleanup EXIT - -dataset=$(ssh -p "$target_port" "$target_host" "findmnt -n -o SOURCE --mountpoint $mount") -if [ "$production" = true ]; then - case $dataset in */prod/pds) ;; *) echo 'Production dataset is not mounted' >&2; exit 1 ;; esac - python3 "$(dirname "$0")/import-pds-secrets.py" -else - case $dataset in */staging/"$instance") ;; *) echo 'Preview dataset is not mounted' >&2; exit 1 ;; esac - origin=$(ssh -p "$target_port" "$target_host" "zfs get -H -o value origin $dataset") - test "$origin" = - || { echo 'Expected a fresh preview dataset' >&2; exit 1; } -fi -snapshot="$dataset@before-pds-import-$(date +%s)-$$" - -source_backup=$(source_ssh python3 - <<'PY' -from pathlib import Path -import shutil -import sqlite3 -import tempfile - -out = Path(tempfile.mkdtemp(prefix="studio-pds-")) -try: - for name in ("account", "sequencer", "did_cache"): - source = sqlite3.connect(f"file:/mnt/storage1/apps/pds/{name}.sqlite?mode=ro", uri=True) - target = sqlite3.connect(out / f"{name}.sqlite") - source.backup(target) - if target.execute("PRAGMA integrity_check").fetchone()[0] != "ok": - raise ValueError(f"Invalid PDS SQLite copy: {name}") - target.close() - source.close() -except BaseException: - shutil.rmtree(out) - raise -print(out) -PY -) -rsync -a -e "$copy_rsh" "$copy_host:$source_backup/" "$scratch/" -rsync -a --exclude='*.sqlite*' -e "$copy_rsh" "$copy_host:/mnt/storage1/apps/pds/" "$scratch/" -drift=$(rsync -rnc --delete --exclude='*.sqlite*' --out-format='%n' -e "$copy_rsh" "$copy_host:/mnt/storage1/apps/pds/" "$scratch/") -test -z "$drift" || { echo 'PDS source changed during copy; retry the import' >&2; exit 1; } - -if [ "$production" = false ]; then - ssh -p "$target_port" "$target_host" "NOMAD_TOKEN=\$(cat /var/lib/studio/nomad.token) nomad job stop -yes $instance" -fi -ssh -p "$target_port" "$target_host" "set -eu; for i in \$(seq 1 30); do running=\$(curl -fsS -H \"X-Nomad-Token: \$(cat /var/lib/studio/nomad.token)\" http://127.0.0.1:4646/v1/job/$instance/allocations | python3 -c 'import json,sys; print(sum(a[\"ClientStatus\"] == \"running\" for a in json.load(sys.stdin)))'); test \"\$running\" = 0 && break; sleep 2; done; test \"\$running\" = 0; zfs snapshot $snapshot" - -rsync -a --delete -e "ssh -p $target_port" "$scratch/" "$target_host:$root/" -owner=$(ssh -p "$target_port" "$target_host" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/identities.json\"))[\"pds\"])'") -ssh -p "$target_port" "$target_host" "chown -R $owner:$owner $root; python3 -c 'import sqlite3; from pathlib import Path; root=Path(\"$root\"); names=(\"account\",\"sequencer\",\"did_cache\"); assert all((root / (name + \".sqlite\")).is_file() and sqlite3.connect(\"file:\" + str(root / (name + \".sqlite\")) + \"?mode=ro&immutable=1\", uri=True).execute(\"PRAGMA integrity_check\").fetchone()[0] == \"ok\" for name in names)'" -drift=$(rsync -rnc --delete --out-format='%n' -e "ssh -p $target_port" "$scratch/" "$target_host:$root/") -test -z "$drift" || { echo "PDS copy differs from source backup; restore $snapshot" >&2; exit 1; } - -if [ "$production" = true ]; then - if [ "$offline" = false ]; then - test "$(ssh "$source_host" "sudo -n docker inspect -f '{{.State.Running}}' pds")" = false || { - echo 'Zenith pds restarted during import; leave the home server stopped' >&2; exit 1; - } - fi - echo "Imported production PDS from Zenith. Previous dataset state: $snapshot" -else - python3 "$(dirname "$0")/deploy.py" stage pds - ssh -p "$target_port" "$target_host" "zfs destroy $snapshot" - echo "Imported and tested $instance" -fi diff --git a/tools/import-personal-files.py b/tools/import-personal-files.py deleted file mode 100644 index aebe1e825e88b223bfcef734d61b13dc6179714d..0000000000000000000000000000000000000000 --- a/tools/import-personal-files.py +++ /dev/null @@ -1,114 +0,0 @@ -#!/usr/bin/env python3 -import argparse -import json -import os -from pathlib import Path -import sqlite3 -import subprocess -import time -import urllib.error -import urllib.request - - -parser = argparse.ArgumentParser(description="Copy retained app files on the new host before starting the service.") -parser.add_argument("service", choices=["tailscale", "ddns-updater", "copyparty", "ytdl", "pds"]) - - -def main(): - service = parser.parse_args().service - if os.geteuid() != 0: - raise SystemExit("Run this importer as root on the production host.") - token = Path("/var/lib/studio/nomad.token").read_text().strip() - request = urllib.request.Request( - "http://127.0.0.1:4646/v1/job/" + service, - headers={"X-Nomad-Token": token}, - ) - try: - with urllib.request.urlopen(request, timeout=10) as response: - job = json.load(response) - if not job.get("Stop"): - raise SystemExit("Stop " + service + " before copying its files.") - except urllib.error.HTTPError as error: - if error.code != 404: - raise - request.full_url += "/allocations" - try: - with urllib.request.urlopen(request, timeout=10) as response: - allocations = json.load(response) - except urllib.error.HTTPError as error: - if error.code != 404: - raise - allocations = [] - if any(item["ClientStatus"] not in ("complete", "failed", "lost") for item in allocations): - raise SystemExit("Wait for " + service + " allocations to stop before copying its files.") - - root = Path("/srv/prod") / service - dataset = subprocess.check_output(["findmnt", "-n", "-o", "SOURCE", "--mountpoint", str(root)], text=True).strip() - if dataset != "globe/prod/" + service: - raise SystemExit("Mount globe/prod/" + service + " at " + str(root) + " before copying files.") - uid = 0 if service == "tailscale" else json.loads(Path("/var/lib/studio/identities.json").read_text())[service] - directories = { - "tailscale": ("tailscale", "var/lib/tailscale"), - "ddns-updater": ("ddns-updater", "updater/data"), - "copyparty": ("copyparty/copyparty", "cfg/copyparty"), - "ytdl": ("ytdl-sub", "config"), - "pds": ("pds", "pds"), - } - source_name, target_name = directories[service] - source = Path("/mnt/storage1/apps") / source_name - target = root / target_name - if not source.is_dir(): - raise SystemExit("Retained app files are missing at " + str(source)) - if service == "tailscale": - json.loads((source / "tailscaled.state").read_text()) - elif service == "ddns-updater": - json.loads((source / "updates.json").read_text()) - elif service == "copyparty": - for database in source.glob("*.db"): - with sqlite3.connect("file:" + str(database) + "?mode=ro", uri=True) as connection: - if connection.execute("PRAGMA integrity_check").fetchone()[0] != "ok": - raise SystemExit("Check the retained Copyparty database before copying " + database.name) - if not (source / "shares.db").is_file(): - raise SystemExit("Restore the retained Copyparty shares.db before importing.") - elif service == "pds": - for name in ("account", "sequencer", "did_cache"): - if not (source / (name + ".sqlite")).is_file(): - raise SystemExit("Restore the retained PDS " + name + " database before importing.") - if list(source.rglob("*-wal")) or list(source.rglob("*-shm")): - raise SystemExit("Back up all PDS SQLite databases, including actor stores, before importing WAL files.") - for database in source.rglob("*.sqlite"): - with sqlite3.connect("file:" + str(database) + "?mode=ro&immutable=1", uri=True) as connection: - if connection.execute("PRAGMA integrity_check").fetchone()[0] != "ok": - raise SystemExit("Check the retained PDS database before importing " + database.name) - - snapshot = dataset + "@before-files-import-" + str(time.time_ns()) - subprocess.run(["zfs", "snapshot", snapshot], check=True) - target.mkdir(parents=True, exist_ok=True) - exclusions = ["--exclude=*.lock", "--exclude=/.ytdl-sub-lock", "--exclude=/.cache/", "--exclude=/work/"] - subprocess.run(["rsync", "-aH", "--delete", *exclusions, str(source) + "/", str(target) + "/"], check=True) - drift = subprocess.check_output([ - "rsync", "-rlnc", "--delete", "--out-format=%n", *exclusions, str(source) + "/", str(target) + "/", - ], text=True) - if drift: - raise SystemExit("Copied files differ from the retained source. Restore " + snapshot + " before retrying.") - os.chown(target, uid, uid) - for directory, children, files in os.walk(target): - for name in children + files: - os.chown(Path(directory) / name, uid, uid, follow_symlinks=False) - if service == "tailscale": - os.chmod(target / "tailscaled.state", 0o600) - elif service == "copyparty": - for database in target.glob("*.db"): - with sqlite3.connect("file:" + str(database) + "?mode=ro", uri=True) as connection: - if connection.execute("PRAGMA integrity_check").fetchone()[0] != "ok": - raise SystemExit("Copied Copyparty database needs recovery from " + snapshot) - elif service == "pds": - for database in target.rglob("*.sqlite"): - with sqlite3.connect("file:" + str(database) + "?mode=ro&immutable=1", uri=True) as connection: - if connection.execute("PRAGMA integrity_check").fetchone()[0] != "ok": - raise SystemExit("Copied PDS database needs recovery from " + snapshot) - print(json.dumps({"service": service, "snapshot": snapshot, "copied": True})) - - -if __name__ == "__main__": - main() diff --git a/tools/import-personal-postgres.py b/tools/import-personal-postgres.py deleted file mode 100644 index 67ff7e6fdd1ff698791c4b1f133d8e0a0e4e2054..0000000000000000000000000000000000000000 --- a/tools/import-personal-postgres.py +++ /dev/null @@ -1,215 +0,0 @@ -#!/usr/bin/env python3 -import argparse -import hashlib -import json -import os -from pathlib import Path -import re -import subprocess -import tempfile -import urllib.error -import urllib.request - - -def identifier(value): - return '"' + value.replace('"', '""') + '"' - - -def digest(path): - result = hashlib.sha256() - with path.open('rb') as stream: - for block in iter(lambda: stream.read(1024 * 1024), b''): - result.update(block) - return result.hexdigest() - - -def fingerprint(sql, database): - relations = json.loads(sql(database, """ - SELECT coalesce(json_agg(x ORDER BY n, c), '[]') FROM ( - SELECT n.nspname n, c.relname c, c.relkind k - FROM pg_class c JOIN pg_namespace n ON n.oid=c.relnamespace - WHERE n.nspname NOT LIKE 'pg_%' AND n.nspname <> 'information_schema' - AND c.relkind IN ('r','m','S') - ) x - """)) - result = {} - for relation in relations: - table = identifier(relation['n']) + '.' + identifier(relation['c']) - query = (f'SELECT last_value,is_called FROM {table}' if relation['k'] == 'S' else - f'COPY (SELECT row_to_json(t)::text FROM {table} t ' - 'ORDER BY row_to_json(t)::text COLLATE "C") TO STDOUT') - content = sql(database, query) - result[table] = {'rows': len(content.splitlines()), - 'sha256': hashlib.sha256(content).hexdigest()} - content = sql(database, "COPY (SELECT loid,pageno,encode(data,'hex') " - 'FROM pg_largeobject ORDER BY loid,pageno) TO STDOUT') - result['large_objects'] = {'rows': len(content.splitlines()), - 'sha256': hashlib.sha256(content).hexdigest()} - return result - - -def main(): - parser = argparse.ArgumentParser(description='Restore a personal database before starting its service.') - parser.add_argument('service', choices=('keycloak', 'dawarich')) - parser.add_argument('handoff', type=Path) - parser.add_argument('--container', help='Restore inside the isolated initializer before deploying Postgres') - args = parser.parse_args() - if os.geteuid() != 0: - parser.error('Run as root on Zenith') - os.umask(0o077) - handoff = args.handoff.resolve() - cluster = json.loads((handoff / 'manifest.json').read_text())['clusters']['18'] - if not cluster.get('restoreVerified'): - raise ValueError('PostgreSQL 18 backup is unverified. Use the verified handoff') - entry = cluster['databases'][args.service] - dump = (handoff / entry['file']).resolve() - if not dump.is_relative_to(handoff / 'pg18') or dump.suffix != '.dump': - raise ValueError('Dump is outside the PostgreSQL 18 handoff') - if dump.stat().st_size != entry['bytes'] or digest(dump) != entry['sha256']: - raise ValueError('Backup checksum differs. Recover the verified handoff before importing') - - token = os.environ.get('NOMAD_TOKEN') or Path('/var/lib/studio/nomad.token').read_text().strip() - - def nomad(path): - request = urllib.request.Request('http://127.0.0.1:4646/v1/' + path, - headers={'X-Nomad-Token': token}) - with urllib.request.urlopen(request, timeout=10) as response: - return json.load(response) - - def stopped(service): - try: - job = nomad('job/' + service) - except urllib.error.HTTPError as error: - if error.code == 404: - return - raise - if not job.get('Stop') or any( - item['ClientStatus'] in ('pending', 'running') - for item in nomad('job/' + service + '/allocations') - ): - raise ValueError(f'Stop {service} and wait for its allocations before importing') - - stopped(args.service) - connection = nomad('var/nomad/jobs/' + args.service + '/inputs/database')['Items'] - database, owner = connection['name'], connection['username'] - expected_database = 'keycloak_next' if args.service == 'keycloak' else 'dawarich' - if database != expected_database or owner != 'svc_' + database: - raise ValueError('Database variable differs from the production service. Allocate its database first') - if args.container: - stopped('postgres') - else: - allocations = [item['ID'] for item in nomad('job/postgres/allocations') - if item['ClientStatus'] == 'running' and item['DesiredStatus'] == 'run'] - if len(allocations) != 1: - raise ValueError('Postgres needs exactly one running allocation') - folder = Path(tempfile.mkdtemp(prefix=args.service + '-import-', dir='/var/lib/studio')) - diagnostics = folder / 'diagnostics.log' - podman = ['podman'] - - def run(command, *, source=None, output=None): - with diagnostics.open('ab') as errors: - result = subprocess.run(command, input=source if isinstance(source, bytes) else None, - stdin=source if source is not None and not isinstance(source, bytes) else None, - stdout=output or subprocess.PIPE, stderr=errors) - if result.returncode: - raise RuntimeError(f'{Path(command[0]).name} failed. See {diagnostics}') - return result.stdout - - if args.container: - container = json.loads(run(podman + ['inspect', args.container]))[0] - if not container['State']['Running'] or container['HostConfig']['NetworkMode'] != 'none': - raise ValueError('Initializer must be running with --network none') - container_id = container['Id'] - else: - containers = [parts[0] for line in run(podman + ['ps', '--format', '{{.ID}} {{.Names}}']).decode().splitlines() - if len(parts := line.split()) == 2 and parts[1].endswith(allocations[0])] - if len(containers) != 1: - raise ValueError('Postgres allocation container is unavailable') - container_id = containers[0] - execute = podman + ['exec', '-i', container_id] - - def sql(db, query): - return run(execute + ['psql', '-X', '-U', 'postgres', '-d', db, - '-At', '-v', 'ON_ERROR_STOP=1'], source=(query + '\n').encode()) - - current_owner = sql('postgres', f"SELECT pg_get_userbyid(datdba) FROM pg_database WHERE datname='{database}'").decode().strip() - if current_owner != owner: - raise ValueError('Production database is missing or has another owner. Allocate it with the Postgres provider') - with dump.open('rb') as source: - listing = run(execute + ['pg_restore', '--list'], source=source).decode() - extensions = re.findall(r'^\d+; \d+ \d+ EXTENSION - (\S+) ', listing, re.MULTILINE) - if set(extensions) - {'plpgsql', 'postgis'} or ('postgis' in extensions) != (args.service == 'dawarich'): - raise ValueError('Backup extensions differ from the service. Review the personal database before importing') - extension_data, application = [], [] - for line in listing.splitlines(): - if re.match(r'^\d+; \d+ \d+ (EXTENSION |COMMENT - EXTENSION |SCHEMA - public )', line): - continue - if re.match(r'^\d+; \d+ \d+ TABLE DATA public spatial_ref_sys ', line): - extension_data.append(line) - else: - application.append(line) - contents = {name: value for name, value in entry['contents'].items() if name != 'large_object_owners'} - restore_list = '/tmp/' + folder.name + '.list' - run(execute + ['tee', restore_list], source=('\n'.join(application) + '\n').encode()) - try: - stopped(args.service) - if args.container: - stopped('postgres') - if sql('postgres', f"SELECT count(*) FROM pg_stat_activity WHERE datname='{database}'").strip() != b'0': - raise ValueError('Database has connected clients. Stop them before importing') - backup = folder / 'before.dump' - with backup.open('wb') as output: - run(execute + ['pg_dump', '-U', 'postgres', '-Fc', database], output=output) - if not backup.stat().st_size: - raise RuntimeError('Target backup is empty. Import stopped') - sql('postgres', f'DROP DATABASE {database};') - sql('postgres', f'CREATE DATABASE {database} OWNER {owner};') - sql(database, f'ALTER SCHEMA public OWNER TO {owner};') - if args.service == 'dawarich': - sql(database, 'CREATE EXTENSION postgis;') - with dump.open('rb') as source: - run(execute + ['pg_restore', '-U', 'postgres', '-d', database, '--no-owner', '--no-acl', - '--role=' + owner, '--single-transaction', '--exit-on-error', - '--use-list=' + restore_list], source=source) - if extension_data: - run(execute + ['tee', restore_list], source=('\n'.join(extension_data) + '\n').encode()) - with dump.open('rb') as source: - extension_sql = run(execute + ['pg_restore', '-U', 'postgres', '--data-only', '--no-owner', - '--no-acl', '--use-list=' + restore_list, '--file=-'], source=source) - srids = [] - copying = False - for line in extension_sql.splitlines(): - if line.startswith(b'COPY public.spatial_ref_sys ('): - copying = True - elif copying and line == b'\\.': - copying = False - elif copying: - srids.append(int(line.split(b'\t', 1)[0])) - # PostGIS dumps include custom SRIDs; the extension supplies built-in definitions. - if srids: - sql(database, 'DELETE FROM public.spatial_ref_sys WHERE srid IN (' + ','.join(map(str, srids)) + ');') - with dump.open('rb') as source: - run(execute + ['pg_restore', '-U', 'postgres', '-d', database, '--no-owner', '--no-acl', - '--data-only', '--single-transaction', '--exit-on-error', - '--use-list=' + restore_list], source=source) - actual = fingerprint(sql, database) - if actual != contents: - changed = sorted(name for name in actual.keys() | contents.keys() if actual.get(name) != contents.get(name)) - (folder / 'differences.json').write_text(json.dumps(changed, indent=2) + '\n') - raise RuntimeError(f'Restored data differs. Keep {args.service} stopped and review {folder}') - if dump.stat().st_size != entry['bytes'] or digest(dump) != entry['sha256']: - raise RuntimeError('Backup changed during import. Keep the service stopped') - (folder / 'verified.json').write_text(json.dumps({ - 'service': args.service, 'database': database, 'source': str(dump), - 'sha256': entry['sha256'], 'contents': actual, - }, indent=2) + '\n') - print(f'Imported and verified {args.service}. Previous database: {backup}') - finally: - run(execute + ['rm', '-f', restore_list]) - - -if __name__ == '__main__': - try: - main() - except (ValueError, RuntimeError, OSError, KeyError, urllib.error.HTTPError) as error: - raise SystemExit(str(error)) from None diff --git a/tools/import-qbittorrent.sh b/tools/import-qbittorrent.sh deleted file mode 100755 index 538f6467dfa03ed95a71980e682756a7e4d62049..0000000000000000000000000000000000000000 --- a/tools/import-qbittorrent.sh +++ /dev/null @@ -1,66 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -: "${STUDIO_DEPLOY_HOST:?Set STUDIO_DEPLOY_HOST to the production target}" -source_host=${STUDIO_MIGRATION_SOURCE:-zenith} -target_host=$STUDIO_DEPLOY_HOST -target_port=${STUDIO_DEPLOY_PORT:-2222} -remote=(ssh -p "$target_port" "$target_host") -root=/srv/prod/qbittorrent -source_copy_host=$source_host -source_rsh=ssh -offline=false -if [[ -n ${STUDIO_LEGACY_HANDOFF:-} ]]; then - sh "$(dirname "$0")/check-legacy-handoff.sh" "$STUDIO_LEGACY_HANDOFF" "$target_host" "$target_port" - source_copy_host=$target_host - source_rsh="ssh -p $target_port" - offline=true -fi - -if [[ $offline == false ]]; then - source_running=$(ssh "$source_host" 'sudo -n docker inspect -f "{{.State.Running}}" qbittorrent') - [[ $source_running == false ]] || { echo 'Stop Zenith qBittorrent before importing its profile' >&2; exit 1; } -fi -"${remote[@]}" 'test "$(findmnt -n -o FSTYPE --mountpoint /srv/clover/Media)" = zfs; test -d /srv/clover/Media/Seedbox; test -L /srv/clover/Media/torrent; test /srv/clover/Media/Seedbox -ef /srv/clover/Media/torrent' || { - echo 'Mount Media and point torrent at Seedbox before importing qBittorrent' >&2; exit 1; -} - -response=$("${remote[@]}" 'curl -s -w "\n%{http_code}" -H "X-Nomad-Token: $(cat /var/lib/studio/nomad.token)" http://127.0.0.1:4646/v1/job/qbittorrent') -status=${response##*$'\n'} -if [[ $status == 200 ]]; then - stopped=$(python3 -c 'import json,sys; print(str(json.load(sys.stdin)["Stop"]).lower())' <<<"${response%$'\n'*}") - [[ $stopped == true ]] || { echo 'Stop qBittorrent on the home server before importing its profile' >&2; exit 1; } -elif [[ $status != 404 ]]; then - echo "Could not verify qBittorrent job state on the home server: $status" >&2 - exit 1 -fi -dataset=$("${remote[@]}" "findmnt -n -o SOURCE --mountpoint $root") -[[ $dataset == */prod/qbittorrent ]] || { echo "Expected a mounted service dataset at $root" >&2; exit 1; } - -scratch=$(mktemp -d) -trap 'rm -rf "$scratch"' EXIT -mkdir "$scratch/qBittorrent" -rsync -a --exclude=/config/ipc-socket --exclude=/config/lockfile --exclude=/data/logs/ --exclude=/cache/ \ - -e "$source_rsh" "$source_copy_host:/mnt/storage1/apps/qbittorrent/qBittorrent/" "$scratch/qBittorrent/" -resume_count=$(find "$scratch/qBittorrent/data/BT_backup" -maxdepth 1 -type f -name '*.fastresume' | wc -l | tr -d ' ') -[[ $resume_count -gt 0 ]] || { echo 'No saved torrents copied from Zenith' >&2; exit 1; } - -snapshot="$dataset@before-qbittorrent-import-$(date +%s)-$$" -"${remote[@]}" "zfs snapshot $snapshot" -rsync -a --delete -e "ssh -p $target_port" "$scratch/qBittorrent/" "$target_host:$root/config/qBittorrent/" -changes=$(rsync -a --checksum --dry-run --itemize-changes -e "ssh -p $target_port" \ - "$scratch/qBittorrent/" "$target_host:$root/config/qBittorrent/") -[[ -z $changes ]] || { echo 'Copied qBittorrent profile differs from source' >&2; exit 1; } - -uid=$("${remote[@]}" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/identities.json\"))[\"qbittorrent\"])'") -"${remote[@]}" "chown -R $uid:$uid $root/config/qBittorrent && chmod -R u+rwX,go-rwx $root/config/qBittorrent && printf '{\"hostRoot\":\"$root\",\"uid\":$uid}' | python3 /opt/studio/current/service/qbittorrent/prepare.py" -copied=$("${remote[@]}" "find $root/config/qBittorrent/data/BT_backup -maxdepth 1 -type f -name '*.fastresume' | wc -l") -[[ $copied -eq $resume_count ]] || { echo 'Saved torrent count changed during import' >&2; exit 1; } -if [[ $offline == false ]]; then - [[ $(ssh "$source_host" 'sudo -n docker inspect -f "{{.State.Running}}" qbittorrent') == false ]] || { - echo 'Zenith qBittorrent restarted during import; leave the home server stopped' >&2 - exit 1 - } -fi -"${remote[@]}" 'python3 /opt/studio/current/tools/studio.py deploy qbittorrent' -echo "Imported $resume_count saved torrents into $dataset; pre-import snapshot: $snapshot" diff --git a/tools/import-shale.sh b/tools/import-shale.sh deleted file mode 100644 index e0721c986f520b48f2b2e388580f12ba229d45b3..0000000000000000000000000000000000000000 --- a/tools/import-shale.sh +++ /dev/null @@ -1,106 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -instance=${1:?usage: tools/import-shale.sh shale|shale-preview-XXXXXXXX} -[[ $instance == shale || $instance =~ ^shale-preview-[0-9a-f]{8}$ ]] || { echo 'Expected Shale or its preview ID' >&2; exit 1; } -source_host=${STUDIO_MIGRATION_SOURCE:-zenith} -target_host=${STUDIO_DEPLOY_HOST:-root@127.0.0.1} -target_port=${STUDIO_DEPLOY_PORT:-2222} -remote=(ssh -p "$target_port" "$target_host") -job_status=200 -handoff=${STUDIO_LEGACY_HANDOFF:-} -if [[ -n $handoff ]]; then - [[ $instance == shale && -n ${STUDIO_DEPLOY_HOST:-} ]] || { - echo 'Offline handoff requires production Shale and a target' >&2 - exit 1 - } - sh "$(dirname "$0")/check-legacy-handoff.sh" "$handoff" "$target_host" "$target_port" - source_copy_host=$target_host - source_rsh="ssh -p $target_port" -else - source_copy_host=$source_host - source_rsh=ssh -fi - -if [[ $instance == shale ]]; then - [[ -n ${STUDIO_DEPLOY_HOST:-} ]] || { echo 'Set STUDIO_DEPLOY_HOST to the production target' >&2; exit 1; } - if [[ -z $handoff ]]; then - [[ $(ssh "$source_host" "sudo -n docker inspect -f '{{.State.Running}}' shale") == false ]] || { echo 'Stop Zenith Shale before importing production data' >&2; exit 1; } - fi - root=/srv/prod/shale - source_root=/mnt/storage1/apps/shale - job_response=$("${remote[@]}" 'curl -s -w "\n%{http_code}" -H "X-Nomad-Token: $(cat /var/lib/studio/nomad.token)" http://127.0.0.1:4646/v1/job/shale') - job_status=${job_response##*$'\n'} - if [[ $job_status == 200 ]]; then - stopped=$(python3 -c 'import json,sys; print(str(json.load(sys.stdin)["Stop"]).lower())' <<<"${job_response%$'\n'*}") - [[ $stopped == true ]] || { echo 'Stop Shale on the home server before importing production data' >&2; exit 1; } - elif [[ $job_status != 404 ]]; then - echo "Could not verify production Shale job: $job_status" >&2 - exit 1 - fi -else - root=/srv/staging/$instance - source_id=$("${remote[@]}" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/stages/$instance.json\"))[\"sourceId\"])'") - [[ $source_id == shale ]] || { echo 'Preview belongs to another service' >&2; exit 1; } - source_snapshot=$(ssh "$source_host" '/usr/sbin/zfs list -H -t snapshot -o name -s creation -r storage1/apps | tail -n 1') - [[ $source_snapshot == storage1/apps@* ]] || { echo 'No Zenith app snapshot is available' >&2; exit 1; } - source_root=/mnt/storage1/apps/.zfs/snapshot/${source_snapshot#*@}/shale -fi - -dataset=$("${remote[@]}" "findmnt -n -o SOURCE --mountpoint $root") -if [[ $instance == shale ]]; then - [[ $dataset == */prod/shale ]] || { echo 'Production Shale dataset is not mounted' >&2; exit 1; } -else - [[ $dataset == */staging/$instance ]] || { echo 'Preview Shale dataset is not mounted' >&2; exit 1; } -fi - -scratch=$(mktemp -d) -trap 'rm -rf "$scratch"' EXIT -for name in data repositories_owned repositories_mirrors; do - rsync -aH -e "$source_rsh" "$source_copy_host:$source_root/$name/" "$scratch/$name/" - drift=$(rsync -rlnc --delete --out-format='%n' -e "$source_rsh" "$source_copy_host:$source_root/$name/" "$scratch/$name/") - [[ -z $drift ]] || { echo "Zenith Shale $name changed during copy" >&2; exit 1; } -done -python3 - "$scratch/data/astheno.shale.db" <<'PY' -import sqlite3 -import sys - -db = sqlite3.connect(f"file:{sys.argv[1]}?mode=ro", uri=True) -assert db.execute("PRAGMA integrity_check").fetchone()[0] == "ok" -PY - -if [[ $instance != shale ]]; then - "${remote[@]}" "NOMAD_TOKEN=\$(cat /var/lib/studio/nomad.token) nomad job stop -yes $instance" -fi -if [[ $job_status == 200 || $instance != shale ]]; then - for _ in {1..30}; do - running=$("${remote[@]}" "NOMAD_TOKEN=\$(cat /var/lib/studio/nomad.token) nomad job allocs -json $instance" | - python3 -c 'import json,sys; print(sum(a["ClientStatus"] == "running" for a in json.load(sys.stdin)))') - [[ $running == 0 ]] && break - sleep 2 - done - [[ $running == 0 ]] || { echo 'Shale allocation did not stop' >&2; exit 1; } -fi - -snapshot=$dataset@before-shale-import-$(date +%s)-$$ -"${remote[@]}" "zfs snapshot $snapshot" -for name in data repositories_owned repositories_mirrors; do - "${remote[@]}" "mkdir -p $root/$name" - rsync -aH --delete -e "ssh -p $target_port" "$scratch/$name/" "$target_host:$root/$name/" - drift=$(rsync -rlnc --delete --out-format='%n' -e "ssh -p $target_port" "$scratch/$name/" "$target_host:$root/$name/") - [[ -z $drift ]] || { echo "Shale $name on the home server differs from the source copy; restore $snapshot" >&2; exit 1; } -done -uid=$("${remote[@]}" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/identities.json\"))[\"shale\"])'") -"${remote[@]}" "chown -R $uid:$uid $root/data $root/repositories_owned $root/repositories_mirrors" -"${remote[@]}" "python3 -c 'import sqlite3; db=sqlite3.connect(\"file:$root/data/astheno.shale.db?mode=ro\", uri=True); assert db.execute(\"PRAGMA integrity_check\").fetchone()[0] == \"ok\"'" - -if [[ $instance == shale ]]; then - if [[ -z $handoff ]]; then - [[ $(ssh "$source_host" "sudo -n docker inspect -f '{{.State.Running}}' shale") == false ]] || { echo 'Zenith Shale restarted during import; leave the home server stopped' >&2; exit 1; } - fi - echo "Imported production Shale. Previous dataset state: $snapshot" -else - python3 "$(dirname "$0")/deploy.py" stage shale - "${remote[@]}" "zfs destroy $snapshot" - echo "Imported and tested $instance from $source_snapshot" -fi diff --git a/tools/import-source-data.sh b/tools/import-source-data.sh deleted file mode 100644 index b766dcc4c8185d5393e8e9f075c294ab402ba3d3..0000000000000000000000000000000000000000 --- a/tools/import-source-data.sh +++ /dev/null @@ -1,68 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -: "${STUDIO_DEPLOY_HOST:?Set STUDIO_DEPLOY_HOST to the production target}" -source_host=${STUDIO_MIGRATION_SOURCE:-zenith} -target_host=$STUDIO_DEPLOY_HOST -target_port=${STUDIO_DEPLOY_PORT:-22} -remote=(ssh -p "$target_port" "$target_host") -target_dir=/srv/prod/clover-source-of-truth/data -tree_hash="$(dirname "$0")/tree-hash.py" - -if [[ -n ${STUDIO_MIGRATION_SNAPSHOT:-} ]]; then - [[ $STUDIO_MIGRATION_SNAPSHOT =~ ^[A-Za-z0-9_-]+$ ]] || { echo 'Invalid source snapshot name' >&2; exit 1; } - source=("${remote[@]}") - source_dir="/srv/clover/.zfs/snapshot/$STUDIO_MIGRATION_SNAPSHOT/Documents/Config/paper clover" - "${source[@]}" "test -d '$source_dir'" -else - source=(ssh "$source_host") - source_dir='/mnt/storage1/clover/Documents/Config/paper clover' - [[ $("${source[@]}" "sudo -n docker inspect -f '{{.State.Running}}' clover-source-of-truth") == false ]] || { - echo 'Stop Zenith Source of Truth before importing production data' >&2 - exit 1 - } -fi -"${remote[@]}" "set -e; test \$(findmnt -n -o FSTYPE --mountpoint /srv/clover) = zfs; test -d /srv/clover/Published; ! findmnt -n -R '$target_dir' >/dev/null" -dataset=$("${remote[@]}" 'findmnt -n -o SOURCE --mountpoint /srv/prod/clover-source-of-truth') -[[ $dataset == */prod/clover-source-of-truth ]] || { echo 'Production service dataset is not mounted' >&2; exit 1; } -job_response=$("${remote[@]}" 'curl -s -w "\n%{http_code}" -H "X-Nomad-Token: $(cat /var/lib/studio/nomad.token)" http://127.0.0.1:4646/v1/job/clover-source-of-truth') -job_status=${job_response##*$'\n'} -if [[ $job_status == 200 ]]; then - stopped=$(python3 -c 'import json,sys; print(str(json.load(sys.stdin)["Stop"]).lower())' <<<"${job_response%$'\n'*}") - [[ $stopped == true ]] || { echo 'Stop Source of Truth on the home server before importing production data' >&2; exit 1; } - for _ in {1..30}; do - running=$("${remote[@]}" 'NOMAD_TOKEN=$(cat /var/lib/studio/nomad.token) nomad job allocs -json clover-source-of-truth' | - python3 -c 'import json,sys; print(sum(a["ClientStatus"] == "running" for a in json.load(sys.stdin)))') - [[ $running == 0 ]] && break - sleep 2 - done - [[ $running == 0 ]] || { echo 'The home server allocation did not stop' >&2; exit 1; } -elif [[ $job_status != 404 ]]; then - echo "Could not verify the home server job: $job_status" >&2 - exit 1 -fi - -source_manifest=$("${source[@]}" "python3 - '$source_dir'" < "$tree_hash") -source_bytes=$(python3 -c 'import json,sys; print(json.load(sys.stdin)["bytes"])' <<<"$source_manifest") -available=$("${remote[@]}" "df -B1 --output=avail /srv/prod/clover-source-of-truth | tail -n 1") -(( available > source_bytes )) || { echo 'Production dataset has too little free space for the source tree' >&2; exit 1; } - -snapshot=$dataset@before-source-data-import-$(date +%s)-$$ -"${remote[@]}" "set -e; zfs snapshot $snapshot; rm -rf -- '$target_dir'; mkdir -p -- '$target_dir'" -if [[ -n ${STUDIO_MIGRATION_SNAPSHOT:-} ]]; then - "${remote[@]}" "bash -c \"set -euo pipefail; tar -C '$source_dir' -cf - . | tar -C '$target_dir' -xf -\"" -else - "${source[@]}" "tar -C '$source_dir' -cf - ." | - "${remote[@]}" "tar -C '$target_dir' -xf -" -fi -target_manifest=$("${remote[@]}" "python3 - '$target_dir'" < "$tree_hash") -[[ $source_manifest == "$target_manifest" ]] || { echo "Source and target trees differ; restore $snapshot" >&2; exit 1; } -uid=$("${remote[@]}" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/identities.json\"))[\"clover-source-of-truth\"])'") -"${remote[@]}" "set -e; chown -R $uid:$uid '$target_dir'; python3 -c 'import sqlite3; db=sqlite3.connect(\"file:$target_dir/cache.sqlite?mode=ro\", uri=True); assert db.execute(\"PRAGMA integrity_check\").fetchone()[0] == \"ok\"'" -if [[ -z ${STUDIO_MIGRATION_SNAPSHOT:-} ]]; then - [[ $("${source[@]}" "sudo -n docker inspect -f '{{.State.Running}}' clover-source-of-truth") == false ]] || { - echo 'Zenith Source of Truth restarted during import; leave the home server stopped' >&2 - exit 1 - } -fi -echo "Imported ${source_bytes} bytes without local staging. Previous dataset state: $snapshot" diff --git a/tools/import-source-index.sh b/tools/import-source-index.sh deleted file mode 100644 index 51b0594138362ee38984e130546d0a5dbc39cc29..0000000000000000000000000000000000000000 --- a/tools/import-source-index.sh +++ /dev/null @@ -1,68 +0,0 @@ -#!/bin/sh -set -eu - -instance=${1:?usage: tools/import-source-index.sh clover-source-of-truth-preview-XXXXXXXX} -printf '%s\n' "$instance" | grep -Eq '^clover-source-of-truth-preview-[0-9a-f]{8}$' || { - echo 'Expected a Clover Source of Truth preview ID' >&2 - exit 1 -} - -source_host=${STUDIO_MIGRATION_SOURCE:-zenith} -target_host=${STUDIO_DEPLOY_HOST:-root@127.0.0.1} -target_port=${STUDIO_DEPLOY_PORT:-2222} -root="/srv/staging/$instance/data" -mount="/srv/staging/$instance" -source_id=$(ssh -p "$target_port" "$target_host" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/stages/$instance.json\"))[\"sourceId\"])'") -test "$source_id" = clover-source-of-truth || { echo 'Preview belongs to another service' >&2; exit 1; } -dataset=$(ssh -p "$target_port" "$target_host" "findmnt -n -o SOURCE --mountpoint $mount") -case $dataset in */staging/"$instance") ;; *) echo 'Preview dataset is not mounted' >&2; exit 1 ;; esac - -scratch=$(mktemp -d) -source_backup= -cleanup() { - rm -rf "$scratch" - if [ -n "$source_backup" ]; then - ssh "$source_host" "rm -rf '$source_backup'" - fi -} -trap cleanup EXIT - -source_backup=$(ssh "$source_host" sh -s <<'REMOTE' -set -eu -out=$(mktemp -d /tmp/studio-source-index-XXXXXX) -trap 'rm -rf "$out"' EXIT -source='/mnt/storage1/clover/Documents/Config/paper clover/cache.sqlite' -sqlite3 -readonly "$source" ".backup '$out/cache.sqlite'" -test "$(sqlite3 -readonly "$out/cache.sqlite" 'PRAGMA integrity_check;')" = ok -trap - EXIT -printf '%s\n' "$out" -REMOTE -) -rsync -a "$source_host:$source_backup/cache.sqlite" "$scratch/cache.sqlite" -python3 - "$scratch/cache.sqlite" <<'PY' -import sqlite3 -import sys - -db = sqlite3.connect('file:' + sys.argv[1] + '?mode=ro&immutable=1', uri=True) -assert db.execute('PRAGMA integrity_check').fetchone()[0] == 'ok' -PY - -ssh -p "$target_port" "$target_host" "NOMAD_TOKEN=\$(cat /var/lib/studio/nomad.token) nomad job stop -yes $instance" -snapshot="$dataset@before-source-index-import-$(date +%s)-$$" -ssh -p "$target_port" "$target_host" "set -eu; for i in \$(seq 1 30); do running=\$(curl -fsS -H \"X-Nomad-Token: \$(cat /var/lib/studio/nomad.token)\" http://127.0.0.1:4646/v1/job/$instance/allocations | python3 -c 'import json,sys; print(sum(a[\"ClientStatus\"] == \"running\" for a in json.load(sys.stdin)))'); test \"\$running\" = 0 && break; sleep 2; done; test \"\$running\" = 0; zfs snapshot $snapshot" - -rsync -a -e "ssh -p $target_port" "$scratch/cache.sqlite" "$target_host:$root/cache.sqlite" -owner=$(ssh -p "$target_port" "$target_host" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/identities.json\"))[\"clover-source-of-truth\"])'") -ssh -p "$target_port" "$target_host" "chown $owner:$owner $root/cache.sqlite; rm -f $root/cache.sqlite-wal $root/cache.sqlite-shm $root/db-snapshot.sqlite" -drift=$(rsync -rnc --out-format='%n' -e "ssh -p $target_port" "$scratch/cache.sqlite" "$target_host:$root/cache.sqlite") -test -z "$drift" || { echo "Index copy differs from source backup; restore $snapshot" >&2; exit 1; } - -python3 "$(dirname "$0")/deploy.py" stage clover-source-of-truth -ssh -p "$target_port" "$target_host" "python3 - <<'PY' -import sqlite3 -source = sqlite3.connect('file:$root/cache.sqlite?mode=ro', uri=True) -assert source.execute('PRAGMA integrity_check').fetchone()[0] == 'ok' -for table in ('media_files', 'derived_files', 'derived_refs'): - print(table, source.execute('SELECT count(*) FROM ' + table).fetchone()[0]) -PY" -echo "Imported the live SQLite index into $instance; previous dataset: $snapshot" diff --git a/tools/import-yt-feed.sh b/tools/import-yt-feed.sh deleted file mode 100644 index 938713bb79bcad6e7c1d218b2e719f5ebc3bf9ae..0000000000000000000000000000000000000000 --- a/tools/import-yt-feed.sh +++ /dev/null @@ -1,109 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -instance=${1:-ytdl} -[[ $instance == ytdl || $instance =~ ^ytdl-preview-[0-9a-f]{8}$ ]] || { - echo 'Expected ytdl or its preview ID' >&2 - exit 1 -} - -source_host=${STUDIO_MIGRATION_SOURCE:-zenith} -target_host=${STUDIO_DEPLOY_HOST:-root@127.0.0.1} -target_port=${STUDIO_DEPLOY_PORT:-2222} -remote=(ssh -p "$target_port" "$target_host") -source_copy_host=$source_host -source_rsh=ssh -offline=false -if [[ -n ${STUDIO_LEGACY_HANDOFF:-} ]]; then - [[ $instance == ytdl && -n ${STUDIO_DEPLOY_HOST:-} ]] || { - echo 'Offline handoff requires production YouTube Triage and a target' >&2; exit 1; - } - sh "$(dirname "$0")/check-legacy-handoff.sh" "$STUDIO_LEGACY_HANDOFF" "$target_host" "$target_port" - source_copy_host=$target_host - source_rsh="ssh -p $target_port" - offline=true -fi -if [[ $instance == ytdl ]]; then - root=/srv/prod/ytdl - "${remote[@]}" 'test "$(findmnt -n -o FSTYPE --mountpoint /srv/clover/Media)" = zfs; test -d "/srv/clover/Media/Intake - Music"; test -d "/srv/clover/Media/Indie Shows"; test -d /srv/clover/Media/Videos/Independent' || { - echo 'Mount Media with Intake - Music, Indie Shows, and Videos/Independent before importing YouTube Triage' >&2 - exit 1 - } - if [[ $offline == false ]]; then - source_running=$(ssh "$source_host" "sudo -n docker inspect -f '{{.State.Running}}' yt-feed") - [[ $source_running == false ]] || { echo 'Stop Zenith yt-feed before importing production data' >&2; exit 1; } - fi - response=$("${remote[@]}" "curl -s -w '\n%{http_code}' -H \"X-Nomad-Token: \$(cat /var/lib/studio/nomad.token)\" http://127.0.0.1:4646/v1/job/ytdl") - status=${response##*$'\n'} - if [[ $status == 200 ]]; then - stopped=$(python3 -c 'import json,sys; print(str(json.load(sys.stdin)["Stop"]).lower())' <<<"${response%$'\n'*}") - [[ $stopped == true ]] || { echo 'Stop production YTDL before importing' >&2; exit 1; } - elif [[ $status != 404 ]]; then - echo "Could not verify production YTDL job state: $status" >&2 - exit 1 - fi -else - root=/srv/staging/$instance - source_id=$("${remote[@]}" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/stages/$instance.json\"))[\"sourceId\"])'") - [[ $source_id == ytdl ]] || { echo 'Preview belongs to another service' >&2; exit 1; } -fi -dataset=$("${remote[@]}" "findmnt -n -o SOURCE --mountpoint $root") -[[ $dataset == */prod/ytdl || $dataset == */staging/"$instance" ]] || { - echo "Expected a mounted YTDL dataset at $root" >&2 - exit 1 -} - -scratch=$(mktemp -d) -trap 'rm -rf "$scratch"' EXIT -rsync -a -e "$source_rsh" "$source_copy_host:/mnt/storage1/apps/yt-feed/" "$scratch/" -python3 - "$scratch" <<'PY' -import json -from pathlib import Path -import sys - -root = Path(sys.argv[1]) -files = list(root.glob("*.json")) -if not files: - raise ValueError("no YouTube Triage JSON state was copied") -for file in files: - json.loads(file.read_text()) -print(f"Copied {len(files)} valid JSON state files") -PY -drift=$(rsync -rnc --delete --out-format='%n' -e "$source_rsh" "$source_copy_host:/mnt/storage1/apps/yt-feed/" "$scratch/") -[[ -z $drift ]] || { echo 'Source changed during copy; retry the import' >&2; exit 1; } - -if [[ $instance != ytdl ]]; then - "${remote[@]}" "NOMAD_TOKEN=\$(cat /var/lib/studio/nomad.token) nomad job stop -yes $instance" -else - "${remote[@]}" 'systemctl stop studio-dashboard' -fi -for _ in {1..30}; do - running=$("${remote[@]}" "curl -s -H \"X-Nomad-Token: \$(cat /var/lib/studio/nomad.token)\" http://127.0.0.1:4646/v1/job/$instance/allocations" | - python3 -c 'import json,sys; print(sum(x["ClientStatus"] == "running" for x in json.load(sys.stdin)))') - [[ $running == 0 ]] && break - sleep 2 -done -[[ $running == 0 ]] || { echo "$instance still has a running allocation" >&2; exit 1; } - -snapshot="$dataset@before-yt-feed-import-$(date +%s)-$$" -"${remote[@]}" "zfs snapshot $snapshot" -rsync -a --delete -e "ssh -p $target_port" "$scratch/" "$target_host:$root/data/" -uid=$("${remote[@]}" "python3 -c 'import json; print(json.load(open(\"/var/lib/studio/identities.json\"))[\"ytdl\"])'") -"${remote[@]}" "chown -R $uid:$uid $root/data" -drift=$(rsync -rnc --delete --out-format='%n' -e "ssh -p $target_port" "$scratch/" "$target_host:$root/data/") -[[ -z $drift ]] || { echo "Copied state differs from source; restore $snapshot" >&2; exit 1; } - -if [[ $instance == ytdl ]]; then - if [[ $offline == false ]]; then - [[ $(ssh "$source_host" "sudo -n docker inspect -f '{{.State.Running}}' yt-feed") == false ]] || { - echo 'Zenith yt-feed restarted during import; leave the home server stopped' >&2 - exit 1 - } - fi - "${remote[@]}" 'systemctl start studio-dashboard' - echo "Imported production YTDL from Zenith. Previous dataset state: $snapshot" -else - python3 "$(dirname "$0")/deploy.py" stage ytdl - "${remote[@]}" "zfs destroy $snapshot" - echo "Imported and tested $instance" -fi diff --git a/tools/intake-migration.md b/tools/intake-migration.md deleted file mode 100644 index 4bd24af70a9e744ade8cbc5992ae7bb8eb10e7a6..0000000000000000000000000000000000000000 --- a/tools/intake-migration.md +++ /dev/null @@ -1,7 +0,0 @@ -# JSON Intake migration - -Zenith's `Documents/Intake` exists but contains no files at the 2026-09-26 inspection. Snow Globe already mounts the equivalent Clover folder, so no application dataset copy is required. - -Existing Siri shortcuts and webhooks send Zenith's `INTAKE_KEY`. During production cutover, set `STUDIO_DEPLOY_HOST` and `STUDIO_DEPLOY_PORT` for the production host, then run `bash tools/import-legacy-secrets.sh intake INTAKE_KEY` to stream that value from Zenith's `.env` into Snow Globe's Nomad secret. Redeploy Intake to load it. The command does not print or save the key locally; it refuses to run without an explicit target. - -On 2026-09-26, `intake-preview-7f29ae9a` inherited the VM production key and cloned the Clover dataset for its writable mount. Its HTTPS route rejected an incorrect key with 401 and accepted a test row with the inherited key; the row appeared only in the clone, then was removed. Release `7b6aac707cf7c55f` promoted successfully after a backup of 22 service datasets and four databases, and all 19 HTTP routes passed. Destroying the stage removed its Nomad job, secret, ZFS clone, and source snapshot. Production Intake remained healthy and its Clover folder stayed empty. diff --git a/tools/jellyfin-migration.md b/tools/jellyfin-migration.md deleted file mode 100644 index 2780ff47451ccefd8d0ced81cf583f968a3c246d..0000000000000000000000000000000000000000 --- a/tools/jellyfin-migration.md +++ /dev/null @@ -1,13 +0,0 @@ -# Jellyfin media migration proof - -An initial small preview used a consistent SQLite backup from Zenith, verified by `PRAGMA integrity_check` before and after transfer. Its 4,879 `/media/...` paths matched Zenith's database by sorted path digest. On Zenith, 4,823 resolved in the existing media dataset; the other 56 were already missing at the source (50 videos, 3 trailers, 2 folders, and 1 photo). - -The VM mounts Zenith's media tree read-only at `/srv/clover/Media`, which Jellyfin receives as `/media`. Twenty evenly spaced source paths that exist on Zenith also resolved through the VM mount. The staged app is healthy without copying the 3.97 TiB media dataset. [The media cutover plan](media-cutover.md) moves that dataset to the final `/srv/clover/Media` mountpoint while preserving hardlinks and snapshots. - -A full preview restore used the existing read-only `storage1/apps@hourly-2026-09-26_01-00` snapshot as its source. [import-jellyfin.sh](import-jellyfin.sh) copied Jellyfin's metadata, artwork, attachments, subtitles, intro-skipper state, and plugins while excluding caches, logs, and transcodes. The copy was 4.7 GB on the Mac and 3.2 GB in the VM's compressed ZFS dataset; checksums matched before the app started. The pinned 10.11.11 image migrated the 10.11.5 database successfully. Both sides had 17,466 base items, 22 users, 7,717 user-data rows, and the same digest for 4,879 media paths after boot. The source and stage also matched counts for 7,149 metadata files, 873 attachments, and 1,574 subtitles. SQLite integrity and HTTPS health passed; the SSO start endpoint redirected to the staged Keycloak client. - -Production import (`STUDIO_DEPLOY_HOST= STUDIO_DEPLOY_PORT=22 sh tools/import-jellyfin.sh jellyfin`) uses the stopped live source, requires the new Jellyfin job stopped and the real Media ZFS filesystem mounted, snapshots the destination, verifies the copy, assigns Jellyfin's allocated UID, and leaves the new job stopped for cutover. During the preview boot, a three-second Nomad health probe timed out under load and Caddy briefly removed the stage route. The probe now allows 15 seconds; the updated preview deployment passed. The imported Jellyfin settings also automatically installed newer AniList and Intro Skipper plugins after startup. - -After a same-machine OS replacement, set `STUDIO_LEGACY_HANDOFF` to the [offline handoff](legacy-handoff.md) directory. The production importer then copies the old app metadata directly between mounted datasets on the new host, without a Mac scratch copy or old Docker daemon. The 3.97 TiB Media dataset remains a ZFS mount and is not copied. - -The service binds the reorganized host folders at their original `/media/jellyfin/...` container paths using the shared [config/site.pkl](../config/site.pkl) mapping. Both existing library settings and new-instance setup keep those paths; no database path rewrite is needed. See the [current service layout](media-cutover.md#service-paths-for-the-reorganized-tree). This follows [Jellyfin's migration guidance](https://jellyfin.org/docs/general/administration/migrate/) to preserve the paths seen by the application. diff --git a/tools/knife.md b/tools/knife.md index 6a99d27fe21b2611262af09bf2018af41e8e2033..81b2a4c55b8dc958d0e68ec4c8ea90e955e5f861 100644 --- a/tools/knife.md +++ b/tools/knife.md @@ -38,7 +38,7 @@ an allocation ID: ```sh ssh -p 22 root@10.0.0.1 -export NOMAD_TOKEN="$(cat /var/lib/studio/nomad.token)" +export NOMAD_TOKEN="$(cat /var/lib/snowglobe/nomad.token)" knife_allocation=$(nomad job allocs -json knife | python3 -c 'import json,sys; a=[r["ID"] for r in json.load(sys.stdin) if r["ClientStatus"]=="running" and r["DesiredStatus"]=="run"]; assert len(a)==1; print(a[0])') knife_container="app-$knife_allocation" podman exec "$knife_container" knife auth devices revoke-all @@ -75,14 +75,3 @@ requirements or corrupt downloads fail startup. For a new image, review the bund and update `service/knife/bundle.json`, then use the normal stage → main → prod workflow. Runtime downloads under `data/runtime/knife-container` survive an infra image rollback; rolling back code does not restore state or archive objects. - -## 2026-10-16 SSO verification - -A fresh Zenith preview completed Clover's real Snow Globe sign-in, verified the -issuer/subject account binding, renewed its browser session, and opened an -authenticated WebSocket through HTTPS. Anonymous access, missing CSRF, invalid -OIDC/native transactions, and corrupt archive objects were rejected. A synthetic -object uploaded and downloaded identically with working byte ranges; temporarily -removing the preview's archive marker returned 503 and its original marker was -restored unchanged. The private allocation JSON matched its separate Nomad client, -used mode 0400 and UID 3132, and its intended bind mount rejected writes. diff --git a/tools/legacy-handoff.md b/tools/legacy-handoff.md deleted file mode 100644 index 3070cb869de952ea36eb3ef3480855007bdbc51d..0000000000000000000000000000000000000000 --- a/tools/legacy-handoff.md +++ /dev/null @@ -1,28 +0,0 @@ -# Same-machine PostgreSQL handoff - -The retained apps dataset is an encrypted ZFS root. Keep its mountpoint explicitly at `/mnt/storage1/apps` after renaming the pool to `globe` and replacing Zenith with NixOS; [media-cutover.md](media-cutover.md) gives the command order. The old app directories and `/mnt/storage1/apps/home-infra/.env` remain there; the new `/srv/prod` root is separate. - -If TrueNAS is still running, stop the old apps being migrated, leaving old PostgreSQL running long enough to export. Run `bash tools/export-legacy-postgres.sh` before replacing the OS. Zenith grants the `clo` account passwordless `sudo docker` access for the export; this online exporter requires the original `storage1/apps` mount and exports Forgejo and HedgeDoc only. It does not stop apps itself. - -If already booted into the installer, [export-offline-postgres.py](export-offline-postgres.py) exports every connectable database and all roles from both retained clusters. It requires all PostgreSQL servers to be stopped, matching-major runtimes with the source extensions installed, and `glibcLocales`. It starts private RAM copies over private Unix sockets, saves complete physical archives including `template0`, checks the original file hashes remain unchanged, and restores every logical dump into fresh temporary clusters. Verification compares every user table's contents, sequence state, and large objects. The originals are never started. Run as root on the installer, with the runtime paths supplied by the pinned Nix builds: - -```sh -python3 tools/export-offline-postgres.py \ - --cluster /mnt/storage1/apps/pg_data "$postgres17_runtime" \ - --cluster /mnt/storage1/apps/pg_data18/18/docker "$postgres18_runtime" \ - --locale-archive "$glibc_locales/lib/locale/locale-archive" -``` - -It publishes a private `studio-handoff/-` directory only after successful restores. Its `pg17` and `pg18` directories contain all database dumps, `globals.sql`, and `cluster.tar.gz`; the manifest records their hashes and restore evidence. Root-level Forgejo and HedgeDoc dump links select the PG18 exports for the existing importers. Dawarich's old database is retained in the full export even though its new service starts fresh. Copy the complete handoff directory, preserving these links, to a private off-server archive and verify the file hashes before replacing the OS. `globals.sql` contains role password hashes; protect the archive like the recovery keys. Failed exports remain under `.pending-*` and cannot be selected by the importers. - -The October 4 installer export restored and verified nine databases and eleven roles from each cluster. Its complete archive is saved on the Mac at `/Users/clo/Library/Application Support/Zenith Recovery/postgres-backups/20261004T230507Z-e75419.tar`; every archived file hash and both physical cluster contents were checked against the server manifest. The production handoff is: - -```sh -export STUDIO_LEGACY_HANDOFF=/mnt/storage1/apps/studio-handoff/20261004T230507Z-e75419 -``` - -After NixOS has imported and unlocked the apps dataset, set `STUDIO_DEPLOY_HOST`, `STUDIO_DEPLOY_PORT`, and `STUDIO_LEGACY_HANDOFF` to that printed directory. The importers require the mounted pool's encrypted apps dataset at `/mnt/storage1/apps` and a stopped destination job. HedgeDoc and evil.inc Forgejo verify dump hashes and source table counts before restoring their PostgreSQL databases. Shale, Jellyfin, Navidrome, PDS, qBittorrent, Sonarr, Radarr, Jackett, and YouTube Triage can copy their retained app data without the old Docker daemon. Each importer prints its pre-import backup or ZFS snapshot. - -`import-legacy-secrets.sh` reads the retained old `.env` from the new host when `STUDIO_LEGACY_HANDOFF` is set. It sends only the requested keys into Snow Globe's Nomad variables; it does not print their values. The handoff directory must exist on the target. Without that variable, the script continues reading from the old Zenith host for a two-host migration. - -The handoff contains no Forgejo repository files. [import-evil-forgejo.sh](import-evil-forgejo.sh) checks its `evil-forgejo.dump`, secret fingerprints, and the stopped Snow Globe job, then copies the old 14 GB app directory directly from the mounted dataset. The Forgejo job stays stopped until its files, database, and account identities have been checked together. diff --git a/tools/local-ai-cache.md b/tools/local-ai-cache.md index 78362dcd5318ef7d9daea77589dc6b6c45908bde..02db1c65a278f55b639096c85f9345bb27efe2e2 100644 --- a/tools/local-ai-cache.md +++ b/tools/local-ai-cache.md @@ -20,7 +20,7 @@ plugin versions, key, KV layout, and parallel configuration. Corrupt blocks fall back to another copy or become misses. Key rotation invalidates reuse; old entries remain encrypted and subject to eviction. -`config/site.pkl` owns the NVMe path (`/var/cache/studio/local-ai` by default). +`config/site.pkl` owns the NVMe path (`/var/cache/snowglobe/local-ai` by default). The prepare hook creates it with the assigned service owner. HDD blocks use the managed service dataset. Fresh previews use separate managed directories for both disk tiers. @@ -48,7 +48,7 @@ blocks totaling 1,912,086,528 bytes. Forced RAM/disk restore probes stopped when the endpoint reported busy; end-to-end disk restore timing remains unverified. -## Zenith measurements — 2026-10-09 +## Snow Globe measurements — 2026-10-09 HyperQwen, vLLM 0.27.1, MTP with three draft tokens, 150,000-token context, GPU prefix caching enabled. Production stayed running; no cache @@ -98,13 +98,13 @@ timings and counters; it never prints prompt text, generated text, or keys. ```sh ssh -o BatchMode=yes root@10.0.0.1 \ - 'pid=$(pgrep -x vllm); nsenter -t "$pid" -n python3 - prefill "$pid" --hostname zenith' \ + 'pid=$(pgrep -x vllm); nsenter -t "$pid" -n python3 - prefill "$pid" --hostname snowglobe' \ < tools/benchmark-local-ai-cache.py ssh -o BatchMode=yes root@10.0.0.1 \ - 'python3 - filesystem /srv/staging --hostname zenith' \ + 'python3 - filesystem /srv/staging --hostname snowglobe' \ < tools/benchmark-local-ai-cache.py ssh -o BatchMode=yes root@10.0.0.1 \ - 'python3 - filesystem /var/tmp --hostname zenith' \ + 'python3 - filesystem /var/tmp --hostname snowglobe' \ < tools/benchmark-local-ai-cache.py ``` diff --git a/tools/local-vm.sh b/tools/local-vm.sh index 718e24d926855412a6c2230158346dcc7c01c5b4..2da2913c67a9ea17d04038d268d4504f17771255 100644 --- a/tools/local-vm.sh +++ b/tools/local-vm.sh @@ -1,7 +1,7 @@ #!/bin/sh set -eu -cache=${STUDIO_VM_CACHE_DIR:-$HOME/Library/Caches/clover-studio-vm-x86} +cache=${SNOWGLOBE_VM_CACHE_DIR:-$HOME/Library/Caches/clover-snowglobe-vm-x86} mode=${1:-} qemu=$(command -v qemu-system-x86_64 || true) if [ -z "$qemu" ]; then @@ -25,14 +25,14 @@ if [ "$mode" != installed ] && [ -f "$cache/nixos-minimal.iso" ]; then set -- -drive "file=$cache/nixos-minimal.iso,media=cdrom,readonly=on" fi if [ "$mode" = installed ]; then - media=${STUDIO_VM_MEDIA_SOURCE:-/Volumes/media} + media=${SNOWGLOBE_VM_MEDIA_SOURCE:-/Volumes/media} if ! mount | grep -F " on $media (" >/dev/null; then echo "Mount the media share at $media before starting the VM" >&2 exit 1 fi set -- "$@" -virtfs "local,path=$media,mount_tag=media,security_model=none,readonly=on" fi -if [ "${STUDIO_VM_HEADLESS:-0}" = 1 ]; then +if [ "${SNOWGLOBE_VM_HEADLESS:-0}" = 1 ]; then set -- "$@" -display none -serial "file:$cache/serial.log" -daemonize -pidfile "$cache/qemu.pid" else set -- "$@" -display cocoa -serial mon:stdio diff --git a/tools/log-shipper.py b/tools/log-shipper.py index 6296687be245aeb1e348a6225d1a2d9eafd863fd..a0f5ce5f93a5d107b0fb930e54b57595a616f1de 100644 --- a/tools/log-shipper.py +++ b/tools/log-shipper.py @@ -9,13 +9,13 @@ import time from urllib.request import Request, urlopen -STATE = Path("/var/lib/studio/log-shipper.json") +STATE = Path("/var/lib/snowglobe/log-shipper.json") LOGS = Path("/var/lib/nomad/alloc") CRI = re.compile(r"^(\S+) (stdout|stderr) [FP] ?(.*)$") def nomad(path): - token = Path("/var/lib/studio/nomad.token").read_text().strip() + token = Path("/var/lib/snowglobe/nomad.token").read_text().strip() with urlopen(Request("http://127.0.0.1:4646" + path, headers={"X-Nomad-Token": token}), timeout=10) as response: return json.load(response) diff --git a/tools/media-cutover.md b/tools/media-cutover.md deleted file mode 100644 index 6559c9a6cdb51c703c81f17c69cf6ac914905ef5..0000000000000000000000000000000000000000 --- a/tools/media-cutover.md +++ /dev/null @@ -1,222 +0,0 @@ -# Media dataset cutover - -Before cutover, TrueNAS mounted `storage1/media` at `/mnt/storage1/media`. It is one 3.97 TiB ZFS filesystem containing `jellyfin`, `music`, `music_intake`, and `torrent`; `jellyfin` and `torrent` report the same filesystem device. The dataset had one snapshot with no clones at inspection. The VM reads it through a read-only mount at `/srv/clover/Media` without copying its contents. - -The Mac mounted the VM's `media` SMB share over Tailscale, listed the source folders, and received a write error for a test file. The `clover` share is writable in the VM; both shares use the same `clo` account. - -Zenith's NFSv4 ACLs need a separate [permission cutover](storage-acl-cutover.md) before this dataset is served by NixOS. - -The new layout needs that same dataset at `/srv/clover/Media`. A disposable VM test moved a ZFS filesystem beneath a different parent with `zfs rename`, changed its mountpoint, and confirmed that its snapshot, inode, and hardlinks survived. A second test renamed an independent encrypted root beneath another encrypted root; it remained its own encryption root. These test the ZFS operations, not the live Zenith cutover. - -Run the layout commands as root in the live installer, one step at a time. Stop on any error. The old writers and share services must be stopped; booting the installer leaves the old TrueNAS services stopped. - -### Reopen the inspected pool for writes - -The recovery inspection imported `storage1` read-only with an alternate root under `/run/infra2-inspection`. Export releases that import; it does not delete the pool. Reimport the verified data pool by its numeric ID, without an alternate root. `-N` leaves every dataset unmounted, `cachefile=none` avoids recording this temporary import, and `-f` permits the installer to take over the pool last used by TrueNAS. This ID identifies the four-disk data pool, not the NVMe's `boot-pool`. - -```sh -zpool export storage1 -zpool import -N -f -d /dev/disk/by-id -o cachefile=none 8944904502819631621 -zpool get readonly,altroot storage1 -``` - -The last command must show `readonly=off` and `altroot=-` before continuing. Leave `boot-pool` alone. - -### Record identity and take a snapshot - -Keep this shell open: `media_guid` records the dataset's identity for the check after renaming. The recursive snapshot captures all existing datasets' file state before cutover. It does not undo dataset names or mountpoint properties. - -```sh -media_guid=$(zfs get -H -o value guid storage1/media) -zfs snapshot -r storage1@before-infra2-layout -``` - -### Rename the pool to globe - -The top-level dataset has the pool's name. Rename it by exporting and importing the pool with a new name, not with `zfs rename`. These commands also work midway through the mountpoint changes below, provided the datasets remain unmounted. - -```sh -zpool export storage1 -zpool import -N -d /dev/disk/by-id -o cachefile=none 8944904502819631621 globe -zpool get guid,readonly,altroot globe -zfs list -r -o name,mountpoint,mounted globe -``` - -Expect the same pool GUID `8944904502819631621`, `readonly=off`, and `altroot=-`. Every dataset and snapshot prefix becomes `globe/`; their GUIDs, encryption keys, and snapshots are retained. The explicit `/srv` mountpoint and pinned `/mnt/storage1/apps` path remain unchanged. The production configuration uses `globe` for imports and service startup checks. See [OpenZFS pool import](https://openzfs.github.io/openzfs-docs/man/master/8/zpool-import.8.html). - -From here onward, use `globe` in ZFS arguments. Keep `/mnt/storage1/apps` as a filesystem path: the legacy importers still read it there. Existing recovery exports label their keys with the original `storage1/...` names; those same keys now unlock the corresponding `globe/...` datasets. - -### Set the layout - -`mountpoint` controls where a dataset appears in the filesystem. `rename` changes its name and parent in the ZFS hierarchy. The `-u` flags update metadata without mounting datasets. Encrypted datasets can be renamed while locked; these commands do not change their keys or copy their file contents. See [OpenZFS rename](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-rename.8.html) and [mountpoint properties](https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html). - -| Command | Effect | -| --- | --- | -| `zfs set -u mountpoint=/mnt/storage1/apps globe/apps` | Pin the retained app tree at the path used by the legacy importers, before moving the pool root. | -| `zfs set -u mountpoint=/srv globe` | Move the pool's root mountpoint; default-mountpoint children such as backups, logs, and mirrors follow it. | -| `zfs set -u mountpoint=/srv/clover globe/clover` | Set Clover's mount location. Its dataset name stays `globe/clover`. | -| `zfs rename -u globe/media globe/clover/Media` | Move the existing Media dataset beneath Clover and capitalize its name. Its snapshots stay attached to the same dataset. | -| `zfs set -u mountpoint=/srv/clover/Media globe/clover/Media` | Set Media's mount location. It remains an independent encryption root with its original key. | - -The destination dataset must not already exist, and the `Media` directory inside Clover must be absent or empty before mounting there. The live inspection checked both on 2026-10-04: the dataset and directory were absent. - -### Verify before mounting - -```sh -test "$(zfs get -H -o value guid globe/clover/Media)" = "$media_guid" -zfs list -o name,mountpoint,mounted globe globe/apps globe/clover globe/clover/Media -zfs get encryptionroot globe/apps globe/clover globe/clover/Media -zfs list -t snapshot -r -o name globe/clover/Media -``` - -The identity check must succeed. Expect these mountpoint properties, with `mounted=no` throughout: - -```text -globe /srv -globe/apps /mnt/storage1/apps -globe/clover /srv/clover -globe/clover/Media /srv/clover/Media -``` - -Media's `encryptionroot` must be `globe/clover/Media`, not `globe/clover`; apps and Clover must remain their own encryption roots. Media's original snapshots should now use its new dataset prefix, alongside `@before-infra2-layout`. Loading keys and mounting the filesystems follow this metadata check; permission conversion is covered in [storage-acl-cutover.md](storage-acl-cutover.md). Preserve the old NVMe until the PostgreSQL handoff in [legacy-handoff.md](legacy-handoff.md) is verified. - -A disposable pool with an altroot reproduced the default child's move when the parent mountpoint changed, then kept the old child path after an explicit mountpoint was set. After loading keys and mounting, Clover and Media must resolve to distinct ZFS filesystems before starting Snow Globe; `tools/studio.py pool` enforces this. Check a known hardlinked download/library pair by device and inode, then verify Jellyfin and Navidrome library paths inside their containers. - -The root change also relocates other default-mountpoint children: `backup` (including the 2.07 TB `backup/sandwich`), `mirrors` (266 GB), `agent`, `homes`, `logs`, and `.ix-virt`. They remain datasets under `/srv` until deliberately renamed; the move does not copy their data. `ix-apps` has a local `/mnt/.ix-apps` mountpoint and `.system` uses `legacy`, so neither follows the root. Preserve these datasets during cutover. - -### Remove retired app clones and the rehearsal store - -The five `apps-hourly-…-clone` entries are cloned filesystems, distinct from the original `globe/apps@hourly-…` snapshots. The rehearsal store is `globe/apps/snowglobe-rehearsal-20261002` (59.8 GiB at inspection). The live installer inspection on 2026-10-04 confirmed that all six are unmounted, and `zfs destroy -nvr` previews succeeded: each would delete only the named filesystem and its own `@before-infra2-layout` snapshot. - -These commands permanently remove those six retired filesystems and their snapshots. They preserve `globe/apps`, its original hourly snapshots, and the other datasets' cutover snapshots. Run one command at a time and stop on an error. To preview a command again, replace `-vr` with `-nvr`; lowercase `-r` includes the target's children and snapshots. Do not use uppercase `-R`, which also destroys dependent clones outside the target hierarchy. See [OpenZFS destroy](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-destroy.8.html). - -```sh -zfs destroy -vr globe/apps-hourly-2026-09-06_21-00-clone -zfs destroy -vr globe/apps-hourly-2026-09-08_23-00-clone -zfs destroy -vr globe/apps-hourly-2026-09-11_07-00-clone -zfs destroy -vr globe/apps-hourly-2026-09-11_12-00-clone -zfs destroy -vr globe/apps-hourly-2026-09-12_14-00-clone -zfs destroy -vr globe/apps/snowglobe-rehearsal-20261002 -``` - -Check the remaining app datasets and source snapshots afterward: - -```sh -zfs list -r -o name,used,mountpoint globe/apps -zfs list -t snapshot -d 1 -o name globe/apps -``` - -### Unlock and mount Clover and Media - -Clover and Media are independent encryption roots, so each needs its original key. The recovery export labels these `storage1/clover` and `storage1/media`, respectively. For the installer commands below, the recovered keys must be available as `/run/infra2-keys/clover.hex` and `/run/infra2-keys/media.hex`, with directory mode `0700` and file mode `0600`, owned by root. These temporary files disappear at reboot; recover them from the private archive again if needed. Keep keys out of this repository and terminal output. - -Run one command at a time and stop on errors. Mount the pool root if needed, then verify that `/srv` is its actual mountpoint: - -```sh -if [ "$(zfs get -H -o value mounted globe)" = no ]; then zfs mount globe; fi -test "$(findmnt -n -o SOURCE --mountpoint /srv)" = globe -``` - -Load the two keys, mount Clover before its Media child, then remove the temporary key files after both mounts succeed. `-L` overrides the key source for this command without changing the datasets' persistent `keylocation=prompt`. Loading keys alone does not mount them; see [OpenZFS load-key](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-load-key.8.html). Use ordinary [ZFS mounts](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-mount.8.html), without overlay options; stop if the Media mountpoint is nonempty. - -```sh -zfs load-key -L file:///run/infra2-keys/clover.hex globe/clover -zfs load-key -L file:///run/infra2-keys/media.hex globe/clover/Media -zfs mount globe/clover -zfs mount globe/clover/Media -rm /run/infra2-keys/clover.hex /run/infra2-keys/media.hex -test "$(findmnt -n -o SOURCE --mountpoint /srv/clover)" = globe/clover -test "$(findmnt -n -o SOURCE --mountpoint /srv/clover/Media)" = globe/clover/Media -findmnt -rn -o SOURCE,TARGET,FSTYPE --mountpoint /srv/clover/Media -``` - -The last command must show `globe/clover/Media /srv/clover/Media zfs`. This mounts only Clover and Media beneath the pool root. Continue to the folder renames below after these checks succeed. Root can perform these renames; service access still requires the separate ACL cutover. - -### Unlock retained apps for the legacy handoff - -The original apps key is labeled `storage1/apps` in the recovery export. Prepare it as `/run/infra2-keys/apps.hex` with the same root-only permissions as the Clover and Media keys. Unlock and mount the retained dataset at its pinned legacy path: - -```sh -zfs load-key -L file:///run/infra2-keys/apps.hex globe/apps -zfs mount globe/apps -rm /run/infra2-keys/apps.hex -test "$(findmnt -n -o SOURCE --mountpoint /mnt/storage1/apps)" = globe/apps -findmnt -rn -o SOURCE,TARGET,FSTYPE --mountpoint /mnt/storage1/apps -``` - -Expect `globe/apps /mnt/storage1/apps zfs`. Keep this path intact for the [legacy handoff](legacy-handoff.md); mounting the dataset does not start the old apps. - -### Keys after installation - -The recovery keys remain in the private recovery archive on the Mac. The `/run/infra2-keys` files above are temporary transfer copies; after `load-key`, ZFS holds the loaded key material in memory, so deleting those files does not relock mounted datasets. Reboot loses the loaded keys. The datasets retain `keylocation=prompt` for manual recovery; the installed target loads TPM-encrypted credentials before mounting them. Nomad's mount checks prevent it from starting against locked data. - -Zenith's enabled TPM 2.0 (`/dev/tpm0` and `/dev/tpmrm0`) sealed and decrypted all six encryption-root keys during installation preparation on 2026-10-04. A systemd service test successfully loaded the encrypted credentials and unlocked the previously locked logs dataset. [zenith-hardware.md](zenith-hardware.md) records their recovery locations and the installed boot service; the root names and credential paths have one home in [zenith.nix](../nixos/zenith.nix). - -[TPM-encrypted systemd credentials](https://github.com/systemd/systemd/blob/main/man/systemd-creds.xml) keep encrypted key blobs on the NVMe and supply decrypted credentials in RAM. These credentials bind to the original TPM and PCR 7. Secure Boot is disabled, so this supplies convenient automatic unlock and protection for detached drives, without establishing a trusted boot chain. Firmware changes affecting PCR 7 or a TPM reset can require resealing the credentials from the recovery keys. - -Manual unlock over root SSH remains the recovery path: recover the relevant key to a private `/run` file, use `zfs load-key -L file:///run/ `, remove that file, then mount the dataset. Keep the off-server keys when replacing hardware; encrypted blobs from the old TPM cannot unlock on a replacement machine. - -### Rename the media folders - -Clover and Media both report `casesensitivity=insensitive` on the live pool. Names differing only by capitalization resolve to the same entry. For a case-only folder rename, use an unused temporary name; ordinary `mv music Music` interprets the existing destination directory as a request to move `music` inside itself. From the Media directory: - -```sh -test ! -e .music-case-rename && test ! -L .music-case-rename && -mv -T -- music .music-case-rename && -mv -T -- .music-case-rename Music -``` - -Lowercase configured paths still resolve after a case-only rename; do not create a separate `music -> Music` or `seedbox -> Seedbox` alias, because those names collide on this filesystem. The distinct `torrent` alias below preserves the old host path. qBittorrent also receives a container mount at `/data/media/torrent` pointing to `Seedbox`, so its saved torrent paths remain valid. - -Zenith's qBittorrent `BT_backup` is 21 MB. A checksum-verified copy contained 544 complete bencoded resume files. Their `save_path` values include 498 at `/data/media/torrent`, three in its subfolders, and 43 in Jellyfin folders; 543 also set `qBt-downloadPath` to `/data/media/torrent`. Keep these saved paths working by renaming the real media folder and leaving a relative compatibility symlink, after the media dataset is mounted at its new path and the old writers are stopped: - -```sh -test -d /srv/clover/Media/torrent -test ! -e /srv/clover/Media/Seedbox -mv /srv/clover/Media/torrent /srv/clover/Media/Seedbox -ln -s Seedbox /srv/clover/Media/torrent -test "$(stat -c %d:%i /srv/clover/Media/Seedbox)" = "$(stat -Lc %d:%i /srv/clover/Media/torrent)" -``` - -The same-dataset rename preserves existing hardlinks. New downloads use `/data/media/Seedbox`; saved torrents continue to resolve through `/data/media/torrent`. If `Seedbox` and the `torrent` alias already exist, skip the rename and verify their device/inode identity. Rename the existing music intake folder after stopping YouTube Triage, then keep its current container path working through a relative symlink: - -```sh -test -d /srv/clover/Media/music_intake -test ! -e '/srv/clover/Media/Intake - Music' -mv /srv/clover/Media/music_intake '/srv/clover/Media/Intake - Music' -ln -s 'Intake - Music' /srv/clover/Media/music_intake -``` - -### Service paths for the reorganized tree - -The service definitions read the current capitalized host folders. Jellyfin and qBittorrent retain their saved paths inside the containers through individual bind mounts; the host needs no `jellyfin` compatibility directory. The shared folder mapping is defined once in [config/site.pkl](../config/site.pkl), under `jellyfinFolders`. Jellyfin mounts its entries beneath `/media/jellyfin`; qBittorrent mounts them beneath `/data/media/jellyfin` and mounts `Seedbox` at both `/data/media/Seedbox` and `/data/media/torrent`. - -Navidrome mounts `Music` at `/music`. Jellyfin and Navidrome mounts remain read-only; qBittorrent follows `site.mediaReadOnly`, so read-only rehearsals stay read-only. Keeping the recorded Jellyfin paths follows its [migration guidance](https://jellyfin.org/docs/general/administration/migrate/) and avoids rewriting its database paths or library settings. - -Before reorganization, YouTube Archiver kept its download archive JSON beside the videos in `Media/jellyfin/Independent`, not in its `/config` cache ([upstream archive behavior](https://github.com/jmbannon/ytdl-sub/wiki/5.-Optimizing-Your-First-Config)). The source inspection found 12 archive files there (52,345 bytes), and all 12 were visible through the VM's read-only media mount. That folder now lives at `Media/Videos/Independent`; preserve its archive files with the videos. The old app directory contained only a lock, cache, and empty work directory. - -The complete 33 MB qBittorrent config was also copied into a disposable ZFS clone in the VM. The pinned container started with `--network none` and no media mount; its API loaded all 544 torrents. All reported `missingFiles`, as expected without `/data/media`. One resume file without `qBt-savePath` adopted the new default `/data/media/seedbox`; the other 543 retained their saved paths. The container, clone, and copied config were removed after the test. - -A second disposable restore copied Zenith's qBittorrent profile into the VM and mounted the real read-only media through a temporary `/data/media/seedbox` alias, with `torrent -> seedbox` alongside it. The container ran as UID 3000 with no network and a read-only root filesystem. Its API loaded all 544 torrents; after resume checks, all 544 reported 100% progress, with 539 `queuedUP`, three `stalledUP`, two `forcedUP`, and none `missingFiles`. The container resolved the legacy symlink, and its temporary profile was removed. This tests the saved paths and source files through the VM mount; the actual Zenith folder has not yet been renamed. - -The old resume files are owned by UID 3000, while Snow Globe assigns qBittorrent UID 3114. [import-qbittorrent.sh](import-qbittorrent.sh) copies the stopped old profile into the stopped Snow Globe service's dataset, snapshots the destination, checks the copy, changes ownership, applies the new save path, and starts the service. It refuses to run while Zenith's qBittorrent is active. In a disposable VM restore, UID 3114 loaded all 544 imported torrents without a media mount; all correctly reported `missingFiles`. With the same read-only media alias and legacy symlink as above, all 544 reached 100% progress and none remained `missingFiles`. - -The production importer requires the real Media ZFS mount and a `torrent` symlink resolving to the same directory as `Seedbox` before it starts qBittorrent. It compares device/inode identity, so `torrent -> Seedbox/` is accepted. For a same-machine OS replacement, set `STUDIO_LEGACY_HANDOFF` to the [offline handoff](legacy-handoff.md) directory; the profile copy then reads the retained old apps dataset without old Docker. - -The PIA credentials are the two lines in `~/pia.txt` on the Mac (username, then password). With `STUDIO_DEPLOY_HOST` and `STUDIO_DEPLOY_PORT` set for production, import them through `python3 tools/deploy.py secrets qbittorrent --file ~/pia.txt --key vpn_user --key vpn_pass` before deploying qBittorrent. The CLI requires a private file, sends its contents to the target over SSH, and does not print the values. - -At the 2026-09-26 check, Zenith's Sonarr and Radarr databases each had an enabled qBittorrent client and enabled RSS indexers; the VM's production and preview copies had both disabled. The preview setup disables every indexer's RSS and automatic search, including Sonarr's built-in EZTV indexer, while leaving the source untouched. A production [ARR import](import-arr.sh) now requires the corresponding Zenith container to be stopped before copying its database, and checks it remains stopped when the copy finishes. Reenable downloads only after the media dataset is writable at the final mountpoint and qBittorrent's imported profile is serving the same paths. - -After a same-machine OS replacement, `STUDIO_LEGACY_HANDOFF` makes the Sonarr and Radarr importer read their stopped databases and app files from the mounted old apps dataset on the new host. A read-only SQLite backup and integrity check passed for each old database. - -The VM runs Sonarr `4.0.20.3014` and Radarr `6.4.4.10685`. [Sonarr v4's External mode](https://wiki.servarr.com/sonarr/settings) delegates authentication to a reverse proxy, and [Radarr's current FAQ](https://wiki.servarr.com/radarr/faq) says native OIDC is unsupported. Sonarr's [v5 development settings](https://github.com/Sonarr/Sonarr/blob/v5-develop/src/Sonarr.Api.V5/Settings/GeneralSettingsResource.cs) contain OIDC fields; the latest stable [Sonarr release](https://github.com/Sonarr/Sonarr/releases) remains v4 at this check. Both pinned services use Caddy Forward Auth with External mode. Unauthenticated UI and API requests redirected to sign-in, and forged identity headers sent from the Mac still received HTTP 302 over trusted TLS. Nomad advertised their application ports only on `127.0.0.1`. - -[import-jackett.sh](import-jackett.sh) makes the Jackett dependency repeatable: previews copy the latest read-only Zenith app snapshot; production import requires both old and new Jackett stopped. The restored preview kept the same API key as Zenith and the VM's production Jackett, preserved all three indexer definitions, and served `t=caps` for Nyaa and Pirate Bay with HTTP 200. The imported Sonarr and Radarr Jackett indexers use that same key and point to the current Nomad Jackett endpoint. - -With `STUDIO_LEGACY_HANDOFF` set, production Jackett import reads its retained config from the new host's mounted old apps dataset after reboot; it no longer needs the old Docker host. - -Sonarr and Radarr persist their Jackett and qBittorrent endpoints in their databases, so those two internal ports are static (`30017` and `30038`) across allocation changes. On the VM, both apps' saved Jackett URLs matched the current reserved port. Their public HTTPS routes still use separate Caddy endpoints and forward auth. - -[import-navidrome.sh](import-navidrome.sh) restores the old `navidrone` app directory into Navidrome's managed dataset. The preview imported Zenith's `storage1/apps@hourly-2026-09-26_02-00` snapshot, migrated the database under the pinned image, and passed SQLite integrity and HTTP health checks. Its three users, 10,945 media files, 1,046 albums, and 2,655 artists matched the source snapshot; `snow` remained an administrator. A `Remote-User: snow` request to the production Navidrome API returned the existing `snow` user, proving that its external-auth setting accepts the proxy's header. The preview's database is retained, but its job is stopped while the Jackett preview uses the VM's limited memory. Production import requires both Navidrome jobs stopped and the real Media dataset mounted as ZFS. - -For a same-machine OS replacement, set `STUDIO_LEGACY_HANDOFF` to the [offline handoff](legacy-handoff.md) directory when importing production Jellyfin or Navidrome. Both importers read their retained app data directly from the mounted old apps dataset on the new host; the Media library stays on its ZFS dataset. diff --git a/tools/namespace-transition-test.py b/tools/namespace-transition-test.py new file mode 100644 index 0000000000000000000000000000000000000000..474a24be211d9d7415ed36c0cd9d4fb7d6001347 --- /dev/null +++ b/tools/namespace-transition-test.py @@ -0,0 +1,338 @@ +#!/usr/bin/env python3 +from importlib import import_module +import io +import json +import os +from pathlib import Path +import shutil +import subprocess +import sys +import tempfile +from types import SimpleNamespace +import unittest +from unittest.mock import patch +import xml.etree.ElementTree as ET + +import release +import snowglobe + + +transition = import_module("namespace-transition") + + +class NamespaceTransition(unittest.TestCase): + def setUp(self): + temporary = tempfile.TemporaryDirectory() + self.addCleanup(temporary.cleanup) + self.root = Path(temporary.name).resolve() + self.directories = tuple((self.root / parent / "snowglobe", self.root / parent / transition.PREVIOUS) + for parent in ("opt", "state", "assets", "cache")) + self.addCleanup(patch.stopall) + patch.object(transition, "DIRECTORIES", self.directories).start() + registry = self.root / "nixos" + registry.mkdir() + patch.object(transition, "NIXOS_STATE", registry).start() + patch.object(transition.pwd, "getpwnam", side_effect=KeyError).start() + patch.object(transition.grp, "getgrnam", side_effect=KeyError).start() + + def test_staging_aliases_share_existing_state_and_never_seed_another_registry(self): + for _, old in self.directories: + old.mkdir(parents=True) + new, old = self.directories[1] + (old / "identities.json").write_text('{"knife":3132}') + transition.prepare() + transition.prepare() + self.assertTrue(new.is_symlink()) + self.assertEqual(new.resolve(), old) + (new / "identities.json").write_text('{"knife":3132,"dashboard":3133}') + self.assertEqual(json.loads((old / "identities.json").read_text()), {"knife": 3132, "dashboard": 3133}) + self.assertTrue(all(new.resolve() == old for new, old in self.directories)) + + def test_staging_bootstrap_starts_the_installed_router_generation_without_restart(self): + state = self.root / "bootstrap" + state.mkdir() + for name in ("nomad", "router", "dashboard"): + (state / (name + ".token")).write_text("disposable-" + name) + original = transition.layout + source = self.root / "installed" + (source / "tools").mkdir(parents=True) + (source / "tools" / (transition.PREVIOUS + ".py")).touch() + with patch.object(snowglobe, "STATE", state), patch.object(snowglobe, "token"), patch.object( + snowglobe.os, "geteuid", return_value=0, + ), patch.object(Path, "is_symlink", return_value=True), patch.object( + transition, "layout", return_value=original(source), + ), patch.object(snowglobe, "command", return_value=subprocess.CompletedProcess([], 0)) as command: + snowglobe.bootstrap([]) + units = [call.args for call in command.call_args_list if call.args[0] == "systemctl"] + self.assertEqual(units, [("systemctl", "start", transition.PREVIOUS + "-router.service")]) + + def test_split_state_is_rejected_before_any_alias_is_created(self): + self.directories[0][1].mkdir(parents=True) + new, old = self.directories[1] + new.mkdir(parents=True) + old.mkdir() + with self.assertRaisesRegex(ValueError, "independent control stores"): + transition.prepare() + self.assertFalse(self.directories[0][0].exists()) + + def test_promotion_preserves_whole_stores_and_release_links_then_is_idempotent(self): + revision = "1" * 16 + for _, old in self.directories: + old.mkdir(parents=True) + (old / "durable").write_text("existing bytes") + (old / "durable").chmod(0o600) + new_root, old_root = self.directories[0] + (old_root / "releases" / revision).mkdir(parents=True) + for name in ("current", "main"): + (old_root / name).symlink_to(old_root / "releases" / revision) + before = [(old / "durable").stat().st_ino for _, old in self.directories] + with patch.object(transition.subprocess, "run", return_value=subprocess.CompletedProcess([], 0)) as run, patch.object( + transition, "migrate_policies", + ) as policies: + transition.promote(self.root) + policies.assert_called_once_with(self.root, self.directories[1][0]) + self.assertEqual(run.call_count, 6) + transition.promote(self.root) + self.assertEqual(run.call_count, 6) + for index, (new, old) in enumerate(self.directories): + self.assertFalse(new.is_symlink()) + self.assertTrue(old.is_symlink()) + self.assertEqual((new / "durable").stat().st_ino, before[index]) + self.assertEqual((new / "durable").stat().st_mode & 0o777, 0o600) + for name in ("current", "main"): + self.assertEqual(os.readlink(new_root / name), str(new_root / "releases" / revision)) + + def test_old_release_manifest_and_canonical_alias_validate_without_rewriting_the_tree(self): + new_root, old_root = self.directories[0] + revision = "a" * 16 + source = old_root / "releases" / revision + (source / "tools").mkdir(parents=True) + (source / "tools" / (transition.PREVIOUS + ".py")).write_text("immutable") + manifest = source / ("." + transition.PREVIOUS + "-release.json") + manifest.write_text(json.dumps({"id": revision, "digest": release.tree_digest(source), "version": 2})) + original = manifest.read_bytes() + (old_root / "current").symlink_to(source) + transition.prepare() + with patch.object(release, "ROOT", new_root), patch.object(release, "RELEASES", new_root / "releases"): + self.assertEqual(release.current_release(), revision) + self.assertEqual(release.check_release(revision).resolve(), source) + self.assertEqual(manifest.read_bytes(), original) + + def test_old_release_activation_uses_its_units_script_and_environment(self): + revision = "b" * 16 + source = self.root / "release" + (source / "tools").mkdir(parents=True) + (source / "tools" / (transition.PREVIOUS + ".py")).touch() + (source / "nixos").mkdir() + (source / "nixos/dashboard.nix").touch() + transition.layout(source)["manifest"].write_text('{"version":2}') + observed = [] + + def run(argv, **kwargs): + observed.append((argv, kwargs)) + return subprocess.CompletedProcess(argv, 0, '"leader"') + + with patch.dict(os.environ, {"SNOWGLOBE_POOL": "globe"}), patch.object(release, "ROOT", self.root), patch.object( + release, "STATE", self.root, + ), patch.object(release, "check_release", return_value=source), patch.object( + release, "current_release", return_value=None, + ), patch.object(release, "host_digest", return_value="different"), patch.object( + release, "host_configuration", return_value=transition.NAMESPACE["previousHost"], + ), patch.object(release, "jobs", return_value=set()), patch.object( + release.os, "uname", return_value=SimpleNamespace(nodename="snowglobe"), + ), patch.object(release.subprocess, "run", side_effect=run), patch.object(transition, "activate") as activate: + release.activate(revision, initial=True) + activate.assert_called_once_with(source) + scripts = [(argv, kwargs) for argv, kwargs in observed if argv[0] == "python3"] + self.assertEqual([argv[-1] for argv, _ in scripts], ["preflight", "pool", "deploy", "check"]) + self.assertTrue(all(argv[1].endswith("/" + transition.PREVIOUS + ".py") for argv, _ in scripts)) + self.assertTrue(all(kwargs["env"][transition.PREVIOUS.upper() + "_POOL"] == "globe" for _, kwargs in scripts)) + self.assertIn(["systemctl", "restart", transition.PREVIOUS + "-router.service"], [argv for argv, _ in observed]) + + def test_policy_attachment_changes_after_new_policy_exists_without_changing_token_files(self): + state = self.root / "state" + state.mkdir() + for name in ("nomad", "router", "dashboard"): + (state / (name + ".token")).write_text("disposable-" + name) + responses = [io.BytesIO(json.dumps({"Name": transition.PREVIOUS + "-" + consumer, + "AccessorID": consumer, "Policies": [transition.PREVIOUS + "-" + consumer, "other"]}).encode()) + for consumer in ("router", "dashboard")] + with patch.object(transition.urllib.request, "urlopen", side_effect=responses), patch.object( + transition.subprocess, "run", return_value=subprocess.CompletedProcess([], 0), + ) as run: + transition.migrate_policies(self.root, state) + commands = [call.args[0] for call in run.call_args_list] + for index, consumer in enumerate(("router", "dashboard")): + self.assertEqual(commands[index * 2][:5], ["nomad", "acl", "policy", "apply", "snowglobe-" + consumer]) + self.assertIn("-policy=snowglobe-" + consumer, commands[index * 2 + 1]) + self.assertIn("-policy=other", commands[index * 2 + 1]) + self.assertFalse(any("disposable" in argument for argument in commands[index * 2 + 1])) + self.assertEqual((state / (consumer + ".token")).read_text(), "disposable-" + consumer) + + def test_existing_vm_owner_profile_and_seed_still_resolve(self): + old = "https://paperclover.net/" + transition.PREVIOUS + root = ET.fromstring(f'owner-idwindows-q35-secure{transition.PREVIOUS}-cloud-init') + transition.normalize_vm(root) + namespace = "{https://paperclover.net/snowglobe}" + self.assertEqual(root.findtext(f"metadata/{namespace}vm/{namespace}owner"), "owner-id") + self.assertEqual(root.findtext(f"metadata/{namespace}vm/{namespace}hardwareProfile"), "windows-q35-secure") + self.assertEqual(root.findtext("devices/disk/serial"), "snowglobe-cloud-init") + + def test_account_and_nixos_registries_round_trip_with_fixed_numeric_ownership(self): + old, new = transition.PREVIOUS + "-dashboard", "snowglobe-dashboard" + users = {old: SimpleNamespace(pw_uid=997, pw_gid=994)} + groups = {old: SimpleNamespace(gr_gid=994)} + registry = transition.NIXOS_STATE + (registry / "uid-map").write_text(json.dumps({old: 997, "other": 888})) + (registry / "gid-map").write_text(json.dumps({old: 994})) + for name in ("declarative-users", "declarative-groups"): + (registry / name).write_text("root " + old + "\n") + + def lookup(values, name): + if name not in values: + raise KeyError(name) + return values[name] + + calls = [] + + def command(argv, **kwargs): + calls.append(argv) + if argv[0] == "groupmod": + groups[argv[2]] = groups.pop(argv[3]) + if argv[0] == "usermod": + users[argv[2]] = users.pop(argv[3]) + return subprocess.CompletedProcess(argv, 0) + + for _, directory in self.directories: + directory.mkdir(parents=True) + release_path = self.directories[0][1] / "releases" / ("c" * 16) + (release_path / "tools").mkdir(parents=True) + (release_path / "tools/snowglobe.py").touch() + transition.prepare() + canonical_release = self.directories[0][0] / "releases" / release_path.name + with patch.object(transition, "NIXOS_STATE", registry), patch.object( + transition.pwd, "getpwnam", side_effect=lambda name: lookup(users, name), + ), patch.object(transition.grp, "getgrnam", side_effect=lambda name: lookup(groups, name)), patch.object( + transition.subprocess, "run", side_effect=command, + ), patch.object(transition, "migrate_policies"): + transition.activate(canonical_release) + self.assertEqual(set(users), {new}) + self.assertEqual((users[new].pw_uid, users[new].pw_gid, groups[new].gr_gid), (997, 994, 994)) + self.assertEqual(json.loads((registry / "uid-map").read_text()), {new: 997, "other": 888}) + self.assertEqual((registry / "declarative-users").read_text().split(), ["root", new]) + previous_release = self.directories[0][0] / "releases" / ("d" * 16) + (previous_release / "tools").mkdir(parents=True) + (previous_release / "tools" / (transition.PREVIOUS + ".py")).touch() + transition.activate(previous_release) + self.assertEqual(set(users), {old}) + self.assertEqual((users[old].pw_uid, users[old].pw_gid, groups[old].gr_gid), (997, 994, 994)) + self.assertEqual(json.loads((registry / "uid-map").read_text()), {old: 997, "other": 888}) + self.assertEqual((registry / "declarative-users").read_text().split(), ["root", old]) + self.assertEqual(calls[0], ["systemctl", "stop", old]) + self.assertEqual(calls[8], ["systemctl", "stop", new]) + + def test_policy_failure_after_directory_move_is_retried_without_moving_state_again(self): + for _, old in self.directories: + old.mkdir(parents=True) + with patch.object(transition.subprocess, "run", return_value=subprocess.CompletedProcess([], 0)) as run, patch.object( + transition, "migrate_policies", side_effect=[RuntimeError("interrupted ACL update"), None], + ) as policies: + with self.assertRaisesRegex(RuntimeError, "ACL update"): + transition.promote(self.root) + before = [new.stat().st_ino for new, _ in self.directories] + transition.promote(self.root) + self.assertEqual(policies.call_count, 2) + self.assertEqual(run.call_count, 6) + self.assertEqual(before, [new.stat().st_ino for new, _ in self.directories]) + + def test_only_legacy_transition_stops_the_retired_window_proxy(self): + with patch.object(transition.subprocess, "run", return_value=subprocess.CompletedProcess([], 0)) as run: + transition.stop_units(transition.PREVIOUS) + previous = [call.args[0] for call in run.call_args_list] + run.reset_mock() + transition.stop_units("snowglobe") + canonical = [call.args[0] for call in run.call_args_list] + self.assertIn(["systemctl", "stop", transition.PREVIOUS + "-vm-window-host-proxy"], previous) + self.assertFalse(any("vm-window-host-proxy" in command[-1] for command in canonical)) + self.assertTrue(all(command[:2] == ["systemctl", "stop"] for command in previous + canonical)) + + def test_missing_reverse_alias_after_directory_move_is_recovered_from_old_releases(self): + new, old = self.directories[0] + source = new / "releases" / ("e" * 16) / "tools" + source.mkdir(parents=True) + (source / (transition.PREVIOUS + ".py")).touch() + transition.prepare() + self.assertTrue(old.is_symlink()) + self.assertEqual(old.resolve(), new) + + @unittest.skipUnless(sys.platform == "linux" and os.geteuid() == 0 and shutil.which("usermod") and shutil.which("groupmod"), + "requires Linux root and shadow tools; all account files stay inside a temporary prefix") + def test_real_shadow_tools_and_interrupted_registry_update_resume_with_only_fixture_accounts(self): + old, new = transition.PREVIOUS + "-dashboard", "snowglobe-dashboard" + prefix = self.root / "accounts" + etc = prefix / "etc" + etc.mkdir(parents=True) + (etc / "passwd").write_text(f"{old}:x:997:994:fixture:/var/empty:/bin/false\n") + (etc / "group").write_text(f"{old}:x:994:\n") + (etc / "shadow").write_text(f"{old}:!:1:0:99999:7:::\n") + (etc / "gshadow").write_text(f"{old}:!::\n") + for name in ("shadow", "gshadow"): + (etc / name).chmod(0o600) + (etc / "login.defs").write_text("") + registry = transition.NIXOS_STATE + for name, value in (("uid-map", 997), ("gid-map", 994)): + (registry / name).write_text(json.dumps({old: value})) + for name in ("declarative-users", "declarative-groups"): + (registry / name).write_text(old + "\n") + command = subprocess.run + replace = Path.replace + fail_login = True + fail_registry = True + + def run(argv, **kwargs): + nonlocal fail_login + self.assertIn(argv[0], {"usermod", "groupmod"}) + if argv[0] == "usermod" and fail_login: + fail_login = False + raise subprocess.CalledProcessError(1, argv) + return command([argv[0], "--prefix", str(prefix), *argv[1:]], **kwargs) + + def atomic_replace(path, target): + nonlocal fail_registry + if Path(target) == registry / "gid-map" and fail_registry: + fail_registry = False + raise OSError("interrupted registry write") + return replace(path, target) + + def user(name): + fields = (etc / "passwd").read_text().splitlines()[0].split(":") + if fields[0] != name: + raise KeyError(name) + return SimpleNamespace(pw_uid=int(fields[2]), pw_gid=int(fields[3])) + + def group(name): + fields = (etc / "group").read_text().splitlines()[0].split(":") + if fields[0] != name: + raise KeyError(name) + return SimpleNamespace(gr_gid=int(fields[2])) + + with patch.object(transition.pwd, "getpwnam", side_effect=user), patch.object( + transition.grp, "getgrnam", side_effect=group, + ), patch.object(transition.subprocess, "run", side_effect=run), patch.object(Path, "replace", atomic_replace): + with self.assertRaises(subprocess.CalledProcessError): + transition.rename_identity(transition.identity_change("snowglobe")) + self.assertEqual(group(new).gr_gid, 994) + self.assertEqual(user(old).pw_uid, 997) + with self.assertRaisesRegex(OSError, "registry write"): + transition.rename_identity(transition.identity_change("snowglobe")) + self.assertEqual(user(new).pw_uid, 997) + transition.rename_identity(transition.identity_change("snowglobe")) + self.assertIsNone(transition.identity_change("snowglobe")) + self.assertEqual(json.loads((registry / "gid-map").read_text()), {new: 994}) + transition.rename_identity(transition.identity_change(transition.PREVIOUS)) + self.assertEqual((user(old).pw_uid, user(old).pw_gid, group(old).gr_gid), (997, 994, 994)) + self.assertIsNone(transition.identity_change(transition.PREVIOUS)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/namespace-transition.py b/tools/namespace-transition.py new file mode 100644 index 0000000000000000000000000000000000000000..8f5ca4de89f108e68d57cb48485c86618da7e5d5 --- /dev/null +++ b/tools/namespace-transition.py @@ -0,0 +1,245 @@ +#!/usr/bin/env python3 +"""Bridge the installed namespace and immutable releases during the rename.""" +import argparse +import grp +import json +import os +from pathlib import Path +import pwd +import subprocess +import urllib.request + + +NAMESPACE = globals().get("NAMESPACE") or json.loads( + (Path(__file__).with_suffix(".json") if Path(__file__).with_suffix(".json").exists() + else Path(__file__).resolve().parent.parent / "dashboard/src/namespace-transition.json").read_text() +) +PREVIOUS = NAMESPACE["previous"] +NIXOS_STATE = Path("/var/lib/nixos") +DIRECTORIES = tuple((Path(parent) / "snowglobe", Path(parent) / PREVIOUS) + for parent in ("/opt", "/var/lib", "/var/lib/caddy", "/var/cache")) + + +def prepare(): + for new, old in DIRECTORIES: + if new.exists() and old.exists() and new.resolve() != old.resolve(): + raise ValueError(f"Two independent control stores exist: {new}, {old}") + new_root, _ = DIRECTORIES[0] + previous_releases = any((new_root / "releases").glob("*/tools/" + PREVIOUS + ".py")) + for new, old in DIRECTORIES: + if old.exists() and not new.exists() and not new.is_symlink(): + new.symlink_to(old, target_is_directory=True) + elif previous_releases and new.exists() and not old.exists() and not old.is_symlink(): + old.symlink_to(new, target_is_directory=True) + + +def inherit_environment(): + for name, value in list(os.environ.items()): + if name.startswith(PREVIOUS.upper() + "_"): + os.environ.setdefault("SNOWGLOBE_" + name.removeprefix(PREVIOUS.upper() + "_"), value) + + +def environment(path, values=None): + values = dict(os.environ if values is None else values) + if layout(path)["prefix"] == PREVIOUS: + for name, value in list(values.items()): + if name.startswith("SNOWGLOBE_"): + values[PREVIOUS.upper() + "_" + name.removeprefix("SNOWGLOBE_")] = value + return values + + +def layout(path): + prefix = "snowglobe" if (path / ".snowglobe-release.json").exists() or (path / "tools/snowglobe.py").exists() else PREVIOUS + return {"prefix": prefix, "script": path / f"tools/{prefix}.py", "manifest": path / f".{prefix}-release.json"} + + +def host_configuration(hostname, available): + hostname = hostname.split(".", 1)[0] + candidates = ("snowglobe", NAMESPACE["previousHost"]) if hostname in {"snowglobe", NAMESPACE["previousHost"]} else ( + "vm" if hostname in {"snowglobe-rehearsal", "clover-demo"} else hostname, + ) + for name in candidates: + if name in available: + return name + raise ValueError(f"release has no NixOS configuration for {hostname}") + + +def host_environment(): + for prefix in ("snowglobe", PREVIOUS): + result = subprocess.run(["systemctl", "show", prefix + "-host", "--property=Environment", "--value"], + capture_output=True, text=True) + if result.stdout.strip(): + return result.stdout + raise ValueError("the host service environment is unavailable") + + +def stop_preview(identity): + for prefix in ("snowglobe", PREVIOUS): + subprocess.run(["systemctl", "stop", prefix + "-preview-" + identity], capture_output=True) + + +def dashboard_container(): + for name in ("snowglobe-dashboard", PREVIOUS + "-dashboard"): + if subprocess.run(["podman", "container", "exists", name], capture_output=True).returncode == 0: + return name + raise ValueError("the dashboard container is unavailable") + + +def normalize_vm(root): + for element in root.iter(): + if element.tag.startswith("{https://paperclover.net/" + PREVIOUS + "}"): + element.tag = element.tag.replace("{https://paperclover.net/" + PREVIOUS + "}", "{https://paperclover.net/snowglobe}", 1) + if element.tag == "serial" and element.text == PREVIOUS + "-cloud-init": + element.text = "snowglobe-cloud-init" + return root + + +def migrate_policies(source, state, prefix="snowglobe"): + management = {**os.environ, "NOMAD_TOKEN": (state / "nomad.token").read_text().strip()} + for consumer in ("router", "dashboard"): + request = urllib.request.Request("http://127.0.0.1:4646/v1/acl/token/self", headers={ + "X-Nomad-Token": (state / (consumer + ".token")).read_text().strip(), + }) + with urllib.request.urlopen(request, timeout=10) as response: + token = json.load(response) + old, new = (PREVIOUS if prefix == "snowglobe" else "snowglobe") + "-" + consumer, prefix + "-" + consumer + subprocess.run(["nomad", "acl", "policy", "apply", new, str(source / "config/policies" / (consumer + ".hcl"))], + env=management, check=True, capture_output=True) + policies = [new if value == old else value for value in token.get("Policies", [])] + if old in token.get("Policies", []) or token.get("Name") == old: + subprocess.run(["nomad", "acl", "token", "update", "-name=" + new, + *("-policy=" + value for value in policies), token["AccessorID"]], + env=management, check=True, capture_output=True) + + +def stop_units(prefix): + units = ("dashboard", "router", "log-shipper", "vm-screen", "host") + if prefix == PREVIOUS: + units += ("vm-window-host-proxy",) + for unit in units: + result = subprocess.run(["systemctl", "stop", prefix + "-" + unit], capture_output=True, text=True) + if result.returncode and "not loaded" not in result.stderr: + result.check_returncode() + + +def identity_change(prefix): + old = (PREVIOUS if prefix == "snowglobe" else "snowglobe") + "-dashboard" + new = prefix + "-dashboard" + users, groups = {}, {} + for name in (old, new): + try: + users[name] = pwd.getpwnam(name) + except KeyError: + pass + try: + groups[name] = grp.getgrnam(name) + except KeyError: + pass + if len(users) > 1 or len(groups) > 1: + raise ValueError("Two dashboard identities exist; refusing to merge identities") + if any((user.pw_uid, user.pw_gid) != (997, 994) for user in users.values()) or any( + group.gr_gid != 994 for group in groups.values() + ): + raise ValueError("The dashboard account must retain UID 997 and GID 994") + registries = False + for filename, expected in (("uid-map", 997), ("gid-map", 994)): + path = NIXOS_STATE / filename + if path.exists(): + values = json.loads(path.read_text()) + if any(name in values and values[name] != expected for name in (old, new)): + raise ValueError("The NixOS identity registry disagrees with the dashboard account") + registries |= old in values + for filename in ("declarative-users", "declarative-groups"): + path = NIXOS_STATE / filename + registries |= path.exists() and old in path.read_text().split() + return (old, new) if old in users or old in groups or registries else None + + +def rename_identity(change): + if change is None: + return + old, new = change + try: + grp.getgrnam(old) + except KeyError: + pass + else: + subprocess.run(["groupmod", "--new-name", new, old], check=True, capture_output=True) + try: + pwd.getpwnam(old) + except KeyError: + pass + else: + subprocess.run(["usermod", "--login", new, old], check=True, capture_output=True) + for filename in ("uid-map", "gid-map", "declarative-users", "declarative-groups"): + path = NIXOS_STATE / filename + if not path.exists(): + continue + if filename.endswith("-map"): + values = json.loads(path.read_text()) + if old not in values: + continue + values[new] = values.pop(old) + content = json.dumps(values) + "\n" + else: + names = path.read_text().split() + if old not in names: + continue + content = " ".join(new if name == old else name for name in names) + "\n" + pending = path.with_suffix(".namespace-pending") + pending.write_text(content) + pending.chmod(path.stat().st_mode & 0o777) + pending.replace(path) + + +def promote(source): + prepare() + change = identity_change("snowglobe") + moving = any(old.exists() and not old.is_symlink() for _, old in DIRECTORIES) + if moving or change is not None: + stop_units(PREVIOUS) + rename_identity(change) + for new, old in DIRECTORIES: + if old.exists() and not old.is_symlink(): + if new.is_symlink(): + new.unlink() + old.rename(new) + old.symlink_to(new, target_is_directory=True) + new_root, old_root = DIRECTORIES[0] + for name in ("current", "main"): + link = new_root / name + if link.is_symlink(): + target = Path(os.readlink(link)) + if target.is_relative_to(old_root): + pending = link.with_suffix(".namespace-pending") + pending.unlink(missing_ok=True) + pending.symlink_to(new_root / target.relative_to(old_root)) + pending.replace(link) + if old_root.exists(): + migrate_policies(source, DIRECTORIES[1][0]) + + +def activate(path): + if layout(path)["prefix"] == "snowglobe": + promote(path) + else: + change = identity_change(PREVIOUS) + stop_units("snowglobe") + rename_identity(change) + migrate_policies(path, DIRECTORIES[1][0], PREVIOUS) + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("mode", choices=("prepare", "promote")) + args = parser.parse_args() + if os.geteuid() != 0: + raise PermissionError("the namespace transition requires root") + if args.mode == "prepare": + prepare() + else: + promote(Path(__file__).resolve().parent.parent) + + +if __name__ == "__main__": + main() diff --git a/tools/oidc-provider.py b/tools/oidc-provider.py index 26e69b32d3d3ee791d156612f7d7cd2b0bf295a6..3d36374c4e925d841ba9a08db50ed519454573ab 100644 --- a/tools/oidc-provider.py +++ b/tools/oidc-provider.py @@ -1,6 +1,7 @@ #!/usr/bin/env python3 """Provision a first-party OIDC client through the dashboard's host-only CLI.""" import json +from importlib import import_module import subprocess import sys @@ -8,7 +9,7 @@ data = json.load(sys.stdin) if data["request"]["kind"] != "client": raise ValueError("unsupported Snowglobe input") result = subprocess.run( - ["podman", "exec", "-i", "studio-dashboard", "/bin/home-dashboard", "--oidc-client"], + ["podman", "exec", "-i", import_module("namespace-transition").dashboard_container(), "/bin/home-dashboard", "--oidc-client"], input=json.dumps(data), text=True, check=True, capture_output=True, ) sys.stdout.write(result.stdout) diff --git a/tools/openclaw-verify.py b/tools/openclaw-verify.py index 05f31ec90119f57d4efa79d1c90772e7e08eedd2..8db7cb5434c18d96ed80b53f307bfc7c5e0a47ff 100644 --- a/tools/openclaw-verify.py +++ b/tools/openclaw-verify.py @@ -5,7 +5,7 @@ import subprocess parser = argparse.ArgumentParser() -parser.add_argument("--host", default="root@zenith") +parser.add_argument("--host", default="root@10.0.0.1") task = parser.add_mutually_exclusive_group() task.add_argument("--prepare-task", action="store_true") task.add_argument("--check-task", action="store_true") diff --git a/tools/pds-migration.md b/tools/pds-migration.md deleted file mode 100644 index 04ac11216293b8dc7bae6497009f7a732a28b0b7..0000000000000000000000000000000000000000 --- a/tools/pds-migration.md +++ /dev/null @@ -1,7 +0,0 @@ -# PDS data migration - -The 2026-09-26 preview restore copied Zenith's three SQLite databases with SQLite's backup API and copied its blocks and actor keys without writing to Zenith. The restored stage passed SQLite integrity checks and HTTPS health. Source and stage had the same table and row counts: `account` 18/47, `sequencer` 3/76, and `did_cache` 3/5. The preview uses a separate hostname and disables PLC and crawlers, so it cannot validate public federation. - -For production, configure the new host's site domain as `paperclover.net` and verify PDS renders `at.paperclover.net`. Stop the old and new PDS instances, then run `STUDIO_DEPLOY_HOST= STUDIO_DEPLOY_PORT=22 sh tools/import-pds.sh pds`. The importer requires both instances stopped, imports the existing JWT/admin/PLC and mail secrets, snapshots the destination, copies and verifies the data, and leaves the new instance stopped. Its snapshot name is printed for recovery. Start the new instance only after checking the account DID and handle against Zenith; switch public routing after its health and identity endpoints agree. The importer rechecks that Zenith did not restart during the copy. - -For a same-machine OS replacement, set `STUDIO_LEGACY_HANDOFF` to the [offline handoff](legacy-handoff.md) directory. The importer then reads the retained PDS SQLite files and `.env` from the new host's mounted old apps dataset, without old Docker. Its SQLite backup operation was checked read-only against all three old databases. diff --git a/tools/personal-cutover.md b/tools/personal-cutover.md deleted file mode 100644 index 65ec21856cd79ab57f195bfb4c04981fd7460bfd..0000000000000000000000000000000000000000 --- a/tools/personal-cutover.md +++ /dev/null @@ -1,45 +0,0 @@ -# Personal service cutover - -The installed host retains the original encrypted app tree at `/mnt/storage1/apps`. Clover and its Media child are already mounted at `/srv/clover` and `/srv/clover/Media`. Copy each app's state into its managed production dataset, verify it, and only then start that app. Old app trees stay untouched. Personal Forgejo repository migration and PostgreSQL restoration are separate from these file imports. - -```sh -export STUDIO_DEPLOY_HOST=root@10.0.0.1 -export STUDIO_DEPLOY_PORT=22 -export STUDIO_LEGACY_HANDOFF=/mnt/storage1/apps/studio-handoff/20261004T230507Z-e75419 -``` - -Provision managed service datasets and identities before import. Register a stopped production Nomad job when required: render HCL, parse with `nomad job run -output`, set the returned `Job.Stop=true`, then POST `/v1/jobs` and confirm zero allocations. Do not deploy an empty app to create its dataset. A missing job is accepted by the Shale, qBittorrent, ARR, YouTube, and personal-files importers. Jellyfin, Navidrome, and Jackett importers require `Stop=true` on an existing job. - -| App | Retained source → managed destination | Import and pre-start proof | -| --- | --- | --- | -| Shale | `apps/shale/{data,repositories_owned,repositories_mirrors}` → `/srv/prod/shale/` | `bash tools/import-shale.sh shale`; three directory checksums and SQLite integrity. Forgejo-only repositories still need import into Shale. | -| Jellyfin | `apps/jellyfin/` → `/srv/prod/jellyfin/config/` | `sh tools/import-jellyfin.sh jellyfin`; excludes caches/logs/transcodes; verifies both SQLite databases and copy checksums. | -| Navidrome | `apps/navidrone/` → `/srv/prod/navidrome/data/` | `sh tools/import-navidrome.sh navidrome`; SQLite integrity, checksum, and retained `snow` administrator. The old directory is spelled `navidrone`. | -| qBittorrent | `apps/qbittorrent/qBittorrent/` → `/srv/prod/qbittorrent/config/qBittorrent/` | `bash tools/import-qbittorrent.sh`; verifies resume-file count and checksum, prepares config, **then starts the service**. Load PIA secrets first. | -| PDS | `apps/pds/` → `/srv/prod/pds/pds/` | `ssh root@10.0.0.1 'python3 - pds' < tools/import-personal-files.py`; preserve original JWT/admin/PLC/mail secrets first. Checks every SQLite database, including actor stores, and copies actor keys/blocks. Refuses WAL files requiring a consistent SQLite backup. | -| Sonarr / Radarr | `apps//` → `/srv/prod//config/` | `bash tools/import-arr.sh `; consistent SQLite backup, integrity and hash; excludes logs/PIDs. Configure current internal endpoints after startup. | -| Jackett | `apps/jackett/` → `/srv/prod/jackett/config/Jackett/` | `sh tools/import-jackett.sh jackett`; retains API key and indexer JSON, verifies checksums. | -| YouTube state | `apps/yt-feed/` → `/srv/prod/ytdl/data/` | `bash tools/import-yt-feed.sh ytdl`; validates JSON and checksums; stops and restarts dashboard's worker around import. Import mail secrets before activating worker. | -| Clover DB | `/srv/clover/.zfs/snapshot//Documents/Config/paper clover/` → `/srv/prod/clover-source-of-truth/data/` | `STUDIO_MIGRATION_SNAPSHOT= bash tools/import-source-data.sh`; direct host-local tar stream, tree digest and SQLite integrity. Import original API key. | -| Tailscale / DDNS / Copyparty / YouTube Archiver / PDS | Sources and mount destinations are defined in [import-personal-files.py](import-personal-files.py). | Run the host-side importer below once per app. No app starts; it checks stopped allocations, snapshots destination, copies, checksum-verifies, and assigns its UID. | - -```sh -ssh root@10.0.0.1 'python3 - tailscale' < tools/import-personal-files.py -ssh root@10.0.0.1 'python3 - ddns-updater' < tools/import-personal-files.py -ssh root@10.0.0.1 'python3 - copyparty' < tools/import-personal-files.py -ssh root@10.0.0.1 'python3 - ytdl' < tools/import-personal-files.py -``` - -Tailscale must retain `tailscaled.state` to preserve its node identity; the copy remains root-owned and mode `0600`. The PDS importer copies every actor store, not only its three root SQLite files; `import-pds.sh` excludes nested SQLite files and is unsuitable for this offline cutover. DDNS retains update history; its provider configuration comes from imported Cloudflare secrets and the service's generated `CONFIG`. Copyparty's active state is the **nested** old `copyparty/copyparty/` directory, including `shares.db`, sessions, IdP database, and salts; it stays nested under the new `/cfg/copyparty`. The [container sets `XDG_CONFIG_HOME=/cfg`](https://github.com/9001/copyparty/blob/hovudstraum/scripts/docker/Dockerfile.ac), and Copyparty appends its app directory. The outer legacy policy config is not copied: `prepare.py` generates the current policy. Its share database contained 17 shares and three selected-file records at the October 5 UTC inspection. Per-volume histories already on Clover/Media stay on those datasets. - -YouTube Archiver's 12 retained download-archive files stay beside videos in `Media/Videos/Independent`; cache and stale locks in its old app directory are disposable. Its retained subscriptions specify `/media/jellyfin/Independent`, so the service binds that container path to the reorganized folder. The upscaler reads `/media/Videos/Independent` directly. Dashboard's worker reads the actual `Indie Shows`, `Videos/Independent`, and `Intake - Music` folders; it needs no media symlinks. - -Samba serves existing Clover/Media directly and needs only its account secret. Intake likewise uses `/srv/clover/Documents/Intake` directly and needs the retained API key. VictoriaMetrics, VictoriaLogs, and VictoriaTraces had no legacy directories at their personal app paths; start new stores. OpenSpeedTest, FlareSolverr, and Forward Auth have no retained application file state. Keycloak and Dawarich require their personal PostgreSQL restores before activation; Dawarich's file-volume migration belongs with its database importer. - -Activate telemetry and restored PostgreSQL first, then restored Keycloak and Forward Auth. Start restored Shale and complete Git repository migration. After personal files/secrets are verified, enable Copyparty, Samba, PDS, Navidrome, Jellyfin, Intake, and Clover DB. Bring up Jackett and qBittorrent before Sonarr/Radarr. Start restored YouTube state, dashboard worker, and Archiver after checking final writable media paths. DDNS starts after wildcard routing is ready; preserved Tailscale state can start once its own copy passes. - -Do not reuse preview imports for production: those branches can start jobs, use old `storage1` snapshot names, or intentionally disable writers. qBittorrent's production importer starts its job at the end; YouTube's importer restarts dashboard and may resume retained queued work immediately. All other production file importers listed above leave the imported app stopped. - -October 5 UTC execution evidence is stored root-only under `/var/lib/studio/migrations/personal-cutover-20261004`; the original app snapshot is `globe/apps@personal-cutover-20261004`. The manual no-start qBittorrent copy retained 550 saved torrents, YouTube retained six valid JSON state files, Copyparty retained 17 shares, PDS retained all four SQLite databases, and pgAdmin passed SQLite integrity. Dawarich retained its storage file, empty watched-import directory, and 95,396-byte Redis dump. YouTube state needs explicit access ACL masks (`m::rwX`) and default masks (`d:m::rwx`) so dashboard and upscaler can both write. - -Before ARR activation, the copied databases received a separate `before-arr-path-cutover-*` snapshot. Sonarr translated two root folders and 89 series paths; Radarr translated one root folder and 56 movie paths from `/data/media/jellyfin/` to `/data/media/`. All other application table values matched their pre-change hashes. Seven download-path mappings per app are derived from qBittorrent and ARR volume definitions, including `torrent` → `Seedbox` and the saved Jellyfin aliases; [configure-arr.py](configure-arr.py) updates their host alongside the download client. [Servarr maps paths only for the matching download-client host](https://github.com/Servarr/Wiki/blob/master/sonarr/settings.md). All mapped root destinations passed service UID/GID 3000 read/search checks. Three Sonarr series directories were already absent in the pre-layout Media snapshot; the other 86 and all 56 Radarr movie directories resolve. Private proofs are `-path-cutover.json`, `-mapped-directory-proof.json`, and `sonarr-missing-directory-origin.json` in the evidence directory above. diff --git a/tools/rehearsal.md b/tools/rehearsal.md new file mode 100644 index 0000000000000000000000000000000000000000..5d59bcbe472e5980d50c81eb0efea5a2c32756b9 --- /dev/null +++ b/tools/rehearsal.md @@ -0,0 +1,26 @@ +# Isolated host rehearsal + +Normal service and dashboard previews run on Snow Globe. Use the optional NixOS +VM only when a change needs an isolated host, OS, storage, or guest boundary. +It has no production GPU; do not download model weights or enable inference. + +The VM configuration is `nixos/vm.nix` (flake `vm`), with hostname `snowglobe-rehearsal`, +domain `snowglobe.test`, and a demo pool. Its SSH endpoint is +`root@127.0.0.1`, port `2222`. Both deployment overrides are required: + +```sh +SNOWGLOBE_DEPLOY_HOST=root@127.0.0.1 SNOWGLOBE_DEPLOY_PORT=2222 python3 tools/deploy.py stage +``` + +`tools/local-vm.sh` launches the VM. `SNOWGLOBE_VM_CACHE_DIR` overrides its cache; +`SNOWGLOBE_VM_HEADLESS=1` selects background operation. Installed mode mounts +`SNOWGLOBE_VM_MEDIA_SOURCE` (default `/Volumes/media`) read-only through 9P at +`/mnt/media-source`, with a bindfs view at `/srv/clover/Media`. +Those media bytes are personal source data, not synthetic fixtures. The VM's +own Clover dataset is separate. Use disposable test data and do not inspect +private source contents without authorization. + +Use the configured DNS and TLS routing for the VM. Do not hand-edit host files, +DNS, or trust to bypass a failed setup. Keep VM console/browser checks inside the +normal browser panel. Rehearsal evidence belongs in its relevant runbook; the +primary agent workflow remains staging on Snow Globe. diff --git a/tools/release-host-test.py b/tools/release-host-test.py new file mode 100644 index 0000000000000000000000000000000000000000..6fb8dce85ddea0ca18cfe7e9558a19c469f16006 --- /dev/null +++ b/tools/release-host-test.py @@ -0,0 +1,90 @@ +#!/usr/bin/env python3 +import json +import os +from pathlib import Path +import subprocess +import tempfile +from types import SimpleNamespace +import unittest +from unittest.mock import patch + +import release + +PREVIOUS_HOST = release.transition.NAMESPACE["previousHost"] + + +class HostConfiguration(unittest.TestCase): + def select(self, hostname, configurations): + with patch.object(release.os, "uname", return_value=SimpleNamespace(nodename=hostname)), patch.object( + release.subprocess, "run", return_value=subprocess.CompletedProcess([], 0, json.dumps(configurations)), + ) as command: + result = release.host_configuration(Path("/release")) + command.assert_called_once_with( + ["nix", "--extra-experimental-features", "nix-command flakes", "eval", "--json", "path:/release#nixosConfigurations", "--apply", "builtins.attrNames"], + check=True, capture_output=True, text=True, + ) + return result + + def test_rename_and_rollback(self): + for hostname in (PREVIOUS_HOST, "snowglobe", "snowglobe.paperclover.net"): + with self.subTest(hostname=hostname): + self.assertEqual(self.select(hostname, ["installer", "snowglobe", "vm"]), "snowglobe") + self.assertEqual(self.select(hostname, ["installer", PREVIOUS_HOST, "vm"]), PREVIOUS_HOST) + + def test_rehearsal_stays_on_vm(self): + self.assertEqual(self.select("snowglobe-rehearsal", ["installer", "snowglobe", "vm"]), "vm") + + def test_unrelated_host_is_not_retargeted(self): + self.assertEqual(self.select("fixture", ["fixture", "vm"]), "fixture") + for hostname in ("fixture", "snowglobe", "snowglobe-rehearsal"): + with self.subTest(hostname=hostname), self.assertRaisesRegex(ValueError, "no NixOS configuration"): + self.select(hostname, ["installer"]) + + +class ActivationEnvironment(unittest.TestCase): + def test_host_switch_refreshes_inherited_name_and_preserves_overrides(self): + for before, after, inherited, expected in ( + (PREVIOUS_HOST, "snowglobe", PREVIOUS_HOST, "snowglobe"), + ("snowglobe", PREVIOUS_HOST, "snowglobe", PREVIOUS_HOST), + (PREVIOUS_HOST, "snowglobe", None, "snowglobe"), + (PREVIOUS_HOST, "snowglobe", "test-override", "test-override"), + ("snowglobe-rehearsal", "snowglobe-rehearsal", "snowglobe-rehearsal", "snowglobe-rehearsal"), + ): + with self.subTest(before=before, inherited=inherited), tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + source = root / "release" + source.mkdir() + (source / "tools").mkdir() + (source / "tools/snowglobe.py").touch() + (source / ".snowglobe-release.json").write_text('{"version":1}') + hostname = before + observed = [] + + def run(argv, **kwargs): + nonlocal hostname + if argv[:2] == ["nixos-rebuild", "switch"]: + hostname = after + if argv[-1] == "deploy": + observed.append(os.environ.get("SNOWGLOBE_NODE_NAME")) + return subprocess.CompletedProcess(argv, 0, '"leader"') + + with patch.dict(os.environ), patch.object(release, "ROOT", root), patch.object( + release, "STATE", root, + ), patch.object(release, "check_release", return_value=source), patch.object( + release, "current_release", return_value=None, + ), patch.object(release, "host_digest", return_value="new"), patch.object( + release, "host_configuration", return_value="fixture", + ), patch.object(release, "jobs", return_value=set()), patch.object( + release.os, "uname", side_effect=lambda: SimpleNamespace(nodename=hostname), + ), patch.object(release.subprocess, "run", side_effect=run): + if inherited is None: + os.environ.pop("SNOWGLOBE_NODE_NAME", None) + else: + os.environ["SNOWGLOBE_NODE_NAME"] = inherited + with patch.object(release.transition, "activate"): + release.activate("0123456789abcdef", initial=True) + self.assertEqual(observed, [expected]) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/release.py b/tools/release.py index 20115985fc33e7f2360ff5c9218a108ee0feca72..48af1a16674bf05a5f45d63492db85d83ff39a41 100644 --- a/tools/release.py +++ b/tools/release.py @@ -4,6 +4,7 @@ from datetime import datetime, timezone import fcntl import hashlib import json +from importlib import import_module import os from pathlib import Path import re @@ -11,34 +12,23 @@ import subprocess import time -ROOT = Path("/opt/studio") +transition = import_module("namespace-transition") + +ROOT = Path("/opt/snowglobe") RELEASES = ROOT / "releases" -STATE = Path("/var/lib/studio") +STATE = Path("/var/lib/snowglobe") HISTORY = STATE / "deployments.json" -SOURCES = ("config", "service", "tools", "nixos", "dashboard", "guest", "flake.nix", "flake.lock", "readme.md") +SOURCES = ("config", "service", "tools", "nixos", "dashboard", "guest", "flake.nix", "flake.lock", "readme.md", "AGENTS.md") RELEASE_ID = re.compile(r"[0-9a-f]{16}\Z") STAGE_ID = re.compile(r"[a-z][a-z0-9-]*\Z") -def excluded_services(root): - path = root / "config/excluded-services.json" - return set(json.loads(path.read_text())) if path.exists() else set() - - def files(root): - excluded = excluded_services(root) for name in SOURCES: source = root / name if not source.exists() and not source.is_symlink(): continue - if name == "service" and source.is_dir(): - paths = [] - for directory, directories, filenames in os.walk(source): - directories[:] = [entry for entry in directories if entry not in excluded] - paths.extend(Path(directory) / entry for entry in [*directories, *filenames] - if not (entry.endswith(".pkl") and entry[:-4] in excluded)) - else: - paths = source.rglob("*") if source.is_dir() else [source] + paths = source.rglob("*") if source.is_dir() else [source] for path in sorted(paths): relative = path.relative_to(root) if relative.parts[:2] in ( @@ -76,7 +66,7 @@ def host_digest(root): digest = hashlib.sha256() paths = sorted(path for path, relative in files(root) if relative.parts[0] in {"nixos", "dashboard", "config", "service"} - or str(relative) in {"flake.nix", "flake.lock", "tools/router.py", "tools/dashboard-host.py", "tools/dashboard-run.py", "tools/release.py", "tools/vms.py", "tools/vm-screen.py", "guest/protocol.json"}) + or str(relative) in {"flake.nix", "flake.lock", "tools/router.py", "tools/dashboard-host.py", "tools/dashboard-run.py", "tools/release.py", "tools/namespace-transition.py", "tools/vms.py", "tools/vm-screen.py", "guest/protocol.json"}) for path in paths: digest.update(str(path.relative_to(root)).encode()) _hash_contents(digest, path) @@ -89,7 +79,7 @@ def check_release(release, legacy=False): path = RELEASES / release if not path.is_dir(): raise ValueError(f"release is missing: {release}") - manifest = path / ".studio-release.json" + manifest = transition.layout(path)["manifest"] if not manifest.exists(): if not legacy: raise ValueError(f"release has no manifest: {release}") @@ -105,7 +95,7 @@ def current_release(link="current"): if not current.is_symlink(): return None target = current.resolve() - if target.parent != RELEASES or not RELEASE_ID.fullmatch(target.name): + if target.parent != RELEASES.resolve() or not RELEASE_ID.fullmatch(target.name): raise ValueError(f"{link} points outside releases: {target}") return target.name @@ -116,7 +106,7 @@ def main_release(version=None): return None if not RELEASE_ID.fullmatch(version): raise ValueError("invalid release ID") - details = json.loads((RELEASES / version / ".studio-release.json").read_text()) + details = json.loads(transition.layout(RELEASES / version)["manifest"].read_text()) revision = details.get("main") if (not isinstance(revision, dict) or not isinstance(revision.get("commit"), str) or not re.fullmatch(r"[0-9a-f]{40}", revision["commit"]) @@ -137,8 +127,8 @@ def save_history(entries): def jobs(path): rendered = subprocess.run( - ["python3", str(path / "tools/studio.py"), "render"], - check=True, capture_output=True, text=True, + ["python3", str(transition.layout(path)["script"]), "render"], + check=True, capture_output=True, text=True, env=transition.environment(path), ).stdout result = set(re.findall(r'^job "([a-z][a-z0-9-]*)" \{$', rendered, re.MULTILINE)) if not result: @@ -146,22 +136,33 @@ def jobs(path): return result +def host_configuration(path): + hostname = os.uname().nodename.split(".", 1)[0] + available = json.loads(subprocess.run( + ["nix", "--extra-experimental-features", "nix-command flakes", "eval", "--json", f"path:{path}#nixosConfigurations", "--apply", "builtins.attrNames"], + check=True, capture_output=True, text=True, + ).stdout) + return transition.host_configuration(hostname, available) + + def activate(release, legacy=False, initial=False): path = check_release(release, legacy) - script = path / "tools/studio.py" + layout = transition.layout(path) + script = layout["script"] + environment = transition.environment(path) + prefix = layout["prefix"] managed = STATE / "managed-jobs.json" current = current_release() previous = set(json.loads(managed.read_text())) if managed.exists() else ( jobs(RELEASES / current) if current and not initial else set() ) digest = host_digest(path) - hostname = os.uname().nodename.split(".", 1)[0] - configuration = "vm" if hostname == "clover-demo" else hostname recorded = ROOT / "host.digest" old_digest = recorded.read_text().strip() if recorded.exists() else None if old_digest != digest: + configuration = host_configuration(path) subprocess.run(["nixos-rebuild", "dry-build", "--flake", f"path:{path}#{configuration}"], check=True) - subprocess.run(["python3", str(script), "preflight"], check=True) + subprocess.run(["python3", str(script), "preflight"], check=True, env=environment) backup = None if current and not initial: backup_script = ROOT / "data.py" @@ -176,7 +177,12 @@ def activate(release, legacy=False, initial=False): if old_digest != digest: if current is None: (ROOT / "current").symlink_to(path) + hostname = os.uname().nodename.split(".", 1)[0] + transition.activate(path) subprocess.run(["nixos-rebuild", "switch", "--flake", f"path:{path}#{configuration}"], check=True) + if os.environ.get("SNOWGLOBE_NODE_NAME", hostname) == hostname: + os.environ["SNOWGLOBE_NODE_NAME"] = os.uname().nodename.split(".", 1)[0] + environment = transition.environment(path) next_link = ROOT / ("next-" + release) next_link.unlink(missing_ok=True) next_link.symlink_to(path) @@ -193,12 +199,12 @@ def activate(release, legacy=False, initial=False): time.sleep(2) else: raise RuntimeError("Nomad API did not become ready") - subprocess.run(["systemctl", "restart", "studio-router.service"], check=True) + subprocess.run(["systemctl", "restart", prefix + "-router.service"], check=True) if (path / "nixos/dashboard.nix").exists(): - subprocess.run(["systemctl", "start", "studio-dashboard.service"], check=True) - subprocess.run(["systemctl", "is-active", "--quiet", "studio-dashboard.service"], check=True) - subprocess.run(["python3", str(script), "pool"], check=True) - subprocess.run(["python3", str(script), "deploy"], check=True) + subprocess.run(["systemctl", "start", prefix + "-dashboard.service"], check=True) + subprocess.run(["systemctl", "is-active", "--quiet", prefix + "-dashboard.service"], check=True) + subprocess.run(["python3", str(script), "pool"], check=True, env=environment) + subprocess.run(["python3", str(script), "deploy"], check=True, env=environment) desired = jobs(path) for name in sorted(previous - desired): subprocess.run( @@ -209,19 +215,29 @@ def activate(release, legacy=False, initial=False): pending.write_text(json.dumps(sorted(desired)) + "\n") pending.replace(managed) if (path / "tools/log-shipper.py").exists(): - subprocess.run(["systemctl", "restart", "studio-log-shipper.service"], check=True) - manifest = path / ".studio-release.json" + subprocess.run(["systemctl", "restart", prefix + "-log-shipper.service"], check=True) + manifest = transition.layout(path)["manifest"] if not legacy and json.loads(manifest.read_text()).get("version", 1) >= 2: - subprocess.run(["python3", str(script), "check"], check=True) + subprocess.run(["python3", str(script), "check"], check=True, env=environment) return backup def main(): parser = argparse.ArgumentParser(description="Deploy main or roll back a home server release") - parser.add_argument("mode", choices=["publish", "deploy", "rollback", "history", "bootstrap", "verify"]) + parser.add_argument("mode", choices=["publish", "deploy", "rollback", "history", "bootstrap", "verify", "stage-destroy"]) parser.add_argument("target", nargs="?") args = parser.parse_args() + transition.prepare() + transition.inherit_environment() STATE.mkdir(parents=True, exist_ok=True) + if args.mode == "stage-destroy": + if not args.target or not STAGE_ID.fullmatch(args.target): + parser.error("stage-destroy requires a stage ID") + saved = json.loads((STATE / "stages" / (args.target + ".json")).read_text()) + path = check_release(saved["release"]) + subprocess.run(["python3", str(transition.layout(path)["script"]), "destroy", args.target], + check=True, env=transition.environment(path)) + return if args.mode == "verify": if not args.target: parser.error("verify requires a release ID") @@ -255,7 +271,7 @@ def main(): entries = history() current = current_release() if current and not entries and args.mode != "bootstrap": - entries.append({"release": current, "source": "previous", "time": int(time.time()), "legacy": not (RELEASES / current / ".studio-release.json").exists()}) + entries.append({"release": current, "source": "previous", "time": int(time.time()), "legacy": not transition.layout(RELEASES / current)["manifest"].exists()}) save_history(entries) if args.mode == "deploy": candidate = main_release() diff --git a/tools/router-cookie-test.py b/tools/router-cookie-test.py index 6a555a2648844889bce45ec2af319f05b0faf7fe..e69d98ac90c90268d819acf30b85051c4e5e2b28 100644 --- a/tools/router-cookie-test.py +++ b/tools/router-cookie-test.py @@ -30,14 +30,14 @@ def main(): parser = argparse.ArgumentParser() parser.add_argument('--caddy', required=True, type=Path) args = parser.parse_args() - with tempfile.TemporaryDirectory(prefix='studio-router-cookies-') as temporary, ThreadingHTTPServer(('127.0.0.1', 0), Cookies) as backend: + with tempfile.TemporaryDirectory(prefix='snowglobe-router-cookies-') as temporary, ThreadingHTTPServer(('127.0.0.1', 0), Cookies) as backend: root = Path(temporary) proof = root / 'proof' proof.write_text('a' * 64) - environment = {**os.environ, 'STUDIO_DOMAIN': 'studio.test', 'STUDIO_DASHBOARD_PORT': '3000', - 'STUDIO_PROXY_TOKEN_FILE': str(proof), 'XDG_DATA_HOME': str(root / 'data'), + environment = {**os.environ, 'SNOWGLOBE_DOMAIN': 'snowglobe.test', 'SNOWGLOBE_DASHBOARD_PORT': '3000', + 'SNOWGLOBE_PROXY_TOKEN_FILE': str(proof), 'XDG_DATA_HOME': str(root / 'data'), 'XDG_CONFIG_HOME': str(root / 'config')} - environment.pop('STUDIO_INTERNAL_PORT', None) + environment.pop('SNOWGLOBE_INTERNAL_PORT', None) def nomad(path, _): if path == '/v1/services': @@ -45,7 +45,7 @@ def main(): if path.startswith('/v1/service/'): service = path.rsplit('/', 1)[1] return [{'ServiceName': service, 'JobID': service, 'AllocID': service, 'Address': '127.0.0.1', - 'Port': backend.server_port, 'Tags': ['caddy-host=' + service + '.studio.test'] + + 'Port': backend.server_port, 'Tags': ['caddy-host=' + service + '.snowglobe.test'] + (['caddy-secure-cookies=true'] if service == 'knife' else [])}] return {'http': {'Status': 'success'}} @@ -61,8 +61,8 @@ def main(): listener.bind(('127.0.0.1', 0)) port = listener.getsockname()[1] runtime = root / 'Caddyfile' - sites = rendered.split('auth.studio.test {', 1)[0] - sites = sites.replace('knife.studio.test {', f'http://127.0.0.1:{port} {{').replace('plain.studio.test {', f'http://localhost:{port} {{').replace(' tls internal\n', '') + sites = rendered.split('auth.snowglobe.test {', 1)[0] + sites = sites.replace('knife.snowglobe.test {', f'http://127.0.0.1:{port} {{').replace('plain.snowglobe.test {', f'http://localhost:{port} {{').replace(' tls internal\n', '') runtime.write_text('{\n admin off\n auto_https off\n}\n' + sites) threading.Thread(target=backend.serve_forever, daemon=True).start() with (root / 'caddy.log').open('w') as log: diff --git a/tools/router.py b/tools/router.py index 6e76921123d708c54d09473768269a0b1136e3b6..26be8c9d00e55ee2666dc48402fcf21b91083870 100644 --- a/tools/router.py +++ b/tools/router.py @@ -10,8 +10,8 @@ import urllib.parse import urllib.request ROUTES = "/var/lib/caddy/routes.caddy" -TOKEN = "/var/lib/studio/router.token" -ROUTE_DIR = "/var/lib/studio/routes" +TOKEN = "/var/lib/snowglobe/router.token" +ROUTE_DIR = "/var/lib/snowglobe/routes" HOST = re.compile(r"[a-z0-9](?:[a-z0-9.-]*[a-z0-9])?\Z") @@ -39,13 +39,13 @@ def proxy(upstreams, indent, uncompressed=False, host=None): def shale_mcp_routes(port): - return [" @shale_mcp_link path_regexp shale_mcp_link ^/-/studio-mcp/([A-Za-z0-9_-]{43})$", + return [" @shale_mcp_link path_regexp shale_mcp_link ^/-/snowglobe-mcp/([A-Za-z0-9_-]{43})$", " handle @shale_mcp_link {", " rewrite * /oauth/shale/link/{re.shale_mcp_link.1}", - " request_header -User-Name", " request_header -User-Groups", " request_header -Studio-Proxy-Token", + " request_header -User-Name", " request_header -User-Groups", " request_header -Snowglobe-Proxy-Token", *proxy(f"127.0.0.1:{port}", " "), " }", - " @shale_mcp_callback {", " path /-/callback", " header Cookie *studio_mcp_shale_link=*", " }", + " @shale_mcp_callback {", " path /-/callback", " header Cookie *snowglobe_mcp_shale_link=*", " }", " handle @shale_mcp_callback {", " rewrite * /oauth/shale/callback", - " request_header -User-Name", " request_header -User-Groups", " request_header -Studio-Proxy-Token", + " request_header -User-Name", " request_header -User-Groups", " request_header -Snowglobe-Proxy-Token", *proxy(f"127.0.0.1:{port}", " "), " }"] @@ -100,13 +100,12 @@ def shale_git_routes(upstreams): def render(token): - with open(os.environ["STUDIO_PROXY_TOKEN_FILE"]) as file: + with open(os.environ["SNOWGLOBE_PROXY_TOKEN_FILE"]) as file: dashboard_proof = file.read().strip() if not re.fullmatch(r"[0-9a-fA-F]{64}", dashboard_proof): raise ValueError("invalid dashboard proxy token") - auth_host = "auth." + os.environ["STUDIO_DOMAIN"] - legacy_host = "keycloak." + os.environ["STUDIO_DOMAIN"] - origins = json.dumps({"origins": ["https://snowglobe." + os.environ["STUDIO_DOMAIN"]]}, separators=(",", ":")) + auth_host = "auth." + os.environ["SNOWGLOBE_DOMAIN"] + origins = json.dumps({"origins": ["https://snowglobe." + os.environ["SNOWGLOBE_DOMAIN"]]}, separators=(",", ":")) webauthn = [" handle /.well-known/webauthn {", ' header Content-Type application/json', f" respond {json.dumps(origins)} 200", " }"] routes = {} @@ -178,7 +177,7 @@ def render(token): lines = [] for (listener, host), route in sorted(routes.items()): if host: - if host in {auth_host, legacy_host}: + if host == auth_host: continue role = route["authRole"] if role and not auth_upstreams: @@ -205,18 +204,18 @@ def render(token): "/srv/staging/" + service) if service == "shale": lines += [" handle /-/logout {", f" reverse_proxy {' '.join(sorted(route['upstreams']))} {{", - f" header_down Location https://snowglobe.{os.environ['STUDIO_DOMAIN']}/auth/shale/sign-out", + f" header_down Location https://snowglobe.{os.environ['SNOWGLOBE_DOMAIN']}/auth/shale/sign-out", " @logged_out status 200 302 303 404", " handle_response @logged_out {", " copy_response 303", " }", " }", " }"] lines += shale_attachment_routes(" ".join(sorted(route["upstreams"]))) lines += [" @userscript path_regexp userscript ^/userscripts/discord-pluralkit-predict(/.*)?$", " redir @userscript /discord-pluralkit-predict{re.userscript.1}?{query} 308"] - port = int(os.environ["STUDIO_DASHBOARD_PORT"]) + port = int(os.environ["SNOWGLOBE_DASHBOARD_PORT"]) if not 1 <= port <= 65535: raise ValueError("invalid dashboard port for Shale linking") lines += shale_mcp_routes(port) lines += [" tracing {", f" span {service}", " span_attributes {", - f" studio.service {service}", " studio.kind edge", " }", " }"] + f" snowglobe.service {service}", " snowglobe.kind edge", " }", " }"] manifest = os.path.join(ROUTE_DIR, service + ".json") assets = {} headers = {} @@ -285,17 +284,17 @@ def render(token): ] elif headers and (auth_upstreams or service == "copyparty"): native = service == "copyparty" - auth = f"127.0.0.1:{int(os.environ['STUDIO_DASHBOARD_PORT'])}" if native else " ".join(sorted(auth_upstreams)) + auth = f"127.0.0.1:{int(os.environ['SNOWGLOBE_DASHBOARD_PORT'])}" if native else " ".join(sorted(auth_upstreams)) if native: lines += [" handle /auth/file/* {", " request_header -User-Name", " request_header -User-Groups", - f" request_header Studio-Proxy-Token {dashboard_proof}", - " request_header X-Studio-Client-IP {remote_host}", *proxy(auth," "), " }"] + f" request_header Snowglobe-Proxy-Token {dashboard_proof}", + " request_header X-Snowglobe-Client-IP {remote_host}", *proxy(auth," "), " }"] else: lines += [" handle /snow.oauth2/* {", *proxy(auth," "), " }"] lines += [" handle {"] lines += [f" request_header -{name}" for name in scrub] lines += [f" reverse_proxy {auth} {{", " lb_try_duration 5s", *passive_health(auth, " "), " method GET", " rewrite " + ("/auth/file/check" if native else "/snow.oauth2/auth"), - *([f" header_up Studio-Proxy-Token {dashboard_proof}"] if native else []), + *([f" header_up Snowglobe-Proxy-Token {dashboard_proof}"] if native else []), " header_up X-Forwarded-Method {method}", " header_up X-Forwarded-Uri {uri}", " @authenticated status 2xx", " handle_response @authenticated {"] lines += [f" request_header {name} {{rp.header.{source}}}" for name, source in headers.items()] @@ -309,13 +308,13 @@ def render(token): shale_page = service == "shale" or re.fullmatch(r"shale-preview-[0-9a-f]{8}", service) page_headers = [] if shale_page: - page_upstream = f"127.0.0.1:{int(os.environ['STUDIO_DASHBOARD_PORT'])}" - page_headers = [f" request_header Studio-Proxy-Token {dashboard_proof}", - f" request_header Studio-Shale-Upstream {json.dumps(sorted(route['upstreams'])[0])}", - " request_header Studio-Shale-Uri {uri}", + page_upstream = f"127.0.0.1:{int(os.environ['SNOWGLOBE_DASHBOARD_PORT'])}" + page_headers = [f" request_header Snowglobe-Proxy-Token {dashboard_proof}", + f" request_header Snowglobe-Shale-Upstream {json.dumps(sorted(route['upstreams'])[0])}", + " request_header Snowglobe-Shale-Uri {uri}", " rewrite * /auth/shale/page"] for index, (request, markup) in enumerate(head_html.items()): - name = f"@studio_head_{index}" + name = f"@snowglobe_head_{index}" lines += [f" {name} path {request}", f" handle {name} {{", " route {", f" replace {json.dumps(markup + '')} {{", " match {", " header Content-Type text/html*", " }", " }", @@ -330,18 +329,16 @@ def render(token): lines.insert(len(lines) - 1, " }") else: lines += [f":{listener} {{", *proxy(" ".join(sorted(route["upstreams"])), " "), "}"] - dashboard_host = "snowglobe." + os.environ["STUDIO_DOMAIN"] - if any(not HOST.fullmatch(host) for host in [auth_host, legacy_host, dashboard_host]): + dashboard_host = "snowglobe." + os.environ["SNOWGLOBE_DOMAIN"] + if any(not HOST.fullmatch(host) for host in [auth_host, dashboard_host]): raise ValueError("invalid login domain") - for host in [auth_host, legacy_host]: - lines += [f"{host} {{", *([" tls internal"] if host.endswith(".test") else []), - *(webauthn if host == auth_host else []), " handle {", - f" redir https://{dashboard_host}/ 308", " }", "}"] + lines += [f"{auth_host} {{", *([" tls internal"] if auth_host.endswith(".test") else []), + *webauthn, " handle {", f" redir https://{dashboard_host}/ 308", " }", "}"] traces = next((route for route in routes.values() if "victoria-traces" in route["services"]), None) if traces: lines += ["http://127.0.0.1:10428 {", " bind 127.0.0.1", *proxy(" ".join(sorted(traces["upstreams"])), " "), "}"] - dashboard_port = int(os.environ["STUDIO_DASHBOARD_PORT"]) + dashboard_port = int(os.environ["SNOWGLOBE_DASHBOARD_PORT"]) if not 1 <= dashboard_port <= 65535: raise ValueError("invalid dashboard route") if (80, dashboard_host) in routes: @@ -349,25 +346,25 @@ def render(token): lines.append(f"{dashboard_host} {{") if dashboard_host.endswith(".test"): lines.append(" tls internal") - lines += [" encode zstd gzip", " tracing {", " span globe", " span_attributes {", " studio.kind edge", " }", " }"] + lines += [" encode zstd gzip", " tracing {", " span globe", " span_attributes {", " snowglobe.kind edge", " }", " }"] lines += [" handle {", " request_header -User-Name", " request_header -User-Groups", - f" request_header Studio-Proxy-Token {dashboard_proof}", - " request_header X-Studio-Client-IP {remote_host}", + f" request_header Snowglobe-Proxy-Token {dashboard_proof}", + " request_header X-Snowglobe-Client-IP {remote_host}", *proxy(f"127.0.0.1:{dashboard_port}", " "), " }", "}"] - internal_port = os.environ.get("STUDIO_INTERNAL_PORT") + internal_port = os.environ.get("SNOWGLOBE_INTERNAL_PORT") if internal_port is not None: - internal_host = "dashboard.internal." + os.environ["STUDIO_DOMAIN"] + internal_host = "dashboard.internal." + os.environ["SNOWGLOBE_DOMAIN"] if not HOST.fullmatch(internal_host) or not internal_port.isdecimal() or not 1 <= int(internal_port) <= 65535: raise ValueError("invalid internal dashboard route") if (80, internal_host) in routes or (int(internal_port), "") in routes: raise ValueError("internal dashboard route conflicts with a Nomad service") lines += [f"{internal_host}:{internal_port} {{", " tls internal", - f" @dashboard header Studio-Proxy-Token {dashboard_proof}", - " handle @dashboard {", " request_header -Studio-Proxy-Token", + f" @dashboard header Snowglobe-Proxy-Token {dashboard_proof}", + " handle @dashboard {", " request_header -Snowglobe-Proxy-Token", " request_header -User-Name", " request_header -User-Groups", " handle_path /nomad/* {", *proxy("127.0.0.1:4646", " ", host="{upstream_hostport}"), " }"] for service, upstreams in sorted(internal_services.items()): - host = f"{service}.{os.environ['STUDIO_DOMAIN']}" if service == "shale" or re.fullmatch(r"shale-preview-[0-9a-f]{8}", service) else "{upstream_hostport}" + host = f"{service}.{os.environ['SNOWGLOBE_DOMAIN']}" if service == "shale" or re.fullmatch(r"shale-preview-[0-9a-f]{8}", service) else "{upstream_hostport}" lines += [f" handle_path /services/{service}/* {{", *proxy(" ".join(sorted(upstreams)), " ", host=host), " }"] lines += [" handle {", " respond 404", " }", " }", " handle {", " respond 403", " }", "}"] diff --git a/tools/samba-macos-test.swift b/tools/samba-macos-test.swift index 043c5221b708086b633c6045659dbfb1b6efe63f..4d92d03993e7e100f854c015cebc22ca8339f1c5 100644 --- a/tools/samba-macos-test.swift +++ b/tools/samba-macos-test.swift @@ -13,7 +13,7 @@ guard CommandLine.arguments.count == 2 else { let fm = FileManager.default let directory = URL(fileURLWithPath: CommandLine.arguments[1], isDirectory: true) - .appendingPathComponent(".studio-smb-test-\(UUID().uuidString)", isDirectory: true) + .appendingPathComponent(".snowglobe-smb-test-\(UUID().uuidString)", isDirectory: true) var failures = 0 func check(_ condition: Bool) throws { @@ -68,7 +68,7 @@ do { try check(Data(contentsOf: package.appendingPathComponent("TXT.rtf")) == first) } test("metadata and large resource fork copy") { - let attributes = ["com.apple.studio-smb-test": first, "com.apple.ResourceFork": Data(repeating: 0x5a, count: 128 * 1024)] + let attributes = ["com.apple.snowglobe-smb-test": first, "com.apple.ResourceFork": Data(repeating: 0x5a, count: 128 * 1024)] for (name, value) in attributes { let result = value.withUnsafeBytes { setxattr(file.path, name, $0.baseAddress, $0.count, 0, 0) } guard result == 0 else { throw NSError(domain: NSPOSIXErrorDomain, code: Int(errno)) } diff --git a/tools/service-dependencies.md b/tools/service-dependencies.md index 5ee7c81656292a2d1c2fcadd3b2b2df276265cae..27ca8f24e1fac3dce8bf752c0a5a75bb8d86bb8f 100644 --- a/tools/service-dependencies.md +++ b/tools/service-dependencies.md @@ -1,37 +1,37 @@ -# Service connection checks - -On 2026-09-26, the VM's pgAdmin database contained one configured server, `Clover Postgres`, at the current Nomad address and port. A query from the pgAdmin container using its rendered environment authenticated to the `postgres` database. The `pg.studio.test` health route passed. Zenith's pgAdmin also had one saved server; Snow Globe creates its connection from the Postgres requirement on boot. - -Redis Insight's `/api/databases/0/info` initially connected to the VM Redis service. After stopping Redis allocation `7ca6e34a`, Nomad placed a new one and changed its port from `26848` to `26913`. Redis Insight's watched `nomadService` template restarted its task once; its saved connection then used `26913` and `/api/databases/0/info` connected again. A temporary Redis key survived the reallocation and was deleted afterward. Both `redis.studio.test` and `pg.studio.test` passed their HTTP checks. [Nomad's template `change_mode` defaults to `restart`](https://developer.hashicorp.com/nomad/docs/job-specification/template), and [Redis Insight applies changed preconfigured connections after a restart](https://redis.io/docs/latest/operate/redisinsight/configuration/). - -After the x86 VM crashed and rebooted on 2026-09-26, Keycloak needed several minutes to start under emulation. Shale's earlier `wait-keycloak-app` prestart task remained marked complete, so Shale restarted 19 times while its OIDC discovery endpoint returned 404. Forward Auth also retried. Both recovered when Keycloak became healthy. [Nomad does not rerun a successful non-sidecar prestart task after a task restart](https://developer.hashicorp.com/nomad/docs/job-specification/lifecycle). - -Evil.inc Forgejo and HedgeDoc remained running but unhealthy after the same reboot. Forgejo had tried PostgreSQL before it was ready and then stopped making progress. Targeted `nomad job restart -all-tasks` calls recovered both; the full 19-route Snow Globe check then passed. Snow Globe now renders a one-minute retry delay for dependent services and [Nomad `check_restart`](https://developer.hashicorp.com/nomad/docs/job-specification/check_restart) for unhealthy services. Disposable one- and two-task Nomad jobs proved that a failing group-level check restarts every running task in its group. Eight generated jobs, including Keycloak, Forgejo, and HedgeDoc, passed `nomad job validate`. - -Release `7ec7b25654702110` staged Shale from its ZFS clone and promoted to the VM after backup `20260926T134201Z-22d930` captured 22 service datasets. The promotion passed all 19 HTTPS routes. Live Nomad inspection confirmed Shale's one-minute retry delay and five-minute health-check grace, plus Forgejo's 20-minute grace from its declared healthy deadline. - -After a clean VM reboot, PostgreSQL passed first, Keycloak became healthy at 14:04 UTC, and Shale recovered on its next one-minute retry. Forgejo remained running but unhealthy after its early database connection failed. At 14:15 UTC, Nomad's group-level `check_restart` restarted both Forgejo and Anubis without intervention; both checks and Forgejo's HTTPS route passed by 14:17 UTC. The full 19-route check passed after reboot. From the Mac, Shale, Keycloak discovery, and the Shale preview returned 200 through `.studio.test` DNS and trusted TLS. The 20-minute grace made Forgejo's recovery too slow, so its service now declares a two-minute health-restart grace, separate from its deployment deadline. - -Keycloak preview `keycloak-preview-dbe9aba6` ran with the production `start` command against a cloned database. Its OIDC discovery advertised the preview hostname. Release `47342c5c49f3eb88` promoted that preview after backup `20260926T141745Z-93733a` captured 22 service datasets. The production Keycloak allocation passed its health check with zero restarts; the promotion completed its dependency allocation and all 19 HTTPS checks. From the Mac, discovery returned HTTP 200 with a trusted certificate and issuer `https://keycloak.studio.test/realms/master`. Live job inspection confirmed `start`, the full production hostname, and Forgejo's two-minute check-restart grace. - -Shale and Jellyfin currently use writable SQLite datasets, so their single-allocation `simple` rollouts cannot overlap old and new writers. The owner accepted a brief Shale restart after staged validation. A controlled restart of the Shale preview on the x86 VM returned 46 failed HTTPS probes from this Mac between 4.87 and 23.14 seconds into a 0.25-second sampling run. Nomad reported the task restart complete after two seconds, before the Caddy route served requests again. The preview subsequently returned HTTP 200 with trusted TLS and one running allocation. This measures a restart under x86 emulation, not a production release rollout or physical-host downtime. - -Open Speed Test has no writable mounts or fixed ports, so its job now uses Nomad's overlapping canary. The first VM promotion automatically promoted a healthy canary, but one of 200 Mac HTTPS probes received HTTP 502 when Caddy dialed the retiring allocation's closed port before a route reload. The router now gives generated reverse proxies a five-second retry window and remembers failed upstream connections for 30 seconds, using [Caddy's documented load balancing options](https://caddyserver.com/docs/caddyfile/directives/reverse_proxy). The next staged release validated the generated Caddyfile and automatically promoted another healthy canary; 510 consecutive Mac HTTPS probes from 03:19:41 to 03:22:11 UTC returned 200, spanning its 03:21:05–03:21:20 rollout. This is a sampled VM result, not a guarantee that every request in future rollouts will succeed. - -For a new clone stage with a PostgreSQL database requirement, Snow Globe now snapshots the service dataset and PostgreSQL dataset in one ZFS operation, which [OpenZFS creates atomically](https://openzfs.github.io/openzfs-docs/man/v2.1/8/zfs-snapshot.8.html). It starts an isolated PostgreSQL container from the latter snapshot and copies the database from that container into a stage database on the live PostgreSQL server. The 2026-09-27 HedgeDoc preview `evil-hedgedoc-preview-31768845` completed its Nomad deployment and returned HTTP 200 from the Mac. Its service snapshot remains for the preview; the temporary PostgreSQL snapshot, clone, and container were removed. This aligns the file and database fork point for these services, while applications remain responsible for their own crash recovery from the snapshots. - -Writable external ZFS mounts now join that same snapshot operation. The recreated YouTube Feed preview `yt-feed-preview-63d74b72` mounted cloned service data and a cloned writable Clover configuration folder, then completed its Nomad deployment. Both source snapshots report `createtxg=35540`, and the preview's VM health check passed. The Mac route redirected to authentication as configured. - -The preview's web container wrote a disposable file through `/yt-config` as its assigned `3118:3000` identity. It appeared in the cloned Clover configuration folder with that owner and was absent from `/srv/clover/Documents/Config/Youtube Downloader`; the file was removed afterward. This verifies writable external mount isolation and group access through the container, beyond snapshot metadata alone. - -An encrypted ZFS rehearsal on the VM created separate source and staging encryption roots, then cloned a source snapshot beneath the staging root. The clone mounted, retained the source encryption root and origin, initially used `0B`, and accepted a write without changing the source file. The temporary datasets and keys were destroyed afterward. Production staging therefore needs the source dataset key loaded while its clone exists; placing the clone below a separately encrypted staging parent does not rekey it. - -Promotion of `c1f546fe904f4c33` stopped during Keycloak configuration. The old task digest included every file in each service folder, so a change to PostgreSQL's `provide.py` restarted PostgreSQL despite no database job configuration change. Keycloak's watched database endpoint then restarted its existing task; its group-level health check restarted that task again before x86 startup completed. A fresh Keycloak allocation completed startup, and all 19 production routes passed. Snow Globe now hashes source files into a task environment variable only for services with a `prepare` script, since those scripts change startup files under managed volumes. Editing PostgreSQL's provider script left its rendered job identical in a temporary release copy; editing qBittorrent's prepare script changed its rendered job. Single-container Nomad services now associate their checks with the task, and setup waits for its route before calling the app API. All 26 generated jobs validated. The new Keycloak preview reached HTTPS health and completed setup with zero restarts; after a manual in-place restart, it had no health-triggered restart through its nine-minute startup. - -Release `c19bee75fad72499` promoted successfully after backup `20260927T042521Z-9fcd75`; all 19 routes passed. Keycloak's production allocation reached health after its slow x86 build and startup with zero restarts. The Mac reached Keycloak and Shale with HTTP 200 and Jellyfin with its expected login redirect. During pgAdmin startup, QEMU paused with a host disk I/O error. Removing the two detached NixOS installer ISOs and moving the unused arm64 demo disk to `/Volumes/Project/Studio VM Archives/clover-studio-vm-arm64.qcow2` freed host space; QEMU resumed and the same promotion process finished. The resulting release history and `/opt/studio/current` both point at `c19bee75fad72499`. - -Read-only `nomad job plan` checks against the active release returned no allocation creates or destroys for PostgreSQL, Keycloak, or Shale. Their rendered HCL files were byte-identical to `nomad job inspect -hcl`. Nomad still described each plan as one in-place update without a field-level diff, so that phrase alone is not evidence of a changed job or a task restart. [The Nomad CLI defines exit code 0 as no allocation creation or destruction](https://developer.hashicorp.com/nomad/commands/job/plan). Resubmitting the identical PostgreSQL job retained its allocation `ff3fc333` and version 14; Keycloak retained allocation `d1c5e27a`. - -Release `9ee7ba520b01b1e5` staged YouTube Triage with its writable external clone, passed the VM NixOS dry build, and promoted after backup `20260927T053411Z-183aca`. The NixOS switch raised the demo pool size to 16 GB; all 19 routes passed. PostgreSQL, Keycloak, Jellyfin, Shale, and Dawarich kept allocation IDs `ff3fc333`, `d1c5e27a`, `c2b6c232`, `25050918`, and `abad9408` through this promotion. The Mac received HTTP 302 at the protected preview URL and HTTP 200 from Shale. - -The manual CLI rolled back from `9ee7ba520b01b1e5` to `c19bee75fad72499` after backup `20260927T054434Z-67d6c0`, switched the NixOS host configuration, and passed all 19 routes. Promoting the same preview again backed up the rollback state as `20260927T055034Z-27d895`, restored NixOS release `9ee7ba520b01b1e5`, and passed the same route sweep. Those five sampled allocations kept their IDs through both switches; the Mac still received Shale HTTP 200 and the protected preview's HTTP 302. Data restore was not invoked by code rollback. +# Service dependencies and rollout + +Declare providers and requirements in Pkl. Provisioning allocates private +inputs, and Nomad templates discover provider endpoints. Watched endpoint +changes restart consumers; applications that persist internal endpoints must +receive repeatable setup updates. [configure-arr.py](configure-arr.py), for +example, derives Jackett/qBittorrent endpoints and remote path mappings from +the service declarations. + +A successful non-sidecar prestart task is not rerun on every application +restart. Dependent jobs therefore also need runtime retry and health recovery. +The schema and renderer own retry delays, health-restart grace, and deployment +deadlines. Verify the actual client connection after provider reallocation; +a healthy application HTTP route alone does not prove database connectivity. + +The default single-allocation rollout stops the old writer before starting the +new one. Writable SQLite stores cannot safely overlap writers. Stateless jobs +can declare canary rollout when their ports and mounts allow it. Generated +Caddy proxies retry connection failures while service discovery catches up; +this does not guarantee uninterrupted long-running requests or WebSockets. + +A new cloned preview snapshots its service dataset, writable external ZFS +mounts, and PostgreSQL dataset in one operation. An isolated PostgreSQL clone +supplies the preview database. The temporary database clone is removed after +copying. Repeating staging retains existing preview data; fresh isolation +starts empty and rejects writable external mounts. Review outbound side +effects independently of filesystem isolation. + +ZFS clones retain their source encryption root, even below the independently +encrypted staging parent. Keep source keys available while clones exist. +Fresh previews inherit staging encryption. Verify actual mount bindings before +allowing disposable writes, especially read-only personal mounts. + +Inspect generated-job changes before expecting a restart. A provider script +change can alter setup without changing a running job. Prepared startup files +are included in task change detection; unrelated files are not. A Nomad plan +reporting an in-place update alone does not establish that tasks will restart. diff --git a/tools/shale-labels-test.py b/tools/shale-labels-test.py index bb302513ebf9b000f798f0204ef4fee7953367bd..51e8696fb6a1e32d061dce6b3a27c7a37ca2a8b6 100644 --- a/tools/shale-labels-test.py +++ b/tools/shale-labels-test.py @@ -15,20 +15,20 @@ class ShaleTests(unittest.TestCase): def setUp(self): self.temp = tempfile.TemporaryDirectory() self.root = Path(self.temp.name) - self.env = patch.dict(os.environ, {'STUDIO_SHALE_DATABASE': str(self.root / 'shale.sqlite'), - 'STUDIO_DASHBOARD_DATA': str(self.root), 'STUDIO_DOMAIN': 'studio.test', 'STUDIO_SHALE_POLICY': str(self.root / 'policy.json')}) + self.env = patch.dict(os.environ, {'SNOWGLOBE_SHALE_DATABASE': str(self.root / 'shale.sqlite'), + 'SNOWGLOBE_DASHBOARD_DATA': str(self.root), 'SNOWGLOBE_DOMAIN': 'snowglobe.test', 'SNOWGLOBE_SHALE_POLICY': str(self.root / 'policy.json')}) self.env.start() (self.root / 'policy.json').write_text(json.dumps({'actorOwners': {'ai': 'owner'}, 'statusDelegates': {'ai': ['chat']}})) with sqlite3.connect(self.root / 'accounts.sqlite') as db: db.execute("CREATE TABLE users(id TEXT PRIMARY KEY,profile TEXT,username TEXT GENERATED ALWAYS AS (json_extract(profile,'$.username')) STORED UNIQUE)") for name in ('owner', 'other', 'ai'): db.execute('INSERT INTO users(id,profile) VALUES(?,?)', (name, json.dumps({'username': name, 'enabled': True}))) - self.grant = {'id': 'fixture', 'user': 'owner', 'resource': 'https://snowglobe.studio.test/mcp/shale', 'scopes': ['shale:write'], 'resources': ['chat']} + self.grant = {'id': 'fixture', 'user': 'owner', 'resource': 'https://snowglobe.snowglobe.test/mcp/shale', 'scopes': ['shale:write'], 'resources': ['chat']} with sqlite3.connect(self.root / 'connections.sqlite') as db: db.execute('CREATE TABLE records(key TEXT PRIMARY KEY,value TEXT,last_updated INTEGER)') db.execute('INSERT INTO records VALUES(?,?,0)', ('grant:fixture', json.dumps(self.grant))) for name in ('owner', 'other', 'ai'): - db.execute('INSERT INTO records VALUES(?,?,0)', ('shale-session:' + name, json.dumps({'origin': 'https://shale.studio.test/'}))) + db.execute('INSERT INTO records VALUES(?,?,0)', ('shale-session:' + name, json.dumps({'origin': 'https://shale.snowglobe.test/'}))) self.db = sqlite3.connect(self.root / 'shale.sqlite') self.db.executescript(''' CREATE TABLE users(id INTEGER PRIMARY KEY,uuid TEXT,name TEXT,provider TEXT,snowflake TEXT,joined_on TEXT,last_updated TEXT); @@ -37,7 +37,7 @@ class ShaleTests(unittest.TestCase): CREATE TABLE issue_labels(id INTEGER PRIMARY KEY,uuid TEXT,repo INTEGER REFERENCES repositories(id),name TEXT,description TEXT,color TEXT,open_issues INTEGER DEFAULT 0,last_updated TEXT); CREATE TABLE issues__labels(id INTEGER PRIMARY KEY,uuid TEXT,issue INTEGER REFERENCES issues(id),label INTEGER REFERENCES issue_labels(id)); CREATE TABLE issue_actions(id INTEGER PRIMARY KEY,uuid TEXT,issue INTEGER REFERENCES issues(id),actor INTEGER REFERENCES users(id),kind TEXT,payload TEXT,last_updated TEXT,added_on TEXT,deleted_on TEXT); - INSERT INTO users(id,name,provider,snowflake) VALUES(1,'owner','snowglobe.studio.test','owner'),(2,'other','snowglobe.studio.test','other'),(3,'ai','snowglobe.studio.test','ai'); + INSERT INTO users(id,name,provider,snowflake) VALUES(1,'owner','snowglobe.snowglobe.test','owner'),(2,'other','snowglobe.snowglobe.test','other'),(3,'ai','snowglobe.snowglobe.test','ai'); INSERT INTO repositories(id,name,owner,last_updated) VALUES(1,'chat',1,''),(2,'other',2,''); INSERT INTO issues(id,repo,number,status,last_updated) VALUES(1,1,1,'todo',''),(2,1,2,'done',''),(3,2,1,'todo',''); ''') diff --git a/tools/shale-labels.py b/tools/shale-labels.py index c672a604b553bd4640b33c0efe5fdaf30ca65dd8..13e5f936ba0bc86717c2d878aaa0099ed1a0230c 100644 --- a/tools/shale-labels.py +++ b/tools/shale-labels.py @@ -15,7 +15,7 @@ Error = import_module('dashboard-run').Error def database(mode, timeout): - path = Path(os.environ.get('STUDIO_SHALE_DATABASE', '/srv/prod/shale/data/astheno.shale.db')) + path = Path(os.environ.get('SNOWGLOBE_SHALE_DATABASE', '/srv/prod/shale/data/astheno.shale.db')) return sqlite3.connect(path.as_uri() + '?mode=' + mode, uri=True, timeout=timeout) @@ -36,7 +36,7 @@ def metadata(request): or any(not isinstance(n, str) or len(n) > 254 for n in names) or request['repository'] is not None and not isinstance(request['repository'], str)): raise Error(400, 'Choose up to 1024 account names.') - accounts = Path(os.environ.get('STUDIO_DASHBOARD_DATA', '/var/lib/studio/dashboard')) + accounts = Path(os.environ.get('SNOWGLOBE_DASHBOARD_DATA', '/var/lib/snowglobe/dashboard')) with closing(database('ro', 2)) as db: db.execute('ATTACH DATABASE ? AS auth', ((accounts / 'accounts.sqlite').as_uri() + '?mode=ro',)) result = {} @@ -46,14 +46,14 @@ def metadata(request): "JOIN auth.users h ON h.id=json_extract(a.profile,'$.owner') " "JOIN users o ON o.snowflake=h.id AND o.provider=u.provider " "WHERE u.provider=? AND json_extract(a.profile,'$.kind')='ai' AND u.name IN (" + ','.join('?' for _ in names) + ")", - ['snowglobe.' + os.environ.get('STUDIO_DOMAIN', 'studio.test'), *names]): + ['snowglobe.' + os.environ.get('SNOWGLOBE_DOMAIN', 'snowglobe.test'), *names]): result[name] = {'model': model, 'owner': owner, 'picture': picture} for name, seed, picture in db.execute( "SELECT u.name,u.uuid,json_extract(a.profile,'$.attributes.picture[0]') " "FROM users u JOIN auth.users a ON a.id=u.snowflake " "WHERE u.provider=? AND coalesce(json_extract(a.profile,'$.kind'),'user') NOT IN ('ai','guest') " "AND u.name IN (" + ','.join('?' for _ in names) + ")", - ['snowglobe.' + os.environ.get('STUDIO_DOMAIN', 'studio.test'), *names]): + ['snowglobe.' + os.environ.get('SNOWGLOBE_DOMAIN', 'snowglobe.test'), *names]): if picture: result[name] = {'avatar': seed, 'picture': picture} row = db.execute("SELECT (SELECT count(*) FROM issues WHERE repo=r.id AND status NOT IN ('done','not_planned','duplicate','invalid')) FROM repositories r WHERE name=?", (request['repository'],)).fetchone() @@ -117,8 +117,8 @@ def handle(request): raise Error(400, 'Supply issue text within the tool limits.') if action == 'create_issue' and args['status'] not in ('needs_triage', 'todo'): raise Error(400, 'Choose triage or todo for a new issue.') - accounts = Path(os.environ.get('STUDIO_DASHBOARD_DATA', '/var/lib/studio/dashboard')) - domain = os.environ.get('STUDIO_DOMAIN', 'studio.test') + accounts = Path(os.environ.get('SNOWGLOBE_DASHBOARD_DATA', '/var/lib/snowglobe/dashboard')) + domain = os.environ.get('SNOWGLOBE_DOMAIN', 'snowglobe.test') with closing(database('rw', 10)) as db, db: db.row_factory = sqlite3.Row db.execute('PRAGMA foreign_keys=ON') @@ -139,7 +139,7 @@ def handle(request): native_caller = db.execute('SELECT id FROM users WHERE provider=? AND snowflake=?', ('snowglobe.' + domain, grant['user'])).fetchone() if not repo or not native_caller or repo[2] == 'off': raise Error(403, 'This Shale connection cannot edit this repository.') - policy = json.loads(Path(os.environ.get('STUDIO_SHALE_POLICY', str(Path(__file__).resolve().parent.parent / 'config/shale.json'))).read_text()) + policy = json.loads(Path(os.environ.get('SNOWGLOBE_SHALE_POLICY', str(Path(__file__).resolve().parent.parent / 'config/shale.json'))).read_text()) owner = policy.get('actorOwners', {}).get(caller[0], grant['user']) issue = None if action == 'create_issue' or 'id' not in args else db.execute('SELECT id,status,author,title FROM issues WHERE repo=? AND number=?', (repo[0], args['id'])).fetchone() if 'id' in args and not issue: diff --git a/tools/shale-migration.md b/tools/shale-migration.md deleted file mode 100644 index 830ef0744201613d7f423dbc87bb40b9de065fc7..0000000000000000000000000000000000000000 --- a/tools/shale-migration.md +++ /dev/null @@ -1,83 +0,0 @@ -# Shale migration - -Shale is the sole Git server on Zenith. Personal Forgejo is retired. Retained Forgejo repositories and database exports remain migration sources; copying the existing Shale directory does not import repositories that only existed in Forgejo. The legacy multi-server SSH router is not part of the target setup. - -The 2026-10-04 personal inventory found 12 Shale owned repositories, no Shale mirrors, and 19 retained Forgejo bare repositories: 18 under `clo` and one under `nix`. Forgejo's retained LFS directory contains no files. Complete both imports before Shale's first production activation. - -The production data migration on 2026-10-04 completed before activating either production Shale or Postgres. The retained Shale copy passed checksum and SQLite checks, then all 19 Forgejo histories imported with verified object-file hashes and ref mappings. The result contains 21 repository records. All 11 private Forgejo sources retain private access; existing Shale identities, tokens, and sessions survive. The original session secret is preserved in the Nomad variable and its private recovery copy. Production import evidence is `/var/lib/studio/shale-forgejo-import-ldfl9zuy/repositories.json`; the earlier service dataset state is `globe/prod/shale@before-shale-import-1791160403-42496`. - -After copying the retained Shale state into `/srv/prod/shale`, [import-forgejo-to-shale.py](import-forgejo-to-shale.py) imports the additional histories while Shale stays stopped. Its metadata JSON comes only from the separately restored **personal** Forgejo PostgreSQL database: - -```sql -SELECT coalesce(json_agg(x), '[]') FROM ( - SELECT r.id, r.owner_id, u.lower_name AS owner_name, - r.lower_name, r.name, r.is_private, r.default_branch, - r.is_empty, r.is_mirror - FROM repository r JOIN "user" u ON u.id = r.owner_id - ORDER BY u.lower_name, r.lower_name -) x; -``` - -```sh -python3 tools/import-forgejo-to-shale.py \ - --target /srv/prod/shale \ - --metadata /run/personal-forgejo-repositories.json -``` - -The importer preserves existing Shale identities and repository records. Clover's repositories retain their names; the other namespace becomes `nix/config`. It copies and hashes Git objects without copying Forgejo hooks or configuration. Missing branches, tags, pull refs, and notes retain their names. A divergent branch or tag is retained under `refs/heads/forgejo//...` or `refs/tags/forgejo//...`; other conflicting refs use `refs/forgejo//...`. Existing Shale refs retain their values, including newer histories already migrated from old mirrors. Former Forgejo mirrors become owned histories, with their original mirror status recorded in the private evidence. That directory contains every original ref, its destination, all copied object hashes, and the previous SQLite database. - -New repository access follows verified Forgejo visibility, with pushes restricted to the owner and issue submission disabled. When private Forgejo history joins an existing repository, public or unlisted permissions are tightened to private **before objects are copied**; existing `off` permissions remain off. An import failure must leave Shale stopped. Restoring only the previous database can re-expose imported private objects through its old public permissions, so recovery must preserve the tightened access or restore the complete service dataset. - -Existing Shale account IDs and OIDC subjects must survive the cutover. A deliberate issuer move uses `service/shale/prepare.py` to rebind the provider before startup; duplicate bindings stop the migration. Shale caches repository metadata and access at startup, so finish SQLite changes before starting the application. An isolated pinned-image test proved direct registration of a new repository: private anonymous web access returned 404 and Git discovery returned 401; after public permissions and a restart, its page and Git advertisement returned 200 with the original branch object ID. The test used copied data, `--network none`, and no published ports. - -A full-data test imported all 19 retained Forgejo histories into a disposable copy of the existing Shale state, yielding 21 repository records. Its conservative fixture metadata marked every incoming repository private. Every copied object file and every source ref passed verification. The pinned application then denied anonymous access to a new private repository and a formerly public collision, while preserving an unaffected public repository. On that isolated copy, public test permissions proved HTTP pages and Git advertisements for `bgds`, `home-infra`, and `nix/config`; `home-infra` advertised the original Forgejo `main` and `vllm`. An actual smart HTTP fetch returned a 53-object pack with a verified pack checksum. Production visibility must come from the restored Forgejo metadata, not this test fixture. - -After migration and authenticated browser checks, push the new infra `main` to `https://shale.paperclover.net/home-infra.git` with a Shale personal access token. The existing `home-infra` record and UUID are retained; the imported Forgejo branches remain available alongside the tested final `main`. - -The October 5 issue migration uses `tools/import-forgejo-issues.py` and the verified personal PostgreSQL export at `/var/lib/studio/forgejo-issue-migration/source.json`. Its sibling `source-proof.json` records the source checksum. The export contains 452 issues, 1,429 comments and events, 60 labels, 24 attachments, and 312 history revisions. Pull request discussions become issues with their original branch and merge metadata. Historical authors retain their names through non-login identities; `paperclover` maps to the existing Clover account. Original creation, update, and comment timestamps are preserved. Every source row is retained in the private `studio_forgejo_records` ledger, including revision history that Shale cannot present as editable comments. - -Run this while the target job is stopped and its allocations have exited: - -```sh -python3 tools/import-forgejo-issues.py \ - --target /srv/prod/shale \ - --source /var/lib/studio/forgejo-issue-migration/source.json -``` - -For an isolated stage, add `--rehearsal` and use `/srv/staging/shale-preview-`. The importer verifies the export checksum, checks that the target job is stopped before writing and immediately before commit, backs up SQLite into a private evidence directory, verifies native issue records remain unchanged, and supports an idempotent rerun of the same export. Take a full service ZFS snapshot before the production import as well. Do not replace production with the stage clone: import again into the stopped production dataset so newer native issues survive. - -Six occupied numbers are remapped: `chat` #1→#6 and #2→#7; `home-infra` #1→#38, #2→#39, and #3→#40; `react-mutation` #1→#19. All other original numbers remain. Explicit Forgejo issue links are rewritten and remapped bodies identify their original number. Bare `#references` and code text remain untouched. Attachments are copied with verified SHA-256 hashes and served through an authorization check against their associated Shale issue. Their local files are readable only by root and Caddy; private issue attachments remain private. - -The writable ZFS rehearsal `shale-preview-0242cee6` starts from all 104 existing native issues and imports to 556 total. It preserves the original Clover OIDC identity. September's `r1616-ga87d2f5.zig.0.16.0` avoids the `r1758` anonymous deleted-comment crash, but its Markdown scanner uses a shared capture buffer and crashes under concurrent fenced-code rendering. The documented `NPROC=1` worker setting avoids that race. With this setting, 904 authenticated/anonymous imported-issue requests passed with eight clients, all 24 attachment hashes and access checks passed, and browser closing of a disposable copy of `chat` #1 succeeded. Production `chat` #1 remains untouched. - -Production issue import completed on October 5 under release `dda941e618934ad9`, from committed main `51be72a9`. Recovery snapshot: `globe/prod/shale@before-forgejo-issues-20261005T080143Z-14c71a`. The private import report and SQLite backup live at `/var/lib/studio/forgejo-issue-migration/issue-import-c726euv2`. All 104 native issues survived alongside the 452 imported issues. After the route reload settled, 904 concurrent HTTPS issue reads passed (644 successful reads, 260 expected private-page denials), all 24 attachment hashes and access checks passed, SQLite integrity passed, and a forged-Origin request returned 403. Browser navigation confirmed historical timestamps, comments, labels, and status on a live imported issue. Native `chat` #1 remains Todo. The disposable Keycloak rehearsal was destroyed after verification; the production recovery snapshot and private evidence remain. - -The r1616 build predated hidden form CSRF tokens. The router's exact HTTPS Origin guard preceded all Shale handlers inside an explicit Caddy `route`; default directive ordering could otherwise bypass it. Missing, wrong, and suffix-forged origins returned 403 without a database change on the clone. The valid site origin allowed a status change, while Basic-auth Git requests still reached Shale. At that time, the MCP adapter accepted tokenless issue forms only with the verified `r1616` structural footer and no CSRF-token input anywhere on the page. - -The October 5 `r1763-g9f5b1c7.zig.0.16.0` upgrade rehearsal passed anonymous reads of the formerly crashing issues, Unicode issue creation and comments, issue closing with a comment Delete form present, access denial, restart persistence, token revocation, and logout. The Rust adapter parsed its actual owner issue markup and accepted its status/comment CSRF fields. All migrated SQLite rows remained identical. Eight-worker concurrent Markdown rendering still crashed; keep `NPROC=1`, which passed 280 concurrent fenced-code requests. The injected issue-form script is no longer needed. Private evidence lives at `/var/lib/studio/shale-upgrade-0680d28c`; its disposable containers were removed after testing. - -Shale allocates an issue number from the issue with the highest SQLite row ID. The importer now inserts by destination number, including remapped collisions. The already-imported database needs `--repair-issue-order` once while its job is stopped: the guarded repair saves a SQLite backup, swaps six imported surrogate IDs, rewrites foreign keys and ledger references, and verifies every table's contents using issue UUIDs. Native IDs, public numbers, timestamps, comments, labels, attachments and history remain intact. It refuses duplicate numbers or changes to native issue IDs. On the repaired native-auth clone, the actual Rust Backend created `home-infra` #41, `react-mutation` #20 and `chat` #8, then commented on and closed the disposable `home-infra` issue. Fresh import, repeated import, repeated repair, and refusal checks passed. - -Production upgraded to `r1763` on October 5 in release `9ef0e1fd9c7a72bc`, from main `38ee0683`. The pre-cutover Shale recovery snapshot is `globe/prod/shale@native-auth-20261005T091312Z`. Fresh normal and guest sign-in callbacks returned 200; the existing Clover session and identity remained intact. All 452 imported issue routes passed concurrent anonymous checks (192 successful reads and 260 expected private-page denials), all 24 attachment hash/access checks passed, and all 556 issues remained unique with valid foreign keys. Native `chat` #1 remains Todo. The live proof is recorded in `/var/lib/studio/shale-upgrade-0680d28c/upgrade-validation.json`. - -The October 4 transport check inspected the then-pinned image in disposable containers without mounting real app data. Its embedded Git endpoint and account settings use HTTP and personal access tokens; no SSH listener, authorized-key interface, or forced-command handler was found. The [official installation](https://astheno.software/shale/installation/) and [configuration reference](https://astheno.software/shale/reference/environment/) also expose HTTP serving and OAuth login without SSH configuration. A `git` account must either use a Shale-aware SSH bridge or await native SSH support. Direct filesystem Git commands would bypass Shale's authorization. - -Zenith's Shale app directory contains a small SQLite database and 419 MB of owned repositories. `bash tools/import-shale.sh shale-preview-4eea0e3b` copied `data`, `repositories_owned`, and `repositories_mirrors` opaquely from the read-only `storage1/apps@hourly-2026-09-26_05-00` snapshot. It verified checksums and SQLite integrity, then restarted the preview. Both sides had 11 top-level owned repository directories; the preview had one healthy Nomad allocation and returned HTTPS 200. Repository contents were not inspected. - -For the production copy, stop Zenith's Shale container and the Snow Globe Shale job, set `STUDIO_DEPLOY_HOST` and `STUDIO_DEPLOY_PORT` for the new host, then run `bash tools/import-shale.sh shale`. The importer checks both jobs remain stopped, snapshots the destination dataset, verifies all three copied directories, and leaves Snow Globe stopped. Start the new job after the copy, check the SQLite state and a known login through the new Keycloak client, then switch the public route. The destination snapshot printed by the importer remains available for recovery. - -After a same-machine OS replacement, set `STUDIO_LEGACY_HANDOFF` to the [offline handoff](legacy-handoff.md) directory as well. The importer then reads the retained Shale directory from the new host's mounted old apps dataset, with no old Docker dependency. - -## Imported repository HTTP pushes - -The Forgejo importer originally omitted `http.receivepack=true`, which Shale sets when it creates an owned repository. Shale checks its push ACL before invoking `git-http-backend`, but does not set `REMOTE_USER`; Git's default therefore rejected authorized pushes to imported repositories. The importer now writes the setting, and Shale's prepare step repairs missing settings without replacing an explicit disable. Production `config` remains owner-only for pushes. - -Shale also forwards CGI `Status` as an ordinary header on HTTP 200. The Git-specific Caddy handler translates Git's error statuses into HTTP statuses and removes the CGI header. A disposable production database/repository copy verified anonymous and invalid credentials return 401, authenticated non-owners return 403, owner discovery advertises `main=37665e69e1aa954f0dec06b1cafa1612f44dc907`, and an actual `jj git push --bookmark main` advances the clone to `b8e734ce6aabd94a7e1aacfc989467662ca27dbe`. Disabled receive-pack returns actual HTTP 403; invalid method/content type return 400/415 without a `Status` header. No production token was added. - -## r1807 JSON upgrade, 2026-10-13 UTC - -The pinned image in [service.pkl](../service/shale/service.pkl) serves repository lists, issue lists, and issue details as JSON with `Accept: application/json` on their existing URLs. The MCP reads those responses, retaining issue numbers as `id` and raw Markdown in comment text. Native JSON omits comment IDs and issue-list creation times, so the MCP omits those fields. Account settings return 406 for JSON, and label JSON omits numeric IDs; those two reads retain their HTML checks. Writes still use the host broker for durable model authorship, repository permissions, delegated statuses, and grant revocation. - -The inherited `shale-preview-8ff39b2c` contained earlier disposable test writes. Rehearsing its previous pinned image followed by r1807 preserved every row of all 10 existing tables, including 559 issues; SQLite integrity passed and its allocation had zero restarts. Preview mirror cloning and polling are disabled. Public browsing, README/file navigation, open/closed issue filters, issue detail, labels, keyboard navigation, and 390-pixel layouts passed browser checks. - -An independent instance used only synthetic databases, credentials, certificates, and native Snowglobe OIDC. The candidate Nix dashboard passed linking, every issue/label operation, raw Markdown and durable model attribution, anonymous/other-user/selected-repository denials, exact-Origin enforcement, uncertain-write handling without replay, 80 concurrent JSON reads, 280 concurrent fenced-Markdown reads, backend/dashboard restart persistence, revocation, unlinking, and SQLite integrity. `NPROC=1` remains in place. The router test also verifies JSON headers and bytes pass through unchanged. Restricted verification evidence is `/var/lib/studio/shale-json-upgrade-20261013`. diff --git a/tools/snowglobe-hardware.md b/tools/snowglobe-hardware.md new file mode 100644 index 0000000000000000000000000000000000000000..c1ada1f9f48668ffe99f2c56cbd8dcb2524e40bc --- /dev/null +++ b/tools/snowglobe-hardware.md @@ -0,0 +1,105 @@ +# Snow Globe hardware and recovery + +## Physical host + +Snow Globe is the existing Ryzen 9 5950X computer, with 128 GB RAM, an RTX 3090, +a 500 GB WD Blue SN5000 NVMe OS disk, and four 8 TB WD80EFPX disks in the +`globe` RAIDZ1 pool. The onboard RTL8168 Ethernet connection is 1 Gb/s. +[The host configuration](../nixos/snowglobe.nix), +[network configuration](../nixos/snowglobe-network.nix), and +[filesystem configuration](../nixos/hardware-configuration.nix) own installed +settings. Use the release deployment workflow for host changes. + +Read-only inspection on 2026-10-16 confirmed the physical machine at +`root@10.0.0.1:22`, a healthy `globe` pool with zero known data errors, and +mounted production, staging, Clover, and Media datasets. The host configuration +names it `snowglobe`. Renaming the computer preserves the existing hardware, +pool, and application state. + +## Storage boundaries + +| Dataset | Mount | Role | +| --- | --- | --- | +| `globe` | `/srv` | Pool root; unencrypted | +| `globe/clover` | `/srv/clover` | Personal files; independent encryption root | +| `globe/clover/Media` | `/srv/clover/Media` | Media; independent encryption root | +| `globe/prod` | `/srv/prod` | Encrypted managed service data | +| `globe/staging` | `/srv/staging` | Encrypted preview data | +| `globe/apps` | `/mnt/storage1/apps` | Retained application recovery source | +| `globe/logs` | `/srv/logs` | Independent encrypted dataset | + +The retained apps tree and other old datasets are not active deployment +sources. Removing obsolete repository import tools does not authorize destroying +those datasets, snapshots, or private migration evidence. + +Clover and Media are case-insensitive POSIX-ACL filesystems. Their current +properties are `acltype=posix`, `aclmode=discard`, `aclinherit=discard`. +Production and staging also use POSIX ACLs. Preserve allocated UIDs and the +shared group 3000. Test permission changes on a clone; do not recursively +change personal ownership to repair one service. + +The [site mapping](../config/site.pkl) owns media folder names. Saved application +paths are preserved by the service bind mounts: qBittorrent sees Seedbox at +both `/data/media/Seedbox` and `/data/media/torrent`; Jellyfin keeps its +`/media/jellyfin/...` paths. Preserve those mappings and YouTube archive files +alongside their videos when reorganizing media. + +Samba serves these datasets directly using POSIX permissions. Its declaration +preserves shared editing with `0660` files and `2770` directories and uses +kernel oplocks for service-write visibility. Run +`swift tools/samba-macos-test.swift ` against disposable files +after changing permissions or SMB configuration. The probe exercises Cocoa +safe saves, packages, metadata/resource forks, and cleanup. + +## Unlock and recovery + +The host configuration lists six encrypted roots and their systemd credential +names. TPM-encrypted blobs live in `/etc/credstore.encrypted`; the unlock unit +runs after pool import and before mounting. Plaintext keys must stay outside +the Nix store and tracked configuration. Nomad refuses to start without the +actual pool, Clover, Media, production, and staging mounts. + +Credentials are bound to the machine's TPM and PCR 7. Clearing/replacing the +TPM or changing firmware can require manual unlock and resealing. Secure Boot +is disabled, so automatic TPM unlock does not establish a trusted boot chain. +Keep independent recovery keys when replacing hardware. + +The private off-server recovery archive on Clover's Mac includes the additional +production/staging keys in `postgres-backups/new-os-dataset-keys.json` and the +verified pre-install PostgreSQL logical/physical exports in +`postgres-backups/20261004T230507Z-e75419.tar`. Original dataset keys +and the former OS configuration are in the same private recovery collection. +These paths document locations only; they do not establish current backup +completeness or authorize opening private contents. + +For manual unlock on a confirmed recovery host, recover the required key into +a root-only file in `/run`, then run these commands one at a time with the +verified dataset substituted: + +```sh +zfs load-key -L file:///run/ +rm /run/ +zfs mount +findmnt --mountpoint +``` + +Loading a key does not mount the filesystem. Mount parents before children; +never overlay a nonempty mountpoint to bypass a mount failure. + +## Control state and backups + +The NVMe contains both Nomad state and `/var/lib/snowglobe`: service identities, +ACL token files, dashboard accounts, release history, and backup manifests and +database dumps. Read-only `findmnt` inspection on 2026-10-16 confirmed both +control-state locations remain on the ext4 OS filesystem, outside the ZFS +service datasets. Replacing the OS disk without recovering this state loses +more than service files. A Nomad snapshot contains jobs, ACLs, and secret +variables; it does not contain the separate dashboard account database or UID +registry. See [data restore](data-restore.md) for service recovery boundaries. + +No automatic ZFS snapshot or release-backup retention schedule is configured. +Release backups do not substitute for independent off-server recovery copies. +Before destructive storage or OS work, verify key recovery and application/ +database restores from at least two restricted copies stored separately off +this computer. Preserve the existing encrypted roots, pool host ID, numeric +owners, SSH trust, and retained recovery sources through a rebuild. diff --git a/tools/snowglobe-secret-files-test.py b/tools/snowglobe-secret-files-test.py new file mode 100644 index 0000000000000000000000000000000000000000..1be2eefec089bff962690a31624ab7a3cbb7b524 --- /dev/null +++ b/tools/snowglobe-secret-files-test.py @@ -0,0 +1,72 @@ +#!/usr/bin/env python3 +import json +import os +from pathlib import Path +import subprocess +import tempfile +import unittest + +import snowglobe + + +class SecretFiles(unittest.TestCase): + @classmethod + def setUpClass(cls): + with tempfile.TemporaryDirectory(prefix="snowglobe-secret-template-") as temporary: + fixture = Path(temporary) / "fixture.pkl" + fixture.write_text('amends ' + json.dumps((snowglobe.REPO / "config/Service.pkl").as_uri()) + '\nname = "Fixture"\ncontainer { image = "localhost/fixture:pinned" }\n') + result = subprocess.run([os.environ.get("PKL", "pkl"), "eval", "-p", "uid=3132", "-p", "serviceId=fixture", str(fixture)], check=True, capture_output=True, text=True) + cls.fixture = json.loads(result.stdout) + cls.fixture.update(uid=3132, sourceId="fixture", inputs={}) + cls.fixture.pop("requirements") + + def setUp(self): + self.data = json.loads(json.dumps(self.fixture)) + self.task = self.data["containers"]["app"] + self.task["secretFiles"] = {"/run/secrets/oidc.json": '{{ with nomadVar "nomad/jobs/fixture/inputs/oidc" }}{"clientSecret":{{ .clientSecret.Value | toJSON }}}{{ end }}'} + + def test_private_allocation_template_and_read_only_file_mount(self): + rendered = snowglobe.render(self.data, {"fixture": self.data}) + self.assertIn('"secrets/oidc.json:/run/secrets/oidc.json:ro"', rendered) + self.assertIn('destination = "secrets/oidc.json"', rendered) + self.assertIn('perms = "0400"', rendered) + self.assertIn('uid = 3132', rendered) + self.assertIn('gid = 3132', rendered) + self.assertIn('error_on_missing_key = true', rendered) + self.assertIn(self.task["secretFiles"]["/run/secrets/oidc.json"], rendered) + self.assertNotIn('env = true', rendered) + self.assertNotIn('change_mode =', rendered) + + def test_services_without_files_gain_no_template_or_mount(self): + self.task["secretFiles"] = {} + rendered = snowglobe.render(self.data, {"fixture": self.data}) + self.assertNotIn('template {', rendered) + self.assertNotIn('secrets/', rendered) + + def test_unsafe_and_colliding_targets_are_rejected(self): + for target in ("relative.json", "/run/../secret.json", "/run/./secret.json", "/run//secret.json", "/run/secrets/", "/secrets/file", "/secrets", "/run/app.env", "/run/app-secret.env", "/run/nomad_token", "/run/line\nbreak"): + with self.subTest(target=target): + self.task["secretFiles"] = {target: "{}"} + with self.assertRaisesRegex(ValueError, "invalid secret file target"): + snowglobe.render(self.data, {"fixture": self.data}) + self.task["secretFiles"] = {"/one/config.json": "{}", "/two/config.json": "{}"} + with self.assertRaisesRegex(ValueError, "invalid secret file target"): + snowglobe.render(self.data, {"fixture": self.data}) + self.task["secretFiles"] = {"/run/secrets/oidc.json": "{}"} + self.task["volumes"] = {"/run/secrets/oidc.json": {}} + with self.assertRaisesRegex(ValueError, "invalid secret file target"): + snowglobe.render(self.data, {"fixture": self.data}) + + def test_image_user_and_heredoc_injection_are_rejected(self): + self.task["imageUser"] = True + with self.assertRaisesRegex(ValueError, "service-owned"): + snowglobe.render(self.data, {"fixture": self.data}) + self.task["imageUser"] = False + for body in ("EOF", "{}\nEOF\nmalicious", "{}\nEOF\r\nmalicious"): + self.task["secretFiles"] = {"/run/secrets/oidc.json": body} + with self.assertRaisesRegex(ValueError, "invalid secret template delimiter"): + snowglobe.render(self.data, {"fixture": self.data}) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/snowglobe-sso-test.py b/tools/snowglobe-sso-test.py new file mode 100644 index 0000000000000000000000000000000000000000..ea5bd0168b18167fea8d580939644e77af97c697 --- /dev/null +++ b/tools/snowglobe-sso-test.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python3 +import json +import os +from pathlib import Path +import subprocess +import tempfile +import unittest +from unittest.mock import patch + +import snowglobe + + +class SsoConnectivity(unittest.TestCase): + client = '''new sso.Client { + clientId = module.id + name = module.name + redirectUris { "https://fixture.example/api/callback" } +}''' + + def load(self, requirements=client, extra_hosts=()): + with tempfile.TemporaryDirectory(prefix="snowglobe-sso-") as temporary: + fixture = Path(temporary) / "fixture.pkl" + fixture.write_text(f'''amends {json.dumps((snowglobe.REPO / "config/Service.pkl").as_uri())} +import {json.dumps((snowglobe.REPO / "config/OpenID.pkl").as_uri())} as sso +name = "Fixture" +requirements {{ {requirements} }} +containers {{ + ["app"] {{ + image = "localhost/fixture:pinned" + extraHosts {{ {"; ".join(json.dumps(host) for host in extra_hosts)} }} + }} + ["worker"] {{ image = "localhost/fixture:pinned" }} +}} +''') + def pkl(*args, capture=False): + return subprocess.run([os.environ.get("PKL", "pkl"), *args[1:]], check=True, capture_output=capture, text=True) + with patch.object(snowglobe, "service_files", return_value={"fixture": fixture}), patch.object(snowglobe, "identity", return_value=3132), patch.object(snowglobe, "command", side_effect=pkl): + return snowglobe.load("fixture", {"domain": "fixture.example"}) + + def render(self, data): + return snowglobe.render(data, {"fixture": data}) + + def test_client_declaration_routes_every_task_without_changing_provider_inputs(self): + data = self.load() + self.assertEqual(data["inputs"]["oidc"], { + "alias": "oidc", "provider": "snowglobe", "kind": "client", "clientId": "fixture", + "name": "Fixture", "redirectUris": ["https://fixture.example/api/callback"], + "usernameAliases": {}, "allowGuests": False, "usernameRequired": False, + }) + rendered = self.render(data) + self.assertEqual(rendered.count('extra_hosts = ["snowglobe.fixture.example:host-gateway"]'), 2) + self.assertNotIn("snowglobe.paperclover.net", rendered) + + def test_explicit_hosts_take_precedence_and_duplicates_are_removed(self): + data = self.load(extra_hosts=[ + "Snowglobe.Fixture.Example:192.0.2.10", + "other.example:2001:db8::10", + "other.example:2001:db8::10", + ]) + rendered = self.render(data) + self.assertIn('extra_hosts = ["Snowglobe.Fixture.Example:192.0.2.10", "other.example:2001:db8::10"]', rendered) + self.assertEqual(rendered.count("snowglobe.fixture.example:host-gateway"), 1) + + def test_multiple_clients_share_one_host_binding_per_task(self): + other = self.client.replace('clientId = module.id', 'alias = "other"; clientId = "other"') + data = self.load(requirements=self.client + "\n" + other) + self.assertEqual(self.render(data).count("snowglobe.fixture.example:host-gateway"), 2) + + def test_services_without_sso_preserve_their_explicit_hosts(self): + data = self.load(requirements="", extra_hosts=["other.example:host-gateway"]) + rendered = self.render(data) + self.assertIn('extra_hosts = ["other.example:host-gateway"]', rendered) + self.assertNotIn("snowglobe.fixture.example", rendered) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/snowglobe.py b/tools/snowglobe.py new file mode 100644 index 0000000000000000000000000000000000000000..7f6397de3a70a367a3b73afe0811731383d638a8 --- /dev/null +++ b/tools/snowglobe.py @@ -0,0 +1,1303 @@ +#!/usr/bin/env python3 +import argparse +from contextlib import nullcontext +import fcntl +import hashlib +import json +from importlib import import_module +import os +from pathlib import Path +import re +import secrets +import shutil +import subprocess +import sys +import tempfile +import time + +from data import cloned_postgres, dataset_for + + +transition = import_module("namespace-transition") +transition.inherit_environment() + +REPO = Path(__file__).resolve().parent.parent +SERVICES = REPO / "service" +STATE = Path("/var/lib/snowglobe") +IDENTITIES = STATE / "identities.json" +ASSETS = Path("/var/lib/caddy/snowglobe") +NAME = re.compile(r"[a-z][a-z0-9-]*\Z") + + +def command(*args, capture=False, **kwargs): + return subprocess.run(args, check=True, text=True, capture_output=capture, **kwargs) + + +def service_files(): + files = {} + for directory in SERVICES.iterdir(): + if not directory.is_dir(): + continue + for path in directory.glob("*.pkl"): + name = directory.name if path.name == "service.pkl" else path.stem + if name in files: + raise ValueError(f"duplicate service definition: {name}") + files[name] = path + return files + + +def services(): + return sorted(service_files()) + + +def service_dir(name): + path = service_files()[name] + return path.parent if path.name == "service.pkl" else path.parent / name + + +def dependencies(data): + return set(data["dependsOn"]) | {item["provider"] for item in data["inputs"].values() if item["provider"] != "snowglobe"} + + +def ordered(data): + pending = {item["id"]: item for item in data} + result = [] + while pending: + ready = [ + item + for item in pending.values() + if not dependencies(item) & pending.keys() + ] + if not ready: + raise ValueError("cycle in service dependencies") + for item in ready: + result.append(item) + del pending[item["id"]] + return result + + +def identity(name): + STATE.mkdir(parents=True, exist_ok=True) + if not IDENTITIES.exists(): + write_private(IDENTITIES, (REPO / "config/identities.json").read_text()) + with (IDENTITIES.parent / ".identities.lock").open("w") as lock: + fcntl.flock(lock, fcntl.LOCK_EX) + with IDENTITIES.open() as file: + ids = json.load(file) + if len(set(ids.values())) != len(ids): + raise ValueError("duplicate service UIDs") + if name not in ids: + used = set(ids.values()) + ids[name] = next(uid for uid in range(3100, 60000) if uid not in used) + pending = IDENTITIES.with_suffix(".pending") + pending.write_text(json.dumps(ids, indent=2) + "\n") + pending.replace(IDENTITIES) + return ids[name] + + +def load(name, properties, instance_id=None): + files = service_files() + if not NAME.fullmatch(name) or name not in files: + raise ValueError(f"unknown service: {name}") + instance_id = instance_id or name + if not NAME.fullmatch(instance_id): + raise ValueError(f"invalid instance ID: {instance_id}") + uid = identity(name) + result = command( + "pkl", + "eval", + *( + part + for key, value in {**properties, "serviceId": instance_id, "uid": uid, "preview": str(instance_id != name).lower()}.items() + for part in ("-p", f"{key}={value}") + ), + str(files[name]), + capture=True, + ) + data = json.loads(result.stdout) + if data["id"] != instance_id: + raise ValueError(f"service ID must match deployment: {instance_id}") + inputs = {} + for requirement in data.pop("requirements"): + alias = requirement["alias"] + if not NAME.fullmatch(alias) or alias == "own" or alias in inputs: + raise ValueError(f"invalid or duplicate requirement alias: {alias}") + inputs[alias] = requirement + issuer_hosts = [request.pop("issuerHostname") for request in inputs.values() + if request["provider"] == "snowglobe" and request["kind"] == "client"] + data["inputs"] = inputs + data["uid"] = uid + data["sourceId"] = name + for task_name, task in data["containers"].items(): + task["extraHosts"] = list(dict.fromkeys(task["extraHosts"])) + for hostname in issuer_hosts: + if all(host.split(":", 1)[0].lower() != hostname.lower() for host in task["extraHosts"]): + task["extraHosts"].append(f"{hostname}:host-gateway") + if bool(task.get("image")) == bool(task.get("build")): + raise ValueError(f"{name}.{task_name} needs one image or build directory") + if task.get("build"): + if REPO.parent == Path("/opt/snowglobe/releases").resolve(): + context = config_path(name, task["build"]) + if not context.is_dir() or not (context / "Dockerfile").is_file(): + raise ValueError(f"invalid build context: {name}.{task_name}") + digest = hashlib.sha256() + for path in sorted(context.rglob("*")): + if path.is_file(): + digest.update(str(path.relative_to(context)).encode() + b"\0") + digest.update(path.read_bytes()) + tag = digest.hexdigest()[:16] + elif not (service_dir(name) / task["build"] / "Dockerfile").is_file() and not (service_dir(name) / "build-source.json").is_file(): + raise ValueError(f"missing build source: {name}.{task_name}") + else: + tag = REPO.name + task["image"] = f"localhost/snowglobe/{name}-{task_name}:{tag}" + return data + + +def q(value): + return json.dumps(value, ensure_ascii=False) + + +def write_private(path, content): + pending = None + try: + with tempfile.NamedTemporaryFile("w", dir=path.parent, delete=False) as file: + pending = Path(file.name) + file.write(content) + pending.replace(path) + finally: + if pending: + pending.unlink(missing_ok=True) + + +def host_path(root, relative): + if relative == ".": + return root + part = Path(relative) + if part.is_absolute() or ".." in part.parts or not part.parts: + raise ValueError(f"invalid relative path: {relative}") + return str(Path(root) / part) + + +def config_path(service, relative): + root = service_dir(service).resolve() + source = (root / relative).resolve() + if not source.is_relative_to(root) or not source.exists(): + raise ValueError(f"invalid config path: {relative}") + return source + + +def config_assets(data): + paths = set() + for task in data["containers"].values(): + paths.update(volume["config"] for volume in task["volumes"].values() if volume.get("config")) + if task.get("http"): + paths.update(task["http"]["overrideFiles"].values()) + sources = {rel: config_path(data.get("sourceId", data["id"]), rel) for rel in paths} + digest = hashlib.sha256() + for rel, source in sorted(sources.items()): + digest.update(rel.encode()) + files = sorted(source.rglob("*")) if source.is_dir() else [source] + for file in files: + if file.is_symlink(): + raise ValueError(f"config asset cannot be a symlink: {file}") + if file.is_file(): + digest.update(str(Path(rel) / file.relative_to(source)).encode() if source.is_dir() else rel.encode()) + digest.update(file.read_bytes()) + return ASSETS / data["id"] / digest.hexdigest()[:16], sources + + +def render(data, definitions): + if not data["containers"]: + return "" + name = data["id"] + asset_dir, _ = config_assets(data) + prepare_digest = None + if data.get("prepare"): + source = service_dir(data.get("sourceId", name)) + digest = hashlib.sha256() + for path in sorted(source.rglob("*")): + if not path.is_file() or "__pycache__" in path.parts or path.suffix == ".pyc": + continue + if path.name == "service.pkl" or path.name.startswith("icon"): + continue + digest.update(str(path.relative_to(source)).encode() + b"\0") + digest.update(path.read_bytes()) + prepare_digest = digest.hexdigest()[:16] + routed = [task["http"] for task in data["containers"].values() + if task.get("http") and task["http"].get("hostname")] + primary = routed[0] if routed else {} + requirements = {provider: "startup" for provider in data["dependsOn"]} + requirements.update({item["provider"]: item["kind"] for item in data["inputs"].values()}) + secrets_meta = [ + {"name": name, "generated": True, "bytes": spec["bytes"]} + for name, spec in data["secrets"].items() + ] + [{"name": name, "generated": False} for name in data["requiredSecrets"]] + lines = [ + f"job {q(name)} {{", + ' datacenters = ["clover"]', + ' type = "service"', + ' meta {', + f" snowglobe_service = {q(data['sourceId'])}", + f" snowglobe_name = {q(data['name'])}", + f" snowglobe_tagline = {q(data['tagline'])}", + f" snowglobe_hostname = {q(primary.get('hostname') or '')}", + f" snowglobe_launcher = {q(str(data['launcher']).lower())}", + f" snowglobe_auth_role = {q(data.get('access') or primary.get('authRole') or '')}", + f" snowglobe_trace_service = {q(data.get('traceServiceName') or '')}", + f" snowglobe_metrics_pushed = {q(str(data['metricsPushed']).lower())}", + f" snowglobe_requires = {q(json.dumps(requirements))}", + f" snowglobe_secrets = {q(json.dumps(secrets_meta))}", + ' }', + ' group "app" {', + ] + if dependencies(data): + lines += [ + " restart {", + " attempts = 3", + ' delay = "1m"', + ' interval = "4m"', + ' mode = "delay"', + " }", + ] + if data["rollout"] == "overlapped": + for task in data["containers"].values(): + if task["hostNetwork"] or any( + endpoint and endpoint.get("hostPort") is not None + for endpoint in (task.get("http"), *task["tcp"]) + ) or any(not volume["readOnly"] for volume in task["volumes"].values()): + raise ValueError(f"overlapped rollout requires dynamic ports and read-only mounts: {name}") + elif data["rollout"] != "simple": + raise ValueError(f"unknown rollout: {data['rollout']}") + if data["rollout"] == "overlapped" or data.get("healthyDeadline"): + lines.append(" update {") + if data["rollout"] == "overlapped": + lines += [" canary = 1", " auto_promote = true", " auto_revert = true"] + if data.get("healthyDeadline"): + lines += [f" healthy_deadline = {q(data['healthyDeadline'])}", ' progress_deadline = "0"'] + lines.append(" }") + if data.get("healthyDeadline"): + lines += [" migrate {", f" healthy_deadline = {q(data['healthyDeadline'])}", " }"] + ports = {} + network = [] + host_network = any(task["hostNetwork"] for task in data["containers"].values()) + if host_network and not all(task["hostNetwork"] for task in data["containers"].values()): + raise ValueError("mixed host and bridge networking in one group") + for task_name, task in data["containers"].items(): + for kind, endpoint in [("http", task.get("http")), *((port["name"], port) for port in task["tcp"])]: + if endpoint: + label = kind if len(data["containers"]) == 1 else f"{task_name}-{kind}" + if label in ports.values(): + raise ValueError(f"duplicate port label: {label}") + ports[(task_name, kind)] = label + network.append(f" port {q(label)} {{") + if endpoint.get("hostPort") is not None: + network.append(f" static = {endpoint['hostPort']}") + if not task["hostNetwork"]: + network.append(f" to = {endpoint['containerPort']}") + if endpoint.get("loopback") or (not task["hostNetwork"] and kind == "http" and endpoint.get("hostPort") is None): + network.append(' host_network = "loopback"') + network.append(" }") + if network: + lines.append(" network {") + if host_network: + lines.append(' mode = "host"') + lines += network + [" }"] + for provider_id in sorted(dependencies(data)): + provider = definitions[provider_id] + for provider_task, container in provider["containers"].items(): + endpoint = container.get("http") + if not endpoint or not endpoint.get("hostname") or endpoint.get("authRole"): + continue + host = endpoint["hostname"] + task_name = f"wait-{provider_id}-{provider_task}" + lines += [ + f" task {q(task_name)} {{", + ' driver = "podman"', + " lifecycle {", + ' hook = "prestart"', + " sidecar = false", + " }", + " config {", + ' image = "docker.io/curlimages/curl@sha256:43ebaa53d3806db6b1ce4353b6b26ae638ec1c167ee351524b05690f988bb20d"', + f" extra_hosts = {q([host + ':host-gateway'])}", + f" args = {q(['--insecure', '--fail', '--silent', '--show-error', '--retry', '999999', '--retry-all-errors', '--retry-delay', '5', '--connect-timeout', '3', '--max-time', '5', 'https://' + host + endpoint['checkPath']])}", + " }", + " resources {", + " cpu = 50", + " memory = 64", + " }", + " }", + ] + for task_name, task in data["containers"].items(): + if task.get("http") or task.get("tcp"): + kind = "http" if task.get("http") else "tcp" + endpoint = task["http"] if kind == "http" else task["tcp"][0] + label = ports[(task_name, "http" if kind == "http" else endpoint["name"])] + lines += [" service {", f" name = {q(name if task_name == 'app' else name + '-' + task_name)}", ' provider = "nomad"', f" port = {q(label)}"] + if len(data["containers"]) == 1: + lines.append(f" task = {q(task_name)}") + tags = [] + if kind == "http" and endpoint.get("metricsPath"): + tags.append("snowglobe-metrics-path=" + endpoint["metricsPath"]) + if kind == "http" and endpoint.get("hostname"): + tags += ["caddy-host=" + host for host in [endpoint["hostname"], *endpoint["plainHostnames"]]] + if endpoint.get("authRole"): + tags.append("caddy-auth-role=" + endpoint["authRole"]) + if endpoint.get("userHeader"): + tags.append("caddy-user-header=" + endpoint["userHeader"]) + if endpoint["forwardRealIp"]: + tags.append("caddy-real-ip=true") + if endpoint.get("secureCookies"): + tags.append("caddy-secure-cookies=true") + if endpoint["tlsInternal"]: + tags.append("caddy-internal=true") + if tags: + lines.append(f" tags = {q(tags)}") + lines += [" check {", f" type = {q(kind)}"] + if kind == "http": + lines.append(f" path = {q(endpoint['checkPath'])}") + if endpoint["checkHeaders"]: + lines.append(" header {") + for key, value in endpoint["checkHeaders"].items(): + if not re.fullmatch(r"[A-Za-z][A-Za-z0-9-]*", key): + raise ValueError(f"invalid health check header: {key}") + lines.append(f" {key} = [{q(value)}]") + lines.append(" }") + lines += [ + ' interval = "10s"', + ' timeout = "15s"', + ' check_restart {', + ' limit = 12', + f" grace = {q(data.get('healthRestartGrace') or data.get('healthyDeadline') or '5m')}", + ' }', + ' }', + ' }', + ] + for task_name, task in data["containers"].items(): + lines += [f" task {q(task_name)} {{", ' driver = "podman"'] + if task["lifecycle"] != "main": + lines += [' lifecycle {', ' hook = "prestart"', + f" sidecar = {str(task['lifecycle'] == 'prestartSidecar').lower()}", ' }'] + if not task["imageUser"]: + uses_clover = any( + volume.get("clover") or ( + volume.get("src") + and Path(volume["src"]).resolve().is_relative_to(Path(data["cloverRoot"]).resolve()) + ) + for volume in task["volumes"].values() + ) + gid = data["cloverGid"] if uses_clover else 0 if task["rootGroup"] else data["uid"] + lines.append(f" user = {q(str(data['uid']) + ':' + str(gid))}") + lines += [" config {", f" image = {q(task['image'])}"] + if task.get("entrypoint"): + lines.append(f" entrypoint = {q(task['entrypoint'])}") + if task["args"]: + lines.append(f" args = {q(task['args'])}") + if task["hostNetwork"]: + lines.append(' network_mode = "host"') + task_ports = [label for (owner, _), label in ports.items() if owner == task_name] + if task_ports and not task["hostNetwork"]: + lines.append(f" ports = {q(task_ports)}") + for field, values in (("cap_add", task["capAdd"]), ("devices", task["devices"]), ("extra_hosts", task["extraHosts"])): + if values: + lines.append(f" {field} = {q(values)}") + volumes = [] + secret_files = task.get("secretFiles", {}) + filenames = set() + for target, template in secret_files.items(): + filename = Path(target).name + if (not re.fullmatch(r"/[A-Za-z0-9_./-]+", target) or target.endswith("/") + or any(part in {".", "..", ""} for part in target[1:].split("/")) + or filename in filenames + or filename in {task_name + ".env", task_name + "-secret.env", "nomad_token"} + or target in task["volumes"] or target == "/secrets" or target.startswith("/secrets/")): + raise ValueError(f"invalid secret file target: {target}") + if task["imageUser"]: + raise ValueError("secret files require a service-owned container user") + if re.search(r"(?m)^EOF\r?$", template): + raise ValueError("invalid secret template delimiter") + filenames.add(filename) + volumes.append(f"secrets/{filename}:{target}:ro") + for target, volume in task["volumes"].items(): + if not target.startswith("/") or ":" in target: + raise ValueError(f"invalid volume target: {target}") + if volume.get("config") is not None: + if volume.get("src") is not None or not volume["readOnly"]: + raise ValueError("config mounts must be read only and have no host source") + config_path(data.get("sourceId", name), volume["config"]) + source = host_path(str(asset_dir), volume["config"]) + elif volume.get("src") is not None: + source = volume["src"] + if not Path(source).is_absolute(): + raise ValueError(f"volume source must be absolute: {source}") + else: + source = host_path(data["hostRoot"], target.lstrip("/")) + if ":" in source: + raise ValueError(f"invalid volume source: {source}") + volumes.append(f"{source}:{target}" + (":ro" if volume["readOnly"] else "")) + if volumes: + lines.append(f" volumes = {q(volumes)}") + if task["tmpfs"]: + lines.append(f" tmpfs = {q(task['tmpfs'])}") + lines.append(" }") + secret_env = [] + plain_env = {} + for key, value in task["env"].items(): + if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", key): + raise ValueError(f"invalid env key: {key}") + parts = re.split(r"(\$\{secret\.[A-Za-z_][A-Za-z0-9_]*\.[A-Za-z_][A-Za-z0-9_]*\})", value) + if len(parts) == 1: + if "${secret." in value: + raise ValueError(f"invalid secret reference: {value}") + plain_env[key] = value + continue + args = [] + paths = {} + for part in parts: + match = re.fullmatch(r"\$\{secret\.([A-Za-z_][A-Za-z0-9_]*)\.([A-Za-z_][A-Za-z0-9_]*)\}", part) + if match: + alias, field = match.groups() + if alias != "own" and alias not in data["inputs"]: + raise ValueError(f"unknown secret alias: {alias}") + path = f"nomad/jobs/{name}" + (f"/inputs/{alias}" if alias != "own" else "") + variable = paths.setdefault(path, f"$s{len(paths)}") + args.append(f"(index {variable} {q(field)}).Value") + elif part: + args.append(q(part)) + start = "".join(f'{{{{ with {variable} := nomadVar {q(path)} }}}}' for path, variable in paths.items()) + end = "{{ end }}" * len(paths) + secret_env.append(f'{start}{key}={{{{ print {" ".join(args)} | toJSON }}}}{end}') + if "SNOWGLOBE_PREPARE_DIGEST" in task["env"]: + raise ValueError("SNOWGLOBE_PREPARE_DIGEST is reserved") + lines.append(" env {") + for key, value in plain_env.items(): + lines.append(f" {key} = {q(value)}") + if prepare_digest: + lines.append(f" SNOWGLOBE_PREPARE_DIGEST = {q(prepare_digest)}") + lines.append(" }") + if secret_env: + lines += [" template {", " data = < 1: + raise ValueError(f"multiple stages exist for {args.name}; destroy one first") + created = not previous + stage = f"{args.name}-preview-{secrets.token_hex(4)}" if created else previous[0][0].stem + http = [task["http"] for task in data["containers"].values() if task.get("http") and task["http"].get("hostname")] + if len(http) > 1: + raise ValueError("preview requires at most one HTTP route") + original_host = http[0].get("hostname") if http else None + hostname = stage + "." + original_host.split(".", 1)[1] if original_host else None + data = load(args.name, properties, stage) + data["rollout"] = "simple" + data["hostRoot"] = str(Path(data["stagingRoot"]) / stage) + if data["stageIsolation"] == "fresh" and any( + volume.get("src") and not volume["readOnly"] + for task in data["containers"].values() + for volume in task["volumes"].values() + ): + raise ValueError("fresh stages cannot use writable external mounts") + for task in data["containers"].values(): + if task.get("tcp") and not task["hostNetwork"]: + for port in task["tcp"]: + port["hostPort"] = None + if hostname: + for task in data["containers"].values(): + if task.get("http") and task["http"].get("hostname") == original_host: + task["http"]["hostname"] = hostname + task["http"]["plainHostnames"] = [stage + "-" + host for host in task["http"]["plainHostnames"]] + task["env"] = {key: value.replace(original_host, hostname) for key, value in task["env"].items()} + if task.get("secretFiles"): + task["secretFiles"] = {key: value.replace(original_host, hostname) for key, value in task["secretFiles"].items()} + for request in data["inputs"].values(): + if request["provider"] == "snowglobe": + request["redirectUris"] = [uri.replace(original_host, hostname) for uri in request["redirectUris"]] + for assignment in args.env: + key, separator, value = assignment.partition("=") + if not separator or len(data["containers"]) != 1: + parser.error(f"unknown environment override: {key}") + env = next(iter(data["containers"].values()))["env"] + if key not in env: + parser.error(f"unknown environment override: {key}") + env[key] = value + metadata = ( + {"mount": data["hostRoot"], "inputs": data["inputs"], + "tasks": list(data["containers"]), "sourceId": args.name, + "stageIsolation": data["stageIsolation"]} + if created else previous[0][1] + ) + if not created: + if metadata.get("stageIsolation", "clone") != data["stageIsolation"]: + raise ValueError("stage isolation changed; destroy the stage before recreating it") + if bool(metadata.get("clone")) != data["hasManagedVolumes"]: + raise ValueError("storage layout changed; destroy the stage before recreating it") + for alias, request in data["inputs"].items(): + if alias in metadata["inputs"] and metadata["inputs"][alias] != request: + raise ValueError(f"requirement {alias} changed; destroy the stage before recreating it") + if alias not in metadata["inputs"] and request["provider"] == "postgres": + raise ValueError("new database requirement needs a new stage") + metadata["inputs"].update(data["inputs"]) + metadata["tasks"] = sorted(set(metadata["tasks"]) | set(data["containers"])) + stage_dir.mkdir(parents=True, exist_ok=True) + postgres_snapshot = None + pending_snapshots = set() + try: + postgres_inputs = created and data["stageIsolation"] == "clone" and any( + request["provider"] == "postgres" and request["kind"] == "database" + for request in data["inputs"].values() + ) + snapshots = [] + source = None + if created and data["hasManagedVolumes"]: + pool = data["pool"] + source = subprocess.run( + ["findmnt", "-n", "-o", "SOURCE", "--mountpoint", definitions[args.name]["hostRoot"]], + capture_output=True, text=True, + ).stdout.strip() + dataset = pool + "/staging/" + stage + if subprocess.run(["zfs", "list", pool + "/staging"], capture_output=True).returncode: + command("zfs", "create", "-o", "mountpoint=none", pool + "/staging") + if source and data["stageIsolation"] == "clone": + if source.split("/", 1)[0] != pool: + raise ValueError(f"production dataset belongs to another pool: {source}") + snapshot = source + "@" + stage + snapshots.append(snapshot) + if postgres_inputs: + postgres_dataset = dataset_for("postgres") + if not postgres_dataset: + raise ValueError("Postgres needs a ZFS dataset for consistent stages") + postgres_snapshot = postgres_dataset + "@" + stage + snapshots.append(postgres_snapshot) + external = {} + for task in data["containers"].values(): + for volume in task["volumes"].values(): + if volume.get("src") and not volume["readOnly"]: + external.setdefault(volume["src"], []).append(volume) + external_mounts = {} + for external_source in external: + path = Path(external_source).resolve(strict=True) + mount = json.loads(command( + "findmnt", "--json", "-o", "SOURCE,TARGET,FSTYPE,OPTIONS", "--target", str(path), capture=True, + ).stdout)["filesystems"][0] + if "ro" not in mount["options"].split(",") and mount["fstype"] != "zfs": + raise ValueError(f"writable external mount is not ZFS: {external_source}") + external_mounts[external_source] = (path, mount) + if created and mount["fstype"] == "zfs" and "ro" not in mount["options"].split(","): + target = mount["source"] + "@" + stage + if target not in snapshots: + snapshots.append(target) + if snapshots: + if len({target.split("/", 1)[0] for target in snapshots}) != 1: + raise ValueError("staged datasets must belong to one ZFS pool") + command("zfs", "snapshot", *snapshots) + pending_snapshots.update(snapshots) + if created and data["hasManagedVolumes"]: + if source and data["stageIsolation"] == "clone": + try: + clone_dataset(source, snapshot, dataset, data["hostRoot"]) + except Exception: + command("zfs", "destroy", snapshot) + pending_snapshots.remove(snapshot) + raise + metadata["snapshot"] = snapshot + pending_snapshots.remove(snapshot) + else: + command("zfs", "create", "-o", "mountpoint=" + data["hostRoot"], dataset) + metadata["clone"] = dataset + bindings = {} + for source, volumes in sorted(external.items()): + path, mount = external_mounts[source] + if "ro" in mount["options"].split(","): + binding = {"src": source, "readOnly": True} + elif mount["fstype"] == "zfs": + dataset = mount["source"] + if metadata.get("snapshot") == dataset + "@" + stage: + clone_mount = data["hostRoot"] + else: + staged = metadata.setdefault("external", {}) + if dataset not in staged: + if not created: + raise ValueError("external storage changed; destroy the stage before recreating it") + suffix = hashlib.sha256(dataset.encode()).hexdigest()[:8] + pool = dataset.split("/", 1)[0] + clone = pool + "/staging/" + stage + "-external-" + suffix + clone_mount = str(Path(data["stagingRoot"]) / (stage + "-external-" + suffix)) + snapshot = dataset + "@" + stage + if subprocess.run(["zfs", "list", pool + "/staging"], capture_output=True).returncode: + command("zfs", "create", "-o", "mountpoint=none", pool + "/staging") + try: + clone_dataset(dataset, snapshot, clone, clone_mount) + except Exception: + command("zfs", "destroy", snapshot) + pending_snapshots.remove(snapshot) + raise + staged[dataset] = {"clone": clone, "snapshot": snapshot, "mount": clone_mount} + pending_snapshots.remove(snapshot) + clone_mount = staged[dataset]["mount"] + relative = path.relative_to(Path(mount["target"]).resolve()) + binding = {"src": str(Path(clone_mount) / relative), "readOnly": False} + else: + raise ValueError(f"writable external mount is not ZFS: {source}") + bindings[source] = binding + for volume in volumes: + if path.is_relative_to(Path(data["cloverRoot"]).resolve()): + volume["clover"] = True + volume.update(binding) + if not created and bindings != metadata.get("externalSources", {}): + raise ValueError("external storage changed; destroy the stage before recreating it") + metadata["externalSources"] = bindings + if REPO.parent == Path("/opt/snowglobe/releases").resolve(): + metadata["release"] = REPO.name + metadata["ready"] = False + write_private(stage_dir / f"{stage}.json", json.dumps(metadata)) + bootstrap([data]) + if created and data["stageIsolation"] == "clone" and (data["secrets"] or data["requiredSecrets"]): + source = get_variable("nomad/jobs/" + args.name) + if source: + put_variable("nomad/jobs/" + stage, source["Items"], 0) + if postgres_snapshot: + postgres_clone = postgres_dataset.rsplit("/prod/", 1)[0] + "/staging/" + stage + "-postgres" + postgres_mount = Path(data["stagingRoot"]) / (stage + "-postgres") + source_context = cloned_postgres(postgres_snapshot, postgres_clone, postgres_mount) + else: + source_context = nullcontext(None) + with source_context as postgres_source: + provision_inputs(data, definitions, stage, postgres_source) + provision(data) + prepare(data) + build_images(data) + submit(data, "run", definitions) + except Exception: + if created: + destroy_stage(stage, metadata, definitions) + raise + finally: + for snapshot in sorted(pending_snapshots): + command("zfs", "destroy", snapshot) + if hostname: + check_http(hostname, http[0]["checkPath"], http[0]["tlsInternal"]) + configure(data) + if REPO.parent == Path("/opt/snowglobe/releases").resolve(): + metadata["ready"] = True + metadata["overrides"] = [assignment.partition("=")[0] for assignment in args.env] + write_private(stage_dir / f"{stage}.json", json.dumps(metadata)) + print(f"stage={stage}\nrelease={metadata.get('release', '')}\nhost={hostname}\ndataset={metadata.get('clone', '')}") + return + definitions = {name: load(name, properties) for name in services()} + if args.name and args.name not in definitions: + raise ValueError(f"unknown service: {args.name}") + if not args.name and args.mode == "deploy": + names = [name for name in names if definitions[name]["containers"]] + selected = set(names) + while True: + required = selected | set().union(*(dependencies(definitions[name]) for name in selected)) + missing = required - definitions.keys() + if missing: + raise ValueError(f"unknown dependency: {sorted(missing)}") + if required == selected: + break + selected = required + data = ordered([definitions[name] for name in sorted(selected)]) + if args.mode == "render": + for item in data: + if item["containers"]: + print(render(item, definitions)) + elif args.mode == "validate": + token() + for item in data: + submit(item, "validate", definitions) + elif args.mode == "check": + for item in data: + for task in item["containers"].values(): + if task.get("http") and task["http"].get("hostname"): + check_http(task["http"]["hostname"], task["http"]["checkPath"], task["http"]["tlsInternal"]) + elif args.mode == "preflight": + check_pool(data) + token() + missing_secrets = {} + for item in data: + if item["containers"] and item["requiredSecrets"]: + variable = get_variable("nomad/jobs/" + item["id"]) + values = variable["Items"] if variable else {} + missing = [name for name in item["requiredSecrets"] if not values.get(name)] + if missing: + missing_secrets[item["id"]] = missing + if missing_secrets: + raise ValueError("missing required secrets: " + "; ".join( + f"{name}: {', '.join(keys)}" for name, keys in missing_secrets.items())) + elif args.mode == "pool": + ensure_pool(data) + elif args.mode == "allocate": + ensure_pool(data) + bootstrap(data) + for item in data: + provision_inputs(item, definitions) + provision(item) + prepare(item) + build_images(item) + elif args.mode == "bootstrap": + bootstrap(data) + elif args.mode == "deploy": + bootstrap(data) + for item in data: + provision_inputs(item, definitions) + provision(item) + prepare(item) + build_images(item) + submit(item, "run", definitions) + configure(item) + + +if __name__ == "__main__": + try: + main() + except (OSError, RuntimeError, ValueError, subprocess.CalledProcessError) as error: + print(error, file=sys.stderr) + sys.exit(1) diff --git a/tools/source-of-truth-migration.md b/tools/source-of-truth-migration.md deleted file mode 100644 index ac633f440d9f0db4e4a8d2328a17a9f5a72b44ee..0000000000000000000000000000000000000000 --- a/tools/source-of-truth-migration.md +++ /dev/null @@ -1,19 +0,0 @@ -# Clover Source of Truth migration - -The existing preview can take a consistent copy of Zenith's live SQLite index without stopping the old service: - -```sh -sh tools/import-source-index.sh clover-source-of-truth-preview-6311ec97 -``` - -The script stops only that preview, snapshots its ZFS dataset, imports a SQLite `.backup`, checks integrity and checksums, then stages and checks the HTTP route. It retains the pre-import snapshot for recovery. It leaves the preview's own API key in place; the old key must be preserved for the production endpoint because existing clients send it as their authorization header. - -On 2026-09-26, Zenith's live index contained 2,592 media files, 136,842 derived file records, and 15,635 derived references. The imported preview passed SQLite integrity, reported the same counts, returned HTTP 200 at `clover-source-of-truth-preview-6311ec97.studio.test`, and had one healthy Nomad allocation. A nonexistent file returned 404. One indexed public raw file and its derived asset were copied from Zenith into the isolated preview, served over HTTPS with HTTP 200, and matched their source SHA-256 hashes; the test files were then removed. The full Published and `derived` trees remain absent from the VM. Zenith's derived tree holds 143,547 files totaling 39,684,661,494 bytes, beyond the demo pool's capacity. - -The `clo` account used by the stream importer had zero unreadable files and zero untraversable directories in the source tree at inspection. - -For the real cutover, keep `storage1/clover` mounted at `/srv/clover` and point the service's `/published` mount at `/srv/clover/Published`. That avoids copying Published at all. Stop the old source-of-truth container and the new Nomad job, then run `bash tools/import-source-data.sh` with `STUDIO_DEPLOY_HOST` and `STUDIO_DEPLOY_PORT` set for the new host while Zenith remains reachable over SSH. The importer checks both jobs, streams `Documents/Config/paper clover/` into the service's managed ZFS dataset without storing 40 GB on the Mac, compares content digests and SQLite integrity, and leaves Snow Globe stopped with its pre-import snapshot available. A small cross-host fixture verified the stream, including a symlink and hardlink; the full 40 GB transfer remains a production cutover operation. Later staging clones the managed service dataset with ZFS. - -If NixOS replaces Zenith on the same machine, stop the old service and take a named `storage1/clover` snapshot before reinstalling. After the pool is mounted at `/srv/clover`, set `STUDIO_MIGRATION_SNAPSHOT=` when running the same importer. It reads the immutable `/srv/clover/.zfs/snapshot//Documents/Config/paper clover/` tree on the new host and streams directly into the service dataset; no old Docker daemon or second machine is required. The importer rejects a missing or malformed snapshot name before changing the destination. - -Run `bash tools/import-legacy-secrets.sh clover-source-of-truth CLOVER_SOT_KEY` during cutover to transfer the old key directly from Zenith's `.env` into Snow Globe's Nomad secret, then deploy the service so its environment receives the imported value. This command changes the production service key; do not run it against the VM while its generated key is in use. Check a representative public raw and derived file over the new hostname before changing the public route. The old container and dataset snapshots remain available until this check passes. diff --git a/tools/storage-acl-cutover.md b/tools/storage-acl-cutover.md deleted file mode 100644 index e5235c80311e6255af31de5a88bd657327cee8b1..0000000000000000000000000000000000000000 --- a/tools/storage-acl-cutover.md +++ /dev/null @@ -1,49 +0,0 @@ -# Storage ACL cutover - -Zenith mounts `storage1/clover` and `storage1/media` with `nfs4acl`. Their NFSv4 ACLs are nontrivial: Clover grants `group:apps` and `user:1000`; Media grants `user:clo` and TrueNAS built-in groups. Media has 2,431 directories and 28,729 files. Of those, 543 directories and 12,885 files are `root:root` with mode `770`, so UID 3000 relies on the NFSv4 ACL to reach them. - -[Stock OpenZFS on Linux does not enforce NFSv4 ACLs](https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Datasets/ACLs.html). The NixOS VM accepted `acltype=nfsv4` on a disposable dataset but rejected `setfacl` and denied UID 3000 access to a root-owned `770` directory. Setting `acltype=posixacl` and granting UID 3000 access made the same check pass. A property value of `nfsv4` alone is therefore insufficient proof of access on NixOS. - -`tools/studio.py pool` rejects a mounted Clover or Media ZFS dataset with `acltype=nfsv4` before provisioning service volumes. It also rejects NFSv4 ACLs on the production or staging dataset parent, including values inherited from the pool; a disposable pool proved all three cases and accepted POSIX ACLs. Promotion runs the Clover/Media and production preflight before switching `/opt/studio/current`; a disposable NFSv4 dataset caused promotion to fail while its release link and history stayed unchanged. - -The media rename and mountpoint change in [media-cutover.md](media-cutover.md) must be paired with a permission migration. Rehearse it on copy-on-write clones of both datasets, leaving the originals and their snapshots intact. A shared numeric group `3000` with group read/write and setgid directories matches the VM's working Clover, Samba, and Copyparty permissions; a clone test must also verify media writers and readers before choosing that policy for the real tree. Changing `acltype` does not translate existing NFSv4 entries into POSIX ACLs, and [TrueNAS warns against recursive ACL changes without a snapshot](https://www.truenas.com/docs/scale/datasets/permissions/permissions/). - -A disposable POSIX-ACL ZFS fixture started with root-owned `0770` directories and a `0660` file that UID 3000 could not read. Granting named group 3000 access and a default ACL on directories, without changing owners, let UID 3000 and Jellyfin UID 3106 read the file. qBittorrent UID 3114 created a file that Copyparty UID 3116 could append to and UID 3000 could edit. The new file kept group 0 from its setgid parent, but inherited the named group 3000 ACL; the fixture was destroyed. This proves local service access on representative modes; SMB has a separate limitation below. - -A macOS SMB mount reached the VM over Tailscale and wrote a file as `3000:3000` with mode `0644`, despite Samba's `force create mode = 0660`; Copyparty's group identity could not edit it. A temporary Samba configuration with `acl_xattr` and `acl_xattr:ignore system acls = yes` yielded `0666`. Adding a parent default POSIX ACL (`user::rwx,group::rwx,other::---`) yielded files at `0660`, but Mac-created directories at `2777`. Adding `inherit permissions = yes` yielded directories at `2770` and files at `0660`; Copyparty's UID 3116/GID 3000 then appended to a Mac-created file and created a sibling in its directory. `inherit permissions` and the default ACL without `acl_xattr` instead yielded `0755` directories and `0644` files. That rehearsal introduced the ACL-xattr configuration superseded by the 2026-10-13 macOS fix below, while retaining the image's `catia fruit streams_xattr` Mac VFS modules. A disposable write through the staged Samba share again produced a `0660` file and `2770` directory, both writable by UID 3116/GID 3000; the fixture was removed. VM promotion `b1a2cdde0f6f12a4` passed all 19 HTTP routes and kept four PostgreSQL dumps in its pre-switch backup. The real dataset cutover still needs inherited POSIX ACLs applied through writable directories and verified on clones. [Samba documents the `acl_xattr` behavior](https://www.samba.org/samba/docs/current/man-html/vfs_acl_xattr.8.html). - -`acl_xattr:ignore system acls = yes` ignores named POSIX ACL grants for SMB permission checks. A staged Samba version without `acl_xattr` let a Linux SMB client write root-owned files through group ACLs, but a macOS mount then created `0644` files and `0755` directories that Copyparty could not edit. That version was rolled back. With the current Samba settings, a Mac mount created `0660` files and `2770` directories in a group-owned `3000` fixture, and Copyparty could append. The current configuration therefore needs group ownership as well as ACLs on a production clone rehearsal; the local UID test alone does not establish Finder access. - -A later disposable Samba test changed only `acl_xattr:ignore system acls` to `no`. In a root-owned, group-0 directory with a named/default POSIX ACL for group 3000, a macOS SMB mount created a file and directory, overwrote an existing root-owned file, and Copyparty UID 3116/GID 3000 appended to the new file. This worked both with and without `inherit permissions`. The new file and directory were `3000:3000` mode `0770`; the existing file retained root ownership. This offers a way to avoid changing ownership across the existing tree, but the executable bits on new regular files need review. The disposable dataset and container were removed, and the managed Samba job was restored without changing its configuration. - -An isolated Samba container on port 1445 was reachable from macOS through an SSH tunnel using `mount_smbfs //clo@127.0.0.1:1445/clover`; the managed share stayed online. In that fixture, disabling DOS archive/hidden/system mode mapping and then tightening `create mask` to `0660`, `directory mask` to `0770`, and `acl map full control` to `no` still produced `0770` Finder files and directories. Copyparty could append. The execute bits therefore appear to come from the ACL mapping in this combination, not the tested mode masks. The isolated container, dataset, and tunnel were removed. - -The same isolated setup without `acl_xattr` and with `inherit permissions = no` let Finder overwrite an existing root-owned file through its named ACL. New Finder files were `0644` and directories `0755` despite Samba's forced `0660`/`2770` modes, and the new file's inherited group ACL was masked to read-only; Copyparty could not append. That configuration cannot satisfy shared editing without a further permission mechanism. The fixture was removed. - -With Samba's default oplocks, a Mac SMB mount returned stale bytes and NULs immediately after Copyparty appended to a mounted file; remounting showed the correct server data. A disposable Samba container with `oplocks = no` returned the correct bytes immediately in the same test. The managed Samba job was restored afterward. Disabling oplocks may reduce SMB client caching performance and has not been promoted. - -Zenith's `storage1` pool root is unencrypted; `storage1/apps`, `storage1/clover`, and `storage1/media` are separate AES-256-GCM encryption roots with `keyformat=hex` and `keylocation=prompt`. Creating `storage1/prod` beneath the pool root without encryption would silently expose new app data. Snow Globe now requires an encrypted `prod` dataset mounted at `/srv/prod` whenever Clover or Media is encrypted. A disposable encrypted Clover dataset failed this preflight against the VM's unencrypted demo `prod`. A fresh disposable pool with separate encrypted Clover and `prod` roots passed: the `shale` service dataset inherited `prod` as its encryption root. The current VM predates the mounted-parent change and still has `studio-demo/prod` with `mountpoint=none`; its service children are mounted individually. - -The real host must load the selected keys before mounting datasets and starting Nomad; [loading a key does not itself mount the dataset](https://openzfs.github.io/openzfs-docs/man/v2.2/8/zfs-load-key.8.html). Keep keys out of the repository. A separate VM test confirmed that renaming one independent encrypted root beneath another preserves its independent encryption root. - -Copyparty's previous `df: 16` reserve rejected every upload on the 8 GB VM pool. Its prepared config now reserves one quarter of the dataset capacity, capped at the original 16 GB. The staged and production VM instances accepted a file and directory upload; the resulting `0664` file and setgid `2775` directory were writable by Clover's separate UID/GID 3000. Test files were removed, the Source of Truth preview was restarted, and the Copyparty preview was stopped to return the VM's reserved memory. Promotion `ec5322fadd8ae7a8` passed all 19 HTTP routes with 22 ZFS snapshots and four PostgreSQL dumps in its backup. - -Snow Globe's `zfs clone` of a disposable encrypted Media source stayed AES-256-GCM encrypted under an unencrypted staging parent and shared the Media encryption root; writes to the clone left the source unchanged. [OpenZFS specifies that clones always share their origin's encryption key](https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html). Fresh stages have no encrypted origin to inherit, so `tools/studio.py stage` now requires an encrypted `pool/staging` mounted at `/srv/staging` when `pool/prod` is encrypted. A disposable pool proved missing and unencrypted staging parents are rejected, while an encrypted parent passes. The VM's existing unencrypted pool still stages normally. - -`storage1/apps` is one encrypted dataset with no child datasets: 47.8 GiB live and 65.3 GiB held by 207 snapshots at inspection. It also contains stacks excluded from Snow Globe. Renaming it to `storage1/prod` would preserve that history but leave the service folders as ordinary directories; moving selected data into per-service child datasets would still be necessary. Creating a separate encrypted `prod` root keeps the old tree and its snapshots available during migration. - -## macOS saves — 2026-10-13 - -Zenith ran Samba 4.23.8 with `acl_xattr` but without `CAP_SYS_ADMIN`. On macOS 15.7.5, ordinary writes succeeded while `NSFileManager.replaceItemAt`, `NSDocument.writeSafely`, package replacement, and `chmod` failed with `EACCES`. The ACL module could not store protected `security.NTACL` attributes; [Samba documents that capability requirement](https://gitlab.com/samba-team/samba/-/merge_requests/3501). Only generated disposable files were opened during these probes. - -The [service declaration](../service/samba/service.pkl) now uses POSIX permissions directly. Disabling NFS ACE mode updates keeps Mac clients from bypassing the creation masks. Removing inherited mode handling leaves new regular files at `0660` and directories at `2770`; UID 3116/GID 3000 can edit them. A capability-only alternative fixed Cocoa saving but produced `0644` files that the shared group could not edit. Allowing AppleDouble names also fixes client temporary files and resource-fork copies; [Samba warns that vetoing these names breaks archive tools](https://www.samba.org/samba/docs/current/man-html/vfs_fruit.8.html). - -[The macOS regression probe](samba-macos-test.swift) creates and removes its own directory: `swift tools/samba-macos-test.swift `. Its nine checks cover atomic writes, permission-preserving replacement, Cocoa Save As/save/autosave, RTFD packages, metadata and 128 KiB resource-fork copies, AppleDouble filenames, permission updates, replacement with an open reader, and cleanup. All passed in an isolated local Linux rehearsal and Zenith's ZFS preview `samba-preview-84726d65`. The preview used disposable authentication. Files owned by root and by UID 3116 were also editable through Cocoa in the local rehearsal. Blender 5.2 saved an asset and backup, linked it into a project, reopened the project, and read an external PNG from the share. - -Kernel oplocks make native service overwrites and appends invalidate SMB caches. The default SMB leases returned stale data even after reopening a file; the revised configuration returned the updated bytes immediately, including service-owned files. UID 3000 granted a kernel read lease on a disposable Zenith ZFS file without extra capabilities. An external atomic replacement retains the old inode while a Mac reader remains open; closing all old handles makes subsequent readers see the replacement. Disabling oplocks entirely showed the same Mac behavior, so the service retains kernel-backed caching. - -The live connection negotiated 1 Gb Ethernet. A disposable 128 MiB synthetic file measured 65.3 MiB/s writes with fsync and 76.6 MiB/s reads with the Mac data cache disabled and the server cache warm. Among 512 random 64 KiB reads, median/p95/maximum latency was 1.23/1.53/178.92 ms. These numbers do not establish real-time sample-library playback. Zenith has four 8 TB WD80EFPX HDDs in one RAIDZ1 vdev with 128 KiB records, a gigabit RTL8168 NIC, and 128 GB RAM; the Mac's unused built-in port supports 10 Gb. [RAIDZ streaming bandwidth scales across data disks, but default-record random-read IOPS are roughly one disk's](https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Pool%20Structure/RAIDZ.html). A cable cannot remove the NAS NIC's speed limit. - -REAPER supports VST3, while macOS VST3 files are executable bundles ([REAPER](https://www.reaper.fm/about.php), [Steinberg](https://steinbergmedia.github.io/vst3_dev_portal/pages/Technical%2BDocumentation/Locations%2BFormat/Plugin%2BFormat.html)). Actual vendor plug-ins and sample libraries need their own playback trial. [Kontakt streams samples during playback](https://support.native-instruments.com/support/solutions/articles/69000879768-what-is-dfd-), and [NI advises against active libraries on NAS](https://support.native-instruments.com/support/solutions/articles/69000879313-notes-on-network-drives-and-disk-formats). No existing plug-in or sample-library contents were opened or moved during this work. - -Production release `94c1aa0a1f6a4a00` activated on 2026-10-14 at 05:45 UTC from main `f9c3a85e`, with pre-switch backup `20261014T054319Z-54be52`. All 22 HTTP routes and the Samba allocation were healthy. The live Mac mount passed all nine regression checks, shared-group writes became visible immediately, and Blender passed the asset round trip. The old mount disconnected during activation; a background NetFS reconnect restored `/Volumes/clover` using saved credentials. Generated files and the ZFS preview were removed afterward. diff --git a/tools/studio-secret-files-test.py b/tools/studio-secret-files-test.py deleted file mode 100644 index 613cad961bd91717aab5fa2b4199be69a3fa85d0..0000000000000000000000000000000000000000 --- a/tools/studio-secret-files-test.py +++ /dev/null @@ -1,72 +0,0 @@ -#!/usr/bin/env python3 -import json -import os -from pathlib import Path -import subprocess -import tempfile -import unittest - -import studio - - -class SecretFiles(unittest.TestCase): - @classmethod - def setUpClass(cls): - with tempfile.TemporaryDirectory(prefix="studio-secret-template-") as temporary: - fixture = Path(temporary) / "fixture.pkl" - fixture.write_text('amends ' + json.dumps((studio.REPO / "config/Service.pkl").as_uri()) + '\nmeta { name = "Fixture" }\ncontainer { image = "localhost/fixture:pinned" }\n') - result = subprocess.run([os.environ.get("PKL", "pkl"), "eval", "-p", "uid=3132", "-p", "serviceId=fixture", str(fixture)], check=True, capture_output=True, text=True) - cls.fixture = json.loads(result.stdout) - cls.fixture.update(uid=3132, sourceId="fixture", inputs={}) - cls.fixture.pop("requirements") - - def setUp(self): - self.data = json.loads(json.dumps(self.fixture)) - self.task = self.data["containers"]["app"] - self.task["secretFiles"] = {"/run/secrets/oidc.json": '{{ with nomadVar "nomad/jobs/fixture/inputs/oidc" }}{"clientSecret":{{ .clientSecret.Value | toJSON }}}{{ end }}'} - - def test_private_allocation_template_and_read_only_file_mount(self): - rendered = studio.render(self.data, {"fixture": self.data}) - self.assertIn('"secrets/oidc.json:/run/secrets/oidc.json:ro"', rendered) - self.assertIn('destination = "secrets/oidc.json"', rendered) - self.assertIn('perms = "0400"', rendered) - self.assertIn('uid = 3132', rendered) - self.assertIn('gid = 3132', rendered) - self.assertIn('error_on_missing_key = true', rendered) - self.assertIn(self.task["secretFiles"]["/run/secrets/oidc.json"], rendered) - self.assertNotIn('env = true', rendered) - self.assertNotIn('change_mode =', rendered) - - def test_services_without_files_gain_no_template_or_mount(self): - self.task["secretFiles"] = {} - rendered = studio.render(self.data, {"fixture": self.data}) - self.assertNotIn('template {', rendered) - self.assertNotIn('secrets/', rendered) - - def test_unsafe_and_colliding_targets_are_rejected(self): - for target in ("relative.json", "/run/../secret.json", "/run/./secret.json", "/run//secret.json", "/run/secrets/", "/secrets/file", "/secrets", "/run/app.env", "/run/app-secret.env", "/run/nomad_token", "/run/line\nbreak"): - with self.subTest(target=target): - self.task["secretFiles"] = {target: "{}"} - with self.assertRaisesRegex(ValueError, "invalid secret file target"): - studio.render(self.data, {"fixture": self.data}) - self.task["secretFiles"] = {"/one/config.json": "{}", "/two/config.json": "{}"} - with self.assertRaisesRegex(ValueError, "invalid secret file target"): - studio.render(self.data, {"fixture": self.data}) - self.task["secretFiles"] = {"/run/secrets/oidc.json": "{}"} - self.task["volumes"] = {"/run/secrets/oidc.json": {}} - with self.assertRaisesRegex(ValueError, "invalid secret file target"): - studio.render(self.data, {"fixture": self.data}) - - def test_image_user_and_heredoc_injection_are_rejected(self): - self.task["imageUser"] = True - with self.assertRaisesRegex(ValueError, "service-owned"): - studio.render(self.data, {"fixture": self.data}) - self.task["imageUser"] = False - for body in ("EOF", "{}\nEOF\nmalicious", "{}\nEOF\r\nmalicious"): - self.task["secretFiles"] = {"/run/secrets/oidc.json": body} - with self.assertRaisesRegex(ValueError, "invalid secret template delimiter"): - studio.render(self.data, {"fixture": self.data}) - - -if __name__ == "__main__": - unittest.main() diff --git a/tools/studio.py b/tools/studio.py deleted file mode 100644 index 55477e09d230c8a1c19eaaabd835043efdc80510..0000000000000000000000000000000000000000 --- a/tools/studio.py +++ /dev/null @@ -1,1286 +0,0 @@ -#!/usr/bin/env python3 -import argparse -from contextlib import nullcontext -import fcntl -import hashlib -import json -import os -from pathlib import Path -import re -import secrets -import shutil -import subprocess -import sys -import tempfile -import time - -from data import cloned_postgres, dataset_for -from release import excluded_services - - -REPO = Path(__file__).resolve().parent.parent -SERVICES = REPO / "service" -STATE = Path("/var/lib/studio") -IDENTITIES = STATE / "identities.json" -ASSETS = Path("/var/lib/caddy/studio") -NAME = re.compile(r"[a-z][a-z0-9-]*\Z") - - -def command(*args, capture=False, **kwargs): - return subprocess.run(args, check=True, text=True, capture_output=capture, **kwargs) - - -def service_files(): - excluded = excluded_services(REPO) - files = {} - for directory in SERVICES.iterdir(): - if directory.name in excluded or not directory.is_dir(): - continue - for path in directory.glob("*.pkl"): - name = directory.name if path.name == "service.pkl" else path.stem - if name in excluded: - continue - if name in files: - raise ValueError(f"duplicate service definition: {name}") - files[name] = path - return files - - -def services(): - return sorted(service_files()) - - -def service_dir(name): - path = service_files()[name] - return path.parent if path.name == "service.pkl" else path.parent / name - - -def dependencies(data): - return set(data["dependsOn"]) | {item["provider"] for item in data["inputs"].values() if item["provider"] != "snowglobe"} - - -def ordered(data): - pending = {item["id"]: item for item in data} - result = [] - while pending: - ready = [ - item - for item in pending.values() - if not dependencies(item) & pending.keys() - ] - if not ready: - raise ValueError("cycle in service dependencies") - for item in ready: - result.append(item) - del pending[item["id"]] - return result - - -def identity(name): - STATE.mkdir(parents=True, exist_ok=True) - if not IDENTITIES.exists(): - write_private(IDENTITIES, (REPO / "config/identities.json").read_text()) - with (IDENTITIES.parent / ".identities.lock").open("w") as lock: - fcntl.flock(lock, fcntl.LOCK_EX) - with IDENTITIES.open() as file: - ids = json.load(file) - if len(set(ids.values())) != len(ids): - raise ValueError("duplicate service UIDs") - if name not in ids: - used = set(ids.values()) - ids[name] = next(uid for uid in range(3100, 60000) if uid not in used) - pending = IDENTITIES.with_suffix(".pending") - pending.write_text(json.dumps(ids, indent=2) + "\n") - pending.replace(IDENTITIES) - return ids[name] - - -def load(name, properties, instance_id=None): - files = service_files() - if not NAME.fullmatch(name) or name not in files: - raise ValueError(f"unknown service: {name}") - instance_id = instance_id or name - if not NAME.fullmatch(instance_id): - raise ValueError(f"invalid instance ID: {instance_id}") - uid = identity(name) - result = command( - "pkl", - "eval", - *( - part - for key, value in {**properties, "serviceId": instance_id, "uid": uid, "preview": str(instance_id != name).lower()}.items() - for part in ("-p", f"{key}={value}") - ), - str(files[name]), - capture=True, - ) - data = json.loads(result.stdout) - if data["id"] != instance_id: - raise ValueError(f"service ID must match deployment: {instance_id}") - inputs = {} - for requirement in data.pop("requirements"): - alias = requirement["alias"] - if not NAME.fullmatch(alias) or alias == "own" or alias in inputs: - raise ValueError(f"invalid or duplicate requirement alias: {alias}") - inputs[alias] = requirement - data["inputs"] = inputs - data["uid"] = uid - data["sourceId"] = name - for task_name, task in data["containers"].items(): - if bool(task.get("image")) == bool(task.get("build")): - raise ValueError(f"{name}.{task_name} needs one image or build directory") - if task.get("build"): - if REPO.parent == Path("/opt/studio/releases"): - context = config_path(name, task["build"]) - if not context.is_dir() or not (context / "Dockerfile").is_file(): - raise ValueError(f"invalid build context: {name}.{task_name}") - digest = hashlib.sha256() - for path in sorted(context.rglob("*")): - if path.is_file(): - digest.update(str(path.relative_to(context)).encode() + b"\0") - digest.update(path.read_bytes()) - tag = digest.hexdigest()[:16] - elif not (service_dir(name) / task["build"] / "Dockerfile").is_file() and not (service_dir(name) / "build-source.json").is_file(): - raise ValueError(f"missing build source: {name}.{task_name}") - else: - tag = REPO.name - task["image"] = f"localhost/studio/{name}-{task_name}:{tag}" - return data - - -def q(value): - return json.dumps(value, ensure_ascii=False) - - -def write_private(path, content): - pending = None - try: - with tempfile.NamedTemporaryFile("w", dir=path.parent, delete=False) as file: - pending = Path(file.name) - file.write(content) - pending.replace(path) - finally: - if pending: - pending.unlink(missing_ok=True) - - -def host_path(root, relative): - if relative == ".": - return root - part = Path(relative) - if part.is_absolute() or ".." in part.parts or not part.parts: - raise ValueError(f"invalid relative path: {relative}") - return str(Path(root) / part) - - -def config_path(service, relative): - root = service_dir(service).resolve() - source = (root / relative).resolve() - if not source.is_relative_to(root) or not source.exists(): - raise ValueError(f"invalid config path: {relative}") - return source - - -def config_assets(data): - paths = set() - for task in data["containers"].values(): - paths.update(volume["config"] for volume in task["volumes"].values() if volume.get("config")) - if task.get("http"): - paths.update(task["http"]["overrideFiles"].values()) - sources = {rel: config_path(data.get("sourceId", data["id"]), rel) for rel in paths} - digest = hashlib.sha256() - for rel, source in sorted(sources.items()): - digest.update(rel.encode()) - files = sorted(source.rglob("*")) if source.is_dir() else [source] - for file in files: - if file.is_symlink(): - raise ValueError(f"config asset cannot be a symlink: {file}") - if file.is_file(): - digest.update(str(Path(rel) / file.relative_to(source)).encode() if source.is_dir() else rel.encode()) - digest.update(file.read_bytes()) - return ASSETS / data["id"] / digest.hexdigest()[:16], sources - - -def render(data, definitions): - if not data["containers"]: - return "" - name = data["id"] - asset_dir, _ = config_assets(data) - prepare_digest = None - if data.get("prepare"): - source = service_dir(data.get("sourceId", name)) - digest = hashlib.sha256() - for path in sorted(source.rglob("*")): - if not path.is_file() or "__pycache__" in path.parts or path.suffix == ".pyc": - continue - if path.name == "service.pkl" or path.name.startswith("icon"): - continue - digest.update(str(path.relative_to(source)).encode() + b"\0") - digest.update(path.read_bytes()) - prepare_digest = digest.hexdigest()[:16] - routed = [task["http"] for task in data["containers"].values() - if task.get("http") and task["http"].get("hostname")] - primary = routed[0] if routed else {} - requirements = {provider: "startup" for provider in data["dependsOn"]} - requirements.update({item["provider"]: item["kind"] for item in data["inputs"].values()}) - secrets_meta = [ - {"name": name, "generated": True, "bytes": spec["bytes"]} - for name, spec in data["secrets"].items() - ] + [{"name": name, "generated": False} for name in data["requiredSecrets"]] - lines = [ - f"job {q(name)} {{", - ' datacenters = ["clover"]', - ' type = "service"', - ' meta {', - f" studio_service = {q(data['sourceId'])}", - f" studio_name = {q(data['name'])}", - f" studio_tagline = {q(data['tagline'])}", - f" studio_hostname = {q(primary.get('hostname') or '')}", - f" studio_launcher = {q(str(data['launcher']).lower())}", - f" studio_auth_role = {q(data.get('access') or primary.get('authRole') or '')}", - f" studio_trace_service = {q(data.get('traceServiceName') or '')}", - f" studio_metrics_pushed = {q(str(data['metricsPushed']).lower())}", - f" studio_requires = {q(json.dumps(requirements))}", - f" studio_secrets = {q(json.dumps(secrets_meta))}", - ' }', - ' group "app" {', - ] - if dependencies(data): - lines += [ - " restart {", - " attempts = 3", - ' delay = "1m"', - ' interval = "4m"', - ' mode = "delay"', - " }", - ] - if data["rollout"] == "overlapped": - for task in data["containers"].values(): - if task["hostNetwork"] or any( - endpoint and endpoint.get("hostPort") is not None - for endpoint in (task.get("http"), *task["tcp"]) - ) or any(not volume["readOnly"] for volume in task["volumes"].values()): - raise ValueError(f"overlapped rollout requires dynamic ports and read-only mounts: {name}") - elif data["rollout"] != "simple": - raise ValueError(f"unknown rollout: {data['rollout']}") - if data["rollout"] == "overlapped" or data.get("healthyDeadline"): - lines.append(" update {") - if data["rollout"] == "overlapped": - lines += [" canary = 1", " auto_promote = true", " auto_revert = true"] - if data.get("healthyDeadline"): - lines += [f" healthy_deadline = {q(data['healthyDeadline'])}", ' progress_deadline = "0"'] - lines.append(" }") - if data.get("healthyDeadline"): - lines += [" migrate {", f" healthy_deadline = {q(data['healthyDeadline'])}", " }"] - ports = {} - network = [] - host_network = any(task["hostNetwork"] for task in data["containers"].values()) - if host_network and not all(task["hostNetwork"] for task in data["containers"].values()): - raise ValueError("mixed host and bridge networking in one group") - for task_name, task in data["containers"].items(): - for kind, endpoint in [("http", task.get("http")), *((port["name"], port) for port in task["tcp"])]: - if endpoint: - label = kind if len(data["containers"]) == 1 else f"{task_name}-{kind}" - if label in ports.values(): - raise ValueError(f"duplicate port label: {label}") - ports[(task_name, kind)] = label - network.append(f" port {q(label)} {{") - if endpoint.get("hostPort") is not None: - network.append(f" static = {endpoint['hostPort']}") - if not task["hostNetwork"]: - network.append(f" to = {endpoint['containerPort']}") - if endpoint.get("loopback") or (not task["hostNetwork"] and kind == "http" and endpoint.get("hostPort") is None): - network.append(' host_network = "loopback"') - network.append(" }") - if network: - lines.append(" network {") - if host_network: - lines.append(' mode = "host"') - lines += network + [" }"] - for provider_id in sorted(dependencies(data)): - provider = definitions[provider_id] - for provider_task, container in provider["containers"].items(): - endpoint = container.get("http") - if not endpoint or not endpoint.get("hostname") or endpoint.get("authRole"): - continue - host = endpoint["hostname"] - task_name = f"wait-{provider_id}-{provider_task}" - lines += [ - f" task {q(task_name)} {{", - ' driver = "podman"', - " lifecycle {", - ' hook = "prestart"', - " sidecar = false", - " }", - " config {", - ' image = "docker.io/curlimages/curl@sha256:43ebaa53d3806db6b1ce4353b6b26ae638ec1c167ee351524b05690f988bb20d"', - f" extra_hosts = {q([host + ':host-gateway'])}", - f" args = {q(['--insecure', '--fail', '--silent', '--show-error', '--retry', '999999', '--retry-all-errors', '--retry-delay', '5', '--connect-timeout', '3', '--max-time', '5', 'https://' + host + endpoint['checkPath']])}", - " }", - " resources {", - " cpu = 50", - " memory = 64", - " }", - " }", - ] - for task_name, task in data["containers"].items(): - if task.get("http") or task.get("tcp"): - kind = "http" if task.get("http") else "tcp" - endpoint = task["http"] if kind == "http" else task["tcp"][0] - label = ports[(task_name, "http" if kind == "http" else endpoint["name"])] - lines += [" service {", f" name = {q(name if task_name == 'app' else name + '-' + task_name)}", ' provider = "nomad"', f" port = {q(label)}"] - if len(data["containers"]) == 1: - lines.append(f" task = {q(task_name)}") - tags = [] - if kind == "http" and endpoint.get("metricsPath"): - tags.append("studio-metrics-path=" + endpoint["metricsPath"]) - if kind == "http" and endpoint.get("hostname"): - tags += ["caddy-host=" + host for host in [endpoint["hostname"], *endpoint["plainHostnames"]]] - if endpoint.get("authRole"): - tags.append("caddy-auth-role=" + endpoint["authRole"]) - if endpoint.get("userHeader"): - tags.append("caddy-user-header=" + endpoint["userHeader"]) - if endpoint["forwardRealIp"]: - tags.append("caddy-real-ip=true") - if endpoint.get("secureCookies"): - tags.append("caddy-secure-cookies=true") - if endpoint["tlsInternal"]: - tags.append("caddy-internal=true") - if tags: - lines.append(f" tags = {q(tags)}") - lines += [" check {", f" type = {q(kind)}"] - if kind == "http": - lines.append(f" path = {q(endpoint['checkPath'])}") - if endpoint["checkHeaders"]: - lines.append(" header {") - for key, value in endpoint["checkHeaders"].items(): - if not re.fullmatch(r"[A-Za-z][A-Za-z0-9-]*", key): - raise ValueError(f"invalid health check header: {key}") - lines.append(f" {key} = [{q(value)}]") - lines.append(" }") - lines += [ - ' interval = "10s"', - ' timeout = "15s"', - ' check_restart {', - ' limit = 12', - f" grace = {q(data.get('healthRestartGrace') or data.get('healthyDeadline') or '5m')}", - ' }', - ' }', - ' }', - ] - for task_name, task in data["containers"].items(): - lines += [f" task {q(task_name)} {{", ' driver = "podman"'] - if task["lifecycle"] != "main": - lines += [' lifecycle {', ' hook = "prestart"', - f" sidecar = {str(task['lifecycle'] == 'prestartSidecar').lower()}", ' }'] - if not task["imageUser"]: - uses_clover = any( - volume.get("clover") or ( - volume.get("src") - and Path(volume["src"]).resolve().is_relative_to(Path(data["cloverRoot"]).resolve()) - ) - for volume in task["volumes"].values() - ) - gid = data["cloverGid"] if uses_clover else 0 if task["rootGroup"] else data["uid"] - lines.append(f" user = {q(str(data['uid']) + ':' + str(gid))}") - lines += [" config {", f" image = {q(task['image'])}"] - if task.get("entrypoint"): - lines.append(f" entrypoint = {q(task['entrypoint'])}") - if task["args"]: - lines.append(f" args = {q(task['args'])}") - if task["hostNetwork"]: - lines.append(' network_mode = "host"') - task_ports = [label for (owner, _), label in ports.items() if owner == task_name] - if task_ports and not task["hostNetwork"]: - lines.append(f" ports = {q(task_ports)}") - for field, values in (("cap_add", task["capAdd"]), ("devices", task["devices"]), ("extra_hosts", task["extraHosts"])): - if values: - lines.append(f" {field} = {q(values)}") - volumes = [] - secret_files = task.get("secretFiles", {}) - filenames = set() - for target, template in secret_files.items(): - filename = Path(target).name - if (not re.fullmatch(r"/[A-Za-z0-9_./-]+", target) or target.endswith("/") - or any(part in {".", "..", ""} for part in target[1:].split("/")) - or filename in filenames - or filename in {task_name + ".env", task_name + "-secret.env", "nomad_token"} - or target in task["volumes"] or target == "/secrets" or target.startswith("/secrets/")): - raise ValueError(f"invalid secret file target: {target}") - if task["imageUser"]: - raise ValueError("secret files require a service-owned container user") - if re.search(r"(?m)^EOF\r?$", template): - raise ValueError("invalid secret template delimiter") - filenames.add(filename) - volumes.append(f"secrets/{filename}:{target}:ro") - for target, volume in task["volumes"].items(): - if not target.startswith("/") or ":" in target: - raise ValueError(f"invalid volume target: {target}") - if volume.get("config") is not None: - if volume.get("src") is not None or not volume["readOnly"]: - raise ValueError("config mounts must be read only and have no host source") - config_path(data.get("sourceId", name), volume["config"]) - source = host_path(str(asset_dir), volume["config"]) - elif volume.get("src") is not None: - source = volume["src"] - if not Path(source).is_absolute(): - raise ValueError(f"volume source must be absolute: {source}") - else: - source = host_path(data["hostRoot"], target.lstrip("/")) - if ":" in source: - raise ValueError(f"invalid volume source: {source}") - volumes.append(f"{source}:{target}" + (":ro" if volume["readOnly"] else "")) - if volumes: - lines.append(f" volumes = {q(volumes)}") - if task["tmpfs"]: - lines.append(f" tmpfs = {q(task['tmpfs'])}") - lines.append(" }") - secret_env = [] - plain_env = {} - for key, value in task["env"].items(): - if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", key): - raise ValueError(f"invalid env key: {key}") - parts = re.split(r"(\$\{secret\.[A-Za-z_][A-Za-z0-9_]*\.[A-Za-z_][A-Za-z0-9_]*\})", value) - if len(parts) == 1: - if "${secret." in value: - raise ValueError(f"invalid secret reference: {value}") - plain_env[key] = value - continue - args = [] - paths = {} - for part in parts: - match = re.fullmatch(r"\$\{secret\.([A-Za-z_][A-Za-z0-9_]*)\.([A-Za-z_][A-Za-z0-9_]*)\}", part) - if match: - alias, field = match.groups() - if alias != "own" and alias not in data["inputs"]: - raise ValueError(f"unknown secret alias: {alias}") - path = f"nomad/jobs/{name}" + (f"/inputs/{alias}" if alias != "own" else "") - variable = paths.setdefault(path, f"$s{len(paths)}") - args.append(f"(index {variable} {q(field)}).Value") - elif part: - args.append(q(part)) - start = "".join(f'{{{{ with {variable} := nomadVar {q(path)} }}}}' for path, variable in paths.items()) - end = "{{ end }}" * len(paths) - secret_env.append(f'{start}{key}={{{{ print {" ".join(args)} | toJSON }}}}{end}') - if "STUDIO_PREPARE_DIGEST" in task["env"]: - raise ValueError("STUDIO_PREPARE_DIGEST is reserved") - lines.append(" env {") - for key, value in plain_env.items(): - lines.append(f" {key} = {q(value)}") - if prepare_digest: - lines.append(f" STUDIO_PREPARE_DIGEST = {q(prepare_digest)}") - lines.append(" }") - if secret_env: - lines += [" template {", " data = < 1: - raise ValueError(f"multiple stages exist for {args.name}; destroy one first") - created = not previous - stage = f"{args.name}-preview-{secrets.token_hex(4)}" if created else previous[0][0].stem - http = [task["http"] for task in data["containers"].values() if task.get("http") and task["http"].get("hostname")] - if len(http) > 1: - raise ValueError("preview requires at most one HTTP route") - original_host = http[0].get("hostname") if http else None - hostname = stage + "." + original_host.split(".", 1)[1] if original_host else None - data = load(args.name, properties, stage) - data["rollout"] = "simple" - data["hostRoot"] = str(Path(data["stagingRoot"]) / stage) - if data["stageIsolation"] == "fresh" and any( - volume.get("src") and not volume["readOnly"] - for task in data["containers"].values() - for volume in task["volumes"].values() - ): - raise ValueError("fresh stages cannot use writable external mounts") - for task in data["containers"].values(): - if task.get("tcp") and not task["hostNetwork"]: - for port in task["tcp"]: - port["hostPort"] = None - if hostname: - for task in data["containers"].values(): - if task.get("http") and task["http"].get("hostname") == original_host: - task["http"]["hostname"] = hostname - task["http"]["plainHostnames"] = [stage + "-" + host for host in task["http"]["plainHostnames"]] - task["env"] = {key: value.replace(original_host, hostname) for key, value in task["env"].items()} - if task.get("secretFiles"): - task["secretFiles"] = {key: value.replace(original_host, hostname) for key, value in task["secretFiles"].items()} - for request in data["inputs"].values(): - if request["provider"] == "snowglobe": - request["redirectUris"] = [uri.replace(original_host, hostname) for uri in request["redirectUris"]] - for assignment in args.env: - key, separator, value = assignment.partition("=") - if not separator or len(data["containers"]) != 1: - parser.error(f"unknown environment override: {key}") - env = next(iter(data["containers"].values()))["env"] - if key not in env: - parser.error(f"unknown environment override: {key}") - env[key] = value - metadata = ( - {"mount": data["hostRoot"], "inputs": data["inputs"], - "tasks": list(data["containers"]), "sourceId": args.name, - "stageIsolation": data["stageIsolation"]} - if created else previous[0][1] - ) - if not created: - if metadata.get("stageIsolation", "clone") != data["stageIsolation"]: - raise ValueError("stage isolation changed; destroy the stage before recreating it") - if bool(metadata.get("clone")) != data["hasManagedVolumes"]: - raise ValueError("storage layout changed; destroy the stage before recreating it") - for alias, request in data["inputs"].items(): - if alias in metadata["inputs"] and metadata["inputs"][alias] != request: - raise ValueError(f"requirement {alias} changed; destroy the stage before recreating it") - if alias not in metadata["inputs"] and request["provider"] == "postgres": - raise ValueError("new database requirement needs a new stage") - metadata["inputs"].update(data["inputs"]) - metadata["tasks"] = sorted(set(metadata["tasks"]) | set(data["containers"])) - stage_dir.mkdir(parents=True, exist_ok=True) - postgres_snapshot = None - pending_snapshots = set() - try: - postgres_inputs = created and data["stageIsolation"] == "clone" and any( - request["provider"] == "postgres" and request["kind"] == "database" - for request in data["inputs"].values() - ) - snapshots = [] - source = None - if created and data["hasManagedVolumes"]: - pool = data["pool"] - source = subprocess.run( - ["findmnt", "-n", "-o", "SOURCE", "--mountpoint", definitions[args.name]["hostRoot"]], - capture_output=True, text=True, - ).stdout.strip() - dataset = pool + "/staging/" + stage - if subprocess.run(["zfs", "list", pool + "/staging"], capture_output=True).returncode: - command("zfs", "create", "-o", "mountpoint=none", pool + "/staging") - if source and data["stageIsolation"] == "clone": - if source.split("/", 1)[0] != pool: - raise ValueError(f"production dataset belongs to another pool: {source}") - snapshot = source + "@" + stage - snapshots.append(snapshot) - if postgres_inputs: - postgres_dataset = dataset_for("postgres") - if not postgres_dataset: - raise ValueError("Postgres needs a ZFS dataset for consistent stages") - postgres_snapshot = postgres_dataset + "@" + stage - snapshots.append(postgres_snapshot) - external = {} - for task in data["containers"].values(): - for volume in task["volumes"].values(): - if volume.get("src") and not volume["readOnly"]: - external.setdefault(volume["src"], []).append(volume) - external_mounts = {} - for external_source in external: - path = Path(external_source).resolve(strict=True) - mount = json.loads(command( - "findmnt", "--json", "-o", "SOURCE,TARGET,FSTYPE,OPTIONS", "--target", str(path), capture=True, - ).stdout)["filesystems"][0] - if "ro" not in mount["options"].split(",") and mount["fstype"] != "zfs": - raise ValueError(f"writable external mount is not ZFS: {external_source}") - external_mounts[external_source] = (path, mount) - if created and mount["fstype"] == "zfs" and "ro" not in mount["options"].split(","): - target = mount["source"] + "@" + stage - if target not in snapshots: - snapshots.append(target) - if snapshots: - if len({target.split("/", 1)[0] for target in snapshots}) != 1: - raise ValueError("staged datasets must belong to one ZFS pool") - command("zfs", "snapshot", *snapshots) - pending_snapshots.update(snapshots) - if created and data["hasManagedVolumes"]: - if source and data["stageIsolation"] == "clone": - try: - clone_dataset(source, snapshot, dataset, data["hostRoot"]) - except Exception: - command("zfs", "destroy", snapshot) - pending_snapshots.remove(snapshot) - raise - metadata["snapshot"] = snapshot - pending_snapshots.remove(snapshot) - else: - command("zfs", "create", "-o", "mountpoint=" + data["hostRoot"], dataset) - metadata["clone"] = dataset - bindings = {} - for source, volumes in sorted(external.items()): - path, mount = external_mounts[source] - if "ro" in mount["options"].split(","): - binding = {"src": source, "readOnly": True} - elif mount["fstype"] == "zfs": - dataset = mount["source"] - if metadata.get("snapshot") == dataset + "@" + stage: - clone_mount = data["hostRoot"] - else: - staged = metadata.setdefault("external", {}) - if dataset not in staged: - if not created: - raise ValueError("external storage changed; destroy the stage before recreating it") - suffix = hashlib.sha256(dataset.encode()).hexdigest()[:8] - pool = dataset.split("/", 1)[0] - clone = pool + "/staging/" + stage + "-external-" + suffix - clone_mount = str(Path(data["stagingRoot"]) / (stage + "-external-" + suffix)) - snapshot = dataset + "@" + stage - if subprocess.run(["zfs", "list", pool + "/staging"], capture_output=True).returncode: - command("zfs", "create", "-o", "mountpoint=none", pool + "/staging") - try: - clone_dataset(dataset, snapshot, clone, clone_mount) - except Exception: - command("zfs", "destroy", snapshot) - pending_snapshots.remove(snapshot) - raise - staged[dataset] = {"clone": clone, "snapshot": snapshot, "mount": clone_mount} - pending_snapshots.remove(snapshot) - clone_mount = staged[dataset]["mount"] - relative = path.relative_to(Path(mount["target"]).resolve()) - binding = {"src": str(Path(clone_mount) / relative), "readOnly": False} - else: - raise ValueError(f"writable external mount is not ZFS: {source}") - bindings[source] = binding - for volume in volumes: - if path.is_relative_to(Path(data["cloverRoot"]).resolve()): - volume["clover"] = True - volume.update(binding) - if not created and bindings != metadata.get("externalSources", {}): - raise ValueError("external storage changed; destroy the stage before recreating it") - metadata["externalSources"] = bindings - if REPO.parent == Path("/opt/studio/releases"): - metadata["release"] = REPO.name - metadata["ready"] = False - write_private(stage_dir / f"{stage}.json", json.dumps(metadata)) - bootstrap([data]) - if created and data["stageIsolation"] == "clone" and (data["secrets"] or data["requiredSecrets"]): - source = get_variable("nomad/jobs/" + args.name) - if source: - put_variable("nomad/jobs/" + stage, source["Items"], 0) - if postgres_snapshot: - postgres_clone = postgres_dataset.rsplit("/prod/", 1)[0] + "/staging/" + stage + "-postgres" - postgres_mount = Path(data["stagingRoot"]) / (stage + "-postgres") - source_context = cloned_postgres(postgres_snapshot, postgres_clone, postgres_mount) - else: - source_context = nullcontext(None) - with source_context as postgres_source: - provision_inputs(data, definitions, stage, postgres_source) - provision(data) - prepare(data) - build_images(data) - submit(data, "run", definitions) - except Exception: - if created: - destroy_stage(stage, metadata, definitions) - raise - finally: - for snapshot in sorted(pending_snapshots): - command("zfs", "destroy", snapshot) - if hostname: - check_http(hostname, http[0]["checkPath"], http[0]["tlsInternal"]) - configure(data) - if REPO.parent == Path("/opt/studio/releases"): - metadata["ready"] = True - metadata["overrides"] = [assignment.partition("=")[0] for assignment in args.env] - write_private(stage_dir / f"{stage}.json", json.dumps(metadata)) - print(f"stage={stage}\nrelease={metadata.get('release', '')}\nhost={hostname}\ndataset={metadata.get('clone', '')}") - return - definitions = {name: load(name, properties) for name in services()} - if args.name and args.name not in definitions: - raise ValueError(f"unknown service: {args.name}") - if not args.name and args.mode == "deploy": - names = [name for name in names if definitions[name]["containers"]] - selected = set(names) - while True: - required = selected | set().union(*(dependencies(definitions[name]) for name in selected)) - missing = required - definitions.keys() - if missing: - raise ValueError(f"unknown dependency: {sorted(missing)}") - if required == selected: - break - selected = required - data = ordered([definitions[name] for name in sorted(selected)]) - if args.mode == "render": - for item in data: - if item["containers"]: - print(render(item, definitions)) - elif args.mode == "validate": - token() - for item in data: - submit(item, "validate", definitions) - elif args.mode == "check": - for item in data: - for task in item["containers"].values(): - if task.get("http") and task["http"].get("hostname"): - check_http(task["http"]["hostname"], task["http"]["checkPath"], task["http"]["tlsInternal"]) - elif args.mode == "preflight": - check_pool(data) - token() - missing_secrets = {} - for item in data: - if item["containers"] and item["requiredSecrets"]: - variable = get_variable("nomad/jobs/" + item["id"]) - values = variable["Items"] if variable else {} - missing = [name for name in item["requiredSecrets"] if not values.get(name)] - if missing: - missing_secrets[item["id"]] = missing - if missing_secrets: - raise ValueError("missing required secrets: " + "; ".join( - f"{name}: {', '.join(keys)}" for name, keys in missing_secrets.items())) - elif args.mode == "pool": - ensure_pool(data) - elif args.mode == "allocate": - ensure_pool(data) - bootstrap(data) - for item in data: - provision_inputs(item, definitions) - provision(item) - prepare(item) - build_images(item) - elif args.mode == "bootstrap": - bootstrap(data) - elif args.mode == "deploy": - bootstrap(data) - for item in data: - provision_inputs(item, definitions) - provision(item) - prepare(item) - build_images(item) - submit(item, "run", definitions) - configure(item) - - -if __name__ == "__main__": - try: - main() - except (OSError, RuntimeError, ValueError, subprocess.CalledProcessError) as error: - print(error, file=sys.stderr) - sys.exit(1) diff --git a/tools/tree-hash.py b/tools/tree-hash.py deleted file mode 100644 index ee2f898a525160548952a8b18dc5a10c61995e3a..0000000000000000000000000000000000000000 --- a/tools/tree-hash.py +++ /dev/null @@ -1,48 +0,0 @@ -#!/usr/bin/env python3 -import hashlib -import json -import os -from pathlib import Path -import stat -import sys - - -root = Path(sys.argv[1]).resolve(strict=True) -if not root.is_dir(): - raise ValueError("Expected a directory") -digest = hashlib.sha256() -files = 0 -size = 0 - - -def fail(error): - raise error - - -for directory, children, names in os.walk(root, followlinks=False, onerror=fail): - children.sort() - for name in sorted([*children, *names]): - entry = Path(directory) / name - relative = os.fsencode(entry.relative_to(root)) - mode = entry.lstat().st_mode - if stat.S_ISDIR(mode): - kind = b"d" - elif stat.S_ISLNK(mode): - kind = b"l" - elif stat.S_ISREG(mode): - kind = b"f" - else: - raise ValueError(f"Unsupported file type: {entry}") - digest.update(kind + len(relative).to_bytes(8, "big") + relative) - if kind == b"l": - target = os.fsencode(os.readlink(entry)) - digest.update(len(target).to_bytes(8, "big") + target) - elif kind == b"f": - files += 1 - length = entry.stat().st_size - size += length - digest.update(length.to_bytes(8, "big")) - with entry.open("rb") as source: - for chunk in iter(lambda: source.read(1024 * 1024), b""): - digest.update(chunk) -print(json.dumps({"sha256": digest.hexdigest(), "files": files, "bytes": size})) diff --git a/tools/verify-legacy-dump.py b/tools/verify-legacy-dump.py deleted file mode 100644 index 6fca6c0c6cca44d8ae6297621f2c7c7cf51281c7..0000000000000000000000000000000000000000 --- a/tools/verify-legacy-dump.py +++ /dev/null @@ -1,37 +0,0 @@ -#!/usr/bin/env python3 -import hashlib -import json -from pathlib import Path -import re -import sys - - -folder = Path(sys.argv[1]) -if not folder.is_dir(): - raise SystemExit("Legacy handoff directory is unavailable") -folder = folder.resolve(strict=True) -database = sys.argv[2] -if folder.parent != Path("/mnt/storage1/apps/studio-handoff") or not re.fullmatch(r"\d{8}T\d{6}Z-[0-9a-f]{6}", folder.name): - raise ValueError("Unexpected legacy handoff directory") -if database not in {"evil-forgejo", "evil-hedgedoc"}: - raise ValueError("Unexpected legacy database") -manifest = json.loads((folder / "manifest.json").read_text()) -if manifest["id"] != folder.name: - raise ValueError("Legacy handoff ID changed") -entry = manifest["databases"][database] -if entry["file"] != database + ".dump": - raise ValueError("Legacy dump filename changed") -dump = folder / entry["file"] -digest = hashlib.sha256() -with dump.open("rb") as file: - for chunk in iter(lambda: file.read(1024 * 1024), b""): - digest.update(chunk) -if dump.stat().st_size != entry["bytes"] or digest.hexdigest() != entry["sha256"]: - raise ValueError("Legacy dump checksum mismatch") -if len(entry["counts"]) != (4 if database == "evil-hedgedoc" else 2) or any(not isinstance(value, int) or value < 0 for value in entry["counts"]): - raise ValueError("Legacy database counts are invalid") -if database == "evil-forgejo": - keys = {"lfs_jwt", "oauth_jwt", "security_key", "internal_token", "anubis_key", "mailer_address", "mailer_username", "mailer_password"} - if set(entry["secretSha256"]) != keys or any(not re.fullmatch(r"[0-9a-f]{64}", value) for value in entry["secretSha256"].values()): - raise ValueError("Legacy Forgejo secret fingerprints are invalid") -print(json.dumps(entry)) diff --git a/tools/vm-screen-test.py b/tools/vm-screen-test.py index b8f7a062bf6be80f88ff1bd843f172c81c7731ca..65c4c40ff9aecf55c13398270d5685b2ffafaf9f 100644 --- a/tools/vm-screen-test.py +++ b/tools/vm-screen-test.py @@ -48,7 +48,7 @@ class LifecycleTests(unittest.IsolatedAsyncioTestCase): with tempfile.TemporaryDirectory() as directory: socket = str(Path(directory) / "screen.sock") process = await asyncio.create_subprocess_exec(sys.executable, str(Path(__file__).with_name("vm-screen.py")), - env={**os.environ, "STUDIO_VM_SCREEN_SOCKET": socket}) + env={**os.environ, "SNOWGLOBE_VM_SCREEN_SOCKET": socket}) writers = [] try: async with asyncio.timeout(5): diff --git a/tools/vm-screen.py b/tools/vm-screen.py index 57afe790c4cfcd6d080dfa2768704d1af98ba1b7..34e81884b78ab2785073a230f222bedacc2d3dfb 100644 --- a/tools/vm-screen.py +++ b/tools/vm-screen.py @@ -24,9 +24,9 @@ from gi.repository import GLib, Gst, GstRtp, GstSdp, GstWebRTC MAX_PIXELS = 3840 * 2160 -STUN = os.environ.get("STUDIO_VM_STUN_SERVER", "stun://stun.cloudflare.com:3478") +STUN = os.environ.get("SNOWGLOBE_VM_STUN_SERVER", "stun://stun.cloudflare.com:3478") SLOTS = asyncio.Semaphore(4) -GUEST_PROTOCOL = json.loads(Path(os.environ.get("STUDIO_VM_GUEST_PROTOCOL", Path(__file__).resolve().parent.parent / "guest/protocol.json")).read_text()) +GUEST_PROTOCOL = json.loads(Path(os.environ.get("SNOWGLOBE_VM_GUEST_PROTOCOL", Path(__file__).resolve().parent.parent / "guest/protocol.json")).read_text()) GUESTS = {} TWCC = "http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01" @@ -312,7 +312,7 @@ class GuestDesktop: writer = None try: reader, writer = await asyncio.wait_for(asyncio.open_unix_connection( - os.environ.get("STUDIO_HOST_SOCKET", "/run/studio-host/host.sock")), 10) + os.environ.get("SNOWGLOBE_HOST_SOCKET", "/run/snowglobe-host/host.sock")), 10) peer = writer.get_extra_info("socket") if struct.unpack("3i", peer.getsockopt(socket.SOL_SOCKET, socket.SO_PEERCRED, 12))[1] != 0: raise ValueError("Untrusted guest display broker") @@ -801,7 +801,7 @@ async def serve(reader, writer): if not isinstance(name, str) or not name or len(name) > 63: raise ValueError("Choose a VM from the list.") host_reader, host_writer = await asyncio.wait_for(asyncio.open_unix_connection( - os.environ.get("STUDIO_HOST_SOCKET", "/run/studio-host/host.sock")), 10) + os.environ.get("SNOWGLOBE_HOST_SOCKET", "/run/snowglobe-host/host.sock")), 10) desktop = Desktop(host_reader, host_writer, lambda message: screen.send(message) if message["type"] != "cursor" or screen.source is desktop else None) host_peer = host_writer.get_extra_info("socket") if struct.unpack("3i", host_peer.getsockopt(socket.SOL_SOCKET, socket.SO_PEERCRED, 12))[1] != 0: @@ -896,7 +896,7 @@ async def main(): task = asyncio.create_task(serve(reader, writer)) clients.add(task) task.add_done_callback(clients.discard) - path = Path(os.environ.get("STUDIO_VM_SCREEN_SOCKET", "/run/studio-vm-screen/screen.sock")) + path = Path(os.environ.get("SNOWGLOBE_VM_SCREEN_SOCKET", "/run/snowglobe-vm-screen/screen.sock")) path.unlink(missing_ok=True) server = await asyncio.start_unix_server(connected, path=str(path), limit=131072) path.chmod(0o600) diff --git a/tools/vms.py b/tools/vms.py index 2659fc3ba52568a29ecf9bab7c211660d699b0cb..62f4186209ec3a902841e392f43fddc5f9c1935b 100644 --- a/tools/vms.py +++ b/tools/vms.py @@ -1,5 +1,6 @@ #!/usr/bin/env python3 import json +from importlib import import_module import os from pathlib import Path import re @@ -15,10 +16,11 @@ import xml.etree.ElementTree as ET DISKS = Path("/srv/vm") -IMAGES = Path(os.environ.get("STUDIO_VM_IMAGES_ROOT", "/srv/clover/Media/vm")) -PRESETS = Path(os.environ.get("STUDIO_VM_PRESETS_ROOT", str(IMAGES / "Presets"))) -UPLOADS = Path(os.environ.get("STUDIO_VM_UPLOADS_ROOT", str(IMAGES / "Install Disks"))) -NS = "{https://paperclover.net/studio}" +IMAGES = Path(os.environ.get("SNOWGLOBE_VM_IMAGES_ROOT", "/srv/clover/Media/vm")) +PRESETS = Path(os.environ.get("SNOWGLOBE_VM_PRESETS_ROOT", str(IMAGES / "Presets"))) +UPLOADS = Path(os.environ.get("SNOWGLOBE_VM_UPLOADS_ROOT", str(IMAGES / "Install Disks"))) +transition = import_module("namespace-transition") +NS = "{https://paperclover.net/snowglobe}" NAME = re.compile(r"[a-z0-9][a-z0-9-]{0,62}\Z") GUEST_CHANNEL = json.loads((Path(__file__).resolve().parent.parent / "guest/protocol.json").read_text())["channel"] PROFILES = { @@ -237,7 +239,7 @@ def domains(): for name in virsh("list", "--all", "--name").splitlines(): if not name: continue - root = ET.fromstring(virsh("dumpxml", name)) + root = transition.normalize_vm(ET.fromstring(virsh("dumpxml", name))) state_line = virsh("domstate", name, "--reason").splitlines()[0].lower() state = next((value for value in ("running", "blocked", "paused", "shutdown", "crashed", "pmsuspended") if state_line.startswith(value)), "shutoff") reason = state_line.split("(", 1)[1].rstrip(")") if state in {"paused", "crashed"} and "(" in state_line else None @@ -246,7 +248,7 @@ def domains(): disks = [] media = [] for element in root.findall("./devices/disk"): - if element.findtext("serial") == "studio-cloud-init": + if element.findtext("serial") == "snowglobe-cloud-init": continue source = element.find("source") target = element.find("target") @@ -456,7 +458,7 @@ def managed_directory(path): def change_media(spec): name = spec["name"] - root = ET.fromstring(virsh("dumpxml", name, "--inactive")) + root = transition.normalize_vm(ET.fromstring(virsh("dumpxml", name, "--inactive"))) drives = root.findall("./devices/disk[@device='cdrom']") source = resolve_image(spec["image"]) if spec["image"] else None details = profile(inferred_profile(root)) @@ -511,7 +513,7 @@ def save_preset(spec): name = spec["name"] if virsh("domstate", name).strip() != "shut off": raise ValueError("Shut down this VM before saving a preset.") - root = ET.fromstring(virsh("dumpxml", name, "--inactive")) + root = transition.normalize_vm(ET.fromstring(virsh("dumpxml", name, "--inactive"))) if root.findtext(f"metadata/{NS}vm/{NS}platform") == "macos": raise ValueError("Use a prepared OS X preset from the OS library.") if root.findtext(f"metadata/{NS}vm/{NS}setup") in {"windows-v2", "windows-v3"} and spec.get("setup") != "windows-v3": @@ -575,7 +577,7 @@ def save_preset(spec): def console_target(name): - root = ET.fromstring(virsh("dumpxml", name)) + root = transition.normalize_vm(ET.fromstring(virsh("dumpxml", name))) graphics = root.find("./devices/graphics[@type='vnc']") if virsh("domstate", name).strip() not in {"running", "paused", "blocked"} or graphics is None: raise ValueError("Start this VM before opening its screen.") @@ -590,8 +592,8 @@ def console_target(name): def secure_firmware(arch): firmware = "AAVMF" if arch == "aarch64" else "OVMF" - code = Path(os.environ.get(f"STUDIO_VM_SECURE_{firmware}_CODE", "")) - variables = Path(os.environ.get(f"STUDIO_VM_SECURE_{firmware}_VARS", "")) + code = Path(os.environ.get(f"SNOWGLOBE_VM_SECURE_{firmware}_CODE", "")) + variables = Path(os.environ.get(f"SNOWGLOBE_VM_SECURE_{firmware}_VARS", "")) if not code.is_absolute() or not variables.is_absolute(): raise ValueError("Secure UEFI firmware is unavailable on this host.") try: @@ -764,7 +766,7 @@ def create(spec): elif not overlay: command("qemu-img", "create", "-f", "qcow2", str(drive), str(spec["disk"])) virsh("pool-refresh", "vms") - domain_type = "qemu" if details_profile.get("emulated") or os.environ.get("STUDIO_VM_ACCEL") == "qemu" else "kvm" + domain_type = "qemu" if details_profile.get("emulated") or os.environ.get("SNOWGLOBE_VM_ACCEL") == "qemu" else "kvm" root = ET.Element("domain", type=domain_type) ET.SubElement(root, "name").text = name ET.SubElement(root, "description").text = spec["description"] @@ -836,7 +838,7 @@ def create(spec): devices = ET.SubElement(root, "devices") emulator = f"/run/current-system/sw/bin/qemu-system-{details_profile['arch']}" if profile_name == "windows-aarch64-nvme": - emulator = os.environ.get("STUDIO_VM_WINDOWS_ARM_EMULATOR", "") + emulator = os.environ.get("SNOWGLOBE_VM_WINDOWS_ARM_EMULATOR", "") if not Path(emulator).is_absolute(): raise ValueError("Windows ARM is unavailable on this host. Choose another OS.") ET.SubElement(devices, "emulator").text = emulator @@ -860,7 +862,7 @@ def create(spec): ET.SubElement(seed, "driver", name="qemu", type="raw") ET.SubElement(seed, "source", file=str(directory / "seed.iso")) ET.SubElement(seed, "target", dev=details_profile["seed"][0], bus=details_profile["seed"][1]) - ET.SubElement(seed, "serial").text = "studio-cloud-init" + ET.SubElement(seed, "serial").text = "snowglobe-cloud-init" ET.SubElement(seed, "readonly") if preset and spec["platform"] == "windows": windows_seed(directory, spec["username"]) @@ -946,14 +948,14 @@ def main(): fields = Path(f"/proc/{pid}/stat").read_text().rsplit(") ", 1)[1].split() resident = int(Path(f"/proc/{pid}/statm").read_text().split()[1]) * os.sysconf("SC_PAGE_SIZE") result[name] = {"cpu": (int(fields[11]) + int(fields[12])) / os.sysconf("SC_CLK_TCK"), "memory": resident, - "vcpus": int(ET.fromstring(virsh("dumpxml", name)).findtext("vcpu", "1"))} + "vcpus": int(transition.normalize_vm(ET.fromstring(virsh("dumpxml", name))).findtext("vcpu", "1"))} return result if action == "library": return library() if action == "console": return console_target(payload["name"]) if action == "guest": - root = ET.fromstring(virsh("dumpxml", payload["name"])) + root = transition.normalize_vm(ET.fromstring(virsh("dumpxml", payload["name"]))) source = root.find(f"./devices/channel[@type='unix']/target[@name='{GUEST_CHANNEL}']/../source") expected = f"/run/libvirt/qemu/channel/{root.get('id')}-{payload['name'][:20]}/{GUEST_CHANNEL}" if virsh("domstate", payload["name"]).strip() != "running" or source is None or source.get("mode") != "bind" or source.get("path") != expected: @@ -962,7 +964,7 @@ def main(): if action == "serial": if virsh("domstate", payload["name"]).strip() not in {"running", "blocked"}: raise ValueError("Start this VM before opening its console.") - root = ET.fromstring(virsh("dumpxml", payload["name"])) + root = transition.normalize_vm(ET.fromstring(virsh("dumpxml", payload["name"]))) console = root.find("./devices/console[@type='pty']") source = console.find("source") if console is not None else None target = console.find("target") if console is not None else None @@ -971,7 +973,7 @@ def main(): raise ValueError("This VM has no serial console. Add one in its hardware settings.") return {"path": path} if action == "owner": - return ET.fromstring(virsh("dumpxml", payload["name"])).findtext(f"metadata/{NS}vm/{NS}owner") + return transition.normalize_vm(ET.fromstring(virsh("dumpxml", payload["name"]))).findtext(f"metadata/{NS}vm/{NS}owner") if action == "access": try: with os.fdopen(open_regular(DISKS / payload["name"] / "access.json")) as incoming: @@ -988,7 +990,7 @@ def main(): name = payload["name"] ensure_name(name) if payload["action"] == "start": - root = ET.fromstring(virsh("dumpxml", name, "--inactive")) + root = transition.normalize_vm(ET.fromstring(virsh("dumpxml", name, "--inactive"))) if root.find("./devices/interface/source[@network='default']") is not None: ensure_network() virsh({"start": "start", "shutdown": "shutdown", "reboot": "reboot", "destroy": "destroy", "resume": "resume"}[payload["action"]], name) @@ -1009,7 +1011,7 @@ def main(): raise ValueError("VM does not exist") if virsh("domstate", name).strip() != "shut off": virsh("destroy", name) - root = ET.fromstring(virsh("dumpxml", name)) + root = transition.normalize_vm(ET.fromstring(virsh("dumpxml", name))) virsh("undefine", name, *(["--nvram"] if root.find("./os/nvram") is not None else []), *(["--tpm"] if root.find("./devices/tpm") is not None else [])) directory = DISKS / name if payload["disks"] and directory.is_dir(): diff --git a/tools/xmpp.md b/tools/xmpp.md index 1a47e9ef7365ed60699e8d5c27b7d5cade10ccc3..f96f7e61bba761ad4bbbe2c61fbc3d0eb5cec2df 100644 --- a/tools/xmpp.md +++ b/tools/xmpp.md @@ -20,62 +20,31 @@ these host ports; stop its Nomad job before launching another on the same host. `setup.py` installs a transient firewall service tagged with the preview ID. It removes its rules within ten seconds of preview destruction, and on service stop. After a host reboot, repeat the staging command to restore these runtime rules. -Production firewall rules are declared in `nixos/zenith.nix`. The DNS provider +Production firewall rules are declared in `nixos/snowglobe.nix`. The DNS provider provisions client, direct-TLS client, and federation SRV records pointing at `xmpp.paperclover.net`; the website keeps its existing apex address. Router forwarding must send the TCP listeners, UDP TURN listeners, and UDP relay range -to Zenith. `xmpp-nat.service` refreshes UPnP mappings on startup and every five +to Snow Globe. `xmpp-nat.service` refreshes UPnP mappings on startup and every five minutes, using the Archer AX3000's description URL at the configured gateway. The description is fetched directly; multicast SSDP discovery failed on this host. Production activation uses the normal described-main deployment: ```sh -STUDIO_DEPLOY_HOST=root@10.0.0.1 STUDIO_DEPLOY_PORT=22 python3 tools/deploy.py prod +SNOWGLOBE_DEPLOY_HOST=root@10.0.0.1 SNOWGLOBE_DEPLOY_PORT=22 python3 tools/deploy.py prod ``` -## Zenith production, 2026-10-10 +## Client connections and verification The owner-readable connection file on Clover's Mac is `~/.local/share/snowglobe/xmpp.json`. Monal can use the production JID and its generated password; the explicit server is `xmpp.paperclover.net`, port `5223`, with direct TLS. Client and federation SRV records resolve to that server. - -Verified through the public address from the home LAN: TLS authentication, two -resources, carbons, MAM retrieval, stream resumption, PEP publication/deletion, -private group creation/history/destruction, PNG upload/download, WebSocket, -authenticated UDP TURN allocation/deallocation, relay address/range, rejection -of a private peer, STARTTLS, and TURN TLS certificate validation. All three tasks -had zero restarts. The normal deployment took backup -`20261011T015942Z-a7e251` and passed all service route checks. A transient systemd -probe confirmed UPnP refresh works with `DynamicUser`. Off-LAN calls, iOS push -delivery, federation with another server, and recovery after a router restart -need client-side verification. - -## Zenith preview, 2026-10-10 - -Stage: `xmpp-preview-c035f9fb` -Hostname: `xmpp-preview-c035f9fb.paperclover.net` -JID: `clover@xmpp-preview-c035f9fb.paperclover.net` -Dataset: `globe/staging/xmpp-preview-c035f9fb` - -The preview Nomad job and its transient firewall service were stopped for the -production cutover. Its dataset, Nomad variables, and owner-readable Mac -connection file at `~/.local/share/snowglobe/xmpp-preview.json` are retained. -Preview account data was not copied into production. - -Delete the preview, including its accounts and history: - -```sh -STUDIO_DEPLOY_HOST=root@10.0.0.1 STUDIO_DEPLOY_PORT=22 python3 tools/deploy.py destroy xmpp-preview-c035f9fb -``` - -Verified with synthetic data: certificate validation and TLS 1.3 login, two -simultaneous resources, carbons, stream resumption after transport loss, MAM -retrieval, PEP publication/deletion, private group creation/history/destruction, -PNG upload/download over public HTTPS, WebSocket stream opening, external-service -credentials, authenticated UDP TURN allocation/deallocation, advertised public -relay address/range, and rejection of a private-network peer. Push support was -discovered; delivery through an iOS client's push provider and calls across the -public router were not exercised. All 25 pre-existing service jobs rendered -byte-identically after the TCP schema migration. The active production release -was unchanged, and Caddy, Nomad, dashboard, and Shale checks passed. +A preview has its own connection file and account domain; never copy its +account database into production. + +Test client authentication, two-resource carbons, MAM retrieval, resumable +streams, OMEMO/PEP keys, private group history, uploads, and WebSocket after +changing the service. Calling also requires authenticated TURN, the configured +public relay range, and an off-LAN client check. Push discovery alone does not +prove delivery through an iOS client's push provider. Recheck external +connectivity after router or forwarding changes. diff --git a/tools/yt-feed-migration.md b/tools/yt-feed-migration.md deleted file mode 100644 index d142ac2592f1a59f73a2973163907eb8cb2600d4..0000000000000000000000000000000000000000 --- a/tools/yt-feed-migration.md +++ /dev/null @@ -1,9 +0,0 @@ -# YouTube Triage migration - -The service keeps six JSON state files in `/mnt/storage1/apps/yt-feed` and editable `feed.yaml` and `subscriptions.yaml` in Clover's `Documents/Config/Youtube Downloader`. Its downloaded videos and upscaler output live in the existing Media dataset, so that dataset moves by ZFS rename as described in [media-cutover.md](media-cutover.md). - -On 2026-09-26, all six JSON files in `yt-feed-preview-e87b1ca1` matched Zenith by SHA-256 and parsed as JSON. Both Clover configuration files had matching SHA-256 hashes in the VM. The preview Media mount was read-only. Globe now owns the review API and worker; Nomad's `ytdl` job runs the thumbnail upscaler and archiver. - -For production, set `STUDIO_DEPLOY_HOST` and `STUDIO_DEPLOY_PORT` for the new host, then run `bash tools/import-legacy-secrets.sh ytdl MAILER_ADDRESS MAILER_USERNAME MAILER_PASSWORD`. The order matches `smtp_host`, `smtp_user`, and `smtp_pass` in `service/ytdl/service.pkl`. Stop Zenith's `yt-feed` and the Snow Globe `ytdl` job before running `bash tools/import-yt-feed.sh ytdl`; the importer stops Globe's worker, snapshots the Snow Globe dataset, verifies the JSON copy, then restarts Globe. Clover's configuration and the Media dataset must be mounted at their final paths before import. Verify Globe's YouTube queue and one manual review action before cutover. - -For a same-machine OS replacement, set `STUDIO_LEGACY_HANDOFF` to the [offline handoff](legacy-handoff.md) directory while importing both secrets and JSON state. The production importer then reads the retained files from the mounted old apps dataset on the new host, without old Docker. diff --git a/tools/zenith-hardware.md b/tools/zenith-hardware.md deleted file mode 100644 index 786b6e58cedd5ccd39cf6c64747cf0272623e4e6..0000000000000000000000000000000000000000 --- a/tools/zenith-hardware.md +++ /dev/null @@ -1,35 +0,0 @@ -# Zenith source host inventory - -NixOS was installed onto the NVMe on October 4, 2026 and passed two boots over SSH at the existing addresses. The final boot reported `running` with zero failed units: the data pool imported healthy, all six encrypted roots unlocked and mounted automatically through the TPM credentials, and Nomad, Caddy, SSH, the host service, and NVIDIA persistence started successfully. Original dataset GUIDs and all PostgreSQL handoff file hashes matched their pre-install records. The RTX 3090 uses driver 595.71.05; `hardware.graphics.enable` supplies the driver library link needed by its persistence daemon. The historical TrueNAS inventory below describes the earlier layout. - -Read-only inspection on 2026-09-27 found a BIOS-booted Ryzen 9 5950X host with a Realtek RTL8111/8168/8411 NIC (`r8169`), RTX 3090, one 500 GB WD Blue SN5000 NVMe boot disk, and four 8 TB WD80EFPX disks. The NVMe is `/dev/disk/by-id/nvme-WD_Blue_SN5000_500GB_24261Z806200`; its GPT has a 1 MB BIOS boot partition, a 512 MB EFI partition, and a TrueNAS `boot-pool` partition with the running root at `boot-pool/ROOT/25.04.2.4`. The four other disks form the healthy `storage1` RAIDZ1 pool. The VM's UEFI boot configuration does not describe this host. - -`enp4s0` has static `10.0.0.1/24` and `192.168.0.1/24` addresses, default gateway `10.0.0.2`, and resolvers `1.1.1.1` and `1.0.0.1`. The NixOS target retains these; DHCP would not preserve the NAS address. - -`storage1` is unencrypted and mounted at `/mnt/storage1`. Its `apps`, `clover`, and `media` children are separate AES-256-GCM encryption roots. `apps` uses POSIX ACLs; `clover` and `media` use NFSv4 ACLs. Their mountpoints are `/mnt/storage1/apps`, `/mnt/storage1/clover`, and `/mnt/storage1/media`. The pool reported about 16.1 TB free. The [ACL cutover](storage-acl-cutover.md) and [Media cutover](media-cutover.md) describe the data-side migration; no disk or pool changes were made during this inventory. - -On 2026-09-27, Zenith ran OpenZFS 2.3.0 and reported `storage1` healthy. The pinned NixOS VM ran OpenZFS 2.4.4; its `zpool upgrade -v` listed every feature currently enabled or active on `storage1`, including encryption and block cloning. This checks feature support, not an actual import of the four-disk pool. - -TrueNAS's previous schedule snapshotted Clover hourly for one week, daily for four weeks, and monthly for two years; apps hourly for one week and daily for one month. Media had no scheduled snapshot task. NixOS does not yet replace this retention policy. The pinned NixOS `services.zfs.autoSnapshot` module has global retention counts, so it cannot express these different dataset schedules. Its `services.sanoid` module supports retention per dataset; the owner is choosing the replacement policy before it is enabled. - -The October 4 installer boots in UEFI mode on the same hardware. The `zenith` target now uses systemd-boot, host ID `4fa19ccb`, the renamed `globe` pool, and `paperclover.net`. The owner completed the dataset renames in [media-cutover.md](media-cutover.md); retained apps stay at `/mnt/storage1/apps`. Encrypted `globe/prod` and `globe/staging` roots were created with POSIX ACLs for the new service volumes. Nomad's startup guard requires the real pool, Clover, Media, production, and staging mounts; deployment separately checks their ACL types. - -The October 4 personal ACL cutover retained `globe/clover@personal-acl-20261004` and `globe/clover/Media@personal-acl-20261004`. Snapshot clones first passed read/write tests for UIDs 3000, 3106, 3114, and 3116 with group 3000. Both original datasets now use `acltype=posix`, `aclmode=discard`, and `aclinherit=discard`; retaining the legacy `aclmode=restricted` caused `EPERM` during rehearsal. One metadata pass processed 1,448,236 Clover entries and 31,187 Media entries without changing owner UIDs: group 3000 receives `g+rwX`, directories have setgid and inheritable group ACLs, and other access is `---`. Traversal stayed on each filesystem and did not follow symlinks. `/var/lib/studio/personal-acl-verified.json` records the completed original passes. The test clones were destroyed after verification; the original snapshots remain. - -The ASUS firmware TPM2 successfully sealed and decrypted every ZFS encryption-root key. [zenith.nix](../nixos/zenith.nix) loads encrypted systemd credentials from `/etc/credstore.encrypted` and runs `studio-zfs-unlock` after pool import and before `zfs-mount`. Plaintext keys remain outside the Nix store and NVMe; the running service's credentials exist in RAM. The credentials are bound to this TPM and PCR 7, so firmware changes affecting Secure Boot or clearing/replacing the TPM can require manual recovery. Secure Boot is currently disabled. The original dataset keys and TrueNAS configuration remain in the private Mac recovery archive; new production/staging keys are backed up separately at `/Users/clo/Library/Application Support/Zenith Recovery/postgres-backups/new-os-dataset-keys.json`. - -The NVMe installation replaces the old TrueNAS boot pool with a 1 GiB FAT32 EFI partition and an ext4 root partition occupying the remaining space. The only erase target is `/dev/disk/by-id/nvme-WD_Blue_SN5000_500GB_24261Z806200`; no HDD partition is formatted. [legacy-handoff.md](legacy-handoff.md) records the verified exports and off-server backups made before replacing the OS. The generated [hardware configuration](../nixos/hardware-configuration.nix) identifies the new root and EFI filesystems. The installed source lives at `/etc/nixos`; subsequent OS changes use `nixos-rebuild switch --flake /etc/nixos#zenith`. - -The target authorizes this Mac's existing ED25519 key for `clo` and root SSH. Its fingerprint `SHA256:52mNGHRsVFBDED9IAX5pe+LRWUefqTbxEReunq21QvU` matches the key Zenith currently accepts from this Mac. The other three keys in Zenith's `clo` authorized-keys file are not copied into the new root account. A synthetic NixOS evaluation confirmed both accounts receive exactly this key. - -The original TrueNAS ED25519, ECDSA, and RSA SSH host keys were recovered from the configuration backup, checked against their public keys, and restored into `/etc/ssh` on NixOS. ED25519 fingerprint `SHA256:P8pRWNrL6tceq9KN41Hdb0v0kmIboMf/zFBgOtHJfm0` matches this Mac's existing `git.paperclover.net` trust entry; a fresh connection with strict host checking passed after reloading SSH. The installer keys used during installation are retained privately in `/root/infra2-installer-host-keys`. [Shale migration](shale-migration.md) records the intended Git service scope, transport check, and replacement of the legacy SSH router. - -[Nomad stores its server state under `data_dir`](https://developer.hashicorp.com/nomad/docs/configuration), including [variables and their encrypted secret values](https://developer.hashicorp.com/nomad/docs/concepts/variables). Snow Globe stores generated service UIDs, deployment history, backup manifests/dumps, and dashboard state under `/var/lib/studio`. Both paths are on the VM's OS disk, outside its ZFS service datasets. Nomad used `66 MB` at inspection; Snow Globe's `8.7 GB` includes an `8.5 GB` VM-only `zpool.img`, while its release backups used `275 MB` across 25 runs. Reusing or replacing Zenith's NVMe would lose the control records unless they are migrated to encrypted ZFS or backed up separately. The pinned NixOS Nomad module accepts `services.nomad.settings.data_dir = "/srv/prod/nomad"` when `dropPrivileges = false`; a synthetic target evaluation passed. The corresponding Snow Globe state mount and dataset boundary await the storage policy decision before the physical cutover. - -A `nomad operator snapshot save` on the VM produced a private, compressed 143,608-byte snapshot in `/run`. Inspection reported 46 variables, 31 jobs, and 25 ACL policies. The snapshot restored into a fresh server-only Nomad agent in an isolated network namespace: its API listed all 31 jobs and 46 variables, and a hash comparison of Shale's secret variable items matched the live server without printing the values. The temporary agent, data, and snapshot were removed; the live leader and jobs stayed healthy. This proves [Nomad's server-state snapshot and restore](https://developer.hashicorp.com/nomad/commands/operator/snapshot/restore) on the pinned version, but Snow Globe's separate UID registry and backup files still need durable storage. - -The production target selects NixOS's stable NVIDIA driver for the RTX 3090 and its headless persistence daemon; the synthetic install evaluated with driver 595.71.05. Jellyfin has no GPU device allocation yet, so hardware transcoding still needs a physical-host test. - -With root SSH access and Nomad running, `STUDIO_DEPLOY_HOST=root@10.0.0.1 STUDIO_DEPLOY_PORT=22 python3 tools/deploy.py bootstrap` creates the Nomad ACL tokens and service datasets before checking external secrets. The first run reports all missing secret names; load those with `tools/import-legacy-secrets.sh` or `deploy.py secrets`, then rerun bootstrap to launch the jobs. Generated service secrets are created automatically. No Snow Globe job starts before the external-secret check passes. A bootstrap interrupted after switching the release link can be retried until deployment history records success. - -The ACL bootstrap ran twice against a disposable Nomad dev agent on the VM, created management, router, and dashboard token files, and left 14 policies. The dev agent and temporary state were removed; the VM's production Nomad agent was untouched.