projectbrain-Public/scripts/legacy-deployment-migration/README.md

Controlled legacy deployment migration

plan.py is a pure planner. execute.py prepare is an explicit opt-in, build-only preparation command. Separate cutover and recover modes require --apply and the exact verified preparation identity. Tests use fixtures/offline Compose rendering. The first verified production migration is recorded below; test results alone must never be presented as proof of a production cutover.

Authoritative broker code: Unraid-AppOps-Gateway/ops/gitea-deploy-broker/deploy.py. Production installs it separately at /mnt/pool/appdata/gitea-deploy-runners/controller/deploy.py. Production projects configuration is intentionally outside Git. Compare normalized source before changes.

Why standard retry/bootstrap cannot migrate these two

VacatureRadar and LumaOps have healthy legacy containers but schema3 failed state with no known live commit or artifacts. capture_compose_rollback rightly refuses unattested rollback. bootstrap-legacy requires exact historical successful legacy state and refuses schema3. Never invent a revision or a successful receipt.

Standard preflight overlays live appdata, and standard migration cleanup removes retained legacy containers. Neither is suitable for a rollback-retaining migration with unknown original provenance. Use a reviewed separate one-shot transaction.

Required transaction design

1. Pin/validate exact Gitea commit and tree. Prepare a separate clean build context; external credentials remain outside it. Build before disrupting production. Require immutable OCI source/revision/tree/version/time labels, not container labels alone. Existing configurations do not enable these checks yet.
2. Acquire the same project then global locks as the broker. Revalidate original container and immutable image IDs, healthy state, exact mount inventory, free disk space, candidate network/ports, and absence of attempt targets. Protect the original image with an immutable rollback reference. Persist a root-only journal and original configuration snapshot without exposing secrets in output.
3. Stop the original, cold-copy and verify all writable data into new attempt-owned paths. VacatureRadar runs PostgreSQL17 and Django migrations on startup: restarting the old image on migrated data is NOT rollback. Its nested local, logs and media bind mounts share one cold copy. LumaOps has bind mounts plus an anonymous /config volume; include both with their nested destination semantics preserved.
4. Retain/rename the exact original container. Start only the candidate under a distinct Compose project (<slug>-managed), preserving production name/ports. The candidate mounts clones exclusively. A distinct project prevents Compose from adopting/removing retained original containers or failing exact-container-set checks.
5. Verify immutable built image IDs, provenance and Docker health plus representative HTTP behavior. Only then obtain the real broker receipt and commit broker config (exact source, external env, explicit cloned data paths, new project identity, OCI verification) and state. Keep the originals and cold-copy journal available.
6. On any failure before commit, remove only recorded attempt-created container IDs after identity checks, restore broker configuration, and rename/restart the exact original container with original untouched data. Verify old health. A failed restore is reported as unknown/failed, never as healthy. Keep partial artifacts for recovery.

Process-crash recovery needs a durable phase journal and idempotent identity checks; exception handlers alone are insufficient. Source-tree, data, broker configuration, runtime and receipt transitions must be separately recorded. Do not automatically clean originals after success. Retention cleanup needs separate authorization.

build_plan accepts a strict sanitized runtime projection: name, containerId, imageId, running, health, composeProject, and mounts (type/source/destination/rw). Never feed or print full Docker inspect/environment output. This planner validates paths lexically; the executor also resolves roots, rejects symlinks/special files, verifies backup checksums/ownership and revalidates state while locks are held.

Run: python -m unittest discover -s scripts/legacy-deployment-migration -p 'test_*.py'.

Build-only preparation

After the two tracked docker-compose.migration.yml files are reviewed and merged, execute.py prepare <slug> <full-current-branch-sha> <unique-attempt> --container-id <expected-id> --image-id <expected-immutable-id> --apply prepares on the controller. Without --apply it performs no work. It validates the reviewed broker artifact, current branch head and original healthy container identity, then holds the normal project/global locks while extracting an isolated exact source tree and building an candidate image through the exact merged Compose configuration. OCI labels are checked on its immutable image ID. Source, original runtime and mounts are rechecked after build. Runtime environment is captured privately for Compose interpolation; credentials remain outside the build context and are not supplied as build arguments. Root-only journal/logs retain interrupted/failed attempts; a repeated attempt refuses to overwrite them. No production configuration/state/data is written.

The build-only journal is NOT a live receipt. cutover starts that same image using Compose up --no-build --pull never, then normal broker verification binds the observed runtime to its prepared immutable image ID. No rebuild occurs during downtime. Only a genuine normal broker receipt can be committed.

Tracked Compose override field contract (all required, no live-data defaults):

• Shared: MIGRATION_ATTEMPT, MIGRATION_IMAGE, MIGRATION_SOURCE_REVISION, MIGRATION_SOURCE_TREE, MIGRATION_BUILD_ID, MIGRATION_BUILD_DATE.
• VacatureRadar: MIGRATION_LOCAL_ROOT, VACATURERADAR_ENV_FILE.
• LumaOps: MIGRATION_CONFIG_ROOT (anonymous /config clone), MIGRATION_OPENRGB_ROOT, MIGRATION_LUMA_CONFIG_ROOT, MIGRATION_DATA_ROOT, MIGRATION_LOGS_ROOT, MIGRATION_ENV_FILE.

Pass the protected external runtime env file as Compose --env-file as well, preserving base-file interpolation values. Before cutover validate rendered absolute mounts against the plan, reject originals, and verify create_host_path:false so a missing copy cannot silently become an empty data directory. The normal broker requires Compose files to exist in the committed tree, which is why these overrides are tracked app files rather than a generated external override.

Cutover and recovery

Use the same slug, revision, attempt and expected original IDs as preparation: execute.py cutover ... --apply. Selected project configuration/state hashes, source, image, runtime env, original IDs/health/mounts and rendered clone mounts/ports are revalidated before stopping. Both normal deployment locks remain held throughout.

The durable journal records intent before stopping, copying, renaming, deploying, verifying, and committing. Cold copies use cp -a --reflink=auto, filesystem sync and SHA256/ownership/mode/xattr/hardlink verification. The original is retained under a distinct name and immutable image tag. Broker config/state and original container configuration are saved root-only. The candidate never mounts original data. The new broker project uses dynamic revision/tree/build placeholders so subsequent normal deployments do not reuse frozen provenance labels.

An ordinary failure attempts verified recovery automatically. After abrupt process termination use execute.py recover ... --apply: it accepts only the saved original or candidate config/state, checks rollback snapshot integrity, and removes only the exact owned candidate (nonce, Compose project, recorded ID where available). It restores original config/state, canonical name and original untouched data, then verifies original immutable image/mounts, Docker health and HTTP readiness. Candidate data, logs and all originals are retained. Unrelated containers/config/state or changed original data block automatic recovery rather than be overwritten/deleted.

Recovery after the final committed journal marker reverifies production and resumes activation; it does not roll back potentially newer application writes. Recovery before that marker restores the original, including a crash after receipt/config write but before final journal commit. Failed recovery remains explicitly recovery_failed, never healthy.

Independent review and actual prepared-image/maintenance-window checks are required before first invocation. Health paths were probed read-only and returned HTTP 200 on 2026-09-09: VacatureRadar port1226 /health/ready/, LumaOps port1223 /health/ready. These endpoints supplement, not replace, immutable image and normal receipt checks.

VacatureRadar's reviewed source must contain both the supervised Celery startup hold and first-position HTTP maintenance middleware. The helper writes an attempt-bound .migration-worker-hold only into the verified cold clone before candidate start. Ordinary traffic receives 503; exact GET/HEAD readiness/liveness probes remain available. After durable commit, activation journals intent, removes only the matching marker, fsyncs its directory, and records completion. Activation failure stays committed: explicit recovery resumes release and never rolls back after workers can run. A missing marker without durable activation intent fails closed.

Both tracked overrides require MIGRATION_NETWORK, bound to the exact inspected original <slug>_default network. Different/additional original networks fail closed; the rendered candidate must retain that network. Network/device compatibility and app-specific migration behavior still require operator review; the helper is not a generalized transaction system for external services.

LumaOps has no application maintenance gate in this migration. Its candidate can serve requests and interact with RGB hardware before the receipt is committed. Cold-data rollback restores the original application/container/data, but cannot undo external hardware effects or preserve candidate writes from that interval. Perform its cutover in the approved maintenance window with no concurrent user changes; do not describe this as a zero-effect or zero-data-loss transaction.

Handoff safety boundary (2026-09-09)

The guarded broker artifact expected by execute.py has normalized SHA256 b5a0467f53acba5c783781a22bf81cf99f8b96ae99535460ad6236fac038330d. This reviewed broker revision was installed atomically on 2026-09-09 after AppOps PR30 passed both validations and merged as 219184267972c596e7b61bbb29f0fab1b6472c9b. The previous controller is retained as deploy.py.before-single-runtime-rollback-20260909. The helper refuses to run until the host artifact matches. Candidate devices, capabilities, read-only root, privilege, security options and tmpfs must match the inspected original exactly. It rejects stale selected-project configuration after acquiring the project lock. Preparation also records selected-project configuration and original state hashes; cutover compares both before making any production change.

Simulated failure tests cover interruption after original stop, after rename and after receipt/config write before journal commit. Negative adapter tests cover unrelated candidate nonce/project/IDs and binding broker verification to the prepared immutable image. Offline Compose tests verify effective clone mounts and missing-input failures. These tests do not claim a live migration or an end-to-end Docker rollback.

The protected-main approval/release gate for the app overrides and other prerequisite branches must be satisfied before selecting a preparation SHA. Never run preparation against an unmerged workaround or claim these local tests prove a live migration.

Verified production outcome, 2026-09-09

VacatureRadar was prepared and committed from merged revision 38e87678e1c5b6aa6de2cb49e49366fce8a3cbf2, image sha256:c410c5ec26deb0cbd6a2992d4741e6c356f537d087de7fade74f214a747cca5a. The protected journal is under the controller's legacy-migrations/vacatureradar/legacy-20260909-a/ directory and records committed, activationIntent:true, activated:true. The original container VacatureRadar-legacy-legacy-20260909-a remains stopped with its original data. Docker health, HTTP readiness, the login page and running Celery processes were checked after activation. Normal Gitea deployment run9748 succeeded against the same verified revision. No post-commit rollback was attempted.

The initial helper's Docker JSON template incorrectly grouped index arguments. Preparation rejected this before production changes. Revision 4d1fc0d fixed it, with a regression test and successful read-only qualification against real Docker. The actual cutover used that reviewed helper revision. Three additional Linux storage tests used isolated temporary fixtures, not application data.

LumaOps has not been migrated. Its original runtime has AUTH_ENABLED=false and no admin token; the merged target requires authentication on its production network binding. Preparation stopped before building or changing production. Provisioning secure login needs an explicit operator decision; never disable the target's authentication guard or describe the old runtime as a verified new build. Failed preparation artifacts are retained; no LumaOps cutover journal exists.