projectbrain-Public/scripts/legacy-deployment-migration/README.md

# Controlled legacy deployment migration

`plan.py` is a **pure planner**. `execute.py prepare` is an explicit opt-in,
build-only preparation command. Separate `cutover` and `recover` modes require
`--apply` and the exact verified preparation identity. Tests use fixtures/offline
Compose rendering. The first verified production migration is recorded below;
test results alone must never be presented as proof of a production cutover.

Authoritative broker code: `Unraid-AppOps-Gateway/ops/gitea-deploy-broker/deploy.py`.
Production installs it separately at
`/mnt/pool/appdata/gitea-deploy-runners/controller/deploy.py`. Production projects
configuration is intentionally outside Git. Compare normalized source before changes.

## Why standard retry/bootstrap cannot migrate these two

VacatureRadar and LumaOps have healthy legacy containers but schema3 failed state
with no known live commit or artifacts. `capture_compose_rollback` rightly refuses
unattested rollback. `bootstrap-legacy` requires exact historical successful legacy
state and refuses schema3. Never invent a revision or a successful receipt.

Standard preflight overlays live appdata, and standard migration cleanup removes
retained legacy containers. Neither is suitable for a rollback-retaining migration
with unknown original provenance. Use a reviewed separate one-shot transaction.

## Required transaction design

1. Pin/validate exact Gitea commit and tree. Prepare a separate clean build context;
   external credentials remain outside it. Build before disrupting production.
   Require immutable OCI source/revision/tree/version/time labels, not container
   labels alone. Existing configurations do not enable these checks yet.
2. Acquire the same project then global locks as the broker. Revalidate original
   container and immutable image IDs, healthy state, exact mount inventory, free
   disk space, candidate network/ports, and absence of attempt targets. Protect the
   original image with an immutable rollback reference. Persist a root-only journal
   and original configuration snapshot without exposing secrets in output.
3. Stop the original, cold-copy and verify all writable data into new attempt-owned
   paths. VacatureRadar runs PostgreSQL17 and Django migrations on startup: restarting
   the old image on migrated data is NOT rollback. Its nested `local`, logs and media
   bind mounts share one cold copy. LumaOps has bind mounts plus an anonymous `/config`
   volume; include both with their nested destination semantics preserved.
4. Retain/rename the exact original container. Start only the candidate under a
   distinct Compose project (`<slug>-managed`), preserving production name/ports.
   The candidate mounts clones exclusively. A distinct project prevents Compose from
   adopting/removing retained original containers or failing exact-container-set checks.
5. Verify immutable built image IDs, provenance and Docker health plus representative
   HTTP behavior. Only then obtain the real broker receipt and commit broker config
   (exact source, external env, explicit cloned data paths, new project identity,
   OCI verification) and state. Keep the originals and cold-copy journal available.
6. On any failure before commit, remove only recorded attempt-created container IDs
   after identity checks, restore broker configuration, and rename/restart the exact
   original container with original untouched data. Verify old health. A failed restore
   is reported as unknown/failed, never as healthy. Keep partial artifacts for recovery.

Process-crash recovery needs a durable phase journal and idempotent identity checks;
exception handlers alone are insufficient. Source-tree, data, broker configuration,
runtime and receipt transitions must be separately recorded. Do not automatically
clean originals after success. Retention cleanup needs separate authorization.

`build_plan` accepts a strict sanitized runtime projection: `name`, `containerId`,
`imageId`, `running`, `health`, `composeProject`, and `mounts` (type/source/destination/rw).
Never feed or print full Docker inspect/environment output. This planner validates
paths lexically; the executor also resolves roots, rejects symlinks/special files,
verifies backup checksums/ownership and revalidates state while locks are held.

Run: `python -m unittest discover -s scripts/legacy-deployment-migration -p 'test_*.py'`.

## Build-only preparation

After the two tracked `docker-compose.migration.yml` files are reviewed and merged,
`execute.py prepare <slug> <full-current-branch-sha> <unique-attempt> --container-id
<expected-id> --image-id <expected-immutable-id> --apply` prepares on the controller.
Without `--apply` it performs no work. It validates the reviewed broker artifact,
current branch head and original healthy container identity, then holds the normal
project/global locks while extracting an isolated exact source tree and building an
candidate image through the exact merged Compose configuration. OCI labels are
checked on its immutable image ID. Source, original runtime and mounts are rechecked
after build. Runtime environment is captured privately for Compose interpolation;
credentials remain outside the build context and are not supplied as build arguments.
Root-only journal/logs retain interrupted/failed attempts; a repeated
attempt refuses to overwrite them. No production configuration/state/data is written.

The build-only journal is NOT a live receipt. `cutover` starts that same image using
Compose `up --no-build --pull never`, then normal broker verification binds the
observed runtime to its prepared immutable image ID. No rebuild occurs during downtime.
Only a genuine normal broker receipt can be committed.

Tracked Compose override field contract (all required, no live-data defaults):

- Shared: `MIGRATION_ATTEMPT`, `MIGRATION_IMAGE`, `MIGRATION_SOURCE_REVISION`, `MIGRATION_SOURCE_TREE`,
  `MIGRATION_BUILD_ID`, `MIGRATION_BUILD_DATE`.
- VacatureRadar: `MIGRATION_LOCAL_ROOT`, `VACATURERADAR_ENV_FILE`.
- LumaOps: `MIGRATION_CONFIG_ROOT` (anonymous `/config` clone),
  `MIGRATION_OPENRGB_ROOT`, `MIGRATION_LUMA_CONFIG_ROOT`, `MIGRATION_DATA_ROOT`,
  `MIGRATION_LOGS_ROOT`, `MIGRATION_ENV_FILE`.

Pass the protected external runtime env file as Compose `--env-file` as well,
preserving base-file interpolation values. Before cutover validate rendered absolute
mounts against the plan, reject originals, and verify `create_host_path:false` so a
missing copy cannot silently become an empty data directory. The normal broker
requires Compose files to exist in the committed tree, which is why these overrides
are tracked app files rather than a generated external override.

## Cutover and recovery

Use the same slug, revision, attempt and expected original IDs as preparation:
`execute.py cutover ... --apply`. Selected project configuration/state hashes,
source, image, runtime env, original IDs/health/mounts and rendered clone mounts/ports
are revalidated before stopping. Both normal deployment locks remain held throughout.

The durable journal records intent before `stopping`, `copying`, `renaming`,
`deploying`, `verifying`, and `committing`. Cold copies use `cp -a --reflink=auto`,
filesystem sync and SHA256/ownership/mode/xattr/hardlink verification. The original
is retained under a distinct name and immutable image tag. Broker config/state and
original container configuration are saved root-only. The candidate never mounts
original data. The new broker project uses dynamic revision/tree/build placeholders
so subsequent normal deployments do not reuse frozen provenance labels.

An ordinary failure attempts verified recovery automatically. After abrupt process
termination use `execute.py recover ... --apply`: it accepts only the saved original
or candidate config/state, checks rollback snapshot integrity, and removes only the
exact owned candidate (nonce, Compose project, recorded ID where available). It
restores original config/state, canonical name and original untouched data, then
verifies original immutable image/mounts, Docker health and HTTP readiness. Candidate
data, logs and all originals are retained. Unrelated containers/config/state or
changed original data block automatic recovery rather than be overwritten/deleted.

Recovery after the final `committed` journal marker reverifies production and resumes activation; it
does not roll back potentially newer application writes. Recovery before that marker
restores the original, including a crash after receipt/config write but before final
journal commit. Failed recovery remains explicitly `recovery_failed`, never healthy.

Independent review and actual prepared-image/maintenance-window checks are required
before first invocation. Health paths were probed read-only and returned HTTP 200 on
2026-09-09: VacatureRadar port1226 `/health/ready/`, LumaOps port1223 `/health/ready`.
These endpoints supplement, not replace, immutable image and normal receipt checks.

VacatureRadar's reviewed source must contain both the supervised Celery startup hold
and first-position HTTP maintenance middleware. The helper writes an attempt-bound
`.migration-worker-hold` only into the verified cold clone before candidate start.
Ordinary traffic receives 503; exact GET/HEAD readiness/liveness probes remain available.
After durable commit, activation journals intent, removes only the matching marker,
fsyncs its directory, and records completion. Activation failure stays committed:
explicit recovery resumes release and never rolls back after workers can run.
A missing marker without durable activation intent fails closed.

Both tracked overrides require `MIGRATION_NETWORK`, bound to the exact inspected
original `<slug>_default` network. Different/additional original networks fail closed;
the rendered candidate must retain that network. Network/device
compatibility and app-specific migration behavior still require operator review;
the helper is not a generalized transaction system for external services.

LumaOps has no application maintenance gate in this migration. Its candidate can
serve requests and interact with RGB hardware before the receipt is committed.
Cold-data rollback restores the original application/container/data, but cannot
undo external hardware effects or preserve candidate writes from that interval.
Perform its cutover in the approved maintenance window with no concurrent user
changes; do not describe this as a zero-effect or zero-data-loss transaction.

## Handoff safety boundary (2026-09-09)

The guarded broker artifact expected by `execute.py` has normalized SHA256
`b5a0467f53acba5c783781a22bf81cf99f8b96ae99535460ad6236fac038330d`.
This reviewed broker revision was installed atomically on 2026-09-09 after
AppOps PR30 passed both validations and merged as `219184267972c596e7b61bbb29f0fab1b6472c9b`.
The previous controller is retained as `deploy.py.before-single-runtime-rollback-20260909`.
The helper refuses to run until the host artifact matches. Candidate devices, capabilities, read-only root,
privilege, security options and tmpfs must match the inspected original exactly.
It rejects stale selected-project configuration after acquiring the project lock.
Preparation also records selected-project configuration and original state hashes;
cutover compares both before making any production change.

Simulated failure tests cover interruption after original stop, after rename and
after receipt/config write before journal commit. Negative adapter tests cover
unrelated candidate nonce/project/IDs and binding broker verification to the prepared
immutable image. Offline Compose tests verify effective clone mounts and missing-input
failures. These tests do not claim a live migration or an end-to-end Docker rollback.

The protected-main approval/release gate for the app overrides and other prerequisite
branches must be satisfied before selecting a preparation SHA. Never run preparation
against an unmerged workaround or claim these local tests prove a live migration.

## Verified production outcome, 2026-09-09

VacatureRadar was prepared and committed from merged revision
`38e87678e1c5b6aa6de2cb49e49366fce8a3cbf2`, image
`sha256:c410c5ec26deb0cbd6a2992d4741e6c356f537d087de7fade74f214a747cca5a`.
The protected journal is under the controller's
`legacy-migrations/vacatureradar/legacy-20260909-a/` directory and records
`committed`, `activationIntent:true`, `activated:true`. The original container
`VacatureRadar-legacy-legacy-20260909-a` remains stopped with its original data.
Docker health, HTTP readiness, the login page and running Celery processes were
checked after activation. Normal Gitea deployment run9748 succeeded against the
same verified revision. No post-commit rollback was attempted.

The initial helper's Docker JSON template incorrectly grouped `index` arguments.
Preparation rejected this before production changes. Revision `4d1fc0d` fixed it,
with a regression test and successful read-only qualification against real Docker.
The actual cutover used that reviewed helper revision. Three additional Linux
storage tests used isolated temporary fixtures, not application data.

LumaOps has **not** been migrated. Its original runtime has `AUTH_ENABLED=false`
and no admin token; the merged target requires authentication on its production
network binding. Preparation stopped before building or changing production.
Provisioning secure login needs an explicit operator decision; never disable the
target's authentication guard or describe the old runtime as a verified new build.
Failed preparation artifacts are retained; no LumaOps cutover journal exists.