Initial curated projectbrain source from 8a7b1e0651dbdab97d744cbd2e653ab3970ac6c5
This commit is contained in:
commit
5aa3e8d685
700 files changed
+132801
No files matched your search
@@ -0,0 +1 @@
|
||||
__pycache__/
|
||||
@@ -0,0 +1,206 @@
|
||||
# Controlled legacy deployment migration
|
||||
|
||||
`plan.py` is a **pure planner**. `execute.py prepare` is an explicit opt-in,
|
||||
build-only preparation command. Separate `cutover` and `recover` modes require
|
||||
`--apply` and the exact verified preparation identity. Tests use fixtures/offline
|
||||
Compose rendering. The first verified production migration is recorded below;
|
||||
test results alone must never be presented as proof of a production cutover.
|
||||
|
||||
Authoritative broker code: `Unraid-AppOps-Gateway/ops/gitea-deploy-broker/deploy.py`.
|
||||
Production installs it separately at
|
||||
`/mnt/pool/appdata/gitea-deploy-runners/controller/deploy.py`. Production projects
|
||||
configuration is intentionally outside Git. Compare normalized source before changes.
|
||||
|
||||
## Why standard retry/bootstrap cannot migrate these two
|
||||
|
||||
VacatureRadar and LumaOps have healthy legacy containers but schema3 failed state
|
||||
with no known live commit or artifacts. `capture_compose_rollback` rightly refuses
|
||||
unattested rollback. `bootstrap-legacy` requires exact historical successful legacy
|
||||
state and refuses schema3. Never invent a revision or a successful receipt.
|
||||
|
||||
Standard preflight overlays live appdata, and standard migration cleanup removes
|
||||
retained legacy containers. Neither is suitable for a rollback-retaining migration
|
||||
with unknown original provenance. Use a reviewed separate one-shot transaction.
|
||||
|
||||
## Required transaction design
|
||||
|
||||
1. Pin/validate exact Gitea commit and tree. Prepare a separate clean build context;
|
||||
external credentials remain outside it. Build before disrupting production.
|
||||
Require immutable OCI source/revision/tree/version/time labels, not container
|
||||
labels alone. Existing configurations do not enable these checks yet.
|
||||
2. Acquire the same project then global locks as the broker. Revalidate original
|
||||
container and immutable image IDs, healthy state, exact mount inventory, free
|
||||
disk space, candidate network/ports, and absence of attempt targets. Protect the
|
||||
original image with an immutable rollback reference. Persist a root-only journal
|
||||
and original configuration snapshot without exposing secrets in output.
|
||||
3. Stop the original, cold-copy and verify all writable data into new attempt-owned
|
||||
paths. VacatureRadar runs PostgreSQL17 and Django migrations on startup: restarting
|
||||
the old image on migrated data is NOT rollback. Its nested `local`, logs and media
|
||||
bind mounts share one cold copy. LumaOps has bind mounts plus an anonymous `/config`
|
||||
volume; include both with their nested destination semantics preserved.
|
||||
4. Retain/rename the exact original container. Start only the candidate under a
|
||||
distinct Compose project (`<slug>-managed`), preserving production name/ports.
|
||||
The candidate mounts clones exclusively. A distinct project prevents Compose from
|
||||
adopting/removing retained original containers or failing exact-container-set checks.
|
||||
5. Verify immutable built image IDs, provenance and Docker health plus representative
|
||||
HTTP behavior. Only then obtain the real broker receipt and commit broker config
|
||||
(exact source, external env, explicit cloned data paths, new project identity,
|
||||
OCI verification) and state. Keep the originals and cold-copy journal available.
|
||||
6. On any failure before commit, remove only recorded attempt-created container IDs
|
||||
after identity checks, restore broker configuration, and rename/restart the exact
|
||||
original container with original untouched data. Verify old health. A failed restore
|
||||
is reported as unknown/failed, never as healthy. Keep partial artifacts for recovery.
|
||||
|
||||
Process-crash recovery needs a durable phase journal and idempotent identity checks;
|
||||
exception handlers alone are insufficient. Source-tree, data, broker configuration,
|
||||
runtime and receipt transitions must be separately recorded. Do not automatically
|
||||
clean originals after success. Retention cleanup needs separate authorization.
|
||||
|
||||
`build_plan` accepts a strict sanitized runtime projection: `name`, `containerId`,
|
||||
`imageId`, `running`, `health`, `composeProject`, and `mounts` (type/source/destination/rw).
|
||||
Never feed or print full Docker inspect/environment output. This planner validates
|
||||
paths lexically; the executor also resolves roots, rejects symlinks/special files,
|
||||
verifies backup checksums/ownership and revalidates state while locks are held.
|
||||
|
||||
Run: `python -m unittest discover -s scripts/legacy-deployment-migration -p 'test_*.py'`.
|
||||
|
||||
## Build-only preparation
|
||||
|
||||
After the two tracked `docker-compose.migration.yml` files are reviewed and merged,
|
||||
`execute.py prepare <slug> <full-current-branch-sha> <unique-attempt> --container-id
|
||||
<expected-id> --image-id <expected-immutable-id> --apply` prepares on the controller.
|
||||
Without `--apply` it performs no work. It validates the reviewed broker artifact,
|
||||
current branch head and original healthy container identity, then holds the normal
|
||||
project/global locks while extracting an isolated exact source tree and building an
|
||||
candidate image through the exact merged Compose configuration. OCI labels are
|
||||
checked on its immutable image ID. Source, original runtime and mounts are rechecked
|
||||
after build. Runtime environment is captured privately for Compose interpolation;
|
||||
credentials remain outside the build context and are not supplied as build arguments.
|
||||
Root-only journal/logs retain interrupted/failed attempts; a repeated
|
||||
attempt refuses to overwrite them. No production configuration/state/data is written.
|
||||
|
||||
The build-only journal is NOT a live receipt. `cutover` starts that same image using
|
||||
Compose `up --no-build --pull never`, then normal broker verification binds the
|
||||
observed runtime to its prepared immutable image ID. No rebuild occurs during downtime.
|
||||
Only a genuine normal broker receipt can be committed.
|
||||
|
||||
Tracked Compose override field contract (all required, no live-data defaults):
|
||||
|
||||
- Shared: `MIGRATION_ATTEMPT`, `MIGRATION_IMAGE`, `MIGRATION_SOURCE_REVISION`, `MIGRATION_SOURCE_TREE`,
|
||||
`MIGRATION_BUILD_ID`, `MIGRATION_BUILD_DATE`.
|
||||
- VacatureRadar: `MIGRATION_LOCAL_ROOT`, `VACATURERADAR_ENV_FILE`.
|
||||
- LumaOps: `MIGRATION_CONFIG_ROOT` (anonymous `/config` clone),
|
||||
`MIGRATION_OPENRGB_ROOT`, `MIGRATION_LUMA_CONFIG_ROOT`, `MIGRATION_DATA_ROOT`,
|
||||
`MIGRATION_LOGS_ROOT`, `MIGRATION_ENV_FILE`.
|
||||
|
||||
Pass the protected external runtime env file as Compose `--env-file` as well,
|
||||
preserving base-file interpolation values. Before cutover validate rendered absolute
|
||||
mounts against the plan, reject originals, and verify `create_host_path:false` so a
|
||||
missing copy cannot silently become an empty data directory. The normal broker
|
||||
requires Compose files to exist in the committed tree, which is why these overrides
|
||||
are tracked app files rather than a generated external override.
|
||||
|
||||
## Cutover and recovery
|
||||
|
||||
Use the same slug, revision, attempt and expected original IDs as preparation:
|
||||
`execute.py cutover ... --apply`. Selected project configuration/state hashes,
|
||||
source, image, runtime env, original IDs/health/mounts and rendered clone mounts/ports
|
||||
are revalidated before stopping. Both normal deployment locks remain held throughout.
|
||||
|
||||
The durable journal records intent before `stopping`, `copying`, `renaming`,
|
||||
`deploying`, `verifying`, and `committing`. Cold copies use `cp -a --reflink=auto`,
|
||||
filesystem sync and SHA256/ownership/mode/xattr/hardlink verification. The original
|
||||
is retained under a distinct name and immutable image tag. Broker config/state and
|
||||
original container configuration are saved root-only. The candidate never mounts
|
||||
original data. The new broker project uses dynamic revision/tree/build placeholders
|
||||
so subsequent normal deployments do not reuse frozen provenance labels.
|
||||
|
||||
An ordinary failure attempts verified recovery automatically. After abrupt process
|
||||
termination use `execute.py recover ... --apply`: it accepts only the saved original
|
||||
or candidate config/state, checks rollback snapshot integrity, and removes only the
|
||||
exact owned candidate (nonce, Compose project, recorded ID where available). It
|
||||
restores original config/state, canonical name and original untouched data, then
|
||||
verifies original immutable image/mounts, Docker health and HTTP readiness. Candidate
|
||||
data, logs and all originals are retained. Unrelated containers/config/state or
|
||||
changed original data block automatic recovery rather than be overwritten/deleted.
|
||||
|
||||
Recovery after the final `committed` journal marker reverifies production and resumes activation; it
|
||||
does not roll back potentially newer application writes. Recovery before that marker
|
||||
restores the original, including a crash after receipt/config write but before final
|
||||
journal commit. Failed recovery remains explicitly `recovery_failed`, never healthy.
|
||||
|
||||
Independent review and actual prepared-image/maintenance-window checks are required
|
||||
before first invocation. Health paths were probed read-only and returned HTTP 200 on
|
||||
2026-09-09: VacatureRadar port1226 `/health/ready/`, LumaOps port1223 `/health/ready`.
|
||||
These endpoints supplement, not replace, immutable image and normal receipt checks.
|
||||
|
||||
VacatureRadar's reviewed source must contain both the supervised Celery startup hold
|
||||
and first-position HTTP maintenance middleware. The helper writes an attempt-bound
|
||||
`.migration-worker-hold` only into the verified cold clone before candidate start.
|
||||
Ordinary traffic receives 503; exact GET/HEAD readiness/liveness probes remain available.
|
||||
After durable commit, activation journals intent, removes only the matching marker,
|
||||
fsyncs its directory, and records completion. Activation failure stays committed:
|
||||
explicit recovery resumes release and never rolls back after workers can run.
|
||||
A missing marker without durable activation intent fails closed.
|
||||
|
||||
Both tracked overrides require `MIGRATION_NETWORK`, bound to the exact inspected
|
||||
original `<slug>_default` network. Different/additional original networks fail closed;
|
||||
the rendered candidate must retain that network. Network/device
|
||||
compatibility and app-specific migration behavior still require operator review;
|
||||
the helper is not a generalized transaction system for external services.
|
||||
|
||||
LumaOps has no application maintenance gate in this migration. Its candidate can
|
||||
serve requests and interact with RGB hardware before the receipt is committed.
|
||||
Cold-data rollback restores the original application/container/data, but cannot
|
||||
undo external hardware effects or preserve candidate writes from that interval.
|
||||
Perform its cutover in the approved maintenance window with no concurrent user
|
||||
changes; do not describe this as a zero-effect or zero-data-loss transaction.
|
||||
|
||||
## Handoff safety boundary (2026-09-09)
|
||||
|
||||
The guarded broker artifact expected by `execute.py` has normalized SHA256
|
||||
`b5a0467f53acba5c783781a22bf81cf99f8b96ae99535460ad6236fac038330d`.
|
||||
This reviewed broker revision was installed atomically on 2026-09-09 after
|
||||
AppOps PR30 passed both validations and merged as `219184267972c596e7b61bbb29f0fab1b6472c9b`.
|
||||
The previous controller is retained as `deploy.py.before-single-runtime-rollback-20260909`.
|
||||
The helper refuses to run until the host artifact matches. Candidate devices, capabilities, read-only root,
|
||||
privilege, security options and tmpfs must match the inspected original exactly.
|
||||
It rejects stale selected-project configuration after acquiring the project lock.
|
||||
Preparation also records selected-project configuration and original state hashes;
|
||||
cutover compares both before making any production change.
|
||||
|
||||
Simulated failure tests cover interruption after original stop, after rename and
|
||||
after receipt/config write before journal commit. Negative adapter tests cover
|
||||
unrelated candidate nonce/project/IDs and binding broker verification to the prepared
|
||||
immutable image. Offline Compose tests verify effective clone mounts and missing-input
|
||||
failures. These tests do not claim a live migration or an end-to-end Docker rollback.
|
||||
|
||||
The protected-main approval/release gate for the app overrides and other prerequisite
|
||||
branches must be satisfied before selecting a preparation SHA. Never run preparation
|
||||
against an unmerged workaround or claim these local tests prove a live migration.
|
||||
|
||||
## Verified production outcome, 2026-09-09
|
||||
|
||||
VacatureRadar was prepared and committed from merged revision
|
||||
`38e87678e1c5b6aa6de2cb49e49366fce8a3cbf2`, image
|
||||
`sha256:c410c5ec26deb0cbd6a2992d4741e6c356f537d087de7fade74f214a747cca5a`.
|
||||
The protected journal is under the controller's
|
||||
`legacy-migrations/vacatureradar/legacy-20260909-a/` directory and records
|
||||
`committed`, `activationIntent:true`, `activated:true`. The original container
|
||||
`VacatureRadar-legacy-legacy-20260909-a` remains stopped with its original data.
|
||||
Docker health, HTTP readiness, the login page and running Celery processes were
|
||||
checked after activation. Normal Gitea deployment run9748 succeeded against the
|
||||
same verified revision. No post-commit rollback was attempted.
|
||||
|
||||
The initial helper's Docker JSON template incorrectly grouped `index` arguments.
|
||||
Preparation rejected this before production changes. Revision `4d1fc0d` fixed it,
|
||||
with a regression test and successful read-only qualification against real Docker.
|
||||
The actual cutover used that reviewed helper revision. Three additional Linux
|
||||
storage tests used isolated temporary fixtures, not application data.
|
||||
|
||||
LumaOps has **not** been migrated. Its original runtime has `AUTH_ENABLED=false`
|
||||
and no admin token; the merged target requires authentication on its production
|
||||
network binding. Preparation stopped before building or changing production.
|
||||
Provisioning secure login needs an explicit operator decision; never disable the
|
||||
target's authentication guard or describe the old runtime as a verified new build.
|
||||
Failed preparation artifacts are retained; no LumaOps cutover journal exists.
|
||||
@@ -0,0 +1,221 @@
|
||||
"""Explicit fixed-target preparation, journaled cutover and crash recovery.
|
||||
|
||||
Run on the reviewed Unraid controller host. Builds a verified exact-source candidate
|
||||
Preparation leaves live config/state/containers/data untouched. Cutover is a
|
||||
separate --apply operation requiring a reviewed, verified preparation. Originals
|
||||
and failed candidate data are retained; recover never undoes a committed migration.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
from contextlib import ExitStack
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import subprocess
|
||||
|
||||
from plan import ATTEMPT, ID, SHA, TARGETS
|
||||
|
||||
|
||||
BROKER_PATH = Path("/mnt/pool/appdata/gitea-deploy-runners/controller/deploy.py")
|
||||
BROKER_SHA = "b5a0467f53acba5c783781a22bf81cf99f8b96ae99535460ad6236fac038330d"
|
||||
JOURNAL_ROOT = Path("/mnt/pool/appdata/gitea-deploy-runners/legacy-migrations")
|
||||
DOCKERFILES = {"vacatureradar": "Dockerfile.unraid", "lumaops": "Dockerfile"}
|
||||
|
||||
|
||||
def durable_json(path, payload):
|
||||
temporary = path.with_suffix(".tmp")
|
||||
descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
with os.fdopen(descriptor, "w") as stream:
|
||||
json.dump(payload, stream, sort_keys=True)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
os.replace(temporary, path)
|
||||
descriptor = os.open(path.parent, os.O_RDONLY)
|
||||
try:
|
||||
os.fsync(descriptor)
|
||||
finally:
|
||||
os.close(descriptor)
|
||||
|
||||
|
||||
def require_real_child(path, parent):
|
||||
resolved = path.resolve()
|
||||
root = parent.resolve()
|
||||
if path.is_symlink() or resolved == root or not resolved.is_relative_to(root):
|
||||
raise ValueError("Candidate path is not a private child of the fixed root")
|
||||
return resolved
|
||||
|
||||
|
||||
def checked(command, *, log=None, timeout=30, env=None, cwd=None):
|
||||
# Do not inherit application credentials into builds or print raw subprocess output.
|
||||
environment = {key: os.environ[key] for key in ("PATH", "HOME", "DOCKER_CONFIG") if key in os.environ}
|
||||
result = subprocess.run(command, env=environment if env is None else env, cwd=cwd, stdout=log or subprocess.PIPE,
|
||||
stderr=log or subprocess.PIPE, text=log is None,
|
||||
timeout=timeout, check=False)
|
||||
if result.returncode:
|
||||
raise RuntimeError("Bounded preparation command failed; inspect the protected attempt log")
|
||||
return result.stdout if log is None else ""
|
||||
|
||||
|
||||
def docker_json_template(fields):
|
||||
# Go templates otherwise pass index/map/key as three separate json args.
|
||||
return "\n".join("{{json (" + field + ")}}" for field in fields)
|
||||
|
||||
|
||||
def original_identity(name, runner=checked):
|
||||
fields = [".Id", ".Image", ".State.Running", ".State.Health.Status",
|
||||
'index .Config.Labels "com.docker.compose.project"']
|
||||
template = docker_json_template(fields)
|
||||
values = runner(["docker", "inspect", "--format", template, name]).splitlines()
|
||||
if len(values) != len(fields):
|
||||
raise ValueError("Unexpected sanitized container inspection")
|
||||
return dict(zip(("containerId", "imageId", "running", "health", "composeProject"),
|
||||
(json.loads(value) for value in values)))
|
||||
|
||||
|
||||
def assert_original(observed, slug, container_id, image_id):
|
||||
if observed != {"containerId": container_id, "imageId": image_id,
|
||||
"running": True, "health": "healthy", "composeProject": slug}:
|
||||
raise ValueError("Original runtime identity/health drift; preparation refused")
|
||||
|
||||
|
||||
def mount_inventory(target, runner=checked):
|
||||
raw = json.loads(runner(["docker", "inspect", "--format", "{{json .Mounts}}", target]))
|
||||
return sorted(({"type": item["Type"], "source": item["Source"],
|
||||
"destination": item["Destination"], "rw": item["RW"]} for item in raw),
|
||||
key=lambda item: item["destination"])
|
||||
|
||||
|
||||
def expected_labels(metadata):
|
||||
return {
|
||||
"org.opencontainers.image.source": metadata["repository"],
|
||||
"org.opencontainers.image.revision": metadata["revision"],
|
||||
"org.opencontainers.image.source-tree": metadata["tree"],
|
||||
"org.opencontainers.image.version": metadata["id"],
|
||||
"org.opencontainers.image.created": metadata["builtAt"],
|
||||
}
|
||||
|
||||
|
||||
def verify_image(image, metadata, runner=checked):
|
||||
image_id = runner(["docker", "image", "inspect", "--format", "{{.Id}}", image]).strip()
|
||||
if not re.fullmatch(r"sha256:[0-9a-f]{64}", image_id):
|
||||
raise ValueError("Candidate immutable image identity is invalid")
|
||||
for key, expected in expected_labels(metadata).items():
|
||||
actual = runner(["docker", "image", "inspect", "--format",
|
||||
'{{index .Config.Labels "' + key + '"}}', image_id]).strip()
|
||||
if actual != expected:
|
||||
raise ValueError("Candidate immutable provenance did not verify")
|
||||
return image_id
|
||||
|
||||
|
||||
def prepare(broker, slug, revision, attempt, container_id, image_id, runner=checked):
|
||||
if (slug not in TARGETS or not SHA.fullmatch(revision) or not ATTEMPT.fullmatch(attempt)
|
||||
or not ID.fullmatch(container_id) or not re.fullmatch(r"sha256:[0-9a-f]{64}", image_id)):
|
||||
raise ValueError("Invalid fixed target, commit, attempt or expected runtime identity")
|
||||
repository = f"jens/{slug}"
|
||||
original_name = TARGETS[slug][0]
|
||||
workspace = require_real_child(JOURNAL_ROOT / slug / attempt, JOURNAL_ROOT)
|
||||
source = require_real_child(broker.EXACT_SOURCE_ROOT / f"{slug}-migration-{attempt}", broker.EXACT_SOURCE_ROOT)
|
||||
with ExitStack() as locks:
|
||||
locks.enter_context(broker.exclusive_lock(broker.STATE_ROOT / "locks" / f"{slug}.lock", "migration project"))
|
||||
locks.enter_context(broker.exclusive_lock(broker.GLOBAL_DEPLOY_LOCK, "migration preparation build"))
|
||||
# Read config only after lock acquisition, unlike starting another broker process.
|
||||
project = dict(broker.load_projects()[repository])
|
||||
repo = broker.repository_path(repository, project)
|
||||
broker.validate_commit(repo, str(project["branch"]), revision)
|
||||
head = broker.output(["git", f"--git-dir={repo}", "rev-parse", f"refs/heads/{project['branch']}"])
|
||||
if head != revision:
|
||||
raise ValueError("Migration candidate must be the current configured branch head")
|
||||
assert_original(original_identity(original_name, runner), slug, container_id, image_id)
|
||||
if workspace.exists() or source.exists():
|
||||
raise ValueError("Attempt artifacts already exist; retain them and review before a new attempt")
|
||||
metadata = broker.deployment_metadata(repo, revision, repository)
|
||||
if not broker.git_path_exists(repo, revision, "docker-compose.migration.yml"):
|
||||
raise ValueError("Candidate commit does not contain the reviewed migration Compose contract")
|
||||
if not broker.git_path_exists(repo, revision, DOCKERFILES[slug]):
|
||||
raise ValueError("Candidate Dockerfile missing")
|
||||
workspace.mkdir(parents=True, mode=0o700)
|
||||
workspace.chmod(0o700)
|
||||
journal_path = workspace / "prepare.json"
|
||||
journal = {"schema": "legacy-prepare.v1", "phase": "preparing", "repository": repository,
|
||||
"revision": revision, "metadata": metadata, "source": str(source),
|
||||
"originalContainerId": container_id, "originalImageId": image_id,
|
||||
"originalMounts": mount_inventory(container_id, runner),
|
||||
"projectConfigSha256": hashlib.sha256(json.dumps(project, sort_keys=True).encode()).hexdigest(),
|
||||
"stateSha256": hashlib.sha256((broker.STATE_ROOT / f"{slug}.json").read_bytes()).hexdigest(),
|
||||
"originalProvenance": "unknown-legacy", "productionMutated": False}
|
||||
durable_json(journal_path, journal)
|
||||
try:
|
||||
broker.sync_source(repo, revision, source, slug, exact=True)
|
||||
if not broker.exact_source_matches_revision(repo, revision, source):
|
||||
raise ValueError("Candidate source is not the exact committed tree")
|
||||
from transaction import prepare_compose
|
||||
candidate, command, cwd, environment, plan, nonce = prepare_compose(
|
||||
broker, slug, revision, attempt, project, journal, workspace, runner)
|
||||
image = environment["MIGRATION_IMAGE"]
|
||||
journal.update(candidateProject=candidate, plan=plan, nonce=nonce)
|
||||
journal["runtimeEnvSha256"] = hashlib.sha256((workspace / "runtime.env").read_bytes()).hexdigest()
|
||||
journal.update(phase="building", candidateImage=image)
|
||||
durable_json(journal_path, journal)
|
||||
descriptor = os.open(workspace / "build.log", os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
with os.fdopen(descriptor, "wb") as log:
|
||||
runner(command, cwd=cwd, env=environment, log=log, timeout=3600)
|
||||
log.flush()
|
||||
os.fsync(log.fileno())
|
||||
immutable = verify_image(image, metadata, runner)
|
||||
if not broker.exact_source_matches_revision(repo, revision, source):
|
||||
raise ValueError("Candidate source changed during build")
|
||||
assert_original(original_identity(original_name, runner), slug, container_id, image_id)
|
||||
if mount_inventory(container_id, runner) != journal["originalMounts"]:
|
||||
raise ValueError("Original mount inventory changed during preparation")
|
||||
journal.update(phase="build_verified", candidateImageId=immutable)
|
||||
durable_json(journal_path, journal)
|
||||
return {"phase": journal["phase"], "repository": repository, "revision": revision,
|
||||
"candidateImageId": immutable, "journal": str(journal_path),
|
||||
"productionMutated": False, "cutoverAvailable": True}
|
||||
except Exception:
|
||||
journal["phase"] = "preparation_failed"
|
||||
durable_json(journal_path, journal)
|
||||
raise
|
||||
|
||||
|
||||
def load_broker():
|
||||
if hashlib.sha256(BROKER_PATH.read_bytes().replace(b"\r\n", b"\n")).hexdigest() != BROKER_SHA:
|
||||
raise ValueError("Live broker source differs from the reviewed artifact")
|
||||
spec = importlib.util.spec_from_file_location("reviewed_deploy_broker", BROKER_PATH)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("mode", choices=["prepare", "cutover", "recover"])
|
||||
parser.add_argument("slug", choices=sorted(TARGETS))
|
||||
parser.add_argument("revision")
|
||||
parser.add_argument("attempt")
|
||||
parser.add_argument("--container-id", required=True)
|
||||
parser.add_argument("--image-id", required=True)
|
||||
parser.add_argument("--apply", action="store_true")
|
||||
args = parser.parse_args()
|
||||
if not args.apply:
|
||||
print(json.dumps({"apply": False, "mode": args.mode, "productionMutated": False}))
|
||||
return
|
||||
if not hasattr(os, "geteuid") or os.geteuid() != 0:
|
||||
raise SystemExit("Run only as root on the configured Unraid controller host")
|
||||
try:
|
||||
broker = load_broker()
|
||||
if args.mode == "prepare":
|
||||
result = prepare(broker, args.slug, args.revision, args.attempt, args.container_id, args.image_id)
|
||||
else:
|
||||
from transaction import operate
|
||||
result = operate(broker, args.mode, args.slug, args.revision, args.attempt, args.container_id, args.image_id)
|
||||
except Exception:
|
||||
raise SystemExit("Operation failed. Inspect the protected attempt journal; do not assume production or rollback succeeded. Use explicit recover for an unfinished cutover.") from None
|
||||
print(json.dumps(result))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,103 @@
|
||||
"""Pure, fail-closed migration planning; this module cannot mutate Docker or files.
|
||||
|
||||
Accept only the sanitized inspect projection documented in README.md. A plan is
|
||||
not deployment authorization, a snapshot, a receipt, or proof of successful backup.
|
||||
"""
|
||||
|
||||
import re
|
||||
from pathlib import PurePosixPath
|
||||
|
||||
|
||||
TARGETS = {
|
||||
"vacatureradar": ("VacatureRadar", "/mnt/pool/appdata/VacatureRadar"),
|
||||
"lumaops": ("lumaops", "/mnt/pool/appdata/lumaops"),
|
||||
}
|
||||
SHA = re.compile(r"[0-9a-f]{40}")
|
||||
ID = re.compile(r"[0-9a-f]{64}")
|
||||
ATTEMPT = re.compile(r"[a-z0-9][a-z0-9-]{7,47}")
|
||||
|
||||
|
||||
def _under(path, parent):
|
||||
value = PurePosixPath(path)
|
||||
return (
|
||||
value.is_absolute()
|
||||
and ".." not in value.parts
|
||||
and value != PurePosixPath(parent)
|
||||
and value.is_relative_to(parent)
|
||||
)
|
||||
|
||||
|
||||
def build_plan(slug, revision, attempt, observed):
|
||||
"""Bind one target to retained runtime identity and isolated candidate data."""
|
||||
if slug not in TARGETS or not SHA.fullmatch(revision) or not ATTEMPT.fullmatch(attempt):
|
||||
raise ValueError("Invalid fixed target, full revision, or attempt identifier")
|
||||
expected_name, appdata = TARGETS[slug]
|
||||
keys = {"name", "containerId", "imageId", "running", "health", "composeProject", "mounts"}
|
||||
if set(observed) != keys:
|
||||
raise ValueError("Expected only the sanitized runtime projection")
|
||||
if (
|
||||
observed["name"] != expected_name
|
||||
or observed["composeProject"] != slug
|
||||
or not ID.fullmatch(observed["containerId"])
|
||||
or not re.fullmatch(r"sha256:[0-9a-f]{64}", observed["imageId"])
|
||||
or observed["running"] is not True
|
||||
or observed["health"] != "healthy"
|
||||
):
|
||||
raise ValueError("Legacy runtime identity or health does not match")
|
||||
mounts = observed["mounts"]
|
||||
if not mounts or len(mounts) > 16:
|
||||
raise ValueError("Expected bounded persistent mounts")
|
||||
destinations = set()
|
||||
sources = set()
|
||||
for mount in mounts:
|
||||
if set(mount) != {"type", "source", "destination", "rw"}:
|
||||
raise ValueError("Unexpected mount fields")
|
||||
destination = mount["destination"]
|
||||
if not _under(destination, "/") or destination in destinations:
|
||||
raise ValueError("Invalid or duplicate mount destination")
|
||||
destinations.add(destination)
|
||||
source = mount["source"]
|
||||
if mount["type"] == "bind":
|
||||
valid = _under(source, appdata)
|
||||
elif mount["type"] == "volume":
|
||||
valid = bool(re.fullmatch(r"/var/lib/docker/volumes/[0-9a-f]{64}/_data", source))
|
||||
else:
|
||||
valid = False
|
||||
if not valid or mount["rw"] is not True:
|
||||
raise ValueError("Unreviewed source, mount type, or access mode")
|
||||
sources.add(source)
|
||||
# Clone a parent only once; preserve nested bind destinations against the
|
||||
# corresponding cloned subdirectory, not unrelated independent snapshots.
|
||||
roots = sorted(source for source in sources if not any(
|
||||
other != source and _under(source, other) for other in sources
|
||||
))
|
||||
candidate_root = f"/mnt/pool/appdata/gitea-deploy-runners/legacy-migrations/{slug}/{attempt}"
|
||||
clones = {source: f"{candidate_root}/data/{index}" for index, source in enumerate(roots)}
|
||||
candidate_mounts = []
|
||||
for mount in mounts:
|
||||
root = next(source for source in roots if mount["source"] == source or _under(mount["source"], source))
|
||||
relative = PurePosixPath(mount["source"]).relative_to(root)
|
||||
candidate_mounts.append({
|
||||
"source": str(PurePosixPath(clones[root]) / relative),
|
||||
"destination": mount["destination"],
|
||||
"type": "bind", "rw": True,
|
||||
})
|
||||
return {
|
||||
"schema": "legacy-migration-plan.v1", "executable": False,
|
||||
"repository": f"jens/{slug}", "candidateRevision": revision,
|
||||
"originalProvenance": "unknown-legacy", "originalRevision": None,
|
||||
"originalContainerId": observed["containerId"], "originalImageId": observed["imageId"],
|
||||
"originalName": expected_name, "retainedName": f"{expected_name}-legacy-{attempt}",
|
||||
"candidateComposeProject": f"{slug}-managed",
|
||||
"workspace": candidate_root,
|
||||
"coldCopyRoots": [{"original": root, "candidate": clones[root]} for root in roots],
|
||||
"candidateMounts": candidate_mounts,
|
||||
"requiredGates": [
|
||||
"exact-commit-and-tree-validated", "candidate-built-before-cutover",
|
||||
"immutable-oci-provenance-verified", "project-and-global-locks-held",
|
||||
"original-identity-revalidated", "legacy-stopped-before-data-copy",
|
||||
"cold-copy-checksums-verified", "original-data-never-mounted-by-candidate",
|
||||
"candidate-container-ids-recorded", "docker-and-http-health-verified",
|
||||
"real-broker-receipt-verified", "broker-config-and-state-committed",
|
||||
],
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
"""Private durable journal and fail-closed cold-copy verification primitives."""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import stat
|
||||
import tempfile
|
||||
|
||||
|
||||
def digest(data):
|
||||
return hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def private_write(path, data):
|
||||
"""Unique temporary files let explicit recovery proceed after process crashes."""
|
||||
path = Path(path)
|
||||
if path.is_symlink():
|
||||
raise ValueError("Refusing a symlink journal/configuration target")
|
||||
descriptor, temporary = tempfile.mkstemp(prefix=f".{path.name}-", dir=path.parent)
|
||||
with os.fdopen(descriptor, "wb") as stream:
|
||||
stream.write(data)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
os.replace(temporary, path)
|
||||
descriptor = os.open(path.parent, os.O_RDONLY)
|
||||
try:
|
||||
os.fsync(descriptor)
|
||||
finally:
|
||||
os.close(descriptor)
|
||||
|
||||
|
||||
def private_json(path, value):
|
||||
private_write(path, json.dumps(value, sort_keys=True).encode())
|
||||
|
||||
|
||||
def dotenv(entries):
|
||||
"""Do not guess escaping for unsupported environment values; never log them."""
|
||||
import re
|
||||
result = {}
|
||||
for entry in entries:
|
||||
key, separator, value = entry.partition("=")
|
||||
if not separator or not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", key):
|
||||
raise ValueError("Unsupported runtime environment entry")
|
||||
if key in result or any(character in value for character in "\x00\n\r'\\"):
|
||||
raise ValueError("Runtime environment needs reviewed dotenv escaping")
|
||||
result[key] = value
|
||||
return "".join(f"{key}='{value}'\n" for key, value in sorted(result.items())).encode()
|
||||
|
||||
|
||||
def tree_manifest(root):
|
||||
"""Hash content plus ownership/modes/xattrs; reject symlinks and special files.
|
||||
|
||||
Preserving an unknown symlink could let the candidate write original data.
|
||||
Hardlink equivalence groups are compared without depending on inode numbers.
|
||||
"""
|
||||
root = Path(root)
|
||||
if root.is_symlink() or not root.is_dir():
|
||||
raise ValueError("Cold-copy root must be a real directory")
|
||||
records = {}
|
||||
inodes = {}
|
||||
for path in [root, *sorted(root.rglob("*"))]:
|
||||
info = path.lstat()
|
||||
relative = str(path.relative_to(root))
|
||||
if not stat.S_ISREG(info.st_mode) and not stat.S_ISDIR(info.st_mode):
|
||||
raise ValueError("Cold-copy tree contains an unreviewed symlink/special file")
|
||||
record = {"mode": stat.S_IMODE(info.st_mode), "uid": info.st_uid, "gid": info.st_gid}
|
||||
if stat.S_ISREG(info.st_mode):
|
||||
hasher = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for block in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
hasher.update(block)
|
||||
record.update(kind="file", size=info.st_size, sha256=hasher.hexdigest())
|
||||
group = (info.st_dev, info.st_ino)
|
||||
record["hardlink"] = inodes.setdefault(group, relative)
|
||||
else:
|
||||
record["kind"] = "directory"
|
||||
if hasattr(os, "listxattr"):
|
||||
record["xattrs"] = {name: digest(os.getxattr(path, name)) for name in sorted(os.listxattr(path))}
|
||||
records[relative] = record
|
||||
return records
|
||||
|
||||
|
||||
def clone_cold(source, target, runner):
|
||||
source, target = Path(source), Path(target)
|
||||
if target.exists() or target.is_symlink():
|
||||
raise ValueError("Cold-copy target already exists")
|
||||
before = tree_manifest(source)
|
||||
target.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
# Destination does not exist: cp -a copies root metadata as well as children.
|
||||
runner(["cp", "-a", "--reflink=auto", "--", str(source), str(target)], timeout=3600)
|
||||
runner(["sync", "-f", "--", str(target)], timeout=3600)
|
||||
copied = tree_manifest(target)
|
||||
if before != copied or before != tree_manifest(source):
|
||||
raise ValueError("Cold copy content/metadata does not match unchanged original")
|
||||
return digest(json.dumps(before, sort_keys=True).encode())
|
||||
@@ -0,0 +1,84 @@
|
||||
"""Offline Compose rendering; never builds images or starts containers."""
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import unittest
|
||||
|
||||
|
||||
PROJECTS = Path(__file__).resolve().parents[3]
|
||||
CASES = {
|
||||
"VacatureRadar": ("app", ["docker-compose.unraid.yml"], {
|
||||
"MIGRATION_LOCAL_ROOT": "/migration-fixture/local",
|
||||
"VACATURERADAR_ENV_FILE": "/migration-fixture/runtime.env",
|
||||
}, {"/app/local", "/app/media", "/app/logs"}),
|
||||
"LumaOps": ("lumaops", ["docker-compose.yml", "docker-compose.unraid.yml"], {
|
||||
"MIGRATION_CONFIG_ROOT": "/migration-fixture/config",
|
||||
"MIGRATION_OPENRGB_ROOT": "/migration-fixture/openrgb",
|
||||
"MIGRATION_LUMA_CONFIG_ROOT": "/migration-fixture/luma-config",
|
||||
"MIGRATION_DATA_ROOT": "/migration-fixture/data",
|
||||
"MIGRATION_LOGS_ROOT": "/migration-fixture/logs",
|
||||
"MIGRATION_ENV_FILE": "/migration-fixture/runtime.env",
|
||||
}, {"/config", "/config/openrgb", "/config/lumaops", "/data", "/logs"}),
|
||||
}
|
||||
|
||||
|
||||
@unittest.skipUnless(shutil.which("docker"), "Docker Compose CLI is required; no daemon used")
|
||||
class ComposeTests(unittest.TestCase):
|
||||
def render(self, name, missing=None):
|
||||
_, files, variables, _ = CASES[name]
|
||||
environment = dict(os.environ)
|
||||
for key in list(environment):
|
||||
if key.startswith("MIGRATION_") or key.startswith("VACATURERADAR_"):
|
||||
environment.pop(key)
|
||||
environment.update({
|
||||
"MIGRATION_IMAGE": "migration-fixture:candidate",
|
||||
"MIGRATION_NETWORK": name.lower() + "_default",
|
||||
"MIGRATION_ATTEMPT": "fixture-migration-attempt",
|
||||
"MIGRATION_SOURCE_REVISION": "a" * 40,
|
||||
"MIGRATION_SOURCE_TREE": "b" * 40,
|
||||
"MIGRATION_BUILD_ID": "a" * 12,
|
||||
"MIGRATION_BUILD_DATE": "2026-09-09T00:00:00.000Z",
|
||||
"LUMAOPS_ADMIN_TOKEN": "fixture-only-not-a-production-token-000000",
|
||||
**variables,
|
||||
})
|
||||
if missing:
|
||||
environment.pop(missing)
|
||||
command = ["docker", "compose", "--env-file", os.devnull]
|
||||
for file in [*files, "docker-compose.migration.yml"]:
|
||||
command.extend(["-f", file])
|
||||
command.extend(["config", "--no-env-resolution", "--no-path-resolution", "--format", "json"])
|
||||
return subprocess.run(command, cwd=PROJECTS / name, env=environment,
|
||||
capture_output=True, text=True, timeout=20)
|
||||
|
||||
def test_merged_mounts_are_exclusively_explicit_clones(self):
|
||||
for name, (service, _, _, destinations) in CASES.items():
|
||||
with self.subTest(name=name):
|
||||
result = self.render(name)
|
||||
self.assertEqual(result.returncode, 0, "Fixture Compose rendering failed")
|
||||
config = json.loads(result.stdout)
|
||||
runtime = config["services"][service]
|
||||
self.assertEqual(runtime["network_mode"], name.lower() + "_default")
|
||||
self.assertEqual(config["name"], name.lower() + "-managed")
|
||||
self.assertEqual({m["target"] for m in runtime["volumes"]}, destinations)
|
||||
for mount in runtime["volumes"]:
|
||||
self.assertEqual(mount["type"], "bind")
|
||||
self.assertTrue(mount["source"].startswith("/migration-fixture/"))
|
||||
self.assertFalse(mount["bind"]["create_host_path"])
|
||||
labels = runtime["build"]["labels"]
|
||||
self.assertEqual(labels["org.opencontainers.image.revision"], "a" * 40)
|
||||
self.assertEqual(labels["org.opencontainers.image.source-tree"], "b" * 40)
|
||||
self.assertEqual(labels["org.opencontainers.image.source"], "jens/" + name.lower())
|
||||
|
||||
def test_missing_required_inputs_fail_closed(self):
|
||||
for name, (_, _, variables, _) in CASES.items():
|
||||
for variable in [*variables, "MIGRATION_NETWORK", "MIGRATION_IMAGE", "MIGRATION_ATTEMPT", "MIGRATION_SOURCE_REVISION",
|
||||
"MIGRATION_SOURCE_TREE", "MIGRATION_BUILD_ID", "MIGRATION_BUILD_DATE"]:
|
||||
with self.subTest(name=name, variable=variable):
|
||||
self.assertNotEqual(self.render(name, variable).returncode, 0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,49 @@
|
||||
import unittest
|
||||
|
||||
from execute import assert_original, docker_json_template, expected_labels, original_identity, verify_image
|
||||
|
||||
|
||||
class ExecuteTests(unittest.TestCase):
|
||||
def test_template_groups_function_arguments_before_json_encoding(self):
|
||||
self.assertEqual(
|
||||
docker_json_template(['.Id', 'index .Config.Labels "com.docker.compose.project"']),
|
||||
'{{json (.Id)}}\n{{json (index .Config.Labels "com.docker.compose.project")}}',
|
||||
)
|
||||
|
||||
def test_inspection_never_requests_environment(self):
|
||||
calls = []
|
||||
|
||||
def runner(command):
|
||||
calls.append(command)
|
||||
return '"' + "a" * 64 + '"\n"sha256:' + "b" * 64 + '"\ntrue\n"healthy"\n"lumaops"'
|
||||
|
||||
observed = original_identity("lumaops", runner)
|
||||
assert_original(observed, "lumaops", "a" * 64, "sha256:" + "b" * 64)
|
||||
self.assertNotIn(".Config.Env", str(calls))
|
||||
self.assertEqual(calls[0][:2], ["docker", "inspect"])
|
||||
|
||||
def test_runtime_drift_blocks(self):
|
||||
with self.assertRaises(ValueError):
|
||||
assert_original({}, "lumaops", "a" * 64, "sha256:" + "b" * 64)
|
||||
|
||||
def test_image_labels_are_checked_on_immutable_id(self):
|
||||
metadata = {"repository": "jens/lumaops", "revision": "a" * 40,
|
||||
"tree": "b" * 40, "id": "a" * 12, "builtAt": "2026-09-09T00:00:00.000Z"}
|
||||
labels = expected_labels(metadata)
|
||||
calls = []
|
||||
|
||||
def runner(command):
|
||||
calls.append(command)
|
||||
if command[-2] == "{{.Id}}":
|
||||
return "sha256:" + "c" * 64
|
||||
key = command[-2].split('"')[1]
|
||||
return labels[key]
|
||||
|
||||
self.assertEqual(verify_image("candidate", metadata, runner), "sha256:" + "c" * 64)
|
||||
self.assertTrue(all(command[-1] == "sha256:" + "c" * 64 for command in calls[1:]))
|
||||
with self.assertRaises(ValueError):
|
||||
verify_image("candidate", metadata, lambda command: "invalid")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,61 @@
|
||||
import unittest
|
||||
|
||||
from plan import build_plan
|
||||
|
||||
|
||||
def observation():
|
||||
return {
|
||||
"name": "VacatureRadar", "containerId": "a" * 64,
|
||||
"imageId": "sha256:" + "b" * 64, "running": True,
|
||||
"health": "healthy", "composeProject": "vacatureradar",
|
||||
"mounts": [
|
||||
{"type": "bind", "source": "/mnt/pool/appdata/VacatureRadar/local",
|
||||
"destination": "/app/local", "rw": True},
|
||||
{"type": "bind", "source": "/mnt/pool/appdata/VacatureRadar/local/logs",
|
||||
"destination": "/app/logs", "rw": True},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
class PlanTests(unittest.TestCase):
|
||||
def test_nested_mounts_share_one_cold_copy(self):
|
||||
plan = build_plan("vacatureradar", "c" * 40, "migration-20260909", observation())
|
||||
self.assertEqual(len(plan["coldCopyRoots"]), 1)
|
||||
self.assertTrue(plan["candidateMounts"][1]["source"].endswith("/data/0/logs"))
|
||||
self.assertFalse(plan["executable"])
|
||||
self.assertIsNone(plan["originalRevision"])
|
||||
self.assertEqual(plan["originalProvenance"], "unknown-legacy")
|
||||
self.assertNotEqual(plan["candidateComposeProject"], "vacatureradar")
|
||||
|
||||
def test_rejects_runtime_drift_and_unsafe_paths(self):
|
||||
for key, value in [("name", "other"), ("containerId", "short"),
|
||||
("running", False), ("health", "starting")]:
|
||||
observed = observation()
|
||||
observed[key] = value
|
||||
with self.subTest(key=key), self.assertRaises(ValueError):
|
||||
build_plan("vacatureradar", "c" * 40, "migration-20260909", observed)
|
||||
for source in ["/", "/mnt/pool/appdata", "/mnt/pool/appdata/VacatureRadar/../other"]:
|
||||
observed = observation()
|
||||
observed["mounts"][0]["source"] = source
|
||||
with self.subTest(source=source), self.assertRaises(ValueError):
|
||||
build_plan("vacatureradar", "c" * 40, "migration-20260909", observed)
|
||||
|
||||
def test_rejects_secret_bearing_full_inspect(self):
|
||||
observed = observation()
|
||||
observed["Config"] = {"Env": ["fixture-only"]}
|
||||
with self.assertRaises(ValueError):
|
||||
build_plan("vacatureradar", "c" * 40, "migration-20260909", observed)
|
||||
|
||||
def test_anonymous_volume_is_cloned_not_reused(self):
|
||||
observed = observation()
|
||||
observed.update(name="lumaops", composeProject="lumaops", mounts=[
|
||||
{"type": "volume", "source": "/var/lib/docker/volumes/" + "d" * 64 + "/_data",
|
||||
"destination": "/config", "rw": True},
|
||||
])
|
||||
plan = build_plan("lumaops", "c" * 40, "migration-20260909", observed)
|
||||
self.assertEqual(plan["candidateMounts"][0]["type"], "bind")
|
||||
self.assertNotIn("/var/lib/docker/volumes/", plan["candidateMounts"][0]["source"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Linux filesystem qualification with isolated temporary data, never appdata."""
|
||||
import os
|
||||
from pathlib import Path
|
||||
import stat
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
from storage import clone_cold, private_write, tree_manifest
|
||||
|
||||
|
||||
@unittest.skipUnless(os.name == "posix", "Linux ownership/fsync qualification")
|
||||
class StorageTests(unittest.TestCase):
|
||||
def test_private_replacement_is_durable_and_private(self):
|
||||
with tempfile.TemporaryDirectory(prefix="migration-storage-test-") as directory:
|
||||
path = Path(directory) / "journal.json"
|
||||
private_write(path, b"first")
|
||||
private_write(path, b"second")
|
||||
self.assertEqual(path.read_bytes(), b"second")
|
||||
self.assertEqual(stat.S_IMODE(path.stat().st_mode), 0o600)
|
||||
target = Path(directory) / "linked"
|
||||
target.symlink_to(path)
|
||||
with self.assertRaises(ValueError):
|
||||
private_write(target, b"must not replace")
|
||||
self.assertEqual(path.read_bytes(), b"second")
|
||||
|
||||
def test_real_cold_copy_preserves_content_modes_and_hardlinks(self):
|
||||
with tempfile.TemporaryDirectory(prefix="migration-storage-test-") as directory:
|
||||
source = Path(directory) / "original"
|
||||
source.mkdir(mode=0o750)
|
||||
payload = source / "database"
|
||||
payload.write_bytes(b"representative fixture data")
|
||||
payload.chmod(0o640)
|
||||
os.link(payload, source / "hardlink")
|
||||
destination = Path(directory) / "candidate"
|
||||
|
||||
def runner(command, **kwargs):
|
||||
subprocess.run(command, check=True, capture_output=True, **kwargs)
|
||||
|
||||
checksum = clone_cold(source, destination, runner)
|
||||
self.assertEqual(tree_manifest(source), tree_manifest(destination))
|
||||
self.assertEqual(len(checksum), 64)
|
||||
self.assertNotEqual(payload.stat().st_ino, (destination / "database").stat().st_ino)
|
||||
(destination / "database").write_bytes(b"candidate mutation")
|
||||
self.assertEqual(payload.read_bytes(), b"representative fixture data")
|
||||
with self.assertRaises(ValueError):
|
||||
clone_cold(source, destination, runner)
|
||||
|
||||
def test_special_files_are_rejected_before_copy(self):
|
||||
with tempfile.TemporaryDirectory(prefix="migration-storage-test-") as directory:
|
||||
root = Path(directory)
|
||||
(root / "link").symlink_to("/tmp")
|
||||
with self.assertRaises(ValueError):
|
||||
tree_manifest(root)
|
||||
(root / "link").unlink()
|
||||
os.mkfifo(root / "fifo")
|
||||
with self.assertRaises(ValueError):
|
||||
tree_manifest(root)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,289 @@
|
||||
"""No Docker/network calls: exercise crash phases and fail-closed recovery gates."""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from storage import clone_cold, digest, dotenv, tree_manifest
|
||||
from transaction import (Production, STEPS, candidate_project,
|
||||
recover_transaction, run_transaction, validate_runtime_safety)
|
||||
|
||||
|
||||
class Crash(BaseException):
|
||||
pass
|
||||
|
||||
|
||||
class FakeOperations:
|
||||
def __init__(self, fail=None, crash=False):
|
||||
self.journal = {"phase": "snapshotted"}
|
||||
self.fail, self.crash = fail, crash
|
||||
self.original_running, self.renamed, self.candidate = True, False, False
|
||||
self.state, self.config = "original", "original"
|
||||
self.clones = False
|
||||
self.foreign = False
|
||||
self.saved = []
|
||||
|
||||
def save(self, phase):
|
||||
self.saved.append(phase)
|
||||
self.journal["phase"] = phase
|
||||
|
||||
def event(self, name):
|
||||
if self.fail == name:
|
||||
raise Crash() if self.crash else ValueError("fixture failure")
|
||||
|
||||
def stopping(self):
|
||||
self.original_running = False
|
||||
self.event("after_stop")
|
||||
|
||||
def copying(self):
|
||||
assert not self.original_running
|
||||
self.clones = True
|
||||
self.event("after_copy")
|
||||
|
||||
def renaming(self):
|
||||
self.renamed = True
|
||||
self.event("after_rename")
|
||||
|
||||
def deploying(self):
|
||||
self.candidate = True
|
||||
self.event("after_deploy")
|
||||
|
||||
def verifying(self):
|
||||
self.event("during_verify")
|
||||
|
||||
def committing(self):
|
||||
self.state = "candidate"
|
||||
self.event("after_receipt")
|
||||
self.config = "candidate"
|
||||
self.event("after_config")
|
||||
|
||||
def recover(self):
|
||||
if self.foreign:
|
||||
raise ValueError("foreign canonical container")
|
||||
self.candidate = False
|
||||
self.state = self.config = "original"
|
||||
self.renamed = False
|
||||
self.original_running = True
|
||||
|
||||
def verify_committed(self):
|
||||
assert self.state == self.config == "candidate"
|
||||
|
||||
def original(self, healthy=False):
|
||||
assert self.original_running
|
||||
|
||||
def http_health(self):
|
||||
pass
|
||||
|
||||
def activate(self):
|
||||
assert self.journal["phase"] == "committed"
|
||||
self.event("during_activation")
|
||||
self.journal["activated"] = True
|
||||
|
||||
|
||||
class TransactionTests(unittest.TestCase):
|
||||
def test_runtime_safety_parity_rejects_each_security_or_device_drift(self):
|
||||
host = {"Devices": [{"PathOnHost": "/dev/example", "PathInContainer": "/dev/example", "CgroupPermissions": "rwm"}],
|
||||
"CapAdd": ["CAP_CHOWN"], "CapDrop": ["ALL"], "ReadonlyRootfs": True,
|
||||
"SecurityOpt": ["no-new-privileges:true"], "Tmpfs": {"/tmp": "size=128m"}}
|
||||
service = {"devices": [{"source": "/dev/example", "target": "/dev/example", "permissions": "rwm"}],
|
||||
"cap_add": ["CHOWN"], "cap_drop": ["ALL"], "read_only": True,
|
||||
"security_opt": ["no-new-privileges:true"], "tmpfs": ["/tmp:size=128m"]}
|
||||
validate_runtime_safety(host, service)
|
||||
for key, value in (("devices", []), ("cap_add", []), ("cap_drop", []), ("read_only", False),
|
||||
("privileged", True), ("security_opt", []), ("tmpfs", [])):
|
||||
with self.subTest(key=key), self.assertRaises(ValueError):
|
||||
validate_runtime_safety(host, {**service, key: value})
|
||||
|
||||
def test_activation_failure_never_rolls_back_and_recovery_resumes(self):
|
||||
ops = FakeOperations(fail="during_activation")
|
||||
with self.assertRaises(ValueError):
|
||||
run_transaction(ops)
|
||||
self.assertEqual(ops.journal["phase"], "committed")
|
||||
self.assertTrue(ops.candidate)
|
||||
self.assertFalse(ops.original_running)
|
||||
ops.fail = None
|
||||
self.assertEqual(recover_transaction(ops), "committed")
|
||||
self.assertTrue(ops.journal["activated"])
|
||||
|
||||
def test_real_marker_release_requires_matching_nonce_and_resumes_after_unlink(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
ops = Production.__new__(Production)
|
||||
ops.workspace = Path(temporary).resolve()
|
||||
root = ops.workspace / "local"
|
||||
root.mkdir()
|
||||
ops.slug = "vacatureradar"
|
||||
ops.plan = {"candidateMounts": [{"destination": "/app/local", "source": str(root)}]}
|
||||
ops.journal = {"phase": "committed", "nonce": "correct", "candidateContainerId": "candidate"}
|
||||
ops.verify_committed = lambda: None
|
||||
ops.assert_candidate = lambda value: None
|
||||
ops.save = lambda phase: None
|
||||
marker = ops.hold_path()
|
||||
marker.write_bytes(b"wrong")
|
||||
with patch("transaction.private_read", side_effect=lambda path: path.read_bytes()):
|
||||
with self.assertRaises(ValueError):
|
||||
ops.activate()
|
||||
self.assertTrue(marker.exists())
|
||||
marker.write_bytes(ops.hold_bytes())
|
||||
with patch("transaction.os.open", return_value=123), patch("transaction.os.fsync"), patch("transaction.os.close"):
|
||||
ops.activate()
|
||||
self.assertFalse(marker.exists())
|
||||
self.assertTrue(ops.journal["activationIntent"])
|
||||
ops.activate()
|
||||
self.assertTrue(ops.journal["activated"])
|
||||
ops.journal.pop("activationIntent")
|
||||
with self.assertRaises(ValueError):
|
||||
ops.activate()
|
||||
|
||||
def test_intent_precedes_each_successful_phase(self):
|
||||
ops = FakeOperations()
|
||||
run_transaction(ops)
|
||||
self.assertEqual(ops.saved, [*STEPS, "committed"])
|
||||
self.assertFalse(ops.original_running)
|
||||
self.assertTrue(ops.renamed and ops.clones and ops.candidate)
|
||||
self.assertEqual(recover_transaction(ops), "committed")
|
||||
self.assertTrue(ops.candidate, "Recovery cannot discard writes after a committed migration")
|
||||
|
||||
def test_each_failure_restores_original_and_keeps_clones(self):
|
||||
for failure in ("after_stop", "after_copy", "after_rename", "after_deploy",
|
||||
"during_verify", "after_receipt", "after_config"):
|
||||
with self.subTest(failure=failure):
|
||||
ops = FakeOperations(fail=failure)
|
||||
with self.assertRaisesRegex(RuntimeError, "original runtime restored"):
|
||||
run_transaction(ops)
|
||||
self.assertTrue(ops.original_running)
|
||||
self.assertEqual(ops.state, "original")
|
||||
self.assertEqual(ops.config, "original")
|
||||
self.assertFalse(ops.candidate)
|
||||
self.assertEqual(ops.clones, failure != "after_stop")
|
||||
self.assertEqual(recover_transaction(ops), "rolled_back")
|
||||
|
||||
def test_abrupt_crash_after_stop_rename_and_receipt_requires_explicit_recovery(self):
|
||||
for failure in ("after_stop", "after_rename", "after_receipt", "after_config"):
|
||||
with self.subTest(failure=failure):
|
||||
ops = FakeOperations(fail=failure, crash=True)
|
||||
with self.assertRaises(Crash):
|
||||
run_transaction(ops)
|
||||
self.assertNotEqual(ops.journal["phase"], "committed")
|
||||
self.assertEqual(recover_transaction(ops), "rolled_back")
|
||||
self.assertTrue(ops.original_running)
|
||||
|
||||
def test_foreign_runtime_blocks_automatic_recovery(self):
|
||||
ops = FakeOperations(fail="during_verify")
|
||||
ops.foreign = True
|
||||
with self.assertRaisesRegex(RuntimeError, "requires explicit recovery"):
|
||||
run_transaction(ops)
|
||||
self.assertEqual(ops.journal["phase"], "recovery_failed")
|
||||
self.assertTrue(ops.candidate)
|
||||
|
||||
def test_production_candidate_gate_rejects_wrong_nonce_project_and_replaced_id(self):
|
||||
ops = Production.__new__(Production)
|
||||
ops.prepared = {"originalContainerId": "original"}
|
||||
ops.plan = {"candidateComposeProject": "lumaops-managed"}
|
||||
ops.journal = {"nonce": "expected", "candidateContainerId": "candidate"}
|
||||
ops.runner = object()
|
||||
for fields in (["candidate", "image", "lumaops-managed", "wrong"],
|
||||
["candidate", "image", "foreign-project", "expected"],
|
||||
["original", "image", "lumaops-managed", "expected"],
|
||||
["replacement", "image", "lumaops-managed", "expected"]):
|
||||
with self.subTest(fields=fields), patch("transaction.inspect_fields", return_value=fields):
|
||||
with self.assertRaises(ValueError):
|
||||
ops.assert_candidate(fields[0])
|
||||
|
||||
def test_future_broker_metadata_remains_dynamic(self):
|
||||
original = {"source_dir": "/old", "compose_files": ["docker-compose.unraid.yml"],
|
||||
"clean_untracked_paths": ["old"], "deploy_script": "old.sh"}
|
||||
prepared = {"source": "/exact/new"}
|
||||
plan = {"repository": "jens/vacatureradar", "candidateComposeProject": "vacatureradar-managed",
|
||||
"candidateMounts": [{"source": "/clone/local", "destination": "/app/local"}]}
|
||||
project = candidate_project(original, prepared, plan, "nonce", Path("/private/runtime.env"))
|
||||
self.assertTrue(project["exact_source"] and project["verify_oci_revision"])
|
||||
self.assertNotIn("deploy_script", project)
|
||||
self.assertEqual(project["environment"]["MIGRATION_SOURCE_REVISION"], "{revision}")
|
||||
self.assertEqual(project["environment"]["MIGRATION_SOURCE_TREE"], "{tree}")
|
||||
self.assertEqual(project["environment"]["MIGRATION_BUILD_ID"], "{build_id}")
|
||||
self.assertEqual(project["environment"]["MIGRATION_BUILD_DATE"], "{build_date}")
|
||||
|
||||
def test_dotenv_is_literal_and_rejects_unsupported_escaping(self):
|
||||
self.assertEqual(dotenv(["KEY=has $literal #value"]), b"KEY='has $literal #value'\n")
|
||||
for entries in (["KEY=one", "KEY=two"], ["BAD KEY=value"], ["KEY=has'quote"], ["KEY=line\nbreak"]):
|
||||
with self.assertRaises(ValueError):
|
||||
dotenv(entries)
|
||||
|
||||
def test_manifest_detects_content_or_mode_change(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
file = root / "data"
|
||||
file.write_bytes(b"fixture-only")
|
||||
before = digest(json.dumps(tree_manifest(root), sort_keys=True).encode())
|
||||
file.write_bytes(b"changed")
|
||||
after = digest(json.dumps(tree_manifest(root), sort_keys=True).encode())
|
||||
self.assertNotEqual(before, after)
|
||||
|
||||
def test_cold_copy_uses_archive_mode_and_rejects_metadata_mismatch(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
source, target = Path(temporary) / "source", Path(temporary) / "target"
|
||||
source.mkdir()
|
||||
calls = []
|
||||
identical = {".": {"mode": 448, "uid": 999, "gid": 999}}
|
||||
with patch("storage.tree_manifest", return_value=identical):
|
||||
clone_cold(source, target, lambda command, **kwargs: calls.append(command))
|
||||
self.assertEqual(calls[0][:4], ["cp", "-a", "--reflink=auto", "--"])
|
||||
with patch("storage.tree_manifest", side_effect=[identical, {}]):
|
||||
with self.assertRaisesRegex(ValueError, "content/metadata"):
|
||||
clone_cold(source, target, lambda *args, **kwargs: None)
|
||||
|
||||
def test_real_recovery_adapter_refuses_foreign_container_before_remove(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
workspace = Path(temporary)
|
||||
config, state = b"{}", b'{"status":"failed"}'
|
||||
for name, value in [("original-projects.json", config), ("original-state.json", state),
|
||||
("projects.json", config), ("state.json", state)]:
|
||||
(workspace / name).write_bytes(value)
|
||||
ops = Production.__new__(Production)
|
||||
ops.workspace = workspace
|
||||
ops.broker = type("Broker", (), {"CONFIG_PATH": workspace / "projects.json"})()
|
||||
ops.state_path = workspace / "state.json"
|
||||
ops.prepared = {"originalContainerId": "original"}
|
||||
ops.plan = {"originalName": "lumaops", "candidateComposeProject": "lumaops-managed"}
|
||||
ops.journal = {"originalConfigSha256": digest(config), "candidateConfigSha256": "other",
|
||||
"originalStateSha256": digest(state), "nonce": "expected"}
|
||||
ops.original = lambda **kwargs: {"running": False}
|
||||
ops.canonical_id = lambda name: "foreign"
|
||||
calls = []
|
||||
ops.runner = lambda command, **kwargs: calls.append(command)
|
||||
with patch("transaction.inspect_fields", return_value=["foreign", "image", "lumaops-managed", "wrong"]):
|
||||
with self.assertRaisesRegex(ValueError, "unrelated"):
|
||||
ops.recover()
|
||||
self.assertEqual(calls, [])
|
||||
|
||||
def test_verification_binds_normal_broker_receipt_to_prepared_image(self):
|
||||
ops = Production.__new__(Production)
|
||||
ops.journal = {"candidateContainerId": "candidate"}
|
||||
ops.slug = "lumaops"
|
||||
ops.plan = {"originalName": "lumaops", "candidateMounts": []}
|
||||
ops.prepared = {"metadata": {"fixture": "metadata"}, "candidateImageId": "immutable", "revision": "rev"}
|
||||
ops.project, ops.repository, ops.runner = {}, "jens/lumaops", None
|
||||
ops.assert_candidate = lambda value: None
|
||||
ops.http_health = lambda: None
|
||||
ops.save = lambda value: None
|
||||
seen = []
|
||||
|
||||
class Broker:
|
||||
def verify_deployment(self, project, metadata, **kwargs):
|
||||
seen.append(kwargs)
|
||||
return {"fixture": "real-broker-receipt"}
|
||||
|
||||
def validate_recorded_receipt(self, *args):
|
||||
return None
|
||||
|
||||
ops.broker = Broker()
|
||||
with patch("transaction.mounts", return_value=[]), patch("transaction.inspect_fields", return_value=[{"lumaops_default": {}}]):
|
||||
ops.verifying()
|
||||
self.assertEqual(seen, [{"expected_image_ids": {"lumaops": "immutable"}}])
|
||||
self.assertEqual(ops.journal["receipt"], {"fixture": "real-broker-receipt"})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,517 @@
|
||||
"""Journaled legacy cutover through normal broker functions; fixed two targets only.
|
||||
|
||||
No provenance is written for the original runtime. Recovery retains every data
|
||||
clone and original image/container. Completed migrations are never automatically
|
||||
rolled back: new writes may have reached candidate data after commit.
|
||||
"""
|
||||
|
||||
from contextlib import ExitStack
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import secrets
|
||||
import shutil
|
||||
import time
|
||||
from urllib.request import urlopen
|
||||
|
||||
from execute import (JOURNAL_ROOT, assert_original, checked, docker_json_template, mount_inventory, original_identity,
|
||||
require_real_child, verify_image)
|
||||
from plan import ATTEMPT, SHA, TARGETS, build_plan
|
||||
from storage import clone_cold, digest, dotenv, private_json, private_write, tree_manifest
|
||||
|
||||
|
||||
STEPS = ("stopping", "copying", "renaming", "deploying", "verifying", "committing")
|
||||
|
||||
|
||||
def run_transaction(ops):
|
||||
"""Write intent before every side effect, allowing recovery after abrupt exit."""
|
||||
try:
|
||||
for phase in STEPS:
|
||||
ops.save(phase)
|
||||
getattr(ops, phase)()
|
||||
ops.save("committed")
|
||||
except Exception:
|
||||
try:
|
||||
ops.recover()
|
||||
ops.save("rolled_back")
|
||||
except Exception:
|
||||
ops.save("recovery_failed")
|
||||
raise RuntimeError("Migration failed and rollback requires explicit recovery") from None
|
||||
raise RuntimeError("Migration failed; original runtime restored and candidate data retained") from None
|
||||
# Outside rollback scope: releasing workers permits new external effects/data.
|
||||
ops.activate()
|
||||
|
||||
|
||||
def recover_transaction(ops):
|
||||
if ops.journal["phase"] == "committed":
|
||||
ops.verify_committed()
|
||||
ops.activate()
|
||||
return "committed"
|
||||
if ops.journal["phase"] == "rolled_back":
|
||||
ops.original(healthy=True)
|
||||
ops.http_health()
|
||||
return "rolled_back"
|
||||
ops.recover()
|
||||
ops.save("rolled_back")
|
||||
return "rolled_back"
|
||||
|
||||
|
||||
def inspect_fields(runner, target, expressions):
|
||||
template = docker_json_template(expressions)
|
||||
return [json.loads(line) for line in runner(["docker", "inspect", "--format", template, target]).splitlines()]
|
||||
|
||||
|
||||
def mounts(runner, target):
|
||||
raw = inspect_fields(runner, target, [".Mounts"])[0]
|
||||
return sorted(({"type": item["Type"], "source": item["Source"],
|
||||
"destination": item["Destination"], "rw": item["RW"]} for item in raw),
|
||||
key=lambda item: item["destination"])
|
||||
|
||||
|
||||
def candidate_project(original, prepared, plan, nonce, env_path):
|
||||
slug = plan["repository"].split("/")[1]
|
||||
by_target = {item["destination"]: item["source"] for item in plan["candidateMounts"]}
|
||||
names = ({"MIGRATION_LOCAL_ROOT": "/app/local"} if slug == "vacatureradar" else {
|
||||
"MIGRATION_CONFIG_ROOT": "/config", "MIGRATION_OPENRGB_ROOT": "/config/openrgb",
|
||||
"MIGRATION_LUMA_CONFIG_ROOT": "/config/lumaops", "MIGRATION_DATA_ROOT": "/data",
|
||||
"MIGRATION_LOGS_ROOT": "/logs"})
|
||||
environment = {**dict(original.get("environment", {})),
|
||||
**{key: by_target[target] for key, target in names.items()},
|
||||
"MIGRATION_ATTEMPT": nonce, "MIGRATION_IMAGE": f"gitea-managed/{slug}:{{revision}}",
|
||||
"MIGRATION_NETWORK": slug + "_default",
|
||||
"MIGRATION_SOURCE_REVISION": "{revision}", "MIGRATION_SOURCE_TREE": "{tree}",
|
||||
"MIGRATION_BUILD_ID": "{build_id}", "MIGRATION_BUILD_DATE": "{build_date}",
|
||||
"MIGRATION_ENV_FILE": str(env_path), "VACATURERADAR_ENV_FILE": str(env_path)}
|
||||
result = dict(original)
|
||||
for key in ("deploy_script", "archive_deploy_script", "manual", "clean_untracked_paths", "legacy_containers"):
|
||||
result.pop(key, None)
|
||||
result.update(source_dir=prepared["source"], compose_project=plan["candidateComposeProject"],
|
||||
exact_source=True, verify_oci_revision=True, env_source=str(env_path),
|
||||
environment=environment,
|
||||
compose_files=[*original["compose_files"], "docker-compose.migration.yml"])
|
||||
return result
|
||||
|
||||
|
||||
def render_contract(broker, project, prepared, plan, runner):
|
||||
cwd, command = broker.compose_command(project)
|
||||
environment = broker.deployment_environment(project, prepared["revision"], plan["repository"], prepared["metadata"])
|
||||
rendered = json.loads(runner(command + ["config", "--format", "json"], cwd=cwd, env=environment))
|
||||
services = rendered.get("services", {})
|
||||
if len(services) != 1:
|
||||
raise ValueError("Migration supports exactly one reviewed application container")
|
||||
service = next(iter(services.values()))
|
||||
if service.get("container_name") != plan["originalName"]:
|
||||
raise ValueError("Candidate canonical name differs from the plan")
|
||||
network = plan["repository"].split("/")[1] + "_default"
|
||||
original_networks = inspect_fields(runner, prepared["originalContainerId"], [".NetworkSettings.Networks"])[0]
|
||||
if set(original_networks) != {network} or service.get("network_mode") != network:
|
||||
raise ValueError("Candidate network must exactly preserve the reviewed original network")
|
||||
if prepared.get("originalNetwork") not in {None, network}:
|
||||
raise ValueError("Original network changed since preparation")
|
||||
host = inspect_fields(runner, prepared["originalContainerId"], [".HostConfig"])[0]
|
||||
validate_runtime_safety(host, service)
|
||||
expected = {item["destination"]: item["source"] for item in plan["candidateMounts"]}
|
||||
observed = {}
|
||||
for item in service.get("volumes", []):
|
||||
if item.get("type") != "bind" or item.get("read_only") or item.get("bind", {}).get("create_host_path") is not False:
|
||||
raise ValueError("Candidate contains an unreviewed writable mount")
|
||||
observed[item["target"]] = item["source"]
|
||||
if observed != expected or len(observed) != len(service.get("volumes", [])):
|
||||
raise ValueError("Rendered candidate mounts do not exactly match cold clones")
|
||||
bindings = inspect_fields(runner, prepared["originalContainerId"], [".HostConfig.PortBindings"])[0]
|
||||
old_ports = sorted((port, item.get("HostIp") or "0.0.0.0", str(item["HostPort"]))
|
||||
for port, items in bindings.items() for item in items or [])
|
||||
new_ports = sorted((f"{item['target']}/{item.get('protocol', 'tcp')}", item.get("host_ip") or "0.0.0.0", str(item["published"]))
|
||||
for item in service.get("ports", []))
|
||||
if old_ports != new_ports:
|
||||
raise ValueError("Candidate published ports differ from the original runtime")
|
||||
if service.get("labels", {}).get("io.itworx.migration-attempt") != environment["MIGRATION_ATTEMPT"]:
|
||||
raise ValueError("Candidate ownership correlation label is absent")
|
||||
return cwd, command, environment
|
||||
|
||||
|
||||
def validate_runtime_safety(host, service):
|
||||
"""Compare nonsecret hardware/security bindings; unsupported shapes fail closed."""
|
||||
old_devices = sorted((item["PathOnHost"], item["PathInContainer"], item["CgroupPermissions"])
|
||||
for item in host.get("Devices") or [])
|
||||
new_devices = sorted((item["source"], item["target"], item.get("permissions", "rwm"))
|
||||
for item in service.get("devices", []))
|
||||
if old_devices != new_devices or host.get("DeviceRequests") or host.get("DeviceCgroupRules"):
|
||||
raise ValueError("Candidate device contract differs or is unsupported")
|
||||
for actual, desired in (("CapAdd", "cap_add"), ("CapDrop", "cap_drop")):
|
||||
normalize = lambda values: sorted(value.upper().removeprefix("CAP_") for value in values or [])
|
||||
if normalize(host.get(actual)) != normalize(service.get(desired)):
|
||||
raise ValueError("Candidate capability contract differs")
|
||||
for actual, desired in (("ReadonlyRootfs", "read_only"), ("Privileged", "privileged")):
|
||||
if bool(host.get(actual)) != bool(service.get(desired)):
|
||||
raise ValueError("Candidate filesystem/privilege contract differs")
|
||||
if sorted(host.get("SecurityOpt") or []) != sorted(service.get("security_opt") or []):
|
||||
raise ValueError("Candidate security options differ")
|
||||
new_tmpfs = {}
|
||||
for item in service.get("tmpfs") or []:
|
||||
target, separator, options = item.partition(":")
|
||||
if target in new_tmpfs:
|
||||
raise ValueError("Duplicate candidate tmpfs target")
|
||||
new_tmpfs[target] = options if separator else ""
|
||||
if (host.get("Tmpfs") or {}) != new_tmpfs:
|
||||
raise ValueError("Candidate tmpfs contract differs")
|
||||
|
||||
|
||||
def prepare_compose(broker, slug, revision, attempt, original, prepared, workspace, runner):
|
||||
if slug == "vacatureradar":
|
||||
source = Path(prepared["source"])
|
||||
required = {
|
||||
"scripts/migration_worker_gate.py": (".migration-worker-hold", "while check():"),
|
||||
"deployment/unraid/supervisord.conf": ("migration_worker_gate.py worker", "migration_worker_gate.py scheduler"),
|
||||
"config/settings.py": ("apps.core.middleware.MigrationMaintenanceMiddleware",),
|
||||
"apps/core/middleware.py": ("class MigrationMaintenanceMiddleware", "migration_maintenance"),
|
||||
}
|
||||
for filename, fragments in required.items():
|
||||
content = (source / filename).read_text()
|
||||
if any(fragment not in content for fragment in fragments):
|
||||
raise ValueError("Exact source lacks the reviewed maintenance contract")
|
||||
observed = {"name": TARGETS[slug][0], **original_identity(prepared["originalContainerId"], runner),
|
||||
"mounts": prepared["originalMounts"]}
|
||||
plan = build_plan(slug, revision, attempt, observed)
|
||||
if Path(plan["workspace"]).resolve() != workspace:
|
||||
raise ValueError("Plan workspace differs from the protected preparation directory")
|
||||
nonce = secrets.token_hex(16)
|
||||
env_path = workspace / "runtime.env"
|
||||
entries = inspect_fields(runner, prepared["originalContainerId"], [".Config.Env"])[0]
|
||||
private_write(env_path, dotenv(entries))
|
||||
prepared["originalNetwork"] = slug + "_default"
|
||||
candidate = candidate_project(original, prepared, plan, nonce, env_path)
|
||||
repo = broker.repository_path(plan["repository"], original)
|
||||
broker.validate_project_config(repo, candidate, revision)
|
||||
cwd, command, environment = render_contract(broker, candidate, prepared, plan, runner)
|
||||
return candidate, command + ["build"], cwd, environment, plan, nonce
|
||||
|
||||
|
||||
class Production:
|
||||
def __init__(self, broker, slug, prepared, workspace, journal, runner=checked):
|
||||
self.broker, self.slug, self.prepared = broker, slug, prepared
|
||||
self.workspace, self.journal, self.runner = workspace, journal, runner
|
||||
self.repository = f"jens/{slug}"
|
||||
self.plan = journal["plan"]
|
||||
self.project = dict(journal["candidateProject"])
|
||||
self.project.update(_repository_name=self.repository,
|
||||
_repository_path=journal["repositoryPath"],
|
||||
_deployment_metadata=prepared["metadata"])
|
||||
self.state_path = broker.STATE_ROOT / f"{slug}.json"
|
||||
|
||||
def save(self, phase):
|
||||
self.journal["phase"] = phase
|
||||
private_json(self.workspace / "cutover.json", self.journal)
|
||||
|
||||
def canonical_id(self, name):
|
||||
result = self.runner(["docker", "container", "ls", "--all", "--quiet", "--no-trunc",
|
||||
"--filter", f"name=^/{name}$"]).strip().splitlines()
|
||||
if len(result) > 1:
|
||||
raise ValueError("Ambiguous container identity")
|
||||
return result[0] if result else None
|
||||
|
||||
def original(self, healthy=False):
|
||||
container_id = self.prepared["originalContainerId"]
|
||||
observed = original_identity(container_id, self.runner)
|
||||
if observed["imageId"] != self.prepared["originalImageId"] or observed["composeProject"] != self.slug:
|
||||
raise ValueError("Original immutable runtime identity changed")
|
||||
if mounts(self.runner, container_id) != self.journal["originalMounts"]:
|
||||
raise ValueError("Original mounts changed")
|
||||
networks = inspect_fields(self.runner, container_id, [".NetworkSettings.Networks"])[0]
|
||||
if set(networks) != {self.slug + "_default"}:
|
||||
raise ValueError("Original network changed")
|
||||
if healthy:
|
||||
assert_original(observed, self.slug, container_id, self.prepared["originalImageId"])
|
||||
return observed
|
||||
|
||||
def stopping(self):
|
||||
self.original(healthy=True)
|
||||
self.runner(["docker", "stop", "--time", "60", self.prepared["originalContainerId"]], timeout=90)
|
||||
if self.original()["running"]:
|
||||
raise ValueError("Original did not stop")
|
||||
|
||||
def copying(self):
|
||||
if self.original()["running"]:
|
||||
raise ValueError("Cold copy requires stopped original")
|
||||
checksums = {}
|
||||
for item in self.plan["coldCopyRoots"]:
|
||||
original = Path(item["original"])
|
||||
if original.is_symlink() or original.resolve() != original:
|
||||
raise ValueError("Unreviewed symlink in original cold-copy root")
|
||||
target = require_real_child(Path(item["candidate"]), self.workspace)
|
||||
checksums[item["original"]] = clone_cold(original, target, self.runner)
|
||||
self.journal["coldCopyChecksums"] = checksums
|
||||
self.save("copying")
|
||||
if self.slug == "vacatureradar":
|
||||
marker = self.hold_path()
|
||||
if marker.exists() or marker.is_symlink():
|
||||
raise ValueError("Cold clone already contains a maintenance hold")
|
||||
private_write(marker, self.hold_bytes())
|
||||
|
||||
def hold_path(self):
|
||||
roots = [item["source"] for item in self.plan["candidateMounts"]
|
||||
if item["destination"] == "/app/local"]
|
||||
if len(roots) != 1:
|
||||
raise ValueError("Maintenance hold requires one exact cloned local root")
|
||||
return require_real_child(Path(roots[0]) / ".migration-worker-hold", self.workspace)
|
||||
|
||||
def hold_bytes(self):
|
||||
return ("legacy-migration.v1:" + self.journal["nonce"] + "\n").encode()
|
||||
|
||||
def verify_hold(self):
|
||||
if self.slug == "vacatureradar" and private_read(self.hold_path()) != self.hold_bytes():
|
||||
raise ValueError("Maintenance hold identity does not match this attempt")
|
||||
|
||||
def activate(self):
|
||||
if self.journal.get("phase") != "committed":
|
||||
raise ValueError("Activation requires a durable committed transaction")
|
||||
self.verify_committed()
|
||||
self.assert_candidate(self.journal["candidateContainerId"])
|
||||
if self.slug == "vacatureradar":
|
||||
marker = self.hold_path()
|
||||
if marker.exists() or marker.is_symlink():
|
||||
self.verify_hold()
|
||||
self.journal["activationIntent"] = True
|
||||
self.save("committed")
|
||||
marker.unlink()
|
||||
descriptor = os.open(marker.parent, os.O_RDONLY)
|
||||
try:
|
||||
os.fsync(descriptor)
|
||||
finally:
|
||||
os.close(descriptor)
|
||||
elif not self.journal.get("activationIntent"):
|
||||
raise ValueError("Maintenance hold disappeared before activation intent")
|
||||
self.journal["activated"] = True
|
||||
self.save("committed")
|
||||
|
||||
def renaming(self):
|
||||
if self.canonical_id(self.plan["retainedName"]):
|
||||
raise ValueError("Retained container name is already occupied")
|
||||
if self.canonical_id(self.plan["originalName"]) != self.prepared["originalContainerId"]:
|
||||
raise ValueError("Canonical container identity changed")
|
||||
self.runner(["docker", "rename", self.prepared["originalContainerId"], self.plan["retainedName"]])
|
||||
|
||||
def deploying(self):
|
||||
self.verify_hold()
|
||||
if self.canonical_id(self.plan["originalName"]):
|
||||
raise ValueError("Canonical name was claimed before candidate deployment")
|
||||
# True first deploy relative to the distinct managed Compose project.
|
||||
if self.broker.capture_compose_rollback(self.project) is not None:
|
||||
raise ValueError("Candidate project is not a first deployment")
|
||||
self.broker.revalidate_deployment_candidate(Path(self.journal["repositoryPath"]), self.project,
|
||||
self.prepared["revision"], self.repository, self.prepared["metadata"])
|
||||
cwd, command, environment = render_contract(self.broker, self.project, self.prepared, self.plan, self.runner)
|
||||
image_id = verify_image(environment["MIGRATION_IMAGE"], self.prepared["metadata"], self.runner)
|
||||
if image_id != self.prepared["candidateImageId"]:
|
||||
raise ValueError("Prepared image tag changed before candidate start")
|
||||
self.runner(command + ["up", "--detach", "--no-build", "--pull", "never"],
|
||||
cwd=cwd, env=environment, timeout=180)
|
||||
container_id = self.canonical_id(self.plan["originalName"])
|
||||
if not container_id:
|
||||
raise ValueError("Candidate deployment produced no canonical container")
|
||||
self.assert_candidate(container_id)
|
||||
self.journal["candidateContainerId"] = container_id
|
||||
self.save("deploying")
|
||||
|
||||
def assert_candidate(self, container_id):
|
||||
values = inspect_fields(self.runner, container_id, [".Id", ".Image",
|
||||
'index .Config.Labels "com.docker.compose.project"',
|
||||
'index .Config.Labels "io.itworx.migration-attempt"'])
|
||||
if (values[0] != container_id or container_id == self.prepared["originalContainerId"]
|
||||
or values[2] != self.plan["candidateComposeProject"] or values[3] != self.journal["nonce"]):
|
||||
raise ValueError("Refusing an unrelated candidate container")
|
||||
recorded = self.journal.get("candidateContainerId")
|
||||
if recorded and recorded != container_id:
|
||||
raise ValueError("Recorded candidate container was replaced")
|
||||
|
||||
def http_health(self):
|
||||
path = "http://127.0.0.1:1226/health/ready/" if self.slug == "vacatureradar" else "http://127.0.0.1:1223/health/ready"
|
||||
with urlopen(path, timeout=10) as response:
|
||||
if response.status != 200 or response.geturl() != path:
|
||||
raise ValueError("Representative readiness probe failed")
|
||||
|
||||
def verifying(self):
|
||||
self.verify_hold()
|
||||
self.assert_candidate(self.journal["candidateContainerId"])
|
||||
expected = sorted(self.plan["candidateMounts"], key=lambda item: item["destination"])
|
||||
if mounts(self.runner, self.journal["candidateContainerId"]) != expected:
|
||||
raise ValueError("Candidate mounts differ from the isolated cold-copy plan")
|
||||
networks = inspect_fields(self.runner, self.journal["candidateContainerId"], [".NetworkSettings.Networks"])[0]
|
||||
if set(networks) != {self.slug + "_default"}:
|
||||
raise ValueError("Candidate runtime network differs from the original")
|
||||
receipt = self.broker.verify_deployment(self.project, self.prepared["metadata"],
|
||||
expected_image_ids={self.plan["originalName"]: self.prepared["candidateImageId"]})
|
||||
self.broker.validate_recorded_receipt(self.project, receipt, self.repository, self.prepared["revision"])
|
||||
self.http_health()
|
||||
self.journal["receipt"] = receipt
|
||||
self.save("verifying")
|
||||
|
||||
def committing(self):
|
||||
self.broker.write_state(self.repository, self.project, self.prepared["revision"], "success",
|
||||
"Controlled legacy migration passed immutable image and readiness verification",
|
||||
live_receipt=self.journal["receipt"], attempt_metadata=self.prepared["metadata"])
|
||||
# Never alter another project's configuration. Both deployment locks held.
|
||||
current = self.broker.CONFIG_PATH.read_bytes()
|
||||
if digest(current) != self.journal["originalConfigSha256"]:
|
||||
raise ValueError("Broker configuration changed during cutover")
|
||||
private_write(self.broker.CONFIG_PATH, (self.workspace / "candidate-projects.json").read_bytes())
|
||||
self.verify_committed()
|
||||
|
||||
def verify_committed(self):
|
||||
if digest(self.broker.CONFIG_PATH.read_bytes()) != self.journal["candidateConfigSha256"]:
|
||||
raise ValueError("Committed broker configuration no longer matches")
|
||||
receipt = self.broker.verified_current_receipt(self.repository, self.project,
|
||||
self.prepared["revision"], self.prepared["metadata"]["tree"])
|
||||
if receipt is None:
|
||||
raise ValueError("Committed runtime receipt does not reverify")
|
||||
self.http_health()
|
||||
|
||||
def recover(self):
|
||||
if self.journal.get("phase") == "committed" or self.journal.get("activationIntent"):
|
||||
raise ValueError("Committed activation must never roll back")
|
||||
for filename, key in (("original-projects.json", "originalConfigSha256"),
|
||||
("original-state.json", "originalStateSha256")):
|
||||
if digest((self.workspace / filename).read_bytes()) != self.journal[key]:
|
||||
raise ValueError("Protected rollback snapshot integrity failed")
|
||||
config = self.broker.CONFIG_PATH.read_bytes()
|
||||
if digest(config) not in {self.journal["originalConfigSha256"], self.journal["candidateConfigSha256"]}:
|
||||
raise ValueError("Unrelated broker configuration change blocks automatic recovery")
|
||||
state = self.state_path.read_bytes()
|
||||
if digest(state) != self.journal["originalStateSha256"]:
|
||||
payload = json.loads(state)
|
||||
if payload.get("live_receipt") != self.journal.get("receipt") or not self.journal.get("receipt"):
|
||||
raise ValueError("Unrelated deployment state blocks automatic recovery")
|
||||
self.original()
|
||||
if not self.journal.get("originalRestartIntent"):
|
||||
for source, expected in self.journal.get("coldCopyChecksums", {}).items():
|
||||
if digest(json.dumps(tree_manifest(source), sort_keys=True).encode()) != expected:
|
||||
raise ValueError("Original data changed after verified cold copy")
|
||||
container_id = self.canonical_id(self.plan["originalName"])
|
||||
if container_id and container_id != self.prepared["originalContainerId"]:
|
||||
self.assert_candidate(container_id)
|
||||
# Persist discovered ownership before removing a crash-created container.
|
||||
self.journal["candidateContainerId"] = container_id
|
||||
self.save("recovering")
|
||||
self.runner(["docker", "rm", "--force", container_id], timeout=90)
|
||||
if self.canonical_id(self.plan["originalName"]) not in {None, self.prepared["originalContainerId"]}:
|
||||
raise ValueError("Canonical name is occupied by unrelated runtime")
|
||||
current_name = inspect_fields(self.runner, self.prepared["originalContainerId"], [".Name"])[0].lstrip("/")
|
||||
if current_name not in {self.plan["originalName"], self.plan["retainedName"]}:
|
||||
raise ValueError("Original runtime was renamed by another operator")
|
||||
if current_name != self.plan["originalName"]:
|
||||
self.runner(["docker", "rename", self.prepared["originalContainerId"], self.plan["originalName"]])
|
||||
private_write(self.broker.CONFIG_PATH, (self.workspace / "original-projects.json").read_bytes())
|
||||
private_write(self.state_path, (self.workspace / "original-state.json").read_bytes())
|
||||
self.journal["originalRestartIntent"] = True
|
||||
self.save("restoring_original")
|
||||
if not self.original()["running"]:
|
||||
self.runner(["docker", "start", self.prepared["originalContainerId"]], timeout=90)
|
||||
deadline = time.monotonic() + 240
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
self.original(healthy=True)
|
||||
self.http_health()
|
||||
return
|
||||
except (ValueError, OSError):
|
||||
time.sleep(3)
|
||||
raise ValueError("Original runtime recovery health did not verify")
|
||||
|
||||
|
||||
def private_read(path):
|
||||
import os
|
||||
import stat
|
||||
info = path.lstat()
|
||||
if not stat.S_ISREG(info.st_mode) or info.st_uid != os.geteuid() or stat.S_IMODE(info.st_mode) & 0o077:
|
||||
raise ValueError("Expected a root-only regular transaction file")
|
||||
return path.read_bytes()
|
||||
|
||||
|
||||
def operate(broker, mode, slug, revision, attempt, container_id, image_id, runner=checked):
|
||||
if mode not in {"cutover", "recover"} or slug not in TARGETS or not SHA.fullmatch(revision) or not ATTEMPT.fullmatch(attempt):
|
||||
raise ValueError("Invalid fixed migration operation")
|
||||
workspace = require_real_child(JOURNAL_ROOT / slug / attempt, JOURNAL_ROOT)
|
||||
repository = f"jens/{slug}"
|
||||
with ExitStack() as locks:
|
||||
locks.enter_context(broker.exclusive_lock(broker.STATE_ROOT / "locks" / f"{slug}.lock", "migration project"))
|
||||
locks.enter_context(broker.exclusive_lock(broker.GLOBAL_DEPLOY_LOCK, "migration production"))
|
||||
prepared_bytes = private_read(workspace / "prepare.json")
|
||||
prepared = json.loads(prepared_bytes)
|
||||
if (prepared.get("phase") != "build_verified" or prepared.get("repository") != repository
|
||||
or prepared.get("revision") != revision or prepared.get("originalContainerId") != container_id
|
||||
or prepared.get("originalImageId") != image_id):
|
||||
raise ValueError("Preparation identity does not match the requested operation")
|
||||
journal_path = workspace / "cutover.json"
|
||||
if mode == "recover":
|
||||
journal = json.loads(private_read(journal_path))
|
||||
if journal.get("preparedSha256") != digest(prepared_bytes):
|
||||
raise ValueError("Prepared artifact changed after transaction snapshot")
|
||||
ops = Production(broker, slug, prepared, workspace, journal, runner)
|
||||
phase = recover_transaction(ops)
|
||||
return {"phase": phase, "repository": repository, "candidateDataRetained": True}
|
||||
if journal_path.exists():
|
||||
raise ValueError("Transaction already exists; use explicit recover, never repeat cutover")
|
||||
projects_bytes = broker.CONFIG_PATH.read_bytes()
|
||||
projects = json.loads(projects_bytes)
|
||||
original = projects[repository]
|
||||
state_bytes = (broker.STATE_ROOT / f"{slug}.json").read_bytes()
|
||||
if (digest(json.dumps(original, sort_keys=True).encode()) != prepared.get("projectConfigSha256")
|
||||
or digest(state_bytes) != prepared.get("stateSha256")):
|
||||
raise ValueError("Original broker config/state changed since preparation")
|
||||
if broker.state_live_revision(json.loads(state_bytes)):
|
||||
raise ValueError("This transaction only supports unattested legacy first deployments")
|
||||
repo = broker.repository_path(repository, original)
|
||||
broker.validate_commit(repo, str(original["branch"]), revision)
|
||||
if broker.output(["git", f"--git-dir={repo}", "rev-parse", f"refs/heads/{original['branch']}"]) != revision:
|
||||
raise ValueError("Prepared revision is no longer the configured branch head")
|
||||
source = require_real_child(Path(prepared["source"]), broker.EXACT_SOURCE_ROOT)
|
||||
if not broker.exact_source_matches_revision(repo, revision, source):
|
||||
raise ValueError("Prepared source changed before cutover")
|
||||
if verify_image(prepared["candidateImage"], prepared["metadata"], runner) != prepared["candidateImageId"]:
|
||||
raise ValueError("Prepared immutable image changed")
|
||||
assert_original(original_identity(container_id, runner), slug, container_id, image_id)
|
||||
if mount_inventory(container_id, runner) != prepared["originalMounts"]:
|
||||
raise ValueError("Original mounts changed after preparation")
|
||||
if digest(private_read(workspace / "runtime.env")) != prepared["runtimeEnvSha256"]:
|
||||
raise ValueError("Private runtime environment changed after preparation")
|
||||
plan = build_plan(slug, revision, attempt, {"name": TARGETS[slug][0],
|
||||
**original_identity(container_id, runner), "mounts": prepared["originalMounts"]})
|
||||
if plan != prepared["plan"]:
|
||||
raise ValueError("Cold-copy plan changed after preparation")
|
||||
candidate = candidate_project(original, prepared, plan, prepared["nonce"], workspace / "runtime.env")
|
||||
if candidate != prepared["candidateProject"]:
|
||||
raise ValueError("Candidate project differs from the build configuration")
|
||||
broker.validate_project_config(repo, candidate, revision)
|
||||
render_contract(broker, candidate, prepared, plan, runner)
|
||||
required = 8 * 1024**3
|
||||
for item in plan["coldCopyRoots"]:
|
||||
root = Path(item["original"])
|
||||
if root.is_symlink() or root.resolve() != root or not root.is_dir():
|
||||
raise ValueError("Cold-copy source is not an unchanged real directory")
|
||||
required += sum(path.stat().st_size for path in root.rglob("*") if path.is_file())
|
||||
if Path(item["candidate"]).exists():
|
||||
raise ValueError("Cold-copy target already exists")
|
||||
if shutil.disk_usage(workspace).free < required:
|
||||
raise ValueError("Insufficient space for retained cold clones plus safety reserve")
|
||||
updated = {**projects, repository: candidate}
|
||||
candidate_bytes = json.dumps(updated, sort_keys=True).encode()
|
||||
private_write(workspace / "original-projects.json", projects_bytes)
|
||||
private_write(workspace / "original-state.json", state_bytes)
|
||||
private_write(workspace / "candidate-projects.json", candidate_bytes)
|
||||
# Configuration may contain secrets: stored privately, never in user output.
|
||||
snapshot = inspect_fields(runner, container_id, [".Config", ".HostConfig", ".NetworkSettings.Networks"])
|
||||
private_json(workspace / "original-container.json", snapshot)
|
||||
journal = {"schema": "legacy-cutover.v1", "phase": "snapshotted", "plan": plan,
|
||||
"preparedSha256": digest(prepared_bytes), "candidateProject": candidate,
|
||||
"repositoryPath": str(repo), "nonce": prepared["nonce"], "originalMounts": prepared["originalMounts"],
|
||||
"originalConfigSha256": digest(projects_bytes), "originalStateSha256": digest(state_bytes),
|
||||
"candidateConfigSha256": digest(candidate_bytes)}
|
||||
ops = Production(broker, slug, prepared, workspace, journal, runner)
|
||||
ops.http_health()
|
||||
if ops.canonical_id(plan["retainedName"]) or broker.compose_project_container_names(plan["candidateComposeProject"]):
|
||||
raise ValueError("Attempt runtime/project is already occupied")
|
||||
runner(["docker", "image", "tag", image_id, f"gitea-legacy-rollback/{slug}:{prepared['nonce']}"])
|
||||
ops.save("snapshotted")
|
||||
run_transaction(ops)
|
||||
return {"phase": "committed", "repository": repository, "revision": revision,
|
||||
"originalContainerRetained": True, "originalDataRetained": True,
|
||||
"receiptVerified": True, "journal": str(journal_path)}
|
||||
Reference in new issue
Block a user