Files
ModelForge-Public/docs/operations/RC_ACCEPTANCE.md
T

7.8 KiB

Isolated public-candidate acceptance

The Gitea workflow .gitea/workflows/public-candidate-acceptance.yml moves the expensive release proof off a developer workstation and onto a Docker-capable server runner. It is manual-only and accepts an exact canonical commit SHA. It does not invoke unraid-deploy.yml, the deploy controller or the production Compose project.

The job creates a curated public export, validates its content manifest, scans it with Gitleaks, then creates a temporary parentless Git commit so release provenance describes the exact exported tree. From that tree it:

  1. builds API, Console, Node Agent and Runtime Worker images with OCI identity;
  2. produces the release manifest, checksums, CycloneDX inventory and image provenance;
  3. scans every exact image ID with checksum-pinned Trivy and rejects ambiguous reports or any HIGH/CRITICAL finding;
  4. starts API, Console, PostgreSQL and Redis with the production overlay under a unique modelforge-validation-* Compose project;
  5. uses newly generated secrets, loopback-only ephemeral host ports and project-scoped volumes;
  6. proves liveness, readiness, version identity, Console serving and both sides of the operator authentication boundary; and
  7. always removes the temporary containers, networks, volumes and four candidate image tags. An independent if: always() workflow cleanup repeats the exact project-label cleanup after a failure, cancellation or timeout; it never runs a host-wide Docker prune.

No Node Agent or Runtime Worker is started by this clean-install gate. Consequently it creates no compute identity and cannot interact with the production GPU node. A later GPU qualification must enrol a disposable validation-canary identity with production_eligible=false and retain that separate evidence.

P12 runtime-isolation preflight

Before a P12 Node Agent or Runtime Worker starts, render its exact candidate Compose projection to JSON and run scripts/rc_isolation_preflight.py. The check requires a bounded modelforge-validation-* project and refuses external/non-project volumes or networks, host networking, global container names, non-loopback or protected ports, Docker-socket mounts, writable bind mounts, missing bind sources, source paths outside the candidate root, and references to named production resources. A read-only bind targeting /app/config must contain models/initial-candidates.yaml under that exact source path. Run the preflight before docker compose up. After up, run it again with --inspect-docker; it records only resources carrying the exact Compose project label and rejects any name outside that project namespace.

This preflight is an admission control, not P12 qualification evidence. A green result does not replace the required fresh secrets, production_eligible=false validation-canary enrollment, real private runtime inference, before/after production readiness, normal lifecycle cleanup, or retained runtime evidence.

Dedicated test ingress for a retained RC stack

The disposable clean-install workflow above removes its stack. For a later retained P12 stack, include docker-compose.rc-ingress.yml after the base and production files. It places PostgreSQL, Redis and the migration job on a project-private backend network; Web is only on ingress, and API joins both. The ingress network is created by the exact modelforge-validation-* project, not shared with production. Bind all published control-plane ports to 127.0.0.1 and choose unused ports; the reverse proxy reaches Web and API over the ingress network, not those host ports.

Port 443 is reserved for the existing production services. Its NPM host is still pending because the current production Console embeds an HTTP API origin and CSP; the candidate does not change it. The candidate must use a separate HTTPS test listener on port 18444 with a certificate valid for modelforge.example.com. Its canonical browser/API origin is https://modelforge.example.com:18444; set both MODELFORGE_CORS_ORIGINS to that value at runtime and use it for VITE_API_BASE_URL when building the candidate Console image. The test listener routes only to the retained validation project.

Render the complete Compose plan with the exact candidate source, four image identities, freshly generated validation secrets and intended public origin. Run rc_isolation_preflight.py on that JSON before starting any container and again with --inspect-docker afterward. Add --public-tls-port 18444 only with docker-compose.validation-tls.yml; the preflight admits this one port solely for the pinned, confined tls service. It must show only project-scoped networks and volumes, loopback control-plane ports and the exact source-bound config mount. The operator must also verify candidate and production stack health before and after startup.

The isolated tls service uses a minimal Caddyfile to route /api/ and /v1/ to its API and / to its Web service on the same private ingress network. It binds only port 18444. NPM remains on its own networks and keeps port 443 for production. Stage only a protected copy of the NPM certificate/key under the private candidate root, outside the image build context; never mount the NPM certificate store into the candidate. Follow docs/operations/VALIDATION_TLS_TEST_PORT.md for exact staging and HTTPS checks. Validate the certificate, Console, API calls and browser CSP without disabling TLS verification.

Candidate stack retention guard

Use scripts/rc_stack_retention.py to review server-side modelforge-validation-* Compose containers whose temporary source tree may have been removed or reduced after a run. Its default mode only inspects containers and prints JSON. It reports the oldest container age (stale after 24 hours by default), missing bind sources, and a missing config/models/initial-candidates.yaml under any /app/config bind source. Set --stale-after-hours to change the age threshold.

python3 scripts/rc_stack_retention.py
python3 scripts/rc_stack_retention.py --stale-after-hours 48

Stopping is limited to one exact stale or incomplete RC project. Before issuing docker stop, the operator must provide a nonempty retained evidence record and explicitly attest that no jobs are active. A running API, one-off job, migration, agent, runtime worker, or unknown service blocks the operation. Only running postgres, redis, or web containers from that inspected project may be stopped. The command does not remove containers, volumes, networks, or images; it never targets modelforge or a project outside the bounded validation namespace.

python3 scripts/rc_stack_retention.py \
  --stop-project modelforge-validation-example \
  --confirm-stop modelforge-validation-example \
  --confirm-no-active-jobs \
  --evidence-record /path/to/retained-rc-evidence.json

The evidence record is a UTF-8 JSON object with exactly these fields: {"schema_version":1,"project":"modelforge-validation-example","retained":true,"evidence_reference":"acceptance-run-id"}. The project must match the selected project, and the evidence reference must identify the retained acceptance output or audit evidence. The operator is responsible for confirming that the reference points to the expected evidence before the stop command.

After the workflow exists on the default branch, dispatch it directly and enter the exact pushed commit. Before merge, dispatch the existing Managed validation workflow on the candidate branch with profile build; that profile calls the same acceptance workflow with gitea.sha. Use the actual intended Console API origin for a publishable artifact; the managed-branch route uses https://modelforge.example.test and therefore produces non-publishable infrastructure rehearsal artifacts. Evidence is retained for 90 days. A failed scan or install is evidence of a blocker, not permission to bypass the gate.