7.8 KiB
Isolated public-candidate acceptance
The Gitea workflow .gitea/workflows/public-candidate-acceptance.yml moves the expensive release
proof off a developer workstation and onto a Docker-capable server runner. It is manual-only and
accepts an exact canonical commit SHA. It does not invoke unraid-deploy.yml, the deploy controller
or the production Compose project.
The job creates a curated public export, validates its content manifest, scans it with Gitleaks, then creates a temporary parentless Git commit so release provenance describes the exact exported tree. From that tree it:
- builds API, Console, Node Agent and Runtime Worker images with OCI identity;
- produces the release manifest, checksums, CycloneDX inventory and image provenance;
- scans every exact image ID with checksum-pinned Trivy and rejects ambiguous reports or any HIGH/CRITICAL finding;
- starts API, Console, PostgreSQL and Redis with the production overlay under a unique
modelforge-validation-*Compose project; - uses newly generated secrets, loopback-only ephemeral host ports and project-scoped volumes;
- proves liveness, readiness, version identity, Console serving and both sides of the operator authentication boundary; and
- always removes the temporary containers, networks, volumes and four candidate image tags. An
independent
if: always()workflow cleanup repeats the exact project-label cleanup after a failure, cancellation or timeout; it never runs a host-wide Docker prune.
No Node Agent or Runtime Worker is started by this clean-install gate. Consequently it creates no
compute identity and cannot interact with the production GPU node. A later GPU qualification must
enrol a disposable validation-canary identity with production_eligible=false and retain that
separate evidence.
P12 runtime-isolation preflight
Before a P12 Node Agent or Runtime Worker starts, render its exact candidate Compose projection to
JSON and run scripts/rc_isolation_preflight.py. The check requires a bounded
modelforge-validation-* project and refuses external/non-project volumes or networks, host
networking, global container names, non-loopback or
protected ports, Docker-socket mounts, writable bind mounts, missing bind sources, source paths
outside the candidate root, and references to named production resources. A read-only bind targeting
/app/config must contain models/initial-candidates.yaml under that exact source path. Run the
preflight before docker compose up. After up, run it again with
--inspect-docker; it records only resources carrying the exact Compose project label and rejects
any name outside that project namespace.
This preflight is an admission control, not P12 qualification evidence. A green result does not
replace the required fresh secrets, production_eligible=false validation-canary enrollment, real
private runtime inference, before/after production readiness, normal lifecycle cleanup, or retained
runtime evidence.
Dedicated test ingress for a retained RC stack
The disposable clean-install workflow above removes its stack. For a later retained P12 stack,
include docker-compose.rc-ingress.yml after the base and production files. It places PostgreSQL,
Redis and the migration job on a project-private backend network; Web is only on ingress, and
API joins both. The ingress network is created by the exact modelforge-validation-* project, not
shared with production. Bind all published control-plane ports to 127.0.0.1 and choose unused
ports; the reverse proxy reaches Web and API over the ingress network, not those host ports.
Port 443 is reserved for the existing production services. Its NPM host is still pending because
the current production Console embeds an HTTP API origin and CSP; the candidate does not change it.
The candidate must use a separate
HTTPS test listener on port 18444 with a certificate valid for modelforge.example.com. Its
canonical browser/API origin is https://modelforge.example.com:18444; set both
MODELFORGE_CORS_ORIGINS to that value at runtime and use it for VITE_API_BASE_URL when building
the candidate Console image.
The test listener routes only to the retained validation project.
Render the complete Compose plan with the exact candidate source, four image identities, freshly
generated validation secrets and intended public origin. Run rc_isolation_preflight.py on that
JSON before starting any container and again with --inspect-docker afterward. Add
--public-tls-port 18444 only with docker-compose.validation-tls.yml; the preflight admits
this one port solely for the pinned, confined tls service. It must show only project-scoped
networks and volumes, loopback control-plane ports and the exact source-bound config mount.
The operator must also verify candidate and production stack health before and after startup.
The isolated tls service uses a minimal Caddyfile to route /api/ and /v1/ to its API and
/ to its Web service on the same private ingress network. It binds only port 18444. NPM remains
on its own networks and keeps port 443 for production. Stage only a protected copy of the NPM
certificate/key under the private candidate root, outside the image build context; never mount
the NPM certificate store into the candidate. Follow
docs/operations/VALIDATION_TLS_TEST_PORT.md for exact staging and HTTPS checks. Validate the
certificate, Console, API calls and browser CSP without disabling TLS verification.
Candidate stack retention guard
Use scripts/rc_stack_retention.py to review server-side modelforge-validation-* Compose
containers whose temporary source tree may have been removed or reduced after a run. Its default
mode only inspects containers and prints JSON. It reports the oldest container age (stale after 24 hours by default),
missing bind sources, and a missing config/models/initial-candidates.yaml under any /app/config
bind source. Set --stale-after-hours to change the age threshold.
python3 scripts/rc_stack_retention.py
python3 scripts/rc_stack_retention.py --stale-after-hours 48
Stopping is limited to one exact stale or incomplete RC project. Before issuing docker stop, the
operator must provide a nonempty retained evidence record and explicitly attest that no jobs are
active. A running API, one-off job, migration, agent, runtime worker, or unknown service blocks the
operation. Only running postgres, redis, or web containers from that inspected project may be
stopped. The command does not remove containers, volumes, networks, or images; it never targets
modelforge or a project outside the bounded validation namespace.
python3 scripts/rc_stack_retention.py \
--stop-project modelforge-validation-example \
--confirm-stop modelforge-validation-example \
--confirm-no-active-jobs \
--evidence-record /path/to/retained-rc-evidence.json
The evidence record is a UTF-8 JSON object with exactly these fields:
{"schema_version":1,"project":"modelforge-validation-example","retained":true,"evidence_reference":"acceptance-run-id"}.
The project must match the selected project, and the evidence reference must identify the retained
acceptance output or audit evidence. The operator is responsible for confirming that the reference
points to the expected evidence before the stop command.
After the workflow exists on the default branch, dispatch it directly and enter the exact pushed
commit. Before merge, dispatch the existing Managed validation workflow on the candidate branch
with profile build; that profile calls the same acceptance workflow with gitea.sha. Use the
actual intended Console API origin for a publishable artifact; the managed-branch route uses
https://modelforge.example.test and therefore produces non-publishable infrastructure rehearsal
artifacts. Evidence is retained for 90 days. A failed scan or install is evidence of a blocker, not
permission to bypass the gate.