123 lines
7.8 KiB
Markdown
123 lines
7.8 KiB
Markdown
# Isolated public-candidate acceptance
|
|
|
|
The Gitea workflow `.gitea/workflows/public-candidate-acceptance.yml` moves the expensive release
|
|
proof off a developer workstation and onto a Docker-capable server runner. It is manual-only and
|
|
accepts an exact canonical commit SHA. It does not invoke `unraid-deploy.yml`, the deploy controller
|
|
or the production Compose project.
|
|
|
|
The job creates a curated public export, validates its content manifest, scans it with Gitleaks,
|
|
then creates a temporary parentless Git commit so release provenance describes the exact exported
|
|
tree. From that tree it:
|
|
|
|
1. builds API, Console, Node Agent and Runtime Worker images with OCI identity;
|
|
2. produces the release manifest, checksums, CycloneDX inventory and image provenance;
|
|
3. scans every exact image ID with checksum-pinned Trivy and rejects ambiguous reports or any
|
|
HIGH/CRITICAL finding;
|
|
4. starts API, Console, PostgreSQL and Redis with the production overlay under a unique
|
|
`modelforge-validation-*` Compose project;
|
|
5. uses newly generated secrets, loopback-only ephemeral host ports and project-scoped volumes;
|
|
6. proves liveness, readiness, version identity, Console serving and both sides of the operator
|
|
authentication boundary; and
|
|
7. always removes the temporary containers, networks, volumes and four candidate image tags. An
|
|
independent `if: always()` workflow cleanup repeats the exact project-label cleanup after a
|
|
failure, cancellation or timeout; it never runs a host-wide Docker prune.
|
|
|
|
No Node Agent or Runtime Worker is started by this clean-install gate. Consequently it creates no
|
|
compute identity and cannot interact with the production GPU node. A later GPU qualification must
|
|
enrol a disposable `validation-canary` identity with `production_eligible=false` and retain that
|
|
separate evidence.
|
|
|
|
## P12 runtime-isolation preflight
|
|
|
|
Before a P12 Node Agent or Runtime Worker starts, render its exact candidate Compose projection to
|
|
JSON and run `scripts/rc_isolation_preflight.py`. The check requires a bounded
|
|
`modelforge-validation-*` project and refuses external/non-project volumes or networks, host
|
|
networking, global container names, non-loopback or
|
|
protected ports, Docker-socket mounts, writable bind mounts, missing bind sources, source paths
|
|
outside the candidate root, and references to named production resources. A read-only bind targeting
|
|
`/app/config` must contain `models/initial-candidates.yaml` under that exact source path. Run the
|
|
preflight before `docker compose up`. After `up`, run it again with
|
|
`--inspect-docker`; it records only resources carrying the exact Compose project label and rejects
|
|
any name outside that project namespace.
|
|
|
|
This preflight is an admission control, not P12 qualification evidence. A green result does not
|
|
replace the required fresh secrets, `production_eligible=false` `validation-canary` enrollment, real
|
|
private runtime inference, before/after production readiness, normal lifecycle cleanup, or retained
|
|
runtime evidence.
|
|
|
|
### Dedicated test ingress for a retained RC stack
|
|
|
|
The disposable clean-install workflow above removes its stack. For a later retained P12 stack,
|
|
include `docker-compose.rc-ingress.yml` after the base and production files. It places PostgreSQL,
|
|
Redis and the migration job on a project-private `backend` network; Web is only on `ingress`, and
|
|
API joins both. The ingress network is created by the exact `modelforge-validation-*` project, not
|
|
shared with production. Bind all published control-plane ports to `127.0.0.1` and choose unused
|
|
ports; the reverse proxy reaches Web and API over the ingress network, not those host ports.
|
|
|
|
Port 443 is reserved for the existing production services. Its NPM host is still pending because
|
|
the current production Console embeds an HTTP API origin and CSP; the candidate does not change it.
|
|
The candidate must use a separate
|
|
HTTPS test listener on port 18444 with a certificate valid for `modelforge.example.com`. Its
|
|
canonical browser/API origin is `https://modelforge.example.com:18444`; set both
|
|
`MODELFORGE_CORS_ORIGINS` to that value at runtime and use it for `VITE_API_BASE_URL` when building
|
|
the candidate Console image.
|
|
The test listener routes only to the retained validation project.
|
|
|
|
Render the complete Compose plan with the exact candidate source, four image identities, freshly
|
|
generated validation secrets and intended public origin. Run `rc_isolation_preflight.py` on that
|
|
JSON before starting any container and again with `--inspect-docker` afterward. Add
|
|
`--public-tls-port 18444` only with `docker-compose.validation-tls.yml`; the preflight admits
|
|
this one port solely for the pinned, confined `tls` service. It must show only project-scoped
|
|
networks and volumes, loopback control-plane ports and the exact source-bound config mount.
|
|
The operator must also verify candidate and production stack health before and after startup.
|
|
|
|
The isolated `tls` service uses a minimal Caddyfile to route `/api/` and `/v1/` to its API and
|
|
`/` to its Web service on the same private ingress network. It binds only port 18444. NPM remains
|
|
on its own networks and keeps port 443 for production. Stage only a protected copy of the NPM
|
|
certificate/key under the private candidate root, outside the image build context; never mount
|
|
the NPM certificate store into the candidate. Follow
|
|
`docs/operations/VALIDATION_TLS_TEST_PORT.md` for exact staging and HTTPS checks. Validate the
|
|
certificate, Console, API calls and browser CSP without disabling TLS verification.
|
|
|
|
## Candidate stack retention guard
|
|
|
|
Use `scripts/rc_stack_retention.py` to review server-side `modelforge-validation-*` Compose
|
|
containers whose temporary source tree may have been removed or reduced after a run. Its default
|
|
mode only inspects containers and prints JSON. It reports the oldest container age (stale after 24 hours by default),
|
|
missing bind sources, and a missing `config/models/initial-candidates.yaml` under any `/app/config`
|
|
bind source. Set `--stale-after-hours` to change the age threshold.
|
|
|
|
```sh
|
|
python3 scripts/rc_stack_retention.py
|
|
python3 scripts/rc_stack_retention.py --stale-after-hours 48
|
|
```
|
|
|
|
Stopping is limited to one exact stale or incomplete RC project. Before issuing `docker stop`, the
|
|
operator must provide a nonempty retained evidence record and explicitly attest that no jobs are
|
|
active. A running API, one-off job, migration, agent, runtime worker, or unknown service blocks the
|
|
operation. Only running `postgres`, `redis`, or `web` containers from that inspected project may be
|
|
stopped. The command does not remove containers, volumes, networks, or images; it never targets
|
|
`modelforge` or a project outside the bounded validation namespace.
|
|
|
|
```sh
|
|
python3 scripts/rc_stack_retention.py \
|
|
--stop-project modelforge-validation-example \
|
|
--confirm-stop modelforge-validation-example \
|
|
--confirm-no-active-jobs \
|
|
--evidence-record /path/to/retained-rc-evidence.json
|
|
```
|
|
|
|
The evidence record is a UTF-8 JSON object with exactly these fields:
|
|
`{"schema_version":1,"project":"modelforge-validation-example","retained":true,"evidence_reference":"acceptance-run-id"}`.
|
|
The project must match the selected project, and the evidence reference must identify the retained
|
|
acceptance output or audit evidence. The operator is responsible for confirming that the reference
|
|
points to the expected evidence before the stop command.
|
|
|
|
After the workflow exists on the default branch, dispatch it directly and enter the exact pushed
|
|
commit. Before merge, dispatch the existing **Managed validation** workflow on the candidate branch
|
|
with profile `build`; that profile calls the same acceptance workflow with `gitea.sha`. Use the
|
|
actual intended Console API origin for a publishable artifact; the managed-branch route uses
|
|
`https://modelforge.example.test` and therefore produces non-publishable infrastructure rehearsal
|
|
artifacts. Evidence is retained for 90 days. A failed scan or install is evidence of a blocker, not
|
|
permission to bypass the gate.
|