ModelForge-Public/docs/security/SUPPLY_CHAIN_RELEASE_GATE.md

# Supply-chain release gate

## Release boundary

RC-DEL-02 is a managed, fail-closed evidence gate. Pull requests, default runs, explicit `full`
runs and explicit `security` runs all require the complete source phase. Only an explicit
`workflow_dispatch` with `profile=security` builds and scans the four shipping images; `full` does
not silently turn into the expensive image profile. Neither profile can pass because a required
scanner was absent, timed out, lost network access, returned malformed data or silently omitted a
profile-required component. The job does not deploy or contact a ModelForge production service.

The checkout uses the commit selected by the Gitea run, `fetch-depth: 0`, tag fetching and
`persist-credentials: false`. An explicit fetch uses full head and tag refspecs.
`scripts/security_gate.py history` separately proves that Git reports the checkout as non-shallow,
that `HEAD` equals `GITHUB_SHA`, and that every advertised remote head and tag has the same local
object. It also records the canonical origin, all-ref and reachable-commit counts, root commits and
object-reachability proof. A non-shallow single-branch clone is incomplete and fails; checkout
configuration alone is not proof.

The job has a 120-minute bound. The explicit worst-case step bounds for the image profile total 108
minutes, including five minutes each for the always-run evaluator and artifact upload, leaving a
12-minute job-level reserve. A bootstrap `security-gate-result.json` starts in the fail state before
checkout. Evidence and tools each use an exclusive `mktemp` directory whose prefix includes run ID,
run attempt and SHA and whose unpredictable suffix prevents stale-path reuse. Both roots must be
real, runner-owned mode-0700 directories outside the workspace. The evidence location survives
checkout cleaning and remains uploadable when checkout fails. The evaluator rejects evidence-root,
directory or file symlinks/reparse points, non-regular files, ownership/mode surprises and paths
outside the evidence root. On Linux it walks with descriptor-relative `openat`/`O_NOFOLLOW`; on
Windows it rejects reparse points and compares pre-open, open-handle and post-read identities. It
validates owner/mode/type on opened handles, reads and hashes the same descriptors, retains one byte
snapshot, then revalidates ancestry, identity, size and timestamps before every consumption. A
same-inode mutation or lstat/open swap therefore fails without consuming replacement content.
Missing status records, timeout exits and interrupted or failed downloads/builds/scans all remain
blocking evidence.

## Pinned scanner acquisition

No floating scanner action, `latest` download or `curl | sh` installer is used. The runner downloads
and verifies these exact artifacts before execution:

| Scanner | Exact version/artifact | SHA-256 |
| --- | --- | --- |
| pip-audit | `pip_audit-2.10.1-py3-none-any.whl` | `99ef3f600a317c1945f1e89e227ef26e1c2d618429b8bd3fa6f4f7c440c4611a` |
| Gitleaks | `gitleaks_8.30.1_linux_x64.tar.gz` | `551f6fc83ea457d62a0d98237cbad105af8d557003051f41f3e7ca7b3f2470eb` |
| Trivy | `trivy_0.74.0_Linux-64bit.tar.gz` | `2ae6fe3ee734b7fdf11335663e18c75ea12dccc76062f09f164a3b0f8be4371a` |

The pip-audit wheel and every dependency in its scanner environment are fully resolved in
`config/security/pip-audit-toolchain.lock`. Each official PyPI wheel has an exact version and
SHA-256; acquisition uses `--require-hashes --only-binary=:all:` from `https://pypi.org/simple`, and
installation is offline from those verified downloads with `--no-index --require-hashes`. The
evaluator verifies reported runtime versions again. Trivy version/DB evidence is collected in the
mandatory source phase, so `full` does not depend on image execution.

Both Trivy DB `UpdatedAt` and `DownloadedAt` must be well-formed explicit UTC timestamps, must not
be future-dated, and `DownloadedAt` cannot predate `UpdatedAt`. The DB update may be at most 48
hours old and the local download at most 24 hours old at evaluation time. Missing, null, malformed,
non-UTC, future or stale metadata fails. npm is runner-supplied and its exact observed version is
retained rather than claimed in advance.

## Required scan matrix

The source phase collects findings without ignore flags or finding-based early exit:

| Target | Evidence |
| --- | --- |
| Complete Git history | Gitleaks `git`, `--log-opts=--all`, `--redact=100` |
| API source | pip-audit of every `[project].dependencies`, optional group and locked build tool |
| Node Agent source | same, including its exact local API dependency |
| Runtime Worker source | same, including both `dev` and `inference` groups |
| Console lock | `npm audit --json` for all dependencies |
| Console production lock | `npm audit --omit=dev --json` |
| Repository filesystem | Trivy `vuln,misconfig`, retaining UNKNOWN through CRITICAL |

Python declarations are read with `tomllib`; a hand-maintained requirement subset is not accepted.
Each PEP 517 backend requirement must have one exact version that matches both gate policy and
`config/security/build-toolchain.lock`. The lock pins `pip==26.2.1`, `setuptools==84.0.0`,
`wheel==0.48.0`, `msgpack==1.2.2`, `hatchling==1.32.4`, `editables==0.5` and Hatchling's complete build-time
dependency closure. Every artifact is installed from PyPI with `--require-hashes --only-binary=:all:`.
The msgpack hashes cover CPython 3.11-3.14 glibc x86_64/aarch64 wheels and
CPython 3.12-3.14 Alpine musl x86_64/aarch64 wheels used by managed validation and release images.
The managed source-validation environment and all Python image builds install this lock before
packaging projects, then use `--no-build-isolation` so pip cannot resolve a second build
environment. The gate audits every locked tool as an exact direct package and compares the PEP 517
requirements against the lock.
The three first-party distributions are injected into their source inventories, and local
cross-component pins must resolve to the matching project version. Source reports therefore show
explicit coverage of `modelforge-api`, `modelforge-node-agent` and
`modelforge-runtime-worker`, even though a public advisory service may correctly report that a
private distribution is not present in its index. Those exact locally proven names are separated
before the public advisory lookup because `pip-audit --strict` correctly rejects a distribution
that PyPI cannot identify; every other declared direct package remains in the strict audit input.
Source pip-audit is intentionally allowed to resolve dependencies: every declared direct package
must appear at its exact manifest version, while every additional resolved transitive package and
version is retained in `resolved_transitive_packages`. Missing or version-drifted direct packages
fail. The resolved source report can omit the build-environment tools `packaging`, `pip`,
`setuptools` and `wheel`. The gate accepts those omissions only for source resolution: all four
remain hash-locked, and shipping-image inventory still requires `packaging` when present while
forbidding retained installers. Trivy scans the repository and immutable images. No other runtime
package receives this exception. The retained pip inputs
must preserve the original PEP 508 declarations, including extras such as `uvicorn[standard]` and
`psycopg[binary]`, plus every locked build-tool entry; reconstructing a bare `name==version` line
for an extra or omitting a lock entry fails. Duplicate project declarations are rejected before
aggregation and name the exact manifest, TOML source and line for both declarations. Identical as
well as conflicting duplicate pip result rows are malformed. Source audits never use `--no-deps`
or `--disable-pip`.

The shipping-image phase builds exactly four non-pushed local tags:

```text
modelforge-security-api:${GITHUB_SHA}
modelforge-security-web:${GITHUB_SHA}
modelforge-security-node-agent:${GITHUB_SHA}
modelforge-security-runtime-worker:${GITHUB_SHA}
```

Every build uses `--pull`, an exact SHA tag and exact `org.opencontainers.image.source`, `revision`
and release `version` labels. Sanitized Docker identity evidence records those labels, the unique
tag and immutable `sha256:` image ID for all four images. The retained set must contain exactly four
release identities, exactly one approved SHA tag per image, no mutable or attacker aliases, no
unexpected digest aliases and no duplicate image IDs. Trivy scans the image ID, not a mutable tag;
a gate-owned normalizer preserves Trivy's component target separately while binding every retained
`Results[].Target` and its own `ScannedImageID` to that exact ID. The evaluator cross-checks
`ArtifactType`, `ArtifactName`, every nested target, `Metadata.ImageID`, repo tags/digests and the
reference against Docker identity evidence; Trivy's distinct config `ArtifactID` must remain an
immutable SHA-256 value.

All four Dockerfiles declare the canonical HTTPS source URL
`https://git.example.com/example/modelforge`. The workflow validates both repository identity
and server URL before building, and the evaluator requires the same canonical URL in each inspected
image. Python images take a repository-root build context so they can consume the shared hash lock.
The curated public-source exporter intentionally rewrites that private static label to its existing
sanitized URL and records it as `sourceRepository` in `PUBLIC_SOURCE_MANIFEST.json`. The manifest
hashes all four transformed Dockerfiles; both the release builder and source gate require their
labels to match that manifest identity. Public release provenance records the same sanitized source
URL alongside the canonical source SHA and manifest digest. Private managed image builds stamp and
evaluate the exact trusted repository URL resolved by their workflow.

Python executes inside the exact API, Node Agent and Runtime Worker image IDs to enumerate installed
distributions through `importlib.metadata`. Each required first-party distribution must equal the
corresponding source project version. Those complete image inventories are audited exactly with
`pip-audit --strict --no-deps --disable-pip`; unexpected or missing image packages fail. The final
Runtime Worker inventory must not contain pip, setuptools, wheel or editables; retaining any
installer/build backend blocks the gate. The image audit permits only the established
`setuptools`/`wheel` report omissions, while its exact image inventory and Trivy evidence remain
mandatory.
The digest-pinned PyTorch base also exposes exact Conda-management distributions and CUDA-local
PyTorch wheels that do not exist at those versions on PyPI. Only the enumerated Runtime Worker
target/name/version tuples are excluded from the PyPI requirements input; every tuple remains
mandatory in the retained image inventory and the whole immutable image remains subject to Trivy.
The allowlist is `conda==25.5.0`, `conda-build==25.5.0`,
`conda-libmamba-solver==25.3.0`, `libmambapy==2.1.1`, `menuinst==2.2.0`,
`torch==2.7.1+cu128`, `torchaudio==2.7.1+cu128`, and `torchvision==0.22.1+cu128`.
Absence or version drift fails before vulnerability evaluation.
All four release images are rebuilt with `--pull --no-cache` for the explicit security profile.
This prevents a previously cached package-manager upgrade layer from masquerading as a current
base-OS scan. The release builder uses the same cache-free policy. Runtime images also remove
installers after their final dependency check.
The Web build sets `VITE_API_BASE_URL=https://security-scan.invalid`; exact image
identity records its canonical nginx bundle path. Evidence collection copies that actual path from
the exact immutable image ID, starts that image, and queries the canonical served build-provenance
asset generated by Vite. The served bytes must equal the hashed bundle asset. That exact-schema
asset binds the intentional API origin with SHA-256; the compiled runtime chunk must reference the
same origin and binding, and both digests are retained inside the whole-bundle digest. Every
JavaScript chunk, source map, HTML and CSS asset is scanned for direct or
split/concatenated/template HTTP(S) and WebSocket origins. The pinned React build's W3 namespace
constants and React error-document origin are separately recorded as exact non-API origins and are
valid only in their expected namespace/error contexts. Missing provenance, any other origin,
additional attacker coexistence, dead expected markers, IPv4/IPv6 loopback and localhost-family
origins all fail. The Web image is static nginx and has no invented Python inventory. Ordinary
validation uses three separate virtual environments so components cannot inherit one another's
resolution.

## Secret evidence and historical false positives

Gitleaks 8.30.1 has upstream false-negative regression #2170 and also drops newly added rules when
they are combined with `extend.useDefault`. The gate therefore scans the same complete history
twice: built-in default rules run on a temporary config-free worktree, then `.gitleaks.toml` runs
narrow no-entropy rules for GitHub's documented `ghp_`, `gho_`, `ghu_`, `ghr_`, `github_pat_` and
both classic and 2026 stateless `ghs_` token formats. The temporary raw reports are merged outside
retained evidence. Neither configuration path contains a path, commit, or global allowlist.

Before the repository scan, the exact pinned Linux binary must independently find canonical
synthetic `ghp_`, `gho_`, `ghu_`, `ghs_`, `ghr_` and `github_pat_` histories through the applicable
custom rules, plus a canonical AWS key through a built-in/default rule. Both configurations must
leave a benign control clean. Trigger values are assembled only at test runtime. Traps remove the
temporary histories and raw reports; scanner, assertion, worktree/prune or `rm` cleanup failure
blocks the gate.

The repository's raw Gitleaks JSON is temporary and is never uploaded. Before retention, the
normalizer emits an exact version-1 object and exact finding keys: a constrained rule ID and a
SHA-256 of Gitleaks' exact fingerprint. Unknown keys and `Secret`, `Match`, `Line`, author, email,
the un-hashed fingerprint, source path or commit metadata fail or are absent from the artifact.

A real pinned Gitleaks 8.30.1 scan on 2026-08-30 proved that the full non-shallow history contains
two findings for one old M15 AES unit-test fixture. Rewriting canonical history is not an acceptable
release operation. The fixture bytes were synthetic test material and never a credential or a
production backup key, so the two observations are classified `FALSE_POSITIVE` in
`config/security/gitleaks-false-positives.json` by exact SHA-256 of the Gitleaks fingerprint and
exact rule ID. The registry contains no path, glob, commit or secret value.

This is deliberately narrower than a scanner allowlist:

- each classification must match exactly one sanitized finding;
- a missing, changed, duplicated or additional fingerprint fails;
- duplicate registry entries and wildcard rule IDs fail;
- owner, classification date, review date, tracking reference and reason are mandatory;
- the review window is at most 30 days and a stale review fails;
- no Gitleaks finding can use the vulnerability accepted-risk mechanism.

Any new leak therefore remains blocking, including another finding from the same rule or file.

## Central decision and accepted risk

Scanners collect facts; only `scripts/security_gate.py evaluate` decides. It rejects missing, empty
or malformed reports, unexpected scanner exits, network/tool failures, timeouts, shallow history,
version drift, source/inventory mismatches and every expected-versus-actual target gap. It then
blocks:

- every unclassified Gitleaks finding;
- every pip-audit vulnerability;
- every npm vulnerability;
- every Trivy UNKNOWN, HIGH or CRITICAL vulnerability or misconfiguration.

LOW and MEDIUM Trivy observations remain in the raw machine-readable reports but are not RC-DEL-02
blocking severities. No scanner is instructed to omit them.

`config/security/accepted-risks.json` is a versioned object. It starts empty and contains only
separately reviewed, temporary exceptions. An entry has exactly these fields:

```json
{
  "scanner": "trivy",
  "target": "node-agent",
  "advisory_id": "CVE-YYYY-NNNN",
  "package": "package-name",
  "installed_version": "1.2.3",
  "classification": "ACCEPTED_TEMPORARY",
  "owner": "named owner",
  "accepted_on": "2026-08-30",
  "expires_on": "2026-09-29",
  "tracking": "https://issue.example/RC-DEL-02",
  "reason": "Evidence-backed reason that the no-fix risk is temporarily acceptable."
}
```

The identity is the exact five-value tuple `(scanner, target, advisory_id, package,
installed_version)`. Classification must be exactly `ACCEPTED_TEMPORARY`; wildcards and extra
fields fail. Only pip-audit, npm-audit and Trivy
vulnerabilities with no available fix are eligible. The window is at most 30 calendar dates;
`expires_on` is the first invalid date. Future,
expired, duplicate, ambiguous, unused/stale, malformed and fix-available entries all fail the gate.
An accepted entry that no longer matches a finding is not harmless configuration; it is stale
policy and blocks the release.

For the v1.2.2 candidate, the registry contains 23 exact Trivy tuples for the Node Agent Debian
Trixie base. Every tuple was reported without an upstream fix by the exact-image run, is owned by
the ModelForge security owner, expires on 2026-10-02, and is tracked in
`https://git.example.com/example/modelforge/issues/24`. The current Bookworm alternative had
additional critical no-fix findings; the tested distroless Debian 13 alternative changes Python,
UID and runtime compatibility. No fix-available finding is accepted. The next scan must consume
every tuple exactly; a fixed, changed, missing or newly introduced tuple fails closed.
The source-only `full` profile validates the registry schema and active dates but defers the
unused/stale decision for image-targeted Trivy entries, because it produces no image evidence.
Only the explicit `security` profile may consume those entries, and it fails when any such tuple is
unused. A passing `full` run therefore never substitutes for the mandatory exact-image run.

Trivy report parsing is strict before risk matching. JSON evidence rejects duplicate object keys;
the pinned Trivy 0.74.0 filesystem scan must report `ArtifactType=repository`, and its
`ArtifactName` must equal the resolved checkout path exactly. Every result target must remain
inside that checkout. Image `ArtifactType`, `ArtifactName` and every result target are likewise
required and bound to the immutable image identity. Each vulnerability must contain
non-empty `VulnerabilityID`, `PkgName`, `InstalledVersion` and a valid `Severity`; malformed values
become schema-policy failures, never fallback identities, and therefore can never be waived.

## Retained result

The evaluator always replaces the bootstrap result when it runs and writes:

- commit, ref, run ID, profile and evaluation time;
- exact scanner versions and Trivy database time;
- every expected operation's exit code;
- SHA-256 for every retained, manifest-approved report (unknown evidence paths fail before bytes are
  read or hashed);
- expected and actual targets by scan class;
- resolved source transitive packages and sanitized image identities when applicable;
- blocked findings;
- applied and unused/invalid false-positive classifications;
- applied and unused/invalid accepted risks;
- the final `PASS` or `FAIL` decision.

The security job uploads its unique retained
`/tmp/modelforge-security-evidence-${run_id}-${run_attempt}-${sha}.${random}` directory with
`gitea-upload-artifact@81f940d004763f986ba3582c007fd842dd5cb0d7`, `if: always()`,
`if-no-files-found: error` and 90-day retention. Its artifact name includes run ID, run attempt and
source SHA. Upload failure is a failed gate, not an evidence-retention warning.

Implementation and synthetic local tests are not release verification. RC-DEL-02 remains
unverified until both an exact-head managed `security` run (including its retained artifact) and an
exact-head managed `full` run succeed. A first real runner execution may legitimately be red because
it sees current dependency/image advisories or runner/network deficiencies; no historical
vulnerability risk has been seeded to manufacture a green result.