Supply-chain release gate

Release boundary

RC-DEL-02 is a managed, fail-closed evidence gate. Pull requests, default runs, explicit full runs and explicit security runs all require the complete source phase. Only an explicit workflow_dispatch with profile=security builds and scans the four shipping images; full does not silently turn into the expensive image profile. Neither profile can pass because a required scanner was absent, timed out, lost network access, returned malformed data or silently omitted a profile-required component. The job does not deploy or contact a ModelForge production service.

The checkout uses the commit selected by the Gitea run, fetch-depth: 0, tag fetching and persist-credentials: false. An explicit fetch uses full head and tag refspecs. scripts/security_gate.py history separately proves that Git reports the checkout as non-shallow, that HEAD equals GITHUB_SHA, and that every advertised remote head and tag has the same local object. It also records the canonical origin, all-ref and reachable-commit counts, root commits and object-reachability proof. A non-shallow single-branch clone is incomplete and fails; checkout configuration alone is not proof.

The job has a 120-minute bound. The explicit worst-case step bounds for the image profile total 108 minutes, including five minutes each for the always-run evaluator and artifact upload, leaving a 12-minute job-level reserve. A bootstrap security-gate-result.json starts in the fail state before checkout. Evidence and tools each use an exclusive mktemp directory whose prefix includes run ID, run attempt and SHA and whose unpredictable suffix prevents stale-path reuse. Both roots must be real, runner-owned mode-0700 directories outside the workspace. The evidence location survives checkout cleaning and remains uploadable when checkout fails. The evaluator rejects evidence-root, directory or file symlinks/reparse points, non-regular files, ownership/mode surprises and paths outside the evidence root. On Linux it walks with descriptor-relative openat/O_NOFOLLOW; on Windows it rejects reparse points and compares pre-open, open-handle and post-read identities. It validates owner/mode/type on opened handles, reads and hashes the same descriptors, retains one byte snapshot, then revalidates ancestry, identity, size and timestamps before every consumption. A same-inode mutation or lstat/open swap therefore fails without consuming replacement content. Missing status records, timeout exits and interrupted or failed downloads/builds/scans all remain blocking evidence.

Pinned scanner acquisition

No floating scanner action, latest download or curl | sh installer is used. The runner downloads and verifies these exact artifacts before execution:

Scanner	Exact version/artifact	SHA-256
pip-audit	`pip_audit-2.10.1-py3-none-any.whl`	`99ef3f600a317c1945f1e89e227ef26e1c2d618429b8bd3fa6f4f7c440c4611a`
Gitleaks	`gitleaks_8.30.1_linux_x64.tar.gz`	`551f6fc83ea457d62a0d98237cbad105af8d557003051f41f3e7ca7b3f2470eb`
Trivy	`trivy_0.74.0_Linux-64bit.tar.gz`	`2ae6fe3ee734b7fdf11335663e18c75ea12dccc76062f09f164a3b0f8be4371a`

The pip-audit wheel and every dependency in its scanner environment are fully resolved in config/security/pip-audit-toolchain.lock. Each official PyPI wheel has an exact version and SHA-256; acquisition uses --require-hashes --only-binary=:all: from https://pypi.org/simple, and installation is offline from those verified downloads with --no-index --require-hashes. The evaluator verifies reported runtime versions again. Trivy version/DB evidence is collected in the mandatory source phase, so full does not depend on image execution.

Both Trivy DB UpdatedAt and DownloadedAt must be well-formed explicit UTC timestamps, must not be future-dated, and DownloadedAt cannot predate UpdatedAt. The DB update may be at most 48 hours old and the local download at most 24 hours old at evaluation time. Missing, null, malformed, non-UTC, future or stale metadata fails. npm is runner-supplied and its exact observed version is retained rather than claimed in advance.

Required scan matrix

The source phase collects findings without ignore flags or finding-based early exit:

Target	Evidence
Complete Git history	Gitleaks `git`, `--log-opts=--all`, `--redact=100`
API source	pip-audit of every `[project].dependencies`, optional group and locked build tool
Node Agent source	same, including its exact local API dependency
Runtime Worker source	same, including both `dev` and `inference` groups
Console lock	`npm audit --json` for all dependencies
Console production lock	`npm audit --omit=dev --json`
Repository filesystem	Trivy `vuln,misconfig`, retaining UNKNOWN through CRITICAL

Python declarations are read with tomllib; a hand-maintained requirement subset is not accepted. Each PEP 517 backend requirement must have one exact version that matches both gate policy and config/security/build-toolchain.lock. The lock pins pip==26.2.1, setuptools==84.0.0, wheel==0.48.0, msgpack==1.2.2, hatchling==1.32.4, editables==0.5 and Hatchling's complete build-time dependency closure. Every artifact is installed from PyPI with --require-hashes --only-binary=:all:. The msgpack hashes cover CPython 3.11-3.14 glibc x86_64/aarch64 wheels and CPython 3.12-3.14 Alpine musl x86_64/aarch64 wheels used by managed validation and release images. The managed source-validation environment and all Python image builds install this lock before packaging projects, then use --no-build-isolation so pip cannot resolve a second build environment. The gate audits every locked tool as an exact direct package and compares the PEP 517 requirements against the lock. The three first-party distributions are injected into their source inventories, and local cross-component pins must resolve to the matching project version. Source reports therefore show explicit coverage of modelforge-api, modelforge-node-agent and modelforge-runtime-worker, even though a public advisory service may correctly report that a private distribution is not present in its index. Those exact locally proven names are separated before the public advisory lookup because pip-audit --strict correctly rejects a distribution that PyPI cannot identify; every other declared direct package remains in the strict audit input. Source pip-audit is intentionally allowed to resolve dependencies: every declared direct package must appear at its exact manifest version, while every additional resolved transitive package and version is retained in resolved_transitive_packages. Missing or version-drifted direct packages fail. The resolved source report can omit the build-environment tools packaging, pip, setuptools and wheel. The gate accepts those omissions only for source resolution: all four remain hash-locked, and shipping-image inventory still requires packaging when present while forbidding retained installers. Trivy scans the repository and immutable images. No other runtime package receives this exception. The retained pip inputs must preserve the original PEP 508 declarations, including extras such as uvicorn[standard] and psycopg[binary], plus every locked build-tool entry; reconstructing a bare name==version line for an extra or omitting a lock entry fails. Duplicate project declarations are rejected before aggregation and name the exact manifest, TOML source and line for both declarations. Identical as well as conflicting duplicate pip result rows are malformed. Source audits never use --no-deps or --disable-pip.

The shipping-image phase builds exactly four non-pushed local tags:

modelforge-security-api:${GITHUB_SHA}
modelforge-security-web:${GITHUB_SHA}
modelforge-security-node-agent:${GITHUB_SHA}
modelforge-security-runtime-worker:${GITHUB_SHA}

Every build uses --pull, an exact SHA tag and exact org.opencontainers.image.source, revision and release version labels. Sanitized Docker identity evidence records those labels, the unique tag and immutable sha256: image ID for all four images. The retained set must contain exactly four release identities, exactly one approved SHA tag per image, no mutable or attacker aliases, no unexpected digest aliases and no duplicate image IDs. Trivy scans the image ID, not a mutable tag; a gate-owned normalizer preserves Trivy's component target separately while binding every retained Results[].Target and its own ScannedImageID to that exact ID. The evaluator cross-checks ArtifactType, ArtifactName, every nested target, Metadata.ImageID, repo tags/digests and the reference against Docker identity evidence; Trivy's distinct config ArtifactID must remain an immutable SHA-256 value.

All four Dockerfiles declare the canonical HTTPS source URL https://git.example.com/example/modelforge. The workflow validates both repository identity and server URL before building, and the evaluator requires the same canonical URL in each inspected image. Python images take a repository-root build context so they can consume the shared hash lock. The curated public-source exporter intentionally rewrites that private static label to its existing sanitized URL and records it as sourceRepository in PUBLIC_SOURCE_MANIFEST.json. The manifest hashes all four transformed Dockerfiles; both the release builder and source gate require their labels to match that manifest identity. Public release provenance records the same sanitized source URL alongside the canonical source SHA and manifest digest. Private managed image builds stamp and evaluate the exact trusted repository URL resolved by their workflow.

Python executes inside the exact API, Node Agent and Runtime Worker image IDs to enumerate installed distributions through importlib.metadata. Each required first-party distribution must equal the corresponding source project version. Those complete image inventories are audited exactly with pip-audit --strict --no-deps --disable-pip; unexpected or missing image packages fail. The final Runtime Worker inventory must not contain pip, setuptools, wheel or editables; retaining any installer/build backend blocks the gate. The image audit permits only the established setuptools/wheel report omissions, while its exact image inventory and Trivy evidence remain mandatory. The digest-pinned PyTorch base also exposes exact Conda-management distributions and CUDA-local PyTorch wheels that do not exist at those versions on PyPI. Only the enumerated Runtime Worker target/name/version tuples are excluded from the PyPI requirements input; every tuple remains mandatory in the retained image inventory and the whole immutable image remains subject to Trivy. The allowlist is conda==25.5.0, conda-build==25.5.0, conda-libmamba-solver==25.3.0, libmambapy==2.1.1, menuinst==2.2.0, torch==2.7.1+cu128, torchaudio==2.7.1+cu128, and torchvision==0.22.1+cu128. Absence or version drift fails before vulnerability evaluation. All four release images are rebuilt with --pull --no-cache for the explicit security profile. This prevents a previously cached package-manager upgrade layer from masquerading as a current base-OS scan. The release builder uses the same cache-free policy. Runtime images also remove installers after their final dependency check. The Web build sets VITE_API_BASE_URL=https://security-scan.invalid; exact image identity records its canonical nginx bundle path. Evidence collection copies that actual path from the exact immutable image ID, starts that image, and queries the canonical served build-provenance asset generated by Vite. The served bytes must equal the hashed bundle asset. That exact-schema asset binds the intentional API origin with SHA-256; the compiled runtime chunk must reference the same origin and binding, and both digests are retained inside the whole-bundle digest. Every JavaScript chunk, source map, HTML and CSS asset is scanned for direct or split/concatenated/template HTTP(S) and WebSocket origins. The pinned React build's W3 namespace constants and React error-document origin are separately recorded as exact non-API origins and are valid only in their expected namespace/error contexts. Missing provenance, any other origin, additional attacker coexistence, dead expected markers, IPv4/IPv6 loopback and localhost-family origins all fail. The Web image is static nginx and has no invented Python inventory. Ordinary validation uses three separate virtual environments so components cannot inherit one another's resolution.

Secret evidence and historical false positives

Gitleaks 8.30.1 has upstream false-negative regression #2170 and also drops newly added rules when they are combined with extend.useDefault. The gate therefore scans the same complete history twice: built-in default rules run on a temporary config-free worktree, then .gitleaks.toml runs narrow no-entropy rules for GitHub's documented ghp_, gho_, ghu_, ghr_, github_pat_ and both classic and 2026 stateless ghs_ token formats. The temporary raw reports are merged outside retained evidence. Neither configuration path contains a path, commit, or global allowlist.

Before the repository scan, the exact pinned Linux binary must independently find canonical synthetic ghp_, gho_, ghu_, ghs_, ghr_ and github_pat_ histories through the applicable custom rules, plus a canonical AWS key through a built-in/default rule. Both configurations must leave a benign control clean. Trigger values are assembled only at test runtime. Traps remove the temporary histories and raw reports; scanner, assertion, worktree/prune or rm cleanup failure blocks the gate.

The repository's raw Gitleaks JSON is temporary and is never uploaded. Before retention, the normalizer emits an exact version-1 object and exact finding keys: a constrained rule ID and a SHA-256 of Gitleaks' exact fingerprint. Unknown keys and Secret, Match, Line, author, email, the un-hashed fingerprint, source path or commit metadata fail or are absent from the artifact.

A real pinned Gitleaks 8.30.1 scan on 2026-08-30 proved that the full non-shallow history contains two findings for one old M15 AES unit-test fixture. Rewriting canonical history is not an acceptable release operation. The fixture bytes were synthetic test material and never a credential or a production backup key, so the two observations are classified FALSE_POSITIVE in config/security/gitleaks-false-positives.json by exact SHA-256 of the Gitleaks fingerprint and exact rule ID. The registry contains no path, glob, commit or secret value.

This is deliberately narrower than a scanner allowlist:

each classification must match exactly one sanitized finding;
a missing, changed, duplicated or additional fingerprint fails;
duplicate registry entries and wildcard rule IDs fail;
owner, classification date, review date, tracking reference and reason are mandatory;
the review window is at most 30 days and a stale review fails;
no Gitleaks finding can use the vulnerability accepted-risk mechanism.

Any new leak therefore remains blocking, including another finding from the same rule or file.

Central decision and accepted risk

Scanners collect facts; only scripts/security_gate.py evaluate decides. It rejects missing, empty or malformed reports, unexpected scanner exits, network/tool failures, timeouts, shallow history, version drift, source/inventory mismatches and every expected-versus-actual target gap. It then blocks:

every unclassified Gitleaks finding;
every pip-audit vulnerability;
every npm vulnerability;
every Trivy UNKNOWN, HIGH or CRITICAL vulnerability or misconfiguration.

LOW and MEDIUM Trivy observations remain in the raw machine-readable reports but are not RC-DEL-02 blocking severities. No scanner is instructed to omit them.

config/security/accepted-risks.json is a versioned object. It starts empty and contains only separately reviewed, temporary exceptions. An entry has exactly these fields:

{
  "scanner": "trivy",
  "target": "node-agent",
  "advisory_id": "CVE-YYYY-NNNN",
  "package": "package-name",
  "installed_version": "1.2.3",
  "classification": "ACCEPTED_TEMPORARY",
  "owner": "named owner",
  "accepted_on": "2026-08-30",
  "expires_on": "2026-09-29",
  "tracking": "https://issue.example/RC-DEL-02",
  "reason": "Evidence-backed reason that the no-fix risk is temporarily acceptable."
}

The identity is the exact five-value tuple (scanner, target, advisory_id, package, installed_version). Classification must be exactly ACCEPTED_TEMPORARY; wildcards and extra fields fail. Only pip-audit, npm-audit and Trivy vulnerabilities with no available fix are eligible. The window is at most 30 calendar dates; expires_on is the first invalid date. Future, expired, duplicate, ambiguous, unused/stale, malformed and fix-available entries all fail the gate. An accepted entry that no longer matches a finding is not harmless configuration; it is stale policy and blocks the release.

For the v1.2.2 candidate, the registry contains 23 exact Trivy tuples for the Node Agent Debian Trixie base. Every tuple was reported without an upstream fix by the exact-image run, is owned by the ModelForge security owner, expires on 2026-10-02, and is tracked in https://git.example.com/example/modelforge/issues/24. The current Bookworm alternative had additional critical no-fix findings; the tested distroless Debian 13 alternative changes Python, UID and runtime compatibility. No fix-available finding is accepted. The next scan must consume every tuple exactly; a fixed, changed, missing or newly introduced tuple fails closed. The source-only full profile validates the registry schema and active dates but defers the unused/stale decision for image-targeted Trivy entries, because it produces no image evidence. Only the explicit security profile may consume those entries, and it fails when any such tuple is unused. A passing full run therefore never substitutes for the mandatory exact-image run.

Trivy report parsing is strict before risk matching. JSON evidence rejects duplicate object keys; the pinned Trivy 0.74.0 filesystem scan must report ArtifactType=repository, and its ArtifactName must equal the resolved checkout path exactly. Every result target must remain inside that checkout. Image ArtifactType, ArtifactName and every result target are likewise required and bound to the immutable image identity. Each vulnerability must contain non-empty VulnerabilityID, PkgName, InstalledVersion and a valid Severity; malformed values become schema-policy failures, never fallback identities, and therefore can never be waived.

Retained result

The evaluator always replaces the bootstrap result when it runs and writes:

commit, ref, run ID, profile and evaluation time;
exact scanner versions and Trivy database time;
every expected operation's exit code;
SHA-256 for every retained, manifest-approved report (unknown evidence paths fail before bytes are read or hashed);
expected and actual targets by scan class;
resolved source transitive packages and sanitized image identities when applicable;
blocked findings;
applied and unused/invalid false-positive classifications;
applied and unused/invalid accepted risks;
the final PASS or FAIL decision.

The security job uploads its unique retained /tmp/modelforge-security-evidence-${run_id}-${run_attempt}-${sha}.${random} directory with gitea-upload-artifact@81f940d004763f986ba3582c007fd842dd5cb0d7, if: always(), if-no-files-found: error and 90-day retention. Its artifact name includes run ID, run attempt and source SHA. Upload failure is a failed gate, not an evidence-retention warning.

Implementation and synthetic local tests are not release verification. RC-DEL-02 remains unverified until both an exact-head managed security run (including its retained artifact) and an exact-head managed full run succeed. A first real runner execution may legitimately be red because it sees current dependency/image advisories or runner/network deficiencies; no historical vulnerability risk has been seeded to manufacture a green result.

20 KiB Raw Blame History