Publish curated ITWorx-ModelForge source from 990401737f96c8d46a9f7b0d84994516cadcb6a0

2026-09-30 09:23:17 +00:00
1 parent 7082ab955a
commit 848f0d8f8a
@@ -1,32 +1,31 @@
-# Compatibility matrix — v1.2.1 plus unreleased RC hardening
+# Compatibility matrix — v1.2.2 source/tag contract

 Nothing here is implied. The executable contract lives in
 `backend/src/modelforge_api/domain/release.py`; startup and release tests enforce it.

-> RC development note: the unreleased product branch now targets additive schema
-> `20260830_0024`, accepts `20260828_0022` and auth-only `20260830_0023` as upgrade sources, and refuses to serve until the
-> audit-chain migration completes. Deployed stable v1.2.1 remains frozen on `20260828_0022`; the
-> released matrix below continues to describe that production artifact until the RC is tagged.
-> Selecting and stamping the post-0024 product version is an explicit pending release tranche; the
-> temporary repository `1.2.1` constant must not be presented as a released schema-0024 artifact.
+> Publication and deployment note: this file records the v1.2.2 source/tag compatibility contract.
+> At the 2026-09-25 reconciliation checkpoint, Gitea's newest published Release object was v1.2.1;
+> v1.2.2 existed as a tag. The observed production record was v1.2.3/schema-0032. The RC source
+> is version 1.3.0-rc.1 with schema target 20260924_0039; it must not be treated as deployed until
+> its own gates pass.

 ## Product and database

 | Contract | Value |
 | --- | --- |
-| Version | `1.2.1` |
+| Version | `1.2.2` |
 | Channel | `stable` |
 | Version source | repository `VERSION` |
-| Runtime schema | `20260830_0023` |
-| Direct upgrade-source schema | `20260827_0021`, `20260828_0022` |
+| Runtime schema | `20260830_0024` |
+| Direct upgrade-source schema | `20260827_0021`, `20260828_0022`, `20260830_0023` |
 | Minimum direct upgrade | `v1.0.0` |
 | PostgreSQL | 16+; Compose ships 17 |
 | SQLite | unit tests only; **NOT SUPPORTED FOR PRODUCTION** |

-The hardened candidate serves only on schema 0023. The upgrade tool accepts 0021 or the production
-0022 head, applies the intervening revisions, validates all existing node credential scopes, and
-reaches 0023 before the API starts. PostgreSQL is the only production database contract; SQLite is
-used only to exercise DDL mechanics in isolated tests.
+v1.2.2 serves only on schema 0024. The upgrade tool accepts 0021, the production 0022 head or the
+intermediate auth-only 0023 head, applies the intervening revisions, validates existing node
+credential scopes and establishes the PostgreSQL-owned audit boundary before the API starts.
+PostgreSQL is the only production database contract; SQLite is used only for isolated tests.

 Compatibility answers remain typed and fail closed:

@@ -45,19 +44,22 @@ Compatibility answers remain typed and fail closed:
 | Accepted protocols | 1 |

 The Node Agent and Runtime Worker remain protocol-compatible with v1.0.0. Release images should
-still be upgraded together so their product and provenance identity is v1.2.1.
+still be upgraded together so their product and provenance identity is v1.2.2. Because the Runtime
+Worker dependency update is behavioral, existing profiles require fresh LAB evidence before
+production promotion.

 ## Upgrade matrix

 | From | Schema | To | Supported |
 | --- | --- | --- | --- |
-| `v1.0.0` | `20260827_0021` | current candidate / `20260830_0023` | yes; backup and rehearsal required |
-| `v1.1.0` | `20260828_0022` | current candidate / `20260830_0023` | yes; scope validation migration |
-| `v1.1.1` | `20260828_0022` | current candidate / `20260830_0023` | yes; scope validation migration |
-| `v1.2.0` | `20260828_0022` | current candidate / `20260830_0023` | yes; scope validation migration |
-| `v1.2.1` | `20260828_0022` | current candidate / `20260830_0023` | yes; scope validation migration |
-| current candidate | `20260830_0023` | current candidate | yes; already current |
-| older than `v1.0.0` | any | `v1.2.1` | no direct upgrade |
+| `v1.0.0` | `20260827_0021` | `v1.2.2` / `20260830_0024` | yes; backup and rehearsal required |
+| `v1.1.0` | `20260828_0022` | `v1.2.2` / `20260830_0024` | yes; scope and audit migration |
+| `v1.1.1` | `20260828_0022` | `v1.2.2` / `20260830_0024` | yes; scope and audit migration |
+| `v1.2.0` | `20260828_0022` | `v1.2.2` / `20260830_0024` | yes; scope and audit migration |
+| `v1.2.1` | `20260828_0022` | `v1.2.2` / `20260830_0024` | yes; scope and audit migration |
+| intermediate RC | `20260830_0023` | `v1.2.2` / `20260830_0024` | yes; audit-boundary migration |
+| `v1.2.2` | `20260830_0024` | `v1.2.2` | yes; already current |
+| older than `v1.0.0` | any | `v1.2.2` | no direct upgrade |

 Because 0022 introduces terminal node and decommission-operation state, application-only rollback
 to v1.0.0 is unsupported. Restore the verified pre-upgrade backup. Alembic downgrade may be used in
@@ -95,6 +97,13 @@ PATCH under SemVer.
 The console image is the one artifact whose *behaviour* differs, because the previous images could
 not reach a non-localhost API at all. That is a defect being repaired, not a capability being added.

+## Why v1.2.2 is a PATCH release
+
+v1.2.2 hardens public packaging, dependency and database-authority boundaries without changing the
+capability API or agent protocol. The Runtime Worker dependency set changes and is therefore
+classified `behavioral` operationally: it remains a patch release, but production promotion is
+blocked until exact LAB probes prove compatibility, output shape and unload/reclaim behavior.
+
 ## Why v1.2.0 is a MINOR release

 v1.2.0 adds substantive, backward-compatible product capability at the console layer: a command
@@ -1,6 +1,6 @@
 # Configuration reference

-Generated from the typed settings by `scripts/generate_configuration_docs.py` for ITWorx ModelForge 1.2.1. Every setting the control plane reads appears here; a setting added without documentation fails the build.
+Generated from the typed settings by `scripts/generate_configuration_docs.py` for ITWorx ModelForge 1.3.0-rc.1. Every setting the control plane reads appears here; a setting added without documentation fails the build.

 All settings are environment variables with the `MODELFORGE_` prefix, read from the process environment or from `.env`.

@@ -10,7 +10,6 @@ All settings are environment variables with the `MODELFORGE_` prefix, read from

 - `MODELFORGE_DATABASE_URL` — SQLAlchemy URL for the API's non-owner modelforge_runtime role. It may read the audit trail and execute the canonical append function, but cannot mutate audit tables directly.
 - `MODELFORGE_REDIS_URL` — Redis URL for transient request payloads and queues.
- `MODELFORGE_OPERATOR_API_KEY` — Operator API key guarding every admin route. Generate at least 32 random characters; ModelForge never mints one for you.
 - `MODELFORGE_BACKUP_ENCRYPTION_KEY` — Base64 AES-256 key for backup encryption. Without it no backup can be produced, and without the same key no backup can be restored — store it outside this deployment.

 Production additionally refuses: a well-known development database password, a wildcard CORS origin, remote model code execution, an unwritable storage root, an unsupported schema revision, a PostgreSQL major below 16, and three policy combinations that cannot all hold at once.
@@ -34,13 +33,17 @@ Losing the backup encryption key makes every existing backup unrecoverable. It i

 ## Sensitivity

-5 settings are credentials. They are never logged, never written to a release artefact and never echoed in an error response:
+9 settings are credentials. They are never logged, never written to a release artefact and never echoed in an error response:

 - `MODELFORGE_DATABASE_URL`
 - `MODELFORGE_MIGRATION_DATABASE_URL`
+- `MODELFORGE_OPERATOR_IDENTITIES`
+- `MODELFORGE_OPERATOR_PASSWORD_HASH`
 - `MODELFORGE_OPERATOR_API_KEY`
 - `MODELFORGE_HF_TOKEN`
+- `MODELFORGE_SECRET_SLOT_ROOT_KEY`
 - `MODELFORGE_BACKUP_ENCRYPTION_KEY`
+- `MODELFORGE_ALERT_WEBHOOK_URL`

 ## Every setting

@@ -56,8 +59,12 @@ Losing the backup encryption key makes every existing backup unrecoverable. It i
 | `MODELFORGE_MIGRATION_DATABASE_URL` | secret, optional | — | no | secret | SQLAlchemy URL for the non-superuser modelforge schema-owner role. Set only in the one-shot migration process; production API startup refuses when this secret is present. |
 | `MODELFORGE_REDIS_URL` | string | `redis://localhost:6379/0` | yes | internal | Redis URL for transient request payloads and queues. |
 | `MODELFORGE_CORS_ORIGINS` | string | `http://localhost:3000` | no | public | Comma-separated exact origins allowed to call the API from a browser. A wildcard is refused in production because requests are credentialed. |
-| `MODELFORGE_OPERATOR_API_KEY` | secret, optional | — | yes | secret | Operator API key guarding every admin route. Generate at least 32 random characters; ModelForge never mints one for you. |
-| `MODELFORGE_HF_TOKEN` | secret, optional | — | no | secret | Optional Hugging Face token, used only for acquiring gated repositories. It is never passed to a runtime and never leaves the control plane. |
+| `MODELFORGE_OPERATOR_IDENTITIES` | secret, optional | — | no | secret | Optional JSON array of named human identities with role, credential ID, SHA-256 key digest and timezone-aware expiry. Provision with scripts/operator_identity.py. Replace this deployment setting and recreate the API to rotate or revoke access. |
+| `MODELFORGE_OPERATOR_PASSWORD_HASH` | secret, optional | — | no | secret | Optional salted scrypt verifier generated by scripts/operator_password.py. Enables password login with 15-minute sessions; requires the separate strong operator API key. Changing either value invalidates password sessions. |
+| `MODELFORGE_OPERATOR_API_KEY` | secret, optional | — | no | secret | Legacy bootstrap administrator key. Generate at least 32 random characters. Production requires this key or an unexpired named administrator in operator_identities. |
+| `MODELFORGE_HF_TOKEN` | secret, optional | — | no | secret | Optional deployment Hugging Face token for control-plane metadata. Compose may also provision it independently to node-agent downloads; API secret activation does not update nodes. |
+| `MODELFORGE_SECRET_SLOT_ROOT_KEY` | secret, optional | — | no | secret | External base64 32-byte AES-256 key for the managed Hugging Face control-plane slot. Keep independent of backup and operator keys; retain it externally for historical restore. |
+| `MODELFORGE_SECRET_SLOT_ROOT_KEY_ID` | string | `hf-control-plane-1` | no | public | External key identifier bound to encrypted HF slot versions. Replacing a key does not rewrap history. |
 | `MODELFORGE_BACKUP_ENCRYPTION_KEY` | secret, optional | — | yes | secret | Base64 AES-256 key for backup encryption. Without it no backup can be produced, and without the same key no backup can be restored — store it outside this deployment. |
 | `MODELFORGE_BACKUP_ENCRYPTION_KEY_ID` | string | `modelforge-backup-key-1` | no | public | Identifier recorded in each backup manifest so a restore can name the key it needs. |
 | `MODELFORGE_HF_HOME` | string | `/data/hf-cache` | no | public | Hugging Face cache root inside the container. |
@@ -108,6 +115,17 @@ Losing the backup encryption key makes every existing backup unrecoverable. It i
 | `MODELFORGE_MIGRATION_RECONCILIATION_ENABLED` | boolean | `false` | no | public | Report interrupted migration cutovers at startup. They are never auto-resolved: external alias truth cannot be inferred after a crash. |
 | `MODELFORGE_OBSERVABILITY_MONITOR_ENABLED` | boolean | `false` | no | public | Run SLO and alert evaluation in this process. |
 | `MODELFORGE_OBSERVABILITY_POLL_INTERVAL_SECONDS` | integer | `60` | no | public | Interval between observability evaluation passes. |
+| `MODELFORGE_ALERT_NOTIFICATIONS_ENABLED` | boolean | `false` | no | public | Queue and dispatch the deployment-managed operational alert webhook. |
+| `MODELFORGE_ALERT_WEBHOOK_URL` | secret, optional | — | no | secret | Deployment-managed HTTPS notification destination. Its exact hostname must also appear in alert_webhook_allowed_hosts; the URL is never returned by the API. |
+| `MODELFORGE_ALERT_WEBHOOK_ALLOWED_HOSTS` | string | — | no | public | Comma-separated exact destination host allowlist for the alert webhook. Wildcards and redirects are not accepted. |
+| `MODELFORGE_ALERT_WEBHOOK_MIN_SEVERITY` | `INFO`, `WARNING`, `CRITICAL` | `WARNING` | no | public | Minimum alert severity sent to the single webhook route: INFO, WARNING or CRITICAL. |
+| `MODELFORGE_ALERT_NOTIFICATION_TIMEOUT_SECONDS` | number | `5.0` | no | public | Total timeout for one webhook delivery. |
+| `MODELFORGE_ALERT_NOTIFICATION_MAX_ATTEMPTS` | integer | `8` | no | public | Maximum webhook attempts before a delivery fails permanently. |
+| `MODELFORGE_ALERT_NOTIFICATION_BATCH_SIZE` | integer | `20` | no | public | Maximum outbox rows claimed by one notification dispatch pass. |
+| `MODELFORGE_ALERT_NOTIFICATION_LEASE_SECONDS` | integer | `60` | no | public | Lease duration before an interrupted delivery can be reclaimed. |
+| `MODELFORGE_ALERT_NOTIFICATION_RETRY_BASE_SECONDS` | integer | `15` | no | public | Base delay for bounded exponential webhook retries. |
+| `MODELFORGE_MODEL_UPDATE_MONITOR_ENABLED` | boolean | `true` | no | public | In production, check explicitly followed model references for changed revisions. Metadata only; never downloads, tests or promotes a model. |
+| `MODELFORGE_MODEL_UPDATE_CHECK_INTERVAL_SECONDS` | integer | `86400` | no | public | Minimum interval between automatic checks of each followed model reference; default one day. Each API process checks at most one due reference per minute. |
 | `MODELFORGE_RECOVERY_RECONCILIATION_ENABLED` | boolean | `false` | no | public | Reconcile interrupted backups and restores at startup. |
 | `MODELFORGE_BACKUP_PG_DUMP_PATH` | string | `pg_dump` | no | public | pg_dump executable. Must match the server major version. |
 | `MODELFORGE_BACKUP_PG_RESTORE_PATH` | string | `pg_restore` | no | public | pg_restore executable. |
@@ -172,4 +190,4 @@ Read by Compose, the Node Agent and the Runtime Worker rather than by the contro

 ## Restarts

-Every setting is read at process start. Changing any of them requires restarting the control plane; none is re-read from the environment while the process is running.
+Deployment environment settings are read at process start; changing an environment value requires a restart. The Console also supports a closed set of numeric serving overrides which take effect for new requests after preview and explicit apply. See the [configuration workflow](operations/CONFIGURATION_WORKFLOW.md) for scope, acknowledgement, rollback and secret-safe transfer. Other settings remain deployment-managed.
@@ -1,7 +1,17 @@
-# Installing ITWorx ModelForge v1.2.1
+# Installing ITWorx ModelForge — historical v1.2.3 procedure

-This is the whole installation. If you have to do something that is not written here, that is a
-defect — please report it rather than working around it.
+At the 2026-09-23 reconciliation checkpoint, the observed production installation ran v1.2.3 at
+schema `20260909_0032` and no v1.2.3 Gitea Release object existed. The current `1.3.0-rc.1` source targets
+`20260924_0039`; its qualification is established by an exact published release record, not this
+historical guide. The steps below preserve the earlier bounded fresh-install
+procedure; do not treat them as validation of the current RC archive, production upgrade, or
+rollback. Select exact source and image identities and complete the [RC acceptance gates](operations/RC_ACCEPTANCE.md)
+before using a candidate package. See the [current-state reconciliation](quality/CURRENT_STATE_RECONCILIATION_2026-09-24.md)
+and [upgrade boundary](UPGRADE.md#source-and-schema-boundary).
+
+
+The final RC installation procedure must be checked against its exact archive and isolated
+clean-install rehearsal; unresolved steps are release blockers.

 The steps below are release-gated against a fresh project with isolated volumes, newly generated
 test secrets and an empty PostgreSQL database.
@@ -74,14 +84,25 @@ MODELFORGE_OPERATOR_API_KEY=<generated, at least 32 characters>
 MODELFORGE_BACKUP_ENCRYPTION_KEY=<generated base64 AES-256 key>
 MODELFORGE_CORS_ORIGINS=https://modelforge.example.internal
 VITE_API_BASE_URL=https://modelforge.example.internal:8000
-MODELFORGE_VERSION=1.2.1
-MODELFORGE_API_IMAGE=modelforge-api:1.2.1
-MODELFORGE_WEB_IMAGE=modelforge-web:1.2.1
+MODELFORGE_VERSION=1.2.3
+MODELFORGE_API_IMAGE=modelforge-api:1.2.3
+MODELFORGE_WEB_IMAGE=modelforge-web:1.2.3
 ```

 Every setting is documented in [CONFIGURATION.md](CONFIGURATION.md), which is generated from the
 typed settings — it cannot drift from what the code actually reads.

+For browser password sign-in, run `python scripts/operator_password.py` and enter your chosen
+password at the hidden prompt. Put only the generated hash in `.env` as
+`MODELFORGE_OPERATOR_PASSWORD_HASH='<generated hash>'`. Keep the single quotes: they prevent
+Compose from interpreting the dollar signs in the hash. Keep the separate generated operator
+API key above for session signing and recovery. Never use your password as that API key.
+Recreate the API after changing either setting. Password sign-in returns a temporary session
+valid for at most 15 minutes; locking or reloading the console clears it. Rotating the password
+hash or API key invalidates existing password sessions. Login attempts require Redis and are
+limited across API processes to five per socket peer and thirty globally per minute.
+The console defaults to **Password**; **Advanced: API key recovery** retains direct key access.
+
 To move a port, set `MODELFORGE_API_PUBLISHED_PORT`, `MODELFORGE_WEB_PORT`,
 `MODELFORGE_POSTGRES_PORT` or `MODELFORGE_REDIS_PORT`. You never have to edit a Compose file.

@@ -119,7 +140,7 @@ path, but never derives a deployable image from a floating `latest` tag.
 To stamp build identity into the images so a running container can say where it came from:

 ```bash
-MODELFORGE_VERSION=1.2.1 \
+MODELFORGE_VERSION=1.2.3 \
 MODELFORGE_COMMIT=$(git rev-parse HEAD) \
 MODELFORGE_BUILT_AT=$(date -u +%FT%TZ) \
 docker compose -f docker-compose.yml -f docker-compose.production.yml up -d --build
@@ -19,6 +19,34 @@ different site work without inbound exposure.

 ## Enrolling

+For the v1.3.0 RC, an administrator can use **Console → Hardware → Enroll node**. The wizard offers
+5-minute, 15-minute, 1-hour, and 24-hour token expiry, and locks the node's API origin to the
+Console's current API origin. RC enrollment always sets `production_eligible=false`. The one-time
+token is held only in the open wizard's component memory; closing the wizard, checking readiness,
+revoking the unused token, or reaching its expiry clears the reveal. The generated `.env` contains
+only non-secret settings. Paste the start command into a Bash shell from the node-agent repository;
+it disables shell tracing, reads the token without echo, uses a temporary mode-0600 env file, and
+removes that file on exit, including error and signal paths.
+
+Use **Check node readiness** after starting the agent. Readiness is tied to the exact node ID returned
+by the enrollment record; it checks connection, inventory age, protocol compatibility, agent health,
+and telemetry age and includes diagnostic next steps. After enrollment is consumed, use the wizard's
+container recreation command to remove the one-time token from the running container environment
+while preserving its persisted identity and credential. A node is not ready merely because it
+connected.
+
+The API flow below remains available for automation and existing installations.
+
+The RC wizard defaults to the local four-image release bundle. Verify the release `SHA256SUMS`, load
+`modelforge-1.3.0-rc.1-images.tar` on the node with `docker image load`, and paste
+`images.modelforge-node-agent` from the matching release manifest. The wizard requires the exact
+`modelforge-node-agent:1.3.0-rc.1` tag, RootFS layer list, architecture, and all recorded OCI labels.
+It ignores the build-host image/config ID because Docker may rewrite it during `docker load`; before
+Compose starts it captures the local image ID and verifies the created container uses that same ID.
+Start and cleanup use `--no-build --pull never`, and the generated configuration sets the local tag
+explicitly. A registry image pinned to `@sha256:<digest>` remains available as an alternate mode.
+The source-build Compose workflow below is separate and intended for development.
+
 1. An operator mints a single-use token:

 ```bash
@@ -79,9 +107,13 @@ bounded diagnostic check that never enrolls is available as:
 ```bash
 docker run --rm --gpus all --network none \
  -e MODELFORGE_AGENT_ACCELERATOR_MODE=nvidia \
-  modelforge-node-agent:1.1.1 modelforge-node-agent --preflight
+  modelforge-node-agent:1.1.1 python3 -m modelforge_node_agent.main --preflight
 ```

+The chiselled runtime intentionally contains no shell or packaging-script directory. Invoke the
+reviewed Python module directly for this bounded diagnostic, just as the image's default command
+does.
+
 ## Identity

 A node's identity is persisted and survives restarts. If a node reappears as a *new* identity, its
@@ -18,9 +18,12 @@ capability key    rag.embedding@1
 If your integration needs a model name, a revision, a node hostname or an artifact path, something
 has gone wrong — those are ModelForge's business, not yours.

+For guided setup, use the [Projects workflow](operations/PROJECT_ONBOARDING.md). Official lightweight
+[Python and TypeScript clients](../clients/README.md) ship in the release archive.
+
 ## Getting a credential

-An operator creates a service client scoped to your project and the capabilities you may call:
+An administrator creates a service client scoped to your project and the capabilities you may call:

 ```bash
 curl -s -X POST -H "X-ModelForge-Admin-Token: $KEY" -H "Content-Type: application/json" \
@@ -29,7 +29,19 @@ manifest. The canonical repository and all of its Git history remain private.
   automatically on persistent private runners. The exporter therefore removes the automatic
   `pull_request` trigger from the public copy of `managed-validation.yml` and leaves only explicit
   owner dispatch. Keep public branch protection independent of private-runner status checks until
-   a public-safe CI runner is configured.
+   a public-safe CI runner is configured on a network segment that cannot reach private
+   infrastructure.
+
+## Current publication state
+
+The parentless public repository is live at `https://git.example.com/Jens/ModelForge-Public`. Its
+protected `main` branch started with one root commit, and annotated tag `v1.2.2` points to that public
+root. Subsequent source snapshots are published from the private canonical branch only after the
+exact public candidate passes server acceptance, manifest validation and secret scanning. The public
+workflow remains owner-dispatch-only: the available server runners share the
+private LAN, so container isolation alone is insufficient for executing arbitrary fork code. A
+future automatic PR runner must also have enforced private-network egress denial; it must never use
+the private validation runner, deploy runner or host Docker socket.

 ## License decision

@@ -9,6 +9,9 @@ evidence before promotion; no production capability is changed automatically.
 - The Runtime Worker now upgrades its digest-pinned Ubuntu base during the image build.
 - Build-only Linux kernel headers are removed after dependency installation instead of remaining in
  the running Runtime Worker image.
+- Explicit shipping-image validation rebuilds all four images without Docker layer cache, refreshes
+  the Node Agent's digest-pinned Python base, upgrades fixed Runtime Worker dependencies, and removes
+  pip/setuptools/wheel after the final dependency check.
 - Pillow, protobuf, SentencePiece, Sentence Transformers, Transformers and inherited Python
  security dependencies are pinned to reviewed fixed versions.
 - The public-candidate acceptance gate builds and scans all four images, binds each Trivy report to
@@ -44,3 +47,23 @@ The control plane and Node Agent change is `transparent`. The Runtime Worker inf
 update is `behavioral`: exact LAB probes, output-shape checks and unload/reclaim evidence are
 required before an existing runtime profile can be promoted. Embedding-space identities and project
 indexes are not reused or changed automatically.
+
+## Post-publication operational verification
+
+The exact Runtime Worker built from `b8755394117affad0178a79a627cf5574d3c8eef` passed a fresh
+GPU Node LAB probe on 2026-09-01, including offline load, finite normalized `[1, 1024]` output and exact
+VRAM return to baseline after unload. Production remained on v1.2.1 pending the separate backup,
+migration and explicit deployment gates. See
+[`quality/V1_2_2_LAB_VERIFICATION.md`](quality/V1_2_2_LAB_VERIFICATION.md).
+
+## Known limitations
+
+- Production upgrade from schema 0022 to 0024 requires downtime, a verified recovery point and
+  separate migration-owner/runtime-role provisioning; publication never performs it automatically.
+- Public pull-request CI remains disabled until an isolated runner can enforce denial of private
+  network egress for untrusted fork code.
+- ExampleRAG's ModelForge shadow candidate is not promotion-eligible after two critical retrieval
+  regressions; the existing production alias and vector space remain unchanged.
+- ExampleVision owner-photo and ASR natural-speech qualification still require owner-provided ground
+  truth. Catalog images and synthetic/library audio are not accepted as substitutes.
+- Point-in-time database recovery remains `NOT_SUPPORTED`.
@@ -0,0 +1,36 @@
+# ModelForge v1.2.3 — operator password sign-in
+
+The console now opens with a Password field. A successful password exchange creates a signed,
+memory-only session lasting at most 15 minutes. Passwords are verified against a salted scrypt
+hash; the existing generated operator API key remains separate for signing and recovery.
+
+Existing API-key clients remain compatible. Set `MODELFORGE_OPERATOR_PASSWORD_HASH` using
+`python scripts/operator_password.py`, single-quote the hash in the Compose environment file,
+and recreate the API. See [installation](INSTALLATION.md). An unset hash preserves existing
+API-key access. Password login fails closed when Redis is unavailable. Attempts are limited
+across API processes, and forwarded headers do not change the peer used by the limiter.
+
+These release notes describe the published hotfix built from v1.2.2. Database schema `20260830_0024`, agent protocol 1, runtime
+dependencies, model artifacts, capability bindings and promotion evidence are unchanged.
+Upgrade classification: transparent. No model or inference runtime promotion is included.
+
+Validation before release: targeted backend authorization, password and release-contract tests;
+frontend full test suite and production build; independent code review; isolated Redis 7 atomic
+parallel attempt, global limit and TTL checks. Deployment and exact-image acceptance must be
+recorded separately; source validation is not evidence of a successful production login.
+
+## Known limitations
+
+Password login requires configured Redis and a password hash; there is no self-service password
+reset. Reloading the browser ends its session. Peers behind one reverse proxy share its attempt
+budget. Runtime and GPU qualification from v1.2.2 remains applicable only to unchanged exact
+runtime images; this hotfix does not qualify or promote new inference-worker images.
+
+
+## RC integration boundary
+
+The development branch also incorporates this hotfix while retaining its newer schema
+`20260908_0028`, named/managed identities and RC features. That source is not the published
+schema-0024 hotfix image and is not qualified by its release evidence. Password sessions on the
+RC branch pass the same identity, role and expiry checks as other operator sessions. Consult
+[the upgrade source boundary](UPGRADE.md#source-and-schema-boundary) before deploying either build.
@@ -0,0 +1,28 @@
+# ModelForge v1.3.0-rc.1 — qualification candidate
+
+`v1.3.0-rc.1` identifies the release-candidate source and artifact set. This file belongs in its
+archive; the annotated tag, Gitea prerelease, exact image identities and retained qualification
+evidence must be verified from the published release record. A prerelease does not authorize a
+production deployment.
+
+The current source targets Alembic schema `20260924_0039` and Node Agent protocol 1. The final
+candidate must contain four first-class images: API, Web, Node Agent and Runtime Worker. P15A must
+bind their exact OCI labels, content-addressable RootFS layer sets, SBOM, provenance, source/archive
+hashes and exact-image security reports into one retained transport bundle. P12–P14 must load that
+bundle without rebuilding it.
+
+The user-visible ModelForge changes are the already merged project-first model management,
+onboarding, model library, immutable dimension pinning and guarded replacement/rollback flows. The
+candidate reports the `rc` release channel so its API health and system metadata cannot be confused
+with the stable production line.
+
+## Known limitations
+
+- The `v1.3.0-rc.1` tag and Gitea prerelease are permitted only after every mandatory readiness
+  row is verified against the exact artifact set; the version string does not prove that gate.
+- The release record must retain proof of the isolated RC stack, real ExampleRAG consumer path,
+  project replacement/rollback, PostgreSQL 0039 recovery, fault journeys and 60-minute soak.
+- A real GPU inference gate requires an available RTX 4080 SUPER window; unmanaged workloads are
+  never stopped or changed by this candidate process.
+- Observed production remains v1.2.3/schema-0032 unless a separate stable promotion procedure
+  explicitly authorizes a change.
@@ -57,8 +57,9 @@ nginx — never a development server.
 Content-Security-Policy with `default-src 'none'`, `script-src 'self'` and no `unsafe-inline` or
 `unsafe-eval`; inline style is permitted only as an attribute via `style-src-attr`, which covers the
 dynamic width bars the console uses while still blocking an injected `<style>` element. `connect-src`
-is derived at build time from the same API base URL compiled into the bundle, so the policy cannot
-drift from the origin the bundle calls.
+is the one exact origin derived at build time from the same API base URL compiled into the bundle;
+it deliberately does not include `'self'`. This prevents a dynamic or injected same-origin-derived
+endpoint from becoming an additional network destination when the API is cross-origin.

 Also: `X-Content-Type-Options`, `X-Frame-Options`, `Referrer-Policy`,
 `Cross-Origin-Opener-Policy`, `Cross-Origin-Resource-Policy` and `Permissions-Policy` — on the entry
@@ -64,5 +64,7 @@ immutable released image and verify its OCI revision after recreation.
 ## Scheduled backups

 M15 delivered scheduled backups; enabling them on GPU Node is an operator action, not something the
-release does for you. `scripts/modelforge_scheduled_backup.sh` is the entry point. Verify one
+release does for you. `scripts/modelforge_scheduled_backup.sh` is the entry point. Set its protected
+`MODELFORGE_BACKUP_ADMIN_API_KEY` environment variable to an existing named administrator key; see
+[Scheduled backups](operations/SCHEDULED_BACKUPS.md) for credential setup and rotation. Verify one
 execution after enabling, and check the recovery dashboard shows a recent verified backup.
@@ -1,9 +1,40 @@
-# Upgrading to the current ModelForge RC candidate
+# Upgrade boundaries — historical v1.2.3 procedure and current RC source

-> RC schema note: the repository's temporary `1.2.1` version has **not** been released with
-> schema `20260830_0024`. Version selection/stamping is a separate pending release tranche. The
-> following preflight is the exact contract for rehearsing an existing stable v1.2.1/schema-0022
-> database before this candidate can be released; it must not be read as a READY announcement.
+> Status reconciled on 2026-09-23: no v1.2.3 Gitea Release object is published. The observed
+> production API is v1.2.3/schema `20260909_0032`; the current source line targets schema
+> `20260923_0037` and has not been production-qualified. The detailed commands below
+> retain the earlier v1.2.3/v1.2.2 schema-0024 and intermediate-RC instructions as historical
+> guidance; they are not an authorization to deploy current master or to claim migration-0037
+> recovery evidence. Use the current-state reconciliation and RC roadmap for the active candidate.
+
+## Source and schema boundary
+
+The schema numbers and commands in this section describe an earlier source snapshot. The current
+`1.3.0-rc.1` source targets `20260923_0037`, while the observed production database remains at
+`20260909_0032`. Rehearse that exact upgrade and recovery path before publishing the RC.
+
+The previously documented v1.2.3 password hotfix was built from the v1.2.2 line and retains schema
+`20260830_0024`. The RC development branch containing this integration requires
+`20260908_0028`. A matching version string does not make these builds interchangeable:
+use the exact commit, image digests and reported schema contract.
+
+For this RC source, the upgrade tool accepts schema 0021 through 0028 and targets 0028.
+In addition to the 0024 role boundary below, it applies migrations 0025 (configuration),
+0026 (project onboarding), 0027 (managed operator identities) and 0028 (platform secret slots).
+Back up and rehearse those migrations in the isolated RC environment before promotion;
+retain the external secret-slot root key for restore. Do not deploy the published schema-0024
+hotfix API against an RC database advanced to 0028. Rollback across that boundary requires
+restoring the matching verified database backup and application images.
+
+The remaining release commands document the previously documented v1.2.3 hotfix. They are not an RC
+promotion instruction; use the separately maintained RC roadmap for its outstanding gates.
+
+
+v1.2.3 retains schema `20260830_0024` from v1.2.2 and adds operator password sign-in.
+An existing v1.2.2/schema-0024 installation needs no schema migration or runtime-worker update.
+Configure the password hash as described in [INSTALLATION.md](INSTALLATION.md), retain the
+bootstrap key, and deploy the verified API and web images. Earlier schema sources still require
+the provisioning and migration procedure below. Publication alone does not promote production.

 ## Required v1.2.1/schema-0022 role provisioning before migration 0024

@@ -19,7 +50,16 @@ application migration must not mint or retain an administrator credential.
   `modelforge` migration/schema owner, and NOINHERIT `modelforge_runtime` API role. Set
   `MODELFORGE_POSTGRES_ADMIN_USER` to a new retained name such as `modelforge_admin`; set the three
   password variables and both owner/runtime SQLAlchemy URLs documented in `CONFIGURATION.md`.
-3. Recreate only the PostgreSQL container so those values are in its environment. It may report
+3. Determine whether the existing `modelforge` role is PostgreSQL's protected bootstrap
+   superuser before changing anything:
+
+   ```sql
+   select oid, rolsuper, oid = 10 as protected_bootstrap
+   from pg_roles where rolname = 'modelforge';
+   ```
+
+   If `protected_bootstrap` is false, recreate only the PostgreSQL container so the new values are
+   in its environment. It may report
   unhealthy until the new admin role exists; that is expected during this bounded downtime. While
   still connecting with the existing v1.2.1 `modelforge` bootstrap superuser, run:

@@ -29,11 +69,34 @@ application migration must not mint or retain an administrator credential.
     < deploy/postgres/provision-existing-1.2.1.sql
   ```

-   The script first creates the retained administrator, then preserves the existing `modelforge`
+   Compose exposes the five `MODELFORGE_*` provisioning variables to this bounded `psql` process;
+   no value is printed. The script first refuses a protected OID-10 owner before any mutation,
+   otherwise creates the retained administrator and preserves the existing `modelforge`
   role OID/table ownership while demoting it to a non-superuser, creates the distinct non-owner
   runtime role, and purges stale memberships that would still permit `SET ROLE`. Passwords are
   read from process environment; none is in the SQL file.
-4. Run only the one-shot migration service. Its environment contains
+4. If `protected_bootstrap` is true, do **not** edit `pg_authid`. PostgreSQL intentionally prevents
+   demotion of OID 10. Use this recoverable cluster migration instead:
+
+   1. Keep the verified ModelForge backup and create a second PostgreSQL custom-format logical dump
+      with `--no-owner --no-privileges --serializable-deferrable`; prove `pg_restore --list` can
+      read it and record its SHA-256.
+   2. Stop all application writers. Preserve the old PostgreSQL volume unchanged as the rollback
+      source. Bind `postgres-data` to a new, empty, explicitly named volume in a deployment-local
+      Compose override.
+   3. Start PostgreSQL with `MODELFORGE_POSTGRES_ADMIN_USER` as the cluster bootstrap user. The
+      committed init script creates the non-superuser `modelforge` owner and NOINHERIT
+      `modelforge_runtime` role.
+   4. Restore the logical dump into the empty `modelforge` database while directly authenticated as
+      `modelforge`, using `pg_restore --no-owner --no-privileges --exit-on-error
+      --single-transaction`. Verify the source schema revision and authoritative row counts before
+      migration.
+   5. Continue with the one-shot migration below. Retain the old volume until the new deployment,
+      state fingerprint and recovery point have passed the operator's rollback-retention window.
+
+   This is the expected path for older installations originally initialized with
+   `POSTGRES_USER=modelforge`; it is not an exceptional permission bypass.
+5. Run only the one-shot migration service. Its environment contains
   `MODELFORGE_MIGRATION_DATABASE_URL` but not the runtime/admin URL:

   ```bash
@@ -44,7 +107,7 @@ application migration must not mint or retain an administrator credential.
   directly authenticated non-superuser owner, `modelforge_runtime` is NOINHERIT/non-admin, and
   neither application role has an explicit role membership.
   The refusal explicitly says to run this v1.2.1-to-schema-0024 provisioning step.
-5. Start the API only after Alembic reaches 0024. Production startup checks live catalog truth:
+6. Start the API only after Alembic reaches 0024. Production startup checks live catalog truth:
   runtime role/ownership/admin flags, table/schema/database privileges, canonical function
   owner/SECURITY DEFINER/fixed search path/grants and exact body digest, PUBLIC execution, four
   exact origin-mode protection-trigger shapes, role memberships, and every unexpected executable
@@ -53,7 +116,7 @@ application migration must not mint or retain an administrator credential.
 The live PostgreSQL privilege/function exercise is a managed-runner release gate. SQLite exercises
 only the defense-in-depth application Engine hooks and is never production proof.

-Supported direct schema sources for this unreleased candidate are `20260827_0021`,
+Supported direct schema sources for v1.2.3 are `20260827_0021`,
 `20260828_0022`, and `20260830_0023`; the target is `20260830_0024`. Older releases require an
 intermediate upgrade to v1.0.0.

@@ -122,16 +185,16 @@ claimed. Record the measured downtime for your deployment.
 python scripts/upgrade.py --database-url postgresql+psycopg://user:pass@host:5432/modelforge --apply
 ```

-For the current candidate this reports `schema migrated: <source> -> 20260830_0024`. Runtime
+For v1.2.3 this reports `schema migrated: <source> -> 20260830_0024`. Runtime
 compatibility begins only after the scope validation, audit-chain checkpoint and PostgreSQL role
 boundary reach `0024`.

 ## 5. Start the new version

 ```bash
-MODELFORGE_VERSION=1.2.1 \
-MODELFORGE_API_IMAGE=modelforge-api:1.2.1 \
-MODELFORGE_WEB_IMAGE=modelforge-web:1.2.1 \
+MODELFORGE_VERSION=1.2.3 \
+MODELFORGE_API_IMAGE=modelforge-api:1.2.3 \
+MODELFORGE_WEB_IMAGE=modelforge-web:1.2.3 \
 docker compose -f docker-compose.yml -f docker-compose.production.yml up -d --no-build api web
 ```

@@ -143,7 +206,7 @@ alternative for development or rehearsal only: pass `MODELFORGE_COMMIT`, `MODELF
 ## 6. Verify

 ```bash
-curl -s http://127.0.0.1:8000/api/v1/version   # expect 1.2.1 and your commit
+curl -s http://127.0.0.1:8000/api/v1/version   # expect 1.2.3 and your commit
 curl -s http://127.0.0.1:8000/api/v1/health/ready
 ```

@@ -12,6 +12,12 @@
 All inference failures use typed codes. `X-Correlation-ID` and the response request ID support
 end-to-end tracing without retaining request content.

+Operator benchmark history is available at `GET /api/v1/lab/benchmark-runs`. The response remains
+the existing JSON array; `limit` stays bounded to 1–100 (default 50). To continue older history,
+send the opaque `cursor` query value from the `X-Next-Cursor` response header. A missing or empty
+header means the page is complete. Pages are ordered by `created_at DESC, id DESC`, and the cursor
+is bound to the authenticated operator. The Console exposes this traversal as “Load more runs.”
+
 This document defines intended API surfaces, not frozen implementation details.

 ## Control-plane API
@@ -32,6 +38,8 @@ POST   /api/v1/admin/node-enrollments
 GET    /api/v1/admin/node-enrollments
 DELETE /api/v1/admin/node-enrollments/{id}
 PATCH  /api/v1/admin/hardware/nodes/{id}
+POST   /api/v1/admin/hardware/nodes/{id}/lifecycle/preview
+POST   /api/v1/admin/hardware/nodes/{id}/lifecycle
 DELETE /api/v1/admin/hardware/nodes/{id}/credential
 POST   /api/v1/admin/hardware/nodes/{id}/credential/rotate
 POST   /api/v1/admin/hardware/nodes/{id}/decommission/preview
@@ -95,6 +95,15 @@ newest *n* verified backups (`minimum_verified_backups`, default 2) are protecte
 age, milestone and legal-hold backups are never expired, and retention refuses to run at all while
 a restore is in flight.

+The admin-only `GET /api/v1/admin/recovery/retention/preview` evaluates the same policy and returns
+the exact backup IDs it would retain or expire, with each decision's reason and the observation
+time. It is read-only: it does not flush pending ORM changes, commit, write an audit event or touch
+backup files, and its response contains no encryption key or destination metadata. The run endpoint
+uses the same decision logic and continues to append `BACKUP_EXPIRED` audit events for each state
+transition. Applying retention evaluates the current rows and policy again, so an older preview is
+not an execution guarantee. `EXPIRED` is a journal state; this retention operation does not
+physically delete backup payloads.
+
 ## Capacity

 Backups check headroom before writing rather than discovering it by filling a disk. The estimate
@@ -45,6 +45,18 @@ Administrative token creation, revocation, node metadata changes and credential
 under `/api/v1/admin`. They require `X-ModelForge-Admin-Token`; the API fails closed when
 `MODELFORGE_OPERATOR_API_KEY` is unset.

+Temporary enable/disable uses an authenticated impact-preview and execute pair:
+`POST /api/v1/admin/hardware/nodes/{id}/lifecycle/preview` and
+`POST /api/v1/admin/hardware/nodes/{id}/lifecycle`. Preview is bound to the node generation and a
+digest of current dependencies. Disable is refused while the dependency scan finds deployments,
+jobs, probes, residencies, leases, requests, pending transitions, or other listed work. Execute
+rechecks under the same dependency-table lock set used by decommission and uses a generation CAS.
+The older metadata `PATCH` still handles display name, labels, role and eligibility, but rejects
+`enabled`; it cannot bypass preview and typed identity confirmation. A remote node can be enabled
+only with an active node credential. Re-enable clears the prior protocol, capability, agent-version
+and heartbeat evidence, sets liveness to offline, and requires a fresh authenticated compatible
+heartbeat before readiness can be inferred. Decommission remains a distinct terminal transition.
+
 `POST /api/v1/admin/hardware/nodes/{id}/credential/rotate` atomically revokes the previous
 credential and returns its replacement once. Install the replacement in the protected agent state
 file before restarting it. Rotation and revocation are separate audited operations.
@@ -0,0 +1,58 @@
+# PrivateMuse audio resource control
+
+ModelForge coordinates PrivateMuse audio providers through the existing
+`creative.resource@1` capability. It does not generate media and it does not infer that a model is
+usable merely because its workload type is supported.
+
+## Workload vocabulary
+
+| Workload | Product meaning |
+| --- | --- |
+| `speech_generation` | Narration and character dialogue synthesis |
+| `voice_clone` | Consent-bound synthesis using an authorized reference voice |
+| `music_generation` | Background music and score generation |
+| `sound_generation` | Text-directed ambience and isolated sound effects |
+| `foley_generation` | Audio derived from an accepted video take |
+| `transcription` | Speech-to-text verification and timing |
+| `audio_enhancement` | Cleanup or source separation that may use the shared GPU |
+| `lip_sync` | A derived video aligned to an accepted speech asset |
+
+The existing `image`, `video`, and `chat` values remain compatible. CPU-only utilities do not need
+a GPU allocation. They may still be represented by a resource binding if a deployment uses the
+shared accelerator; callers must request a positive measured or configured VRAM budget.
+
+## Admission invariant
+
+An allocation is admitted only when the operator has approved the exact tuple of project,
+resource revision, workload, and provider resource identity. Supporting a workload name never
+creates a resource record and never approves an uninstalled or unqualified model. PrivateMuse must
+send a distinct resource route for ordinary speech synthesis and voice cloning even when both are
+served by one model, because cloned-voice use has separate product governance.
+
+The first research candidates are Chatterbox Multilingual V3 for speech and authorized cloning,
+ACE-Step 1.5 for music, and Dasheng-AudioGen for ambience and effects. These names are planning
+inputs only. They must not appear in the live ModelForge catalog until their exact revisions,
+runtime manifests, local measurements, licenses, and provider receipts have passed qualification.
+
+## Lifecycle behavior
+
+Audio allocations use the same durable behavior as image and video work:
+
+- idempotency covers the complete request, including workload;
+- a timeout after submission is reconciled as uncertain and never fails over blindly;
+- a heartbeat extends only a granted lease;
+- drain blocks new work while admitted work finishes or reconciles;
+- measured peak VRAM is attached to the exact approved route;
+- releasing one route may promote the next queued job without interrupting ComfyUI or Ollama.
+
+An allocation granted from already observed schedulable free VRAM carries no
+automatic unload instruction. A resident model is not discarded just because
+a new PrivateMuse job starts. Queued work can be promoted only after capacity is
+actually observed; PrivateMuse may request a guarded idle ComfyUI cache release
+when its own queue is empty. Residency reuse and restoration of another
+application's default model require an explicit, measured warm lease protocol;
+an allocation grant alone is not proof that ComfyUI is holding the same model.
+
+Director can orchestrate several independent PrivateMuse jobs, but ModelForge observes each
+provider allocation separately. A failed music request therefore cannot revoke accepted dialogue,
+and an uncertain sound-effect submission keeps only its own capacity fenced.
@@ -17,3 +17,8 @@ M11 operational consumers are ExampleVision shadow/evaluation clients for `visio
 versus interactive QoS without accepting a priority from the request body. Planning-only OCR bindings
 remain visibly distinct from active consumers.

+
+RC-02 adds persisted project onboarding and explicit binding selection. The plural
+`/credentials/rotate` route supports up to 24 hours of predecessor overlap with an expected
+credential ID; `/credentials/revoke` uses that same claim and typed client-name confirmation to
+revoke every key. See ADR-0050 and the project onboarding runbook. Legacy rotation remains immediate.
@@ -5,6 +5,11 @@ contains a `Project`, `ProjectBinding`, optional `ProjectEvaluationBinding`, ser
 project-owned data plane. It never contains a concrete model path, runtime, node or artifact
 selector.

+`ProjectBindingManifest.channel` records the requested route. A `stable` channel on a capability
+whose estate stability is `planned` or `blocked` describes a desired future integration; startup
+sync reports it as unavailable and does not materialize it. Startup sync preserves existing
+operator-owned bindings if the declared stability later changes.
+
 ExampleRAG is the first operational integration. Its production retrieval remains `CURRENT`; M6 adds
 an operator-only `SHADOW` target. Both targets are server-owned identifiers, not public arbitrary
 collection parameters. Text is transient in the Gateway payload store and never copied into
@@ -0,0 +1,32 @@
+# ADR-0048: Transactional serving configuration
+
+Status: accepted for RC-01, 2026-09-07.
+
+The operator override registry lives in PostgreSQL, alongside its revision history and audit.
+Only explicitly enumerated, non-secret numeric settings consumed by `ServingService` may be
+overridden. A serving request resolves one immutable snapshot on its first policy access, after
+authentication. In-flight requests retain their snapshot. API middleware, process startup, agents and
+workers are not hot-reloaded by this contract; their settings remain deployment-managed.
+
+Precedence is product defaults, Pydantic deployment sources, then operator overrides. The catalog
+does not guess provenance that Pydantic discarded. Preview binds the complete override set,
+effective numeric baseline and active revision to a SHA-256 digest. Apply repeats validation,
+requires that exact preview, and advances the singleton using compare-and-swap. Revision, pointer
+and attributable audit commit together. Audit failure rolls everything back. Concurrent or stale
+previews fail with conflict. Rollback is a new previewed revision, never deletion of history.
+
+Acknowledgement means the control-plane serving settings consumer accepted the typed snapshot for
+new requests. It does not mean an existing request or private worker restarted. Export contains
+portable safe overrides, compatibility metadata and names of secret slots, never credentials or
+credential-bearing connection strings. Import follows the same preview/apply path. Bootstrap
+secrets and deployment settings cannot enter override persistence through import or ordinary edits.
+
+Configuration state and history are authoritative recovery state. A new migration is required;
+qualification must exercise PostgreSQL migration round-trips and concurrent writes. The remaining
+write-only secret lifecycle and interactive identity work are separate RC-01 acceptance criteria.
+
+Implementation note (2026-09-24): the fixed Hugging Face control-plane secret slot was delivered
+under PR #32 and passed exact-source managed full/security run 9327. Together with the safe
+configuration transfer and browser evidence above, this closes RC-CFG-03's functional contract.
+The implementation and its limits are documented in [P01 secret-slot evidence](../../quality/RC_P01_SECRET_SLOTS.md).
+Interactive identity and isolated-RC live provider/root-key qualification remain separate gates.
@@ -0,0 +1,30 @@
+# ADR-0049: Named operator keys and bounded Console access
+
+Status: accepted for RC-01, 2026-09-08. Online managed identities extend this deployment-owned
+provider under ADR-0052; its distinct ownership and recovery contract supersede only the online
+identity-administration limitation below. The deployment registry contract remains unchanged.
+
+Support a small deployment-owned registry of named human credentials as the first attributable
+viewer/operator/admin authentication provider. Each immutable deployment record identifies a
+credential, subject, role, SHA-256 digest and required UTC-aware expiry. Generate keys with 256 bits
+of randomness and the distinct `mfop_` namespace. Validate the complete bounded registry at startup;
+reject duplicate JSON fields, credential IDs, digests, conflicting roles and reserved subjects.
+Rotation may temporarily retain two records for the same subject and role. Removing a record takes
+effect when the API is recreated with the replacement configuration. This is a deployment operation,
+not an online secret-management API. There is no new database state or migration.
+
+Keep the legacy administrator header and bootstrap key compatible. Named credentials use that header
+but never fall back to bootstrap privileges, including after expiry/revocation. Machine keys and
+single-use enrollment tokens cannot authenticate as humans. The existing role dependencies remain
+the authorization authority, and audit writers receive the authenticated subject and correlation ID.
+An installation may disable bootstrap access once it has an unexpired named administrator.
+
+The Console verifies identity before loading protected data, exposes subject, role and key expiry,
+and clears its memory-only key on manual lock, reload, credential refusal or after at most fifteen
+minutes. This local lock deadline is distinct from server-enforced key expiry. Protected responses
+carry `Cache-Control: no-store`; requests do not use cookies. Stale responses cannot restore data or
+invalidate a newly unlocked session. A dependency outage and a forbidden operation are not logout.
+
+OIDC/Authentik, server-issued browser sessions, online identity administration and an audited
+write-only deployment-secret lifecycle remain explicit RC limitations. This provider establishes
+real least privilege and attribution without claiming enterprise IAM or SSO completion.
@@ -0,0 +1,33 @@
+# ADR-0050: Persisted project onboarding and bounded service-key overlap
+
+Status: accepted for RC-02, 2026-09-08.
+
+Persist operator-created projects in the existing registry. A separate immutable onboarding record
+owns owner, environment and labels; existing manifest-backed projects retain their identities and
+bindings. Startup reconciliation must never overwrite operator bindings. New requirements target
+the stable capability gateway, not a model or candidate. Require explicit reindex/schema migration
+support when the contract requires it. A production environment label never grants promotion.
+
+Use the existing hash-only service credential domain for online project-key administration. This
+does not depend on a generic secret vault or change the deployment-owned human identity provider.
+Credential administration remains administrator-only. Project creation and requirements require
+operator; viewers can inspect non-secret metadata and readiness.
+
+Add `superseded_at` and enforce one current non-revoked credential per client with a partial unique
+index. Rotation conditionally claims the exact current credential and creates its replacement in
+the same transaction as audit. A bounded overlap (zero to 24 hours) may retain one predecessor;
+it never extends the predecessor's existing expiry. A subsequent rotation retires older overlaps.
+Typed revocation claims the same current credential and revokes every key of that service identity.
+Concurrent rotation/revocation has one winner. Historical secrets cannot be restored by rotation.
+
+Readiness is derived, not a writable boolean: current credential authentication plus a successful
+gateway request from that exact credential, the current stable deployment and the last 24 hours.
+Persist the credential ID in gateway decision evidence; no plaintext key or input is added. A test
+performed with the previous overlapping key cannot qualify the new key. Missing deployment or
+fresh evidence remains a visible blocker. Engineering readiness never substitutes for lifecycle
+production approval or real-consumer certification.
+
+Migration 0026 preserves existing projects and credential hashes. A mechanical downgrade is
+allowed only before onboarding or rotation history exists. Thereafter recover the verified
+pre-migration backup. Recovery permits exactly an empty onboarding table and null supersession
+metadata during an older-schema upgrade; current-schema restore preserves both exactly.
@@ -0,0 +1,30 @@
+# ADR-0051: Manual canary observations are rehearsal-only
+
+Status: accepted for P00 containment, 2026-09-08.
+
+The operational Console previously offered fixed successful and failed canary observations.
+The lifecycle service accepted those counters and health fields as canary results. Reaching the
+target count could satisfy the canary review gate without a measured request. Other promotion
+and approval checks remained; this finding does not imply unrestricted production promotion.
+
+Remove both synthetic controls from the Console. Keep manual observation compatibility only for
+an explicitly persisted LAB rehearsal: a LAB_REHEARSAL subject in LAB, a consistent LAB approval
+with LAB_PROMOTION policy, and a plan targeting LAB_STABLE. Client labels and observation fields
+cannot grant that scope. Refuse other manual observations before changing counters, status or
+audit state. Attribute accepted manual observations to the authenticated operator and request
+correlation, and label their result as manual rehearsal evidence.
+
+Also refuse non-rehearsal canary commits, including existing READY records with no authoritative
+evidence. Do not invent a trusted source flag that a client can submit to bypass the restriction.
+This deliberately leaves real canary qualification unavailable until P05 adds an authoritative
+evidence ingestion contract. Existing non-canary transitions retain their independent checks.
+
+P05 must consume deduplicated real gateway requests for the exact candidate/deployment and time
+window, combine observed runtime/scheduler health without converting unknown to healthy, and use
+transactional concurrency control. Production qualification requires that follow-on implementation
+and isolated real-workload evidence. Manual LAB rehearsal proves only the lifecycle mechanics.
+
+No schema change is needed for containment; existing result metadata can label the evidence.
+Rollback to the old implementation would restore the unverified-observation path and is not a
+safe remedy for blocked production canaries. Existing explicit abort/rollback operations remain
+available for reconciliation.
@@ -0,0 +1,63 @@
+# ADR-0052: Managed personal keys and recovery admission
+
+Status: accepted design for P01; implementation qualification in progress.
+
+## Decision
+
+The bounded personal-header-key provider is the supported RC interactive provider. Preserve
+ADR-0049 deployment keys and the bootstrap administrator unchanged. Add administrator-managed
+identities under `managed:<slug>` with a distinct `mfopm_` plus 43 URL-safe-character credential
+namespace. Existing deployment subjects cannot contain `:`; authentication of deployment keys
+does not require a managed-identity database lookup. A managed key never falls back to deployment
+or machine privileges. The existing header, role boundaries and memory-only Console remain.
+This decision does not claim OIDC, SSO logout, cookie sessions or enterprise IAM.
+
+Store only the SHA-256 digest of a random 256-bit key, never plaintext or recoverable ciphertext.
+No encryption root key or additional dependency is needed. Issue plaintext exactly once after
+commit. A lost response requires a new rotation. Bound expiry to 1–365 days, rotation overlap to
+0–900 seconds, identities to 100, and displayed credential history to the latest ten records.
+History remains authoritative; retired keys cannot be restored by another rotation. Metadata
+exposes role, source, expiry, version, last use, activation and usability, never digests. Mutation
+reasons cannot contain ModelForge credential namespaces and validation errors omit rejected inputs.
+
+Keep deployment identities read-only. Production startup still requires a secure deployment
+administrator/recovery anchor; this PR does not introduce managed-only bootstrap or automatic
+environment migration. First managed identity must be an administrator. A singleton transaction
+lock serializes every managed mutation; expected registry/identity versions reject stale requests.
+Under that lock recheck the initiating managed administrator's credential, role, activation and
+expiry. Once initialized, every successful transaction leaves an enabled administrator with an
+activated, unrevoked, unexpired key. Expiry can still cause future loss of managed administration;
+the deployment recovery anchor remains necessary. Audit and identity mutation commit atomically.
+
+## Recovery
+
+`managed_operator_state`, `managed_operator_identities` and `managed_operator_credentials` are
+AUTHORITATIVE. Migration 0027 introduces the exact initial singleton and empty identity/credential
+tables. Older-schema restore permits only that exact empty introduction; same-schema restoration
+preserves all authoritative metadata. Credential digests stay excluded from public fingerprints
+under the established secret-column policy. Mechanical downgrade refuses existing identity history;
+recover the verified pre-migration backup instead.
+
+`managed_operator_credential_activations` is CURRENT_TRUTH admission state. Normal issue/rotation
+adds an activation only for the new key. Authentication requires it in addition to authoritative
+credential validity. Supported restore clears activations alongside other stale current truth;
+the independent READY validation rejects any retained activation. This quarantines every restored
+managed key, including keys revoked after the backup, without rewriting historical credentials
+or permitting fingerprint drift. Interrupted restore targets remain isolated and must not serve.
+
+After supported recovery, an external deployment administrator rotates an enabled managed
+administrator or creates a new administrator to restore admission. The new key alone gains
+activation; old keys remain quarantined even during requested overlap. Reconciliation never
+rehydrates activations automatically. A raw `pg_restore` bypassing ModelForge reconciliation and
+READY validation is not a supported recovery procedure and must not be attached to the serving API.
+
+## Evidence boundary
+
+Pre-body managed authentication uses the existing bounded thread-pool/session pattern and returns
+a typed 503 on database failure. No cached successful credential can override revocation.
+Already-authorized ordinary requests may finish; managed administration additionally rechecks its
+initiator while holding the mutation lock. Tests must cover real PostgreSQL cross-admin races,
+rotation/revoke CAS, last-admin protection, migration downgrade/recovery, restored admission
+quarantine, source namespace separation, audit rollback, and Console one-time secret handling.
+Local SQLite tests are not PostgreSQL qualification. Exact-head managed and browser gates remain
+required before this ADR's implementation is marked verified.
@@ -0,0 +1,51 @@
+# ADR-0053: Bounded Hugging Face control-plane secret lifecycle
+
+Status: accepted design; implementation and qualification in progress.
+
+Only `huggingface_control_plane` is API-managed. It supplies the control-plane Hugging Face
+discovery and repository metadata adapter. Node-agent downloads retain their independently
+deployed credential; no operation here acknowledges node credential activation. Database,
+migration, operator/recovery, backup encryption and Redis credentials remain deployment-managed.
+
+Deployment mode remains until the first explicit set. Thereafter no missing, revoked, restored,
+undecryptable or inactive managed state falls back to deployment or ambient HF credentials.
+Set persists a new encrypted candidate and switches to managed mode; an existing active version
+continues serving during rotation. Test calls only the fixed HTTPS Hugging Face whoami-v2 endpoint
+with bounded time/body, no redirects or environment proxies, and redacted results. It proves token
+authentication, not repository entitlement or node downloads. Activation requires a successful
+test within five minutes for the exact candidate version and current revision. Each mutation
+uses CAS and atomic attributable audit. Network testing occurs outside the mutation transaction;
+the result cannot bless a concurrently replaced/revoked candidate. Revocation disables all managed
+versions; recovery requires setting a new candidate, testing and activating it.
+
+Use AES-256-GCM from the existing cryptography dependency, fresh 96-bit nonces and associated data
+binding slot, version and key ID. `MODELFORGE_SECRET_SLOT_ROOT_KEY` is strictly a base64 32-byte key,
+supplied externally, independent of backup encryption and operator credentials. Its explicit ID
+is stored on every version. This bounded first implementation supports one mounted/deployment
+key at a time; replacing that key does not silently rewrap history. Retain old external keys for
+historical restore, or set/test/activate a new version before retiring the prior deployment key.
+The root key cannot be managed through the slot API. No plaintext reads, exports or audit fields.
+
+State and encrypted versions are authoritative. Test results and active admission are current
+truth, cleared by supported restore and independently checked before READY. Restored candidates
+require a fresh test and explicit activation. Old-schema restore permits only the exact initial
+empty state introduced by migration 0028. A history-bearing downgrade requires verified backup
+recovery. Retired ciphertext remains historical; provider revocation cannot be undone locally.
+
+New provider instances resolve one snapshot when upstream access starts. Persisted history and
+node job progress, completion and cleanup do not require upstream credential resolution; a revoked
+slot must not strand an existing job's bookkeeping. Activation acknowledges this
+local resolver, not remote downloads or running requests. Provider failure never becomes a
+successful test. Missing root-key disables the managed consumer without blocking deployment
+administrator access needed for recovery. SQLite qualification is not PostgreSQL race/restore
+qualification; live HF account checks and isolated RC provisioning remain external gates.
+
+Persistence contract: migration 0028 introduces two authoritative tables and two current-truth
+admission tables. The runtime role can insert encrypted versions but can update only their
+`revoked_at` column; it cannot rewrite version numbers, key IDs, nonces, ciphertext or creation
+timestamps, nor delete/truncate historical rows. The recovery owner retains the isolated
+maintenance authority required by supported restore. Metadata fingerprints explicitly exclude
+ciphertext and nonces; encrypted archive authentication and AES-GCM validate secret contents.
+`SECRET_STRICT` retains all secret metadata while delegating the existing verified reconciliation
+marker exception to `MANAGED_STRICT`. Earlier schemas accept only the exact uninitialized seed
+and empty version history introduced by 0028, never arbitrary newly introduced secret state.
@@ -0,0 +1,42 @@
+# ADR 0054: Model Lab preflight and execution are separate contracts
+
+Status: accepted for P03A; persistent execution remains outstanding.
+
+## Context
+
+The existing serving gateway has private worker jobs, GPU leases and transient payloads.
+Its invoke methods synchronously await completion, use service-client authorization and,
+for several modalities, resolve an experiment rather than an explicitly chosen candidate.
+Wrapping these methods in an operator endpoint would conflate credential domains and would
+not provide a durable, restart-safe Model Lab run.
+
+## Decision
+
+First expose an operator-authorized, read-only `POST /api/v1/lab/preflight`. Requests bind to
+exact candidate, artifact, runtime profile, node and LAB approval IDs. Inputs discriminate
+the five supported capability families and reuse existing typed serving validators. Execution,
+queue, retention and output budgets are bounded; concurrency is one.
+
+Recompute current assessment and approval evidence through the runtime service. Check candidate,
+completed probe, profile, node and contract coherence. Responses contain only typed binding,
+input digest/count/size, bounded limits and blocker codes. No raw content, arbitrary review text,
+service credential, payload-store reference or inferred GPU measurement is returned or persisted.
+Validation errors for this namespace also omit caller-controlled input, locations and messages.
+
+Preflight does not create an experiment, approval, gateway request, job, payload or lease. Both
+`execution_started` and `permission_granted` are always false. Passing is a current read-only
+observation; a future start operation must repeat admission and evidence checks atomically.
+The existing control-plane HTTP body cap applies, so initial inline inputs must fit that cap.
+No service-key privilege or machine credential is converted into an operator principal.
+
+## Subsequent implementation
+
+Persist LabRun identity, actor, immutable provenance and idempotency separately from transient
+input/output content. Extract explicit-target dispatch into the existing scheduler/private-worker
+pipeline rather than create another scheduler or fabricate a ServiceClient. Add leases, bounded
+retry, deadline/retention cleanup and restart reconciliation with PostgreSQL migration/recovery proof.
+
+Running cancellation must distinguish requested cancellation from worker acknowledgement.
+An API or database status alone cannot establish that GPU execution or reclamation has completed.
+Then expose run history/results and connect the Console. Real hardware, consumer journeys and
+the frozen RC artifactset remain separate qualification gates.
@@ -0,0 +1,42 @@
+# ADR 0055: Selected-model update inbox
+
+Status: accepted for selected-model tracking; replacement execution remains separate.
+
+The isolated integration introduces schema `20260909_0029` directly after `20260908_0028`.
+Watch and notice history are authoritative recovery state. An older backup may gain these tables
+only in their exact empty initial state; a current backup must retain their complete semantic
+fingerprints. Temporary check tokens and deadlines are excluded and cleared during restore,
+with an independent readiness check refusing remaining restored claims.
+
+ModelForge is a central hub for selected models and project capabilities. Upstream monitoring
+is opt-in, bounded to 100 explicit model/ref watches. A watch records an existing exact revision
+as its comparison baseline. It does not imply that this revision is deployed or that
+a subsequently observed commit is newer or better. Alternative repositories require their own
+explicit selection; this is not a global model-ranking system.
+
+Manual checks reuse the authenticated Hugging Face metadata and immutable revision boundary.
+They create no download plan, worker execution, approval or deployment. Changed commits produce
+one durable notice per watch/commit. Dismissal preserves history and repeated checks do not revive
+dismissed notices. Lists are bounded and paginated. Provider failure records an error while
+preserving the baseline and existing notices.
+
+An atomic database claim bounds concurrent checks. A five-minute lease allows restart recovery;
+token fencing prevents expired requests from overwriting a successor's state. The audit chain
+serializes creation admission so the 100-watch bound also applies to concurrent creators.
+Reads require viewer and mutations require operator. Nested provider audit events use the same
+authenticated actor and correlation as the watch operation.
+
+Production processes check enabled watches daily by default, at most one due watch per minute
+per API process. Atomic due checks avoid a duplicate scheduled call after another process finishes
+the same watch. This is a per-process throughput limit, not a global cross-process scheduler.
+Development/test lifespans do not start this monitor. Errors are recorded or logged without
+credential content; a monitor configuration flag is not proof of a successful provider check.
+
+Tracking changes use an explicit audited compare-and-swap on the observed version and baseline.
+Baseline changes validate the same model and refuse an active check. Pausing fences in-flight
+check publication; already-started metadata retrieval may finish, but cannot produce a notice
+or overwrite successor state. Historical notice provenance is unchanged. Changing tracking never
+changes a deployment. Broad model search, automatic tests, automatic replacement and model-quality
+claims are not introduced. Project-bound comparison and evidence-backed replacement
+continue through existing capability, evaluation and lifecycle boundaries, including embedding
+migration and rollback. This tranche does not complete that replacement workflow.
@@ -0,0 +1,84 @@
+# ADR 0056: Durable selected-model comparison intents
+
+Status: accepted for the persisted operator/consumer protocol. Consumer execution integration and
+release qualification remain open.
+
+Migration note: the original canonical 0030 implementation below is retained as
+design history. [ADR 0057](0057-deployed-lineage-convergence.md) integrates these
+tables through additive 0033 while preserving the separately deployed lineage.
+
+## Context
+
+[ADR 0055](0055-selected-model-update-inbox.md) separates an upstream notice from model quality and
+replacement. [ADR 0054](0054-lab-preflight-before-persistent-execution.md) separates admission from durable execution.
+The selected-project hub needs one current-versus-candidate comparison with attributable evidence,
+without introducing a model tournament, another retrieval engine or another promotion engine.
+
+## Decision
+
+Schema `20260909_0030`, after canonical `20260909_0029`, adds `selected_model_comparisons`.
+An operator creates an intent for one stable project binding, one designated service client,
+one immutable project suite and two distinct exact capability deployments. The current leg must
+be a production `stable` ModelForge deployment; the candidate must be a nonproduction `candidate`
+with an active experiment route for the same `rag.embedding@1` contract. Distinct index references
+prevent silent reuse of the current vector collection. Different embedding spaces are allowed;
+this does not authorize reindex, cutover or replacement.
+
+Creation records immutable revision, artifact-set, runtime-profile, embedding-space, configuration,
+suite, corpus and index identities. Every reviewed case must name the requested corpus digest.
+The retrieval configuration digest must match the immutable suite settings. Claim and result
+publication revalidate the selected identities and route. A suite contains 1–100 reviewed cases.
+One active intent per project binding and a unique binding/idempotency key bound admission.
+Identical retries return the existing intent; changing its payload conflicts.
+
+The operator designates an existing active project-bound `rag.embedding@1` client. Consumer
+endpoints authenticate its service credential and require that exact client and binding on every
+operation. A claim token alone grants no authority. Operator credentials are not converted into
+service credentials. Assigned discovery is bounded and scoped to the designated client. Suite
+case definitions accompany its successful claim; this protocol adds no general consumer suite API.
+
+There is one claim, with a 120-second renewable lease and a fixed one-hour deadline from claim.
+No automatic reclaim or replay occurs. Progress counts are bounded and monotonic. Expired claims
+become `interrupted`; history, assignment reads and subsequent creation reconcile expiration.
+Admission retains its database locks until the decision commits. An explicit new intent is required
+for another attempt, and does not prove an earlier external process stopped.
+
+Pending cancellation becomes `cancelled`. Running cancellation becomes `cancel_requested` until
+the designated consumer acknowledges it. Results cannot publish after cancellation, expiry or
+loss of the claim. Expiry and interruption describe coordination state, not GPU reclamation.
+
+The consumer submits both legs together with ranked per-case results, its implementation digest
+and mappings from suite case IDs to public gateway `request_id` values. Successful cases require
+unique completed gateway requests for the designated client, exact leg deployment, capability and
+unchanged single-case query hash. Server-owned request timestamps must lie within this claim and
+deadline. Failed cases may omit a request, but cannot make the comparison eligible. Request evidence
+proves the matching embedding invocation; retrieval rankings remain explicitly attributed to the
+designated consumer rather than represented as independently observed hub retrieval.
+
+`EvaluationService(commit=False)` creates and completes the existing two evaluation runs and
+comparison inside the locked intent transaction. The server recomputes metrics and eligibility;
+caller aggregate scores are not accepted. Any publication failure rolls back the pair and links.
+An identical result retry returns the existing result, while a changed retry conflicts. The
+response derives a compact outcome from the linked comparison and fails closed on missing evidence.
+Eligibility is evaluation evidence, never production authorization or a promotion operation.
+
+## Recovery and boundaries
+
+Following [ADR 0041](0041-authoritative-versus-rebuildable-recovery-state.md), pins, operation history
+and linked results are authoritative. Claim hashes and expiry are transient. Restore clears these
+claims and transforms `pending`, `running` and `cancel_requested` intents to `interrupted` with
+`COMPARISON_RESTORE_INTERRUPTED`; it does not replay work. A deterministic fingerprint projection
+permits precisely that transformation while retaining the other provenance. An older schema can
+gain the new table only in its exact empty initial state. The readiness gate refuses residual
+restored claims or active intents. Populated comparison history blocks downgrade.
+
+The API does not launch a background consumer, run the ExampleRAG CLI, invoke a provider, claim GPU
+capacity or promote a model. The current ExampleRAG CLI dynamically selects its legacy CURRENT profile
+and does not yet implement this claim/result protocol. Its historical evaluation cannot satisfy a
+fresh intent. A consumer adapter must first validate the actual stable/candidate collection and
+profile identities, execute through the existing retrieval engine, and retain matching public
+gateway request IDs. Real inference, managed PostgreSQL concurrency, upgrade/restore and the frozen
+release artifact set remain qualification gates; this ADR does not assert those gates passed.
+
+The actionable protocol is documented in
+[Selected model comparisons](../../operations/SELECTED_MODEL_COMPARISONS.md).
@@ -0,0 +1,64 @@
+# ADR 0057: Preserve the deployed migration lineage when integrating the model hub
+
+Status: accepted for implementation; deployment qualification pending.
+
+## Context
+
+The canonical hub and a separately deployed creative branch diverged after schema
+`20260908_0028`. Read-only inspection on 2026-09-09 found image
+`modelforge-api:dfec816` and production schema `20260909_0032`. In that deployed
+lineage, 0029 creates creative allocations, 0030 creates model watches, 0031 adds
+creative governance, and 0032 widens audio workload names. Canonical hub 0029 and
+0030 instead represented newer watches and selected comparisons. The shared
+revision strings do not establish compatible database state.
+
+## Decision
+
+Preserve the deployed 0029–0032 migration artifacts byte-for-byte. Add 0033 after
+0032 to introduce watch `enabled=true` and `version=1`, runtime history protections,
+and selected-comparison storage. Do not stamp, rename applied migrations, or
+downgrade populated production data to resolve the conflict. New installations
+and upgrades from the common 0028-or-earlier lineage follow this same linear graph.
+
+The earlier canonical comparison schema was development qualification, not an
+alternative supported production upgrade source. An isolated database built with
+that conflicting graph must retain its original migration bundle; it cannot be
+upgraded using a revision number alone. Convergence does not authorize discarding
+its data. Any required migration of that separate lineage needs its own explicit,
+verified transfer procedure.
+
+Retain canonical project-first navigation, model-watch pause/CAS and bounded
+comparison execution alongside deployed creative history. ModelForge remains the
+central capability hub. This integration adds no model tournament or external
+creative tool execution.
+
+Native inference and external grants share accelerator admission locks and account
+for all granted or uncertain reservations. Capacity reads inside admission do not
+commit the owning transaction. Unattributed GPU telemetry is not proof that a
+particular external grant materialized; retain its full reservation until explicit
+release. Queued work must revalidate current client, project, priority and approved
+resource before receiving a grant. Expiry cannot be reversed by a heartbeat.
+
+The governance tables belong to the typed persistence model, retaining the exact
+deployed constraints. Downgrading 0033 refuses populated hub or creative history,
+preventing traversal into older destructive downgrade paths.
+
+## Recovery and acceptance
+
+Recovery preserves creative provenance and explicitly marks restored queued/granted
+allocations uncertain. That state does not claim an external process stopped.
+Selected comparisons remain interrupted with their claims cleared. Deterministic
+projections computed from the source rows must prove these exact changes; missing,
+changed or unclassified history blocks readiness.
+
+A deployed watch schema without the new control columns requires a fresh source
+fingerprint with the explicit default-column projection. An old opaque fingerprint
+cannot be rewritten to remove execution-claim fields or invent the new controls.
+Unprojectable archived history remains a recovery blocker requiring its original
+qualified artifact; no hash exception is introduced.
+
+Qualify populated 0032→0033 upgrade and actual dump/restore, audit integrity,
+runtime privileges, competing admission transactions, revoked queued authority,
+expiry and UI stale-response handling. All four immutable images, the actual
+deployment upgrade/rollback plan and the release soak remain separate gates.
+Merging this source does not deploy it or establish RC readiness.
@@ -0,0 +1,81 @@
+# ADR 0058: Execute stable route replacement through the approved lifecycle engine
+
+Status: implemented for identical embedding spaces; isolated PostgreSQL qualification passed; production deployment pending.
+
+The project comparison establishes evidence. It grants no production authority. Existing
+lifecycle subjects, plans and rollback snapshots previously recorded state without switching
+the gateway. The new transaction-only serving adapter connects those existing operations to
+the actual capability deployments; there is no second promotion engine.
+
+An approval's typed `evidence.serving_transition` selects current and candidate deployments,
+the exact production execution approval, expected capability `serving_generation`, and one
+selected-comparison ID for every affected stable project binding. The server derives the
+complete binding set and identity snapshots. Missing projects, reused comparisons, changed
+bindings, stale comparison pins, incomplete/errorful runs and revoked production evidence
+block execution. Creating a new comparison supersedes the older binding evidence for this
+purpose. The existing production validation engine rechecks the probe and supply chain.
+Unlike a capability-only readiness record, an operational transition requires the approval's
+declared `production_validation=SATISFIED`; explicitly deferred, required or not-required
+validation cannot activate a project-serving route. This requirement is also carried in the
+immutable serving contract and rechecked at commit.
+
+For each affected binding, the latest ProjectFitEvidence for that exact candidate is included
+in the approved contract when present. Negative/deferred conclusions, incomplete integration,
+blockers or critical errors stop the operation. A later record changes the contract and needs
+fresh review. Evidence for another deployment is not silently applied to this candidate.
+Absence of a separate project-fit record does not fabricate one: reviewed per-binding selected
+comparisons and explicit satisfied production validation remain required. This adapter does
+not impose ExampleVision's OWNER_PHOTO evidence class on RAG projects, or create separate
+PROJECT_PRODUCTION approvals. Generic capability-only/LAB policy interpretation remains intact.
+
+The reviewed contract is stored in the approval's resolved evidence and copied into the
+immutable promotion plan. Production scope, exact candidate/rollback targets and an explicit
+plan approval are required. This first operational adapter supports only `transparent` or
+`behavioral` plans targeting STABLE from a LAB_READY/PROMOTION_ELIGIBLE deployment subject.
+Manual canary observations do not acquire operational authority.
+
+Execution captures actual deployment/routing identities in the existing immutable rollback
+snapshot. The separate explicit lifecycle commit locks the capability contract, revalidates
+all evidence, compares/increments its durable route generation, drains the old route and
+activates the next route in one transaction with lifecycle state and audit. The capability
+endpoint and project credentials do not change. Existing admitted calls retain their old
+deployment; changing the route neither kills workers nor claims memory was reclaimed.
+
+Rollback checks the exact installed route generation, both deployment snapshots and the
+affected binding set. It restores the actual route and subject in one transaction and
+increments generation again. A later route/configuration change produces a conflict; the
+old operation never overwrites it. The previous artifact set, runtime profile and deployment
+receive explicit retention records. Nonzero route generations block schema downgrade.
+Rollback rechecks the retained target's actual production approval, supply-chain/probe facts,
+node availability and runtime readiness. A later revocation or quarantine is an explicit
+blocker, not permission to reactivate unsafe assets. Same-operation commit retries verify the
+exact installed state and return the persisted result without another generation increment.
+Activation timestamps come from the actual commit journal, not the earlier preparation time.
+
+Embedding identity equality means the exact existing embedding-space ID, not matching vector
+dimensions or apparently identical model files. A changed identity still requires a trusted
+consumer reindex/cutover adapter and is blocked by this packet. The old bootstrap promotion
+and newer experiment identity serializers differ; those existing identities are not rewritten
+or silently treated as equivalent. The same-space path therefore does not make arbitrary
+new-model replacement available and does not imply that consumer cutover is complete.
+
+The contract generation starts at zero for existing schemas and is authoritative thereafter.
+All stable-route writers, including initial promotion and explicit drain, share the same
+contract lock and generation increment. Recovery must preserve this value and exact rollback
+provenance; an opaque legacy populated fingerprint is not a substitute for a qualified
+default-zero source projection.
+Operators obtain `expected_route_generation` from `route_generation` on the existing deployment
+response. They must refresh and review after a conflict rather than guessing a generation.
+
+Validation includes real offline lifecycle commit/rollback, stale authority/binding/identity
+negative tests, transaction rollback on subject conflict, and an optional disposable PostgreSQL
+two-committer race with runtime-role writes. No inference, external index operation or production
+deployment is performed by this implementation qualification.
+
+The project comparison view offers a read-only replacement preview. It shows every affected
+stable binding, current/candidate identities, route generation, valid production approval
+choices and actionable blockers. Stale historical production approvals are excluded without
+invalidating another valid choice. The preview neither declares production validation satisfied
+nor creates lifecycle authority. Historical lifecycle approvals remain readable after a route
+change; expected qualification failures mark them STALE with an audited reason, and previously
+approved stale authority remains explicitly revocable.
@@ -0,0 +1,66 @@
+# ADR 0059: Isolated RC embedding comparison and consumer cutover
+
+Status: proposed; implementation and runtime qualification pending.
+
+## Context
+
+The RC product contract requires a disposable `validation-canary` node with
+`production_eligible=false`. Stable serving promotion and the approved lifecycle
+transition require a production-eligible node and production approval. Relaxing
+those checks for RC testing would grant a LAB deployment production authority.
+
+The selected-comparison workflow currently compares a stable deployment with
+one LAB experiment route. An isolated RC stack needs two LAB routes, one for its
+current embedding identity and one for the candidate. Different embedding-space
+IDs require separately pinned Qdrant collections even when vector dimensions
+match. The experiment routes are capability-wide; changing their targets would
+also affect any other client using the same key.
+
+## Decision
+
+- Keep stable deployment selection and production lifecycle rules unchanged.
+- In the isolated RC stack, create two distinct LAB_READY deployments with
+  approved LAB execution and opaque, immutable experiment routes. Give the
+  designated ExampleRAG project one scoped service client credential.
+- Permit a selected comparison with an explicit current experiment route ID.
+  Pin and revalidate both route-to-deployment mappings, full embedding
+  identities, reviewed suite, corpus, retrieval settings, consumer and distinct
+  index references. A stale route, credential, approval or identity fails closed.
+- ExampleRAG builds the candidate collection separately and executes both legs
+  through the pinned ModelForge routes. The current collection is never
+  overwritten during comparison. Gateway request IDs and retrieved chunk IDs
+  are retained as evidence.
+- For a different embedding space, the RC cutover authority is a single durable
+  ExampleRAG selection record that names **both** the active embedding profile
+  (including its experiment route key and embedding-space ID) and the Qdrant
+  collection. Ingest and query resolve that same record. Activation and rollback
+  use its monotonically increasing generation and compare-and-swap. They do not
+  retarget a ModelForge experiment route or its production stable route.
+- Activation requires the completed, reviewed ModelForge comparison and an
+  explicit operator decision covering impact and reindex. The candidate
+  collection must have a complete authoritative reindex, correct identity pins,
+  and a successful consumer smoke check before the selection changes. A retry
+  with the same operation ID returns the recorded result; a stale generation or
+  different payload conflicts. Rollback must verify the retained previous
+  collection/profile and current generation before restoring the prior
+  selection. Calls already admitted may finish against the old selection;
+  subsequent calls use the new generation.
+- A failed or interrupted operation leaves the previous selection usable or
+  enters an explicit blocked state with a recovery journal. It never silently
+  falls back to Ollama or an unpinned collection.
+
+## Qualification boundary
+
+This is an RC consumer transition on disposable state. It does not claim
+production promotion, a stable route change or a production rollout. The final
+candidate must prove the full comparison, activation, real request, rollback,
+second real request, duplicate/retry behavior, stale CAS, concurrent operators,
+and process-restart recovery on the same exact images and source revision.
+
+## Consequences
+
+ModelForge remains the source of model identity, LAB execution, gateway,
+scheduler and comparison evidence. ExampleRAG owns the atomic choice of its
+embedding route and matching index. A global experiment-route remap is not used
+as a shortcut for consumer cutover. The RC cannot be marked qualified until the
+selection record and its runtime tests are implemented and verified.
@@ -0,0 +1,54 @@
+# Managing serving configuration
+
+Open **Configuration** after unlocking the Console. The catalog lists typed values, deployment
+source where provable, sensitivity, validation bounds and whether the value is editable.
+
+Five numeric controls have a live serving consumer: embedding batch size, input character limit,
+gateway request timeout, gateway queue timeout, and telemetry staleness threshold. Other entries
+remain deployment-managed. The separate Scheduler policy workflow remains authoritative for its
+policy values; this editor cannot create a competing scheduler policy.
+
+1. Enter one or more safe overrides. Queue timeout must remain below request timeout. Use
+   **Use deployment value** to remove an existing override.
+2. Choose **Preview changes**. Review before/after values, source changes and impact.
+3. Acknowledge the impact and choose **Apply reviewed changes**. Administrator authority is
+   required; operators may preview and viewers may inspect.
+4. The result names the accepted revision and consumer. The API's serving policy reader uses that
+   snapshot for new requests. Requests that already consumed their configuration retain their
+   snapshot; no worker restart or model promotion is implied.
+
+If another operator applies a change or deployment-resolved values change after preview, apply
+fails with a conflict. Refresh and preview again; the Console does not retry mutations silently.
+
+**Revision history and rollback** retains actor, time, request correlation and complete override
+sets. Restoring an earlier revision or deployment defaults creates another preview followed by
+explicit apply. It does not erase history. Revision zero means the current deployment defaults;
+it does not restore an old environment file.
+
+**Export and import safe overrides** exports JSON with format/product compatibility metadata,
+safe overrides and names of secret slots. It excludes connection strings, credentials and unrelated
+deployment topology. Import replaces the complete override set after a compatibility check and
+review. Slot names are references only; imports never set, rotate or revoke secrets.
+
+Bootstrap and deployment-managed credentials still require the deployment's documented procedure.
+The fixed Hugging Face control-plane slot has a separate administrator-only write-only workflow for
+candidate creation, testing, activation, rotation and revocation; it never reveals or exports a
+token. It does not manage node download credentials. See the [secret lifecycle procedure](SECRET_LIFECYCLE.md).
+Do not paste credentials into the numeric editor or a configuration export.
+
+## Persistence, upgrade and recovery
+
+Schema `20260907_0025` adds `configuration_state` and `configuration_revisions`. Both are
+authoritative PostgreSQL backup state. A validated override, active pointer and attributable audit
+event commit together. Failed audit writes roll back the configuration change. Runtime credentials
+can append revision history but cannot update, delete or truncate it.
+
+The migration leaves existing model, project, credential and audit evidence unchanged and initially
+introduces two empty tables. It supports a mechanical downgrade only while there is no operator
+revision history. After configuration changes, recovery uses the verified pre-migration backup;
+the downgrade guard refuses to destroy the history. Restore compatibility accepts older-schema
+backups only when the added configuration tables are exactly empty, and requires exact retained
+configuration fingerprints for schema-0025 restores.
+
+This source milestone does not create a release or change the deployed 1.2.2 schema. A release
+requires exact-head validation and the separate install/upgrade/rollback qualification.
@@ -0,0 +1,38 @@
+# Updates for chosen project models
+
+ModelForge follows explicitly selected Hugging Face model references. It compares
+the resolved commit with the exact baseline you selected. A changed commit is an
+opportunity to review, not proof that a model is newer, safer or better.
+
+In Discover, create a watch for a registered model, an exact baseline revision and
+the upstream reference to follow. Checks fetch metadata only. Each different commit
+produces one stored notice; dismissing it preserves history and prevents repeated
+checks from reviving the same notice. Provider errors remain visible beside the last
+check and never imply that the current model is up to date.
+
+In production, enabled watches are checked daily by default. Each API process checks
+at most one due watch per minute, and a database claim prevents processes from repeating
+the same completed scheduled check. There are at most 100 explicit model/reference
+watches. Development and test processes do not start background checks. The Console
+reports the configured monitor mode; individual last-check times and errors show
+what actually happened. You can always request a manual check with operator access.
+
+`MODELFORGE_MODEL_UPDATE_MONITOR_ENABLED=false` disables automatic checks after an
+API restart. `MODELFORGE_MODEL_UPDATE_CHECK_INTERVAL_SECONDS` controls the per-watch
+interval (default 86400; permitted range 3600–604800). Compose passes both settings
+to the API. Pausing one watch stops that reference without affecting model serving.
+
+Use Review candidate to select the exact observed model and SHA in acquisition.
+Review its metadata and impact before explicitly preparing a download. This action
+clears any previous acquisition plan; it cannot approve, download or replace a model.
+Tracking-baseline changes use a separate audited concurrency check and preserve
+old notices. They change what is compared during future checks, not the deployment.
+
+Real project comparison, recommendation and replacement remain separate evidence
+and lifecycle steps. The update inbox cannot certify quality or bypass approval,
+private-worker scheduling, embedding reindexing or rollback requirements.
+
+Schema 0029 stores watches and notices. Upgrade from schema0028 introduces empty
+tables. Once tracking history exists, rollback uses a pre-migration backup rather
+than deleting that history through downgrade. Restore preserves tracking state and
+notice provenance, but clears temporary check claims before allowing READY.
@@ -0,0 +1,108 @@
+# Operator access
+
+Use a separate personal key for each human. Viewer reads protected evidence; operator can additionally
+perform permitted operational actions and previews; administrator controls configuration apply,
+credentials, destructive lifecycle and recovery. The API enforces every route's declared role.
+Project `mfsvc_`, node `mfnode_` and enrollment keys never unlock the Console.
+
+## Deployment-owned personal identities
+
+In a private local terminal run:
+
+```sh
+python scripts/operator_identity.py --subject alice --role viewer --days 30
+```
+
+Save `one_time_key` in your password manager and give it only to that person through your approved
+private channel. Add **only** `deployment_record` to the JSON array in `MODELFORGE_OPERATOR_IDENTITIES`.
+The array must contain 1–100 records. A `.env` assignment uses single quotes around the JSON array.
+Never put the complete command output in `.env`, Git, a ticket, an export or a release artifact.
+The stored record contains a credential ID, subject, role, digest and timezone-aware expiry; it does
+not contain the usable key. Configuration reads redact the entire registry.
+
+Apply the deployment setting using the normal controlled API recreation procedure. On the next
+request the API uses the new registry. Enter the personal key in the Console's Operator API key
+field. Verify the displayed subject and role before acting. `GET /api/v1/operator/identity`, with
+`X-ModelForge-Admin-Token` supplied from your private credential store, provides the same information.
+There is no API for reading a key back.
+
+## Deployment-owned rotation, revocation and recovery
+
+Generate a replacement record with the same subject and role and a new credential ID. Add it alongside
+the old record, recreate the API, and verify the replacement login and role. Remove the old record
+and recreate the API again to end the overlap. The old key then receives 401. To revoke immediately,
+remove the record and recreate the API without an overlap. Record the subject, credential IDs,
+approver, reason and application time in the private deployment change record; never record keys.
+These deployment operations do not currently produce online identity-lifecycle audit events.
+
+Changes require recreation, not just editing the host environment. In-flight operations already
+authorized may finish. The expiry itself is checked on every request without a restart. Retain an
+independent administrator or controlled recovery credential during rotation. Conflicting roles for
+one subject are rejected; change every overlapping record together when changing a role.
+
+The legacy `MODELFORGE_OPERATOR_API_KEY` remains the `legacy-operator` administrator. Once a personal
+administrator is proven, clear that setting and recreate the API to disable shared access. Production
+startup refuses an installation without either a valid bootstrap key or an unexpired named admin.
+Keep recovery material outside the database backup and protect it according to your deployment policy.
+Managed database identities do not replace this independent startup/recovery requirement.
+
+## Managed personal identities
+
+The RC personal-key provider also supports online identity administration. An administrator can
+list identity metadata, create a managed identity, rotate its key, change its role or enabled state,
+and revoke its credentials. Deployment-owned entries remain read-only in this interface. Managed
+subjects use `managed:<slug>` and keys use the distinct `mfopm_` namespace. Existing `mfop_`
+deployment keys retain their authentication and recreation behavior.
+
+Create the first managed identity as an administrator. Choose the minimum necessary role and a
+key lifetime of 1–365 days. Issuance returns the key once; save it in an approved private credential
+store before leaving the page. A lost response cannot be recovered from the database: rotate to
+issue another key. Only a digest is persisted. Never paste the issuance response into a ticket,
+audit reason, repository, configuration export or diagnostic bundle.
+
+Rotation accepts an overlap of 0–900 seconds. Verify the replacement key promptly; zero overlap
+invalidates the predecessor immediately. Revocation requires the exact canonical subject and a
+reason. These mutations use the displayed version, recheck administrator authority, and record an
+attributed audit event in the same transaction. A stale version requires refresh and a new review.
+Once initialized, administration cannot remove the last currently usable managed administrator.
+Expiry can still exhaust access later; maintain the independent deployment recovery authority.
+
+The Console requires at least 120 seconds of overlap when rotating your own key, and at least
+120 seconds remaining before both the current key expiry and local Console lock. Re-unlock first
+if the local deadline is too close; use another administrator or the recovery authority when the
+key itself is near expiry. The replacement is displayed for 60 seconds without a follow-up read
+that could invalidate the issuance response. Save it, then lock and unlock with the replacement
+before the overlap ends. Navigation, form changes and locking clear the displayed key.
+
+The metadata endpoint is `GET /api/v1/operator/managed-identities`. Creation uses POST to that
+collection; rotation and revocation use POST to `/{slug}/rotate` and `/{slug}/revoke`, and role or
+enabled-state changes use PATCH to `/{slug}`. All require administrator authority. Keys never appear
+in metadata, and a managed-key lookup failure cannot fall back to the deployment administrator.
+
+## Managed keys after database recovery
+
+Supported recovery restores the authoritative identity and credential history, then clears managed
+credential activations during reconciliation. Validation independently checks that these current
+activation rows are absent. A restored key is therefore quarantined even if the backup predates
+its later revocation. Do not reopen access to a raw `pg_restore` result: use the complete isolated
+ModelForge recovery workflow and its validation gate.
+
+After validated recovery, authenticate using the independent deployment administrator and issue a
+fresh managed administrator key by rotation or creation. Verify that new key before resuming
+ordinary identity administration. Restored keys remain unusable; preserving their metadata is not
+permission to reactivate them. Keep the restored environment isolated until these checks complete.
+
+## Console locking and limitations
+
+The Console displays identity, role, key expiry and its local lock time. It locks at the earlier of
+fifteen minutes after unlock or key expiry. Lock console and reload both clear the memory-only key.
+Public liveness and version remain visible while locked. A 403 keeps the session and reports a
+forbidden operation; a database/service outage also keeps the key. A 401 clears it. No credential is
+stored in cookies, localStorage, sessionStorage or a URL, and protected API responses are not cached.
+
+The local lock is not a server-issued session or key revocation: a copied key remains usable until
+its expiry or removal. OIDC/Authentik and SSO logout are not implemented; personal header keys are
+the explicitly supported RC provider. Deployment-secret lifecycle is a separate feature.
+Use HTTPS outside an explicitly trusted private lab, and protect the deployment
+registry so an untrusted user cannot replace roles or credential digests. The full RC authentication
+and secret-lifecycle acceptance remains open until those documented decisions and journeys close.
@@ -0,0 +1,11 @@
+# ModelForge platform update preflight
+
+The Console's **Update preflight** view reads `GET /api/v1/admin/updates/preflight`. The endpoint is administrator-only and read-only. It reports the current running API release identity and the evidence available for a proposed target.
+
+The current schema revision in this view is the schema target declared by the running application build. The preflight does not read the database's applied Alembic revision; use the database migration checks when that value is required.
+
+This source line does not provide an approved platform-update manifest. Until one is configured, the approved version, release notes, target schema, schema and breaking impact, backup requirement, and approved-artifact provenance are `UNKNOWN`. Missing current API build provenance fields are also `UNKNOWN`. A preflight with these gaps is `INCOMPLETE`; `UNKNOWN` does not mean that no backup is required.
+
+The current provenance status is `AVAILABLE` only when source commit, build timestamp, and image digest are all supplied by the API build. These values are displayed build metadata, not an independent signature or artifact verification. The approved target's provenance remains `UNKNOWN` without approved release metadata.
+
+The endpoint does not fetch packages, persist a plan, write release state, create or verify backups, or deploy an update. It has no execution operation. An operator must review approved release evidence, impact, and verified backup readiness before planning maintenance; execution remains human-controlled.
@@ -0,0 +1,61 @@
+# Project onboarding and service credentials
+
+Open **Projects** with a personal operator or administrator key.
+
+1. Create a project with a stable slug, name, description, owner, environment and optional labels.
+2. Add an exact versioned capability requirement. Select reindex support for incompatible embedding
+   spaces, or schema migration support when required. The requirement uses the stable gateway route.
+3. As administrator, issue a service identity for one explicit binding. Choose a unique client name,
+   purpose and expiry. The binding fixes its capability and workload priority.
+4. Save the returned key in the consumer's secret store or a protected token file. It appears once
+   and the Console hides it after 60 seconds. The database retains a digest, never plaintext.
+5. Expand the integration examples. Set the gateway origin, token-file location and capability in
+   the consumer environment. The examples never contain the issued key.
+6. Paste the service key into the memory-only test field and run **Check connectivity**. This verifies
+   its exact scope without starting inference. The key must belong to the selected identity.
+7. Supply non-sensitive capability-specific JSON and run **Run capability check**. This executes one
+   real gateway request and may load the governed deployment. Image/audio templates require real
+   base64 input; placeholder strings cannot produce a successful test.
+
+The workspace shows blockers, deployment health, current key expiry, last use and the successful
+gateway request ID. Readiness requires current key/current deployment evidence from the last 24
+hours. Rotation, a deployment change, expiry, revocation or stale evidence invalidates the relevant
+checks. Production eligibility is deliberately not inferred; use the project-fit and lifecycle
+approval workflows. A fresh installation with no eligible stable deployment stays visibly blocked.
+
+## Rotation and revocation
+
+Choose immediate replacement, 15 minutes, one hour or 24 hours of overlap. Both the expected
+credential ID and the replacement expiry are sent to the API. Refresh after a conflict. The old
+key expires no later than its original expiry or the overlap deadline; another rotation ends any
+older overlap. Replacement keys are also revealed once.
+
+**Revoke all keys** shows the client, capability and last use. Type the exact client name and
+acknowledge the interruption. Current and overlapping keys immediately stop authorizing new
+requests. Already admitted work may finish. A revoked identity cannot be revived through rotation;
+issue a new service identity if access is needed again. Historical metadata remains auditable.
+
+Legacy single-key rotate/delete routes remain available for compatible automation and use immediate
+revocation; they also use the same transactional claim. New integrations should use the guarded
+plural `/credentials/rotate` and `/credentials/revoke` routes.
+
+## API surface
+
+| Route | Authority | Result |
+| --- | --- | --- |
+| `GET /api/v1/project-workspace` and `/{key}` | Viewer | Persisted requirements, safe client metadata, observed readiness |
+| `POST /api/v1/project-workspace` | Operator | Persist project identity/owner/environment/labels with audit |
+| `POST /api/v1/project-workspace/{key}/requirements` | Operator | Bind an exact compatible requirement with audit |
+| `POST /api/v1/admin/service-clients` | Admin | Issue key once; accepts explicit `project_binding_id` |
+| `GET /api/v1/admin/service-clients/{id}/credentials` | Admin | Most recent 100 key metadata records, never digests/plaintext |
+| `POST /api/v1/admin/service-clients/{id}/credentials/rotate` | Admin | Expected credential ID, overlap seconds and new UTC-aware expiry |
+| `POST /api/v1/admin/service-clients/{id}/credentials/revoke` | Admin | Expected credential ID and exact client-name confirmation |
+| `GET /api/v1/capabilities/{capability}/check` | Scoped service key | Connectivity, client identity and correlation ID; no inference |
+
+Connectivity checks exist for the five implemented embedding, reranking, OCR, vision and
+transcription capability routes. Human keys cannot authenticate there, and service keys cannot
+open the Console. Protected control-plane responses use `Cache-Control: no-store`.
+
+Migration `20260908_0026` adds onboarding metadata and credential supersession. Restore the verified
+pre-migration backup if rollback is necessary after new history exists; do not drop that history.
+See [verification](../quality/RC02_PROJECT_VERIFICATION.md) and [clients](../../clients/README.md).
@@ -14,7 +14,7 @@ tree. From that tree it:
 3. scans every exact image ID with checksum-pinned Trivy and rejects ambiguous reports or any
   HIGH/CRITICAL finding;
 4. starts API, Console, PostgreSQL and Redis with the production overlay under a unique
-   `modelforge-rc-*` Compose project;
+   `modelforge-validation-*` Compose project;
 5. uses newly generated secrets, loopback-only ephemeral host ports and project-scoped volumes;
 6. proves liveness, readiness, version identity, Console serving and both sides of the operator
   authentication boundary; and
@@ -24,8 +24,94 @@ tree. From that tree it:

 No Node Agent or Runtime Worker is started by this clean-install gate. Consequently it creates no
 compute identity and cannot interact with the production GPU node. A later GPU qualification must
-enrol a disposable `rc-canary` identity with `production_eligible=false` and retain that separate
-evidence.
+enrol a disposable `validation-canary` identity with `production_eligible=false` and retain that
+separate evidence.
+
+## P12 runtime-isolation preflight
+
+Before a P12 Node Agent or Runtime Worker starts, render its exact candidate Compose projection to
+JSON and run `scripts/rc_isolation_preflight.py`. The check requires a bounded
+`modelforge-validation-*` project and refuses external/non-project volumes or networks, host
+networking, global container names, non-loopback or
+protected ports, Docker-socket mounts, writable bind mounts, missing bind sources, source paths
+outside the candidate root, and references to named production resources. A read-only bind targeting
+`/app/config` must contain `models/initial-candidates.yaml` under that exact source path. Run the
+preflight before `docker compose up`. After `up`, run it again with
+`--inspect-docker`; it records only resources carrying the exact Compose project label and rejects
+any name outside that project namespace.
+
+This preflight is an admission control, not P12 qualification evidence. A green result does not
+replace the required fresh secrets, `production_eligible=false` `validation-canary` enrollment, real
+private runtime inference, before/after production readiness, normal lifecycle cleanup, or retained
+runtime evidence.
+
+### Dedicated test ingress for a retained RC stack
+
+The disposable clean-install workflow above removes its stack. For a later retained P12 stack,
+include `docker-compose.rc-ingress.yml` after the base and production files. It places PostgreSQL,
+Redis and the migration job on a project-private `backend` network; Web is only on `ingress`, and
+API joins both. The ingress network is created by the exact `modelforge-validation-*` project, not
+shared with production. Bind all published control-plane ports to `127.0.0.1` and choose unused
+ports; the reverse proxy reaches Web and API over the ingress network, not those host ports.
+
+Port 443 is reserved for the existing production services. Its NPM host is still pending because
+the current production Console embeds an HTTP API origin and CSP; the candidate does not change it.
+The candidate must use a separate
+HTTPS test listener on port 18444 with a certificate valid for `modelforge.example.com`. Its
+canonical browser/API origin is `https://modelforge.example.com:18444`; set both
+`MODELFORGE_CORS_ORIGINS` to that value at runtime and use it for `VITE_API_BASE_URL` when building
+the candidate Console image.
+The test listener routes only to the retained validation project.
+
+Render the complete Compose plan with the exact candidate source, four image identities, freshly
+generated validation secrets and intended public origin. Run `rc_isolation_preflight.py` on that
+JSON before starting any container and again with `--inspect-docker` afterward. Add
+`--public-tls-port 18444` only with `docker-compose.validation-tls.yml`; the preflight admits
+this one port solely for the pinned, confined `tls` service. It must show only project-scoped
+networks and volumes, loopback control-plane ports and the exact source-bound config mount.
+The operator must also verify candidate and production stack health before and after startup.
+
+The isolated `tls` service uses a minimal Caddyfile to route `/api/` and `/v1/` to its API and
+`/` to its Web service on the same private ingress network. It binds only port 18444. NPM remains
+on its own networks and keeps port 443 for production. Stage only a protected copy of the NPM
+certificate/key under the private candidate root, outside the image build context; never mount
+the NPM certificate store into the candidate. Follow
+`docs/operations/VALIDATION_TLS_TEST_PORT.md` for exact staging and HTTPS checks. Validate the
+certificate, Console, API calls and browser CSP without disabling TLS verification.
+
+## Candidate stack retention guard
+
+Use `scripts/rc_stack_retention.py` to review server-side `modelforge-validation-*` Compose
+containers whose temporary source tree may have been removed or reduced after a run. Its default
+mode only inspects containers and prints JSON. It reports the oldest container age (stale after 24 hours by default),
+missing bind sources, and a missing `config/models/initial-candidates.yaml` under any `/app/config`
+bind source. Set `--stale-after-hours` to change the age threshold.
+
+```sh
+python3 scripts/rc_stack_retention.py
+python3 scripts/rc_stack_retention.py --stale-after-hours 48
+```
+
+Stopping is limited to one exact stale or incomplete RC project. Before issuing `docker stop`, the
+operator must provide a nonempty retained evidence record and explicitly attest that no jobs are
+active. A running API, one-off job, migration, agent, runtime worker, or unknown service blocks the
+operation. Only running `postgres`, `redis`, or `web` containers from that inspected project may be
+stopped. The command does not remove containers, volumes, networks, or images; it never targets
+`modelforge` or a project outside the bounded validation namespace.
+
+```sh
+python3 scripts/rc_stack_retention.py \
+  --stop-project modelforge-validation-example \
+  --confirm-stop modelforge-validation-example \
+  --confirm-no-active-jobs \
+  --evidence-record /path/to/retained-rc-evidence.json
+```
+
+The evidence record is a UTF-8 JSON object with exactly these fields:
+`{"schema_version":1,"project":"modelforge-validation-example","retained":true,"evidence_reference":"acceptance-run-id"}`.
+The project must match the selected project, and the evidence reference must identify the retained
+acceptance output or audit evidence. The operator is responsible for confirming that the reference
+points to the expected evidence before the stop command.

 After the workflow exists on the default branch, dispatch it directly and enter the exact pushed
 commit. Before merge, dispatch the existing **Managed validation** workflow on the candidate branch
@@ -0,0 +1,32 @@
+# Runbook: temporarily enable or disable a compute node
+
+Use the lifecycle action for planned temporary maintenance. Permanent removal belongs in
+`RUNBOOK_NODE_DECOMMISSION.md`; credential loss belongs in `RUNBOOK_NODE_LOSS.md`.
+
+## Disable
+
+1. Drain and stop workloads before starting. The Console's admin-only **Review disable impact**
+   preview checks active deployments, requests, jobs, runtime probes, leases, residencies, pending
+   transitions, storage dependencies and other known node references. A blocker means resolve the
+   dependency and request a fresh preview; there is no force option.
+2. Confirm the preview names the intended persisted node identity. Enter the operator identity, a
+   reason of at least ten characters, and that persisted identity exactly.
+3. The API rechecks the dependency digest while holding the node and dependency locks, then advances
+   the node generation with compare-and-swap. A 409 means the review is stale or a dependency still
+   exists; inspect the new state and review again.
+4. Disabled nodes reject agent heartbeats and cannot claim new agent work. Do not treat the action
+   as a graceful drain; the preview refuses known active workloads instead.
+
+## Enable
+
+1. Resolve a lost/revoked credential first using `RUNBOOK_NODE_LOSS.md`. Remote nodes without an
+   active node credential cannot be enabled.
+2. Use **Review enable impact**, inspect the current state and type the persisted identity with the
+   operator identity and reason.
+3. A successful enable leaves the remote node `offline` and clears old heartbeat/protocol/capability
+   facts. The agent must authenticate and send a fresh compatible heartbeat before health or
+   scheduler readiness can be inferred. The action does not start or restart the agent on its host.
+
+Never change `enabled` through the metadata `PATCH`; the API rejects that field. Use the preview and
+execute lifecycle endpoints so impact, generation and audit checks cannot be skipped. Do not rehearse
+on production nodes.
@@ -31,9 +31,17 @@ matches and the node reconnects on its own.
 identity file, so it re-enrols onto the same node id with its accelerators, storage roots and
 history intact. Enrolment revokes the previous credential automatically.

+If the node's identity volume is still present and an administrator can install a replacement
+credential directly, open the node detail and use **Rotate node credential**. Rotation revokes the
+old key immediately. Replace `/data/state/node-credential` with the one-time value and recreate the
+agent before expecting heartbeats or inventory to resume. The Console cannot install the value on
+the node. If the identity volume is missing, use fresh enrolment instead.
+
 **The credential is believed stolen.** Revoke it first
 (`DELETE /api/v1/admin/hardware/nodes/{node_id}/credential`), then re-enrol. Revocation is
-permanent: the old credential stays refused after re-enrolment.
+permanent: the old credential stays refused after re-enrolment. If a replacement can be installed
+immediately, the node detail's rotation action combines revocation and replacement; the old key is
+still permanently refused.

 **The identity file is also lost.** The agent enrols as a new node, because ModelForge genuinely
 cannot tell it apart from different hardware. If the old identity is permanently superseded, use
@@ -0,0 +1,45 @@
+# Scheduled backups
+
+The scheduled backup helper creates a recovery backup, verifies it, then applies retention. The
+recovery endpoints require the administrator role. Supply a key for an existing named administrator
+when the deployment no longer uses the legacy global operator key. Named deployment credentials and
+their lifecycle are described in [Operator access](OPERATOR_ACCESS.md).
+
+## Authentication settings
+
+Set `MODELFORGE_BACKUP_ADMIN_API_KEY` to a named administrator key for the scheduler. The key is
+sent only to the ModelForge API in the existing `X-ModelForge-Admin-Token` header. The helper does
+not print it. Keep the value in the deployment's protected environment or `.env` file; do not commit
+it. The API must already recognize the credential as an administrator.
+
+`MODELFORGE_OPERATOR_API_KEY` remains a backward-compatible fallback for existing deployments. If
+both settings are present, `MODELFORGE_BACKUP_ADMIN_API_KEY` takes precedence. If neither has a
+non-empty value, the helper exits before making an API request, so no backup or retention operation
+can be authorized.
+
+For a Compose deployment, add the dedicated key to the protected `.env` file used by Compose:
+
+```dotenv
+MODELFORGE_BACKUP_ADMIN_API_KEY=<existing named administrator key>
+```
+
+Then start or recreate the scheduler with the usual stack files:
+
+```sh
+docker compose -p modelforge --env-file .env \
+  -f docker-compose.yml -f docker-compose.backup.yml \
+  up -d --force-recreate backup-scheduler
+```
+
+For a host timer or Unraid User Script, export `MODELFORGE_BACKUP_ADMIN_API_KEY` in the protected
+job environment before invoking `scripts/modelforge_scheduled_backup.sh`. The legacy variable can
+continue to be used for direct invocations as well.
+
+## Credential lifecycle
+
+Use an administrator identity dedicated to scheduled recovery work. Backup and recovery endpoints
+currently require the administrator role; there is no narrower backup-only role. Keep the credential
+valid and unexpired, and rotate or revoke it through the named identity's supported lifecycle. After
+changing the scheduler's environment, recreate `backup-scheduler`. For deployment-owned named keys,
+apply the API's identity registry change and recreate the API as described in the operator access
+runbook.
@@ -0,0 +1,57 @@
+# Platform secret lifecycle
+
+Implementation qualification in progress. This procedure describes the P01 control-plane
+Hugging Face slot; it does not certify live provider access or download-node credentials.
+
+## Boundaries
+
+The managed slot supplies Hugging Face authentication for control-plane discovery and repository
+metadata. A successful test confirms that the provider accepts that exact candidate credential;
+it does not establish a repository licence, gated-model approval, artifact safety or node download
+readiness. Download nodes retain their separate deployment-managed credential.
+
+Database bootstrap, migration access, deployment administrator recovery credentials and backup
+encryption remain outside the managed slot. Their metadata must show external ownership and
+unknown versions or test state truthfully. Do not put the root key or bootstrap credentials into
+the slot they would protect.
+
+## Operator workflow
+
+1. Provision and retain the separate encryption root key outside database backups, using the
+   documented deployment settings. Keep its identifier stable for the ciphertext it protects.
+   `MODELFORGE_SECRET_SLOT_ROOT_KEY` contains a base64-encoded random 32-byte AES key;
+   `MODELFORGE_SECRET_SLOT_ROOT_KEY_ID` identifies it. Recreate the API after changing these
+   deployment settings. Historical ciphertext still needs its original key and identifier.
+2. As an administrator, enter the Hugging Face token and an audit reason. Review the current
+   revision. Saving creates an encrypted candidate; it does not replace the active credential.
+   The first save opts into managed mode immediately: metadata/discovery access is blocked until
+   that candidate is tested and activated. Plan that initial switch as a short maintenance action.
+3. Test that candidate. A rejected token, provider outage or stale candidate blocks activation
+   and leaves the existing active credential intact. Review the recorded result and time.
+4. Explicitly activate the successfully tested version. New control-plane requests resolve that
+   version; already-started requests may retain their credential snapshot.
+   The successful test must be less than five minutes old and refer to the latest candidate.
+5. To rotate, repeat candidate creation, testing and activation with the replacement token.
+   Confirm ordinary discovery/metadata access before revoking the old token at the provider.
+6. To disable use, explicitly revoke the managed slot. ModelForge must not silently fall back
+   to a deployment token. Provider-side token revocation is a separate action.
+
+The token cannot be read back. Configuration export contains only the slot reference, never
+ciphertext, plaintext or the root key. Never include tokens in reasons, logs, tickets or reports.
+Use the Console's lock action when finished; token input must be cleared after submission and
+when leaving the workflow.
+
+## Failure and recovery
+
+Missing encryption material, a wrong key or invalid ciphertext makes a managed credential
+unavailable. Restore the externally retained matching root key through the deployment process;
+do not replace it with an unrelated key merely to make startup appear healthy.
+
+A supported database restore preserves encrypted history but invalidates prior test and activation
+evidence. Keep the restored environment isolated. An administrator must test and explicitly
+activate the intended version again. Old provider credentials may already have expired or been
+revoked; database rollback cannot undo that external state. A raw database restore is not proof
+that a secret is safe to reactivate.
+
+Final RC evidence must include live provider qualification and safe root-key provisioning on the
+isolated RC host. Node credential activation and gated downloads require their own evidence.
@@ -0,0 +1,132 @@
+# Selected model comparisons
+
+The project hub can persist one comparison of its current ModelForge embedding deployment with one
+selected candidate. Creating an intent leaves it `pending` until its designated project consumer
+claims the work. The API and Console do not start an evaluator automatically. A dedicated ExampleRAG
+consumer operator implements this protocol on `main` after PR #9 (`64c75be`); its targeted and
+managed full tests are green, but no live consumer qualification is implied.
+
+The existing ExampleRAG `examplerag-modelforge-evaluate` CLI remains outside this protocol: its CURRENT leg
+uses a dynamically selected legacy profile, and its output does not implement these claim and gateway
+evidence requirements. Historical M6 results cannot be imported as fresh execution of a new intent.
+
+## Prepare an intent
+
+Select an active stable `rag.embedding@1` project binding, an active service client belonging to
+that exact binding, the current production stable deployment, and a nonproduction candidate with
+an active experiment route. Prepare a separately identified candidate vector index; matching vector
+dimensions do not authorize reuse of the current index or replace embedding migration.
+
+Select a project-owned immutable suite containing 1–100 reviewed cases. Each case's
+`label_provenance.corpus_revision` must equal the supplied lowercase SHA-256 corpus digest.
+The operator creation payload contains:
+
+- `project_binding_id`, `consumer_client_id`, `suite_revision_id`;
+- `current_deployment_id`, `candidate_deployment_id`, `candidate_route_id`;
+- `corpus_sha256`, `retrieval_config_sha256`;
+- distinct `current_index_ref` and `candidate_index_ref`;
+- a unique `idempotency_key` for this intended attempt.
+
+The server pins the deployment/runtime/artifact/embedding identities and the immutable embedding
+dimension, then checks them again at claim and result publication. A changed target requires a new
+intent. One pending, running or
+cancellation-requested intent can exist per binding. An identical creation retry returns the same
+record; reuse of its key with different input conflicts.
+
+## API and authority
+
+All paths below are relative to `/api/v1/selected-model-comparisons`.
+
+| Method/path | Authority | Purpose |
+| --- | --- | --- |
+| `POST /` | Operator | Persist the exact comparison intent |
+| `GET /` and `GET /{id}` | Viewer | History/detail, including the derived evaluation summary |
+| `POST /{id}/cancel` | Operator | Request cancellation using `expected_version` |
+| `GET /assigned` | Designated project service client | Discover its bounded assigned work |
+| `POST /{id}/claim` | Designated project service client | Obtain the one claim token, pins and suite cases |
+| `POST /{id}/progress` | Same client plus claim token | Renew the lease and report bounded counts |
+| `POST /{id}/result` | Same client plus claim token | Publish both legs atomically |
+| `POST /{id}/finish` | Same client plus claim token | Report failure or acknowledge requested cancellation |
+
+Use the collection path without a trailing slash when making requests. History accepts
+`project_binding_id`, `limit` (1–100, default 50) and `offset` (0–10000). Assignment accepts the same
+limit bound. The service credential must authorize `rag.embedding@1`; another client in the same
+project cannot claim or report this intent. Operator credentials cannot call the consumer protocol.
+
+The claim returns `comparison`, `claim_token`, `claim_expires_at` and `cases`. Retain the token only
+in the consumer's private execution context; do not put it in logs or result artifacts. Normal
+history and audit do not return it. Losing the successful claim response does not create a second
+claim: the original claim expires, after which an operator can create a new attempt explicitly.
+
+## Execute and report
+
+Before any inference, the adapter must validate its actual corpus, index, embedding profiles and
+retrieval settings against the pinned identities. Reuse the consumer's retrieval engine. For every
+successful case, send its exact unchanged query as a single embedding input through the appropriate
+ModelForge stable or candidate route, and retain the gateway's public `request_id`.
+
+Send progress with `claim_token`, `current_completed_cases` and `candidate_completed_cases`.
+The lease lasts 120 seconds; renew before it expires. Counts can only increase and cannot exceed
+the suite count. Renewal never extends the one-hour deadline from initial claim. Stop starting
+additional work when the response reports `cancel_requested`.
+
+Result submission contains `claim_token`, the returned `pins_digest`, a lowercase SHA-256
+`consumer_revision_sha256`, and `current`/`candidate` objects each containing `results`. Every result
+contains `case_id`, bounded `ranked_results` (`chunk_id`, optional `document_id`, finite `score`),
+`latency_ms` and optional `error_code`. Submit exactly one result per suite case for each leg.
+
+`current_gateway_requests` and `candidate_gateway_requests` map each successful case UUID to its
+gateway **public `request_id`**, not the database row `id`. Each mapping value must be unique across
+the pair, belong to the designated client and exact deployment, and reference a completed
+`rag.embedding@1` request. Its single input must hash to the suite query. Server request timestamps
+must be within this claim and deadline; old, unrelated or reused request IDs are rejected. A failed
+case may omit a gateway mapping and must report its failure honestly.
+
+The server computes quality metrics and critical regressions from the ranked IDs and suite labels.
+It stores the consumer provenance and gateway mappings with the existing evaluation runs. Both
+runs, their comparison and the intent completion commit together. An identical result retry does
+not add runs; a changed retry conflicts. `evaluation_summary` exposes comparability, eligibility,
+metric deltas and critical regression count. An eligible evaluation does not authorize replacement;
+`production_promotion_performed` remains false.
+
+## Digest canonicalization
+
+`retrieval_config_sha256` is SHA-256 of the immutable suite revision's `retrieval_settings`, encoded
+as UTF-8 JSON with sorted keys, compact separators and ASCII escaping. The server's exact Python
+equivalent is:
+
+```python
+hashlib.sha256(json.dumps(settings, sort_keys=True, separators=(",", ":"),
+                          default=str, allow_nan=False).encode("utf-8")).hexdigest()
+```
+
+Do not hash pretty-printed JSON or rely on a different serializer's Unicode/number formatting.
+Use the returned `pins_digest` unchanged. The gateway query digest applies the same canonicalization
+to `[case.query]`, so an adapter-added prefix or changed whitespace will not match this contract.
+The consumer revision digest identifies the implementation producing the report; it does not turn
+consumer-reported retrieval rankings into independently measured hub results.
+
+## Failure, restart and cancellation
+
+Pending cancellation is immediately `cancelled`. Running cancellation remains `cancel_requested`
+until the current consumer calls finish with `outcome="cancelled"`. A failed consumer can report
+`outcome="failed"` with an optional bounded uppercase `error_code`. A status change cannot prove
+an external process stopped or GPU memory was reclaimed.
+
+Expiry becomes `interrupted`, fences old tokens and refuses late results. Bounded history/assignment
+reads and new intent admission reconcile expired claims. There is no automatic retry or takeover.
+After an API restart a still-valid claim can continue; an expired claim cannot be revived.
+
+Database restore deliberately interrupts pending, running and cancellation-requested intents,
+clears claim credentials and records `COMPARISON_RESTORE_INTERRUPTED`. It preserves pinned history
+and completed evidence. Restore does not launch work. Review any external process before explicitly
+creating a new intent with a new key. Populated history blocks downgrade; never stamp around that
+guard or rewrite exact revision identities.
+
+The ModelForge protocol and the ExampleRAG operator's local contract are implemented, but this does not
+establish real consumer execution, production promotion or RC-ready qualification. See [ADR 0056](../architecture/adr/0056-selected-model-comparison-intents.md)
+for the architecture boundary and the main release ledger for qualification evidence.
+
+## Deployed schema lineage
+
+The integrated deployment introduces comparison storage in `20260909_0033`, after the frozen deployed creative/watch migrations 0029–0032. The original comparison ADR described a conflicting development-only 0030 lineage. Revision strings alone are not sufficient to identify compatible databases; see [ADR 0057](../architecture/adr/0057-deployed-lineage-convergence.md). Never stamp an alternative lineage onto this graph.
@@ -0,0 +1,63 @@
+# Candidate HTTPS on the canonical hostname
+
+Port 443 is reserved for the existing production stack. A retained, isolated
+candidate uses `https://modelforge.example.com:18444`. The two ports never share
+a proxy route. Before configuring port 443, verify that the existing production
+Console uses the HTTPS API origin and matching CSP through a separately authorized
+production release.
+The candidate's `tls` service belongs to its own Compose project and only joins that
+project's `ingress` network. It sends `/api/*` and `/v1/*` to that project's API and
+everything else to that project's Web service.
+
+Use a valid Nginx Proxy Manager certificate for the canonical hostname. Its key
+must not be mounted directly from NPM or copied into the source tree or image
+build context. For each retained
+candidate, stage only `fullchain.pem` and `privkey.pem` under a root-owned `tls`
+directory inside that candidate's private root but outside its `source` directory.
+After choosing an exact candidate root and certificate directory:
+
+```sh
+candidate_root=/path/to/private/modelforge-validation/EXACT_PROJECT
+npm_live_dir=/path/to/npm/letsencrypt/live/CERTIFICATE_ID
+tls_dir="${candidate_root}/tls"
+install -d -m 0700 "${candidate_root}"
+install -d -m 0700 "${tls_dir}"
+install -m 0600 "${npm_live_dir}/fullchain.pem" "${tls_dir}/fullchain.pem"
+install -m 0600 "${npm_live_dir}/privkey.pem" "${tls_dir}/privkey.pem"
+openssl x509 -in "${tls_dir}/fullchain.pem" -noout -checkend 86400
+openssl x509 -in "${tls_dir}/fullchain.pem" -noout -ext subjectAltName
+test "$(openssl x509 -in "${tls_dir}/fullchain.pem" -pubkey -noout | sha256sum | cut -d' ' -f1)" = \
+     "$(openssl pkey -in "${tls_dir}/privkey.pem" -pubout | sha256sum | cut -d' ' -f1)"
+```
+
+Confirm the SAN contains exactly the intended hostname and the public keys match.
+Set `MODELFORGE_CANDIDATE_TLS_DIR` to this absolute `tls_dir`, and build the Web
+image with `VITE_API_BASE_URL=https://modelforge.example.com:18444`. The API's
+`MODELFORGE_CORS_ORIGINS` must be the same origin. Use the base, production,
+`docker-compose.rc-ingress.yml`, and `docker-compose.validation-tls.yml` overlays
+in that order. The `rc` in the historical overlay filename is not a runtime name.
+The Compose project must use the neutral `modelforge-validation-*` namespace.
+
+Before `up`, render the exact Compose JSON and pass it through
+`scripts/rc_isolation_preflight.py` with `--public-tls-port 18444`, candidate root,
+protected production ports and resource names. The preflight admits this one
+public port only for the isolated `tls` service; all other published ports must
+remain loopback only. Run the observed-resource preflight again after `up`.
+Scan the exact pinned Caddy image ID with the release Trivy severity policy and
+retain its report alongside the four application-image scans. Do not start the
+candidate while the host resource guard is active.
+
+Verify the hostname and certificate without `-k` or `--insecure`:
+
+```sh
+server_ip=SERVER_LAN_IP
+curl --fail --show-error --resolve "modelforge.example.com:18444:${server_ip}" \
+  https://modelforge.example.com:18444/api/v1/health/live
+```
+
+Verify the Console, its browser API requests, CSP and authentication against the
+same origin. Record the exact image IDs, source SHA, certificate expiry, and before
+and after production health. The staged key is per candidate and must be removed
+with its candidate root only after retained evidence and cleanup checks pass.
+If NPM renews the certificate while a candidate remains live, copy the new pair
+and restart only that candidate's `tls` service after checking the SAN and key.
@@ -0,0 +1,3 @@
+# Image provenance
+
+`console-demo.png` and `models-demo.png` were captured from the real ModelForge Vite frontend at 1440 × 900 pixels. A local browser supplied synthetic authentication and two synthetic model records through intercepted API responses; the demo credential was not connected to a backend. The images show the operator console and model library without real projects, models, operators or infrastructure. They do not imply a validated deployment.
@@ -5,8 +5,11 @@ returns an `mfsvc_…` bearer secret once. Only SHA-256, a non-secret prefix, ti
 status, rate/concurrency policy and explicit capability scopes are stored. Secrets are absent from
 audit events and ordinary logs.

+Human credentials use the separate `mfop_` namespace and deployment-owned identity registry described
+in [Operator access](../operations/OPERATOR_ACCESS.md). They never authenticate capability or node
+requests; machine credentials never become human principals or Console sessions.
+
 Gateway authentication rejects missing, malformed, unknown, revoked or expired credentials,
 disabled clients and missing capability scope. Each client has a sliding one-minute request limit
 and a concurrent-request limit. Credential creation and revocation are audited. The frontend keeps
 the one-time value in memory only and never reloads plaintext.
-
@@ -1,102 +1,301 @@
 # Supply-chain release gate

-## What is inventoried
+## Release boundary
+
+RC-DEL-02 is a managed, fail-closed evidence gate. Pull requests, default runs, explicit `full`
+runs and explicit `security` runs all require the complete source phase. Only an explicit
+`workflow_dispatch` with `profile=security` builds and scans the four shipping images; `full` does
+not silently turn into the expensive image profile. Neither profile can pass because a required
+scanner was absent, timed out, lost network access, returned malformed data or silently omitted a
+profile-required component. The job does not deploy or contact a ModelForge production service.
+
+The checkout uses the commit selected by the Gitea run, `fetch-depth: 0`, tag fetching and
+`persist-credentials: false`. An explicit fetch uses full head and tag refspecs.
+`scripts/security_gate.py history` separately proves that Git reports the checkout as non-shallow,
+that `HEAD` equals `GITHUB_SHA`, and that every advertised remote head and tag has the same local
+object. It also records the canonical origin, all-ref and reachable-commit counts, root commits and
+object-reachability proof. A non-shallow single-branch clone is incomplete and fails; checkout
+configuration alone is not proof.
+
+The job has a 120-minute bound. The explicit worst-case step bounds for the image profile total 108
+minutes, including five minutes each for the always-run evaluator and artifact upload, leaving a
+12-minute job-level reserve. A bootstrap `security-gate-result.json` starts in the fail state before
+checkout. Evidence and tools each use an exclusive `mktemp` directory whose prefix includes run ID,
+run attempt and SHA and whose unpredictable suffix prevents stale-path reuse. Both roots must be
+real, runner-owned mode-0700 directories outside the workspace. The evidence location survives
+checkout cleaning and remains uploadable when checkout fails. The evaluator rejects evidence-root,
+directory or file symlinks/reparse points, non-regular files, ownership/mode surprises and paths
+outside the evidence root. On Linux it walks with descriptor-relative `openat`/`O_NOFOLLOW`; on
+Windows it rejects reparse points and compares pre-open, open-handle and post-read identities. It
+validates owner/mode/type on opened handles, reads and hashes the same descriptors, retains one byte
+snapshot, then revalidates ancestry, identity, size and timestamps before every consumption. A
+same-inode mutation or lstat/open swap therefore fails without consuming replacement content.
+Missing status records, timeout exits and interrupted or failed downloads/builds/scans all remain
+blocking evidence.
+
+## Pinned scanner acquisition
+
+No floating scanner action, `latest` download or `curl | sh` installer is used. The runner downloads
+and verifies these exact artifacts before execution:
+
+| Scanner | Exact version/artifact | SHA-256 |
+| --- | --- | --- |
+| pip-audit | `pip_audit-2.10.1-py3-none-any.whl` | `99ef3f600a317c1945f1e89e227ef26e1c2d618429b8bd3fa6f4f7c440c4611a` |
+| Gitleaks | `gitleaks_8.30.1_linux_x64.tar.gz` | `551f6fc83ea457d62a0d98237cbad105af8d557003051f41f3e7ca7b3f2470eb` |
+| Trivy | `trivy_0.74.0_Linux-64bit.tar.gz` | `2ae6fe3ee734b7fdf11335663e18c75ea12dccc76062f09f164a3b0f8be4371a` |
+
+The pip-audit wheel and every dependency in its scanner environment are fully resolved in
+`config/security/pip-audit-toolchain.lock`. Each official PyPI wheel has an exact version and
+SHA-256; acquisition uses `--require-hashes --only-binary=:all:` from `https://pypi.org/simple`, and
+installation is offline from those verified downloads with `--no-index --require-hashes`. The
+evaluator verifies reported runtime versions again. Trivy version/DB evidence is collected in the
+mandatory source phase, so `full` does not depend on image execution.
+
+Both Trivy DB `UpdatedAt` and `DownloadedAt` must be well-formed explicit UTC timestamps, must not
+be future-dated, and `DownloadedAt` cannot predate `UpdatedAt`. The DB update may be at most 48
+hours old and the local download at most 24 hours old at evaluation time. Missing, null, malformed,
+non-UTC, future or stale metadata fails. npm is runner-supplied and its exact observed version is
+retained rather than claimed in advance.
+
+## Required scan matrix
+
+The source phase collects findings without ignore flags or finding-based early exit:
+
+| Target | Evidence |
+| --- | --- |
+| Complete Git history | Gitleaks `git`, `--log-opts=--all`, `--redact=100` |
+| API source | pip-audit of every `[project].dependencies`, optional group and locked build tool |
+| Node Agent source | same, including its exact local API dependency |
+| Runtime Worker source | same, including both `dev` and `inference` groups |
+| Console lock | `npm audit --json` for all dependencies |
+| Console production lock | `npm audit --omit=dev --json` |
+| Repository filesystem | Trivy `vuln,misconfig`, retaining UNKNOWN through CRITICAL |
+
+Python declarations are read with `tomllib`; a hand-maintained requirement subset is not accepted.
+Each PEP 517 backend requirement must have one exact version that matches both gate policy and
+`config/security/build-toolchain.lock`. The lock pins `pip==26.2.1`, `setuptools==84.0.0`,
+`wheel==0.48.0`, `msgpack==1.2.2`, `hatchling==1.32.4`, `editables==0.5` and Hatchling's complete build-time
+dependency closure. Every artifact is installed from PyPI with `--require-hashes --only-binary=:all:`.
+The msgpack hashes cover CPython 3.11-3.14 glibc x86_64/aarch64 wheels and
+CPython 3.12-3.14 Alpine musl x86_64/aarch64 wheels used by managed validation and release images.
+The managed source-validation environment and all Python image builds install this lock before
+packaging projects, then use `--no-build-isolation` so pip cannot resolve a second build
+environment. The gate audits every locked tool as an exact direct package and compares the PEP 517
+requirements against the lock.
+The three first-party distributions are injected into their source inventories, and local
+cross-component pins must resolve to the matching project version. Source reports therefore show
+explicit coverage of `modelforge-api`, `modelforge-node-agent` and
+`modelforge-runtime-worker`, even though a public advisory service may correctly report that a
+private distribution is not present in its index. Those exact locally proven names are separated
+before the public advisory lookup because `pip-audit --strict` correctly rejects a distribution
+that PyPI cannot identify; every other declared direct package remains in the strict audit input.
+Source pip-audit is intentionally allowed to resolve dependencies: every declared direct package
+must appear at its exact manifest version, while every additional resolved transitive package and
+version is retained in `resolved_transitive_packages`. Missing or version-drifted direct packages
+fail. The resolved source report can omit the build-environment tools `packaging`, `pip`,
+`setuptools` and `wheel`. The gate accepts those omissions only for source resolution: all four
+remain hash-locked, and shipping-image inventory still requires `packaging` when present while
+forbidding retained installers. Trivy scans the repository and immutable images. No other runtime
+package receives this exception. The retained pip inputs
+must preserve the original PEP 508 declarations, including extras such as `uvicorn[standard]` and
+`psycopg[binary]`, plus every locked build-tool entry; reconstructing a bare `name==version` line
+for an extra or omitting a lock entry fails. Duplicate project declarations are rejected before
+aggregation and name the exact manifest, TOML source and line for both declarations. Identical as
+well as conflicting duplicate pip result rows are malformed. Source audits never use `--no-deps`
+or `--disable-pip`.
+
+The shipping-image phase builds exactly four non-pushed local tags:

 ```text
-Python packages    read from the built images, not from a manifest
-Node packages      read from the lock file, which is what `npm ci` installs
-base images        pinned per component
-CI actions         pinned by commit SHA
-runtime images     pinned by digest
+modelforge-security-api:${GITHUB_SHA}
+modelforge-security-web:${GITHUB_SHA}
+modelforge-security-node-agent:${GITHUB_SHA}
+modelforge-security-runtime-worker:${GITHUB_SHA}
 ```

-Reading the Python inventory from the image rather than the manifest matters: a manifest states
-intent, an image states what shipped. A build that silently resolved a different transitive version
-is exactly the case an SBOM exists to catch.
+Every build uses `--pull`, an exact SHA tag and exact `org.opencontainers.image.source`, `revision`
+and release `version` labels. Sanitized Docker identity evidence records those labels, the unique
+tag and immutable `sha256:` image ID for all four images. The retained set must contain exactly four
+release identities, exactly one approved SHA tag per image, no mutable or attacker aliases, no
+unexpected digest aliases and no duplicate image IDs. Trivy scans the image ID, not a mutable tag;
+a gate-owned normalizer preserves Trivy's component target separately while binding every retained
+`Results[].Target` and its own `ScannedImageID` to that exact ID. The evaluator cross-checks
+`ArtifactType`, `ArtifactName`, every nested target, `Metadata.ImageID`, repo tags/digests and the
+reference against Docker identity evidence; Trivy's distinct config `ArtifactID` must remain an
+immutable SHA-256 value.

-## SBOM
+All four Dockerfiles declare the canonical HTTPS source URL
+`https://git.example.com/example/modelforge`. The workflow validates both repository identity
+and server URL before building, and the evaluator requires the same canonical URL in each inspected
+image. Python images take a repository-root build context so they can consume the shared hash lock.
+The curated public-source exporter intentionally rewrites that private static label to its existing
+sanitized URL and records it as `sourceRepository` in `PUBLIC_SOURCE_MANIFEST.json`. The manifest
+hashes all four transformed Dockerfiles; both the release builder and source gate require their
+labels to match that manifest identity. Public release provenance records the same sanitized source
+URL alongside the canonical source SHA and manifest digest. Private managed image builds stamp and
+evaluate the exact trusted repository URL resolved by their workflow.

-`scripts/m16_sbom.py` produces CycloneDX 1.5 at `docs/security/sbom/modelforge-cyclonedx.json`.
-It lists components and versions with package URLs, not dependency contents: a release inventory
-must be readable and diffable, so no binaries are stored.
+Python executes inside the exact API, Node Agent and Runtime Worker image IDs to enumerate installed
+distributions through `importlib.metadata`. Each required first-party distribution must equal the
+corresponding source project version. Those complete image inventories are audited exactly with
+`pip-audit --strict --no-deps --disable-pip`; unexpected or missing image packages fail. The final
+Runtime Worker inventory must not contain pip, setuptools, wheel or editables; retaining any
+installer/build backend blocks the gate. The image audit permits only the established
+`setuptools`/`wheel` report omissions, while its exact image inventory and Trivy evidence remain
+mandatory.
+The digest-pinned PyTorch base also exposes exact Conda-management distributions and CUDA-local
+PyTorch wheels that do not exist at those versions on PyPI. Only the enumerated Runtime Worker
+target/name/version tuples are excluded from the PyPI requirements input; every tuple remains
+mandatory in the retained image inventory and the whole immutable image remains subject to Trivy.
+The allowlist is `conda==25.5.0`, `conda-build==25.5.0`,
+`conda-libmamba-solver==25.3.0`, `libmambapy==2.1.1`, `menuinst==2.2.0`,
+`torch==2.7.1+cu128`, `torchaudio==2.7.1+cu128`, and `torchvision==0.22.1+cu128`.
+Absence or version drift fails before vulnerability evaluation.
+All four release images are rebuilt with `--pull --no-cache` for the explicit security profile.
+This prevents a previously cached package-manager upgrade layer from masquerading as a current
+base-OS scan. The release builder uses the same cache-free policy. Runtime images also remove
+installers after their final dependency check.
+The Web build sets `VITE_API_BASE_URL=https://security-scan.invalid`; exact image
+identity records its canonical nginx bundle path. Evidence collection copies that actual path from
+the exact immutable image ID, starts that image, and queries the canonical served build-provenance
+asset generated by Vite. The served bytes must equal the hashed bundle asset. That exact-schema
+asset binds the intentional API origin with SHA-256; the compiled runtime chunk must reference the
+same origin and binding, and both digests are retained inside the whole-bundle digest. Every
+JavaScript chunk, source map, HTML and CSS asset is scanned for direct or
+split/concatenated/template HTTP(S) and WebSocket origins. The pinned React build's W3 namespace
+constants and React error-document origin are separately recorded as exact non-API origins and are
+valid only in their expected namespace/error contexts. Missing provenance, any other origin,
+additional attacker coexistence, dead expected markers, IPv4/IPv6 loopback and localhost-family
+origins all fail. The Web image is static nginx and has no invented Python inventory. Ordinary
+validation uses three separate virtual environments so components cannot inherit one another's
+resolution.

-Each component carries the image it belongs to, so a finding can be traced to what actually ships
-it rather than to the repository as a whole.
+## Secret evidence and historical false positives

-## Image provenance
+Gitleaks 8.30.1 has upstream false-negative regression #2170 and also drops newly added rules when
+they are combined with `extend.useDefault`. The gate therefore scans the same complete history
+twice: built-in default rules run on a temporary config-free worktree, then `.gitleaks.toml` runs
+narrow no-entropy rules for GitHub's documented `ghp_`, `gho_`, `ghu_`, `ghr_`, `github_pat_` and
+both classic and 2026 stateless `ghs_` token formats. The temporary raw reports are merged outside
+retained evidence. Neither configuration path contains a path, commit, or global allowlist.

-`docs/security/sbom/image-provenance.json` binds each built image to its image id, repository
-digest where one exists, creation timestamp and the exact source commit. Provenance is what makes
-the SBOM verifiable rather than merely present: without it, an inventory describes an image nobody
-can identify again.
+Before the repository scan, the exact pinned Linux binary must independently find canonical
+synthetic `ghp_`, `gho_`, `ghu_`, `ghs_`, `ghr_` and `github_pat_` histories through the applicable
+custom rules, plus a canonical AWS key through a built-in/default rule. Both configurations must
+leave a benign control clean. Trigger values are assembled only at test runtime. Traps remove the
+temporary histories and raw reports; scanner, assertion, worktree/prune or `rm` cleanup failure
+blocks the gate.

-## CVE triage
+The repository's raw Gitleaks JSON is temporary and is never uploaded. Before retention, the
+normalizer emits an exact version-1 object and exact finding keys: a constrained rule ID and a
+SHA-256 of Gitleaks' exact fingerprint. Unknown keys and `Secret`, `Match`, `Line`, author, email,
+the un-hashed fingerprint, source path or commit metadata fail or are absent from the artifact.

-Findings are classified, and the classification is evidence rather than a label:
+A real pinned Gitleaks 8.30.1 scan on 2026-08-30 proved that the full non-shallow history contains
+two findings for one old M15 AES unit-test fixture. Rewriting canonical history is not an acceptable
+release operation. The fixture bytes were synthetic test material and never a credential or a
+production backup key, so the two observations are classified `FALSE_POSITIVE` in
+`config/security/gitleaks-false-positives.json` by exact SHA-256 of the Gitleaks fingerprint and
+exact rule ID. The registry contains no path, glob, commit or secret value.

-```text
-EXPLOITABLE           reachable from an untrusted input in this deployment
-NOT_REACHABLE         the vulnerable path is not executed by ModelForge
-DEV_ONLY              present only in a development or test dependency
-FALSE_POSITIVE        the advisory does not apply to this version or configuration
-ACCEPTED_TEMPORARY    accepted with an owner, a reason and a review date
-MUST_FIX              blocks the release
+This is deliberately narrower than a scanner allowlist:
+
+- each classification must match exactly one sanitized finding;
+- a missing, changed, duplicated or additional fingerprint fails;
+- duplicate registry entries and wildcard rule IDs fail;
+- owner, classification date, review date, tracking reference and reason are mandatory;
+- the review window is at most 30 days and a stale review fails;
+- no Gitleaks finding can use the vulnerability accepted-risk mechanism.
+
+Any new leak therefore remains blocking, including another finding from the same rule or file.
+
+## Central decision and accepted risk
+
+Scanners collect facts; only `scripts/security_gate.py evaluate` decides. It rejects missing, empty
+or malformed reports, unexpected scanner exits, network/tool failures, timeouts, shallow history,
+version drift, source/inventory mismatches and every expected-versus-actual target gap. It then
+blocks:
+
+- every unclassified Gitleaks finding;
+- every pip-audit vulnerability;
+- every npm vulnerability;
+- every Trivy UNKNOWN, HIGH or CRITICAL vulnerability or misconfiguration.
+
+LOW and MEDIUM Trivy observations remain in the raw machine-readable reports but are not RC-DEL-02
+blocking severities. No scanner is instructed to omit them.
+
+`config/security/accepted-risks.json` is a versioned object. It starts empty and contains only
+separately reviewed, temporary exceptions. An entry has exactly these fields:
+
+```json
+{
+  "scanner": "trivy",
+  "target": "node-agent",
+  "advisory_id": "CVE-YYYY-NNNN",
+  "package": "package-name",
+  "installed_version": "1.2.3",
+  "classification": "ACCEPTED_TEMPORARY",
+  "owner": "named owner",
+  "accepted_on": "2026-08-30",
+  "expires_on": "2026-09-29",
+  "tracking": "https://issue.example/RC-DEL-02",
+  "reason": "Evidence-backed reason that the no-fix risk is temporarily acceptable."
+}
 ```

-"Zero CVEs" is never claimed without the scan output that supports it, and an aggregate count never
-replaces the per-finding record for a HIGH or CRITICAL.
+The identity is the exact five-value tuple `(scanner, target, advisory_id, package,
+installed_version)`. Classification must be exactly `ACCEPTED_TEMPORARY`; wildcards and extra
+fields fail. Only pip-audit, npm-audit and Trivy
+vulnerabilities with no available fix are eligible. The window is at most 30 calendar dates;
+`expires_on` is the first invalid date. Future,
+expired, duplicate, ambiguous, unused/stale, malformed and fix-available entries all fail the gate.
+An accepted entry that no longer matches a finding is not harmless configuration; it is stale
+policy and blocks the release.

-## Dependency pinning
+For the v1.2.2 candidate, the registry contains 23 exact Trivy tuples for the Node Agent Debian
+Trixie base. Every tuple was reported without an upstream fix by the exact-image run, is owned by
+the ModelForge security owner, expires on 2026-10-02, and is tracked in
+`https://git.example.com/example/modelforge/issues/24`. The current Bookworm alternative had
+additional critical no-fix findings; the tested distroless Debian 13 alternative changes Python,
+UID and runtime compatibility. No fix-available finding is accepted. The next scan must consume
+every tuple exactly; a fixed, changed, missing or newly introduced tuple fails closed.
+The source-only `full` profile validates the registry schema and active dates but defers the
+unused/stale decision for image-targeted Trivy entries, because it produces no image evidence.
+Only the explicit `security` profile may consume those entries, and it fails when any such tuple is
+unused. A passing `full` run therefore never substitutes for the mandatory exact-image run.

-Base images are pinned to a major or minor tag so they continue to receive security patches on a
-compatible line; the runtime worker is pinned by digest because its CUDA and PyTorch versions are
-part of the reproducibility contract for a benchmark. CI actions are pinned by commit SHA.
+Trivy report parsing is strict before risk matching. JSON evidence rejects duplicate object keys;
+the pinned Trivy 0.74.0 filesystem scan must report `ArtifactType=repository`, and its
+`ArtifactName` must equal the resolved checkout path exactly. Every result target must remain
+inside that checkout. Image `ArtifactType`, `ArtifactName` and every result target are likewise
+required and bound to the immutable image identity. Each vulnerability must contain
+non-empty `VulnerabilityID`, `PkgName`, `InstalledVersion` and a valid `Severity`; malformed values
+become schema-policy failures, never fallback identities, and therefore can never be waived.

-No dependency is declared as `latest`. A lock file makes a build reproducible in practice, but a
-floating specifier means the next `npm install` can pull a different major version — or a
-compromised release — without anyone deciding to.
+## Retained result

-## Container posture
+The evaluator always replaces the bootstrap result when it runs and writes:

-```text
-non-root            api, node-agent, web; postgres and redis drop to their own users
-no privileged mode  none
-no Docker socket    none
-capabilities        dropped entirely on api, node-agent and web
-no-new-privileges   api, node-agent, web
-read-only rootfs    node-agent, web; api writes only to its mounted data volumes
-healthchecks        api, postgres, redis, web
-```
+- commit, ref, run ID, profile and evaluation time;
+- exact scanner versions and Trivy database time;
+- every expected operation's exit code;
+- SHA-256 for every retained, manifest-approved report (unknown evidence paths fail before bytes are
+  read or hashed);
+- expected and actual targets by scan class;
+- resolved source transitive packages and sanitized image identities when applicable;
+- blocked findings;
+- applied and unused/invalid false-positive classifications;
+- applied and unused/invalid accepted risks;
+- the final `PASS` or `FAIL` decision.

-## Network boundaries
+The security job uploads its unique retained
+`/tmp/modelforge-security-evidence-${run_id}-${run_attempt}-${sha}.${random}` directory with
+`gitea-upload-artifact@81f940d004763f986ba3582c007fd842dd5cb0d7`, `if: always()`,
+`if-no-files-found: error` and 90-day retention. Its artifact name includes run ID, run attempt and
+source SHA. Upload failure is a failed gate, not an evidence-retention warning.

-```text
-PostgreSQL   loopback only by default
-Redis        loopback only by default
-console      loopback only by default
-API          published deliberately; every admin route is operator-authenticated
-runtime      no public port; reached only through the gateway
-Node Agent   outbound only; no inbound listener
-```
-
-PostgreSQL and Redis defaulting to every interface was the most serious finding of the M16 review:
-the control-plane database, holding provenance, credential hashes and the audit trail, was reachable
-from the LAN behind a development password. Both now bind to loopback, overridable per deployment.
-
-## Secrets
-
-`gitleaks` runs over the full history, not the working tree. A finding is investigated rather than
-suppressed; the single allowlist entry names one exact string and says why it is not a credential.
-
-Generated artefacts are checked too: logs, backup manifests, audit exports and the OpenAPI document
-must contain no operator key, encryption key, node credential or database password.
-
-## Cryptography
-
-Backup encryption is AES-256-GCM from `cryptography`. ModelForge designs no primitive of its own;
-`BackupCipher` is an adapter boundary so a KMS or HSM can replace the local key.
-
-The M16 review verified, and static tests now enforce, that a fresh nonce is drawn inside the chunk
-loop rather than once per file, that each chunk is authenticated with associated data binding it to
-its key id and index, and that a decryption failure removes both the partial file and the
-destination so no readable plaintext survives a wrong key.
+Implementation and synthetic local tests are not release verification. RC-DEL-02 remains
+unverified until both an exact-head managed `security` run (including its retained artifact) and an
+exact-head managed `full` run succeed. A first real runner execution may legitimately be red because
+it sees current dependency/image advisories or runner/network deficiencies; no historical
+vulnerability risk has been seeded to manufacture a green result.