Publish curated ITWorx-ModelForge source from 990401737f96c8d46a9f7b0d84994516cadcb6a0

docs/operations/CONFIGURATION_WORKFLOW.md

@@ -0,0 +1,54 @@
+# Managing serving configuration
+
+Open **Configuration** after unlocking the Console. The catalog lists typed values, deployment
+source where provable, sensitivity, validation bounds and whether the value is editable.
+
+Five numeric controls have a live serving consumer: embedding batch size, input character limit,
+gateway request timeout, gateway queue timeout, and telemetry staleness threshold. Other entries
+remain deployment-managed. The separate Scheduler policy workflow remains authoritative for its
+policy values; this editor cannot create a competing scheduler policy.
+
+1. Enter one or more safe overrides. Queue timeout must remain below request timeout. Use
+   **Use deployment value** to remove an existing override.
+2. Choose **Preview changes**. Review before/after values, source changes and impact.
+3. Acknowledge the impact and choose **Apply reviewed changes**. Administrator authority is
+   required; operators may preview and viewers may inspect.
+4. The result names the accepted revision and consumer. The API's serving policy reader uses that
+   snapshot for new requests. Requests that already consumed their configuration retain their
+   snapshot; no worker restart or model promotion is implied.
+
+If another operator applies a change or deployment-resolved values change after preview, apply
+fails with a conflict. Refresh and preview again; the Console does not retry mutations silently.
+
+**Revision history and rollback** retains actor, time, request correlation and complete override
+sets. Restoring an earlier revision or deployment defaults creates another preview followed by
+explicit apply. It does not erase history. Revision zero means the current deployment defaults;
+it does not restore an old environment file.
+
+**Export and import safe overrides** exports JSON with format/product compatibility metadata,
+safe overrides and names of secret slots. It excludes connection strings, credentials and unrelated
+deployment topology. Import replaces the complete override set after a compatibility check and
+review. Slot names are references only; imports never set, rotate or revoke secrets.
+
+Bootstrap and deployment-managed credentials still require the deployment's documented procedure.
+The fixed Hugging Face control-plane slot has a separate administrator-only write-only workflow for
+candidate creation, testing, activation, rotation and revocation; it never reveals or exports a
+token. It does not manage node download credentials. See the [secret lifecycle procedure](SECRET_LIFECYCLE.md).
+Do not paste credentials into the numeric editor or a configuration export.
+
+## Persistence, upgrade and recovery
+
+Schema `20260907_0025` adds `configuration_state` and `configuration_revisions`. Both are
+authoritative PostgreSQL backup state. A validated override, active pointer and attributable audit
+event commit together. Failed audit writes roll back the configuration change. Runtime credentials
+can append revision history but cannot update, delete or truncate it.
+
+The migration leaves existing model, project, credential and audit evidence unchanged and initially
+introduces two empty tables. It supports a mechanical downgrade only while there is no operator
+revision history. After configuration changes, recovery uses the verified pre-migration backup;
+the downgrade guard refuses to destroy the history. Restore compatibility accepts older-schema
+backups only when the added configuration tables are exactly empty, and requires exact retained
+configuration fingerprints for schema-0025 restores.
+
+This source milestone does not create a release or change the deployed 1.2.2 schema. A release
+requires exact-head validation and the separate install/upgrade/rollback qualification.

docs/operations/MODEL_UPDATES.md

@@ -0,0 +1,38 @@
+# Updates for chosen project models
+
+ModelForge follows explicitly selected Hugging Face model references. It compares
+the resolved commit with the exact baseline you selected. A changed commit is an
+opportunity to review, not proof that a model is newer, safer or better.
+
+In Discover, create a watch for a registered model, an exact baseline revision and
+the upstream reference to follow. Checks fetch metadata only. Each different commit
+produces one stored notice; dismissing it preserves history and prevents repeated
+checks from reviving the same notice. Provider errors remain visible beside the last
+check and never imply that the current model is up to date.
+
+In production, enabled watches are checked daily by default. Each API process checks
+at most one due watch per minute, and a database claim prevents processes from repeating
+the same completed scheduled check. There are at most 100 explicit model/reference
+watches. Development and test processes do not start background checks. The Console
+reports the configured monitor mode; individual last-check times and errors show
+what actually happened. You can always request a manual check with operator access.
+
+`MODELFORGE_MODEL_UPDATE_MONITOR_ENABLED=false` disables automatic checks after an
+API restart. `MODELFORGE_MODEL_UPDATE_CHECK_INTERVAL_SECONDS` controls the per-watch
+interval (default 86400; permitted range 3600–604800). Compose passes both settings
+to the API. Pausing one watch stops that reference without affecting model serving.
+
+Use Review candidate to select the exact observed model and SHA in acquisition.
+Review its metadata and impact before explicitly preparing a download. This action
+clears any previous acquisition plan; it cannot approve, download or replace a model.
+Tracking-baseline changes use a separate audited concurrency check and preserve
+old notices. They change what is compared during future checks, not the deployment.
+
+Real project comparison, recommendation and replacement remain separate evidence
+and lifecycle steps. The update inbox cannot certify quality or bypass approval,
+private-worker scheduling, embedding reindexing or rollback requirements.
+
+Schema 0029 stores watches and notices. Upgrade from schema0028 introduces empty
+tables. Once tracking history exists, rollback uses a pre-migration backup rather
+than deleting that history through downgrade. Restore preserves tracking state and
+notice provenance, but clears temporary check claims before allowing READY.

docs/operations/OPERATOR_ACCESS.md

@@ -0,0 +1,108 @@
+# Operator access
+
+Use a separate personal key for each human. Viewer reads protected evidence; operator can additionally
+perform permitted operational actions and previews; administrator controls configuration apply,
+credentials, destructive lifecycle and recovery. The API enforces every route's declared role.
+Project `mfsvc_`, node `mfnode_` and enrollment keys never unlock the Console.
+
+## Deployment-owned personal identities
+
+In a private local terminal run:
+
+```sh
+python scripts/operator_identity.py --subject alice --role viewer --days 30
+```
+
+Save `one_time_key` in your password manager and give it only to that person through your approved
+private channel. Add **only** `deployment_record` to the JSON array in `MODELFORGE_OPERATOR_IDENTITIES`.
+The array must contain 1–100 records. A `.env` assignment uses single quotes around the JSON array.
+Never put the complete command output in `.env`, Git, a ticket, an export or a release artifact.
+The stored record contains a credential ID, subject, role, digest and timezone-aware expiry; it does
+not contain the usable key. Configuration reads redact the entire registry.
+
+Apply the deployment setting using the normal controlled API recreation procedure. On the next
+request the API uses the new registry. Enter the personal key in the Console's Operator API key
+field. Verify the displayed subject and role before acting. `GET /api/v1/operator/identity`, with
+`X-ModelForge-Admin-Token` supplied from your private credential store, provides the same information.
+There is no API for reading a key back.
+
+## Deployment-owned rotation, revocation and recovery
+
+Generate a replacement record with the same subject and role and a new credential ID. Add it alongside
+the old record, recreate the API, and verify the replacement login and role. Remove the old record
+and recreate the API again to end the overlap. The old key then receives 401. To revoke immediately,
+remove the record and recreate the API without an overlap. Record the subject, credential IDs,
+approver, reason and application time in the private deployment change record; never record keys.
+These deployment operations do not currently produce online identity-lifecycle audit events.
+
+Changes require recreation, not just editing the host environment. In-flight operations already
+authorized may finish. The expiry itself is checked on every request without a restart. Retain an
+independent administrator or controlled recovery credential during rotation. Conflicting roles for
+one subject are rejected; change every overlapping record together when changing a role.
+
+The legacy `MODELFORGE_OPERATOR_API_KEY` remains the `legacy-operator` administrator. Once a personal
+administrator is proven, clear that setting and recreate the API to disable shared access. Production
+startup refuses an installation without either a valid bootstrap key or an unexpired named admin.
+Keep recovery material outside the database backup and protect it according to your deployment policy.
+Managed database identities do not replace this independent startup/recovery requirement.
+
+## Managed personal identities
+
+The RC personal-key provider also supports online identity administration. An administrator can
+list identity metadata, create a managed identity, rotate its key, change its role or enabled state,
+and revoke its credentials. Deployment-owned entries remain read-only in this interface. Managed
+subjects use `managed:<slug>` and keys use the distinct `mfopm_` namespace. Existing `mfop_`
+deployment keys retain their authentication and recreation behavior.
+
+Create the first managed identity as an administrator. Choose the minimum necessary role and a
+key lifetime of 1–365 days. Issuance returns the key once; save it in an approved private credential
+store before leaving the page. A lost response cannot be recovered from the database: rotate to
+issue another key. Only a digest is persisted. Never paste the issuance response into a ticket,
+audit reason, repository, configuration export or diagnostic bundle.
+
+Rotation accepts an overlap of 0–900 seconds. Verify the replacement key promptly; zero overlap
+invalidates the predecessor immediately. Revocation requires the exact canonical subject and a
+reason. These mutations use the displayed version, recheck administrator authority, and record an
+attributed audit event in the same transaction. A stale version requires refresh and a new review.
+Once initialized, administration cannot remove the last currently usable managed administrator.
+Expiry can still exhaust access later; maintain the independent deployment recovery authority.
+
+The Console requires at least 120 seconds of overlap when rotating your own key, and at least
+120 seconds remaining before both the current key expiry and local Console lock. Re-unlock first
+if the local deadline is too close; use another administrator or the recovery authority when the
+key itself is near expiry. The replacement is displayed for 60 seconds without a follow-up read
+that could invalidate the issuance response. Save it, then lock and unlock with the replacement
+before the overlap ends. Navigation, form changes and locking clear the displayed key.
+
+The metadata endpoint is `GET /api/v1/operator/managed-identities`. Creation uses POST to that
+collection; rotation and revocation use POST to `/{slug}/rotate` and `/{slug}/revoke`, and role or
+enabled-state changes use PATCH to `/{slug}`. All require administrator authority. Keys never appear
+in metadata, and a managed-key lookup failure cannot fall back to the deployment administrator.
+
+## Managed keys after database recovery
+
+Supported recovery restores the authoritative identity and credential history, then clears managed
+credential activations during reconciliation. Validation independently checks that these current
+activation rows are absent. A restored key is therefore quarantined even if the backup predates
+its later revocation. Do not reopen access to a raw `pg_restore` result: use the complete isolated
+ModelForge recovery workflow and its validation gate.
+
+After validated recovery, authenticate using the independent deployment administrator and issue a
+fresh managed administrator key by rotation or creation. Verify that new key before resuming
+ordinary identity administration. Restored keys remain unusable; preserving their metadata is not
+permission to reactivate them. Keep the restored environment isolated until these checks complete.
+
+## Console locking and limitations
+
+The Console displays identity, role, key expiry and its local lock time. It locks at the earlier of
+fifteen minutes after unlock or key expiry. Lock console and reload both clear the memory-only key.
+Public liveness and version remain visible while locked. A 403 keeps the session and reports a
+forbidden operation; a database/service outage also keeps the key. A 401 clears it. No credential is
+stored in cookies, localStorage, sessionStorage or a URL, and protected API responses are not cached.
+
+The local lock is not a server-issued session or key revocation: a copied key remains usable until
+its expiry or removal. OIDC/Authentik and SSO logout are not implemented; personal header keys are
+the explicitly supported RC provider. Deployment-secret lifecycle is a separate feature.
+Use HTTPS outside an explicitly trusted private lab, and protect the deployment
+registry so an untrusted user cannot replace roles or credential digests. The full RC authentication
+and secret-lifecycle acceptance remains open until those documented decisions and journeys close.

docs/operations/PLATFORM_UPDATE_PREFLIGHT.md

@@ -0,0 +1,11 @@
+# ModelForge platform update preflight
+
+The Console's **Update preflight** view reads `GET /api/v1/admin/updates/preflight`. The endpoint is administrator-only and read-only. It reports the current running API release identity and the evidence available for a proposed target.
+
+The current schema revision in this view is the schema target declared by the running application build. The preflight does not read the database's applied Alembic revision; use the database migration checks when that value is required.
+
+This source line does not provide an approved platform-update manifest. Until one is configured, the approved version, release notes, target schema, schema and breaking impact, backup requirement, and approved-artifact provenance are `UNKNOWN`. Missing current API build provenance fields are also `UNKNOWN`. A preflight with these gaps is `INCOMPLETE`; `UNKNOWN` does not mean that no backup is required.
+
+The current provenance status is `AVAILABLE` only when source commit, build timestamp, and image digest are all supplied by the API build. These values are displayed build metadata, not an independent signature or artifact verification. The approved target's provenance remains `UNKNOWN` without approved release metadata.
+
+The endpoint does not fetch packages, persist a plan, write release state, create or verify backups, or deploy an update. It has no execution operation. An operator must review approved release evidence, impact, and verified backup readiness before planning maintenance; execution remains human-controlled.

docs/operations/PROJECT_ONBOARDING.md

@@ -0,0 +1,61 @@
+# Project onboarding and service credentials
+
+Open **Projects** with a personal operator or administrator key.
+
+1. Create a project with a stable slug, name, description, owner, environment and optional labels.
+2. Add an exact versioned capability requirement. Select reindex support for incompatible embedding
+   spaces, or schema migration support when required. The requirement uses the stable gateway route.
+3. As administrator, issue a service identity for one explicit binding. Choose a unique client name,
+   purpose and expiry. The binding fixes its capability and workload priority.
+4. Save the returned key in the consumer's secret store or a protected token file. It appears once
+   and the Console hides it after 60 seconds. The database retains a digest, never plaintext.
+5. Expand the integration examples. Set the gateway origin, token-file location and capability in
+   the consumer environment. The examples never contain the issued key.
+6. Paste the service key into the memory-only test field and run **Check connectivity**. This verifies
+   its exact scope without starting inference. The key must belong to the selected identity.
+7. Supply non-sensitive capability-specific JSON and run **Run capability check**. This executes one
+   real gateway request and may load the governed deployment. Image/audio templates require real
+   base64 input; placeholder strings cannot produce a successful test.
+
+The workspace shows blockers, deployment health, current key expiry, last use and the successful
+gateway request ID. Readiness requires current key/current deployment evidence from the last 24
+hours. Rotation, a deployment change, expiry, revocation or stale evidence invalidates the relevant
+checks. Production eligibility is deliberately not inferred; use the project-fit and lifecycle
+approval workflows. A fresh installation with no eligible stable deployment stays visibly blocked.
+
+## Rotation and revocation
+
+Choose immediate replacement, 15 minutes, one hour or 24 hours of overlap. Both the expected
+credential ID and the replacement expiry are sent to the API. Refresh after a conflict. The old
+key expires no later than its original expiry or the overlap deadline; another rotation ends any
+older overlap. Replacement keys are also revealed once.
+
+**Revoke all keys** shows the client, capability and last use. Type the exact client name and
+acknowledge the interruption. Current and overlapping keys immediately stop authorizing new
+requests. Already admitted work may finish. A revoked identity cannot be revived through rotation;
+issue a new service identity if access is needed again. Historical metadata remains auditable.
+
+Legacy single-key rotate/delete routes remain available for compatible automation and use immediate
+revocation; they also use the same transactional claim. New integrations should use the guarded
+plural `/credentials/rotate` and `/credentials/revoke` routes.
+
+## API surface
+
+| Route | Authority | Result |
+| --- | --- | --- |
+| `GET /api/v1/project-workspace` and `/{key}` | Viewer | Persisted requirements, safe client metadata, observed readiness |
+| `POST /api/v1/project-workspace` | Operator | Persist project identity/owner/environment/labels with audit |
+| `POST /api/v1/project-workspace/{key}/requirements` | Operator | Bind an exact compatible requirement with audit |
+| `POST /api/v1/admin/service-clients` | Admin | Issue key once; accepts explicit `project_binding_id` |
+| `GET /api/v1/admin/service-clients/{id}/credentials` | Admin | Most recent 100 key metadata records, never digests/plaintext |
+| `POST /api/v1/admin/service-clients/{id}/credentials/rotate` | Admin | Expected credential ID, overlap seconds and new UTC-aware expiry |
+| `POST /api/v1/admin/service-clients/{id}/credentials/revoke` | Admin | Expected credential ID and exact client-name confirmation |
+| `GET /api/v1/capabilities/{capability}/check` | Scoped service key | Connectivity, client identity and correlation ID; no inference |
+
+Connectivity checks exist for the five implemented embedding, reranking, OCR, vision and
+transcription capability routes. Human keys cannot authenticate there, and service keys cannot
+open the Console. Protected control-plane responses use `Cache-Control: no-store`.
+
+Migration `20260908_0026` adds onboarding metadata and credential supersession. Restore the verified
+pre-migration backup if rollback is necessary after new history exists; do not drop that history.
+See [verification](../quality/RC02_PROJECT_VERIFICATION.md) and [clients](../../clients/README.md).

docs/operations/RC_ACCEPTANCE.md

@@ -14,7 +14,7 @@ tree. From that tree it:
 3. scans every exact image ID with checksum-pinned Trivy and rejects ambiguous reports or any
    HIGH/CRITICAL finding;
 4. starts API, Console, PostgreSQL and Redis with the production overlay under a unique
-   `modelforge-rc-*` Compose project;
+   `modelforge-validation-*` Compose project;
 5. uses newly generated secrets, loopback-only ephemeral host ports and project-scoped volumes;
 6. proves liveness, readiness, version identity, Console serving and both sides of the operator
    authentication boundary; and
@@ -24,8 +24,94 @@ tree. From that tree it:
 
 No Node Agent or Runtime Worker is started by this clean-install gate. Consequently it creates no
 compute identity and cannot interact with the production GPU node. A later GPU qualification must
-enrol a disposable `rc-canary` identity with `production_eligible=false` and retain that separate
-evidence.
+enrol a disposable `validation-canary` identity with `production_eligible=false` and retain that
+separate evidence.
+
+## P12 runtime-isolation preflight
+
+Before a P12 Node Agent or Runtime Worker starts, render its exact candidate Compose projection to
+JSON and run `scripts/rc_isolation_preflight.py`. The check requires a bounded
+`modelforge-validation-*` project and refuses external/non-project volumes or networks, host
+networking, global container names, non-loopback or
+protected ports, Docker-socket mounts, writable bind mounts, missing bind sources, source paths
+outside the candidate root, and references to named production resources. A read-only bind targeting
+`/app/config` must contain `models/initial-candidates.yaml` under that exact source path. Run the
+preflight before `docker compose up`. After `up`, run it again with
+`--inspect-docker`; it records only resources carrying the exact Compose project label and rejects
+any name outside that project namespace.
+
+This preflight is an admission control, not P12 qualification evidence. A green result does not
+replace the required fresh secrets, `production_eligible=false` `validation-canary` enrollment, real
+private runtime inference, before/after production readiness, normal lifecycle cleanup, or retained
+runtime evidence.
+
+### Dedicated test ingress for a retained RC stack
+
+The disposable clean-install workflow above removes its stack. For a later retained P12 stack,
+include `docker-compose.rc-ingress.yml` after the base and production files. It places PostgreSQL,
+Redis and the migration job on a project-private `backend` network; Web is only on `ingress`, and
+API joins both. The ingress network is created by the exact `modelforge-validation-*` project, not
+shared with production. Bind all published control-plane ports to `127.0.0.1` and choose unused
+ports; the reverse proxy reaches Web and API over the ingress network, not those host ports.
+
+Port 443 is reserved for the existing production services. Its NPM host is still pending because
+the current production Console embeds an HTTP API origin and CSP; the candidate does not change it.
+The candidate must use a separate
+HTTPS test listener on port 18444 with a certificate valid for `modelforge.example.com`. Its
+canonical browser/API origin is `https://modelforge.example.com:18444`; set both
+`MODELFORGE_CORS_ORIGINS` to that value at runtime and use it for `VITE_API_BASE_URL` when building
+the candidate Console image.
+The test listener routes only to the retained validation project.
+
+Render the complete Compose plan with the exact candidate source, four image identities, freshly
+generated validation secrets and intended public origin. Run `rc_isolation_preflight.py` on that
+JSON before starting any container and again with `--inspect-docker` afterward. Add
+`--public-tls-port 18444` only with `docker-compose.validation-tls.yml`; the preflight admits
+this one port solely for the pinned, confined `tls` service. It must show only project-scoped
+networks and volumes, loopback control-plane ports and the exact source-bound config mount.
+The operator must also verify candidate and production stack health before and after startup.
+
+The isolated `tls` service uses a minimal Caddyfile to route `/api/` and `/v1/` to its API and
+`/` to its Web service on the same private ingress network. It binds only port 18444. NPM remains
+on its own networks and keeps port 443 for production. Stage only a protected copy of the NPM
+certificate/key under the private candidate root, outside the image build context; never mount
+the NPM certificate store into the candidate. Follow
+`docs/operations/VALIDATION_TLS_TEST_PORT.md` for exact staging and HTTPS checks. Validate the
+certificate, Console, API calls and browser CSP without disabling TLS verification.
+
+## Candidate stack retention guard
+
+Use `scripts/rc_stack_retention.py` to review server-side `modelforge-validation-*` Compose
+containers whose temporary source tree may have been removed or reduced after a run. Its default
+mode only inspects containers and prints JSON. It reports the oldest container age (stale after 24 hours by default),
+missing bind sources, and a missing `config/models/initial-candidates.yaml` under any `/app/config`
+bind source. Set `--stale-after-hours` to change the age threshold.
+
+```sh
+python3 scripts/rc_stack_retention.py
+python3 scripts/rc_stack_retention.py --stale-after-hours 48
+```
+
+Stopping is limited to one exact stale or incomplete RC project. Before issuing `docker stop`, the
+operator must provide a nonempty retained evidence record and explicitly attest that no jobs are
+active. A running API, one-off job, migration, agent, runtime worker, or unknown service blocks the
+operation. Only running `postgres`, `redis`, or `web` containers from that inspected project may be
+stopped. The command does not remove containers, volumes, networks, or images; it never targets
+`modelforge` or a project outside the bounded validation namespace.
+
+```sh
+python3 scripts/rc_stack_retention.py \
+  --stop-project modelforge-validation-example \
+  --confirm-stop modelforge-validation-example \
+  --confirm-no-active-jobs \
+  --evidence-record /path/to/retained-rc-evidence.json
+```
+
+The evidence record is a UTF-8 JSON object with exactly these fields:
+`{"schema_version":1,"project":"modelforge-validation-example","retained":true,"evidence_reference":"acceptance-run-id"}`.
+The project must match the selected project, and the evidence reference must identify the retained
+acceptance output or audit evidence. The operator is responsible for confirming that the reference
+points to the expected evidence before the stop command.
 
 After the workflow exists on the default branch, dispatch it directly and enter the exact pushed
 commit. Before merge, dispatch the existing **Managed validation** workflow on the candidate branch

docs/operations/RUNBOOK_NODE_LIFECYCLE.md

@@ -0,0 +1,32 @@
+# Runbook: temporarily enable or disable a compute node
+
+Use the lifecycle action for planned temporary maintenance. Permanent removal belongs in
+`RUNBOOK_NODE_DECOMMISSION.md`; credential loss belongs in `RUNBOOK_NODE_LOSS.md`.
+
+## Disable
+
+1. Drain and stop workloads before starting. The Console's admin-only **Review disable impact**
+   preview checks active deployments, requests, jobs, runtime probes, leases, residencies, pending
+   transitions, storage dependencies and other known node references. A blocker means resolve the
+   dependency and request a fresh preview; there is no force option.
+2. Confirm the preview names the intended persisted node identity. Enter the operator identity, a
+   reason of at least ten characters, and that persisted identity exactly.
+3. The API rechecks the dependency digest while holding the node and dependency locks, then advances
+   the node generation with compare-and-swap. A 409 means the review is stale or a dependency still
+   exists; inspect the new state and review again.
+4. Disabled nodes reject agent heartbeats and cannot claim new agent work. Do not treat the action
+   as a graceful drain; the preview refuses known active workloads instead.
+
+## Enable
+
+1. Resolve a lost/revoked credential first using `RUNBOOK_NODE_LOSS.md`. Remote nodes without an
+   active node credential cannot be enabled.
+2. Use **Review enable impact**, inspect the current state and type the persisted identity with the
+   operator identity and reason.
+3. A successful enable leaves the remote node `offline` and clears old heartbeat/protocol/capability
+   facts. The agent must authenticate and send a fresh compatible heartbeat before health or
+   scheduler readiness can be inferred. The action does not start or restart the agent on its host.
+
+Never change `enabled` through the metadata `PATCH`; the API rejects that field. Use the preview and
+execute lifecycle endpoints so impact, generation and audit checks cannot be skipped. Do not rehearse
+on production nodes.

docs/operations/RUNBOOK_NODE_LOSS.md

@@ -31,9 +31,17 @@ matches and the node reconnects on its own.
 identity file, so it re-enrols onto the same node id with its accelerators, storage roots and
 history intact. Enrolment revokes the previous credential automatically.
 
+If the node's identity volume is still present and an administrator can install a replacement
+credential directly, open the node detail and use **Rotate node credential**. Rotation revokes the
+old key immediately. Replace `/data/state/node-credential` with the one-time value and recreate the
+agent before expecting heartbeats or inventory to resume. The Console cannot install the value on
+the node. If the identity volume is missing, use fresh enrolment instead.
+
 **The credential is believed stolen.** Revoke it first
 (`DELETE /api/v1/admin/hardware/nodes/{node_id}/credential`), then re-enrol. Revocation is
-permanent: the old credential stays refused after re-enrolment.
+permanent: the old credential stays refused after re-enrolment. If a replacement can be installed
+immediately, the node detail's rotation action combines revocation and replacement; the old key is
+still permanently refused.
 
 **The identity file is also lost.** The agent enrols as a new node, because ModelForge genuinely
 cannot tell it apart from different hardware. If the old identity is permanently superseded, use

docs/operations/SCHEDULED_BACKUPS.md

@@ -0,0 +1,45 @@
+# Scheduled backups
+
+The scheduled backup helper creates a recovery backup, verifies it, then applies retention. The
+recovery endpoints require the administrator role. Supply a key for an existing named administrator
+when the deployment no longer uses the legacy global operator key. Named deployment credentials and
+their lifecycle are described in [Operator access](OPERATOR_ACCESS.md).
+
+## Authentication settings
+
+Set `MODELFORGE_BACKUP_ADMIN_API_KEY` to a named administrator key for the scheduler. The key is
+sent only to the ModelForge API in the existing `X-ModelForge-Admin-Token` header. The helper does
+not print it. Keep the value in the deployment's protected environment or `.env` file; do not commit
+it. The API must already recognize the credential as an administrator.
+
+`MODELFORGE_OPERATOR_API_KEY` remains a backward-compatible fallback for existing deployments. If
+both settings are present, `MODELFORGE_BACKUP_ADMIN_API_KEY` takes precedence. If neither has a
+non-empty value, the helper exits before making an API request, so no backup or retention operation
+can be authorized.
+
+For a Compose deployment, add the dedicated key to the protected `.env` file used by Compose:
+
+```dotenv
+MODELFORGE_BACKUP_ADMIN_API_KEY=<existing named administrator key>
+```
+
+Then start or recreate the scheduler with the usual stack files:
+
+```sh
+docker compose -p modelforge --env-file .env \
+  -f docker-compose.yml -f docker-compose.backup.yml \
+  up -d --force-recreate backup-scheduler
+```
+
+For a host timer or Unraid User Script, export `MODELFORGE_BACKUP_ADMIN_API_KEY` in the protected
+job environment before invoking `scripts/modelforge_scheduled_backup.sh`. The legacy variable can
+continue to be used for direct invocations as well.
+
+## Credential lifecycle
+
+Use an administrator identity dedicated to scheduled recovery work. Backup and recovery endpoints
+currently require the administrator role; there is no narrower backup-only role. Keep the credential
+valid and unexpired, and rotate or revoke it through the named identity's supported lifecycle. After
+changing the scheduler's environment, recreate `backup-scheduler`. For deployment-owned named keys,
+apply the API's identity registry change and recreate the API as described in the operator access
+runbook.

docs/operations/SECRET_LIFECYCLE.md

@@ -0,0 +1,57 @@
+# Platform secret lifecycle
+
+Implementation qualification in progress. This procedure describes the P01 control-plane
+Hugging Face slot; it does not certify live provider access or download-node credentials.
+
+## Boundaries
+
+The managed slot supplies Hugging Face authentication for control-plane discovery and repository
+metadata. A successful test confirms that the provider accepts that exact candidate credential;
+it does not establish a repository licence, gated-model approval, artifact safety or node download
+readiness. Download nodes retain their separate deployment-managed credential.
+
+Database bootstrap, migration access, deployment administrator recovery credentials and backup
+encryption remain outside the managed slot. Their metadata must show external ownership and
+unknown versions or test state truthfully. Do not put the root key or bootstrap credentials into
+the slot they would protect.
+
+## Operator workflow
+
+1. Provision and retain the separate encryption root key outside database backups, using the
+   documented deployment settings. Keep its identifier stable for the ciphertext it protects.
+   `MODELFORGE_SECRET_SLOT_ROOT_KEY` contains a base64-encoded random 32-byte AES key;
+   `MODELFORGE_SECRET_SLOT_ROOT_KEY_ID` identifies it. Recreate the API after changing these
+   deployment settings. Historical ciphertext still needs its original key and identifier.
+2. As an administrator, enter the Hugging Face token and an audit reason. Review the current
+   revision. Saving creates an encrypted candidate; it does not replace the active credential.
+   The first save opts into managed mode immediately: metadata/discovery access is blocked until
+   that candidate is tested and activated. Plan that initial switch as a short maintenance action.
+3. Test that candidate. A rejected token, provider outage or stale candidate blocks activation
+   and leaves the existing active credential intact. Review the recorded result and time.
+4. Explicitly activate the successfully tested version. New control-plane requests resolve that
+   version; already-started requests may retain their credential snapshot.
+   The successful test must be less than five minutes old and refer to the latest candidate.
+5. To rotate, repeat candidate creation, testing and activation with the replacement token.
+   Confirm ordinary discovery/metadata access before revoking the old token at the provider.
+6. To disable use, explicitly revoke the managed slot. ModelForge must not silently fall back
+   to a deployment token. Provider-side token revocation is a separate action.
+
+The token cannot be read back. Configuration export contains only the slot reference, never
+ciphertext, plaintext or the root key. Never include tokens in reasons, logs, tickets or reports.
+Use the Console's lock action when finished; token input must be cleared after submission and
+when leaving the workflow.
+
+## Failure and recovery
+
+Missing encryption material, a wrong key or invalid ciphertext makes a managed credential
+unavailable. Restore the externally retained matching root key through the deployment process;
+do not replace it with an unrelated key merely to make startup appear healthy.
+
+A supported database restore preserves encrypted history but invalidates prior test and activation
+evidence. Keep the restored environment isolated. An administrator must test and explicitly
+activate the intended version again. Old provider credentials may already have expired or been
+revoked; database rollback cannot undo that external state. A raw database restore is not proof
+that a secret is safe to reactivate.
+
+Final RC evidence must include live provider qualification and safe root-key provisioning on the
+isolated RC host. Node credential activation and gated downloads require their own evidence.

docs/operations/SELECTED_MODEL_COMPARISONS.md

@@ -0,0 +1,132 @@
+# Selected model comparisons
+
+The project hub can persist one comparison of its current ModelForge embedding deployment with one
+selected candidate. Creating an intent leaves it `pending` until its designated project consumer
+claims the work. The API and Console do not start an evaluator automatically. A dedicated ExampleRAG
+consumer operator implements this protocol on `main` after PR #9 (`64c75be`); its targeted and
+managed full tests are green, but no live consumer qualification is implied.
+
+The existing ExampleRAG `examplerag-modelforge-evaluate` CLI remains outside this protocol: its CURRENT leg
+uses a dynamically selected legacy profile, and its output does not implement these claim and gateway
+evidence requirements. Historical M6 results cannot be imported as fresh execution of a new intent.
+
+## Prepare an intent
+
+Select an active stable `rag.embedding@1` project binding, an active service client belonging to
+that exact binding, the current production stable deployment, and a nonproduction candidate with
+an active experiment route. Prepare a separately identified candidate vector index; matching vector
+dimensions do not authorize reuse of the current index or replace embedding migration.
+
+Select a project-owned immutable suite containing 1–100 reviewed cases. Each case's
+`label_provenance.corpus_revision` must equal the supplied lowercase SHA-256 corpus digest.
+The operator creation payload contains:
+
+- `project_binding_id`, `consumer_client_id`, `suite_revision_id`;
+- `current_deployment_id`, `candidate_deployment_id`, `candidate_route_id`;
+- `corpus_sha256`, `retrieval_config_sha256`;
+- distinct `current_index_ref` and `candidate_index_ref`;
+- a unique `idempotency_key` for this intended attempt.
+
+The server pins the deployment/runtime/artifact/embedding identities and the immutable embedding
+dimension, then checks them again at claim and result publication. A changed target requires a new
+intent. One pending, running or
+cancellation-requested intent can exist per binding. An identical creation retry returns the same
+record; reuse of its key with different input conflicts.
+
+## API and authority
+
+All paths below are relative to `/api/v1/selected-model-comparisons`.
+
+| Method/path | Authority | Purpose |
+| --- | --- | --- |
+| `POST /` | Operator | Persist the exact comparison intent |
+| `GET /` and `GET /{id}` | Viewer | History/detail, including the derived evaluation summary |
+| `POST /{id}/cancel` | Operator | Request cancellation using `expected_version` |
+| `GET /assigned` | Designated project service client | Discover its bounded assigned work |
+| `POST /{id}/claim` | Designated project service client | Obtain the one claim token, pins and suite cases |
+| `POST /{id}/progress` | Same client plus claim token | Renew the lease and report bounded counts |
+| `POST /{id}/result` | Same client plus claim token | Publish both legs atomically |
+| `POST /{id}/finish` | Same client plus claim token | Report failure or acknowledge requested cancellation |
+
+Use the collection path without a trailing slash when making requests. History accepts
+`project_binding_id`, `limit` (1–100, default 50) and `offset` (0–10000). Assignment accepts the same
+limit bound. The service credential must authorize `rag.embedding@1`; another client in the same
+project cannot claim or report this intent. Operator credentials cannot call the consumer protocol.
+
+The claim returns `comparison`, `claim_token`, `claim_expires_at` and `cases`. Retain the token only
+in the consumer's private execution context; do not put it in logs or result artifacts. Normal
+history and audit do not return it. Losing the successful claim response does not create a second
+claim: the original claim expires, after which an operator can create a new attempt explicitly.
+
+## Execute and report
+
+Before any inference, the adapter must validate its actual corpus, index, embedding profiles and
+retrieval settings against the pinned identities. Reuse the consumer's retrieval engine. For every
+successful case, send its exact unchanged query as a single embedding input through the appropriate
+ModelForge stable or candidate route, and retain the gateway's public `request_id`.
+
+Send progress with `claim_token`, `current_completed_cases` and `candidate_completed_cases`.
+The lease lasts 120 seconds; renew before it expires. Counts can only increase and cannot exceed
+the suite count. Renewal never extends the one-hour deadline from initial claim. Stop starting
+additional work when the response reports `cancel_requested`.
+
+Result submission contains `claim_token`, the returned `pins_digest`, a lowercase SHA-256
+`consumer_revision_sha256`, and `current`/`candidate` objects each containing `results`. Every result
+contains `case_id`, bounded `ranked_results` (`chunk_id`, optional `document_id`, finite `score`),
+`latency_ms` and optional `error_code`. Submit exactly one result per suite case for each leg.
+
+`current_gateway_requests` and `candidate_gateway_requests` map each successful case UUID to its
+gateway **public `request_id`**, not the database row `id`. Each mapping value must be unique across
+the pair, belong to the designated client and exact deployment, and reference a completed
+`rag.embedding@1` request. Its single input must hash to the suite query. Server request timestamps
+must be within this claim and deadline; old, unrelated or reused request IDs are rejected. A failed
+case may omit a gateway mapping and must report its failure honestly.
+
+The server computes quality metrics and critical regressions from the ranked IDs and suite labels.
+It stores the consumer provenance and gateway mappings with the existing evaluation runs. Both
+runs, their comparison and the intent completion commit together. An identical result retry does
+not add runs; a changed retry conflicts. `evaluation_summary` exposes comparability, eligibility,
+metric deltas and critical regression count. An eligible evaluation does not authorize replacement;
+`production_promotion_performed` remains false.
+
+## Digest canonicalization
+
+`retrieval_config_sha256` is SHA-256 of the immutable suite revision's `retrieval_settings`, encoded
+as UTF-8 JSON with sorted keys, compact separators and ASCII escaping. The server's exact Python
+equivalent is:
+
+```python
+hashlib.sha256(json.dumps(settings, sort_keys=True, separators=(",", ":"),
+                          default=str, allow_nan=False).encode("utf-8")).hexdigest()
+```
+
+Do not hash pretty-printed JSON or rely on a different serializer's Unicode/number formatting.
+Use the returned `pins_digest` unchanged. The gateway query digest applies the same canonicalization
+to `[case.query]`, so an adapter-added prefix or changed whitespace will not match this contract.
+The consumer revision digest identifies the implementation producing the report; it does not turn
+consumer-reported retrieval rankings into independently measured hub results.
+
+## Failure, restart and cancellation
+
+Pending cancellation is immediately `cancelled`. Running cancellation remains `cancel_requested`
+until the current consumer calls finish with `outcome="cancelled"`. A failed consumer can report
+`outcome="failed"` with an optional bounded uppercase `error_code`. A status change cannot prove
+an external process stopped or GPU memory was reclaimed.
+
+Expiry becomes `interrupted`, fences old tokens and refuses late results. Bounded history/assignment
+reads and new intent admission reconcile expired claims. There is no automatic retry or takeover.
+After an API restart a still-valid claim can continue; an expired claim cannot be revived.
+
+Database restore deliberately interrupts pending, running and cancellation-requested intents,
+clears claim credentials and records `COMPARISON_RESTORE_INTERRUPTED`. It preserves pinned history
+and completed evidence. Restore does not launch work. Review any external process before explicitly
+creating a new intent with a new key. Populated history blocks downgrade; never stamp around that
+guard or rewrite exact revision identities.
+
+The ModelForge protocol and the ExampleRAG operator's local contract are implemented, but this does not
+establish real consumer execution, production promotion or RC-ready qualification. See [ADR 0056](../architecture/adr/0056-selected-model-comparison-intents.md)
+for the architecture boundary and the main release ledger for qualification evidence.
+
+## Deployed schema lineage
+
+The integrated deployment introduces comparison storage in `20260909_0033`, after the frozen deployed creative/watch migrations 0029–0032. The original comparison ADR described a conflicting development-only 0030 lineage. Revision strings alone are not sufficient to identify compatible databases; see [ADR 0057](../architecture/adr/0057-deployed-lineage-convergence.md). Never stamp an alternative lineage onto this graph.

docs/operations/VALIDATION_TLS_TEST_PORT.md

@@ -0,0 +1,63 @@
+# Candidate HTTPS on the canonical hostname
+
+Port 443 is reserved for the existing production stack. A retained, isolated
+candidate uses `https://modelforge.example.com:18444`. The two ports never share
+a proxy route. Before configuring port 443, verify that the existing production
+Console uses the HTTPS API origin and matching CSP through a separately authorized
+production release.
+The candidate's `tls` service belongs to its own Compose project and only joins that
+project's `ingress` network. It sends `/api/*` and `/v1/*` to that project's API and
+everything else to that project's Web service.
+
+Use a valid Nginx Proxy Manager certificate for the canonical hostname. Its key
+must not be mounted directly from NPM or copied into the source tree or image
+build context. For each retained
+candidate, stage only `fullchain.pem` and `privkey.pem` under a root-owned `tls`
+directory inside that candidate's private root but outside its `source` directory.
+After choosing an exact candidate root and certificate directory:
+
+```sh
+candidate_root=/path/to/private/modelforge-validation/EXACT_PROJECT
+npm_live_dir=/path/to/npm/letsencrypt/live/CERTIFICATE_ID
+tls_dir="${candidate_root}/tls"
+install -d -m 0700 "${candidate_root}"
+install -d -m 0700 "${tls_dir}"
+install -m 0600 "${npm_live_dir}/fullchain.pem" "${tls_dir}/fullchain.pem"
+install -m 0600 "${npm_live_dir}/privkey.pem" "${tls_dir}/privkey.pem"
+openssl x509 -in "${tls_dir}/fullchain.pem" -noout -checkend 86400
+openssl x509 -in "${tls_dir}/fullchain.pem" -noout -ext subjectAltName
+test "$(openssl x509 -in "${tls_dir}/fullchain.pem" -pubkey -noout | sha256sum | cut -d' ' -f1)" = \
+     "$(openssl pkey -in "${tls_dir}/privkey.pem" -pubout | sha256sum | cut -d' ' -f1)"
+```
+
+Confirm the SAN contains exactly the intended hostname and the public keys match.
+Set `MODELFORGE_CANDIDATE_TLS_DIR` to this absolute `tls_dir`, and build the Web
+image with `VITE_API_BASE_URL=https://modelforge.example.com:18444`. The API's
+`MODELFORGE_CORS_ORIGINS` must be the same origin. Use the base, production,
+`docker-compose.rc-ingress.yml`, and `docker-compose.validation-tls.yml` overlays
+in that order. The `rc` in the historical overlay filename is not a runtime name.
+The Compose project must use the neutral `modelforge-validation-*` namespace.
+
+Before `up`, render the exact Compose JSON and pass it through
+`scripts/rc_isolation_preflight.py` with `--public-tls-port 18444`, candidate root,
+protected production ports and resource names. The preflight admits this one
+public port only for the isolated `tls` service; all other published ports must
+remain loopback only. Run the observed-resource preflight again after `up`.
+Scan the exact pinned Caddy image ID with the release Trivy severity policy and
+retain its report alongside the four application-image scans. Do not start the
+candidate while the host resource guard is active.
+
+Verify the hostname and certificate without `-k` or `--insecure`:
+
+```sh
+server_ip=SERVER_LAN_IP
+curl --fail --show-error --resolve "modelforge.example.com:18444:${server_ip}" \
+  https://modelforge.example.com:18444/api/v1/health/live
+```
+
+Verify the Console, its browser API requests, CSP and authentication against the
+same origin. Record the exact image IDs, source SHA, certificate expiry, and before
+and after production health. The staged key is per candidate and must be removed
+with its candidate root only after retained evidence and cleanup checks pass.
+If NPM renews the certificate while a candidate remains live, copy the new pair
+and restart only that candidate's `tls` service after checking the SAN and key.