Publish curated ITWorx-ModelForge source from 990401737f96c8d46a9f7b0d84994516cadcb6a0

docs/operations/SELECTED_MODEL_COMPARISONS.md

@@ -0,0 +1,132 @@
+# Selected model comparisons
+
+The project hub can persist one comparison of its current ModelForge embedding deployment with one
+selected candidate. Creating an intent leaves it `pending` until its designated project consumer
+claims the work. The API and Console do not start an evaluator automatically. A dedicated ExampleRAG
+consumer operator implements this protocol on `main` after PR #9 (`64c75be`); its targeted and
+managed full tests are green, but no live consumer qualification is implied.
+
+The existing ExampleRAG `examplerag-modelforge-evaluate` CLI remains outside this protocol: its CURRENT leg
+uses a dynamically selected legacy profile, and its output does not implement these claim and gateway
+evidence requirements. Historical M6 results cannot be imported as fresh execution of a new intent.
+
+## Prepare an intent
+
+Select an active stable `rag.embedding@1` project binding, an active service client belonging to
+that exact binding, the current production stable deployment, and a nonproduction candidate with
+an active experiment route. Prepare a separately identified candidate vector index; matching vector
+dimensions do not authorize reuse of the current index or replace embedding migration.
+
+Select a project-owned immutable suite containing 1–100 reviewed cases. Each case's
+`label_provenance.corpus_revision` must equal the supplied lowercase SHA-256 corpus digest.
+The operator creation payload contains:
+
+- `project_binding_id`, `consumer_client_id`, `suite_revision_id`;
+- `current_deployment_id`, `candidate_deployment_id`, `candidate_route_id`;
+- `corpus_sha256`, `retrieval_config_sha256`;
+- distinct `current_index_ref` and `candidate_index_ref`;
+- a unique `idempotency_key` for this intended attempt.
+
+The server pins the deployment/runtime/artifact/embedding identities and the immutable embedding
+dimension, then checks them again at claim and result publication. A changed target requires a new
+intent. One pending, running or
+cancellation-requested intent can exist per binding. An identical creation retry returns the same
+record; reuse of its key with different input conflicts.
+
+## API and authority
+
+All paths below are relative to `/api/v1/selected-model-comparisons`.
+
+| Method/path | Authority | Purpose |
+| --- | --- | --- |
+| `POST /` | Operator | Persist the exact comparison intent |
+| `GET /` and `GET /{id}` | Viewer | History/detail, including the derived evaluation summary |
+| `POST /{id}/cancel` | Operator | Request cancellation using `expected_version` |
+| `GET /assigned` | Designated project service client | Discover its bounded assigned work |
+| `POST /{id}/claim` | Designated project service client | Obtain the one claim token, pins and suite cases |
+| `POST /{id}/progress` | Same client plus claim token | Renew the lease and report bounded counts |
+| `POST /{id}/result` | Same client plus claim token | Publish both legs atomically |
+| `POST /{id}/finish` | Same client plus claim token | Report failure or acknowledge requested cancellation |
+
+Use the collection path without a trailing slash when making requests. History accepts
+`project_binding_id`, `limit` (1–100, default 50) and `offset` (0–10000). Assignment accepts the same
+limit bound. The service credential must authorize `rag.embedding@1`; another client in the same
+project cannot claim or report this intent. Operator credentials cannot call the consumer protocol.
+
+The claim returns `comparison`, `claim_token`, `claim_expires_at` and `cases`. Retain the token only
+in the consumer's private execution context; do not put it in logs or result artifacts. Normal
+history and audit do not return it. Losing the successful claim response does not create a second
+claim: the original claim expires, after which an operator can create a new attempt explicitly.
+
+## Execute and report
+
+Before any inference, the adapter must validate its actual corpus, index, embedding profiles and
+retrieval settings against the pinned identities. Reuse the consumer's retrieval engine. For every
+successful case, send its exact unchanged query as a single embedding input through the appropriate
+ModelForge stable or candidate route, and retain the gateway's public `request_id`.
+
+Send progress with `claim_token`, `current_completed_cases` and `candidate_completed_cases`.
+The lease lasts 120 seconds; renew before it expires. Counts can only increase and cannot exceed
+the suite count. Renewal never extends the one-hour deadline from initial claim. Stop starting
+additional work when the response reports `cancel_requested`.
+
+Result submission contains `claim_token`, the returned `pins_digest`, a lowercase SHA-256
+`consumer_revision_sha256`, and `current`/`candidate` objects each containing `results`. Every result
+contains `case_id`, bounded `ranked_results` (`chunk_id`, optional `document_id`, finite `score`),
+`latency_ms` and optional `error_code`. Submit exactly one result per suite case for each leg.
+
+`current_gateway_requests` and `candidate_gateway_requests` map each successful case UUID to its
+gateway **public `request_id`**, not the database row `id`. Each mapping value must be unique across
+the pair, belong to the designated client and exact deployment, and reference a completed
+`rag.embedding@1` request. Its single input must hash to the suite query. Server request timestamps
+must be within this claim and deadline; old, unrelated or reused request IDs are rejected. A failed
+case may omit a gateway mapping and must report its failure honestly.
+
+The server computes quality metrics and critical regressions from the ranked IDs and suite labels.
+It stores the consumer provenance and gateway mappings with the existing evaluation runs. Both
+runs, their comparison and the intent completion commit together. An identical result retry does
+not add runs; a changed retry conflicts. `evaluation_summary` exposes comparability, eligibility,
+metric deltas and critical regression count. An eligible evaluation does not authorize replacement;
+`production_promotion_performed` remains false.
+
+## Digest canonicalization
+
+`retrieval_config_sha256` is SHA-256 of the immutable suite revision's `retrieval_settings`, encoded
+as UTF-8 JSON with sorted keys, compact separators and ASCII escaping. The server's exact Python
+equivalent is:
+
+```python
+hashlib.sha256(json.dumps(settings, sort_keys=True, separators=(",", ":"),
+                          default=str, allow_nan=False).encode("utf-8")).hexdigest()
+```
+
+Do not hash pretty-printed JSON or rely on a different serializer's Unicode/number formatting.
+Use the returned `pins_digest` unchanged. The gateway query digest applies the same canonicalization
+to `[case.query]`, so an adapter-added prefix or changed whitespace will not match this contract.
+The consumer revision digest identifies the implementation producing the report; it does not turn
+consumer-reported retrieval rankings into independently measured hub results.
+
+## Failure, restart and cancellation
+
+Pending cancellation is immediately `cancelled`. Running cancellation remains `cancel_requested`
+until the current consumer calls finish with `outcome="cancelled"`. A failed consumer can report
+`outcome="failed"` with an optional bounded uppercase `error_code`. A status change cannot prove
+an external process stopped or GPU memory was reclaimed.
+
+Expiry becomes `interrupted`, fences old tokens and refuses late results. Bounded history/assignment
+reads and new intent admission reconcile expired claims. There is no automatic retry or takeover.
+After an API restart a still-valid claim can continue; an expired claim cannot be revived.
+
+Database restore deliberately interrupts pending, running and cancellation-requested intents,
+clears claim credentials and records `COMPARISON_RESTORE_INTERRUPTED`. It preserves pinned history
+and completed evidence. Restore does not launch work. Review any external process before explicitly
+creating a new intent with a new key. Populated history blocks downgrade; never stamp around that
+guard or rewrite exact revision identities.
+
+The ModelForge protocol and the ExampleRAG operator's local contract are implemented, but this does not
+establish real consumer execution, production promotion or RC-ready qualification. See [ADR 0056](../architecture/adr/0056-selected-model-comparison-intents.md)
+for the architecture boundary and the main release ledger for qualification evidence.
+
+## Deployed schema lineage
+
+The integrated deployment introduces comparison storage in `20260909_0033`, after the frozen deployed creative/watch migrations 0029–0032. The original comparison ADR described a conflicting development-only 0030 lineage. Revision strings alone are not sufficient to identify compatible databases; see [ADR 0057](../architecture/adr/0057-deployed-lineage-convergence.md). Never stamp an alternative lineage onto this graph.