133 lines
8.6 KiB
Markdown
133 lines
8.6 KiB
Markdown
# Selected model comparisons
|
||
|
||
The project hub can persist one comparison of its current ModelForge embedding deployment with one
|
||
selected candidate. Creating an intent leaves it `pending` until its designated project consumer
|
||
claims the work. The API and Console do not start an evaluator automatically. A dedicated ExampleRAG
|
||
consumer operator implements this protocol on `main` after PR #9 (`64c75be`); its targeted and
|
||
managed full tests are green, but no live consumer qualification is implied.
|
||
|
||
The existing ExampleRAG `examplerag-modelforge-evaluate` CLI remains outside this protocol: its CURRENT leg
|
||
uses a dynamically selected legacy profile, and its output does not implement these claim and gateway
|
||
evidence requirements. Historical M6 results cannot be imported as fresh execution of a new intent.
|
||
|
||
## Prepare an intent
|
||
|
||
Select an active stable `rag.embedding@1` project binding, an active service client belonging to
|
||
that exact binding, the current production stable deployment, and a nonproduction candidate with
|
||
an active experiment route. Prepare a separately identified candidate vector index; matching vector
|
||
dimensions do not authorize reuse of the current index or replace embedding migration.
|
||
|
||
Select a project-owned immutable suite containing 1–100 reviewed cases. Each case's
|
||
`label_provenance.corpus_revision` must equal the supplied lowercase SHA-256 corpus digest.
|
||
The operator creation payload contains:
|
||
|
||
- `project_binding_id`, `consumer_client_id`, `suite_revision_id`;
|
||
- `current_deployment_id`, `candidate_deployment_id`, `candidate_route_id`;
|
||
- `corpus_sha256`, `retrieval_config_sha256`;
|
||
- distinct `current_index_ref` and `candidate_index_ref`;
|
||
- a unique `idempotency_key` for this intended attempt.
|
||
|
||
The server pins the deployment/runtime/artifact/embedding identities and the immutable embedding
|
||
dimension, then checks them again at claim and result publication. A changed target requires a new
|
||
intent. One pending, running or
|
||
cancellation-requested intent can exist per binding. An identical creation retry returns the same
|
||
record; reuse of its key with different input conflicts.
|
||
|
||
## API and authority
|
||
|
||
All paths below are relative to `/api/v1/selected-model-comparisons`.
|
||
|
||
| Method/path | Authority | Purpose |
|
||
| --- | --- | --- |
|
||
| `POST /` | Operator | Persist the exact comparison intent |
|
||
| `GET /` and `GET /{id}` | Viewer | History/detail, including the derived evaluation summary |
|
||
| `POST /{id}/cancel` | Operator | Request cancellation using `expected_version` |
|
||
| `GET /assigned` | Designated project service client | Discover its bounded assigned work |
|
||
| `POST /{id}/claim` | Designated project service client | Obtain the one claim token, pins and suite cases |
|
||
| `POST /{id}/progress` | Same client plus claim token | Renew the lease and report bounded counts |
|
||
| `POST /{id}/result` | Same client plus claim token | Publish both legs atomically |
|
||
| `POST /{id}/finish` | Same client plus claim token | Report failure or acknowledge requested cancellation |
|
||
|
||
Use the collection path without a trailing slash when making requests. History accepts
|
||
`project_binding_id`, `limit` (1–100, default 50) and `offset` (0–10000). Assignment accepts the same
|
||
limit bound. The service credential must authorize `rag.embedding@1`; another client in the same
|
||
project cannot claim or report this intent. Operator credentials cannot call the consumer protocol.
|
||
|
||
The claim returns `comparison`, `claim_token`, `claim_expires_at` and `cases`. Retain the token only
|
||
in the consumer's private execution context; do not put it in logs or result artifacts. Normal
|
||
history and audit do not return it. Losing the successful claim response does not create a second
|
||
claim: the original claim expires, after which an operator can create a new attempt explicitly.
|
||
|
||
## Execute and report
|
||
|
||
Before any inference, the adapter must validate its actual corpus, index, embedding profiles and
|
||
retrieval settings against the pinned identities. Reuse the consumer's retrieval engine. For every
|
||
successful case, send its exact unchanged query as a single embedding input through the appropriate
|
||
ModelForge stable or candidate route, and retain the gateway's public `request_id`.
|
||
|
||
Send progress with `claim_token`, `current_completed_cases` and `candidate_completed_cases`.
|
||
The lease lasts 120 seconds; renew before it expires. Counts can only increase and cannot exceed
|
||
the suite count. Renewal never extends the one-hour deadline from initial claim. Stop starting
|
||
additional work when the response reports `cancel_requested`.
|
||
|
||
Result submission contains `claim_token`, the returned `pins_digest`, a lowercase SHA-256
|
||
`consumer_revision_sha256`, and `current`/`candidate` objects each containing `results`. Every result
|
||
contains `case_id`, bounded `ranked_results` (`chunk_id`, optional `document_id`, finite `score`),
|
||
`latency_ms` and optional `error_code`. Submit exactly one result per suite case for each leg.
|
||
|
||
`current_gateway_requests` and `candidate_gateway_requests` map each successful case UUID to its
|
||
gateway **public `request_id`**, not the database row `id`. Each mapping value must be unique across
|
||
the pair, belong to the designated client and exact deployment, and reference a completed
|
||
`rag.embedding@1` request. Its single input must hash to the suite query. Server request timestamps
|
||
must be within this claim and deadline; old, unrelated or reused request IDs are rejected. A failed
|
||
case may omit a gateway mapping and must report its failure honestly.
|
||
|
||
The server computes quality metrics and critical regressions from the ranked IDs and suite labels.
|
||
It stores the consumer provenance and gateway mappings with the existing evaluation runs. Both
|
||
runs, their comparison and the intent completion commit together. An identical result retry does
|
||
not add runs; a changed retry conflicts. `evaluation_summary` exposes comparability, eligibility,
|
||
metric deltas and critical regression count. An eligible evaluation does not authorize replacement;
|
||
`production_promotion_performed` remains false.
|
||
|
||
## Digest canonicalization
|
||
|
||
`retrieval_config_sha256` is SHA-256 of the immutable suite revision's `retrieval_settings`, encoded
|
||
as UTF-8 JSON with sorted keys, compact separators and ASCII escaping. The server's exact Python
|
||
equivalent is:
|
||
|
||
```python
|
||
hashlib.sha256(json.dumps(settings, sort_keys=True, separators=(",", ":"),
|
||
default=str, allow_nan=False).encode("utf-8")).hexdigest()
|
||
```
|
||
|
||
Do not hash pretty-printed JSON or rely on a different serializer's Unicode/number formatting.
|
||
Use the returned `pins_digest` unchanged. The gateway query digest applies the same canonicalization
|
||
to `[case.query]`, so an adapter-added prefix or changed whitespace will not match this contract.
|
||
The consumer revision digest identifies the implementation producing the report; it does not turn
|
||
consumer-reported retrieval rankings into independently measured hub results.
|
||
|
||
## Failure, restart and cancellation
|
||
|
||
Pending cancellation is immediately `cancelled`. Running cancellation remains `cancel_requested`
|
||
until the current consumer calls finish with `outcome="cancelled"`. A failed consumer can report
|
||
`outcome="failed"` with an optional bounded uppercase `error_code`. A status change cannot prove
|
||
an external process stopped or GPU memory was reclaimed.
|
||
|
||
Expiry becomes `interrupted`, fences old tokens and refuses late results. Bounded history/assignment
|
||
reads and new intent admission reconcile expired claims. There is no automatic retry or takeover.
|
||
After an API restart a still-valid claim can continue; an expired claim cannot be revived.
|
||
|
||
Database restore deliberately interrupts pending, running and cancellation-requested intents,
|
||
clears claim credentials and records `COMPARISON_RESTORE_INTERRUPTED`. It preserves pinned history
|
||
and completed evidence. Restore does not launch work. Review any external process before explicitly
|
||
creating a new intent with a new key. Populated history blocks downgrade; never stamp around that
|
||
guard or rewrite exact revision identities.
|
||
|
||
The ModelForge protocol and the ExampleRAG operator's local contract are implemented, but this does not
|
||
establish real consumer execution, production promotion or RC-ready qualification. See [ADR 0056](../architecture/adr/0056-selected-model-comparison-intents.md)
|
||
for the architecture boundary and the main release ledger for qualification evidence.
|
||
|
||
## Deployed schema lineage
|
||
|
||
The integrated deployment introduces comparison storage in `20260909_0033`, after the frozen deployed creative/watch migrations 0029–0032. The original comparison ADR described a conflicting development-only 0030 lineage. Revision strings alone are not sufficient to identify compatible databases; see [ADR 0057](../architecture/adr/0057-deployed-lineage-convergence.md). Never stamp an alternative lineage onto this graph.
|