ModelForge-Public/docs/operations/SELECTED_MODEL_COMPARISONS.md

Selected model comparisons

The project hub can persist one comparison of its current ModelForge embedding deployment with one selected candidate. Creating an intent leaves it pending until its designated project consumer claims the work. The API and Console do not start an evaluator automatically. A dedicated ExampleRAG consumer operator implements this protocol on main after PR #9 (64c75be); its targeted and managed full tests are green, but no live consumer qualification is implied.

The existing ExampleRAG examplerag-modelforge-evaluate CLI remains outside this protocol: its CURRENT leg uses a dynamically selected legacy profile, and its output does not implement these claim and gateway evidence requirements. Historical M6 results cannot be imported as fresh execution of a new intent.

Prepare an intent

Select an active stable rag.embedding@1 project binding, an active service client belonging to that exact binding, the current production stable deployment, and a nonproduction candidate with an active experiment route. Prepare a separately identified candidate vector index; matching vector dimensions do not authorize reuse of the current index or replace embedding migration.

Select a project-owned immutable suite containing 1–100 reviewed cases. Each case's label_provenance.corpus_revision must equal the supplied lowercase SHA-256 corpus digest. The operator creation payload contains:

• project_binding_id, consumer_client_id, suite_revision_id;
• current_deployment_id, candidate_deployment_id, candidate_route_id;
• corpus_sha256, retrieval_config_sha256;
• distinct current_index_ref and candidate_index_ref;
• a unique idempotency_key for this intended attempt.

The server pins the deployment/runtime/artifact/embedding identities and the immutable embedding dimension, then checks them again at claim and result publication. A changed target requires a new intent. One pending, running or cancellation-requested intent can exist per binding. An identical creation retry returns the same record; reuse of its key with different input conflicts.

API and authority

All paths below are relative to /api/v1/selected-model-comparisons.

Method/path	Authority	Purpose
POST /	Operator	Persist the exact comparison intent
GET / and GET /{id}	Viewer	History/detail, including the derived evaluation summary
POST /{id}/cancel	Operator	Request cancellation using expected_version
GET /assigned	Designated project service client	Discover its bounded assigned work
POST /{id}/claim	Designated project service client	Obtain the one claim token, pins and suite cases
POST /{id}/progress	Same client plus claim token	Renew the lease and report bounded counts
POST /{id}/result	Same client plus claim token	Publish both legs atomically
POST /{id}/finish	Same client plus claim token	Report failure or acknowledge requested cancellation

Use the collection path without a trailing slash when making requests. History accepts project_binding_id, limit (1–100, default 50) and offset (0–10000). Assignment accepts the same limit bound. The service credential must authorize rag.embedding@1; another client in the same project cannot claim or report this intent. Operator credentials cannot call the consumer protocol.

The claim returns comparison, claim_token, claim_expires_at and cases. Retain the token only in the consumer's private execution context; do not put it in logs or result artifacts. Normal history and audit do not return it. Losing the successful claim response does not create a second claim: the original claim expires, after which an operator can create a new attempt explicitly.

Execute and report

Before any inference, the adapter must validate its actual corpus, index, embedding profiles and retrieval settings against the pinned identities. Reuse the consumer's retrieval engine. For every successful case, send its exact unchanged query as a single embedding input through the appropriate ModelForge stable or candidate route, and retain the gateway's public request_id.

Send progress with claim_token, current_completed_cases and candidate_completed_cases. The lease lasts 120 seconds; renew before it expires. Counts can only increase and cannot exceed the suite count. Renewal never extends the one-hour deadline from initial claim. Stop starting additional work when the response reports cancel_requested.

Result submission contains claim_token, the returned pins_digest, a lowercase SHA-256 consumer_revision_sha256, and current/candidate objects each containing results. Every result contains case_id, bounded ranked_results (chunk_id, optional document_id, finite score), latency_ms and optional error_code. Submit exactly one result per suite case for each leg.

current_gateway_requests and candidate_gateway_requests map each successful case UUID to its gateway public request_id, not the database row id. Each mapping value must be unique across the pair, belong to the designated client and exact deployment, and reference a completed rag.embedding@1 request. Its single input must hash to the suite query. Server request timestamps must be within this claim and deadline; old, unrelated or reused request IDs are rejected. A failed case may omit a gateway mapping and must report its failure honestly.

The server computes quality metrics and critical regressions from the ranked IDs and suite labels. It stores the consumer provenance and gateway mappings with the existing evaluation runs. Both runs, their comparison and the intent completion commit together. An identical result retry does not add runs; a changed retry conflicts. evaluation_summary exposes comparability, eligibility, metric deltas and critical regression count. An eligible evaluation does not authorize replacement; production_promotion_performed remains false.

Digest canonicalization

retrieval_config_sha256 is SHA-256 of the immutable suite revision's retrieval_settings, encoded as UTF-8 JSON with sorted keys, compact separators and ASCII escaping. The server's exact Python equivalent is:

hashlib.sha256(json.dumps(settings, sort_keys=True, separators=(",", ":"),
                          default=str, allow_nan=False).encode("utf-8")).hexdigest()

Do not hash pretty-printed JSON or rely on a different serializer's Unicode/number formatting. Use the returned pins_digest unchanged. The gateway query digest applies the same canonicalization to [case.query], so an adapter-added prefix or changed whitespace will not match this contract. The consumer revision digest identifies the implementation producing the report; it does not turn consumer-reported retrieval rankings into independently measured hub results.

Failure, restart and cancellation

Pending cancellation is immediately cancelled. Running cancellation remains cancel_requested until the current consumer calls finish with outcome="cancelled". A failed consumer can report outcome="failed" with an optional bounded uppercase error_code. A status change cannot prove an external process stopped or GPU memory was reclaimed.

Expiry becomes interrupted, fences old tokens and refuses late results. Bounded history/assignment reads and new intent admission reconcile expired claims. There is no automatic retry or takeover. After an API restart a still-valid claim can continue; an expired claim cannot be revived.

Database restore deliberately interrupts pending, running and cancellation-requested intents, clears claim credentials and records COMPARISON_RESTORE_INTERRUPTED. It preserves pinned history and completed evidence. Restore does not launch work. Review any external process before explicitly creating a new intent with a new key. Populated history blocks downgrade; never stamp around that guard or rewrite exact revision identities.

The ModelForge protocol and the ExampleRAG operator's local contract are implemented, but this does not establish real consumer execution, production promotion or RC-ready qualification. See ADR 0056 for the architecture boundary and the main release ledger for qualification evidence.

Deployed schema lineage

The integrated deployment introduces comparison storage in 20260909_0033, after the frozen deployed creative/watch migrations 0029–0032. The original comparison ADR described a conflicting development-only 0030 lineage. Revision strings alone are not sufficient to identify compatible databases; see ADR 0057. Never stamp an alternative lineage onto this graph.