85 lines
5.9 KiB
Markdown
85 lines
5.9 KiB
Markdown
# ADR 0056: Durable selected-model comparison intents
|
||
|
||
Status: accepted for the persisted operator/consumer protocol. Consumer execution integration and
|
||
release qualification remain open.
|
||
|
||
Migration note: the original canonical 0030 implementation below is retained as
|
||
design history. [ADR 0057](0057-deployed-lineage-convergence.md) integrates these
|
||
tables through additive 0033 while preserving the separately deployed lineage.
|
||
|
||
## Context
|
||
|
||
[ADR 0055](0055-selected-model-update-inbox.md) separates an upstream notice from model quality and
|
||
replacement. [ADR 0054](0054-lab-preflight-before-persistent-execution.md) separates admission from durable execution.
|
||
The selected-project hub needs one current-versus-candidate comparison with attributable evidence,
|
||
without introducing a model tournament, another retrieval engine or another promotion engine.
|
||
|
||
## Decision
|
||
|
||
Schema `20260909_0030`, after canonical `20260909_0029`, adds `selected_model_comparisons`.
|
||
An operator creates an intent for one stable project binding, one designated service client,
|
||
one immutable project suite and two distinct exact capability deployments. The current leg must
|
||
be a production `stable` ModelForge deployment; the candidate must be a nonproduction `candidate`
|
||
with an active experiment route for the same `rag.embedding@1` contract. Distinct index references
|
||
prevent silent reuse of the current vector collection. Different embedding spaces are allowed;
|
||
this does not authorize reindex, cutover or replacement.
|
||
|
||
Creation records immutable revision, artifact-set, runtime-profile, embedding-space, configuration,
|
||
suite, corpus and index identities. Every reviewed case must name the requested corpus digest.
|
||
The retrieval configuration digest must match the immutable suite settings. Claim and result
|
||
publication revalidate the selected identities and route. A suite contains 1–100 reviewed cases.
|
||
One active intent per project binding and a unique binding/idempotency key bound admission.
|
||
Identical retries return the existing intent; changing its payload conflicts.
|
||
|
||
The operator designates an existing active project-bound `rag.embedding@1` client. Consumer
|
||
endpoints authenticate its service credential and require that exact client and binding on every
|
||
operation. A claim token alone grants no authority. Operator credentials are not converted into
|
||
service credentials. Assigned discovery is bounded and scoped to the designated client. Suite
|
||
case definitions accompany its successful claim; this protocol adds no general consumer suite API.
|
||
|
||
There is one claim, with a 120-second renewable lease and a fixed one-hour deadline from claim.
|
||
No automatic reclaim or replay occurs. Progress counts are bounded and monotonic. Expired claims
|
||
become `interrupted`; history, assignment reads and subsequent creation reconcile expiration.
|
||
Admission retains its database locks until the decision commits. An explicit new intent is required
|
||
for another attempt, and does not prove an earlier external process stopped.
|
||
|
||
Pending cancellation becomes `cancelled`. Running cancellation becomes `cancel_requested` until
|
||
the designated consumer acknowledges it. Results cannot publish after cancellation, expiry or
|
||
loss of the claim. Expiry and interruption describe coordination state, not GPU reclamation.
|
||
|
||
The consumer submits both legs together with ranked per-case results, its implementation digest
|
||
and mappings from suite case IDs to public gateway `request_id` values. Successful cases require
|
||
unique completed gateway requests for the designated client, exact leg deployment, capability and
|
||
unchanged single-case query hash. Server-owned request timestamps must lie within this claim and
|
||
deadline. Failed cases may omit a request, but cannot make the comparison eligible. Request evidence
|
||
proves the matching embedding invocation; retrieval rankings remain explicitly attributed to the
|
||
designated consumer rather than represented as independently observed hub retrieval.
|
||
|
||
`EvaluationService(commit=False)` creates and completes the existing two evaluation runs and
|
||
comparison inside the locked intent transaction. The server recomputes metrics and eligibility;
|
||
caller aggregate scores are not accepted. Any publication failure rolls back the pair and links.
|
||
An identical result retry returns the existing result, while a changed retry conflicts. The
|
||
response derives a compact outcome from the linked comparison and fails closed on missing evidence.
|
||
Eligibility is evaluation evidence, never production authorization or a promotion operation.
|
||
|
||
## Recovery and boundaries
|
||
|
||
Following [ADR 0041](0041-authoritative-versus-rebuildable-recovery-state.md), pins, operation history
|
||
and linked results are authoritative. Claim hashes and expiry are transient. Restore clears these
|
||
claims and transforms `pending`, `running` and `cancel_requested` intents to `interrupted` with
|
||
`COMPARISON_RESTORE_INTERRUPTED`; it does not replay work. A deterministic fingerprint projection
|
||
permits precisely that transformation while retaining the other provenance. An older schema can
|
||
gain the new table only in its exact empty initial state. The readiness gate refuses residual
|
||
restored claims or active intents. Populated comparison history blocks downgrade.
|
||
|
||
The API does not launch a background consumer, run the ExampleRAG CLI, invoke a provider, claim GPU
|
||
capacity or promote a model. The current ExampleRAG CLI dynamically selects its legacy CURRENT profile
|
||
and does not yet implement this claim/result protocol. Its historical evaluation cannot satisfy a
|
||
fresh intent. A consumer adapter must first validate the actual stable/candidate collection and
|
||
profile identities, execute through the existing retrieval engine, and retain matching public
|
||
gateway request IDs. Real inference, managed PostgreSQL concurrency, upgrade/restore and the frozen
|
||
release artifact set remain qualification gates; this ADR does not assert those gates passed.
|
||
|
||
The actionable protocol is documented in
|
||
[Selected model comparisons](../../operations/SELECTED_MODEL_COMPARISONS.md).
|