Initial curated RAGcore source from c16667f0c9397e2bfc8dd0255b114fba23c8d05e

runbooks/REINDEX.md

@@ -0,0 +1,56 @@
+# Reindex runbook
+
+## Purpose
+
+Rebuild the derived Qdrant index from PostgreSQL and immutable blob authority without exposing a partially built collection to readers. PostgreSQL and immutable blobs remain authoritative; Qdrant is rebuildable derived state.
+
+## Preconditions
+
+- Confirm the requested collection profile, embedding model digest, dimensions and chunk profile version.
+- Confirm the authoritative source is available and the effective knowledge-space policy is unchanged for the build.
+- Use the application/worker entry point (`IndexBuildService`) and its injected `QdrantRebuildStore`; do not write directly to Qdrant from a route or maintenance shell.
+- Confirm the Qdrant API key is supplied through the runtime secret mechanism. Never place it in this file, logs or evidence.
+
+## Distributed alias locking
+
+Every build, rollback and reconcile call is serialized per alias by a PostgreSQL-backed lease lock (`indexes.index_build_operations`), not an in-process lock, so it is safe with multiple app/worker replicas. Acquisition is a bounded, polled retry (default 30s timeout) that raises `IndexLockTimeoutError` rather than hanging. A lease auto-expires if the holder crashes; the next acquisition attempt reclaims it, tags the stale row `needs_repair` so the crash stays visible, and proceeds. The same row is the saga log: every meaningful step (`created_shadow`, `streamed_build`, `validated`, `smoke_passed`, `activated`, `rolled_back`, `repaired`) is appended with a timestamp for audit and diagnosis.
+
+## Rebuild and activation
+
+1. Read the authoritative point stream from PostgreSQL and immutable blobs, bounded by both the per-point-count limit and the cumulative per-build byte quota (`MAX_BUILD_BYTES`); many small points cannot evade the quota, since every batch's actual serialized bytes (all payload fields, sparse vectors and the fingerprint fields) are counted, not just text length.
+2. Create the versioned shadow collection `rag_<kind>_<profile>_<version>` and validate its vector configuration and payload indexes.
+3. Upsert bounded batches. The adapter adds an authoritative point fingerprint to each payload; application code does not accept arbitrary Qdrant payload keys.
+4. Before activation, validate exact point count, the full deterministic ID/fingerprint digest, the deterministic sample digest and the required golden-query smoke test. Any malformed point discovered during this listing fails the build outright (a shadow collection must be perfectly clean before it can ever be activated).
+5. Activate `rag_<kind>_<profile>_active` atomically only after all checks pass.
+6. Keep the previous collection until the retention policy permits retirement.
+
+A failed validation leaves the active alias unchanged. Do not force activation after a failed count, digest, sample or smoke check.
+
+### Compensating rollback on failure
+
+Any failure before activation is attempted deletes the orphaned shadow collection (nothing was ever aliased, so nothing is unsafe to reclaim) and records `failed_compensated`. Any failure during or after the activation call is treated as dangerous even if the call itself raised: the service forces the alias back to the last known-good `previous_collection` and re-verifies it before drawing any conclusion.
+
+- If that forced restore is confirmed, the operation is `failed_compensated` and the orphaned shadow is best-effort deleted.
+- If the forced restore itself cannot be confirmed, the operation is marked `needs_repair` and the call raises `IndexRepairError` instead of a plain build error — this must never be reported as a clean failure, since the live alias state is unconfirmed and requires manual repair.
+- If compensation runs but the shadow collection cannot be deleted, the operation is `failed_compensation_incomplete` (a resource leak, not a correctness risk, since the alias was never pointed at it).
+
+## Rollback
+
+Use the recorded `previous_collection` from the build report. `IndexBuildService.rollback` deletes the current alias target and recreates the alias for that previous collection through the application port, then verifies the alias target before reporting success. A rollback whose own verification fails raises `IndexRepairError` and marks the operation `needs_repair`; it is never reported as succeeded. If there is no previous target, rollback removes the alias and leaves the failed shadow collection available for diagnosis and explicit lifecycle cleanup.
+
+## Reconciliation and repair
+
+Run reconciliation against the same authoritative source when monitoring detects divergence. The process compares point IDs and authoritative fingerprints, repairs missing and changed points with bounded upserts, deletes only explicit orphan IDs, and verifies convergence. Investigate repeated divergence before retrying; do not repair by broad collection replacement or unscoped filters.
+
+Listing points during reconciliation never aborts on a single corrupt or legacy point: any point that fails identity, wire-integrity or shape checks is reported as a diagnostic (`MalformedIndexPoint`) instead of raising.
+
+- A malformed point whose id matches a currently expected authoritative chunk is **repairable**: it is deterministically re-upserted from the same authoritative source that repairs missing/changed points, using bounded batches.
+- A malformed point whose id matches nothing expected (including one whose id could not even be parsed as the deterministic `UUID` identity, e.g. a legacy integer id) is **unrepairable**: it is quarantined by deleting it via the exact id Qdrant stored (`delete_raw_points`), never by a broad filter.
+- If any malformed point still remains after a repair pass, reconciliation raises `IndexRepairError` and marks the operation `needs_repair` rather than reporting success.
+- Repeating reconciliation after a repair is idempotent: already-repaired points are simply valid, and already-quarantined ids are simply absent, on the next pass.
+
+## Evidence and operational checks
+
+Record the profile, source snapshot/version, counts, full and sample digests, smoke result, alias transition, rollback/reconciliation result, command exit codes and remaining limitations under `artifacts/evidence/<task-id>/`. Redact credentials, tokens and document content that is not needed to prove the acceptance criteria.
+
+For a milestone gate, run the canonical commands from `docs/10_TESTING/ACCEPTANCE_GATES.md` and confirm repository hygiene, dependency readiness and reproducibility from a clean checkout.