RAGcore-Public/runbooks/REINDEX.md

Reindex runbook

Purpose

Rebuild the derived Qdrant index from PostgreSQL and immutable blob authority without exposing a partially built collection to readers. PostgreSQL and immutable blobs remain authoritative; Qdrant is rebuildable derived state.

Preconditions

• Confirm the requested collection profile, embedding model digest, dimensions and chunk profile version.
• Confirm the authoritative source is available and the effective knowledge-space policy is unchanged for the build.
• Use the application/worker entry point (IndexBuildService) and its injected QdrantRebuildStore; do not write directly to Qdrant from a route or maintenance shell.
• Confirm the Qdrant API key is supplied through the runtime secret mechanism. Never place it in this file, logs or evidence.

Distributed alias locking

Every build, rollback and reconcile call is serialized per alias by a PostgreSQL-backed lease lock (indexes.index_build_operations), not an in-process lock, so it is safe with multiple app/worker replicas. Acquisition is a bounded, polled retry (default 30s timeout) that raises IndexLockTimeoutError rather than hanging. A lease auto-expires if the holder crashes; the next acquisition attempt reclaims it, tags the stale row needs_repair so the crash stays visible, and proceeds. The same row is the saga log: every meaningful step (created_shadow, streamed_build, validated, smoke_passed, activated, rolled_back, repaired) is appended with a timestamp for audit and diagnosis.

Rebuild and activation

1. Read the authoritative point stream from PostgreSQL and immutable blobs, bounded by both the per-point-count limit and the cumulative per-build byte quota (MAX_BUILD_BYTES); many small points cannot evade the quota, since every batch's actual serialized bytes (all payload fields, sparse vectors and the fingerprint fields) are counted, not just text length.
2. Create the versioned shadow collection rag_<kind>_<profile>_<version> and validate its vector configuration and payload indexes.
3. Upsert bounded batches. The adapter adds an authoritative point fingerprint to each payload; application code does not accept arbitrary Qdrant payload keys.
4. Before activation, validate exact point count, the full deterministic ID/fingerprint digest, the deterministic sample digest and the required golden-query smoke test. Any malformed point discovered during this listing fails the build outright (a shadow collection must be perfectly clean before it can ever be activated).
5. Activate rag_<kind>_<profile>_active atomically only after all checks pass.
6. Keep the previous collection until the retention policy permits retirement.

A failed validation leaves the active alias unchanged. Do not force activation after a failed count, digest, sample or smoke check.

Compensating rollback on failure

Any failure before activation is attempted deletes the orphaned shadow collection (nothing was ever aliased, so nothing is unsafe to reclaim) and records failed_compensated. Any failure during or after the activation call is treated as dangerous even if the call itself raised: the service forces the alias back to the last known-good previous_collection and re-verifies it before drawing any conclusion.

• If that forced restore is confirmed, the operation is failed_compensated and the orphaned shadow is best-effort deleted.
• If the forced restore itself cannot be confirmed, the operation is marked needs_repair and the call raises IndexRepairError instead of a plain build error — this must never be reported as a clean failure, since the live alias state is unconfirmed and requires manual repair.
• If compensation runs but the shadow collection cannot be deleted, the operation is failed_compensation_incomplete (a resource leak, not a correctness risk, since the alias was never pointed at it).

Rollback

Use the recorded previous_collection from the build report. IndexBuildService.rollback deletes the current alias target and recreates the alias for that previous collection through the application port, then verifies the alias target before reporting success. A rollback whose own verification fails raises IndexRepairError and marks the operation needs_repair; it is never reported as succeeded. If there is no previous target, rollback removes the alias and leaves the failed shadow collection available for diagnosis and explicit lifecycle cleanup.

Reconciliation and repair

Run reconciliation against the same authoritative source when monitoring detects divergence. The process compares point IDs and authoritative fingerprints, repairs missing and changed points with bounded upserts, deletes only explicit orphan IDs, and verifies convergence. Investigate repeated divergence before retrying; do not repair by broad collection replacement or unscoped filters.

Listing points during reconciliation never aborts on a single corrupt or legacy point: any point that fails identity, wire-integrity or shape checks is reported as a diagnostic (MalformedIndexPoint) instead of raising.

• A malformed point whose id matches a currently expected authoritative chunk is repairable: it is deterministically re-upserted from the same authoritative source that repairs missing/changed points, using bounded batches.
• A malformed point whose id matches nothing expected (including one whose id could not even be parsed as the deterministic UUID identity, e.g. a legacy integer id) is unrepairable: it is quarantined by deleting it via the exact id Qdrant stored (delete_raw_points), never by a broad filter.
• If any malformed point still remains after a repair pass, reconciliation raises IndexRepairError and marks the operation needs_repair rather than reporting success.
• Repeating reconciliation after a repair is idempotent: already-repaired points are simply valid, and already-quarantined ids are simply absent, on the next pass.

Evidence and operational checks

Record the profile, source snapshot/version, counts, full and sample digests, smoke result, alias transition, rollback/reconciliation result, command exit codes and remaining limitations under artifacts/evidence/<task-id>/. Redact credentials, tokens and document content that is not needed to prove the acceptance criteria.

For a milestone gate, run the canonical commands from docs/10_TESTING/ACCEPTANCE_GATES.md and confirm repository hygiene, dependency readiness and reproducibility from a clean checkout.