Initial curated itworx-knowledge-hub source from 3b1751d0d9dee551aa6994eee0093704dc334817

docs/10-search-and-indexing.md

@@ -0,0 +1,132 @@
+# Search and indexing
+
+## Goals
+
+Search must find mixed Dutch, English and technical content quickly without leaking
+inaccessible titles, snippets, facets or counts.
+
+The first release uses PostgreSQL full-text search and `pg_trgm`. Embeddings and external
+search services are out of scope.
+
+## Indexed sources
+
+### Page document
+
+Weighted fields:
+
+- A: title
+- B: tags, document type, breadcrumb labels
+- C: stripped Markdown headings and body
+- D: selected metadata such as course/module and attachment descriptions
+
+### Project document
+
+- name and summary;
+- current state;
+- status and technologies;
+- project updates;
+- linked decision summaries;
+- linked document titles.
+
+### Attachment metadata
+
+Only safe metadata is indexed:
+
+- original filename;
+- description;
+- caption;
+- detected media type.
+
+Attachment body extraction and OCR are deferred.
+
+## Text normalization
+
+- Store original content unchanged.
+- Build a normalized search projection.
+- Use PostgreSQL `simple` configuration initially to avoid damaging code, commands,
+  product names and bilingual terms through aggressive stemming.
+- Normalize Unicode and case for fuzzy title matching.
+- Strip Markdown syntax without concatenating unrelated tokens.
+- Keep code identifiers searchable.
+- Never index hidden secrets or operator-only source paths.
+
+## Query behavior
+
+1. Parse and bound the query.
+2. Apply workspace, visibility, trash and permission filters.
+3. Generate workspace- and permission-filtered full-text candidates.
+4. If exact candidates exist, rank those first and skip workspace-wide fuzzy work; otherwise use the indexed `pg_trgm` title predicate at the measured 0.2 threshold for typo-only searches.
+5. Rank with title/exact phrase/recency weights.
+6. Generate snippets only from permitted candidates.
+7. Return deterministic `(rank, updatedAt, id)` cursor ordering.
+
+Exact title and exact quoted phrase matches outrank fuzzy matches.
+
+This exact-first fallback is intentional: it keeps common-term latency bounded while preserving typo tolerance. The threshold is applied with PostgreSQL `SET LOCAL` in the request transaction; see ADR-020 and the 2026-08-01 production audit.
+
+## Filters
+
+- document type;
+- document status;
+- collection/root;
+- tag;
+- project;
+- author/editor;
+- changed date range;
+- content kind;
+- stale/review state.
+
+Facets are calculated from the permitted candidate population.
+
+## Index consistency
+
+Publishing a page updates its current search projection transactionally where practical.
+Background repair handles:
+
+- parser/version changes;
+- project context rollups;
+- bulk import;
+- failed projection updates;
+- full reindex.
+
+Every projection stores a source version/hash. Administrators can see lag and repair it.
+
+## Permission safety
+
+Search authorization is part of SQL selection. It is not acceptable to:
+
+1. retrieve the top global results;
+2. filter inaccessible rows in application memory;
+3. return fewer results or leaked facets.
+
+Tests must include private pages whose titles and rare terms would reveal their existence.
+
+## Scale verification
+
+Synthetic data must include:
+
+- 100,000 current pages;
+- mixed Dutch/English prose;
+- code and command blocks;
+- 1,000,000 revisions not all included in current-content search;
+- tags and project links;
+- deliberately inaccessible content;
+- typo and phrase queries.
+
+Capture:
+
+- p50/p95/p99 latency;
+- query plans;
+- index sizes;
+- result correctness;
+- no-leak assertions.
+
+## Search API safeguards
+
+- query length bound;
+- result limit cap;
+- cursor expiry/versioning;
+- rate limit;
+- cancellation/timeout;
+- safe highlight tags generated by server or client—not arbitrary HTML;
+- generic not-found/forbidden behavior where existence is sensitive.