itworx-knowledge-hub-Public/docs/10-search-and-indexing.md

Search and indexing

Goals

Search must find mixed Dutch, English and technical content quickly without leaking inaccessible titles, snippets, facets or counts.

The first release uses PostgreSQL full-text search and pg_trgm. Embeddings and external search services are out of scope.

Indexed sources

Page document

Weighted fields:

• A: title
• B: tags, document type, breadcrumb labels
• C: stripped Markdown headings and body
• D: selected metadata such as course/module and attachment descriptions

Project document

• name and summary;
• current state;
• status and technologies;
• project updates;
• linked decision summaries;
• linked document titles.

Attachment metadata

Only safe metadata is indexed:

• original filename;
• description;
• caption;
• detected media type.

Attachment body extraction and OCR are deferred.

Text normalization

• Store original content unchanged.
• Build a normalized search projection.
• Use PostgreSQL simple configuration initially to avoid damaging code, commands, product names and bilingual terms through aggressive stemming.
• Normalize Unicode and case for fuzzy title matching.
• Strip Markdown syntax without concatenating unrelated tokens.
• Keep code identifiers searchable.
• Never index hidden secrets or operator-only source paths.

Query behavior

1. Parse and bound the query.
2. Apply workspace, visibility, trash and permission filters.
3. Generate workspace- and permission-filtered full-text candidates.
4. If exact candidates exist, rank those first and skip workspace-wide fuzzy work; otherwise use the indexed pg_trgm title predicate at the measured 0.2 threshold for typo-only searches.
5. Rank with title/exact phrase/recency weights.
6. Generate snippets only from permitted candidates.
7. Return deterministic (rank, updatedAt, id) cursor ordering.

Exact title and exact quoted phrase matches outrank fuzzy matches.

This exact-first fallback is intentional: it keeps common-term latency bounded while preserving typo tolerance. The threshold is applied with PostgreSQL SET LOCAL in the request transaction; see ADR-020 and the 2026-08-01 production audit.

Filters

• document type;
• document status;
• collection/root;
• tag;
• project;
• author/editor;
• changed date range;
• content kind;
• stale/review state.

Facets are calculated from the permitted candidate population.

Index consistency

Publishing a page updates its current search projection transactionally where practical. Background repair handles:

• parser/version changes;
• project context rollups;
• bulk import;
• failed projection updates;
• full reindex.

Every projection stores a source version/hash. Administrators can see lag and repair it.

Permission safety

Search authorization is part of SQL selection. It is not acceptable to:

1. retrieve the top global results;
2. filter inaccessible rows in application memory;
3. return fewer results or leaked facets.

Tests must include private pages whose titles and rare terms would reveal their existence.

Scale verification

Synthetic data must include:

• 100,000 current pages;
• mixed Dutch/English prose;
• code and command blocks;
• 1,000,000 revisions not all included in current-content search;
• tags and project links;
• deliberately inaccessible content;
• typo and phrase queries.

Capture:

• p50/p95/p99 latency;
• query plans;
• index sizes;
• result correctness;
• no-leak assertions.

Search API safeguards

• query length bound;
• result limit cap;
• cursor expiry/versioning;
• rate limit;
• cancellation/timeout;
• safe highlight tags generated by server or client—not arbitrary HTML;
• generic not-found/forbidden behavior where existence is sensitive.