# Search and indexing

## Goals

Search must find mixed Dutch, English and technical content quickly without leaking
inaccessible titles, snippets, facets or counts.

The first release uses PostgreSQL full-text search and `pg_trgm`. Embeddings and external
search services are out of scope.

## Indexed sources

### Page document

Weighted fields:

- A: title
- B: tags, document type, breadcrumb labels
- C: stripped Markdown headings and body
- D: selected metadata such as course/module and attachment descriptions

### Project document

- name and summary;
- current state;
- status and technologies;
- project updates;
- linked decision summaries;
- linked document titles.

### Attachment metadata

Only safe metadata is indexed:

- original filename;
- description;
- caption;
- detected media type.

Attachment body extraction and OCR are deferred.

## Text normalization

- Store original content unchanged.
- Build a normalized search projection.
- Use PostgreSQL `simple` configuration initially to avoid damaging code, commands,
  product names and bilingual terms through aggressive stemming.
- Normalize Unicode and case for fuzzy title matching.
- Strip Markdown syntax without concatenating unrelated tokens.
- Keep code identifiers searchable.
- Never index hidden secrets or operator-only source paths.

## Query behavior

1. Parse and bound the query.
2. Apply workspace, visibility, trash and permission filters.
3. Generate workspace- and permission-filtered full-text candidates.
4. If exact candidates exist, rank those first and skip workspace-wide fuzzy work; otherwise use the indexed `pg_trgm` title predicate at the measured 0.2 threshold for typo-only searches.
5. Rank with title/exact phrase/recency weights.
6. Generate snippets only from permitted candidates.
7. Return deterministic `(rank, updatedAt, id)` cursor ordering.

Exact title and exact quoted phrase matches outrank fuzzy matches.

This exact-first fallback is intentional: it keeps common-term latency bounded while preserving typo tolerance. The threshold is applied with PostgreSQL `SET LOCAL` in the request transaction; see ADR-020 and the 2026-08-01 production audit.

## Filters

- document type;
- document status;
- collection/root;
- tag;
- project;
- author/editor;
- changed date range;
- content kind;
- stale/review state.

Facets are calculated from the permitted candidate population.

## Index consistency

Publishing a page updates its current search projection transactionally where practical.
Background repair handles:

- parser/version changes;
- project context rollups;
- bulk import;
- failed projection updates;
- full reindex.

Every projection stores a source version/hash. Administrators can see lag and repair it.

## Permission safety

Search authorization is part of SQL selection. It is not acceptable to:

1. retrieve the top global results;
2. filter inaccessible rows in application memory;
3. return fewer results or leaked facets.

Tests must include private pages whose titles and rare terms would reveal their existence.

## Scale verification

Synthetic data must include:

- 100,000 current pages;
- mixed Dutch/English prose;
- code and command blocks;
- 1,000,000 revisions not all included in current-content search;
- tags and project links;
- deliberately inaccessible content;
- typo and phrase queries.

Capture:

- p50/p95/p99 latency;
- query plans;
- index sizes;
- result correctness;
- no-leak assertions.

## Search API safeguards

- query length bound;
- result limit cap;
- cursor expiry/versioning;
- rate limit;
- cancellation/timeout;
- safe highlight tags generated by server or client—not arbitrary HTML;
- generic not-found/forbidden behavior where existence is sensitive.