133 lines
3.6 KiB
Markdown
133 lines
3.6 KiB
Markdown
# Search and indexing
|
|
|
|
## Goals
|
|
|
|
Search must find mixed Dutch, English and technical content quickly without leaking
|
|
inaccessible titles, snippets, facets or counts.
|
|
|
|
The first release uses PostgreSQL full-text search and `pg_trgm`. Embeddings and external
|
|
search services are out of scope.
|
|
|
|
## Indexed sources
|
|
|
|
### Page document
|
|
|
|
Weighted fields:
|
|
|
|
- A: title
|
|
- B: tags, document type, breadcrumb labels
|
|
- C: stripped Markdown headings and body
|
|
- D: selected metadata such as course/module and attachment descriptions
|
|
|
|
### Project document
|
|
|
|
- name and summary;
|
|
- current state;
|
|
- status and technologies;
|
|
- project updates;
|
|
- linked decision summaries;
|
|
- linked document titles.
|
|
|
|
### Attachment metadata
|
|
|
|
Only safe metadata is indexed:
|
|
|
|
- original filename;
|
|
- description;
|
|
- caption;
|
|
- detected media type.
|
|
|
|
Attachment body extraction and OCR are deferred.
|
|
|
|
## Text normalization
|
|
|
|
- Store original content unchanged.
|
|
- Build a normalized search projection.
|
|
- Use PostgreSQL `simple` configuration initially to avoid damaging code, commands,
|
|
product names and bilingual terms through aggressive stemming.
|
|
- Normalize Unicode and case for fuzzy title matching.
|
|
- Strip Markdown syntax without concatenating unrelated tokens.
|
|
- Keep code identifiers searchable.
|
|
- Never index hidden secrets or operator-only source paths.
|
|
|
|
## Query behavior
|
|
|
|
1. Parse and bound the query.
|
|
2. Apply workspace, visibility, trash and permission filters.
|
|
3. Generate workspace- and permission-filtered full-text candidates.
|
|
4. If exact candidates exist, rank those first and skip workspace-wide fuzzy work; otherwise use the indexed `pg_trgm` title predicate at the measured 0.2 threshold for typo-only searches.
|
|
5. Rank with title/exact phrase/recency weights.
|
|
6. Generate snippets only from permitted candidates.
|
|
7. Return deterministic `(rank, updatedAt, id)` cursor ordering.
|
|
|
|
Exact title and exact quoted phrase matches outrank fuzzy matches.
|
|
|
|
This exact-first fallback is intentional: it keeps common-term latency bounded while preserving typo tolerance. The threshold is applied with PostgreSQL `SET LOCAL` in the request transaction; see ADR-020 and the 2026-08-01 production audit.
|
|
|
|
## Filters
|
|
|
|
- document type;
|
|
- document status;
|
|
- collection/root;
|
|
- tag;
|
|
- project;
|
|
- author/editor;
|
|
- changed date range;
|
|
- content kind;
|
|
- stale/review state.
|
|
|
|
Facets are calculated from the permitted candidate population.
|
|
|
|
## Index consistency
|
|
|
|
Publishing a page updates its current search projection transactionally where practical.
|
|
Background repair handles:
|
|
|
|
- parser/version changes;
|
|
- project context rollups;
|
|
- bulk import;
|
|
- failed projection updates;
|
|
- full reindex.
|
|
|
|
Every projection stores a source version/hash. Administrators can see lag and repair it.
|
|
|
|
## Permission safety
|
|
|
|
Search authorization is part of SQL selection. It is not acceptable to:
|
|
|
|
1. retrieve the top global results;
|
|
2. filter inaccessible rows in application memory;
|
|
3. return fewer results or leaked facets.
|
|
|
|
Tests must include private pages whose titles and rare terms would reveal their existence.
|
|
|
|
## Scale verification
|
|
|
|
Synthetic data must include:
|
|
|
|
- 100,000 current pages;
|
|
- mixed Dutch/English prose;
|
|
- code and command blocks;
|
|
- 1,000,000 revisions not all included in current-content search;
|
|
- tags and project links;
|
|
- deliberately inaccessible content;
|
|
- typo and phrase queries.
|
|
|
|
Capture:
|
|
|
|
- p50/p95/p99 latency;
|
|
- query plans;
|
|
- index sizes;
|
|
- result correctness;
|
|
- no-leak assertions.
|
|
|
|
## Search API safeguards
|
|
|
|
- query length bound;
|
|
- result limit cap;
|
|
- cursor expiry/versioning;
|
|
- rate limit;
|
|
- cancellation/timeout;
|
|
- safe highlight tags generated by server or client—not arbitrary HTML;
|
|
- generic not-found/forbidden behavior where existence is sensitive.
|