# Ingestion Pipeline

## Stages

1. **Discover:** connector levert typed source items en cursor.
2. **Acquire:** begrensde read/download met checksum en MIME-detectie.
3. **Classify:** allowlist, size, extension/MIME mismatch en threat checks.
4. **Persist blob:** content-addressed en immutable.
5. **Parse:** gespecialiseerde parser maakt canonical document.
6. **Normalize:** encoding, whitespace, metadata en structure paths.
7. **Chunk:** documenttypebewuste deterministische chunks.
8. **Embed:** profile-pinned dense/sparse representations.
9. **Index:** upsert naar shadow/target build.
10. **Validate:** counts, referential checks, sample retrieval en policychecks.
11. **Publish:** active version en index visibility atomair coördineren.
12. **Reconcile:** verwijderingen en orphan cleanup.

## Idempotency

Iedere stage gebruikt immutable input-ID + stage/profileversion als idempotency key. Retry mag geen dubbele blobs, versions, chunks of points creëren.

## Resource limits

Maximale file size, archive depth, uncompressed ratio, parser timeout, page count, token count en concurrentie zijn configurabel per source/policy.