25 lines
1.1 KiB
Markdown
25 lines
1.1 KiB
Markdown
# Ingestion Pipeline
|
|
|
|
## Stages
|
|
|
|
1. **Discover:** connector levert typed source items en cursor.
|
|
2. **Acquire:** begrensde read/download met checksum en MIME-detectie.
|
|
3. **Classify:** allowlist, size, extension/MIME mismatch en threat checks.
|
|
4. **Persist blob:** content-addressed en immutable.
|
|
5. **Parse:** gespecialiseerde parser maakt canonical document.
|
|
6. **Normalize:** encoding, whitespace, metadata en structure paths.
|
|
7. **Chunk:** documenttypebewuste deterministische chunks.
|
|
8. **Embed:** profile-pinned dense/sparse representations.
|
|
9. **Index:** upsert naar shadow/target build.
|
|
10. **Validate:** counts, referential checks, sample retrieval en policychecks.
|
|
11. **Publish:** active version en index visibility atomair coördineren.
|
|
12. **Reconcile:** verwijderingen en orphan cleanup.
|
|
|
|
## Idempotency
|
|
|
|
Iedere stage gebruikt immutable input-ID + stage/profileversion als idempotency key. Retry mag geen dubbele blobs, versions, chunks of points creëren.
|
|
|
|
## Resource limits
|
|
|
|
Maximale file size, archive depth, uncompressed ratio, parser timeout, page count, token count en concurrentie zijn configurabel per source/policy.
|