Initial curated RAGcore source from c16667f0c9397e2bfc8dd0255b114fba23c8d05e

docs/04_INGESTION/PIPELINE.md

@@ -0,0 +1,24 @@
+# Ingestion Pipeline
+
+## Stages
+
+1. **Discover:** connector levert typed source items en cursor.
+2. **Acquire:** begrensde read/download met checksum en MIME-detectie.
+3. **Classify:** allowlist, size, extension/MIME mismatch en threat checks.
+4. **Persist blob:** content-addressed en immutable.
+5. **Parse:** gespecialiseerde parser maakt canonical document.
+6. **Normalize:** encoding, whitespace, metadata en structure paths.
+7. **Chunk:** documenttypebewuste deterministische chunks.
+8. **Embed:** profile-pinned dense/sparse representations.
+9. **Index:** upsert naar shadow/target build.
+10. **Validate:** counts, referential checks, sample retrieval en policychecks.
+11. **Publish:** active version en index visibility atomair coördineren.
+12. **Reconcile:** verwijderingen en orphan cleanup.
+
+## Idempotency
+
+Iedere stage gebruikt immutable input-ID + stage/profileversion als idempotency key. Retry mag geen dubbele blobs, versions, chunks of points creëren.
+
+## Resource limits
+
+Maximale file size, archive depth, uncompressed ratio, parser timeout, page count, token count en concurrentie zijn configurabel per source/policy.