RAGcore-Public/docs/04_INGESTION/PIPELINE.md

Ingestion Pipeline

Stages

1. Discover: connector levert typed source items en cursor.
2. Acquire: begrensde read/download met checksum en MIME-detectie.
3. Classify: allowlist, size, extension/MIME mismatch en threat checks.
4. Persist blob: content-addressed en immutable.
5. Parse: gespecialiseerde parser maakt canonical document.
6. Normalize: encoding, whitespace, metadata en structure paths.
7. Chunk: documenttypebewuste deterministische chunks.
8. Embed: profile-pinned dense/sparse representations.
9. Index: upsert naar shadow/target build.
10. Validate: counts, referential checks, sample retrieval en policychecks.
11. Publish: active version en index visibility atomair coördineren.
12. Reconcile: verwijderingen en orphan cleanup.

Idempotency

Iedere stage gebruikt immutable input-ID + stage/profileversion als idempotency key. Retry mag geen dubbele blobs, versions, chunks of points creëren.

Resource limits

Maximale file size, archive depth, uncompressed ratio, parser timeout, page count, token count en concurrentie zijn configurabel per source/policy.