35 lines
7.0 KiB
Markdown
35 lines
7.0 KiB
Markdown
# ChatGPT to Knowledge Hub document intake
|
|
|
|
The separate MCP Hub exposes ten focused tools at `/mcp/knowledge-hub/prod`: search and project context; scanned attachment text extraction; workspace roots, children and current page reads; attachment and caller-supplied Gitea text proposals; review submission; and status. `analyze_knowledge_files` and `extract_knowledge_files` advertise `_meta.openai/fileParams: ["files"]`. ContextForge omits connector tool metadata, so the dataplane restores this exact binding only for these two vetted attachment schemas on `knowledge-hub/prod`. The connector scans every source within fixed limits. Extraction returns real text in bounded pages with per-file coverage and an actor-bound authenticated continuation cursor; it has no proposal side effects. A proposal includes the selected create or update publication, and only a human reviewer in Knowledge Hub can approve or publish it.
|
|
|
|
For early attachment validation failures, connector audit may include a fixed `input_reason` label such as `attachment_file_id_invalid`. The label identifies the rejected check only; attachment IDs, URLs, names, and content are never recorded.
|
|
|
|
## Access boundaries
|
|
|
|
- The ChatGPT OAuth identity must be bound in MCP Hub to the exact `knowledge-hub` project and `prod` environment. MCP Hub signs the connector identity assertion and applies the tool policy. A supplied target Knowledge Hub `project_id` must be in `KNOWLEDGE_HUB_ALLOWED_PROJECT_IDS`. Project context always requires this ID. When `project_id` is omitted, search, extraction, proposal creation, review submission and status use the token-scoped workspace only if `KNOWLEDGE_HUB_WORKSPACE_READ_ENABLED=true` and the signed connector identity is exactly `knowledge-hub/prod`. This setting defaults to `false`; an empty target project allowlist is accepted only when workspace access is enabled.
|
|
- Workspace roots and children are cursor paged. Current page Markdown is character paged and carries an ETag; later slices require the same ETag. Select a parent page or workspace root for a create, or compare the complete current page and include its ETag for an update.
|
|
- For Gitea sources, first read the complete file through the separate Gitea MCP. Pass the bounded UTF-8 text and caller-supplied `itworx-git://` URI, ref, commit and full Git blob SHA to `create_text_source_proposal`. The intake connector checks that the supplied text matches that SHA and scans it. `source_complete=true` and repository provenance remain caller assertions; the connector does not fetch a repository or independently verify its contents.
|
|
- A publication Markdown body is limited to 32,000 characters and analysis to 32,000 characters. An attachment or PDF with no extractable text is rejected before a proposal can be created; OCR is outside this path. DOCX extraction reads the main document text and may omit header/footer content, so review those sections separately before submitting a proposal.
|
|
- The connector uses a separate Knowledge Hub service account with exactly `knowledge:search`, `projects:read`, `pages:read`, and `ingestion:write`. It cannot approve or publish proposals. Its identity assertion uses a dedicated `knowledge_hub_ingestion_identity_key` secret mounted only in the dataplane and intake connector, separate from the other connector HMAC key.
|
|
- The Knowledge Hub token is loaded only from `/run/secrets/knowledge_hub_ingestion_token`. Signed attachment download URLs and token values must not appear in logs, audit events, or persisted proposal analysis.
|
|
- An internal retry of a failed proposal is available only with the current ETag, `retryEligible=true`, and a fresh operation key. It is not exposed as an MCP tool or a publication action.
|
|
- The connector and scanner are opt in. The default `INGESTION_CONNECTOR_ENABLED=false` excludes the new project and gateway from reconciliation, leaving existing MCP routes available during staging. The default dataplane does not mount or require the new identity key; the final intake production overlay adds that mount when the intake route is activated.
|
|
|
|
## Production activation
|
|
|
|
The live `wiki.example.com` Authentik proxy requires a narrow unauthenticated-path exception for the connector's API routes. Its proxy provider uses:
|
|
|
|
```text
|
|
^/api/v1/(?:search|projects/[0-9a-fA-F-]{36}/context|pages(?:/roots|/[0-9a-fA-F-]{36}(?:/children)?)|ingestion-proposals(?:/[0-9a-fA-F-]{36}(?:/(?:sources|retry))?)?)$
|
|
```
|
|
|
|
This only passes requests to Knowledge Hub's own bearer authorization. An anonymous JSON request to each allowed route must return 401; `/` and unrelated API routes must continue to redirect to Authentik. Keep the exception aligned with the connector's fixed route allowlist.
|
|
|
|
1. Deploy the Knowledge Hub admin preset that issues the four-scope intake token, including `pages:read`. Create the dedicated token in Beheer and place it in the operator-owned secret directory as `knowledge_hub_ingestion_token` with ownership matching the MCP Hub service UID/GID. Generate a separate random 32-byte-or-longer `knowledge_hub_ingestion_identity_key` in that directory. Keep both secrets out of Git and deployment logs.
|
|
2. Verify a current backup and the pinned images. Set `KNOWLEDGE_HUB_ENDPOINT` to the actual HTTPS Knowledge Hub origin. Set `KNOWLEDGE_HUB_ALLOWED_PROJECT_IDS` to comma-separated UUIDs if project-scoped intake is needed; for workspace-only courses, keep it empty and explicitly set `KNOWLEDGE_HUB_WORKSPACE_READ_ENABLED=true` after reviewing the workspace-wide permission. Pin `CONNECTOR_INGESTION_IMAGE` to an immutable digest. Render Compose with `docker-compose.ingestion.prod.yml` **last**, after `docker-compose.prod.yml` and `docker-compose.autodeploy.yml`; inspect that the final connector image is the digest and `build` is absent before starting it.
|
|
3. Start `ingestion-clamd` and `connector-ingestion` with the `knowledge-intake` Compose profile while `INGESTION_CONNECTOR_ENABLED=false`. Wait for current scanner signatures and connector readiness. Verify the four token scopes, a project context read when project intake is configured, and roots/current-page reads when workspace access is enabled.
|
|
4. Add an enabled, exact OAuth client binding for `knowledge-hub/prod`. Set `INGESTION_CONNECTOR_ENABLED=true` on control API and worker, then restart only those services and verify gateway tool parity, project route, and all ten advertised tool schemas.
|
|
5. Connect the MCP URL in ChatGPT. Exercise search, roots/children and a complete current-page read; attach a benign test PDF and read every extraction page; then create a complete proposal and submit it. Confirm it reaches `ReadyForReview`; approve/publish through the Knowledge Hub UI only. Check that a rejected file leaves no proposal or unscanned persisted source.
|
|
|
|
If staging fails, disable the `knowledge-hub/prod` OAuth client binding, set `INGESTION_CONNECTOR_ENABLED=false`, and reconcile again before stopping the connector. This disables an already created intake virtual server as well. Retain proposal IDs and audit evidence for human review. Do not delete Knowledge Hub data to roll back the connector.
|