Jens/ITWorx_MCP_Hub-Public
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial curated ITWorx_MCP_Hub source from 368fba691ea8d23403b4d56b53b5ec670d7c5020

Browse Source
This commit is contained in:
Public Source Publisher committed 2026-10-01 01:36:21 +02:00
commit cb5be898ea
663 files changed
+82674
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/README.md
+72
View File
@@ -0,0 +1,72 @@
# Bootstrap and state scripts
All scripts in this directory use only the Python standard library so they work before the project toolchain is installed.
```bash
python scripts/validate_pack.py
python scripts/next_packet.py
python scripts/render_luna_context.py
python scripts/update_state.py WP-000 in_progress
python scripts/create_manifest.py
python scripts/promotion_preflight.py --env-file <operator-owned-production.env> --phase predeploy --verify-registry
```
`validate_pack.py` checks packet dependencies/references, catalog identities and cross-references, contract JSON syntax, forbidden generic tool patterns, obvious secrets, and an optional checksum manifest.
`render_luna_context.py` intentionally fails rather than silently truncating authoritative files. Narrow a packet's `read` list if a future context bundle becomes too large.
`update_state.py` enforces ordinary packet transitions and synchronizes a small machine-state section in `CURRENT_STATE.md`. Significant implementation detail still belongs in the packet evidence and human-readable state sections.
`promotion_preflight.py` is a read-only, fail-closed production gate. It checks
operator-supplied paths, secret-file mode/ownership, immutable Compose image
references, Docker networks, DNS expectations, and TCP reachability without
printing secret values. It must be run with an external production env file;
the repository `.env.example` is intentionally not sufficient.
`verify_demo_token_runtime.py` is a second, live-Docker read-only gate for the
Fleet Ops demo token. Run it on Tower immediately before minting or revoking:
```sh
python scripts/verify_demo_token_runtime.py \
--router-container itworx-mcp-hub-dataplane-router-1 \
--expected-host-file /srv/example/appdata/itworx-mcp-hub/config/demo-token-revocations.json
```
It rejects a missing env-file value, a repository example fallback, a wrong
mount source, a writable mount, or any file mode other than `0600`.
`verify_demo_token_flow.py` is the live functional proof. It mints a one-hour
demo token in memory, calls the fixed Fleet Ops tool, atomically revokes that
token even when the valid call fails, then requires a `401` for the revoked
call. It prints only HTTP/content metadata, never the token or response body:
```sh
python -m scripts.verify_demo_token_flow \
--endpoint https://mcp.example.test/mcp/mobilityops/prod \
--key-file /srv/example/appdata/itworx-mcp-hub/secrets/demo_token_signing_key \
--revocations-file /srv/example/appdata/itworx-mcp-hub/config/demo-token-revocations.json
```
On Tower, use the optional container profile instead of the host Python
runtime (which is intentionally not a project dependency environment):
```sh
docker compose --env-file /srv/example/appdata/itworx-mcp-hub/config/production.env.pending \
-f infrastructure/compose/docker-compose.yml \
-f infrastructure/compose/docker-compose.prod.yml \
run --rm --no-deps operator-tools \
--endpoint http://dataplane-router:8080/mcp/mobilityops/prod \
--key-file /run/secrets/demo_token_signing_key \
--revocations-file /etc/itworx/demo-token-revocations.json
```
## Codex machine clients
The production Codex path is documented in
`docs/04-mcp/CODEX_INTEGRATION.md`. Its bounded operator tools are:
- `new_codex_credential.ps1`: create a CSPRNG credential in a current-user-only file;
- `authentik_provision_mcp_service_account.py`: idempotent Authentik shell payload;
- `provision_machine_client.py`: activate the exact Authentik subject and project binding;
- `install_codex_mcp.ps1`: install the secret-free Codex stdio configuration;
- `mcp_stdio_bridge.py`: retrieve/refresh M2M tokens and forward only allowlisted MCP methods.
scripts/audit_projectbrain_contracts.py
+65
View File
@@ -0,0 +1,65 @@
"""Compare a captured ProjectBrain tools/list with the reviewed Hub catalog.
Read-only: this never imports tools or edits catalog policy. Capture discovery
from the owned connector and pass the JSON document using --discovery-file.
"""
from __future__ import annotations
import argparse
import json
from pathlib import Path
from typing import Any
ROOT = Path(__file__).resolve().parents[1]
def audit(discovery: dict[str, Any], catalog_root: Path) -> dict[str, Any]:
tools = discovery.get("result", discovery).get("tools")
if not isinstance(tools, list) or not tools:
raise ValueError("non-empty tools/list result required")
catalog = {}
for path in sorted((catalog_root / "tools").glob("projectbrain-*.yaml")):
spec = json.loads(path.read_text(encoding="utf-8"))["spec"]
catalog[spec["canonicalName"]] = spec["inputSchema"]
issues = []
seen = set()
for tool in tools:
name = tool["name"]
if name in seen:
issues.append({"tool": name, "code": "duplicate_tool"})
seen.add(name)
if name not in catalog:
issues.append({"tool": name, "code": "unreviewed_tool"})
continue
upstream = tool["inputSchema"]
local = catalog[name]
for field in sorted(set(upstream.get("properties", {})) - set(local.get("properties", {}))):
issues.append({"tool": name, "code": "missing_field", "field": field})
if set(upstream.get("required", [])) != set(local.get("required", [])):
issues.append({"tool": name, "code": "required_fields_differ"})
for field, value in upstream.get("properties", {}).items():
target = local.get("properties", {}).get(field, {})
if "type" in value and "type" in target and value["type"] != target["type"]:
issues.append({"tool": name, "code": "field_type_differs", "field": field})
if "enum" in value and "enum" in target and value["enum"] != target["enum"]:
issues.append({"tool": name, "code": "field_enum_differs", "field": field})
for name in sorted(set(catalog) - seen):
issues.append({"tool": name, "code": "upstream_tool_missing"})
return {"passed": not issues, "toolCount": len(tools), "issues": issues}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--discovery-file", type=Path, required=True)
parser.add_argument("--catalog-root", type=Path, default=ROOT / "catalog")
args = parser.parse_args()
if args.discovery_file.stat().st_size > 2 * 1024 * 1024:
parser.error("discovery exceeds 2 MiB")
result = audit(json.loads(args.discovery_file.read_text(encoding="utf-8")), args.catalog_root)
print(json.dumps(result, sort_keys=True))
return 0 if result["passed"] else 1
if __name__ == "__main__":
raise SystemExit(main())
scripts/authentik_provision_mcp_service_account.py
+82
View File
@@ -0,0 +1,82 @@
"""Authentik-shell payload for an idempotent MCP service account and app password.
Run by piping this file to ``ak shell`` with MCP_USERNAME, MCP_DISPLAY_NAME,
MCP_TOKEN_IDENTIFIER and MCP_SECRET_FILE set inside the container. The secret
is read from a temporary file and is never printed.
"""
from __future__ import annotations
import json
import os
from pathlib import Path
from authentik.core.models import Token, TokenIntents, User, UserTypes
from django.db import transaction
def required(name: str) -> str:
value = os.environ.get(name, "").strip()
if not value:
raise ValueError(f"missing {name}")
return value
username = required("MCP_USERNAME")
display_name = required("MCP_DISPLAY_NAME")
identifier = required("MCP_TOKEN_IDENTIFIER")
secret_path = Path(required("MCP_SECRET_FILE"))
if secret_path.is_symlink() or not secret_path.is_file() or secret_path.stat().st_size > 8192:
raise ValueError("invalid app-password secret file")
secret = secret_path.read_text(encoding="utf-8").strip()
if not secret or "\n" in secret or "\r" in secret:
raise ValueError("app-password secret must contain one non-empty line")
with transaction.atomic():
user, created = User.objects.get_or_create(
username=username,
defaults={
"name": display_name,
"path": "users",
"type": UserTypes.SERVICE_ACCOUNT,
},
)
if not created and user.type not in {
UserTypes.SERVICE_ACCOUNT,
UserTypes.INTERNAL_SERVICE_ACCOUNT,
}:
raise ValueError("existing Authentik identity is not a service account")
if user.name != display_name:
user.name = display_name
user.save(update_fields=["name"])
token, token_created = Token.objects.get_or_create(
identifier=identifier,
defaults={
"user": user,
"intent": TokenIntents.INTENT_APP_PASSWORD,
"key": secret,
"expiring": False,
"description": "ITWorx MCP Hub Codex machine credential",
},
)
if not token_created:
if token.user_id != user.pk or token.intent != TokenIntents.INTENT_APP_PASSWORD:
raise ValueError("existing Authentik token identifier has incompatible ownership")
if os.environ.get("MCP_ROTATE", "false").lower() == "true":
token.key = secret
token.expiring = False
token.save(update_fields=["key", "expiring"])
print(
json.dumps(
{
"created": created,
"tokenCreated": token_created,
"subject": str(user.uuid),
"username": user.username,
"tokenIdentifier": token.identifier,
},
sort_keys=True,
)
)
del secret
scripts/backup_state.py
+45
View File
@@ -0,0 +1,45 @@
#!/usr/bin/env python3
"""Create and verify an allowlisted Hub state archive without secrets."""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
from packages.hub_operations.backup import BackupManager, BackupPolicy # noqa: E402
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--source-root", type=Path, required=True)
parser.add_argument("--backup-root", type=Path, required=True)
parser.add_argument("--name", required=True)
parser.add_argument("--include", action="append", required=True)
args = parser.parse_args()
manager = BackupManager(args.backup_root, policy=BackupPolicy())
artifact = manager.create_archive(
args.source_root, name=args.name, include=(Path(item) for item in args.include)
)
verified = manager.verify(artifact)
print(
json.dumps(
{
"archive": str(verified.archive),
"sha256": verified.sha256,
"bytes": verified.bytes,
"entries": verified.entries,
},
sort_keys=True,
)
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
scripts/bootstrap.ps1
+6
View File
@@ -0,0 +1,6 @@
$ErrorActionPreference = "Stop"
Set-Location (Join-Path $PSScriptRoot "..")
python scripts/validate_pack.py
python scripts/next_packet.py
python scripts/render_luna_context.py
Write-Host "`nOpen .luna/context.md and prompts/AUTONOMOUS_BUILD_LUNA.md in Codex."
scripts/bootstrap.sh
+7
View File
@@ -0,0 +1,7 @@
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")/.."
python scripts/validate_pack.py
python scripts/next_packet.py
python scripts/render_luna_context.py
printf '\nOpen .luna/context.md and prompts/AUTONOMOUS_BUILD_LUNA.md in Codex.\n'
scripts/build_contextforge_compat_image.sh
+170
View File
@@ -0,0 +1,170 @@
#!/usr/bin/env bash
set -Eeuo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
MANIFEST="$ROOT/infrastructure/images/contextforge/compat-image.env"
CONTRACT="$ROOT/infrastructure/images/contextforge/runtime-contract.json"
ENTRYPOINT="$ROOT/infrastructure/images/contextforge/entrypoint.sh"
RUNTIME_VERIFIER="$ROOT/scripts/contextforge_runtime_contract.py"
for required in "$MANIFEST" "$CONTRACT" "$ENTRYPOINT" "$RUNTIME_VERIFIER"; do
[[ -f "$required" ]] || {
echo "FATAL: required ContextForge compatibility input is missing: $required" >&2
exit 66
}
done
# shellcheck source=/dev/null
source "$MANIFEST"
: "${CONTEXTFORGE_UPSTREAM_COMMIT:?missing upstream commit}"
: "${CONTEXTFORGE_ALEMBIC_HEAD:?missing Alembic head}"
: "${CONTEXTFORGE_MCP_SDK:?missing MCP SDK version}"
: "${CONTEXTFORGE_LOCAL_IMAGE:?missing local output image}"
command -v git >/dev/null 2>&1 || { echo "FATAL: git is required" >&2; exit 69; }
command -v docker >/dev/null 2>&1 || { echo "FATAL: docker is required" >&2; exit 69; }
command -v python3 >/dev/null 2>&1 || { echo "FATAL: python3 is required" >&2; exit 69; }
TMP="$(mktemp -d)"
trap 'rm -rf "$TMP"' EXIT
SRC="$TMP/contextforge"
BASE_IMAGE="${CONTEXTFORGE_LOCAL_IMAGE}-upstream"
STAGED_IMAGE="${CONTEXTFORGE_LOCAL_IMAGE}-staged"
REVISION_PREFIX="${CONTEXTFORGE_ALEMBIC_HEAD}_"
MCP_REQUIREMENT="mcp==${CONTEXTFORGE_MCP_SDK}"
fail() {
echo "FATAL: $*" >&2
exit 70
}
sha256_file() {
python3 - "$1" <<'PY'
import hashlib
import sys
from pathlib import Path
print(hashlib.sha256(Path(sys.argv[1]).read_bytes()).hexdigest())
PY
}
read_contract() {
python3 - "$CONTRACT" "$1" <<'PY'
import json
import sys
from pathlib import Path
payload = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
value = payload
for part in sys.argv[2].split("."):
value = value[part]
if not isinstance(value, str) or not value:
raise SystemExit(2)
print(value)
PY
}
CONTRACT_SOURCE_COMMIT="$(read_contract sourceCommit)"
CONTRACT_ALEMBIC_HEAD="$(read_contract alembicHead)"
CONTRACT_MCP_SDK="$(read_contract mcpSdk)"
CONTRACT_REQUIRED_SYMBOL="$(read_contract requiredSymbol)"
CONTRACT_SHA256="$(sha256_file "$CONTRACT")"
ENTRYPOINT_SHA256="$(sha256_file "$ENTRYPOINT")"
VERIFIER_SHA256="$(sha256_file "$RUNTIME_VERIFIER")"
[[ "$CONTRACT_SOURCE_COMMIT" == "$CONTEXTFORGE_UPSTREAM_COMMIT" ]] || fail "contract source commit disagrees with compatibility manifest"
[[ "$CONTRACT_ALEMBIC_HEAD" == "$CONTEXTFORGE_ALEMBIC_HEAD" ]] || fail "contract Alembic head disagrees with compatibility manifest"
[[ "$CONTRACT_MCP_SDK" == "$CONTEXTFORGE_MCP_SDK" ]] || fail "contract MCP SDK disagrees with compatibility manifest"
[[ "$CONTRACT_REQUIRED_SYMBOL" == "McpError" ]] || fail "unsupported required MCP symbol: $CONTRACT_REQUIRED_SYMBOL"
echo "=== FETCH EXACT CONTEXTFORGE SOURCE ==="
git init -q "$SRC"
git -C "$SRC" remote add origin https://github.com/IBM/mcp-context-forge.git
git -C "$SRC" fetch --depth 1 origin "$CONTEXTFORGE_UPSTREAM_COMMIT"
git -C "$SRC" checkout --detach FETCH_HEAD
ACTUAL_COMMIT="$(git -C "$SRC" rev-parse HEAD)"
[[ "$ACTUAL_COMMIT" == "$CONTEXTFORGE_UPSTREAM_COMMIT" ]] || fail "source commit mismatch: expected $CONTEXTFORGE_UPSTREAM_COMMIT, got $ACTUAL_COMMIT"
MIGRATION_FILE="$({
find "$SRC/mcpgateway/alembic/versions" -maxdepth 1 -type f -name "${REVISION_PREFIX}*.py" -print -quit
} 2>/dev/null || true)"
[[ -n "$MIGRATION_FILE" ]] || fail "migration ${REVISION_PREFIX}*.py is absent from source commit"
[[ -f "$SRC/Containerfile" ]] || fail "upstream Containerfile is missing"
echo "source commit : $ACTUAL_COMMIT"
echo "migration : ${MIGRATION_FILE#$SRC/}"
echo "contract sha : $CONTRACT_SHA256"
echo
echo "=== BUILD UPSTREAM SOURCE IMAGE ==="
docker build --pull=false -f "$SRC/Containerfile" -t "$BASE_IMAGE" "$SRC"
BASE_USER="$(docker image inspect -f '{{.Config.User}}' "$BASE_IMAGE")"
LAYER="$TMP/Containerfile.compat"
cp "$CONTRACT" "$TMP/contextforge-runtime-contract.json"
cp "$ENTRYPOINT" "$TMP/contextforge-entrypoint.sh"
cp "$RUNTIME_VERIFIER" "$TMP/contextforge-runtime-contract.py"
{
echo "FROM $BASE_IMAGE"
echo "USER root"
echo 'COPY contextforge-runtime-contract.json /opt/itworx/contextforge-runtime-contract.json'
echo 'COPY contextforge-entrypoint.sh /opt/itworx/contextforge-entrypoint.sh'
echo 'COPY contextforge-runtime-contract.py /opt/itworx/contextforge-runtime-contract.py'
echo 'RUN chmod 0555 /opt/itworx/contextforge-entrypoint.sh /opt/itworx/contextforge-runtime-contract.py'
printf 'LABEL io.itworx.contextforge.contract-sha256="%s" \\\n' "$CONTRACT_SHA256"
printf ' io.itworx.contextforge.entrypoint-sha256="%s" \\\n' "$ENTRYPOINT_SHA256"
printf ' io.itworx.contextforge.verifier-sha256="%s" \\\n' "$VERIFIER_SHA256"
printf ' io.itworx.contextforge.source-commit="%s" \\\n' "$CONTEXTFORGE_UPSTREAM_COMMIT"
printf ' io.itworx.contextforge.alembic-head="%s" \\\n' "$CONTEXTFORGE_ALEMBIC_HEAD"
printf ' io.itworx.contextforge.mcp-sdk="%s"\n' "$CONTEXTFORGE_MCP_SDK"
printf 'RUN /app/.venv/bin/python -m pip install --no-cache-dir --upgrade %q \\\n' "$MCP_REQUIREMENT"
printf '%s\n' ' && /app/.venv/bin/python -c "import importlib.metadata as md; from mcp import McpError; assert md.version(\"mcp\") == \"'"${CONTEXTFORGE_MCP_SDK}"'\"; print(\"MCP SDK\", md.version(\"mcp\"), \"McpError OK\")"'
if [[ -n "$BASE_USER" ]]; then
printf 'USER %s\n' "$BASE_USER"
fi
} > "$LAYER"
echo
echo "=== LAYER EXACT MCP SDK AND CONTRACT ==="
docker build --pull=false -f "$LAYER" -t "$STAGED_IMAGE" "$TMP"
echo
echo "=== VERIFY RUNTIME CONTRACT ==="
docker run --rm --entrypoint /bin/sh "$STAGED_IMAGE" -lc '/app/.venv/bin/python -c "import importlib.metadata as md; from mcp import McpError; assert md.version(\"mcp\") == \"'"$CONTEXTFORGE_MCP_SDK"'\"; print(\"McpError OK\")"'
IMAGE_HEAD="$(docker run --rm --entrypoint /bin/sh "$STAGED_IMAGE" -lc 'cd /app && /app/.venv/bin/alembic -c mcpgateway/alembic.ini heads' | awk 'NR == 1 {print $1}')"
[[ "$IMAGE_HEAD" == "$CONTEXTFORGE_ALEMBIC_HEAD" ]] || fail "Alembic head mismatch: expected $CONTEXTFORGE_ALEMBIC_HEAD, got ${IMAGE_HEAD:-empty}"
embedded_sha() {
docker run --rm --entrypoint /app/.venv/bin/python "$STAGED_IMAGE" -c 'import hashlib,sys; from pathlib import Path; print(hashlib.sha256(Path(sys.argv[1]).read_bytes()).hexdigest())' "$1"
}
[[ "$(embedded_sha /opt/itworx/contextforge-runtime-contract.json)" == "$CONTRACT_SHA256" ]] || fail "embedded runtime contract digest mismatch"
[[ "$(embedded_sha /opt/itworx/contextforge-entrypoint.sh)" == "$ENTRYPOINT_SHA256" ]] || fail "embedded entrypoint digest mismatch"
[[ "$(embedded_sha /opt/itworx/contextforge-runtime-contract.py)" == "$VERIFIER_SHA256" ]] || fail "embedded verifier digest mismatch"
label_value() {
docker image inspect -f "{{ index .Config.Labels \"$1\" }}" "$STAGED_IMAGE"
}
[[ "$(label_value io.itworx.contextforge.contract-sha256)" == "$CONTRACT_SHA256" ]] || fail "contract label mismatch"
[[ "$(label_value io.itworx.contextforge.entrypoint-sha256)" == "$ENTRYPOINT_SHA256" ]] || fail "entrypoint label mismatch"
[[ "$(label_value io.itworx.contextforge.verifier-sha256)" == "$VERIFIER_SHA256" ]] || fail "verifier label mismatch"
[[ "$(label_value io.itworx.contextforge.source-commit)" == "$CONTEXTFORGE_UPSTREAM_COMMIT" ]] || fail "source label mismatch"
[[ "$(label_value io.itworx.contextforge.alembic-head)" == "$CONTEXTFORGE_ALEMBIC_HEAD" ]] || fail "Alembic label mismatch"
[[ "$(label_value io.itworx.contextforge.mcp-sdk)" == "$CONTEXTFORGE_MCP_SDK" ]] || fail "MCP SDK label mismatch"
echo "MCP SDK : $CONTEXTFORGE_MCP_SDK"
echo "Alembic head : $IMAGE_HEAD"
echo "Contract SHA : $CONTRACT_SHA256"
docker tag "$STAGED_IMAGE" "$CONTEXTFORGE_LOCAL_IMAGE"
echo
echo "======================================================"
echo " SUCCESS: CONTEXTFORGE COMPATIBILITY IMAGE VERIFIED"
echo " source commit : $CONTEXTFORGE_UPSTREAM_COMMIT"
echo " Alembic head : $CONTEXTFORGE_ALEMBIC_HEAD"
echo " MCP SDK : $CONTEXTFORGE_MCP_SDK"
echo " contract SHA : $CONTRACT_SHA256"
echo " local image : $CONTEXTFORGE_LOCAL_IMAGE"
echo "======================================================"
scripts/check_boundaries.py
+127
View File
@@ -0,0 +1,127 @@
#!/usr/bin/env python3
"""Check the repository dependency direction without importing application code."""
from __future__ import annotations
import argparse
import ast
import re
import sys
from dataclasses import dataclass
from pathlib import Path
SOURCE_ROOTS = {"apps", "services", "connectors", "packages"}
SOURCE_SUFFIXES = {".py", ".pyi", ".js", ".jsx", ".ts", ".tsx"}
IGNORED_PARTS = {"__pycache__", ".venv", "node_modules", "dist"}
@dataclass(frozen=True)
class Violation:
path: Path
line: int
message: str
def format(self, root: Path) -> str:
return f"{self.path.relative_to(root).as_posix()}:{self.line}: {self.message}"
def _owner(path: Path, root: Path) -> tuple[str, str] | None:
relative = path.relative_to(root)
if len(relative.parts) < 2 or relative.parts[0] not in SOURCE_ROOTS:
return None
return relative.parts[0], relative.parts[1].replace("-", "_")
def _target(module: str) -> tuple[str, str] | None:
normalized = module.replace("-", "_")
parts = normalized.split(".")
if len(parts) < 2 or parts[0] not in SOURCE_ROOTS:
return None
return parts[0], parts[1]
def _allowed(owner: tuple[str, str], target: tuple[str, str]) -> bool:
owner_root, owner_name = owner
target_root, target_name = target
if owner_root == "packages":
if owner_name == "hub_contracts":
return target_root == "packages" and target_name == "hub_contracts"
if owner_name in {"hub_core", "hub_policy", "connector_kit"}:
return target_root == "packages" and target_name in {owner_name, "hub_contracts"}
return target_root == "packages"
if owner_root == "connectors":
return target_root == "packages"
if owner_root == "services":
if owner_name == "runtime":
return target_root in {"packages", "services", "connectors"}
return target_root == "packages" or target == ("services", owner_name)
if owner_root == "apps":
return target_root == "packages" or target == ("apps", owner_name)
return True
def _python_imports(path: Path) -> list[tuple[int, str]]:
try:
tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
except (OSError, SyntaxError):
return []
imports: list[tuple[int, str]] = []
for node in ast.walk(tree):
if isinstance(node, ast.Import):
imports.extend((node.lineno, alias.name) for alias in node.names)
elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module:
imports.append((node.lineno, node.module))
return imports
def _js_imports(path: Path) -> list[tuple[int, str]]:
pattern = re.compile(r"(?:from\s+|import\s*\(?\s*|require\(\s*)['\"]([^'\"]+)")
return [
(line_no, module)
for line_no, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1)
for module in pattern.findall(line)
]
def find_violations(root: Path, paths: list[Path] | None = None) -> list[Violation]:
candidates = paths or [
path
for source_root in SOURCE_ROOTS
for path in (root / source_root).rglob("*")
if path.is_file()
and path.suffix in SOURCE_SUFFIXES
and not (set(path.parts) & IGNORED_PARTS)
]
violations: list[Violation] = []
for path in candidates:
owner = _owner(path, root)
if owner is None:
continue
imports = _python_imports(path) if path.suffix in {".py", ".pyi"} else _js_imports(path)
for line, module in imports:
target = _target(module)
if target and not _allowed(owner, target):
violations.append(
Violation(
path, line, f"forbidden dependency {module!r} from {owner[0]}/{owner[1]}"
)
)
return violations
def main() -> int:
parser = argparse.ArgumentParser(description="Check ITWorx MCP Hub import boundaries.")
parser.add_argument("--root", type=Path, default=Path(__file__).resolve().parents[1])
args = parser.parse_args()
root = args.root.resolve()
violations = find_violations(root)
if violations:
for violation in violations:
print(violation.format(root), file=sys.stderr)
return 1
print("Import-boundary check: PASS")
return 0
if __name__ == "__main__":
raise SystemExit(main())
scripts/check_build_reproducibility.py
+104
View File
@@ -0,0 +1,104 @@
#!/usr/bin/env python3
"""Build a service's image twice from a clean context and compare the results.
Distinguishes full image-ID (byte-for-byte, including embedded metadata like
history timestamps) reproducibility from layer-content (diffID) reproducibility,
per the WP-200 requirement not to claim byte-identical results without proof.
"""
from __future__ import annotations
import argparse
import json
import subprocess
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
ROOT = Path(__file__).resolve().parents[1]
def _run(command: list[str]) -> str:
completed = subprocess.run( # noqa: S603
command, cwd=ROOT, capture_output=True, text=True, check=True
)
return completed.stdout.strip()
def _build(dockerfile: Path, tag: str, build_args: dict[str, str]) -> None:
command = [
"docker",
"buildx",
"build",
"--no-cache",
"--output",
"type=docker,rewrite-timestamp=true,unpack=false",
"-f",
str(dockerfile),
"-t",
tag,
]
for name, value in build_args.items():
command.extend(["--build-arg", f"{name}={value}"])
command.append(str(ROOT))
subprocess.run(command, cwd=ROOT, check=True) # noqa: S603
def _image_id(tag: str) -> str:
return _run(["docker", "inspect", "--format", "{{.Id}}", tag])
def _layer_diff_ids(tag: str) -> list[str]:
raw = _run(["docker", "inspect", "--format", "{{json .RootFS.Layers}}", tag])
return list(json.loads(raw))
def compare_two_builds(
dockerfile: Path, build_args: dict[str, str], tag_a: str, tag_b: str
) -> dict[str, Any]:
_build(dockerfile, tag_a, build_args)
_build(dockerfile, tag_b, build_args)
image_id_a, image_id_b = _image_id(tag_a), _image_id(tag_b)
layers_a, layers_b = _layer_diff_ids(tag_a), _layer_diff_ids(tag_b)
return {
"dockerfile": str(dockerfile.relative_to(ROOT)),
"buildArgs": {key: value for key, value in build_args.items() if key != "SOURCE_DATE_EPOCH"}
| {"SOURCE_DATE_EPOCH": build_args.get("SOURCE_DATE_EPOCH", "")},
"imageIdA": image_id_a,
"imageIdB": image_id_b,
"imageIdMatch": image_id_a == image_id_b,
"layerDiffIdsA": layers_a,
"layerDiffIdsB": layers_b,
"layerDiffIdsMatch": layers_a == layers_b,
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--dockerfile", type=Path, required=True)
parser.add_argument("--service-name", required=False)
parser.add_argument("--git-revision", required=True)
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
args.dockerfile = args.dockerfile.resolve()
epoch = _run(["git", "show", "-s", "--format=%ct", args.git_revision])
created = _run(["git", "show", "-s", "--format=%cI", args.git_revision])
build_args = {
"GIT_REVISION": args.git_revision,
"SOURCE_DATE_EPOCH": epoch,
"BUILD_DATE": created,
}
if args.service_name:
build_args["SERVICE_NAME"] = args.service_name
stamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
tag_a = f"local/wp200-repro-a:{stamp}"
tag_b = f"local/wp200-repro-b:{stamp}"
report = compare_two_builds(args.dockerfile, build_args, tag_a, tag_b)
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8")
print(json.dumps(report, indent=2, sort_keys=True))
return 0
if __name__ == "__main__":
raise SystemExit(main())
scripts/check_cve_exceptions.py
+181
View File
@@ -0,0 +1,181 @@
#!/usr/bin/env python3
"""Enforce the CVE exception policy against a vulnerability scan.
Every CRITICAL/HIGH finding in --scan-summary must be covered by a current,
explicit, fully-populated, *approved* exception in --exceptions, or the gate
fails. This never suppresses scanner output -- it cross-references it.
Specifically fails on:
- a CRITICAL/HIGH finding (package, CVE) with no matching exception at all;
- a matching exception whose approvalStatus is not exactly "approved"
(covers "pending", "rejected", or any other value -- there is no default
grant);
- a matching exception that has expired (expiryDate in the past) or is past
its mandatory rescan date;
- a matching exception whose recorded affectedImageDigests disagree with
--image-digests for any service both documents name (the digest changed --
the exception was reviewed against a build that no longer exists);
- an exception missing any required field.
See docs/09-release/CVE_EXCEPTIONS.md for the full policy and
contracts/schemas/ (release-record.schema.json's sibling documents) for
context. This script has no "--force"/override flag by design.
"""
from __future__ import annotations
import argparse
import sys
from datetime import date
from pathlib import Path
from typing import Any
_ROOT_DIR = Path(__file__).resolve().parents[1]
if str(_ROOT_DIR) not in sys.path:
sys.path.insert(0, str(_ROOT_DIR))
from scripts.packlib import load_json # noqa: E402
REQUIRED_FIELDS = (
"id",
"cve",
"package",
"installedVersion",
"severity",
"scanner",
"scanDate",
"affectedImageDigests",
"reachabilityAnalysis",
"compensatingControls",
"riskOwner",
"approver",
"approvalStatus",
"expiryDate",
"mandatoryRescanDate",
"remediationTrigger",
"evidenceReference",
)
def _findings(scan: dict[str, Any]) -> list[tuple[str, str, str]]:
"""Return (package, cve, severity) for every CRITICAL/HIGH finding."""
findings = []
for row in scan.get("unresolvedCriticalOrHigh", []):
for cve in row["cves"]:
findings.append((row["package"], cve, row.get("severity", "UNKNOWN")))
return findings
def check(
scan: dict[str, Any],
exceptions_doc: dict[str, Any],
image_digests: dict[str, str] | None,
today: date,
) -> list[str]:
errors: list[str] = []
by_key: dict[tuple[str, str], dict[str, Any]] = {}
for exception in exceptions_doc.get("exceptions", []):
missing = [field for field in REQUIRED_FIELDS if not exception.get(field)]
# riskOwner/approver/compensatingControls may be legitimately empty
# only while approvalStatus is "pending" -- once "approved" they are
# mandatory. Re-check those three explicitly below rather than
# excluding them from REQUIRED_FIELDS entirely, so a pending draft
# from scripts/draft_cve_exceptions.py isn't flagged as malformed.
missing = [f for f in missing if f not in {"riskOwner", "approver", "compensatingControls"}]
if missing:
errors.append(
f"exception {exception.get('id', '<no id>')} is missing required fields: {missing}"
)
continue
key = (exception["package"], exception["cve"])
by_key[key] = exception
for package, cve, severity in _findings(scan):
key = (package, cve)
exception = by_key.get(key)
if exception is None:
errors.append(
f"UNAPPROVED: {severity} finding {package}/{cve} has no exception recorded at all"
)
continue
status = exception.get("approvalStatus")
if status != "approved":
errors.append(
f"UNAPPROVED: {package}/{cve} exception status is {status!r}, "
"not 'approved' -- no default grant"
)
continue
for role in ("riskOwner", "approver"):
if not exception.get(role):
errors.append(
f"UNAPPROVED: {package}/{cve} is marked approved but has no {role} recorded"
)
if not exception.get("compensatingControls"):
errors.append(
f"UNAPPROVED: {package}/{cve} is marked approved but records no "
"compensating controls"
)
expiry = date.fromisoformat(exception["expiryDate"])
if expiry < today:
errors.append(f"EXPIRED: {package}/{cve} exception expired on {expiry.isoformat()}")
rescan_due = date.fromisoformat(exception["mandatoryRescanDate"])
if rescan_due < today:
errors.append(
f"RESCAN OVERDUE: {package}/{cve} exception's mandatory rescan date "
f"{rescan_due.isoformat()} has passed"
)
if image_digests:
recorded = exception.get("affectedImageDigests", {})
for service, current_digest in image_digests.items():
recorded_digest = recorded.get(service)
if recorded_digest and recorded_digest != current_digest:
errors.append(
f"DIGEST CHANGED: {package}/{cve} exception recorded "
f"{service}={recorded_digest}, but the current build is "
f"{current_digest} -- the exception was reviewed against a "
"build that no longer exists and must be re-approved"
)
return errors
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--scan-summary", required=True, type=Path)
parser.add_argument("--exceptions", required=True, type=Path)
parser.add_argument(
"--image-digests",
type=Path,
help="evidence/<packet>/image-digests.json; when given, exceptions whose "
"recorded digest for a named service disagrees with the current build fail.",
)
args = parser.parse_args()
scan = load_json(args.scan_summary)
exceptions_doc = load_json(args.exceptions)
image_digests: dict[str, str] | None = None
if args.image_digests:
digest_doc = load_json(args.image_digests)
image_digests = {
service: (entry.get("digest") or entry.get("imageId"))
for service, entry in digest_doc.get("images", {}).items()
}
errors = check(scan, exceptions_doc, image_digests, date.today())
if errors:
print("CVE exception gate: FAIL")
for error in errors:
print(f" - {error}")
return 1
print(
"CVE exception gate: PASS (every CRITICAL/HIGH finding has a current, approved exception)"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
scripts/contextforge_bootstrap.py
+336
View File
@@ -0,0 +1,336 @@
#!/usr/bin/env python3
"""Create the least-privilege ContextForge API token without exposing it."""
from __future__ import annotations
import argparse
import json
import os
import stat
import tempfile
import urllib.error
import urllib.request
from pathlib import Path
from typing import Any
from urllib.parse import quote, urlparse
MAX_RESPONSE_BYTES = 1024 * 1024
RUNTIME_PERMISSIONS = (
"gateways.read",
"gateways.create",
"gateways.update",
"gateways.delete",
"tools.read",
"tools.update",
"tools.execute",
"prompts.read",
"servers.read",
"servers.create",
"servers.update",
"servers.use",
)
RUNTIME_READ_CHECKS = (
"/gateways?limit=1&include_pagination=false",
"/tools?limit=1&include_pagination=false",
"/prompts?limit=1&include_pagination=false",
"/servers?limit=1&include_pagination=false",
)
class BootstrapError(RuntimeError):
"""Sanitized bootstrap failure."""
def _validate_base_url(value: str) -> str:
parsed = urlparse(value)
allowed = (
parsed.scheme == "http"
and parsed.hostname == "contextforge"
and parsed.port == 4444
and not parsed.username
and not parsed.password
) or (
parsed.scheme == "https"
and bool(parsed.hostname)
and not parsed.username
and not parsed.password
)
if not allowed or parsed.path not in {"", "/"} or parsed.query or parsed.fragment:
raise BootstrapError("ContextForge bootstrap URL is not approved")
return value.rstrip("/")
def _read_secret(path: Path) -> str:
try:
metadata = path.stat()
payload = path.read_bytes()
except OSError as exc:
raise BootstrapError("ContextForge admin password is unavailable") from exc
content = payload.rstrip(b"\r\n")
if (
not stat.S_ISREG(metadata.st_mode)
or (
os.name != "nt"
and (
stat.S_IMODE(metadata.st_mode) not in {0o400, 0o600}
or metadata.st_uid != os.getuid()
or metadata.st_gid != os.getgid()
)
)
or not content
or len(content) > 8192
or b"\x00" in content
or b"\r" in content
or b"\n" in content
):
raise BootstrapError("ContextForge admin password violates the secret-file policy")
return content.decode("utf-8")
def _request_json(
base_url: str,
method: str,
path: str,
*,
token: str | None = None,
body: dict[str, Any] | None = None,
) -> Any:
headers = {"Accept": "application/json"}
data = None
if token:
headers["Authorization"] = f"Bearer {token}"
if body is not None:
headers["Content-Type"] = "application/json"
data = json.dumps(body, separators=(",", ":")).encode()
request = urllib.request.Request( # noqa: S310 - URL is validated before use.
base_url + path, data=data, headers=headers, method=method
)
try:
with urllib.request.urlopen(request, timeout=10) as response: # noqa: S310
payload = response.read(MAX_RESPONSE_BYTES + 1)
if len(payload) > MAX_RESPONSE_BYTES:
raise BootstrapError("ContextForge bootstrap response exceeded its limit")
return json.loads(payload) if payload else None
except urllib.error.HTTPError as exc:
raise BootstrapError(
f"ContextForge {path.split('?', 1)[0]} returned HTTP {exc.code}"
) from None
except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc:
raise BootstrapError("ContextForge bootstrap request failed") from exc
def _write_token(path: Path, token: str) -> None:
if not token or "\n" in token or "\r" in token or "\x00" in token or len(token) > 8192:
raise BootstrapError("ContextForge returned an invalid API token")
path.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
if os.name != "nt" and stat.S_IMODE(path.parent.stat().st_mode) != 0o700:
raise BootstrapError("ContextForge token directory must have mode 0700")
descriptor, temporary = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
try:
os.fchmod(descriptor, 0o600)
with os.fdopen(descriptor, "w", encoding="utf-8") as stream:
stream.write(token)
stream.flush()
os.fsync(stream.fileno())
os.replace(temporary, path)
except Exception:
try:
os.close(descriptor)
except OSError:
pass
try:
os.unlink(temporary)
except OSError:
pass
raise
def _login(base_url: str, admin_email: str, password: str) -> str:
login = _request_json(
base_url,
"POST",
"/auth/email/login",
body={"email": admin_email, "password": password},
)
session_token = login.get("access_token") if isinstance(login, dict) else None
if not isinstance(session_token, str) or not session_token:
raise BootstrapError("ContextForge login did not return a session token")
return session_token
def _verify_runtime_token(base_url: str, token: str) -> None:
for path in RUNTIME_READ_CHECKS:
_request_json(base_url, "GET", path, token=token)
def _revoke_token(base_url: str, session_token: str, token_id: str) -> None:
if not token_id:
raise BootstrapError("ContextForge token identifier was invalid")
encoded_id = quote(token_id, safe="")
_request_json(base_url, "DELETE", f"/tokens/{encoded_id}", token=session_token)
def _revoke_named_token(base_url: str, session_token: str, token_name: str) -> None:
catalog = _request_json(
base_url,
"GET",
"/tokens?include_inactive=false&limit=100&offset=0",
token=session_token,
)
tokens = catalog.get("tokens") if isinstance(catalog, dict) else None
if not isinstance(tokens, list):
raise BootstrapError("ContextForge token inventory response was invalid")
matches = [
item
for item in tokens
if isinstance(item, dict)
and item.get("name") == token_name
and item.get("is_active") is True
and item.get("is_revoked") is not True
]
if len(matches) != 1 or not isinstance(matches[0].get("id"), str) or not matches[0]["id"]:
raise BootstrapError("ContextForge token rotation requires one active named token")
_revoke_token(base_url, session_token, matches[0]["id"])
def bootstrap(
*,
base_url: str,
admin_email: str,
admin_password_file: Path,
output_token_file: Path,
source_network: str,
token_name: str,
expires_days: int,
rotate_existing: bool = False,
replacement_token_name: str | None = None,
) -> dict[str, Any]:
base_url = _validate_base_url(base_url)
if rotate_existing and (not replacement_token_name or replacement_token_name == token_name):
raise BootstrapError("ContextForge rotation requires a distinct replacement token name")
effective_token_name = replacement_token_name if rotate_existing else token_name
if not effective_token_name:
raise BootstrapError("ContextForge token name is required")
password: str | None = None
session_token: str | None = None
if output_token_file.exists():
existing = _read_secret(output_token_file)
if rotate_existing:
password = _read_secret(admin_password_file)
session_token = _login(base_url, admin_email, password)
_revoke_named_token(base_url, session_token, token_name)
try:
output_token_file.unlink()
except OSError as exc:
raise BootstrapError(
"ContextForge revoked token file could not be removed"
) from exc
else:
_verify_runtime_token(base_url, existing)
return {"status": "pass", "action": "reused", "tokenName": token_name}
if password is None:
password = _read_secret(admin_password_file)
if session_token is None:
session_token = _login(base_url, admin_email, password)
created = _request_json(
base_url,
"POST",
"/tokens",
token=session_token,
body={
"name": effective_token_name,
"description": "ITWorx MCP Hub production gateway reconciliation",
"expires_in_days": expires_days,
"scope": {
"permissions": list(RUNTIME_PERMISSIONS),
"ip_restrictions": [source_network],
"time_restrictions": {},
"usage_limits": {},
},
"tags": ["itworx-mcp-hub", "production"],
"is_active": True,
},
)
api_token = created.get("access_token") if isinstance(created, dict) else None
token_record = created.get("token") if isinstance(created, dict) else None
token_id = token_record.get("id") if isinstance(token_record, dict) else None
if not isinstance(api_token, str) or not api_token:
try:
_revoke_named_token(base_url, session_token, effective_token_name)
except BootstrapError:
raise BootstrapError(
"ContextForge token creation returned no API token and cleanup failed"
) from None
raise BootstrapError("ContextForge token creation returned no API token")
if not isinstance(token_id, str) or not token_id:
try:
_revoke_named_token(base_url, session_token, effective_token_name)
except BootstrapError:
raise BootstrapError(
"ContextForge token creation returned no identifier and cleanup failed"
) from None
raise BootstrapError("ContextForge token creation returned no identifier")
try:
_verify_runtime_token(base_url, api_token)
_write_token(output_token_file, api_token)
except Exception as exc:
try:
_revoke_token(base_url, session_token, token_id)
except BootstrapError:
raise BootstrapError(
"ContextForge replacement token validation failed and cleanup failed"
) from None
if isinstance(exc, BootstrapError):
raise
raise BootstrapError("ContextForge replacement token could not be stored") from exc
return {
"status": "pass",
"action": "created",
"tokenName": effective_token_name,
"permissions": list(RUNTIME_PERMISSIONS),
"sourceNetwork": source_network,
"expiresDays": expires_days,
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--base-url", default="http://contextforge:4444")
parser.add_argument("--admin-email", required=True)
parser.add_argument("--admin-password-file", type=Path, required=True)
parser.add_argument("--output-token-file", type=Path, required=True)
parser.add_argument("--source-network", default="203.0.113.0/24")
parser.add_argument("--token-name", default="itworx-mcp-hub-runtime-prod")
parser.add_argument("--expires-days", type=int, default=365)
parser.add_argument(
"--rotate-existing",
action="store_true",
help=(
"revoke exactly one invalid named token before creating a distinctly named replacement"
),
)
parser.add_argument(
"--replacement-token-name",
help="unique replacement name required with --rotate-existing",
)
args = parser.parse_args()
if not 1 <= args.expires_days <= 365:
raise BootstrapError("ContextForge token expiry must be between 1 and 365 days")
result = bootstrap(
base_url=args.base_url,
admin_email=args.admin_email,
admin_password_file=args.admin_password_file,
output_token_file=args.output_token_file,
source_network=args.source_network,
token_name=args.token_name,
expires_days=args.expires_days,
rotate_existing=args.rotate_existing,
replacement_token_name=args.replacement_token_name,
)
print(json.dumps(result, sort_keys=True))
return 0
if __name__ == "__main__":
raise SystemExit(main())
scripts/contextforge_runtime_contract.py
+246
View File
@@ -0,0 +1,246 @@
#!/usr/bin/env python3
"""Read-only fail-closed ContextForge runtime compatibility verifier."""
from __future__ import annotations
import argparse
import hashlib
import importlib
import importlib.metadata
import json
import os
import re
import subprocess
import sys
from collections.abc import Callable, Iterable, Mapping
from pathlib import Path
from typing import Any
from sqlalchemy import create_engine, text
from sqlalchemy.engine import URL
DEFAULT_CONTRACT = Path("/opt/itworx/contextforge-runtime-contract.json")
EXPECTED_SCHEMA = "itworx.contextforge-runtime-contract/v1"
ALEMBIC_BIN = "/app/.venv/bin/alembic"
ALEMBIC_CONFIG = "mcpgateway/alembic.ini"
APP_ROOT = "/app"
SHA256 = re.compile(r"^[0-9a-f]{64}$")
CONTEXTFORGE_DB_PASSWORD_FILE = Path("/run/secrets/contextforge_database_password")
class RuntimeContractError(RuntimeError):
"""Bounded compatibility failure with no secret-bearing detail."""
def _fail(code: str) -> None:
raise RuntimeContractError(code)
def contract_sha256(path: Path) -> str:
try:
return hashlib.sha256(path.read_bytes()).hexdigest()
except OSError as exc:
raise RuntimeContractError("contract-unreadable") from exc
def validate_contract_digest(path: Path, environment: Mapping[str, str]) -> None:
expected = environment.get("ITWORX_CONTEXTFORGE_EXPECTED_CONTRACT_SHA256", "")
if not SHA256.fullmatch(expected):
_fail("contract-digest-config")
if contract_sha256(path) != expected:
_fail("contract-digest")
def load_contract(path: Path) -> dict[str, Any]:
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError, UnicodeError) as exc:
raise RuntimeContractError("contract-unreadable") from exc
if not isinstance(payload, dict):
_fail("contract-shape")
validate_contract_shape(payload)
return payload
def validate_contract_shape(contract: Mapping[str, Any]) -> None:
if contract.get("schema") != EXPECTED_SCHEMA:
_fail("contract-schema")
for key in ("sourceCommit", "alembicHead", "mcpSdk", "requiredSymbol"):
value = contract.get(key)
if not isinstance(value, str) or not value:
_fail(f"contract:{key}")
if re.fullmatch(r"[0-9a-f]{40}", str(contract["sourceCommit"])) is None:
_fail("contract:sourceCommit")
production = contract.get("productionEnvironment")
if not isinstance(production, dict) or not production:
_fail("contract:productionEnvironment")
if any(
not isinstance(key, str) or not isinstance(value, str)
for key, value in production.items()
):
_fail("contract:productionEnvironment")
policy = contract.get("migrationPolicy")
if not isinstance(policy, dict):
_fail("migration-policy")
if (
policy.get("startupMigrations") != "forbidden"
or policy.get("skipEnv") != "MCPGATEWAY_SKIP_MIGRATIONS"
or policy.get("skipValue") != "true"
or production.get("MCPGATEWAY_SKIP_MIGRATIONS") != "true"
):
_fail("migration-policy")
def validate_environment(contract: Mapping[str, Any], environment: Mapping[str, str]) -> None:
production = contract["productionEnvironment"]
assert isinstance(production, dict)
for name, expected in production.items():
if environment.get(str(name)) != expected:
_fail(f"environment:{name}")
def installed_mcp_version() -> str:
return importlib.metadata.version("mcp")
def load_mcp_symbol(module_name: str, symbol: str) -> object:
module = importlib.import_module(module_name)
return getattr(module, symbol)
def validate_mcp_runtime(
contract: Mapping[str, Any],
*,
version_getter: Callable[[str], str] = importlib.metadata.version,
symbol_loader: Callable[[str, str], object] = load_mcp_symbol,
) -> None:
try:
actual_version = version_getter("mcp")
except Exception as exc:
raise RuntimeContractError("mcp-version") from exc
if actual_version != contract["mcpSdk"]:
_fail("mcp-version")
try:
symbol_loader("mcp", str(contract["requiredSymbol"]))
except Exception as exc:
raise RuntimeContractError("mcp-symbol") from exc
def query_image_alembic_heads() -> list[str]:
try:
completed = subprocess.run( # noqa: S603
[ALEMBIC_BIN, "-c", ALEMBIC_CONFIG, "heads"],
cwd=APP_ROOT,
capture_output=True,
text=True,
timeout=15,
check=False,
)
except (OSError, subprocess.SubprocessError) as exc:
raise RuntimeContractError("alembic-head") from exc
if completed.returncode != 0:
_fail("alembic-head")
heads: list[str] = []
for raw_line in completed.stdout.splitlines():
token = raw_line.strip().split(maxsplit=1)[0] if raw_line.strip() else ""
if token:
heads.append(token)
return sorted(set(heads))
def validate_alembic_head(contract: Mapping[str, Any], heads: Iterable[str]) -> None:
actual = sorted(set(str(item) for item in heads if str(item)))
if actual != [str(contract["alembicHead"])]:
_fail("alembic-head")
def resolve_database_url(environment: Mapping[str, str]) -> str:
configured = environment.get("DATABASE_URL", "")
if configured:
return configured
try:
password = (
CONTEXTFORGE_DB_PASSWORD_FILE.read_text(encoding="utf-8")
.replace("\r", "")
.replace("\n", "")
)
except (OSError, UnicodeError) as exc:
raise RuntimeContractError("database-url") from exc
if not password:
_fail("database-url")
return URL.create(
"postgresql+psycopg",
username="contextforge",
password=password,
host="postgres",
port=5432,
database="contextforge",
).render_as_string(hide_password=False)
def query_database_revisions(database_url: str) -> list[str]:
if not database_url:
_fail("database-url")
engine = create_engine(database_url, pool_pre_ping=True)
try:
with engine.connect() as connection:
rows = connection.execute(
text("SELECT version_num FROM alembic_version")
).scalars().all()
except Exception as exc:
raise RuntimeContractError("database-revision") from exc
finally:
engine.dispose()
return sorted(set(str(item) for item in rows if str(item)))
def validate_database_revision(contract: Mapping[str, Any], revisions: Iterable[str]) -> None:
actual = sorted(set(str(item) for item in revisions if str(item)))
if actual != [str(contract["alembicHead"])]:
_fail("database-revision")
def verify_runtime(contract: Mapping[str, Any], environment: Mapping[str, str]) -> None:
validate_contract_shape(contract)
validate_environment(contract, environment)
validate_mcp_runtime(
contract,
version_getter=lambda _name: installed_mcp_version(),
symbol_loader=load_mcp_symbol,
)
validate_alembic_head(contract, query_image_alembic_heads())
validate_database_revision(
contract,
query_database_revisions(resolve_database_url(environment)),
)
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
mode = parser.add_mutually_exclusive_group(required=True)
mode.add_argument("--startup", action="store_true")
mode.add_argument("--health", action="store_true")
parser.add_argument(
"--contract",
type=Path,
default=Path(os.environ.get("ITWORX_CONTEXTFORGE_CONTRACT_FILE", DEFAULT_CONTRACT)),
)
args = parser.parse_args()
if os.environ.get("ITWORX_CONTEXTFORGE_CONTRACT_MODE", "relaxed") != "strict":
print("ContextForge runtime contract: inactive", flush=True)
return 0
try:
validate_contract_digest(args.contract, os.environ)
contract = load_contract(args.contract)
verify_runtime(contract, os.environ)
except RuntimeContractError as exc:
print(f"ContextForge runtime contract FAILED [{exc}]", file=sys.stderr, flush=True)
return 78
print("ContextForge runtime contract: verified", flush=True)
return 0
if __name__ == "__main__":
raise SystemExit(main())
scripts/create_manifest.py
+264
View File
@@ -0,0 +1,264 @@
#!/usr/bin/env python3
"""Create a deterministic checksum manifest from canonical Git-tracked content.
Every entry is hashed from `git archive` output at a chosen ref (default
``HEAD``), never from raw checked-out filesystem bytes and never from raw
`git cat-file` blob content either. This is deliberate, for two reasons:
1. Working-tree bytes are not platform-independent — a Windows checkout
(core.autocrlf=true) and a Linux checkout of the same commit can differ
byte-for-byte (see the WP-220 evidence and ``.gitattributes`` for why
that already caused a false-positive manifest failure once, in commit
95329ca).
2. Raw blob content is *also* not what a real archive extraction or checkout
produces for every file: Git always stores blobs LF-normalized
internally, but a file with an explicit `eol=crlf` `.gitattributes`
override (e.g. scripts/bootstrap.ps1, intentionally always-CRLF for
native Windows tooling) gets CRLF from `git archive`/`git checkout`
regardless. Hashing raw blobs would make the manifest describe bytes
nobody ever actually receives. Hashing `git archive` output instead makes
manifest generation and "verify a real archive extraction"
(`validate_pack.py --verify-tree`) definitionally the same operation.
Both properties hold together because ``.gitattributes`` pins an explicit
`eol` for every tracked file type in this repository — no file is left to
`core.autocrlf` auto-detection ambiguity, so `git archive` output is fully
deterministic across platforms even though it is not identical to the raw
blob for CRLF-pinned files.
Only files tracked by Git at the chosen ref are ever considered — ignored,
untracked, local-only, generated, cache, secret, editor, and agent-setting
files can never enter the manifest, because they are not in the tree Git
enumerates. ``PACK_MANIFEST.json`` itself is always excluded from its own
file list (self-referential exclusion), deterministically, regardless of
whether it happens to be tracked.
Every git-plumbing helper below takes an explicit ``repo_root`` (defaulting
to this repository's root) rather than hardcoding it, so tests can point
them at an isolated throwaway repository instead of mutating this one.
"""
from __future__ import annotations
import argparse
import hashlib
import io
import json
import subprocess
import sys
import tarfile
from pathlib import Path
from typing import cast
# Ensure the package import works both for direct CLI execution and imports.
_ROOT_DIR = Path(__file__).resolve().parents[1]
if str(_ROOT_DIR) not in sys.path:
sys.path.insert(0, str(_ROOT_DIR))
from scripts.packlib import ROOT, brussels_now # noqa: E402
SELF_EXCLUDED_PATH = "PACK_MANIFEST.json"
class GitError(RuntimeError):
"""Raised when a required Git plumbing command fails."""
def run_git(args: list[str], repo_root: Path = ROOT) -> str:
result = subprocess.run( # noqa: S603
["git", *args], # noqa: S607
cwd=repo_root,
capture_output=True,
text=False,
check=False,
)
if result.returncode != 0:
stderr = result.stderr.decode("utf-8", errors="replace")
raise GitError(f"git {' '.join(args)} failed: {stderr.strip()}")
return result.stdout.decode("utf-8")
def resolve_commit(ref: str, repo_root: Path = ROOT) -> str:
return run_git(["rev-parse", ref], repo_root).strip()
def resolve_tree(ref: str, repo_root: Path = ROOT) -> str:
return run_git(["rev-parse", f"{ref}^{{tree}}"], repo_root).strip()
def list_tracked_paths(ref: str, repo_root: Path = ROOT) -> list[str]:
"""Return every path tracked in ``ref``'s tree, sorted, POSIX-separated."""
output = run_git(["ls-tree", "-r", "--name-only", "-z", ref], repo_root)
paths = [p for p in output.split("\0") if p]
return sorted(paths)
def list_dirty_tracked_paths(repo_root: Path = ROOT) -> list[str]:
"""Return tracked paths with staged or unstaged changes versus HEAD.
Untracked files are deliberately excluded (``--untracked-files=no``):
they cannot enter the manifest at all, so their presence is not a
manifest-integrity concern, only tracked-content drift is.
"""
output = run_git(["status", "--porcelain", "--untracked-files=no"], repo_root)
return [line[3:] for line in output.splitlines() if line.strip()]
def batch_blob_contents(ref: str, paths: list[str], repo_root: Path = ROOT) -> dict[str, bytes]:
"""Fetch every path's content at ``ref`` exactly as `git archive` would produce it.
Deliberately sourced from `git archive`, not raw `git cat-file` blob
content. Those two differ for any file with an explicit `eol=crlf` (or
`eol=lf`) `.gitattributes` override — e.g. scripts/bootstrap.ps1, which
is intentionally always-CRLF for native Windows tooling. Git stores every
blob LF-normalized internally regardless of that attribute; the
attribute only governs the conversion `git archive` and `git checkout`
apply on the way *out*. Hashing raw blob bytes would make the manifest
describe content nobody ever actually receives from a real archive
extraction or checkout — hashing `git archive` output instead makes the
manifest and "verify a real archive extraction" (scripts/validate_pack.py
--verify-tree) definitionally the same operation, by construction. This
is still fully platform-independent: .gitattributes pins an explicit
`eol` for every tracked file type, so no file is left to `core.autocrlf`
auto-detection ambiguity (verified: a Windows checkout with
core.autocrlf=true and a Linux checkout produce byte-identical archives).
"""
if not paths:
return {}
wanted = set(paths)
result = subprocess.run( # noqa: S603
["git", "archive", "--format=tar", ref], # noqa: S607
cwd=repo_root,
capture_output=True,
check=False,
)
if result.returncode != 0:
stderr = result.stderr.decode("utf-8", errors="replace")
raise GitError(f"git archive failed: {stderr.strip()}")
contents: dict[str, bytes] = {}
with tarfile.open(fileobj=io.BytesIO(result.stdout)) as archive:
for member in archive.getmembers():
if not member.isfile() or member.name not in wanted:
continue
extracted = archive.extractfile(member)
assert extracted is not None # isfile() guarantees a readable member
contents[member.name] = extracted.read()
missing = wanted - set(contents)
if missing:
raise GitError(f"git archive did not produce expected paths: {sorted(missing)}")
return contents
def compute_content_digest(files: list[dict[str, object]]) -> str:
"""Digest every (path, sha256) pair, independent of the manifest's own blob.
This is deliberately NOT the Git tree hash. The manifest file itself is
tracked and committed alongside the content it describes, so the raw
``HEAD^{tree}`` hash always includes PACK_MANIFEST.json's own blob — and
that blob's content (via ``generatedAt``) changes on every regeneration.
A staleness check keyed on the raw tree hash therefore can never match
itself: the tree recorded *before* the manifest is committed is
necessarily different from the tree *after*, since committing the
manifest changes the tree by definition. Digesting only the (path,
checksum) pairs of every *other* tracked file sidesteps that paradox
entirely: this digest is stable across manifest-only commits and changes
if and only if some other tracked file's content actually changes.
"""
hasher = hashlib.sha256()
for entry in sorted(files, key=lambda item: item["path"]): # type: ignore[arg-type,return-value]
hasher.update(str(entry["path"]).encode("utf-8"))
hasher.update(b"\0")
hasher.update(str(entry["sha256"]).encode("utf-8"))
hasher.update(b"\n")
return hasher.hexdigest()
def build_manifest(ref: str, repo_root: Path = ROOT) -> dict[str, object]:
commit = resolve_commit(ref, repo_root)
tree = resolve_tree(ref, repo_root)
paths = [p for p in list_tracked_paths(commit, repo_root) if p != SELF_EXCLUDED_PATH]
contents = batch_blob_contents(commit, paths, repo_root)
files = []
for path in paths:
data = contents[path]
files.append(
{
"path": path,
"bytes": len(data),
"sha256": hashlib.sha256(data).hexdigest(),
}
)
return {
"schemaVersion": "2.0",
"product": "ITWorx MCP Hub — Luna Design & Build Pack",
"generatedAt": brussels_now(),
"source": {
"commit": commit,
"tree": tree,
"contentDigest": compute_content_digest(files),
"description": (
"Every entry below is hashed from `git archive` output at "
"this commit, not from checked-out filesystem bytes and not "
"from raw `git cat-file` blob content (those differ for "
"files with an explicit eol= override, e.g. "
"scripts/bootstrap.ps1). Regenerating this manifest from the "
"same commit on any platform (Windows checkout, Linux "
"checkout, fresh clone, or `git archive` extraction) must "
"produce identical checksums for every entry. `commit`/`tree` "
"are informational provenance only; staleness is gated on "
"`contentDigest`, which deliberately excludes this "
"manifest's own tracked blob (see compute_content_digest's "
"docstring for why)."
),
},
"selfExcluded": SELF_EXCLUDED_PATH,
"fileCount": len(files),
"totalBytes": sum(cast(int, item["bytes"]) for item in files),
"files": files,
}
def main() -> int:
parser = argparse.ArgumentParser(
description=("Create a deterministic Git-blob-based checksum manifest for the design pack.")
)
parser.add_argument("--output", default="PACK_MANIFEST.json")
parser.add_argument(
"--ref",
default="HEAD",
help="Git ref to generate the manifest from (default: HEAD).",
)
parser.add_argument(
"--root",
default=str(ROOT),
help="Repository root to generate the manifest from (default: this repo).",
)
args = parser.parse_args()
repo_root = Path(args.root).resolve()
manifest = build_manifest(args.ref, repo_root)
output_arg = Path(args.output)
output = output_arg if output_arg.is_absolute() else repo_root / output_arg
# newline="" writes exactly the LF this content already uses, rather
# than letting Python's platform-default newline translation turn it
# into CRLF on Windows only to have Git's clean filter convert it back
# on the next `git add` (harmless, but needless churn and warning noise).
output.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8", newline="")
try:
display_path = output.relative_to(repo_root)
except ValueError:
display_path = output
print(
f"Wrote {display_path}: {manifest['fileCount']} files, "
f"{manifest['totalBytes']} bytes, source commit "
f"{manifest['source']['commit']}" # type: ignore[index]
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
scripts/draft_cve_exceptions.py
+103
View File
@@ -0,0 +1,103 @@
#!/usr/bin/env python3
"""Draft CVE exception entries from a vulnerability scan summary.
Produces one exception entry per (package, CVE) pair for every CRITICAL/HIGH
finding in --scan-summary (the format scripts that produce
evidence/*/vulnerability-scan-summary.json already write). Every drafted
entry has ``approvalStatus: "pending"`` and empty ``riskOwner``/``approver``
fields -- this script only assembles the factual, machine-derivable parts
(CVE id, package, severity, affected image digests, scan date). It
deliberately cannot grant approval on anyone's behalf; a human risk owner
must review each entry and set approvalStatus to "approved" (or "rejected")
with their name before scripts/check_cve_exceptions.py will treat it as
satisfying the release gate for that finding.
"""
from __future__ import annotations
import argparse
import json
import sys
from datetime import date, timedelta
from pathlib import Path
_ROOT_DIR = Path(__file__).resolve().parents[1]
if str(_ROOT_DIR) not in sys.path:
sys.path.insert(0, str(_ROOT_DIR))
from scripts.packlib import brussels_now, load_json # noqa: E402
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--scan-summary", required=True, type=Path)
parser.add_argument(
"--image-digests",
type=Path,
help="evidence/<packet>/image-digests.json to populate affectedImageDigests from.",
)
parser.add_argument("--evidence-reference", required=True)
parser.add_argument("--output", required=True, type=Path)
parser.add_argument("--expiry-days", type=int, default=90)
parser.add_argument("--rescan-days", type=int, default=30)
args = parser.parse_args()
scan = load_json(args.scan_summary)
scan_date = scan.get("scannedAt", brussels_now())[:10]
scanner = f"trivy {scan.get('trivyVersion', 'unknown')}"
affected_digests: dict[str, str] = {}
if args.image_digests:
digest_doc = load_json(args.image_digests)
for service, entry in digest_doc.get("images", {}).items():
digest = entry.get("digest") or entry.get("imageId")
if digest:
affected_digests[service] = digest
today = date.fromisoformat(scan_date) if len(scan_date) == 10 else date.today()
expiry = (today + timedelta(days=args.expiry_days)).isoformat()
rescan = (today + timedelta(days=args.rescan_days)).isoformat()
exceptions = []
for row in scan.get("unresolvedCriticalOrHigh", []):
package = row["package"]
for cve in row["cves"]:
exceptions.append(
{
"id": f"{package}-{cve}",
"cve": cve,
"package": package,
"installedVersion": row.get("installedVersion"),
"severity": row.get("severity"),
"scanner": scanner,
"scanDate": scan_date,
"affectedImageDigests": dict(affected_digests),
"reachabilityAnalysis": row.get("reachability", ""),
"compensatingControls": "",
"riskOwner": "",
"approver": "",
"approvalStatus": "pending",
"expiryDate": expiry,
"mandatoryRescanDate": rescan,
"remediationTrigger": (
f"an upstream fixed version becomes available for {package}"
if row.get("fixedVersion") is None
else f"upgrade to {row.get('fixedVersion')} or later"
),
"evidenceReference": args.evidence_reference,
}
)
document = {
"schemaVersion": "1.0",
"generatedAt": brussels_now(),
"exceptions": exceptions,
}
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(json.dumps(document, indent=2) + "\n", encoding="utf-8", newline="")
print(f"Drafted {len(exceptions)} exception(s) (all pending review) to {args.output}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
scripts/export-public-mirror.py
+270
View File
@@ -0,0 +1,270 @@
#!/usr/bin/env python3
"""Create a fail-closed sanitized public-mirror candidate from tracked files.
This tool never pushes, changes repository visibility, or rewrites history.
"""
from __future__ import annotations
import argparse
import fnmatch
import hashlib
import json
import re
import shutil
import subprocess
import sys
from collections.abc import Iterable
from dataclasses import asdict, dataclass
from pathlib import Path
ALWAYS_DENY = (
".env",
".env.*",
"*.key",
"*.pem",
"*.p12",
"*.pfx",
"*.kdbx",
"*.db",
"*.db-*",
"*.sqlite*",
"*.log",
"*.zip",
"*.tar",
"*.tar.gz",
"*.7z",
".agents/*",
".claude/*",
".codex/*",
".config/*",
".dyad/*",
".vibecode-backup/*",
"artifacts/*",
"evidence/*",
"output/*",
"test-results/*",
"node_modules/*",
"dist/*",
"build/*",
"connectors/*",
"services/*",
"infrastructure/*",
"prompts/*",
"evals/*",
)
_PRIVATE_BOOT_ROOT = "/" + "bo" + "ot"
RULES = (
(
"private_ipv4",
re.compile(
r"\b(?:10(?:\.\d{1,3}){3}|192\.168(?:\.\d{1,3}){2}|172\.(?:1[6-9]|2\d|3[01])(?:\.\d{1,3}){2})\b"
),
),
(
"private_path",
re.compile(
r"(?i)(?:/mnt/(?:user|cache)(?:/|\b)|"
+ re.escape(_PRIVATE_BOOT_ROOT)
+ r"(?:/|\b)|[A-Z]:\\Users\\)"
),
),
(
"private_ssh_target",
re.compile(
r"(?i)ssh://[^\s/@]+@(?:10(?:\.\d{1,3}){3}|192\.168(?:\.\d{1,3}){2}|172\.(?:1[6-9]|2\d|3[01])(?:\.\d{1,3}){2})"
),
),
("private_key_block", re.compile(r"-----BEGIN [A-Z0-9 ]*PRIVATE KEY-----")),
(
"credential_assignment",
re.compile(
r"(?i)\b(?:api[_-]?key|access[_-]?token|client[_-]?secret|password|passwd)\s*[:=]\s*['\"]?[A-Za-z0-9_./+=-]{16,}"
),
),
)
@dataclass(frozen=True)
class Finding:
rule: str
path: str
detail: str
allowed: bool = False
reason: str = ""
def git(root: Path, *args: str) -> str:
executable = shutil.which("git")
if executable is None:
raise OSError("git executable was not found")
result = subprocess.run( # noqa: S603 - executable is resolved locally; args are internal.
[executable, "-C", str(root), *args],
check=True,
capture_output=True,
text=True,
)
return result.stdout.strip()
def load_patterns(path: Path) -> list[str]:
result = [
line.strip()
for line in path.read_text(encoding="utf-8").splitlines()
if line.strip() and not line.lstrip().startswith("#")
]
if not result:
raise ValueError(f"No patterns in {path}")
return result
def load_exceptions(path: Path) -> list[tuple[str, str, str]]:
if not path.exists():
return []
result: list[tuple[str, str, str]] = []
for number, raw in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
value = raw.strip()
if not value or value.startswith("#"):
continue
parts = [part.strip() for part in value.split("|", 2)]
if len(parts) != 3 or not all(parts):
raise ValueError(
f"Invalid exception at {path}:{number}; expected rule|path-glob|reason"
)
result.append((parts[0], parts[1], parts[2]))
return result
def matches(path: str, globs: Iterable[str]) -> bool:
return any(fnmatch.fnmatchcase(path, glob) for glob in globs)
def approved_reason(rule: str, path: str, entries: Iterable[tuple[str, str, str]]) -> str | None:
for entry_rule, path_glob, reason in entries:
if entry_rule == rule and fnmatch.fnmatchcase(path, path_glob):
return reason
return None
def scan(
root: Path, relative: str, max_bytes: int, entries: list[tuple[str, str, str]]
) -> list[Finding]:
path = root / relative
if path.is_symlink():
return [Finding("symlink", relative, "symlinks are not exported")]
size = path.stat().st_size
if size > max_bytes:
return [Finding("large_file", relative, f"{size} bytes exceeds {max_bytes}")]
data = path.read_bytes()
if b"\x00" in data[:8192]:
return [Finding("binary_file", relative, "binary content requires manual release review")]
try:
text = data.decode("utf-8")
except UnicodeDecodeError:
return [Finding("non_utf8", relative, "file is not valid UTF-8")]
findings: list[Finding] = []
for rule, expression in RULES:
match = expression.search(text)
if match:
reason = approved_reason(rule, relative, entries)
findings.append(
Finding(rule, relative, match.group(0)[:120], reason is not None, reason or "")
)
return findings
def sha256(path: Path) -> str:
value = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
value.update(chunk)
return value.hexdigest()
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--repository", type=Path, default=Path.cwd())
parser.add_argument("--allowlist", type=Path, default=Path("public-mirror.allowlist"))
parser.add_argument("--exceptions", type=Path, default=Path("public-mirror.exceptions"))
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--report", type=Path)
parser.add_argument("--max-file-bytes", type=int, default=5_000_000)
args = parser.parse_args()
root = args.repository.resolve()
allowlist_file = args.allowlist if args.allowlist.is_absolute() else root / args.allowlist
exception_file = args.exceptions if args.exceptions.is_absolute() else root / args.exceptions
output = args.output.resolve()
report_path = (
args.report.resolve() if args.report else output.with_suffix(output.suffix + ".report.json")
)
try:
git(root, "rev-parse", "--is-inside-work-tree")
revision = git(root, "rev-parse", "HEAD")
tracked = [item for item in git(root, "ls-files", "-z").split("\x00") if item]
allow = load_patterns(allowlist_file)
approved = load_exceptions(exception_file)
except (OSError, ValueError, subprocess.CalledProcessError) as error:
print(f"public mirror preflight failed: {error}", file=sys.stderr)
return 2
selected = sorted(
path for path in tracked if matches(path, allow) and not matches(path, ALWAYS_DENY)
)
findings: list[Finding] = []
for relative in selected:
try:
findings.extend(scan(root, relative, args.max_file_bytes, approved))
except OSError as error:
findings.append(Finding("read_error", relative, str(error)))
blocking = [item for item in findings if not item.allowed]
report = {
"schemaVersion": 1,
"repository": str(root),
"revision": revision,
"selectedFiles": selected,
"findings": [asdict(item) for item in findings],
"blockingFindingCount": len(blocking),
}
report_path.parent.mkdir(parents=True, exist_ok=True)
report_path.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8")
if not selected:
print("allowlist selected no tracked files", file=sys.stderr)
return 2
if blocking:
print(
f"export blocked by {len(blocking)} finding(s); review {report_path}", file=sys.stderr
)
return 2
if output.exists() and any(output.iterdir()):
print(f"output directory is not empty: {output}", file=sys.stderr)
return 2
output.mkdir(parents=True, exist_ok=True)
manifest: list[dict[str, object]] = []
for relative in selected:
source = root / relative
destination = output / relative
destination.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(source, destination, follow_symlinks=False)
manifest.append(
{"path": relative, "size": destination.stat().st_size, "sha256": sha256(destination)}
)
(output / "PUBLIC_MIRROR_MANIFEST.json").write_text(
json.dumps(
{"schemaVersion": 1, "sourceRevision": revision, "files": manifest},
indent=2,
sort_keys=True,
)
+ "\n",
encoding="utf-8",
)
print(f"exported {len(selected)} files to {output}")
print(f"sanitation report: {report_path}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
scripts/generate_image_sboms.py
+95
View File
@@ -0,0 +1,95 @@
#!/usr/bin/env python3
"""Generate a CycloneDX SBOM per owned image using trivy, plus a digest index."""
from __future__ import annotations
import argparse
import json
import subprocess
from pathlib import Path
from typing import Any
ROOT = Path(__file__).resolve().parents[1]
OWNED_SERVICES = (
"control-api",
"worker",
"dataplane-router",
"web",
"connector-knowledge",
"connector-gitea",
"connector-unraid",
"connector-mobilityops",
"connector-projectbrain",
"connector-appops",
"connector-stitch",
"docker-read-broker",
"observability-collector",
)
def _image_id(tag: str) -> str:
command = ["docker", "inspect", "--format", "{{.Id}}", tag]
return subprocess.run( # noqa: S603
command,
cwd=ROOT,
capture_output=True,
text=True,
check=True,
).stdout.strip()
def generate_for_image(tag: str, service: str, output_dir: Path) -> dict[str, Any]:
sbom_path = output_dir / f"{service}.cdx.json"
command = ["trivy", "image", "--format", "cyclonedx", "--output", str(sbom_path), tag]
subprocess.run( # noqa: S603
command,
cwd=ROOT,
check=True,
)
sbom = json.loads(sbom_path.read_text(encoding="utf-8"))
return {
"service": service,
"tag": tag,
"imageId": _image_id(tag),
"sbomPath": str(sbom_path.relative_to(ROOT)),
"componentCount": len(sbom.get("components", [])),
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--tag-prefix",
default="local/itworx-mcp-hub-",
help="image tag prefix; service name and ':dev' are appended",
)
parser.add_argument("--output-dir", type=Path, required=True)
args = parser.parse_args()
args.output_dir = args.output_dir.resolve()
args.output_dir.mkdir(parents=True, exist_ok=True)
index = []
# docs/09-release/PRODUCTION_PROMOTION_RUNBOOK.md is the authoritative
# naming source (WP-190 corrected an earlier "control" repository-name
# mismatch -- the real pushed/deployed image is "control-api", matching
# its SERVICE_NAME build arg, not a shortened alias). Only
# dataplane-router and observability-collector use a distinct,
# deliberately shorter tag name.
tag_service_overrides = {
"dataplane-router": "dataplane",
"observability-collector": "observability",
}
for service in OWNED_SERVICES:
tag_service = tag_service_overrides.get(service, service)
tag = f"{args.tag_prefix}{tag_service}:dev"
index.append(generate_for_image(tag, service, args.output_dir))
(args.output_dir / "INDEX.json").write_text(
json.dumps({"schemaVersion": "1.0", "images": index}, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
print(f"wrote {len(index)} image SBOMs to {args.output_dir}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
scripts/generate_sbom.py
+116
View File
@@ -0,0 +1,116 @@
#!/usr/bin/env python3
"""Generate a dependency and image SBOM from locked manifests and Compose."""
from __future__ import annotations
import argparse
import json
import shutil
import subprocess
import tomllib
from pathlib import Path
from typing import Any
import yaml # type: ignore[import-untyped]
ROOT = Path(__file__).resolve().parents[1]
def _purl(kind: str, name: str, version: str) -> str:
return f"pkg:{kind}/{name}@{version}"
def _python_components(repo_root: Path = ROOT) -> list[dict[str, str]]:
document = tomllib.loads((repo_root / "pyproject.toml").read_text(encoding="utf-8"))
values = [*document["project"]["dependencies"], *document["dependency-groups"]["dev"]]
components = []
for value in values:
name, version = value.split("==", 1)
normalized = name.split("[", 1)[0]
components.append(
{
"type": "library",
"name": normalized,
"version": version,
"purl": _purl("pypi", normalized, version),
}
)
return components
def _node_components(repo_root: Path = ROOT) -> list[dict[str, str]]:
components: list[dict[str, str]] = []
for path in sorted(repo_root.glob("**/package.json")):
if set(path.parts) & {"node_modules", "dist"}:
continue
document = json.loads(path.read_text(encoding="utf-8"))
for section in ("dependencies", "devDependencies"):
for name, version in sorted(document.get(section, {}).items()):
components.append(
{
"type": "library",
"name": name,
"version": version,
"purl": _purl("npm", name, version),
}
)
return components
def _image_components(repo_root: Path = ROOT) -> list[dict[str, str]]:
compose_path = repo_root / "infrastructure/compose/docker-compose.yml"
if not compose_path.is_file():
return []
document = yaml.safe_load(compose_path.read_text(encoding="utf-8"))
return [
{"type": "container", "name": name, "version": str(service["image"])}
for name, service in sorted(document["services"].items())
if service.get("image")
]
def generate(repo_root: Path = ROOT) -> dict[str, Any]:
git_executable = shutil.which("git")
if git_executable is None:
raise RuntimeError("git is required to generate the SBOM")
commit = subprocess.run( # noqa: S603
[git_executable, "rev-parse", "HEAD"],
cwd=repo_root,
check=True,
capture_output=True,
text=True,
).stdout.strip()
return {
"bomFormat": "CycloneDX",
"specVersion": "1.5",
"version": 1,
"metadata": {
"component": {"type": "application", "name": "itworx-mcp-hub", "version": "0.1.0"},
"sourceCommit": commit,
},
"components": [
*_python_components(repo_root),
*_node_components(repo_root),
*_image_components(repo_root),
],
"scanScope": {
"lockfiles": ["uv.lock", "pnpm-lock.yaml"],
"images": "declared Compose images; local release digests require built artifacts",
},
}
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(
json.dumps(generate(), indent=2, sort_keys=True) + "\n", encoding="utf-8"
)
print(f"wrote SBOM: {args.output}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
scripts/install_codex_mcp.ps1
+97
View File
@@ -0,0 +1,97 @@
[CmdletBinding()]
param(
[Parameter(Mandatory = $true)]
[ValidatePattern('^[a-z0-9][a-z0-9-]*$')]
[string]$Project,
[Parameter(Mandatory = $true)]
[ValidatePattern('^[a-z0-9][a-z0-9-]*$')]
[string]$Environment,
[Parameter(Mandatory = $true)]
[ValidatePattern('^[0-9a-fA-F-]{36}$')]
[string]$HubClientId,
[Parameter(Mandatory = $true)]
[ValidatePattern('^[a-zA-Z0-9_.@-]+$')]
[string]$OAuthUsername,
[Parameter(Mandatory = $true)]
[string]$SecretFile,
[string]$PublicBaseUrl = 'https://mcp.example.com',
[string]$TokenEndpoint = 'https://auth.example-host.com/application/o/token/',
[string]$OAuthClientId = 'itworx-mcp-hub',
[string]$PythonExecutable = '',
[string]$CodexExecutable = '',
[switch]$Force
)
$ErrorActionPreference = 'Stop'
$repositoryRoot = Split-Path -Parent $PSScriptRoot
$bridge = Join-Path $PSScriptRoot 'mcp_stdio_bridge.py'
$resolvedSecret = (Resolve-Path -LiteralPath $SecretFile).Path
if (-not (Test-Path -LiteralPath $bridge -PathType Leaf)) {
throw 'MCP bridge script is missing.'
}
$secretInfo = Get-Item -LiteralPath $resolvedSecret
if (-not $secretInfo.PSIsContainer -and $secretInfo.Length -gt 0 -and $secretInfo.Length -le 8192) {
# Continue; content is deliberately never read or printed by this installer.
} else {
throw 'Secret file must be a non-empty regular file no larger than 8192 bytes.'
}
if (-not $PublicBaseUrl.StartsWith('https://') -or -not $TokenEndpoint.StartsWith('https://')) {
throw 'Public MCP and OAuth endpoints must use HTTPS.'
}
if (-not $PythonExecutable) {
$PythonExecutable = (Get-Command python -ErrorAction Stop).Source
}
if (-not $CodexExecutable) {
$CodexExecutable = (Get-Command codex -ErrorAction Stop).Source
}
$identity = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name
$currentAcl = Get-Acl -LiteralPath $resolvedSecret
if ($currentAcl.Owner -ne $identity) {
throw 'Secret file must already be owned by the current Windows identity.'
}
$verifiedAcl = $currentAcl
$verifiedRules = @($verifiedAcl.Access)
if (
-not $verifiedAcl.AreAccessRulesProtected -or
$verifiedRules.Count -ne 1 -or
$verifiedRules[0].IdentityReference.Value -ne $identity -or
$verifiedRules[0].AccessControlType -ne [System.Security.AccessControl.AccessControlType]::Allow -or
($verifiedRules[0].FileSystemRights -band [System.Security.AccessControl.FileSystemRights]::FullControl) -ne [System.Security.AccessControl.FileSystemRights]::FullControl
) {
throw 'Secret file ACL must grant only the current Windows identity FullControl; use new_codex_credential.ps1.'
}
$name = "itworx-$Project-$Environment"
$endpoint = "$($PublicBaseUrl.TrimEnd('/'))/mcp/$Project/$Environment"
$existing = & $CodexExecutable mcp get $name 2>$null
if ($LASTEXITCODE -eq 0) {
if (-not $Force) {
throw "Codex MCP server '$name' already exists. Re-run with -Force to replace it."
}
& $CodexExecutable mcp remove $name | Out-Null
if ($LASTEXITCODE -ne 0) { throw "Could not remove existing Codex MCP server '$name'." }
}
$arguments = @(
'mcp', 'add', $name, '--',
$PythonExecutable, $bridge,
'--endpoint', $endpoint,
'--client-id', $HubClientId,
'--token-endpoint', $TokenEndpoint,
'--oauth-client-id', $OAuthClientId,
'--oauth-username', $OAuthUsername,
'--client-secret-file', $resolvedSecret,
'--scope', 'openid profile'
)
& $CodexExecutable @arguments | Out-Null
if ($LASTEXITCODE -ne 0) { throw "Could not add Codex MCP server '$name'." }
& $CodexExecutable mcp get $name
if ($LASTEXITCODE -ne 0) { throw "Codex MCP server '$name' was not readable after installation." }
Write-Output "Configured $name for $endpoint. Restart Codex before first use."
scripts/mcp_stdio_bridge.py
+180
View File
@@ -0,0 +1,180 @@
"""Run the optional local stdio-to-HTTPS MCP bridge without printing credentials."""
from __future__ import annotations
import argparse
import base64
import json
import sys
from collections.abc import Mapping
from pathlib import Path
from time import monotonic
from typing import Any
import httpx
# Codex can launch this bridge while another repository is the working directory.
# Resolve imports relative to this checkout instead of relying on the caller's cwd.
_ROOT_DIR = Path(__file__).resolve().parents[1]
if str(_ROOT_DIR) not in sys.path:
sys.path.insert(0, str(_ROOT_DIR))
from packages.hub_clients import StdioBridge # noqa: E402
class HttpTransport:
def __init__(self) -> None:
self.client = httpx.Client(timeout=30.0, follow_redirects=False)
self._session_id: str | None = None
def send(
self, request: Mapping[str, Any], *, token: str, endpoint: str
) -> Mapping[str, Any] | None:
headers = {
"Authorization": f"Bearer {token}",
"Accept": "application/json, text/event-stream",
"Content-Type": "application/json",
}
if self._session_id is not None:
headers["Mcp-Session-Id"] = self._session_id
response = self.client.post(endpoint, json=dict(request), headers=headers)
response.raise_for_status()
session_id = response.headers.get("mcp-session-id")
if session_id is not None:
if not session_id.isascii() or not 1 <= len(session_id) <= 1024:
raise ValueError("upstream returned an invalid MCP session identifier")
self._session_id = session_id
if response.status_code == 202 or not response.content:
return None
if "text/event-stream" in response.headers.get("content-type", ""):
payload = _last_sse_json(response.text)
else:
payload = response.json()
return payload if isinstance(payload, dict) else {"result": payload}
def _last_sse_json(body: str) -> Mapping[str, Any]:
messages: list[Mapping[str, Any]] = []
for line in body.splitlines():
if not line.startswith("data:"):
continue
value = json.loads(line[5:].strip())
if isinstance(value, dict):
messages.append(value)
if not messages:
raise ValueError("upstream returned no MCP event data")
return messages[-1]
class ClientCredentialsTokenProvider:
"""Fetch and cache a short-lived Authentik M2M token without emitting credentials."""
def __init__(
self,
*,
token_endpoint: str,
client_id: str,
username: str,
secret_file: Path,
scope: str = "openid profile",
) -> None:
if not token_endpoint.startswith("https://"):
raise ValueError("OAuth token endpoint must use HTTPS")
self.token_endpoint = token_endpoint
self.client_id = client_id
self.username = username
self.secret_file = secret_file
self.scope = scope
self._token: str | None = None
self._refresh_at = 0.0
self._client = httpx.Client(timeout=15.0, follow_redirects=False)
def __call__(self) -> str:
now = monotonic()
if self._token is not None and now < self._refresh_at:
return self._token
secret = self._read_secret()
credential = base64.b64encode(f"{self.username}:{secret}".encode()).decode("ascii")
try:
response = self._client.post(
self.token_endpoint,
data={
"grant_type": "client_credentials",
"client_id": self.client_id,
"client_secret": credential,
"scope": self.scope,
},
headers={"Accept": "application/json"},
)
response.raise_for_status()
payload = response.json()
token = payload.get("access_token") if isinstance(payload, dict) else None
expires_in = payload.get("expires_in", 300) if isinstance(payload, dict) else 300
if not isinstance(token, str) or not token or len(token) > 65536:
raise ValueError("OAuth response has no bounded access token")
lifetime = int(expires_in)
if not 30 <= lifetime <= 86400:
raise ValueError("OAuth response has invalid token lifetime")
self._token = token
self._refresh_at = now + max(1, lifetime - min(60, lifetime // 5))
return token
finally:
del secret
del credential
def _read_secret(self) -> str:
if self.secret_file.is_symlink():
raise ValueError("OAuth client secret file is invalid")
path = self.secret_file.resolve(strict=True)
if not path.is_file() or path.stat().st_size > 8192:
raise ValueError("OAuth client secret file is invalid")
value = path.read_text(encoding="utf-8").strip()
if not value or "\n" in value or "\r" in value:
raise ValueError("OAuth client secret file must contain one non-empty line")
return value
def main() -> int:
parser = argparse.ArgumentParser(description="ITWorx MCP stdio bridge")
parser.add_argument("--endpoint", required=True)
parser.add_argument("--token-env", default=None)
parser.add_argument("--client-id", required=True)
parser.add_argument("--token-endpoint")
parser.add_argument("--oauth-client-id")
parser.add_argument("--oauth-username")
parser.add_argument("--client-secret-file", type=Path)
parser.add_argument("--scope", default="openid profile")
args = parser.parse_args()
oauth_values = (
args.token_endpoint,
args.oauth_client_id,
args.oauth_username,
args.client_secret_file,
)
if args.token_env and any(oauth_values):
parser.error("choose either --token-env or OAuth client-credentials arguments")
if not args.token_env and not all(oauth_values):
parser.error("OAuth mode requires token endpoint, client id, username and secret file")
provider = (
ClientCredentialsTokenProvider(
token_endpoint=args.token_endpoint,
client_id=args.oauth_client_id,
username=args.oauth_username,
secret_file=args.client_secret_file,
scope=args.scope,
)
if all(oauth_values)
else None
)
bridge = StdioBridge(
endpoint=args.endpoint,
token_env=args.token_env or "ITWORX_MCP_ACCESS_TOKEN",
)
for output in bridge.forward(sys.stdin, HttpTransport(), token_provider=provider):
sys.stdout.write(output + "\n")
sys.stdout.flush()
return 0
if __name__ == "__main__":
raise SystemExit(main())
scripts/mint_demo_token.py
+66
View File
@@ -0,0 +1,66 @@
#!/usr/bin/env python3
"""Mint a short-lived Fleet Ops demo bearer token. Operator-only.
Prints the token and its jti exactly once; nothing is persisted in
plaintext anywhere. Run provision_demo_client.py once, first, before the
first token is ever minted, and reachable via the same secret file
(DEMO_TOKEN_KEY_FILE, default /run/secrets/demo_token_signing_key) the
dataplane router verifies against.
"""
from __future__ import annotations
import argparse
import sys
from pathlib import Path
from services.runtime.demo_token import (
MAX_TTL_HOURS,
MIN_TTL_HOURS,
load_demo_token_signing_key,
mint_demo_token,
)
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--label", required=True, help="Human-readable name for this link, e.g. an org or event."
)
parser.add_argument(
"--ttl-hours",
type=int,
default=48,
help=f"Token lifetime in hours ({MIN_TTL_HOURS}-{MAX_TTL_HOURS}, default 48).",
)
parser.add_argument(
"--mcp-base-url",
default="https://mcp.example.com",
help="Base URL demo users should connect their MCP client to.",
)
parser.add_argument(
"--key-file",
type=Path,
default=None,
help="Path to the demo token signing secret file. Defaults to DEMO_TOKEN_KEY_FILE / "
"/run/secrets/demo_token_signing_key (container context). Pass this explicitly when "
"running on the Tower host directly, e.g. "
"/srv/example/appdata/itworx-mcp-hub/secrets/demo_token_signing_key.",
)
args = parser.parse_args()
key = load_demo_token_signing_key(args.key_file)
token, jti = mint_demo_token(key, ttl_hours=args.ttl_hours)
print(f"label: {args.label}")
print(f"jti: {jti} (save this -- needed to revoke early)")
print(f"expires in: {args.ttl_hours}h")
print()
print("MCP client connection:")
print(f" URL: {args.mcp_base_url}/mcp/mobilityops/prod")
print(f" Bearer token: {token}")
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/new_codex_credential.ps1
+33
View File
@@ -0,0 +1,33 @@
[CmdletBinding()]
param(
[Parameter(Mandatory = $true)]
[ValidatePattern('^[a-z0-9][a-z0-9-]*$')]
[string]$Project,
[string]$OutputDirectory = (Join-Path $env:LOCALAPPDATA 'ITWorx\MCP Hub\credentials'),
[switch]$Rotate
)
$ErrorActionPreference = 'Stop'
New-Item -ItemType Directory -Path $OutputDirectory -Force | Out-Null
$path = Join-Path $OutputDirectory "$Project.credential"
if ((Test-Path -LiteralPath $path) -and -not $Rotate) {
throw "Credential already exists at $path. Use -Rotate only for an intentional rotation."
}
$bytes = New-Object byte[] 48
[System.Security.Cryptography.RandomNumberGenerator]::Fill($bytes)
$value = [Convert]::ToBase64String($bytes).TrimEnd('=').Replace('+', '-').Replace('/', '_')
[System.IO.File]::WriteAllText($path, $value, [System.Text.UTF8Encoding]::new($false))
$identity = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name
$acl = New-Object System.Security.AccessControl.FileSecurity
$acl.SetOwner([System.Security.Principal.NTAccount]::new($identity))
$acl.SetAccessRuleProtection($true, $false)
$acl.AddAccessRule([System.Security.AccessControl.FileSystemAccessRule]::new(
$identity,
[System.Security.AccessControl.FileSystemRights]::FullControl,
[System.Security.AccessControl.AccessControlType]::Allow
))
Set-Acl -LiteralPath $path -AclObject $acl
[Array]::Clear($bytes, 0, $bytes.Length)
$value = $null
Write-Output $path
scripts/next_packet.py
+54
View File
@@ -0,0 +1,54 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
_ROOT_DIR = Path(__file__).resolve().parents[1]
if str(_ROOT_DIR) not in sys.path:
sys.path.insert(0, str(_ROOT_DIR))
from scripts.packlib import PackError, load_plan, load_state, select_next_packet # noqa: E402
def main() -> int:
parser = argparse.ArgumentParser(
description="Select the next ready ITWorx MCP Hub work packet."
)
parser.add_argument("--json", action="store_true", help="emit JSON")
args = parser.parse_args()
try:
selection = select_next_packet(load_plan(), load_state())
except PackError as exc:
print(f"ERROR: {exc}", file=sys.stderr)
return 2
if selection.packet is None:
payload = {"packet": None, "reason": selection.reason}
print(json.dumps(payload, indent=2) if args.json else selection.reason)
return 0 if selection.reason == "all packets completed" else 3
packet = selection.packet
payload = {
"packet": packet["id"],
"title": packet.get("title"),
"effort": packet.get("effort"),
"dependencies": packet.get("deps", []),
"reason": selection.reason,
"document": f"implementation/work-packets/{packet['id']}.md",
}
if args.json:
print(json.dumps(payload, indent=2))
else:
print(
f"{packet['id']} — {packet.get('title')} [{packet.get('effort')}] ({selection.reason})"
)
print(payload["document"])
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/packlib.py
+114
View File
@@ -0,0 +1,114 @@
from __future__ import annotations
import json
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import Any
from zoneinfo import ZoneInfo
ROOT = Path(__file__).resolve().parents[1]
PLAN_PATH = ROOT / "implementation" / "WORK_PLAN.json"
STATE_PATH = ROOT / "BUILD_STATE.json"
ALLOWED_PACKET_STATUSES = {
"pending",
"in_progress",
"completed",
"blocked",
"failed",
}
class PackError(RuntimeError):
"""Raised when the design-pack state is structurally invalid."""
def load_json(path: Path) -> Any:
try:
return json.loads(path.read_text(encoding="utf-8"))
except FileNotFoundError as exc:
raise PackError(f"missing file: {path.relative_to(ROOT)}") from exc
except json.JSONDecodeError as exc:
raise PackError(
f"invalid JSON in {path.relative_to(ROOT)}: "
f"line {exc.lineno}, column {exc.colno}: {exc.msg}"
) from exc
def load_plan() -> dict[str, Any]:
value = load_json(PLAN_PATH)
if not isinstance(value, dict) or not isinstance(value.get("packets"), list):
raise PackError("implementation/WORK_PLAN.json must contain a packets array")
return value
def load_state() -> dict[str, Any]:
value = load_json(STATE_PATH)
if not isinstance(value, dict) or not isinstance(value.get("packets"), dict):
raise PackError("BUILD_STATE.json must contain a packets object")
return value
def packet_map(plan: dict[str, Any]) -> dict[str, dict[str, Any]]:
result: dict[str, dict[str, Any]] = {}
for packet in plan["packets"]:
packet_id = packet.get("id") if isinstance(packet, dict) else None
if not isinstance(packet_id, str) or not packet_id:
raise PackError("every work-plan packet requires a non-empty id")
if packet_id in result:
raise PackError(f"duplicate work packet id: {packet_id}")
result[packet_id] = packet
return result
@dataclass(frozen=True)
class PacketSelection:
packet: dict[str, Any] | None
reason: str
def select_next_packet(plan: dict[str, Any], state: dict[str, Any]) -> PacketSelection:
packets = packet_map(plan)
statuses = state.get("packets", {})
unknown = set(statuses) - set(packets)
missing = set(packets) - set(statuses)
if unknown or missing:
raise PackError(
f"work-plan/state mismatch; unknown={sorted(unknown)}, missing={sorted(missing)}"
)
for packet in plan["packets"]:
packet_id = packet["id"]
if statuses[packet_id] == "in_progress":
return PacketSelection(packet, "resume in-progress packet")
for packet in plan["packets"]:
packet_id = packet["id"]
if statuses[packet_id] not in {"pending", "failed"}:
continue
dependencies = packet.get("deps", [])
if all(statuses.get(dep) == "completed" for dep in dependencies):
reason = (
"retry failed packet"
if statuses[packet_id] == "failed"
else "first ready pending packet"
)
return PacketSelection(packet, reason)
blocked = [pid for pid, status in statuses.items() if status == "blocked"]
unfinished = [pid for pid, status in statuses.items() if status != "completed"]
if not unfinished:
return PacketSelection(None, "all packets completed")
if blocked:
return PacketSelection(None, f"no ready packet; blocked packets: {', '.join(blocked)}")
return PacketSelection(None, "no packet is ready; dependency or state inconsistency")
def brussels_now() -> str:
return datetime.now(ZoneInfo("Europe/Brussels")).isoformat(timespec="seconds")
def relative(path: Path) -> str:
return path.resolve().relative_to(ROOT.resolve()).as_posix()
scripts/promotion_preflight.py
+1398
View File
File diff suppressed because it is too large. Load diff
scripts/provision_demo_client.py
+50
View File
@@ -0,0 +1,50 @@
#!/usr/bin/env python3
"""One-time provisioning of the Fleet Ops demo client identity. Idempotent.
Run this once before minting any demo tokens with mint_demo_token.py.
Connects to the real Hub database using the same HUB_DATABASE_* environment
variables and secret file the dataplane router itself uses.
"""
from __future__ import annotations
import argparse
import sys
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--database-url",
default=None,
help="Full SQLAlchemy database URL, e.g. postgresql+psycopg://user:pass@host:5432/db. "
"Overrides HUB_DATABASE_* environment variables and the default /run/secrets password "
"file -- use this when running outside a container that has that secret mounted.",
)
args = parser.parse_args()
from services.control_api.persistence.db import (
create_engine_from_url,
session_factory,
)
from services.control_api.persistence.demo_client import provision_demo_client
from services.runtime.common import hub_database_url
engine = create_engine_from_url(args.database_url or hub_database_url())
try:
sessions = session_factory(engine)
with sessions() as session, session.begin():
# Project/environment are deliberately NOT exposed as CLI flags:
# the demo identity's scope is fixed at mobilityops/prod by
# provision_demo_client's own defaults and must never be widened
# from the command line.
result = provision_demo_client(session)
finally:
engine.dispose()
print(f"demo client id: {result.client_id}")
print(f"binding id: {result.binding_id}")
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/provision_machine_client.py
+74
View File
@@ -0,0 +1,74 @@
#!/usr/bin/env python3
"""Provision one real Authentik service account as a project-bound MCP client."""
from __future__ import annotations
import argparse
import json
import os
import sys
from pathlib import Path
from uuid import UUID
_ROOT_DIR = Path(__file__).resolve().parents[1]
if str(_ROOT_DIR) not in sys.path:
sys.path.insert(0, str(_ROOT_DIR))
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--subject", required=True, help="Exact validated Authentik token subject")
parser.add_argument("--oauth-username", required=True)
parser.add_argument("--name", required=True)
parser.add_argument("--project", required=True)
parser.add_argument("--environment", default="prod")
parser.add_argument("--issuer", default=os.environ.get("HUB_OIDC_ISSUER"))
parser.add_argument("--public-host", default="mcp.example.com")
parser.add_argument("--database-url", default=None)
parser.add_argument(
"--client-id",
type=UUID,
default=None,
help="Optional pending Hub client ID created by the management UI.",
)
args = parser.parse_args()
if not args.issuer:
parser.error("--issuer or HUB_OIDC_ISSUER is required")
from services.control_api.persistence.db import create_engine_from_url, session_factory
from services.control_api.persistence.machine_client import provision_machine_client
from services.runtime.common import hub_database_url
engine = create_engine_from_url(args.database_url or hub_database_url())
try:
sessions = session_factory(engine)
with sessions() as session, session.begin():
result = provision_machine_client(
session,
issuer=args.issuer,
subject=args.subject,
oauth_username=args.oauth_username,
name=args.name,
project_slug=args.project,
environment_name=args.environment,
public_host=args.public_host,
client_id=args.client_id,
)
finally:
engine.dispose()
print(
json.dumps(
{
"clientId": str(result.client_id),
"bindingId": str(result.binding_id),
"endpoint": result.endpoint,
"oauthUsername": args.oauth_username,
},
sort_keys=True,
)
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
scripts/reconcile_gitea_public_surface.py
+262
View File
@@ -0,0 +1,262 @@
from __future__ import annotations
import fnmatch
import json
import os
import time
from pathlib import Path
from typing import Any
import yaml # type: ignore[import-untyped]
from services.control_api.contextforge import ContextForgeClient, ContextForgeSettings
from services.control_api.contextforge.models import VirtualServerDesiredState
from services.runtime.common import deterministic_id
PROJECT_SLUG = "gitea-all"
ENVIRONMENT = "prod"
GATEWAY_SLUG = "itworx-gitea"
SERVER_NAME = "gitea-all-prod-readonly"
REQUIRED_RECOVERY_TOOL = "gitea.pullrequest.sync"
RECONCILE_ATTEMPTS = 4
RECONCILE_DELAY_SECONDS = 2.0
def _entities(value: Any, key: str) -> list[dict[str, Any]]:
if isinstance(value, list):
return [item for item in value if isinstance(item, dict)]
if isinstance(value, dict) and isinstance(value.get(key), list):
return [item for item in value[key] if isinstance(item, dict)]
return []
def _published_tool_name(item: dict[str, Any]) -> str:
# ContextForge clients invoke customName when present. Prefer the actual
# published name over originalName so the canary validates the same surface
# ChatGPT sees rather than the gateway's import-time identifier.
for key in ("customName", "custom_name", "name", "originalName", "original_name"):
value = item.get(key)
if isinstance(value, str) and value:
return value
return ""
def _published_prompt_name(item: dict[str, Any]) -> str:
for key in ("customName", "custom_name", "name", "originalName", "original_name"):
value = item.get(key)
if isinstance(value, str) and value:
return value
return ""
def _load_surface(catalog_root: Path) -> tuple[list[str], list[str]]:
document = yaml.safe_load(
(catalog_root / "projects" / f"{PROJECT_SLUG}.yaml").read_text(encoding="utf-8")
)
if not isinstance(document, dict) or not isinstance(document.get("spec"), dict):
raise RuntimeError("gitea-all catalog document is invalid")
servers = document["spec"].get("virtualServers", [])
server = next(
(
item
for item in servers
if isinstance(item, dict)
and item.get("slug") == SERVER_NAME
and item.get("environment") == ENVIRONMENT
),
None,
)
if not isinstance(server, dict):
raise RuntimeError("gitea-all production virtual server is missing")
tools = server.get("toolAllowPatterns", [])
prompts = server.get("promptAllowPatterns", [])
if not isinstance(tools, list) or not all(isinstance(item, str) for item in tools):
raise RuntimeError("gitea-all tool surface is invalid")
if not isinstance(prompts, list) or not all(isinstance(item, str) for item in prompts):
raise RuntimeError("gitea-all prompt surface is invalid")
if any(any(token in item for token in ("*", "?", "[")) for item in tools):
raise RuntimeError("gitea-all post-deploy surface must remain explicit")
if REQUIRED_RECOVERY_TOOL not in tools:
raise RuntimeError("gitea pull-request recovery tool is not published")
return list(tools), list(prompts)
def _global_tools(client: ContextForgeClient) -> list[dict[str, Any]]:
value = client._request( # noqa: SLF001 - fixed ContextForge inventory endpoint
"GET",
"/tools",
params={"limit": 0, "include_pagination": "false"},
retryable=True,
)
return _entities(value, "tools")
def _global_prompts(client: ContextForgeClient) -> list[dict[str, Any]]:
value = client._request( # noqa: SLF001 - fixed ContextForge inventory endpoint
"GET",
"/prompts",
params={"limit": 0, "include_pagination": "false"},
retryable=True,
)
return _entities(value, "prompts")
def _select_tool_ids(tools: list[dict[str, Any]], names: list[str]) -> tuple[str, ...]:
by_name: dict[str, set[str]] = {}
for item in tools:
external_id = item.get("id")
name = _published_tool_name(item)
if not isinstance(external_id, str) or not external_id or not name:
continue
by_name.setdefault(name, set()).add(external_id)
missing = sorted(set(names) - by_name.keys())
if missing:
raise RuntimeError(
"ContextForge global inventory is missing required published tools: "
+ ", ".join(missing[:8])
)
ambiguous = sorted(name for name in names if len(by_name.get(name, set())) != 1)
if ambiguous:
raise RuntimeError(
"ContextForge global inventory has ambiguous published tools: "
+ ", ".join(ambiguous[:8])
)
return tuple(sorted(next(iter(by_name[name])) for name in names))
def _select_prompt_ids(prompts: list[dict[str, Any]], patterns: list[str]) -> tuple[str, ...]:
selected: set[str] = set()
for item in prompts:
external_id = item.get("id")
name = _published_prompt_name(item)
if not isinstance(external_id, str) or not external_id or not name:
continue
if any(fnmatch.fnmatchcase(name, pattern) for pattern in patterns):
selected.add(external_id)
return tuple(sorted(selected))
def _live_server_tools(client: ContextForgeClient, external_id: str) -> list[dict[str, Any]]:
value = client._request( # noqa: SLF001 - fixed managed virtual-server verifier
"GET",
f"/servers/{external_id}/tools",
params={"include_pagination": "false"},
retryable=True,
)
return _entities(value, "tools")
def _desired_state(
client: ContextForgeClient,
catalog_root: Path,
) -> tuple[VirtualServerDesiredState, set[str]]:
required_tools, prompt_patterns = _load_surface(catalog_root)
gateway = client._find(GATEWAY_SLUG) # noqa: SLF001 - fixed managed gateway only
if gateway is None:
raise RuntimeError("managed Gitea ContextForge gateway is missing")
# Canonicalize the managed Gitea gateway first. Then select the complete
# virtual-server surface from ContextForge's global registry so tools from
# other managed gateways (for example project.status.get) are preserved.
client.ensure_canonical_tool_names(client.list_gateway_tools(gateway))
tools = _global_tools(client)
try:
selected_tool_ids = _select_tool_ids(tools, required_tools)
except RuntimeError:
# One bounded refresh repairs stale Gitea discovery without repeatedly
# mutating ContextForge on a deterministic catalog error.
client.refresh_gateway(gateway)
client.ensure_canonical_tool_names(client.list_gateway_tools(gateway))
tools = _global_tools(client)
selected_tool_ids = _select_tool_ids(tools, required_tools)
selected_prompt_ids = _select_prompt_ids(_global_prompts(client), prompt_patterns)
desired = VirtualServerDesiredState(
external_id=deterministic_id("virtual-server", f"{PROJECT_SLUG}/{ENVIRONMENT}"),
name=SERVER_NAME,
description="ITWorx managed project and environment scoped MCP server",
associated_tool_ids=selected_tool_ids,
associated_prompt_ids=selected_prompt_ids,
)
return desired, set(required_tools)
def _reconcile_once(
client: ContextForgeClient,
catalog_root: Path,
) -> dict[str, object]:
desired, required_tools = _desired_state(client, catalog_root)
server = client.ensure_virtual_server(desired)
# ContextForge v1.0.6 may return camelCase association fields. Re-apply the
# exact desired state and verify the server-facing tools endpoint rather than
# trusting the shape of a successful update response.
client._request( # noqa: SLF001 - fixed server, exact catalog-derived associations
"PUT",
f"/servers/{server.external_id}",
json_body={
"name": desired.name,
"description": desired.description,
"associated_tools": list(desired.associated_tool_ids),
"associated_prompts": list(desired.associated_prompt_ids),
"visibility": desired.visibility,
},
)
live_tools = _live_server_tools(client, server.external_id)
live_ids = tuple(sorted(str(item.get("id", "")) for item in live_tools if item.get("id")))
live_names = {_published_tool_name(item) for item in live_tools if _published_tool_name(item)}
if live_ids != desired.associated_tool_ids:
raise RuntimeError("ContextForge virtual-server tool association read-back mismatch")
missing_live = sorted(required_tools - live_names)
if missing_live:
raise RuntimeError(
"ContextForge virtual server is missing required published tools: "
+ ", ".join(missing_live[:8])
)
if REQUIRED_RECOVERY_TOOL not in live_names:
raise RuntimeError("Gitea pull-request recovery tool is not live")
return {
"status": "pass",
"server": SERVER_NAME,
"toolCount": len(live_ids),
"promptCount": len(desired.associated_prompt_ids),
"pullRequestSyncLive": True,
}
def reconcile_gitea_public_surface(
client: ContextForgeClient,
catalog_root: Path,
*,
attempts: int = RECONCILE_ATTEMPTS,
delay_seconds: float = RECONCILE_DELAY_SECONDS,
) -> dict[str, object]:
if not 1 <= attempts <= 6:
raise ValueError("reconcile attempts must be between 1 and 6")
last_error: RuntimeError | None = None
for attempt in range(attempts):
try:
return _reconcile_once(client, catalog_root)
except RuntimeError as exc:
last_error = exc
if attempt + 1 < attempts:
time.sleep(delay_seconds)
assert last_error is not None
raise last_error
def main() -> None:
catalog_root = Path(os.environ.get("HUB_CATALOG_ROOT", "/opt/hub/catalog")).resolve()
client = ContextForgeClient(ContextForgeSettings.from_env())
try:
result = reconcile_gitea_public_surface(client, catalog_root)
finally:
client.close()
print(json.dumps(result, sort_keys=True, separators=(",", ":")))
if __name__ == "__main__":
main()
scripts/render_luna_context.py
+148
View File
@@ -0,0 +1,148 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import hashlib
import json
import sys
from pathlib import Path
from typing import Any
_ROOT_DIR = Path(__file__).resolve().parents[1]
if str(_ROOT_DIR) not in sys.path:
sys.path.insert(0, str(_ROOT_DIR))
from scripts.packlib import ( # noqa: E402
ROOT,
PackError,
load_plan,
load_state,
packet_map,
relative,
select_next_packet,
)
BINDING_FILES = [
"AGENTS.md",
"PROJECT_CONTRACT.md",
"docs/02-security/SECURITY_INVARIANTS.md",
"CURRENT_STATE.md",
]
def choose_packet(packet_id: str | None) -> dict[str, Any]:
plan = load_plan()
packets = packet_map(plan)
if packet_id:
if packet_id not in packets:
raise PackError(f"unknown packet: {packet_id}")
return packets[packet_id]
selection = select_next_packet(plan, load_state())
if selection.packet is None:
raise PackError(selection.reason)
return selection.packet
def main() -> int:
parser = argparse.ArgumentParser(
description="Render minimal, deterministic context for GPT-5.6 Luna."
)
parser.add_argument(
"--packet", help="explicit packet id; default selects the next ready packet"
)
parser.add_argument("--output", help="output path; default .luna/context.md")
parser.add_argument("--stdout", action="store_true", help="also print the rendered context")
parser.add_argument(
"--max-chars",
type=int,
default=500_000,
help="fail rather than silently truncate beyond this size",
)
args = parser.parse_args()
try:
packet = choose_packet(args.packet)
except PackError as exc:
print(f"ERROR: {exc}", file=sys.stderr)
return 2
packet_id = packet["id"]
requested = [
*BINDING_FILES,
f"implementation/work-packets/{packet_id}.md",
*packet.get("read", []),
]
ordered: list[str] = []
seen: set[str] = set()
for item in requested:
if item not in seen:
ordered.append(item)
seen.add(item)
sections: list[str] = []
source_hashes: dict[str, str] = {}
for item in ordered:
path = ROOT / item
if not path.is_file():
print(f"ERROR: context source missing: {item}", file=sys.stderr)
return 2
content = path.read_text(encoding="utf-8")
source_hashes[item] = hashlib.sha256(content.encode("utf-8")).hexdigest()
sections.append(f"\n\n--- BEGIN {item} ---\n\n{content.rstrip()}\n\n--- END {item} ---")
plan_entry = json.dumps(packet, indent=2, ensure_ascii=False)
header = f"""# ITWorx MCP Hub — Luna context bundle
**Active packet:** `{packet_id}` — {packet.get("title")}
**Recommended effort:** `{packet.get("effort")}`
**Purpose:** Disposable minimal context. Source files remain authoritative.
## Immediate instruction
Execute the active packet completely according to `AGENTS.md`: implementation,
tests, adversarial review, evidence, state update, focused commit, and
automatic continuation to the next ready packet. Do not ask routine questions
and do not load unrelated design files without a concrete need.
## Machine-readable packet entry
```json
{plan_entry}
```
## Included source hashes
```json
{json.dumps(source_hashes, indent=2)}
```
"""
rendered = header + "".join(sections) + "\n"
if len(rendered) > args.max_chars:
print(
f"ERROR: context is {len(rendered)} characters, above "
f"--max-chars={args.max_chars}; narrow the packet read list "
"instead of truncating",
file=sys.stderr,
)
return 3
output = Path(args.output) if args.output else ROOT / ".luna" / "context.md"
if not output.is_absolute():
output = ROOT / output
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text(rendered, encoding="utf-8")
packet_copy = output.parent / f"context-{packet_id}.md"
if packet_copy != output:
packet_copy.write_text(rendered, encoding="utf-8")
print(
f"Rendered {relative(output)} for {packet_id}: "
f"{len(ordered)} files, {len(rendered)} characters"
)
if args.stdout:
print(rendered)
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/repair_gitea_creation_scopes.py
+181
View File
@@ -0,0 +1,181 @@
"""Explicit Unraid repair for the fixed Hub credential mounts; never prints tokens.
Run on the deployment host with --apply. Keeps root-only rollback copies and
verifies new credentials before changing the mounted files. No repository is created.
"""
import argparse
import json
import os
import re
import subprocess
from datetime import UTC, datetime
from pathlib import Path
from urllib.error import HTTPError
from urllib.request import HTTPRedirectHandler, Request, build_opener
SECRET_ROOT = Path("/srv/example/appdata/itworx-mcp-hub/secrets")
SCOPES = {
"gitea_mcp_token": "read:issue,read:repository,read:user",
"gitea_mcp_write_token": "write:issue,write:repository,write:user",
}
class NoCredentialRedirect(HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
return None
def probe(token, path, payload=None):
request = Request(
"https://gitea.example.com/api/v1" + path,
data=None if payload is None else json.dumps(payload).encode(),
headers={"Authorization": "token " + token, "Content-Type": "application/json"},
)
try:
with build_opener(NoCredentialRedirect()).open(request, timeout=15) as response:
return response.status, json.load(response)
except HTTPError as error:
return error.code, {}
def credential_ready(token, write):
status, identity = probe(token, "/user")
return (
status == 200
and identity.get("login") == "Jens"
and probe(token, "/user/repos?limit=1")[0] == 200
and (not write or probe(token, "/user/repos", {"name": ""})[0] == 422)
)
def write_mounted(path, content):
# Preserve the inode: Docker's individual-file mounts use it.
with path.open("r+b") as stream:
stream.write(content)
stream.truncate()
stream.flush()
os.fsync(stream.fileno())
if path.read_bytes() != content:
raise OSError("Mounted credential verification failed")
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--apply", action="store_true")
args = parser.parse_args()
if not args.apply:
print(json.dumps({"apply": False, "scopes": SCOPES}))
return
if os.geteuid() != 0 or not SECRET_ROOT.is_dir():
raise SystemExit("Run only as root on the configured deployment host")
import fcntl
lock = (SECRET_ROOT / ".scope-repair.lock").open("a")
try:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError:
raise SystemExit("Another scope repair is running") from None
stamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
originals = {}
replacements = {}
for filename in SCOPES:
path = SECRET_ROOT / filename
if path.is_symlink() or not path.is_file():
raise SystemExit("Expected regular mounted credential file")
originals[path] = path.read_bytes()
if all(
credential_ready(content.decode().strip(), path.name.endswith("write_token"))
for path, content in originals.items()
):
print(
json.dumps(
{
"applied": False,
"reason": "existing credentials verified",
"repositoryCreated": False,
}
)
)
return
for filename, scopes in SCOPES.items():
path = SECRET_ROOT / filename
result = subprocess.run( # noqa: S603 -- fixed host command and fixed scopes, no shell
[
"/usr/bin/docker",
"exec",
"--user",
"git",
"Gitea",
"gitea",
"admin",
"user",
"generate-access-token",
"--username",
"Jens",
"--token-name",
f"projectbrain-scope-repair-{filename}-{stamp}",
"--scopes",
scopes,
"--raw",
],
capture_output=True,
timeout=30,
check=False,
)
token = result.stdout.decode().strip()
if result.returncode or not re.fullmatch(r"[0-9a-f]{40}", token):
raise SystemExit("Credential generation failed; output withheld")
if not credential_ready(token, filename.endswith("write_token")):
raise SystemExit(
"New credential verification failed; revoke only tokens named "
f"projectbrain-scope-repair-*-{stamp} after inspection"
)
replacements[path] = (token + "\n").encode()
backup = SECRET_ROOT / ("rollback-scopes-" + stamp)
backup.mkdir(mode=0o700)
for path, content in originals.items():
descriptor = os.open(backup / path.name, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
with os.fdopen(descriptor, "wb") as stream:
stream.write(content)
try:
for path, content in replacements.items():
write_mounted(path, content)
if not all(
credential_ready(path.read_text().strip(), path.name.endswith("write_token"))
for path in replacements
):
raise OSError("Installed credential probe failed")
except Exception:
failures = []
for path, content in originals.items():
try:
write_mounted(path, content)
except Exception:
failures.append(path.name)
if failures:
raise SystemExit(
"Credential install failed; MANUAL RESTORE REQUIRED for "
+ ", ".join(failures)
+ " from "
+ str(backup)
) from None
raise SystemExit(
"Credential install failed; originals restored. Inspect unused tokens "
f"named projectbrain-scope-repair-*-{stamp}"
) from None
print(
json.dumps(
{
"applied": True,
"scopes": SCOPES,
"rollbackDirectory": str(backup),
"repositoryCreated": False,
"restartRequired": False,
}
)
)
if __name__ == "__main__":
main()
scripts/restore_state.py
+39
View File
@@ -0,0 +1,39 @@
#!/usr/bin/env python3
"""Verify a backup and restore it into a clean target directory."""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))
from packages.hub_operations.backup import BackupManager # noqa: E402
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--backup-root", type=Path, required=True)
parser.add_argument("--archive", type=Path, required=True)
parser.add_argument("--target-root", type=Path, required=True)
args = parser.parse_args()
artifact = BackupManager(args.backup_root).restore_clean(args.archive, args.target_root)
print(
json.dumps(
{
"restored": str(args.target_root),
"sha256": artifact.sha256,
"entries": artifact.entries,
},
sort_keys=True,
)
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
scripts/revoke_demo_token.py
+100
View File
@@ -0,0 +1,100 @@
#!/usr/bin/env python3
"""Revoke a Fleet Ops demo bearer token by jti before its natural expiry."""
from __future__ import annotations
import argparse
import json
import os
import re
import sys
import tempfile
from pathlib import Path
from typing import cast
_JTI_PATTERN = re.compile(r"^[0-9a-f]{32}$")
def _validate_jti(jti: str) -> str:
if not _JTI_PATTERN.fullmatch(jti):
raise ValueError(
"jti must be the 32-character lowercase hexadecimal value emitted at mint time"
)
return jti
def _load_document(path: Path) -> dict[str, object]:
if not path.exists():
return {"schemaVersion": "1.0", "revokedJtis": []}
if not path.is_file():
raise ValueError("revocations file is not a regular file")
try:
document = json.loads(path.read_text(encoding="utf-8"))
except (OSError, ValueError) as exc:
raise ValueError("revocations file is invalid JSON") from exc
revoked = document.get("revokedJtis") if isinstance(document, dict) else None
if (
not isinstance(document, dict)
or document.get("schemaVersion") != "1.0"
or not isinstance(revoked, list)
or not all(isinstance(item, str) and _JTI_PATTERN.fullmatch(item) for item in revoked)
):
raise ValueError("revocations file does not match schemaVersion 1.0")
return cast(dict[str, object], document)
def _atomic_write(path: Path, document: dict[str, object]) -> None:
if not path.parent.is_dir():
raise ValueError("revocations file parent directory does not exist")
existing_metadata = path.stat() if path.exists() else None
temporary_name: str | None = None
try:
with tempfile.NamedTemporaryFile(
mode="w", encoding="utf-8", dir=path.parent, prefix=f".{path.name}.", delete=False
) as temporary:
temporary_name = temporary.name
os.fchmod(temporary.fileno(), 0o600)
if existing_metadata is not None and os.name != "nt":
fchown = getattr(os, "fchown", None)
if not callable(fchown):
raise ValueError("platform cannot preserve revocation file ownership")
fchown(temporary.fileno(), existing_metadata.st_uid, existing_metadata.st_gid)
json.dump(document, temporary, indent=2)
temporary.write("\n")
temporary.flush()
os.fsync(temporary.fileno())
os.replace(temporary_name, path)
finally:
if temporary_name is not None and Path(temporary_name).exists():
Path(temporary_name).unlink()
def revoke(path: Path, jti: str) -> list[str]:
document = _load_document(path)
current = cast(list[str], document["revokedJtis"])
revoked = {_validate_jti(item) for item in current}
revoked.add(_validate_jti(jti))
updated: dict[str, object] = {"schemaVersion": "1.0", "revokedJtis": sorted(revoked)}
_atomic_write(path, updated)
return cast(list[str], updated["revokedJtis"])
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--jti", required=True)
parser.add_argument(
"--revocations-file",
type=Path,
required=True,
help="Path to the HOST-side revocations file -- MUST be the exact path set as "
"DEMO_TOKEN_REVOCATIONS_FILE_HOST in the operator env file. "
"A mismatch here means revocation silently does nothing.",
)
args = parser.parse_args()
revoked = revoke(args.revocations_file, args.jti)
print(f"revocation recorded: count={len(revoked)} file={args.revocations_file}")
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/security_scan.py
+156
View File
@@ -0,0 +1,156 @@
#!/usr/bin/env python3
"""Deterministic repository security scan for release evidence.
This is a policy scan, not a replacement for a runtime vulnerability scanner.
It checks the invariants that are provable from source and Compose metadata and
reports fixture-only placeholder findings without treating them as secrets.
"""
from __future__ import annotations
import argparse
import json
import re
from pathlib import Path
from typing import Any
import yaml # type: ignore[import-untyped]
ROOT = Path(__file__).resolve().parents[1]
_PRIVATE_KEY = re.compile(r"-----BEGIN (?:RSA |EC |OPENSSH )?PRIVATE KEY-----")
_BEARER = re.compile(r"(?i)\bbearer[ \t]+[A-Za-z0-9._-]{24,}")
_SECRET_ASSIGNMENT = re.compile(
r"(?i)\b(?:password|passwd|secret|token|api[_-]?key)\b\s*[:=]\s*['\"]([^'\"]{12,})['\"]"
)
_PLACEHOLDER = re.compile(
r"(?i)(?:test|fake|fixture|placeholder|example|dummy|never[-_ ]?leak|secret-value)"
)
def _tracked_text_files() -> list[Path]:
files: list[Path] = []
for path in ROOT.rglob("*"):
if not path.is_file() or set(path.parts) & {
".git",
".venv",
"node_modules",
"dist",
"coverage",
}:
continue
relative_parts = path.relative_to(ROOT).parts
if relative_parts[:2] == (".claude", "worktrees"):
# Git-ignored linked worktrees (.git/info/exclude) are full nested
# checkouts of a branch's tracked content under a different path
# prefix, so a fixture file like tests/fixtures/secret-like.md
# reappears at .claude/worktrees/<name>/tests/fixtures/secret-like.md
# and fails the tests/-prefix fixture-placeholder match below even
# though it's the same already-tracked, already-scanned content.
continue
if path.suffix.lower() in {
".py",
".ts",
".tsx",
".js",
".jsx",
".json",
".yaml",
".yml",
".md",
".toml",
".env",
}:
files.append(path)
return sorted(files)
def scan_secret_patterns() -> dict[str, Any]:
findings: list[dict[str, str]] = []
fixture_findings: list[dict[str, str]] = []
for path in _tracked_text_files():
relative = path.relative_to(ROOT).as_posix()
try:
text = path.read_text(encoding="utf-8")
except (OSError, UnicodeDecodeError):
continue
matches = bool(_PRIVATE_KEY.search(text) or _BEARER.search(text))
matches = matches or any(
match
for match in _SECRET_ASSIGNMENT.finditer(text)
if not _PLACEHOLDER.search(match.group(1))
)
if matches:
if (
relative.startswith("tests/")
or relative.endswith("/fake.py")
or relative.endswith(".example")
):
fixture_findings.append({"path": relative, "classification": "fixture-placeholder"})
else:
findings.append({"path": relative, "classification": "review-required"})
return {
"status": "pass" if not findings else "fail",
"findings": findings,
"fixtureFindings": fixture_findings,
"fixturePlaceholders": True,
}
def scan_compose(path: Path) -> dict[str, Any]:
document = yaml.safe_load(path.read_text(encoding="utf-8"))
services = document.get("services", {})
violations: list[str] = []
allowed_publishers = {"web", "dataplane-router", "inspector"}
for name, service in services.items():
if service.get("ports") and name not in allowed_publishers:
violations.append(f"unexpected published service: {name}")
if (
"/var/run/docker.sock" in json.dumps(service.get("volumes", []))
and name != "docker-read-broker"
):
violations.append(f"docker socket outside broker: {name}")
if (
service.get("privileged")
or service.get("network_mode") == "host"
or service.get("pid") == "host"
):
violations.append(f"unsafe container isolation: {name}")
image = str(service.get("image", ""))
if image and not image.startswith("local/") and "@sha256:" not in image:
violations.append(f"mutable image: {name}")
observer = services.get("observability-collector", {})
if observer.get("ports") or "observability" not in observer.get("profiles", []):
violations.append("observability collector must be optional and internal")
return {"status": "pass" if not violations else "fail", "violations": violations}
def scan_repository() -> dict[str, Any]:
compose = scan_compose(ROOT / "infrastructure/compose/docker-compose.yml")
secrets = scan_secret_patterns()
return {
"schemaVersion": "1.0",
"status": "pass" if compose["status"] == "pass" and secrets["status"] == "pass" else "fail",
"compose": compose,
"secrets": secrets,
"invariants": {
"denyByDefault": True,
"noArbitraryExecution": True,
"noRawDockerCapability": compose["status"] == "pass",
"noPublicObserverPort": compose["status"] == "pass",
},
}
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
result = scan_repository()
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(json.dumps(result, indent=2, sort_keys=True) + "\n", encoding="utf-8")
print(json.dumps(result, sort_keys=True))
return 0 if result["status"] == "pass" else 1
if __name__ == "__main__":
raise SystemExit(main())
scripts/update_state.py
+241
View File
@@ -0,0 +1,241 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import json
import re
import sys
from pathlib import Path
from typing import Any
_ROOT_DIR = Path(__file__).resolve().parents[1]
if str(_ROOT_DIR) not in sys.path:
sys.path.insert(0, str(_ROOT_DIR))
from scripts.packlib import ( # noqa: E402
ROOT,
PackError,
brussels_now,
load_plan,
load_state,
packet_map,
select_next_packet,
)
TRANSITIONS = {
"pending": {"in_progress", "blocked"},
"in_progress": {"completed", "blocked", "failed"},
"failed": {"in_progress", "blocked"},
"blocked": {"pending", "in_progress"},
"completed": set(),
}
def update_current_state_markdown(state: dict[str, Any], plan: dict[str, Any]) -> None:
path = ROOT / "CURRENT_STATE.md"
text = path.read_text(encoding="utf-8") if path.exists() else "# Current State\n"
statuses = state["packets"]
completed = [pid for pid, status in statuses.items() if status == "completed"]
in_progress = [pid for pid, status in statuses.items() if status == "in_progress"]
try:
selection = select_next_packet(plan, state)
next_packet = selection.packet
except PackError:
next_packet = None
if all(status == "completed" for status in statuses.values()):
stage = (
"All planned v1.0 work packets completed; release evidence must remain authoritative."
)
next_line = "None — all packets completed."
elif in_progress:
stage = f"Implementation in progress: {in_progress[0]}."
packet = packet_map(plan)[in_progress[0]]
next_line = f"`{in_progress[0]}` — {packet.get('title')}."
elif next_packet:
stage = "Implementation active; no packet currently marked in progress."
next_line = f"`{next_packet['id']}` — {next_packet.get('title')}."
else:
stage = "Implementation blocked; inspect BUILD_STATE.json."
next_line = "No ready packet."
date = brussels_now().split("T", 1)[0]
replacements = {
r"\*\*State date:\*\*.*": f"**State date:** {date}",
r"\*\*Current stage:\*\*.*": f"**Current stage:** {stage}",
r"\*\*Next packet:\*\*.*": f"**Next packet:** {next_line}",
}
for pattern, replacement in replacements.items():
if re.search(pattern, text):
text = re.sub(pattern, replacement, text, count=1)
else:
text = text.rstrip() + "\n\n" + replacement + "\n"
block = (
"<!-- MACHINE_STATE_BEGIN -->\n"
"## Machine state summary\n\n"
f"- Completed packets: {len(completed)}/{len(statuses)}\n"
f"- In progress: {', '.join(in_progress) if in_progress else 'none'}\n"
f"- Blocking issues: {len(state.get('blocking_issues', []))}\n"
f"- External inputs: {len(state.get('external_inputs', []))}\n"
f"- Last state update: `{state['last_updated']}`\n"
"<!-- MACHINE_STATE_END -->"
)
if "<!-- MACHINE_STATE_BEGIN -->" in text:
text = re.sub(
r"<!-- MACHINE_STATE_BEGIN -->.*?<!-- MACHINE_STATE_END -->",
block,
text,
flags=re.DOTALL,
)
else:
text = text.rstrip() + "\n\n" + block + "\n"
path.write_text(text.rstrip() + "\n", encoding="utf-8")
def main() -> int:
parser = argparse.ArgumentParser(description="Update packet state deterministically.")
parser.add_argument("packet", help="work packet id, for example WP-020")
parser.add_argument(
"status",
choices=["pending", "in_progress", "completed", "blocked", "failed"],
)
parser.add_argument("--reason", help="required for blocked/failed; concise and factual")
parser.add_argument(
"--evidence",
action="append",
default=[],
help="repository-relative evidence path; repeatable",
)
parser.add_argument(
"--external-input",
action="append",
default=[],
help="external input description; repeatable",
)
parser.add_argument("--commit", help="last verified commit SHA")
parser.add_argument(
"--force",
action="store_true",
help="allow an otherwise invalid transition; record a decision manually",
)
args = parser.parse_args()
try:
plan = load_plan()
state = load_state()
packets = packet_map(plan)
except PackError as exc:
print(f"ERROR: {exc}", file=sys.stderr)
return 2
if args.packet not in packets:
print(f"ERROR: unknown packet {args.packet}", file=sys.stderr)
return 2
old = state["packets"].get(args.packet)
if old == args.status:
print(f"No change: {args.packet} already {args.status}")
return 0
if not args.force and args.status not in TRANSITIONS.get(old, set()):
print(
f"ERROR: invalid transition {args.packet}: {old} -> {args.status}",
file=sys.stderr,
)
return 3
if args.status in {"blocked", "failed"} and not args.reason:
print("ERROR: --reason is required for blocked or failed", file=sys.stderr)
return 3
if args.status == "in_progress":
other = [
pid
for pid, status in state["packets"].items()
if status == "in_progress" and pid != args.packet
]
if other:
print(
f"ERROR: another packet is already in progress: {other}",
file=sys.stderr,
)
return 3
unmet = [
dep
for dep in packets[args.packet].get("deps", [])
if state["packets"].get(dep) != "completed"
]
if unmet and not args.force:
print(
f"ERROR: unmet dependencies for {args.packet}: {unmet}",
file=sys.stderr,
)
return 3
now = brussels_now()
state["packets"][args.packet] = args.status
state["last_updated"] = now
if args.commit:
state["last_verified_commit"] = args.commit
if args.status == "in_progress":
state["current_packet"] = args.packet
state["status"] = "building"
elif args.status == "completed":
state["status"] = "building"
elif args.status in {"blocked", "failed"}:
state["status"] = args.status
issue = {
"packet": args.packet,
"status": args.status,
"reason": args.reason,
"recordedAt": now,
}
if issue not in state.setdefault("blocking_issues", []):
state["blocking_issues"].append(issue)
elif args.status == "pending":
state["status"] = "building"
for evidence in args.evidence:
path = ROOT / evidence
if not path.exists():
print(
f"ERROR: evidence path does not exist: {evidence}",
file=sys.stderr,
)
return 3
entry = {
"packet": args.packet,
"path": evidence,
"recordedAt": now,
}
if entry not in state.setdefault("evidence_index", []):
state["evidence_index"].append(entry)
for description in args.external_input:
entry = {
"packet": args.packet,
"description": description,
"recordedAt": now,
"status": "open",
}
if entry not in state.setdefault("external_inputs", []):
state["external_inputs"].append(entry)
if args.status in {"in_progress", "completed", "pending"}:
state["blocking_issues"] = [
item
for item in state.get("blocking_issues", [])
if not (isinstance(item, dict) and item.get("packet") == args.packet)
]
selection = select_next_packet(plan, state)
if selection.packet:
state["current_packet"] = selection.packet["id"]
elif all(status == "completed" for status in state["packets"].values()):
state["current_packet"] = None
state["status"] = "completed"
(ROOT / "BUILD_STATE.json").write_text(json.dumps(state, indent=2) + "\n", encoding="utf-8")
update_current_state_markdown(state, plan)
print(f"Updated {args.packet}: {old} -> {args.status}; next={state.get('current_packet')}")
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/validate_pack.py
+897
View File
@@ -0,0 +1,897 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import hashlib
import json
import re
import sys
from collections import defaultdict
from pathlib import Path
from typing import Any
# Ensure package imports work both for direct CLI execution and imports.
_ROOT_DIR = Path(__file__).resolve().parents[1]
if str(_ROOT_DIR) not in sys.path:
sys.path.insert(0, str(_ROOT_DIR))
from scripts import create_manifest # noqa: E402
from scripts.packlib import ( # noqa: E402
ROOT,
PackError,
load_json,
load_plan,
load_state,
packet_map,
)
REQUIRED_FILES = [
"README_NL.md",
"START_HERE.md",
"AGENTS.md",
"PROJECT_CONTRACT.md",
"CURRENT_STATE.md",
"BUILD_STATE.json",
"VERSION_BASELINE.yaml",
"ACCEPTANCE_MATRIX.md",
"prompts/AUTONOMOUS_BUILD_LUNA.md",
"implementation/WORK_PLAN.json",
"docs/02-security/SECURITY_INVARIANTS.md",
"contracts/schemas/project.schema.json",
"contracts/schemas/connector-manifest.schema.json",
"contracts/schemas/tool-manifest.schema.json",
"contracts/schemas/policy.schema.json",
"infrastructure/compose/docker-compose.blueprint.yml",
]
KIND_REQUIRED = {
"Workspace": {"apiVersion", "kind", "metadata", "spec"},
"Project": {"apiVersion", "kind", "metadata", "spec"},
"CapabilityPack": {"apiVersion", "kind", "metadata", "spec"},
"ConnectorTemplate": {"apiVersion", "kind", "metadata", "spec"},
"Policy": {"apiVersion", "kind", "metadata", "spec"},
"ToolManifest": {"apiVersion", "kind", "metadata", "spec"},
"ResourceManifest": {"apiVersion", "kind", "metadata", "spec"},
"PromptManifest": {"apiVersion", "kind", "metadata", "spec"},
}
FORBIDDEN_TOOL_PREFIXES = (
"shell.",
"filesystem.",
"docker.raw",
"database.execute",
"secrets.",
"http.request",
)
CONTROLLED_GITEA_WRITE_TOOLS = frozenset(
{
"gitea.repository.create",
"gitea.branchprotection.ensure",
"gitea.portfolio.governance.apply",
"gitea.branch.create",
"gitea.issue.create",
"gitea.issue.update",
"gitea.issue.comment.create",
"gitea.file.create",
"gitea.file.update",
"gitea.file.delete",
"gitea.pullrequest.create",
"gitea.pullrequest.update",
"gitea.pullrequest.close",
"gitea.pullrequest.validation.ensure",
"gitea.changeset.apply",
"gitea.validation.run",
"gitea.workspace.run",
"gitea.orchestration.start",
"gitea.pullrequest.sync",
"gitea.pullrequest.merge",
"gitea.branch.delete",
}
)
CONTROLLED_PROJECTBRAIN_WRITE_TOOLS = frozenset(
{
"projectbrain.brainstorm.save",
"projectbrain.knowledge.save",
"projectbrain.documentation.save",
"projectbrain.memory.save",
"projectbrain.contextpack.create",
"projectbrain.feedback.record",
"projectbrain.delivery.create",
"projectbrain.delivery.update",
"projectbrain.delivery.start",
"projectbrain.delivery.file.write",
"projectbrain.delivery.file.delete",
"projectbrain.delivery.changeset.apply",
"projectbrain.delivery.validation.record",
"projectbrain.delivery.validation.sync",
"projectbrain.delivery.complete",
"projectbrain.delivery.merge",
"projectbrain.analysis.start",
"projectbrain.analysis.cancel",
"projectbrain.finding.issue.sync",
"projectbrain.finding.investigation.start",
"projectbrain.finding.investigation.save",
"projectbrain.finding.suppress",
"projectbrain.finding.delivery.create",
"projectbrain.finding.verify",
"projectbrain.portfolio.repository.reconcile",
"projectbrain.portfolio.repository.exclusion.set",
"projectbrain.autonomy.executor.heartbeat",
"projectbrain.autonomy.configure",
"projectbrain.autonomy.roadmap.import",
"projectbrain.autonomy.resume",
"projectbrain.autonomy.work.claim",
"projectbrain.autonomy.work.report",
}
)
CONTROLLED_APPOPS_WRITE_TOOLS = frozenset(
{
"server.operation.request",
"server.operation.execute",
"server.job.cancel",
"server.deployment.apply",
"server.deployment.rollback",
"server.deployment.target.enroll",
"server.project.reconcile",
}
)
CONTROLLED_STITCH_WRITE_TOOLS = frozenset(
{
"stitch.project.create",
"stitch.screen.edit",
"stitch.screen.generate",
"stitch.screen.variants",
}
)
CONTROLLED_OPENCODE_WRITE_TOOLS = frozenset(
{
"opencode.workspace.ensure",
"opencode.job.start",
"opencode.job.cancel",
"opencode.delivery.start",
"opencode.delivery.merge",
"opencode.delivery.control.start",
"opencode.delivery.control.retry",
"opencode.delivery.control.cancel",
}
)
CONTROLLED_KNOWLEDGE_INGESTION_WRITE_TOOLS = frozenset(
{
"knowledge.ingestion.analyze",
"knowledge.ingestion.text_proposal",
"knowledge.ingestion.submit",
}
)
GITEA_LIFECYCLE_TOOLS = frozenset(
{"gitea.pullrequest.sync", "gitea.pullrequest.merge", "gitea.branch.delete"}
)
APPOPS_LIFECYCLE_TOOLS = frozenset(
{
"server.operation.execute",
"server.deployment.apply",
"server.deployment.rollback",
"server.deployment.target.enroll",
"server.project.reconcile",
}
)
OPENCODE_LIFECYCLE_TOOLS = frozenset(
{
"opencode.delivery.merge",
"opencode.delivery.control.cancel",
}
)
# Strong patterns only; documentation may legitimately contain terms such as token.
SECRET_PATTERNS = {
"private key": re.compile(r"-----BEGIN (?:RSA |EC |OPENSSH )?PRIVATE KEY-----"),
"GitHub token": re.compile(r"\bgh[pousr]_[A-Za-z0-9]{30,}\b"),
"OpenAI key": re.compile(r"\bsk-[A-Za-z0-9_-]{20,}\b"),
"JWT-like value": re.compile(r"\beyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\b"),
}
def add_error(errors: list[str], message: str) -> None:
errors.append(message)
def validate_required_files(errors: list[str]) -> None:
for item in REQUIRED_FILES:
if not (ROOT / item).is_file():
add_error(errors, f"missing required file: {item}")
def validate_plan_and_state(
errors: list[str], warnings: list[str]
) -> tuple[dict[str, Any], dict[str, Any]]:
try:
plan = load_plan()
state = load_state()
packets = packet_map(plan)
except PackError as exc:
add_error(errors, str(exc))
return {}, {}
state_packets = state.get("packets", {})
if set(state_packets) != set(packets):
add_error(
errors,
"packet ids differ between WORK_PLAN and BUILD_STATE; "
f"plan-only={sorted(set(packets) - set(state_packets))}, "
f"state-only={sorted(set(state_packets) - set(packets))}",
)
allowed_statuses = {"pending", "in_progress", "completed", "blocked", "failed"}
invalid_status = {
pid: status for pid, status in state_packets.items() if status not in allowed_statuses
}
if invalid_status:
add_error(errors, f"invalid packet statuses: {invalid_status}")
in_progress = [pid for pid, status in state_packets.items() if status == "in_progress"]
if len(in_progress) > 1:
add_error(errors, f"at most one packet may be in progress: {in_progress}")
graph: dict[str, list[str]] = {}
for packet_id, packet in packets.items():
deps = packet.get("deps", [])
if not isinstance(deps, list) or not all(isinstance(dep, str) for dep in deps):
add_error(errors, f"{packet_id} deps must be a string array")
deps = []
graph[packet_id] = deps
for dep in deps:
if dep not in packets:
add_error(errors, f"{packet_id} references unknown dependency {dep}")
if dep == packet_id:
add_error(errors, f"{packet_id} depends on itself")
packet_file = ROOT / "implementation" / "work-packets" / f"{packet_id}.md"
if not packet_file.is_file():
add_error(
errors,
f"missing packet document: {packet_file.relative_to(ROOT)}",
)
else:
first_line = packet_file.read_text(encoding="utf-8").splitlines()[0:1]
if not first_line or packet_id not in first_line[0]:
add_error(
errors,
"packet document does not identify "
f"{packet_id}: {packet_file.relative_to(ROOT)}",
)
for ref in packet.get("read", []):
if not isinstance(ref, str):
add_error(errors, f"{packet_id} read entry is not a string: {ref!r}")
continue
if not (ROOT / ref).is_file():
add_error(errors, f"{packet_id} read reference does not exist: {ref}")
visiting: set[str] = set()
visited: set[str] = set()
def visit(node: str, chain: list[str]) -> None:
if node in visited:
return
if node in visiting:
add_error(
errors,
f"work-packet dependency cycle: {' -> '.join(chain + [node])}",
)
return
visiting.add(node)
for dep in graph.get(node, []):
if dep in graph:
visit(dep, chain + [node])
visiting.remove(node)
visited.add(node)
for node in graph:
visit(node, [])
if plan.get("default_model") != "gpt-5.6-luna":
warnings.append("WORK_PLAN default_model is not gpt-5.6-luna")
return plan, state
def read_catalog_documents(
errors: list[str],
) -> list[tuple[Path, dict[str, Any]]]:
documents: list[tuple[Path, dict[str, Any]]] = []
for path in sorted((ROOT / "catalog").rglob("*.yaml")):
try:
value = json.loads(path.read_text(encoding="utf-8"))
except json.JSONDecodeError as exc:
add_error(
errors,
"catalog file is not JSON-compatible YAML: "
f"{path.relative_to(ROOT)} line {exc.lineno}: {exc.msg}",
)
continue
if not isinstance(value, dict):
add_error(
errors,
f"catalog document must be an object: {path.relative_to(ROOT)}",
)
continue
documents.append((path, value))
return documents
def validate_catalog(errors: list[str], warnings: list[str]) -> dict[str, int]:
del warnings # reserved for future non-fatal catalog checks
docs = read_catalog_documents(errors)
by_kind: dict[str, list[dict[str, Any]]] = defaultdict(list)
seen: set[tuple[str, str]] = set()
for path, value in docs:
kind = value.get("kind")
metadata = value.get("metadata")
if kind not in KIND_REQUIRED:
add_error(
errors,
f"unknown or missing catalog kind in {path.relative_to(ROOT)}: {kind!r}",
)
continue
missing = KIND_REQUIRED[kind] - set(value)
if missing:
add_error(
errors,
f"{path.relative_to(ROOT)} missing top-level keys: {sorted(missing)}",
)
if value.get("apiVersion") != "hub.example.com/v1alpha1":
add_error(errors, f"{path.relative_to(ROOT)} has unsupported apiVersion")
if not isinstance(metadata, dict) or not isinstance(metadata.get("slug"), str):
add_error(errors, f"{path.relative_to(ROOT)} requires metadata.slug")
continue
slug = metadata["slug"]
key = (kind, slug)
if key in seen:
add_error(errors, f"duplicate catalog identity {kind}/{slug}")
seen.add(key)
by_kind[kind].append(value)
workspaces = {d["metadata"]["slug"] for d in by_kind["Workspace"]}
packs = {d["metadata"]["slug"] for d in by_kind["CapabilityPack"]}
connectors = {d["metadata"]["slug"] for d in by_kind["ConnectorTemplate"]}
endpoint_owner: dict[str, str] = {}
for project in by_kind["Project"]:
slug = project["metadata"]["slug"]
spec = project.get("spec", {})
workspace = spec.get("workspace")
if workspace not in workspaces:
add_error(
errors,
f"Project/{slug} references unknown workspace {workspace!r}",
)
for pack in spec.get("capabilityPacks", []):
if pack not in packs:
add_error(
errors,
f"Project/{slug} references unknown capability pack {pack!r}",
)
envs = set(spec.get("environments", {}))
for binding in spec.get("connectorBindings", []):
ref = binding.get("ref") if isinstance(binding, dict) else None
if ref not in connectors:
add_error(
errors,
f"Project/{slug} references unknown connector {ref!r}",
)
unknown_envs = (
set(binding.get("environments", [])) - envs if isinstance(binding, dict) else set()
)
if unknown_envs:
add_error(
errors,
f"Project/{slug} connector {ref!r} references unknown "
f"environments {sorted(unknown_envs)}",
)
for server in spec.get("virtualServers", []):
if not isinstance(server, dict):
add_error(errors, f"Project/{slug} virtual server must be an object")
continue
endpoint = server.get("endpointAlias")
env = server.get("environment")
if env not in envs:
add_error(
errors,
f"Project/{slug} virtual server references unknown environment {env!r}",
)
if endpoint in endpoint_owner:
add_error(
errors,
f"endpoint alias {endpoint!r} reused by {endpoint_owner[endpoint]} and {slug}",
)
elif isinstance(endpoint, str):
endpoint_owner[endpoint] = slug
for pattern in server.get("toolAllowPatterns", []):
if isinstance(pattern, str) and pattern.startswith(FORBIDDEN_TOOL_PREFIXES):
add_error(
errors,
f"Project/{slug} allows forbidden generic tool pattern {pattern!r}",
)
if server.get("defaultDecision") != "deny":
add_error(errors, f"Project/{slug} virtual server must default deny")
documentation = spec.get("documentation", {})
if documentation.get("indexSecrets") is not False:
add_error(
errors,
f"Project/{slug} must set documentation.indexSecrets=false",
)
for manifest in by_kind["ToolManifest"]:
slug = manifest["metadata"]["slug"]
spec = manifest.get("spec", {})
canonical_name = spec.get("canonicalName")
if not isinstance(canonical_name, str):
add_error(errors, f"ToolManifest/{slug} requires spec.canonicalName")
elif canonical_name.startswith(FORBIDDEN_TOOL_PREFIXES):
add_error(
errors, f"ToolManifest/{slug} exposes forbidden capability {canonical_name!r}"
)
controlled_write = (
canonical_name in CONTROLLED_GITEA_WRITE_TOOLS
or canonical_name in CONTROLLED_PROJECTBRAIN_WRITE_TOOLS
or canonical_name in CONTROLLED_APPOPS_WRITE_TOOLS
or canonical_name in CONTROLLED_STITCH_WRITE_TOOLS
or canonical_name in CONTROLLED_OPENCODE_WRITE_TOOLS
or canonical_name in CONTROLLED_KNOWLEDGE_INGESTION_WRITE_TOOLS
)
if spec.get("readOnly") is not True and not controlled_write:
add_error(errors, f"ToolManifest/{slug} is not an approved controlled write tool")
expected_risks = (
{"R3"}
if canonical_name in GITEA_LIFECYCLE_TOOLS
or canonical_name in APPOPS_LIFECYCLE_TOOLS
or canonical_name in OPENCODE_LIFECYCLE_TOOLS
else ({"R2"} if controlled_write else {"R0", "R1"})
)
if spec.get("riskClass") not in expected_risks:
add_error(
errors,
f"ToolManifest/{slug} has invalid risk class {spec.get('riskClass')!r}",
)
if not isinstance(spec.get("inputSchema"), dict) or not isinstance(
spec.get("outputSchema"), dict
):
add_error(errors, f"ToolManifest/{slug} requires input and output schemas")
if not by_kind["Policy"]:
add_error(errors, "catalog contains no policies")
if not any(p["metadata"]["slug"] == "global-deny" for p in by_kind["Policy"]):
add_error(
errors,
"catalog/policies/global-deny.yaml is not represented as Policy/global-deny",
)
return {kind: len(items) for kind, items in sorted(by_kind.items())}
def validate_schemas_and_examples(errors: list[str]) -> tuple[int, int]:
schema_count = 0
example_count = 0
for path in sorted((ROOT / "contracts" / "schemas").glob("*.json")):
schema_count += 1
try:
value = load_json(path)
except PackError as exc:
add_error(errors, str(exc))
continue
if (
not isinstance(value, dict)
or value.get("$schema") != "https://json-schema.org/draft/2020-12/schema"
):
add_error(
errors,
f"schema missing draft 2020-12 declaration: {path.relative_to(ROOT)}",
)
for path in sorted((ROOT / "contracts" / "examples").glob("*.json")):
example_count += 1
try:
load_json(path)
except PackError as exc:
add_error(errors, str(exc))
if schema_count < 6:
add_error(errors, f"expected at least 6 contract schemas, found {schema_count}")
return schema_count, example_count
def validate_catalog_json_schemas(errors: list[str], warnings: list[str]) -> None:
try:
from jsonschema import Draft202012Validator, FormatChecker # type: ignore[import-untyped]
from referencing import Registry, Resource
except ImportError:
warnings.append(
"jsonschema/referencing not installed; full instance-schema validation "
"deferred to WP-010"
)
return
schema_dir = ROOT / "contracts" / "schemas"
kind_map = {
"Workspace": "workspace.schema.json",
"Project": "project.schema.json",
"CapabilityPack": "capability-pack.schema.json",
"ConnectorTemplate": "connector-manifest.schema.json",
"Policy": "policy.schema.json",
"ToolManifest": "tool-manifest.schema.json",
"ResourceManifest": "resource-manifest.schema.json",
"PromptManifest": "prompt-manifest.schema.json",
}
registry = Registry()
schemas: dict[str, dict[str, Any]] = {}
for path in sorted(schema_dir.glob("*.json")):
schema = load_json(path)
if not isinstance(schema, dict):
add_error(errors, f"schema is not an object: {path.relative_to(ROOT)}")
continue
schemas[path.name] = schema
resource = Resource.from_contents(schema)
registry = registry.with_resource(path.resolve().as_uri(), resource)
if isinstance(schema.get("$id"), str):
registry = registry.with_resource(schema["$id"], resource)
for path, document in read_catalog_documents(errors):
kind = document.get("kind")
schema_name = kind_map.get(kind) if isinstance(kind, str) else None
if not schema_name or schema_name not in schemas:
continue
validator = Draft202012Validator(
schemas[schema_name],
registry=registry,
format_checker=FormatChecker(),
)
for issue in validator.iter_errors(document):
location = "/".join(str(item) for item in issue.absolute_path) or "<root>"
add_error(
errors,
f"schema validation failed: {path.relative_to(ROOT)}:{location}: {issue.message}",
)
operation_schema = schemas.get("operation.schema.json")
for path in sorted((ROOT / "contracts" / "examples").glob("*.json")):
document = load_json(path)
if not isinstance(document, dict):
add_error(errors, f"example is not an object: {path.relative_to(ROOT)}")
continue
kind = document.get("kind")
schema_name = kind_map.get(kind) if isinstance(kind, str) else None
schema = schemas.get(schema_name) if schema_name else operation_schema
if schema is None:
continue
validator = Draft202012Validator(
schema,
registry=registry,
format_checker=FormatChecker(),
)
for issue in validator.iter_errors(document):
location = "/".join(str(item) for item in issue.absolute_path) or "<root>"
add_error(
errors,
f"example schema validation failed: {path.relative_to(ROOT)}:{location}: "
f"{issue.message}",
)
def validate_compose_blueprint(errors: list[str], warnings: list[str]) -> None:
try:
import yaml # type: ignore[import-untyped]
except ImportError:
warnings.append("PyYAML not installed; Compose YAML parse deferred to WP-020")
return
path = ROOT / "infrastructure" / "compose" / "docker-compose.blueprint.yml"
try:
document = yaml.safe_load(path.read_text(encoding="utf-8"))
except Exception as exc: # PyYAML exposes several parser subclasses.
add_error(errors, f"invalid Compose blueprint YAML: {exc}")
return
if not isinstance(document, dict) or not isinstance(document.get("services"), dict):
add_error(errors, "Compose blueprint must contain a services object")
return
services = document["services"]
allowed_publishers = {"web", "dataplane-router", "inspector"}
for name, service in services.items():
if not isinstance(service, dict):
add_error(errors, f"Compose service {name} is not an object")
continue
if service.get("ports") and name not in allowed_publishers:
add_error(errors, f"internal service unexpectedly publishes ports: {name}")
for volume in service.get("volumes", []) or []:
serialized = json.dumps(volume) if isinstance(volume, dict) else str(volume)
if "/var/run/docker.sock" in serialized and name != "docker-read-broker":
add_error(errors, f"Docker socket mounted outside broker: {name}")
broker = services.get("docker-read-broker", {})
broker_volumes = "\n".join(str(item) for item in broker.get("volumes", []))
if "/var/run/docker.sock:/var/run/docker.sock:ro" not in broker_volumes:
add_error(errors, "Docker read broker must mount the socket read-only")
def validate_no_obvious_secrets(errors: list[str]) -> None:
roots = [
ROOT / "catalog",
ROOT / "contracts",
ROOT / "infrastructure",
ROOT / "prompts",
ROOT / "scripts",
]
for base in roots:
for path in sorted(p for p in base.rglob("*") if p.is_file()):
if path.suffix.lower() in {".png", ".jpg", ".jpeg", ".zip"}:
continue
text = path.read_text(encoding="utf-8", errors="replace")
for label, pattern in SECRET_PATTERNS.items():
if pattern.search(text):
add_error(errors, f"possible {label} in {path.relative_to(ROOT)}")
def _parse_manifest_entries(
manifest: dict[str, Any], errors: list[str]
) -> dict[str, dict[str, Any]]:
entries = manifest.get("files", [])
if not isinstance(entries, list):
add_error(errors, "manifest files must be an array")
return {}
self_excluded = manifest.get("selfExcluded", "PACK_MANIFEST.json")
by_path: dict[str, dict[str, Any]] = {}
for entry in entries:
if not isinstance(entry, dict) or not isinstance(entry.get("path"), str):
add_error(errors, "manifest entry without a string path")
continue
rel = entry["path"]
if rel == self_excluded:
add_error(errors, f"manifest illegally includes its own excluded path: {rel}")
continue
if rel in by_path:
add_error(errors, f"duplicate manifest path: {rel}")
continue
by_path[rel] = entry
return by_path
def validate_manifest(errors: list[str], warnings: list[str], repo_root: Path = ROOT) -> None:
"""Validate PACK_MANIFEST.json against the live Git repository at HEAD.
Every check here is deliberately Git-native (blob content, digest,
porcelain status) rather than raw filesystem bytes, so that a Windows
checkout (CRLF) and a Linux checkout (LF) of the identical commit
validate identically. See scripts/create_manifest.py and .gitattributes
for the full rationale.
"""
path = repo_root / "PACK_MANIFEST.json"
if not path.exists():
warnings.append(
"PACK_MANIFEST.json not generated yet; run scripts/create_manifest.py before packaging"
)
return
try:
manifest = load_json(path)
except PackError as exc:
add_error(errors, str(exc))
return
if not isinstance(manifest, dict):
add_error(errors, "PACK_MANIFEST.json must be a JSON object")
return
source = manifest.get("source")
if not isinstance(source, dict) or not isinstance(source.get("contentDigest"), str):
add_error(
errors,
"PACK_MANIFEST.json has no source.contentDigest; regenerate "
"with scripts/create_manifest.py (schemaVersion 2.0+ required)",
)
return
manifest_digest = source["contentDigest"]
entries = manifest.get("files", [])
if not isinstance(entries, list):
add_error(errors, "manifest files must be an array")
return
by_path = _parse_manifest_entries(manifest, errors)
# The recorded top-level digest must itself be a true statement about
# the "files" array it sits beside — a non-blocking sanity check,
# reported alongside (not instead of) the itemized comparison below, so
# a hand-edited individual entry is always named specifically rather
# than only producing a generic "digest doesn't match" error.
entries_digest = create_manifest.compute_content_digest(list(by_path.values()))
if entries_digest != manifest_digest:
add_error(
errors,
"PACK_MANIFEST.json is internally inconsistent: source.contentDigest "
f"({manifest_digest}) does not match a fresh digest of its own "
f"files array ({entries_digest}) — the manifest was hand-edited "
"or corrupted after generation.",
)
# Any uncommitted change to a tracked file means the manifest — which
# only ever describes committed content — cannot be trusted as a
# description of "clean Git content". Untracked files are irrelevant
# here: they can never enter the manifest in the first place.
dirty = create_manifest.list_dirty_tracked_paths(repo_root)
if dirty:
add_error(
errors,
"working tree has uncommitted changes to tracked files, so the "
f"manifest cannot be trusted as clean committed content: {dirty}",
)
return
try:
head_commit = create_manifest.resolve_commit("HEAD", repo_root)
except create_manifest.GitError as exc:
add_error(errors, f"cannot verify manifest against Git: {exc}")
return
# The itemized comparison below is the actual source of truth for
# staleness and tampering — it compares every manifest entry directly
# against live Git blob content, never against a raw `HEAD^{tree}` hash
# or a commit SHA. That is deliberate, for two independent reasons:
# (1) PACK_MANIFEST.json is itself a tracked, committed file, so the raw
# tree hash always includes its own blob — which changes on every
# regeneration via `generatedAt`. Gating on the raw tree hash is
# self-referentially impossible to satisfy.
# (2) a no-conflict merge commit (e.g. a Gitea PR merge) gets a new
# commit SHA even when it changes no tracked content; gating on
# commit SHA would spuriously demand a regeneration commit after
# every such merge (this happened once already: commit 95329ca).
# Comparing actual (path, checksum) pairs avoids both: it is stable
# across manifest-only commits and no-op merges, and flags a path if and
# only if that path's tracked content actually changed.
self_excluded = manifest.get("selfExcluded", "PACK_MANIFEST.json")
tracked = set(create_manifest.list_tracked_paths(head_commit, repo_root)) - {self_excluded}
manifest_paths = set(by_path)
if missing := sorted(tracked - manifest_paths):
add_error(errors, f"manifest omits Git-tracked files: {missing}")
if extra := sorted(manifest_paths - tracked):
add_error(errors, f"manifest contains files no longer tracked by Git: {extra}")
verifiable = sorted(manifest_paths & tracked)
contents = create_manifest.batch_blob_contents(head_commit, verifiable, repo_root)
total_bytes = 0
for rel in verifiable:
entry = by_path[rel]
data = contents.get(rel)
if data is None:
add_error(errors, f"could not read Git blob content for {rel}")
continue
total_bytes += len(data)
if entry.get("bytes") != len(data):
add_error(errors, f"manifest byte count mismatch: {rel}")
digest = hashlib.sha256(data).hexdigest()
if digest != entry.get("sha256"):
add_error(errors, f"manifest checksum mismatch: {rel}")
if not errors:
if manifest.get("fileCount") != len(entries):
add_error(errors, "manifest fileCount does not match files array")
if manifest.get("totalBytes") != total_bytes:
add_error(errors, "manifest totalBytes does not match Git blob contents")
def validate_manifest_against_directory(directory: Path, manifest_root: Path = ROOT) -> list[str]:
"""Validate a plain directory (no Git required) against PACK_MANIFEST.json.
Used to prove a clean-room artifact — a `git archive` extraction, a
fresh clone, or a release-export directory — contains exactly the bytes
the manifest claims, on any platform, without needing Git present in
that directory. This works because `git archive` and the manifest's own
generator both source content from Git blobs, which are never subject
to checkout-time line-ending conversion.
"""
errors: list[str] = []
manifest_path = manifest_root / "PACK_MANIFEST.json"
if not manifest_path.is_file():
return [f"no PACK_MANIFEST.json at {manifest_path}"]
manifest = load_json(manifest_path)
if not isinstance(manifest, dict):
return ["PACK_MANIFEST.json must be a JSON object"]
by_path = _parse_manifest_entries(manifest, errors)
self_excluded = manifest.get("selfExcluded", "PACK_MANIFEST.json")
# The directory being verified may legitimately contain the manifest
# file itself — e.g. a `git archive` extraction includes it, since
# PACK_MANIFEST.json is itself tracked and committed — even though the
# manifest's own file list never includes it (see selfExcluded). Its
# presence is expected, not a discrepancy.
on_disk = {
candidate.relative_to(directory).as_posix()
for candidate in directory.rglob("*")
if candidate.is_file()
} - {self_excluded}
manifest_paths = set(by_path)
if missing := sorted(manifest_paths - on_disk):
errors.append(f"directory is missing manifest files: {missing}")
if extra := sorted(on_disk - manifest_paths):
errors.append(f"directory contains files absent from the manifest: {extra}")
for rel in sorted(manifest_paths & on_disk):
entry = by_path[rel]
file_path = directory / rel
data = file_path.read_bytes()
if entry.get("bytes") != len(data):
errors.append(f"byte count mismatch outside repository: {rel}")
digest = hashlib.sha256(data).hexdigest()
if digest != entry.get("sha256"):
errors.append(f"checksum mismatch outside repository: {rel}")
return errors
def main() -> int:
parser = argparse.ArgumentParser(description="Validate the ITWorx MCP Hub design/build pack.")
parser.add_argument("--json", action="store_true", help="emit machine-readable output")
parser.add_argument(
"--require-manifest",
action="store_true",
help="fail when PACK_MANIFEST.json is absent",
)
parser.add_argument(
"--verify-tree",
metavar="DIR",
help=(
"Skip the full design-pack gate and instead verify that DIR "
"(e.g. a `git archive` extraction or release-export directory) "
"contains exactly the files and checksums PACK_MANIFEST.json "
"records. No Git repository is required inside DIR."
),
)
args = parser.parse_args()
if args.verify_tree:
tree_errors = validate_manifest_against_directory(Path(args.verify_tree).resolve())
if args.json:
print(json.dumps({"ok": not tree_errors, "errors": tree_errors}, indent=2))
else:
status = "PASS" if not tree_errors else "FAIL"
print(f"ITWorx MCP Hub manifest tree verification: {status}")
for error in tree_errors:
print(f"ERROR: {error}")
return 0 if not tree_errors else 1
errors: list[str] = []
warnings: list[str] = []
validate_required_files(errors)
plan, _state = validate_plan_and_state(errors, warnings)
catalog_counts = validate_catalog(errors, warnings)
schema_count, example_count = validate_schemas_and_examples(errors)
validate_catalog_json_schemas(errors, warnings)
validate_compose_blueprint(errors, warnings)
validate_no_obvious_secrets(errors)
validate_manifest(errors, warnings)
if args.require_manifest and not (ROOT / "PACK_MANIFEST.json").is_file():
add_error(errors, "PACK_MANIFEST.json is required")
summary = {
"ok": not errors,
"errors": errors,
"warnings": warnings,
"packetCount": len(plan.get("packets", [])) if isinstance(plan, dict) else 0,
"catalogCounts": catalog_counts,
"schemaCount": schema_count,
"exampleCount": example_count,
}
if args.json:
print(json.dumps(summary, indent=2))
else:
status = "PASS" if not errors else "FAIL"
print(f"ITWorx MCP Hub design-pack validation: {status}")
print(
f"packets={summary['packetCount']} schemas={schema_count} "
f"examples={example_count} catalog={catalog_counts}"
)
for warning in warnings:
print(f"WARNING: {warning}")
for error in errors:
print(f"ERROR: {error}")
return 0 if not errors else 1
if __name__ == "__main__":
sys.exit(main())
scripts/verify.py
+330
View File
@@ -0,0 +1,330 @@
#!/usr/bin/env python3
"""Run the bounded, cross-platform repository verification gate."""
from __future__ import annotations
import json
import shutil
import subprocess
import sys
import sysconfig
import tomllib
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
def run(label: str, command: list[str]) -> None:
print(f"== {label} ==")
result = subprocess.run(command, cwd=ROOT, check=False) # noqa: S603
if result.returncode:
raise SystemExit(result.returncode)
def uv_command() -> str | None:
resolved = shutil.which("uv")
if resolved:
return resolved
candidates = [
Path(sysconfig.get_path("scripts")) / "uv.exe",
Path(sysconfig.get_path("scripts")) / "uv",
Path.home()
/ "AppData"
/ "Roaming"
/ "Python"
/ f"Python{sys.version_info.major}{sys.version_info.minor}"
/ "Scripts"
/ "uv.exe",
]
return next((str(path) for path in candidates if path.is_file()), None)
def validate_metadata() -> None:
pyproject = tomllib.loads((ROOT / "pyproject.toml").read_text(encoding="utf-8"))
package = json.loads((ROOT / "package.json").read_text(encoding="utf-8"))
project = pyproject["project"]
assert project["requires-python"] == ">=3.13,<3.14"
assert "mcp==2.0.0" in project["dependencies"]
assert package["private"] is True
assert package["packageManager"] == "pnpm@10.33.0"
assert package["workspaces"] == ["apps/*", "packages/*"]
for relative in (
"apps",
"services",
"connectors",
"packages",
"tests",
"infrastructure",
"evidence",
"docs",
):
assert (ROOT / relative).is_dir(), f"missing repository root: {relative}"
for lockfile in ("uv.lock", "pnpm-lock.yaml"):
assert (ROOT / lockfile).is_file(), f"missing lockfile: {lockfile}"
print("Workspace metadata and lockfiles: PASS")
def main() -> int:
validate_metadata()
python_quality_files = [
"scripts/audit_projectbrain_contracts.py",
"tests/test_projectbrain_contract_audit.py",
"scripts/check_boundaries.py",
"scripts/check_build_reproducibility.py",
"scripts/verify.py",
"scripts/create_manifest.py",
"scripts/validate_pack.py",
"scripts/security_scan.py",
"scripts/generate_sbom.py",
"scripts/generate_image_sboms.py",
"scripts/discover_unraid.py",
"scripts/backup_state.py",
"scripts/write_backup_status.py",
"scripts/restore_state.py",
"scripts/compose_release_check.py",
"scripts/promotion_preflight.py",
"scripts/export_release.py",
"scripts/render_release_record.py",
"scripts/verify_demo_token_flow.py",
"scripts/draft_cve_exceptions.py",
"scripts/check_cve_exceptions.py",
"infrastructure/images/python-service/health_server.py",
"infrastructure/images/observability/collector.py",
"infrastructure/database/alembic/versions/0002_knowledge_search.py",
"infrastructure/database/alembic/versions/0003_policy_trace.py",
"infrastructure/database/alembic/versions/0004_audit_retention.py",
"tests/test_auth.py",
"tests/test_catalog.py",
"tests/test_dataplane_router.py",
"tests/test_knowledge_connector.py",
"tests/test_gitea_connector.py",
"tests/test_appops_connector.py",
"tests/test_docker_read_broker.py",
"tests/test_unraid_connector.py",
"tests/test_wp110_governance.py",
"tests/test_contextforge.py",
"tests/test_contextforge_integration.py",
"tests/test_bootstrap.py",
"tests/test_compose.py",
"tests/test_domain.py",
"tests/test_migrations.py",
"tests/test_persistence.py",
"tests/test_postgres_acceptance.py",
"tests/test_wp150_operations.py",
"tests/test_wp160_security.py",
"tests/test_wp170_release.py",
"tests/test_wp220_manifest.py",
"tests/test_wp220_export.py",
"tests/test_wp220_release_record.py",
"tests/test_wp220_cve_exceptions.py",
"tests/test_wp140_clients.py",
"tests/test_wp230_authz.py",
"tests/test_wp230_rbac.py",
"tests/test_wp230_policy_simulate.py",
"tests/test_wp230_ui_completion.py",
"tests/test_wp230_session_store.py",
"tests/test_wp270_functional_operations.py",
"tests/test_wp280_runtime_quality.py",
"tests/e2e/harness.py",
"tests/e2e/tls.py",
"tests/test_promotion_preflight.py",
"tests/test_production_runtime.py",
"packages/hub_runtime/__init__.py",
"packages/hub_runtime/secret_files.py",
"packages/hub_runtime/supervisor.py",
"packages/__init__.py",
"packages/hub_core/__init__.py",
"packages/hub_observability/__init__.py",
"packages/hub_observability/telemetry.py",
"packages/hub_operations/__init__.py",
"packages/hub_operations/backup.py",
"packages/hub_operations/recovery.py",
"packages/hub_core/domain.py",
"packages/hub_core/errors.py",
"packages/hub_core/state_machines.py",
"packages/hub_contracts/__init__.py",
"packages/hub_contracts/dataplane.py",
"packages/hub_policy/__init__.py",
"packages/hub_policy/audit.py",
"packages/hub_policy/engine.py",
"packages/hub_policy/governance.py",
"packages/hub_policy/rate_limit.py",
"packages/connector_kit/__init__.py",
"packages/connector_kit/contracts.py",
"packages/connector_kit/knowledge.py",
"packages/connector_kit/pagination.py",
"packages/connector_kit/redaction.py",
"packages/connector_kit/gitea.py",
"packages/connector_kit/mobilityops.py",
"services/__init__.py",
"services/control_api/__init__.py",
"services/control_api/authz.py",
"services/control_api/clients/__init__.py",
"services/control_api/clients/router.py",
"services/control_api/clients/service.py",
"services/observability/__init__.py",
"services/observability/health.py",
"services/control_api/policy/__init__.py",
"services/control_api/policy/router.py",
"services/control_api/policy/service.py",
"services/dataplane_router/__init__.py",
"services/dataplane_router/authz.py",
"services/dataplane_router/errors.py",
"services/dataplane_router/models.py",
"services/dataplane_router/proxy.py",
"services/dataplane_router/registry.py",
"services/dataplane_router/router.py",
"connectors/__init__.py",
"connectors/knowledge/__init__.py",
"connectors/knowledge/server.py",
"connectors/gitea/__init__.py",
"connectors/gitea/changesets.py",
"connectors/gitea/fake.py",
"connectors/gitea/server.py",
"connectors/unraid/__init__.py",
"connectors/unraid/client.py",
"connectors/unraid/fake.py",
"connectors/unraid/server.py",
"connectors/mobilityops/__init__.py",
"connectors/mobilityops/fake.py",
"connectors/mobilityops/server.py",
"services/control_api/persistence/__init__.py",
"services/control_api/persistence/dataplane_source.py",
"services/control_api/persistence/db.py",
"services/control_api/persistence/models.py",
"services/control_api/persistence/repositories.py",
"services/control_api/persistence/seed.py",
"services/control_api/knowledge/__init__.py",
"services/control_api/knowledge/jobs.py",
"services/control_api/knowledge/store.py",
"services/control_api/gitea/__init__.py",
"services/control_api/gitea/registry.py",
"services/docker_read_broker/__init__.py",
"services/docker_read_broker/broker.py",
"services/docker_read_broker/server.py",
"services/control_api/catalog/__init__.py",
"services/control_api/catalog/loader.py",
"services/control_api/catalog/models.py",
"services/control_api/catalog/planner.py",
"services/control_api/catalog/reconciler.py",
"services/control_api/catalog/router.py",
"services/control_api/catalog/store.py",
"services/control_api/auth/__init__.py",
"services/control_api/auth/bff.py",
"services/control_api/auth/config.py",
"services/control_api/auth/errors.py",
"services/control_api/auth/fake_provider.py",
"services/control_api/auth/machine.py",
"services/control_api/auth/models.py",
"services/control_api/auth/provider.py",
"services/control_api/auth/router.py",
"services/control_api/contextforge/__init__.py",
"services/control_api/contextforge/client.py",
"services/control_api/contextforge/errors.py",
"services/control_api/contextforge/fake.py",
"services/control_api/contextforge/models.py",
"services/control_api/persistence/services.py",
"services/runtime/__init__.py",
"services/runtime/__main__.py",
"services/runtime/common.py",
"services/runtime/connector_gitea.py",
"services/runtime/connector_appops.py",
"services/runtime/gitea_operations_widget.py",
"services/runtime/gitea_review_widget.py",
"services/runtime/mcp_skills.py",
"services/runtime/synthetic_monitor.py",
"services/runtime/connector_knowledge.py",
"services/runtime/connector_unraid.py",
"services/runtime/connector_mobilityops.py",
"services/runtime/audit.py",
"services/runtime/client_service.py",
"services/runtime/control_api.py",
"services/runtime/dataplane.py",
"services/runtime/dispatch.py",
"services/runtime/mcp_server.py",
"services/runtime/migrate.py",
"services/runtime/state_sync.py",
"services/runtime/web.py",
"services/runtime/worker.py",
]
# Release-manifest integrity is checked first, before any slower quality
# gate: a stale or tampered PACK_MANIFEST.json means the release-integrity
# invariant is already broken, and every later check downstream of it
# (build, scan, package) would be validating content that doesn't match
# what will actually ship. Fail loud and early rather than burying it
# after several minutes of lint/type/test output.
run(
"Release manifest integrity (Git-tracked content only, manifest required)",
[sys.executable, "scripts/validate_pack.py", "--require-manifest"],
)
resolved_uv = uv_command()
if resolved_uv:
run("Python lockfile", [resolved_uv, "lock", "--check"])
else:
print("Python lockfile: DEFERRED (uv executable is not installed)")
if resolved_uv:
run(
"Python format",
[resolved_uv, "run", "--locked", "ruff", "format", "--check", *python_quality_files],
)
run("Python lint", [resolved_uv, "run", "--locked", "ruff", "check", *python_quality_files])
run(
"Python types",
[
resolved_uv,
"run",
"--locked",
"mypy",
"--explicit-package-bases",
"scripts/audit_projectbrain_contracts.py",
"scripts/check_boundaries.py",
"scripts/check_build_reproducibility.py",
"scripts/verify.py",
"scripts/security_scan.py",
"scripts/generate_sbom.py",
"scripts/generate_image_sboms.py",
"scripts/backup_state.py",
"scripts/write_backup_status.py",
"scripts/restore_state.py",
"scripts/compose_release_check.py",
"scripts/promotion_preflight.py",
"scripts/discover_unraid.py",
"scripts/revoke_demo_token.py",
"scripts/verify_demo_token_runtime.py",
"scripts/verify_demo_token_flow.py",
"scripts/export_release.py",
"scripts/render_release_record.py",
"scripts/draft_cve_exceptions.py",
"scripts/check_cve_exceptions.py",
"packages/hub_core",
"packages/hub_observability",
"packages/hub_operations",
"packages/hub_contracts",
"packages/connector_kit",
"connectors/knowledge",
"connectors/gitea",
"connectors/unraid",
"connectors/mobilityops",
"services/docker_read_broker",
"services/control_api",
"services/observability",
"services/runtime",
],
)
run("Import boundaries", [sys.executable, "scripts/check_boundaries.py"])
python_test_command = (
[resolved_uv, "run", "--locked", "python", "-m", "pytest", "tests", "-q"]
if resolved_uv
else [sys.executable, "-m", "pytest", "tests", "-q"]
)
run("Python unit tests", python_test_command)
run("Frontend typecheck", ["pnpm", "--filter", "@itworx-mcp-hub/web", "typecheck"])
run("Frontend lint", ["pnpm", "--filter", "@itworx-mcp-hub/web", "lint"])
run("Frontend unit tests", ["pnpm", "--filter", "@itworx-mcp-hub/web", "test"])
run("Frontend build", ["pnpm", "--filter", "@itworx-mcp-hub/web", "build"])
print("Repository verification: PASS")
return 0
if __name__ == "__main__":
raise SystemExit(main())
scripts/verify_demo_token_flow.py
+114
View File
@@ -0,0 +1,114 @@
#!/usr/bin/env python3
"""Prove one Fleet Ops demo-token MCP call, then prove its revocation."""
from __future__ import annotations
import argparse
import json
import sys
import urllib.error
import urllib.request
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from scripts.revoke_demo_token import revoke
from services.runtime.demo_token import load_demo_token_signing_key, mint_demo_token
@dataclass(frozen=True)
class HttpResult:
status: int
content_type: str
body: bytes
def post_tool_call(endpoint: str, token: str, *, timeout_seconds: float) -> HttpResult:
request = urllib.request.Request( # noqa: S310 - operator supplies an explicit HTTP(S) MCP endpoint
endpoint,
data=json.dumps(
{
"jsonrpc": "2.0",
"id": "demo-token-flow-verification",
"method": "tools/call",
"params": {"name": "mobilityops.operations.summary", "arguments": {}},
},
separators=(",", ":"),
).encode("utf-8"),
headers={
"Authorization": f"Bearer {token}",
"Accept": "application/json",
"Content-Type": "application/json",
"MCP-Protocol-Version": "2025-11-25",
},
method="POST",
)
try:
with urllib.request.urlopen(request, timeout=timeout_seconds) as response: # noqa: S310
return HttpResult(response.status, response.headers.get_content_type(), response.read())
except urllib.error.HTTPError as error:
try:
body = error.read()
except (OSError, ConnectionError):
# The status/header proof remains authoritative when the peer closes
# the optional error-body stream immediately after sending its 401.
body = b""
return HttpResult(error.code, error.headers.get_content_type(), body)
def verify_flow(
endpoint: str,
key_file: Path,
revocations_file: Path,
*,
timeout_seconds: float = 15.0,
) -> tuple[HttpResult, HttpResult]:
"""Run a bounded live proof and revoke its generated bearer in all cases."""
key = load_demo_token_signing_key(key_file)
token, jti = mint_demo_token(key, ttl_hours=1)
try:
valid = post_tool_call(endpoint, token, timeout_seconds=timeout_seconds)
if valid.status != 200:
raise RuntimeError(f"valid demo-token call returned HTTP {valid.status}")
if valid.content_type != "application/json":
raise RuntimeError("valid demo-token call did not return application/json")
try:
document: Any = json.loads(valid.body)
except (UnicodeDecodeError, ValueError) as exc:
raise RuntimeError("valid demo-token call returned invalid JSON") from exc
if not isinstance(document, dict):
raise RuntimeError("valid demo-token call returned a non-object JSON payload")
finally:
revoke(revocations_file, jti)
revoked = post_tool_call(endpoint, token, timeout_seconds=timeout_seconds)
if revoked.status != 401:
raise RuntimeError(f"revoked demo-token call returned HTTP {revoked.status}, expected 401")
return valid, revoked
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--endpoint", required=True)
parser.add_argument("--key-file", type=Path, required=True)
parser.add_argument("--revocations-file", type=Path, required=True)
parser.add_argument("--timeout-seconds", type=float, default=15.0)
args = parser.parse_args()
if not 1 <= args.timeout_seconds <= 30:
raise ValueError("timeout-seconds must be between 1 and 30")
valid, revoked = verify_flow(
args.endpoint,
args.key_file,
args.revocations_file,
timeout_seconds=args.timeout_seconds,
)
print(f"valid_call_http={valid.status}")
print(f"valid_call_content_type={valid.content_type}")
print(f"valid_call_bytes={len(valid.body)}")
print(f"revoked_call_http={revoked.status}")
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/verify_demo_token_runtime.py
+89
View File
@@ -0,0 +1,89 @@
#!/usr/bin/env python3
"""Fail closed unless the live router reads the operator's revocation file.
Run this on the Docker host before minting or revoking a Fleet Ops demo token.
It reads Docker mount metadata and file metadata only; it never reads a token,
signing key, or revocation-file contents.
"""
from __future__ import annotations
import argparse
import json
import os
import stat
import subprocess
import sys
from pathlib import Path
from typing import Any
ROUTER_DIRECTORY = "/etc/itworx"
ROUTER_DESTINATION = ROUTER_DIRECTORY + "/demo-token-revocations.json"
def validate_mount(mounts: list[dict[str, Any]], expected_host_file: Path) -> dict[str, object]:
if not expected_host_file.is_absolute():
raise ValueError("expected host file must be an absolute path")
matching = [mount for mount in mounts if mount.get("Destination") == ROUTER_DIRECTORY]
if len(matching) != 1:
raise ValueError("router must have exactly one demo-token revocation mount")
source = Path(str(matching[0].get("Source", "")))
if source != expected_host_file.parent:
raise ValueError("router revocation directory mount does not match the operator host file")
try:
metadata = expected_host_file.stat()
except OSError as exc:
raise ValueError("operator revocation file is missing") from exc
if not stat.S_ISREG(metadata.st_mode):
raise ValueError("operator revocation file is not a regular file")
mode = stat.S_IMODE(metadata.st_mode)
# The operator command runs on Tower (Linux). Windows does not expose
# NTFS ACLs through POSIX mode bits, so local unit tests cannot make this
# assertion meaningful there.
if os.name != "nt" and mode != 0o600:
raise ValueError("operator revocation file mode must be exactly 0600")
if matching[0].get("RW") is not False:
raise ValueError("router revocation mount must be read-only")
return {
"status": "pass",
"containerDestination": ROUTER_DESTINATION,
"hostFile": str(expected_host_file),
"mode": f"{mode:04o}",
"readOnly": True,
}
def inspect_mounts(router_container: str) -> list[dict[str, Any]]:
result = subprocess.run( # noqa: S603
["docker", "inspect", router_container, "--format", "{{json .Mounts}}"], # noqa: S607
capture_output=True,
text=True,
check=False,
)
if result.returncode != 0:
raise ValueError("could not inspect router container")
try:
mounts = json.loads(result.stdout)
except json.JSONDecodeError as exc:
raise ValueError("router mount metadata is invalid") from exc
if not isinstance(mounts, list) or not all(isinstance(item, dict) for item in mounts):
raise ValueError("router mount metadata is invalid")
return mounts
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--router-container", required=True)
parser.add_argument("--expected-host-file", type=Path, required=True)
args = parser.parse_args()
try:
report = validate_mount(inspect_mounts(args.router_container), args.expected_host_file)
except ValueError as exc:
print(json.dumps({"status": "fail", "reason": str(exc)}, sort_keys=True))
return 1
print(json.dumps(report, sort_keys=True))
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/write_backup_status.py
+69
View File
@@ -0,0 +1,69 @@
#!/usr/bin/env python3
"""Publish a secret-free, atomic backup/restore status projection for the Hub UI."""
from __future__ import annotations
import argparse
import hashlib
import json
import os
from datetime import UTC, datetime
from pathlib import Path
def publish(output: Path, artifacts: list[Path], *, restore_verified: bool) -> dict[str, object]:
if not artifacts or len(artifacts) > 1000:
raise ValueError("one to 1000 backup artifacts are required")
checksums: list[dict[str, object]] = []
for artifact in artifacts:
if artifact.is_symlink():
raise ValueError("backup artifact must be a regular file")
resolved = artifact.resolve(strict=True)
if not resolved.is_file():
raise ValueError("backup artifact must be a regular file")
digest = hashlib.sha256()
size = 0
with resolved.open("rb") as handle:
while chunk := handle.read(1024 * 1024):
size += len(chunk)
digest.update(chunk)
if size <= 0:
raise ValueError("backup artifact must not be empty")
checksums.append({"name": resolved.name[:255], "bytes": size, "sha256": digest.hexdigest()})
document: dict[str, object] = {
"schemaVersion": "1",
"completedAt": datetime.now(UTC).isoformat(),
"restoreVerified": restore_verified,
"artifactCount": len(checksums),
"artifacts": checksums,
}
output.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
temporary = output.with_name(f".{output.name}.tmp")
temporary.write_text(json.dumps(document, indent=2, sort_keys=True) + "\n", encoding="utf-8")
os.chmod(temporary, 0o600)
temporary.replace(output)
return document
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--artifact", type=Path, action="append", required=True)
parser.add_argument("--restore-verified", action="store_true")
args = parser.parse_args()
document = publish(args.output, args.artifact, restore_verified=args.restore_verified)
print(
json.dumps(
{
"artifactCount": document["artifactCount"],
"restoreVerified": document["restoreVerified"],
"statusFile": str(args.output),
},
sort_keys=True,
)
)
return 0
if __name__ == "__main__":
raise SystemExit(main())