RepoPulse-Public/docs/ARCHITECTURE.md

RepoPulse Architecture

Runtime shape

RepoPulse is a Gitea-native deterministic analysis service. Source code is read by the backend only; the browser receives repository summaries, derived metrics, findings and bounded graph metadata. Gitea is a read-only provider, while SQLite remains the source of RepoPulse catalog and history state during provider outages.

Gitea (read-only API + Git)
       │ catalog + default-branch SHA
       ▼
FastAPI ── SQLite catalog / durable job queue / scan history (/data)
  │                         │
  │                         ├── bounded mirror cache (/data/repos/mirrors)
  │                         └── detached worktrees (/data/repos/worktrees)
  │                                      │
  │                                      ▼
  ├── single-process worker → deterministic scanner → report persistence
  │
  └── /api → Nginx / React repository cockpit
                              │
                              └── Advanced local provider → read-only /scan

The worker starts and stops through FastAPI lifespan. SQLite claim transactions and leases make queued work restart-safe; the current deployment intentionally uses one worker and one active scan by default, without Redis or Celery.

Repository identity and lifecycle

Gitea repositories are first-class records keyed by id = gitea:<numeric-provider-id> and constrained by (provider, provider_repository_id). Owner/name, description, URLs, visibility, archive state, default branch and observed HEAD metadata are catalog attributes; filesystem paths are not primary identity. Local scans use a separate stable local:<path-hash> identity for v1 compatibility.

Catalog synchronization is bounded and paginated. After a successful listing it resolves each default-branch HEAD independently with a bounded REPOPULSE_GITEA_BRANCH_WORKERS pool (default 4). A branch failure is persisted as a sanitized repository-specific sync_error, while every other repository is upserted and remains usable. A failed or truncated listing preserves all prior catalog/history rows; only a complete listing may mark an absent repository unavailable. Archived/unavailable records are retained and are not cascade-deleted.

The computed repository state is one of never_scanned, current, stale, queued, scanning, failed, unavailable or archived. The scheduler reconciles latest_revision against last_scan_revision, respects policy.auto_scan, queues new/stale revisions, and retries after the next sync interval. It does not queue a repository whose observed HEAD already has a successful scan.

Exact source acquisition

For a Gitea job, the requested SHA is captured at enqueue time. The acquirer uses the configured Gitea origin only, passes the token through a subprocess environment header rather than a URL, fetches the selected default branch into a provider-ID mirror, resolves the requested immutable commit, and creates a detached worktree under the managed cache. The scanner sees that one worktree only; cleanup removes it and prunes stale Git registrations. Corrupt mirrors are discarded only after a local bare-repository validation failure, so a remote outage does not erase a usable cache.

Scan boundary

When REPOPULSE_SCAN_ROOT is configured, every local scan path is resolved and checked against that root before analysis. Compose sets the root to /scan and mounts the user-selected source directory read-only. The scanner then resolves every candidate file, rejects symlinked files and directories (including symlink ancestors), and requires the resolved file to remain below the scan root. This prevents the API from being used as a generic filesystem browser outside the configured analysis boundary. Gitea worktrees are acquired into /data, not /scan, and are controlled by repository-ID/job-ID validation.

Persistence and deployment boundary

The exact-source deployment controller replaces the registered repository source directory on each deployment. SQLite history, the catalog, queue and mirror cache must therefore not live below that directory. Compose uses the Docker-managed named volume repopulse_data for /data when REPOPULSE_DATA_ROOT is unset (the production configuration); this volume is independent of source refreshes and container recreation. The checked-in .env.example opts local Compose into REPOPULSE_DATA_ROOT=./data for convenient checkout-local development. /scan remains the read-only source boundary and /data is the only writable application mount.

The AppOps-managed runtime candidate is defined separately in .itworx/deployment.yml and built with deploy/managed/Dockerfile; the legacy root docker-compose.yml keeps its existing topology and now forwards the optional REPOPULSE_MAINTENANCE_TOKEN so the old writer can be authenticated and quiesced before cutover. Dockerfile.unraid remains unchanged pending cutover. The managed target uses the immutable controller-selected source revision to build image repopulse-managed:production, runs as UID/GID 10001:10001 with a read-only root filesystem and one existing NPM network, binds /mnt/pool/appdata/repopulse/data to writable /data, and binds only the existing controller-owned repopulse and privatemuse source directories read-only at /scan/repopulse and /scan/privatemuse. The fixed AppOps profile allows only these exact external source mounts and rejects other paths. Git mirrors and worktrees stay under /data/repos, while process scratch files use the bounded /tmp tmpfs. Runtime secrets are restricted to the fixed Gitea URL/token and a dedicated RepoPulse maintenance token in deploy.managed.env; the legacy deploy.env remains available for rollback. No Gitea webhook secret is configured.

Inventory pipeline and analysis coverage

The scanner prefers git ls-files for a top-level worktree so ignored/untracked build artifacts do not distort results. An empty Git worktree remains empty rather than silently falling back. Non-Git directories use a filesystem traversal that excludes common generated/vendor directories and applies bounded root/nested .gitignore rules. Text files larger than the configured internal read threshold are not parsed as source, and binary files are skipped.

Every report includes ScanCoverage: bounded file, per-file-text, total-text, finding and credential-finding limits plus the observed skipped counts. A limit produces a deterministic partial result and an analysis-coverage warning rather than an unqualified completeness claim or an artificial blocker.

Signals

The v1 scan combines:

• language and non-empty LOC inventory;
• test-file, test-framework, CI and coverage-artifact detection;
• dependency manifests and lockfiles across JavaScript, Python, Go, Rust and Maven;
• TODO/FIXME/HACK/XXX debt;
• large files/modules and Python function-size/branch heuristics;
• basic credential-assignment detection with values redacted;
• bounded recent Git commit/contributor/churn information.

Malformed dependency manifests are represented as warning findings instead of aborting the complete scan. CI workflows are classified from supported commands as validation, build or deploy-only rather than file presence alone. Requirements includes and workspace-root lockfiles are handled safely and boundedly. Credential heuristics redact values, ignore obvious templates/environment references, and still flag real-looking fixture literals for review.

Each finding is enriched by a backend rule definition with confidence, why-it-matters text, recommended action, remediation class and explainable priority. The browser renders retained intelligence; it does not duplicate rule knowledge or mutate external repositories.

Architecture graph

Python imports are parsed using the standard-library AST. Local JavaScript and TypeScript relative imports, bounded baseUrl/paths aliases from tsconfig.json/jsconfig.json, and simple local workspace package imports are resolved against known source files. Only local edges are retained; this remains heuristic source analysis, not a TypeScript compiler. The displayed graph is bounded to 160 nodes and 600 edges to keep large repositories responsive; the response explicitly indicates truncation, reports total discovered nodes/edges and distinguishes visible coupling from total coupling used in hotspot ranking. Extensionless, index and .js-specifier-to-TypeScript resolution are supported.

Hotspot model

Hotspot score is deterministic and combines source size, debt markers, recent changed-line churn and local inbound/outbound coupling. The UI surfaces both the score and reasons so it is not a black-box ranking.

Health and release gate

Category scores start at 100 and are reduced only by findings with an explicit deduction. The overall health score is the average category score. The release gate separately evaluates blocker security findings, test presence, CI, dependency hygiene (including unbounded and unpinned declarations), repository health and engineering warnings, producing READY, READY WITH WARNINGS or BLOCKED.

History and comparison

Every report is serialized as a snapshot in SQLite using WAL. Lightweight scan summary columns (readiness, counts, primary signal, deltas and triage priority) drive catalog CTE queries without deserializing full reports. Comparison calculates metric/category deltas and added/resolved findings against the chronological previous scan of the same repository identity. Findings carry stable semantic fingerprints so line movement does not appear as a new risk. Trend endpoints return bounded chronological snapshots for UI visualization. By default, up to 200 snapshots are retained per repository, the latest 30 remain detailed, older retained payloads compact to summary evidence, and snapshots older than 365 days are removed. v1 path history is backfilled into a local catalog identity and is never deleted solely for migration.

Reporting

RepoPulse can export the complete JSON report or a standalone HTML report. HTML output escapes repository-derived text, export headers use sanitized RFC 5987 filenames, and neither format includes detected secret values or Gitea credentials. A root .repopulse.yml/.yaml may configure automatic scanning, minimum health, blocker allowance, required tests/CI and finding-code suppressions. Suppressed observations remain visible and are excluded from deductions and gate blockers.

Job state and failure isolation

The persisted queue records repository, trigger, requested/actual revision, phase, progress, attempt, timestamps, resulting scan and sanitized error information. Claims use BEGIN IMMEDIATE plus a status guard, so two processes cannot own the same queued job. They move through queued → acquiring_source → scanning → analyzing → persisting → completed or an isolated failed outcome. A heartbeat renews each active lease; only a crashed worker’s expired lease returns work to queued for recovery. Older unstarted revisions become superseded; explicit cancellation states remain representable. One repository failure does not terminate the worker loop or affect other queued repositories.

Operator maintenance mode

The app_state key maintenance_mode durably records whether operator maintenance is enabled; it uses the existing state table and does not change the SQLite schema version. Operator enter/exit endpoints require REPOPULSE_MAINTENANCE_TOKEN (or the existing general operator token on legacy installations). Entering is serialized against API mutations and catalog synchronization so an accepted external write completes before the pause is committed. While paused, mutation requests and webhooks return 423 Locked; the scheduler skips heartbeat writes, Gitea synchronization, queue claims and new scan-job enqueue operations. Existing queued jobs remain queued for later resumption. A job claimed before the mode transition may complete and persist its result. Queue depth, active count, active-drain state and full-drain state are reported by the authenticated maintenance endpoint and the read-only meta endpoint. A malformed persisted maintenance value fails closed as paused.

Operational configuration

REPOPULSE_GITEA_URL and REPOPULSE_GITEA_TOKEN enable the provider. REPOPULSE_GITEA_SYNC_INTERVAL_SECONDS, REPOPULSE_GITEA_BRANCH_WORKERS, REPOPULSE_MAX_QUEUED_SCANS, REPOPULSE_JOB_LEASE_SECONDS, scan-budget and retention variables, and REPOPULSE_REPOSITORY_CACHE bound reconciliation, queue depth, active leases, analysis and cache placement. The durable Gitea runner deliberately has one worker; /api/v1/meta exposes requested versus actual capacity. REPOPULSE_GITEA_WEBHOOK_SECRET enables signed default-branch push intake; non-push, non-default-branch, invalid and branch-deletion events are acknowledged without scheduling work. REPOPULSE_OPERATOR_TOKEN protects mutating endpoints when an authenticated proxy perimeter is not the selected boundary. /api/v1/meta exposes repository state counts, queue depth, storage/retention metrics, recovery status, worker heartbeat, connector URL (without token) and last sync result.