101 lines
13 KiB
Markdown
101 lines
13 KiB
Markdown
# RepoPulse Architecture
|
||
|
||
## Runtime shape
|
||
|
||
RepoPulse is a Gitea-native deterministic analysis service. Source code is read by the backend only; the browser receives repository summaries, derived metrics, findings and bounded graph metadata. Gitea is a read-only provider, while SQLite remains the source of RepoPulse catalog and history state during provider outages.
|
||
|
||
```text
|
||
Gitea (read-only API + Git)
|
||
│ catalog + default-branch SHA
|
||
▼
|
||
FastAPI ── SQLite catalog / durable job queue / scan history (/data)
|
||
│ │
|
||
│ ├── bounded mirror cache (/data/repos/mirrors)
|
||
│ └── detached worktrees (/data/repos/worktrees)
|
||
│ │
|
||
│ ▼
|
||
├── single-process worker → deterministic scanner → report persistence
|
||
│
|
||
└── /api → Nginx / React repository cockpit
|
||
│
|
||
└── Advanced local provider → read-only /scan
|
||
```
|
||
|
||
The worker starts and stops through FastAPI lifespan. SQLite claim transactions and leases make queued work restart-safe; the current deployment intentionally uses one worker and one active scan by default, without Redis or Celery.
|
||
|
||
## Repository identity and lifecycle
|
||
|
||
Gitea repositories are first-class records keyed by `id = gitea:<numeric-provider-id>` and constrained by `(provider, provider_repository_id)`. Owner/name, description, URLs, visibility, archive state, default branch and observed HEAD metadata are catalog attributes; filesystem paths are not primary identity. Local scans use a separate stable `local:<path-hash>` identity for v1 compatibility.
|
||
|
||
Catalog synchronization is bounded and paginated. After a successful listing it resolves each default-branch HEAD independently with a bounded `REPOPULSE_GITEA_BRANCH_WORKERS` pool (default 4). A branch failure is persisted as a sanitized repository-specific `sync_error`, while every other repository is upserted and remains usable. A failed or truncated listing preserves all prior catalog/history rows; only a complete listing may mark an absent repository unavailable. Archived/unavailable records are retained and are not cascade-deleted.
|
||
|
||
The computed repository state is one of `never_scanned`, `current`, `stale`, `queued`, `scanning`, `failed`, `unavailable` or `archived`. The scheduler reconciles `latest_revision` against `last_scan_revision`, respects `policy.auto_scan`, queues new/stale revisions, and retries after the next sync interval. It does not queue a repository whose observed HEAD already has a successful scan.
|
||
|
||
## Exact source acquisition
|
||
|
||
For a Gitea job, the requested SHA is captured at enqueue time. The acquirer uses the configured Gitea origin only, passes the token through a subprocess environment header rather than a URL, fetches the selected default branch into a provider-ID mirror, resolves the requested immutable commit, and creates a detached worktree under the managed cache. The scanner sees that one worktree only; cleanup removes it and prunes stale Git registrations. Corrupt mirrors are discarded only after a local bare-repository validation failure, so a remote outage does not erase a usable cache.
|
||
|
||
## Scan boundary
|
||
|
||
When `REPOPULSE_SCAN_ROOT` is configured, every local scan path is resolved and checked against that root before analysis. Compose sets the root to `/scan` and mounts the user-selected source directory read-only. The scanner then resolves every candidate file, rejects symlinked files and directories (including symlink ancestors), and requires the resolved file to remain below the scan root. This prevents the API from being used as a generic filesystem browser outside the configured analysis boundary. Gitea worktrees are acquired into `/data`, not `/scan`, and are controlled by repository-ID/job-ID validation.
|
||
|
||
## Persistence and deployment boundary
|
||
|
||
The exact-source deployment controller replaces the registered repository source directory on each deployment. SQLite history, the catalog, queue and mirror cache must therefore not live below that directory. Compose uses the Docker-managed named volume `repopulse_data` for `/data` when `REPOPULSE_DATA_ROOT` is unset (the production configuration); this volume is independent of source refreshes and container recreation. The checked-in `.env.example` opts local Compose into `REPOPULSE_DATA_ROOT=./data` for convenient checkout-local development. `/scan` remains the read-only source boundary and `/data` is the only writable application mount.
|
||
|
||
The AppOps-managed runtime candidate is defined separately in `.itworx/deployment.yml` and built with `deploy/managed/Dockerfile`; the legacy root `docker-compose.yml` keeps its existing topology and now forwards the optional `REPOPULSE_MAINTENANCE_TOKEN` so the old writer can be authenticated and quiesced before cutover. `Dockerfile.unraid` remains unchanged pending cutover. The managed target uses the immutable controller-selected source revision to build image `repopulse-managed:production`, runs as UID/GID `10001:10001` with a read-only root filesystem and one existing NPM network, binds `/mnt/pool/appdata/repopulse/data` to writable `/data`, and binds only the existing controller-owned `repopulse` and `privatemuse` source directories read-only at `/scan/repopulse` and `/scan/privatemuse`. The fixed AppOps profile allows only these exact external source mounts and rejects other paths. Git mirrors and worktrees stay under `/data/repos`, while process scratch files use the bounded `/tmp` tmpfs. Runtime secrets are restricted to the fixed Gitea URL/token and a dedicated RepoPulse maintenance token in `deploy.managed.env`; the legacy `deploy.env` remains available for rollback. No Gitea webhook secret is configured.
|
||
|
||
## Inventory pipeline and analysis coverage
|
||
|
||
The scanner prefers `git ls-files` for a top-level worktree so ignored/untracked build artifacts do not distort results. An empty Git worktree remains empty rather than silently falling back. Non-Git directories use a filesystem traversal that excludes common generated/vendor directories and applies bounded root/nested `.gitignore` rules. Text files larger than the configured internal read threshold are not parsed as source, and binary files are skipped.
|
||
|
||
Every report includes `ScanCoverage`: bounded file, per-file-text, total-text, finding and credential-finding limits plus the observed skipped counts. A limit produces a deterministic partial result and an analysis-coverage warning rather than an unqualified completeness claim or an artificial blocker.
|
||
|
||
## Signals
|
||
|
||
The v1 scan combines:
|
||
|
||
- language and non-empty LOC inventory;
|
||
- test-file, test-framework, CI and coverage-artifact detection;
|
||
- dependency manifests and lockfiles across JavaScript, Python, Go, Rust and Maven;
|
||
- TODO/FIXME/HACK/XXX debt;
|
||
- large files/modules and Python function-size/branch heuristics;
|
||
- basic credential-assignment detection with values redacted;
|
||
- bounded recent Git commit/contributor/churn information.
|
||
|
||
Malformed dependency manifests are represented as warning findings instead of aborting the complete scan. CI workflows are classified from supported commands as validation, build or deploy-only rather than file presence alone. Requirements includes and workspace-root lockfiles are handled safely and boundedly. Credential heuristics redact values, ignore obvious templates/environment references, and still flag real-looking fixture literals for review.
|
||
|
||
Each finding is enriched by a backend rule definition with confidence, why-it-matters text, recommended action, remediation class and explainable priority. The browser renders retained intelligence; it does not duplicate rule knowledge or mutate external repositories.
|
||
|
||
## Architecture graph
|
||
|
||
Python imports are parsed using the standard-library AST. Local JavaScript and TypeScript relative imports, bounded `baseUrl`/`paths` aliases from `tsconfig.json`/`jsconfig.json`, and simple local workspace package imports are resolved against known source files. Only local edges are retained; this remains heuristic source analysis, not a TypeScript compiler. The displayed graph is bounded to 160 nodes and 600 edges to keep large repositories responsive; the response explicitly indicates truncation, reports total discovered nodes/edges and distinguishes visible coupling from total coupling used in hotspot ranking. Extensionless, index and `.js`-specifier-to-TypeScript resolution are supported.
|
||
|
||
## Hotspot model
|
||
|
||
Hotspot score is deterministic and combines source size, debt markers, recent changed-line churn and local inbound/outbound coupling. The UI surfaces both the score and reasons so it is not a black-box ranking.
|
||
|
||
## Health and release gate
|
||
|
||
Category scores start at 100 and are reduced only by findings with an explicit deduction. The overall health score is the average category score. The release gate separately evaluates blocker security findings, test presence, CI, dependency hygiene (including unbounded and unpinned declarations), repository health and engineering warnings, producing `READY`, `READY WITH WARNINGS` or `BLOCKED`.
|
||
|
||
## History and comparison
|
||
|
||
Every report is serialized as a snapshot in SQLite using WAL. Lightweight scan summary columns (readiness, counts, primary signal, deltas and triage priority) drive catalog CTE queries without deserializing full reports. Comparison calculates metric/category deltas and added/resolved findings against the chronological previous scan of the same repository identity. Findings carry stable semantic fingerprints so line movement does not appear as a new risk. Trend endpoints return bounded chronological snapshots for UI visualization. By default, up to 200 snapshots are retained per repository, the latest 30 remain detailed, older retained payloads compact to summary evidence, and snapshots older than 365 days are removed. v1 path history is backfilled into a local catalog identity and is never deleted solely for migration.
|
||
|
||
## Reporting
|
||
|
||
RepoPulse can export the complete JSON report or a standalone HTML report. HTML output escapes repository-derived text, export headers use sanitized RFC 5987 filenames, and neither format includes detected secret values or Gitea credentials. A root `.repopulse.yml`/`.yaml` may configure automatic scanning, minimum health, blocker allowance, required tests/CI and finding-code suppressions. Suppressed observations remain visible and are excluded from deductions and gate blockers.
|
||
|
||
## Job state and failure isolation
|
||
|
||
The persisted queue records repository, trigger, requested/actual revision, phase, progress, attempt, timestamps, resulting scan and sanitized error information. Claims use `BEGIN IMMEDIATE` plus a status guard, so two processes cannot own the same queued job. They move through `queued → acquiring_source → scanning → analyzing → persisting → completed` or an isolated `failed` outcome. A heartbeat renews each active lease; only a crashed worker’s expired lease returns work to `queued` for recovery. Older unstarted revisions become `superseded`; explicit cancellation states remain representable. One repository failure does not terminate the worker loop or affect other queued repositories.
|
||
|
||
## Operator maintenance mode
|
||
|
||
The `app_state` key `maintenance_mode` durably records whether operator maintenance is enabled; it uses the existing state table and does not change the SQLite schema version. Operator enter/exit endpoints require `REPOPULSE_MAINTENANCE_TOKEN` (or the existing general operator token on legacy installations). Entering is serialized against API mutations and catalog synchronization so an accepted external write completes before the pause is committed. While paused, mutation requests and webhooks return `423 Locked`; the scheduler skips heartbeat writes, Gitea synchronization, queue claims and new scan-job enqueue operations. Existing queued jobs remain queued for later resumption. A job claimed before the mode transition may complete and persist its result. Queue depth, active count, active-drain state and full-drain state are reported by the authenticated maintenance endpoint and the read-only meta endpoint. A malformed persisted maintenance value fails closed as paused.
|
||
|
||
## Operational configuration
|
||
|
||
`REPOPULSE_GITEA_URL` and `REPOPULSE_GITEA_TOKEN` enable the provider. `REPOPULSE_GITEA_SYNC_INTERVAL_SECONDS`, `REPOPULSE_GITEA_BRANCH_WORKERS`, `REPOPULSE_MAX_QUEUED_SCANS`, `REPOPULSE_JOB_LEASE_SECONDS`, scan-budget and retention variables, and `REPOPULSE_REPOSITORY_CACHE` bound reconciliation, queue depth, active leases, analysis and cache placement. The durable Gitea runner deliberately has one worker; `/api/v1/meta` exposes requested versus actual capacity. `REPOPULSE_GITEA_WEBHOOK_SECRET` enables signed default-branch push intake; non-push, non-default-branch, invalid and branch-deletion events are acknowledged without scheduling work. `REPOPULSE_OPERATOR_TOKEN` protects mutating endpoints when an authenticated proxy perimeter is not the selected boundary. `/api/v1/meta` exposes repository state counts, queue depth, storage/retention metrics, recovery status, worker heartbeat, connector URL (without token) and last sync result.
|