120 lines
14 KiB
Markdown
120 lines
14 KiB
Markdown
# ServiceMap architecture
|
|
|
|
## Runtime shape
|
|
|
|
ServiceMap remains deliberately small: one FastAPI process, one React single-page application and one SQLite database. Compose serves React through Nginx; the Unraid image bundles the same frontend and serves it from FastAPI.
|
|
|
|
```text
|
|
Browser
|
|
│ REST + SSE
|
|
▼
|
|
React workspaces
|
|
│
|
|
▼
|
|
FastAPI
|
|
├── service inventory and dependency validation
|
|
├── probe engine (HTTP, TCP, DNS, TLS)
|
|
├── state machine and flapping observation
|
|
├── health-evidence resolution and provider health
|
|
├── incident correlation and event journal
|
|
├── SSE fan-out bus
|
|
├── atomic config import/export
|
|
├── Docker/AppOps discovery and reconciliation
|
|
├── evidence-backed canonical topology overrides
|
|
└── optional per-service scheduler
|
|
│
|
|
▼
|
|
SQLite on /data
|
|
```
|
|
|
|
## Data ownership and migration
|
|
|
|
`Store` owns SQLite access and opens short-lived connections with WAL mode, a 5-second busy timeout and foreign keys enabled. Startup creates missing discovery tables and adds versioned metadata in place. v1.6 adds health evidence, evidence-to-service links and provider-status rows; the one-time migration projects only legitimate existing latest probe/runtime facts with their original timestamps. Existing rows receive safe defaults; the migration never deletes or rewrites topology data.
|
|
|
|
v1.6.1 adds no schema migration. Its health-only observation loop is distinct from discovery: every configured cadence it reads Docker runtime/healthcheck facts and AppOps application inventory, renews only source-scoped evidence/provider status, and skips demo state. It does not create services, suggestions, relationship evidence, snapshots, probe endpoints or operator overrides. The shared observation TTL is at least 150 seconds and three normal cycles, so a delayed provider cycle ages evidence rather than immediately turning the topology unknown. Provider failures are isolated; they update that provider's availability only and leave other current evidence intact.
|
|
|
|
v1.6.2 keeps the same schema and evidence scopes. An AppOps `warning` observation is degraded application evidence; it cannot resolve a component. Evidence freshness is recomputed when persisted claims are read, so an expired claim cannot appear fresh in the raw inspector/API payload. The current MCP Hub Docker read broker cannot replace the production socket provider yet: its registration covers two unrelated projects, its ServiceMap-scoped inventory is empty, and its short-lived HMAC service-token contract rejects a raw bearer secret. Cutover requires a dedicated ServiceMap read identity, full bounded inventory parity and an explicit network path; broker mode remains fail-closed with no socket fallback.
|
|
|
|
The HTTP discovery routes accept only the Docker host and AppOps URL selected in deployment configuration. A request body cannot turn the container into a general-purpose client for another Docker daemon or internal HTTP endpoint. The private-LAN deployment currently keeps OIDC disabled because it has no dedicated browser client or tested owner login path; mutation APIs therefore depend on the LAN perimeter. Broker mode must be deployed without the legacy Docker socket bind even though the current socket-mode Compose file retains it.
|
|
|
|
The v1.4 semantic projection is deliberately not a parallel datastore. `semantic.py` derives Domain → Application → Component from canonical services, relations, runtime/monitoring state and incident analysis. Logical relations are grouped at query time so one graph edge retains all independent evidence records and their first/last-seen provenance. v1.6.1 resolves application health separately: fresh aggregate AppOps evidence can answer an application but never a component; direct component failures override weak aggregate health and conflicting claims remain visible. Semantic application identity uses locked operator grouping, Compose provenance and stable source references before a display name, preventing generic provider labels from collapsing unrelated services. Bulk evidence lookup serves the complete semantic projection in one bounded SQL query rather than an application-by-application scan.
|
|
|
|
The service record stores descriptive metadata, group/environment/tags, criticality, source reference, application, runtime state/health, monitoring state, probe source/reachability/confidence, enablement, maintenance state, probe policy and state-machine counters. Probe rows remain immutable and retain `trigger` and bounded JSON `metadata` fields. `HealthEvidence` retains provider, subject, scope, confidence, observation/receipt/expiry timestamps, metadata and supersession. Application-scoped claims may be linked to component inspectors but cannot resolve a component's direct health.
|
|
|
|
Indexes cover service/time probe history, global probe time, event time/service/incident and incident status/root lookups. Availability is calculated with SQL aggregates over a bounded time window rather than loading all rows into Python. Evidence timestamp renewals retain a stable row and do not publish SSE; material provider status, evidence status, confidence, summary or metadata changes do.
|
|
|
|
## Health state and dependency semantics
|
|
|
|
The direct state machine is backend-owned, then exposed through an evidence policy:
|
|
|
|
```text
|
|
unknown ──success──> healthy
|
|
healthy ──failure threshold──> degraded/down
|
|
down/degraded ──success #1──> recovering
|
|
recovering ──recovery threshold──> healthy
|
|
any monitored state ──maintenance──> maintenance
|
|
```
|
|
|
|
The first failure can be held at the prior/unknown state when a failure threshold is greater than one. Repeated state changes in a bounded 30-minute query set the `flapping` observation and its count; flapping is not a competing health state.
|
|
|
|
Effective health is calculated recursively from direct state and dependencies. A down dependency makes a healthy dependent degraded; the stored probe observation is never rewritten. Cycles and missing dependencies are rejected before writes.
|
|
|
|
Maintenance and disabled monitoring do not produce a new incident. A running Docker component without a verified endpoint or Docker healthcheck is `unmonitored`, not `down`; this state cannot fabricate a root cause. Docker `unhealthy` and stopped/dead runtime observations are valid direct-down evidence without a synthetic network probe. The resolver selects fresh component-scoped evidence deterministically, exposes conflicts rather than collapsing them and excludes stale/expired claims. AppOps inventory health is application-scoped provenance, not a component probe.
|
|
|
|
## Probe engine
|
|
|
|
`probes.py` uses bounded async operations and one asyncio lock per service. Scheduler, probe-all and manual probe calls therefore coalesce at the service boundary rather than running duplicate network checks concurrently.
|
|
|
|
- HTTP follows redirects, supports expected status ranges and records status/latency.
|
|
- TCP opens and closes a bounded connection.
|
|
- DNS resolves a hostname, returns at most 20 unique addresses and reports no-answer/timeout failures.
|
|
- TLS performs a direct hostname-verified handshake, reports expiry metadata and applies a configurable warning threshold (`SERVICEMAP_TLS_WARNING_DAYS`, default 30).
|
|
|
|
Targets are explicitly typed. HTTP accepts only `http`/`https` URLs; TCP/TLS require host and port; DNS requires a host and does not accept arbitrary URI schemes. There is no `file://` or generic URL fetcher.
|
|
|
|
Per-service interval, timeout and retry settings override the global defaults. Scheduled observations are marked `scheduled`; UI-triggered observations are marked `manual`.
|
|
|
|
## Incidents and events
|
|
|
|
`incidents.py` deterministically finds a down service with no effectively-down dependency, then traverses the reverse graph for blast radius. `sync_incidents()` reconciles candidates with the durable journal:
|
|
|
|
- one active incident per root service;
|
|
- repeated probe cycles update impact instead of opening duplicates;
|
|
- acknowledgement changes `open` to `acknowledged` and persists a timestamp;
|
|
- recovery changes the record to `resolved` and retains `resolved_at`;
|
|
- state transitions, flapping, incident lifecycle and maintenance changes are journaled.
|
|
|
|
Incident timeline entries are events tied to an incident. Raw probe rows are retained independently under the configured probe retention; non-incident events use `SERVICEMAP_EVENT_RETENTION_DAYS` (default 90). Incident lifecycle events are retained with the incident record.
|
|
|
|
## SSE and frontend refresh
|
|
|
|
The durable SQLite event journal is the source of truth. A small thread-safe bounded queue is allocated per SSE client and receives notifications for probe completion, state changes, topology changes and incident lifecycle changes. Slow clients lose only transient wakeups, not durable history. The stream emits keepalives and unregisters queues on disconnect/shutdown.
|
|
|
|
React subscribes with SSE, debounces event refreshes and reconnects with backoff. It enables bounded fallback refresh only while the stream is unavailable; a healthy quiet stream does not trigger periodic full reloads. Authenticated clients use the same SSE protocol through `fetch` with a bearer header. The frontend never invents monitoring state optimistically; every node, counter and inspector value comes from the API.
|
|
|
|
## Production discovery and reconciliation
|
|
|
|
`DiscoveryStore` owns durable runs, source records, observations, relation evidence, suggestions, overrides, topology-change events and compact canonical topology snapshots. `DiscoveryManager` reads the configured Docker endpoint and optional read-only AppOps inventory, then separates observed candidates from inferred relationships and canonical services. Candidates include stable Compose/name identity, image, state, health, labels, networks, ports, Compose metadata, healthcheck presence and sanitized endpoint-host evidence. ServiceMap labels (`servicemap.*`), Compose `depends_on`, safe environment hostnames and AppOps mappings produce explainable suggestions. Numeric AppOps `discovery_confidence` gates classification strength, while its app health remains scoped provenance. Shared networks are low-confidence review hints only.
|
|
|
|
Discovery runs do not mutate canonical topology. Reconciliation updates existing observed records and only imports new services or accepts high-confidence dependency relationships when the user explicitly requests it. Compose references resolve only inside the same project; environment references use project/network/global-unique scope and ambiguous matches remain review suggestions. Manual overrides and locked edges are preserved across later runs. Missing Docker identities move through `missing`, `stale` and `retired` without deleting probe history. Health propagation uses canonical dependency edges only; evidence-only or rejected relationships cannot affect health.
|
|
|
|
Published ports become active candidates only when `SERVICEMAP_PROBE_HOST` is explicitly configured. Internal Docker names are retained as evidence with `internal_only` reachability, never sent to the network probe engine by default. The reachability-repair endpoint provides a dry-run report before it disables unsafe legacy candidates or selects a verified published endpoint; it does not delete history and resolves false incidents with an endpoint-superseded event reason.
|
|
|
|
After reconciliation, a revision-hashed snapshot is captured only when canonical services or dependency edges change. The Changes workspace compares two snapshots and makes a service/relationship change actionable without auto-applying anything. The production UI starts with the canonical topology, exposes discovery diffs and evidence, supports individual accept/reject decisions, and offers an explicit edit mode for pinned node layout and manual dependency edges. Hidden/ignored lifecycle records remain available through the `include_hidden` API for reconciliation and audit context. ELK route sections are passed to `LivingEdge`, so its directed SVG path follows the actual orthogonal layout.
|
|
|
|
## Configuration
|
|
|
|
Config v1 is still accepted. Config v2 serializes the service metadata and policy fields by human-readable service name. Parsing validates endpoint fields, duplicate names, duplicate/self edges and document-contained cycles. Import then validates the projected merge or replace graph and commits it in one `BEGIN IMMEDIATE` transaction.
|
|
|
|
## Demo safety
|
|
|
|
The Northstar demo is built from deterministic UUIDs and synthetic probe records. Entering demo stores JSON snapshots of services, probes, events, incidents, discovery evidence, health evidence and provider state in the durable `demo_sessions` row. Reset replaces only the synthetic workspace while preserving the original snapshot. Exit restores the snapshot and deletes the session. Scheduled and manual live probes and discovery reconciliation are blocked while the demo is active.
|
|
|
|
## Deployment trust boundary
|
|
|
|
The Unraid target uses `/data` on the named `servicemap_data` volume, port mapping `1253:8000`, a healthcheck on `/health`, `no-new-privileges`, dropped capabilities, a read-only root filesystem and bounded `/tmp` tmpfs. Managed validation gates exact-SHA autoredeploy. Socket discovery remains a high-trust Docker API boundary even with a read-only bind. The opt-in broker provider requires a dedicated short-lived ServiceMap token, registration-scoped read capability and shared network, and never falls back to the socket. AppOps access is read-only enrichment.
|
|
|
|
## Optional authorization
|
|
|
|
`SERVICEMAP_AUTH_MODE=disabled` preserves the deployed v1.5 behavior. In `oidc` mode FastAPI validates issuer/audience/signed bearer JWTs against OIDC discovery and JWKS; it maps explicit roles or configured Authentik groups to viewer, operator and admin. `/health` remains anonymous for deployment recovery, while APIs fail closed when OIDC configuration is incomplete. No client credentials, issuer secrets or tokens are stored in ServiceMap's SQLite database or frontend configuration.
|