Initial curated itworx-knowledge-hub source from 3b1751d0d9dee551aa6994eee0093704dc334817

docs/19-observability.md

@@ -0,0 +1,145 @@
+# Observability
+
+## Goals
+
+Operators must be able to answer:
+
+- Is the application alive and ready?
+- Is PostgreSQL available?
+- Are migrations compatible?
+- Are background jobs progressing?
+- Did an import, backup or integrity scan fail?
+- Is search projection current?
+- Are attachment files missing or corrupt?
+- Which release and schema are running?
+- What request/job correlation identifies a failure?
+
+Observability must not expose private knowledge or secrets.
+
+## Health endpoints
+
+### `/health/live`
+
+Answers only whether the process can respond and has not entered a fatal internal state.
+It must not fail because PostgreSQL is temporarily unavailable.
+
+### `/health/ready`
+
+Checks required runtime dependencies:
+
+- compatible database/schema;
+- database connectivity;
+- writable required storage;
+- data-protection key access;
+- critical startup jobs complete;
+- no maintenance lock preventing normal traffic.
+
+Return minimal public details. Detailed diagnostics require administrator authentication.
+
+### `/api/v1/system/info`
+
+Permissioned information:
+
+- product/version/commit;
+- schema version;
+- uptime;
+- configured locale/timezone;
+- storage/job/search status;
+- feature flags;
+- safe dependency versions.
+
+## Logging
+
+Production uses structured JSON to stdout/stderr for container collection.
+
+Core properties:
+
+- timestamp;
+- level;
+- event ID/name;
+- message template;
+- correlation/request ID;
+- actor kind and safe ID where authenticated;
+- route/status/duration;
+- job/import/backup ID;
+- exception type and safe stack in operator logs.
+
+Never log page content, passwords, tokens, cookies, authorization headers, connection
+strings or attachment bytes.
+
+Use explicit redaction and tests for sensitive property names.
+
+## Audit vs logs
+
+- Diagnostic logs help operate and debug.
+- Audit events prove who did what to protected resources.
+
+Do not rely on transient container logs as the audit trail.
+
+## Metrics
+
+Expose bounded metrics to administrators and optionally a protected scrape endpoint:
+
+- request count/duration/status;
+- active sessions (approximate, non-identifying);
+- DB pool saturation;
+- job queue counts and oldest age;
+- import/backup outcomes;
+- search projection lag;
+- attachment storage bytes/count;
+- integrity failures;
+- login failure/lockout counters without username labels.
+
+Avoid high-cardinality labels such as page ID, user ID or query text.
+
+## Tracing
+
+Use correlation IDs across HTTP, application commands and jobs. OpenTelemetry-compatible
+instrumentation is acceptable but no external collector is required.
+
+## Background-job visibility
+
+Admin UI shows:
+
+- type;
+- state;
+- queued/started/completed time;
+- progress counts;
+- attempts;
+- next retry;
+- cancellation availability;
+- safe failure summary;
+- correlation ID.
+
+Raw stack traces remain in operator logs, not ordinary UI.
+
+## Alerts
+
+The initial system provides persistent in-app administrator alerts for:
+
+- repeated backup failure;
+- failed restore rehearsal;
+- import validation failure;
+- storage unwritable/low;
+- missing/corrupt blobs;
+- incompatible schema;
+- stuck critical job;
+- token/security event requiring attention.
+
+External alert integrations are deferred.
+
+## Retention
+
+### Trusted-intake signals
+
+Record proposal count, processing-stage duration, review duration and publication
+count with only bounded `stage`, `outcome` and `problem_type` labels. Correlation
+IDs connect requests to safe operational events. Never put source bodies, source
+URLs, filenames, bearer tokens, credentials or private paths in metrics, logs or
+release evidence. Alert on sustained failed proposals, stuck review state and
+publication failures without exposing proposal content.
+
+- Configure log rotation at Docker/Unraid level.
+- Keep audit retention in the application/database.
+- Prune metrics/job detail according to policy.
+- Preserve final import/backup/restore reports longer than ordinary job logs.