3.8 KiB
Observability
Goals
Operators must be able to answer:
- Is the application alive and ready?
- Is PostgreSQL available?
- Are migrations compatible?
- Are background jobs progressing?
- Did an import, backup or integrity scan fail?
- Is search projection current?
- Are attachment files missing or corrupt?
- Which release and schema are running?
- What request/job correlation identifies a failure?
Observability must not expose private knowledge or secrets.
Health endpoints
/health/live
Answers only whether the process can respond and has not entered a fatal internal state. It must not fail because PostgreSQL is temporarily unavailable.
/health/ready
Checks required runtime dependencies:
- compatible database/schema;
- database connectivity;
- writable required storage;
- data-protection key access;
- critical startup jobs complete;
- no maintenance lock preventing normal traffic.
Return minimal public details. Detailed diagnostics require administrator authentication.
/api/v1/system/info
Permissioned information:
- product/version/commit;
- schema version;
- uptime;
- configured locale/timezone;
- storage/job/search status;
- feature flags;
- safe dependency versions.
Logging
Production uses structured JSON to stdout/stderr for container collection.
Core properties:
- timestamp;
- level;
- event ID/name;
- message template;
- correlation/request ID;
- actor kind and safe ID where authenticated;
- route/status/duration;
- job/import/backup ID;
- exception type and safe stack in operator logs.
Never log page content, passwords, tokens, cookies, authorization headers, connection strings or attachment bytes.
Use explicit redaction and tests for sensitive property names.
Audit vs logs
- Diagnostic logs help operate and debug.
- Audit events prove who did what to protected resources.
Do not rely on transient container logs as the audit trail.
Metrics
Expose bounded metrics to administrators and optionally a protected scrape endpoint:
- request count/duration/status;
- active sessions (approximate, non-identifying);
- DB pool saturation;
- job queue counts and oldest age;
- import/backup outcomes;
- search projection lag;
- attachment storage bytes/count;
- integrity failures;
- login failure/lockout counters without username labels.
Avoid high-cardinality labels such as page ID, user ID or query text.
Tracing
Use correlation IDs across HTTP, application commands and jobs. OpenTelemetry-compatible instrumentation is acceptable but no external collector is required.
Background-job visibility
Admin UI shows:
- type;
- state;
- queued/started/completed time;
- progress counts;
- attempts;
- next retry;
- cancellation availability;
- safe failure summary;
- correlation ID.
Raw stack traces remain in operator logs, not ordinary UI.
Alerts
The initial system provides persistent in-app administrator alerts for:
- repeated backup failure;
- failed restore rehearsal;
- import validation failure;
- storage unwritable/low;
- missing/corrupt blobs;
- incompatible schema;
- stuck critical job;
- token/security event requiring attention.
External alert integrations are deferred.
Retention
Trusted-intake signals
Record proposal count, processing-stage duration, review duration and publication
count with only bounded stage, outcome and problem_type labels. Correlation
IDs connect requests to safe operational events. Never put source bodies, source
URLs, filenames, bearer tokens, credentials or private paths in metrics, logs or
release evidence. Alert on sustained failed proposals, stuck review state and
publication failures without exposing proposal content.
- Configure log rotation at Docker/Unraid level.
- Keep audit retention in the application/database.
- Prune metrics/job detail according to policy.
- Preserve final import/backup/restore reports longer than ordinary job logs.