146 lines
3.8 KiB
Markdown
146 lines
3.8 KiB
Markdown
# Observability
|
|
|
|
## Goals
|
|
|
|
Operators must be able to answer:
|
|
|
|
- Is the application alive and ready?
|
|
- Is PostgreSQL available?
|
|
- Are migrations compatible?
|
|
- Are background jobs progressing?
|
|
- Did an import, backup or integrity scan fail?
|
|
- Is search projection current?
|
|
- Are attachment files missing or corrupt?
|
|
- Which release and schema are running?
|
|
- What request/job correlation identifies a failure?
|
|
|
|
Observability must not expose private knowledge or secrets.
|
|
|
|
## Health endpoints
|
|
|
|
### `/health/live`
|
|
|
|
Answers only whether the process can respond and has not entered a fatal internal state.
|
|
It must not fail because PostgreSQL is temporarily unavailable.
|
|
|
|
### `/health/ready`
|
|
|
|
Checks required runtime dependencies:
|
|
|
|
- compatible database/schema;
|
|
- database connectivity;
|
|
- writable required storage;
|
|
- data-protection key access;
|
|
- critical startup jobs complete;
|
|
- no maintenance lock preventing normal traffic.
|
|
|
|
Return minimal public details. Detailed diagnostics require administrator authentication.
|
|
|
|
### `/api/v1/system/info`
|
|
|
|
Permissioned information:
|
|
|
|
- product/version/commit;
|
|
- schema version;
|
|
- uptime;
|
|
- configured locale/timezone;
|
|
- storage/job/search status;
|
|
- feature flags;
|
|
- safe dependency versions.
|
|
|
|
## Logging
|
|
|
|
Production uses structured JSON to stdout/stderr for container collection.
|
|
|
|
Core properties:
|
|
|
|
- timestamp;
|
|
- level;
|
|
- event ID/name;
|
|
- message template;
|
|
- correlation/request ID;
|
|
- actor kind and safe ID where authenticated;
|
|
- route/status/duration;
|
|
- job/import/backup ID;
|
|
- exception type and safe stack in operator logs.
|
|
|
|
Never log page content, passwords, tokens, cookies, authorization headers, connection
|
|
strings or attachment bytes.
|
|
|
|
Use explicit redaction and tests for sensitive property names.
|
|
|
|
## Audit vs logs
|
|
|
|
- Diagnostic logs help operate and debug.
|
|
- Audit events prove who did what to protected resources.
|
|
|
|
Do not rely on transient container logs as the audit trail.
|
|
|
|
## Metrics
|
|
|
|
Expose bounded metrics to administrators and optionally a protected scrape endpoint:
|
|
|
|
- request count/duration/status;
|
|
- active sessions (approximate, non-identifying);
|
|
- DB pool saturation;
|
|
- job queue counts and oldest age;
|
|
- import/backup outcomes;
|
|
- search projection lag;
|
|
- attachment storage bytes/count;
|
|
- integrity failures;
|
|
- login failure/lockout counters without username labels.
|
|
|
|
Avoid high-cardinality labels such as page ID, user ID or query text.
|
|
|
|
## Tracing
|
|
|
|
Use correlation IDs across HTTP, application commands and jobs. OpenTelemetry-compatible
|
|
instrumentation is acceptable but no external collector is required.
|
|
|
|
## Background-job visibility
|
|
|
|
Admin UI shows:
|
|
|
|
- type;
|
|
- state;
|
|
- queued/started/completed time;
|
|
- progress counts;
|
|
- attempts;
|
|
- next retry;
|
|
- cancellation availability;
|
|
- safe failure summary;
|
|
- correlation ID.
|
|
|
|
Raw stack traces remain in operator logs, not ordinary UI.
|
|
|
|
## Alerts
|
|
|
|
The initial system provides persistent in-app administrator alerts for:
|
|
|
|
- repeated backup failure;
|
|
- failed restore rehearsal;
|
|
- import validation failure;
|
|
- storage unwritable/low;
|
|
- missing/corrupt blobs;
|
|
- incompatible schema;
|
|
- stuck critical job;
|
|
- token/security event requiring attention.
|
|
|
|
External alert integrations are deferred.
|
|
|
|
## Retention
|
|
|
|
### Trusted-intake signals
|
|
|
|
Record proposal count, processing-stage duration, review duration and publication
|
|
count with only bounded `stage`, `outcome` and `problem_type` labels. Correlation
|
|
IDs connect requests to safe operational events. Never put source bodies, source
|
|
URLs, filenames, bearer tokens, credentials or private paths in metrics, logs or
|
|
release evidence. Alert on sustained failed proposals, stuck review state and
|
|
publication failures without exposing proposal content.
|
|
|
|
- Configure log rotation at Docker/Unraid level.
|
|
- Keep audit retention in the application/database.
|
|
- Prune metrics/job detail according to policy.
|
|
- Preserve final import/backup/restore reports longer than ordinary job logs.
|