itworx-knowledge-hub-Public/docs/18-backup-restore.md

Backup and restore

Principle

A backup is not considered valid until it is internally verified, and the backup system is not considered production-ready until a clean-target restore is tested.

Backup contents

A full backup set contains:

1. PostgreSQL logical dump in a supported restorable format.
2. Attachment blobs.
3. Optional generated exports required by policy.
4. Application configuration manifest without secrets.
5. Product, schema and container/image versions.
6. File inventory with sizes and SHA-256 hashes.
7. Backup job metadata and timestamps.
8. Encryption metadata when operator-managed encryption is enabled.

Do not include runtime session secrets or plaintext API tokens.

Consistency strategy

Preferred v1 approach:

• create a PostgreSQL consistent logical dump;
• snapshot/copy immutable content-addressed attachment blobs;
• capture attachment metadata and blob manifest from the same defined backup window;
• validate that every active DB blob reference exists in the manifest;
• record files added after the cutoff for the next backup.

Because committed blobs are immutable, exact transaction coordination is simpler than for mutable files. Deletions use retention, preventing a blob from disappearing during backup.

Backup workflow

1. Acquire a backup job/lease.
2. Record application/schema version and cutoff.
3. Run PostgreSQL dump with failure-on-error.
4. Inventory referenced attachment hashes.
5. copy/link/archive required blobs without following unsafe paths;
6. create manifest;
7. verify dump readability and every file hash;
8. optionally encrypt with an operator-supplied mechanism;
9. atomically publish completed backup;
10. apply retention only after a verified new backup exists;
11. record audit and administrator-visible result.

A failed backup never replaces the last known-good backup.

Retention defaults

Configurable starting policy:

• 7 daily;
• 4 weekly;
• 6 monthly;
• at least one manually protected backup.

Retention must account for available storage and never delete an in-progress restore source.

Restore workflow

Restore is deliberately separate from normal application startup.

1. Select backup.
2. Verify manifest, hashes, version compatibility and encryption.
3. Create clean PostgreSQL database and empty target directories.
4. Restore database.
5. Restore attachment blobs by hash.
6. Start the recorded/current compatible application in maintenance mode.
7. Apply supported migrations.
8. Run integrity checks:
   • FK/schema;
   • user/membership;
   • page/revision counts;
   • current content hashes;
   • link counts;
   • project counts;
   • attachment references vs blobs;
   • import/audit consistency.
9. Run authenticated smoke tests.
10. Produce restore report.
11. Promote target only through an explicit operator step.

Never test restore by overwriting the only production instance.

Backup encryption

The application may integrate an operator-supplied encryption command or library after threat review. Requirements:

• key is not stored only inside the backup;
• command arguments do not expose secrets in process listings where avoidable;
• restore can verify authentication/integrity before extraction;
• failure is explicit;
• documentation includes key-loss consequences.

Unencrypted backups must be clearly labeled and protected by host/storage controls.

Disaster recovery targets

Initial targets, to be confirmed on real hardware:

• RPO: last successful scheduled daily backup by default.
• RTO: documented single-host restore, target under 2 hours for typical personal dataset, excluding transfer of very large attachments.

Do not claim an RTO until a measured production-like rehearsal exists.

Tests

Trusted-intake pilot recovery

Before enabling intake, capture a verified backup and restore it into an isolated, empty target. Record the source commit, image digest, migration range, backup ID, table counts and attachment hashes. A rollback of intake starts by setting IngestionProposal:Enabled=false and revoking the separate client's service account. Keep proposals, source references, attachment usages and audit history; disabling the feature is not a retention-cleanup instruction. Published attachments remain referenced by their pages. Restore a pre-release backup only after an isolated restore proves it is necessary and the operator has assessed newer writes.

• create backup with representative data;
• verify manifest;
• corrupt dump and blob tests;
• clean-target restore;
• compare logical counts and hashes;
• restore older supported schema then migrate;
• missing key/encrypted backup failure;
• interrupted backup cleanup;
• retention policy;
• no secret inclusion;
• large attachment set.