itworx-knowledge-hub-Public/docs/18-backup-restore.md

Backup and restore

Principle

A backup is not considered valid until it is internally verified, and the backup system is not considered production-ready until a clean-target restore is tested.

Backup contents

A full backup set contains:

1. PostgreSQL logical dump in a supported restorable format.
2. Attachment blobs.
3. Optional generated exports required by policy.
4. Application configuration manifest without secrets.
5. Product, schema and container/image versions.
6. File inventory with sizes and SHA-256 hashes.
7. Backup job metadata and timestamps.
8. A signed integrity receipt authenticating the exact manifest.
9. Encryption metadata when operator-managed encryption is enabled.

Do not include runtime session secrets or plaintext API tokens.

Consistency strategy

Preferred v1 approach:

• create a PostgreSQL consistent logical dump;
• snapshot/copy immutable content-addressed attachment blobs;
• capture attachment metadata and blob manifest from the same defined backup window;
• validate that every active DB blob reference exists in the manifest;
• record files added after the cutoff for the next backup.

Because committed blobs are immutable, exact transaction coordination is simpler than for mutable files. Deletions use retention, preventing a blob from disappearing during backup.

Backup workflow

1. Acquire a backup job/lease.
2. Record application/schema version and cutoff.
3. Run PostgreSQL dump with failure-on-error.
4. Inventory referenced attachment hashes.
5. copy/link/archive required blobs without following unsafe paths;
6. create manifest;
7. verify dump readability and every file hash;
8. authenticate the manifest hash with HMAC-SHA-256;
9. optionally encrypt with an operator-supplied mechanism;
10. atomically publish completed backup;
11. apply retention only after a verified new backup exists;
12. record audit and administrator-visible result.

A failed backup never replaces the last known-good backup.

New backup directories and files use private Unix permissions (0700 and 0600). Permission failures stop creation before publication. On Windows, provision and verify restrictive NTFS ACLs for backups, keys and restore targets before use; Unix mode checks do not enforce Windows ACLs. HMAC authenticates backups but does not encrypt their contents, so archive/share access must remain restricted.

Backup authenticity key and recovery

The .integrity.json receipt beside a backup is not trusted by itself. The application authenticates its backup ID, exact manifest hash, completion time and verification result with HMAC-SHA-256. The random 32-byte key is stored at KnowledgeHub:Paths:Keys/backup-integrity.key; in the supplied Compose deployment this is /var/lib/knowledge-hub/keys/backup-integrity.key. This location is outside KnowledgeHub:Paths:Backups, so someone who can rewrite a backup and its receipt cannot create a valid signature. The application creates the key when integrity evidence is first written and restricts it to the service account on Unix hosts.

Treat this key as essential recovery material. Copy the exact key file into an encrypted, access-controlled escrow location separate from the backup storage, and include it in the operator's disaster-recovery procedure. To restore on a replacement host, recover the key to the configured Keys directory before verifying or restoring backups. Never put the key in a backup directory or in the same backup archive as the signed receipts. Protect access to the escrow copy as carefully as access to the backups.

If the key is missing or unreadable, backup integrity is reported as unknown and restore is refused. Do not replace a lost key with a newly generated one and expect existing backups to validate: a new key authenticates only backups created after it. Recover the original key from escrow first. If no copy exists, the existing backups cannot be cryptographically authenticated by the application and must not be treated as verified. Backups made before authenticated receipts were introduced also remain unverified; create a new backup with this version before relying on application-verified restore.

Retention defaults

Configurable starting policy:

• 7 daily;
• 4 weekly;
• 6 monthly;
• at least one manually protected backup.

Retention must account for available storage and never delete an in-progress restore source.

Restore workflow

Restore is deliberately separate from normal application startup.

1. Select backup. Copy read-only media to a writable recovery directory first, preserving the manifest, signed receipts and file bytes. Recover the exact backup-integrity.key from protected external escrow into the configured Keys directory; never take it from the backup archive. The application acquires an exclusive storage lease there to prevent retention from removing a restore source, and writes its restore receipt there. Keep the original media unchanged.
2. Use the backup's recorded application image to create a clean PostgreSQL database at exactly the manifest's schema version. Create an existing, empty target attachments directory on writable storage with room for the full payload.
3. Copy the manifest and every listed database and attachment file into a private staging directory inside the target attachments volume. Verify the HMAC against the staged manifest bytes and verify each staged file's size and SHA-256 before opening or changing the target database. Restore reads only from this verified snapshot, so source-volume changes after verification cannot change restored bytes.
4. Restore database. The restore service refuses a target whose latest applied migration differs from the authenticated manifest, before inserting any rows.
5. Restore attachment blobs by hash.
6. Verify the restored data with the recorded application version.
7. Upgrade through supported migrations using the intended newer image, then repeat integrity checks before promotion.
8. Run integrity checks:
   • FK/schema;
   • user/membership;
   • page/revision counts;
   • current content hashes;
   • link counts;
   • project counts;
   • attachment references vs blobs;
   • import/audit consistency.
9. Run authenticated smoke tests.
10. Produce restore report.
11. Promote target only through an explicit operator step.

Never test restore by overwriting the only production instance.

Pass target PostgreSQL credentials through --target-connection-file=<path>, which reads the connection string from a protected file. On Unix, the file must be readable only by the restore account (mode 0600 or 0400); the restore command rejects files with group or other access. On Windows, apply an ACL that grants access only to the restore operator/service account. Do not put the connection string in command-line arguments. Restore errors deliberately omit provider exception details because they can contain connection data.

The real-backup drill also requires the exact integrity key recovered from protected external escrow. Pass only its file path through KNOWLEDGE_HUB_BACKUP_INTEGRITY_KEY_FILE; the script copies the key with mode 0600 into its temporary working directory, uses that directory as the restore Keys path, and removes the temporary copy during cleanup. The key contents are never placed in an argument or printed by the script.

Backup encryption

The application may integrate an operator-supplied encryption command or library after threat review. Requirements:

• key is not stored only inside the backup;
• command arguments do not expose secrets in process listings where avoidable;
• restore can verify authentication/integrity before extraction;
• failure is explicit;
• documentation includes key-loss consequences.

Unencrypted backups must be clearly labeled and protected by host/storage controls.

Disaster recovery targets

Initial targets, to be confirmed on real hardware:

• RPO: last successful scheduled daily backup by default.
• RTO: documented single-host restore, target under 2 hours for typical personal dataset, excluding transfer of very large attachments.

Do not claim an RTO until a measured production-like rehearsal exists.

Tests

Trusted-intake pilot recovery

Before enabling intake, capture a verified backup and restore it into an isolated, empty target. Record the source commit, image digest, migration range, backup ID, table counts and attachment hashes. A rollback of intake starts by setting IngestionProposal:Enabled=false and revoking the separate client's service account. Keep proposals, source references, attachment usages and audit history; disabling the feature is not a retention-cleanup instruction. Published attachments remain referenced by their pages. Restore a pre-release backup only after an isolated restore proves it is necessary and the operator has assessed newer writes.

• create backup with representative data;
• verify manifest;
• reject a forged manifest and matching unsigned/invalid integrity receipt;
• corrupt dump and blob tests;
• clean-target restore;
• compare logical counts and hashes;
• restore older supported schema then migrate;
• missing integrity key or encryption key failure;
• interrupted backup cleanup;
• retention policy;
• no secret inclusion;
• large attachment set.