200 lines
9.1 KiB
Markdown
200 lines
9.1 KiB
Markdown
# Backup and restore
|
|
|
|
## Principle
|
|
|
|
A backup is not considered valid until it is internally verified, and the backup system is
|
|
not considered production-ready until a clean-target restore is tested.
|
|
|
|
## Backup contents
|
|
|
|
A full backup set contains:
|
|
|
|
1. PostgreSQL logical dump in a supported restorable format.
|
|
2. Attachment blobs.
|
|
3. Optional generated exports required by policy.
|
|
4. Application configuration manifest without secrets.
|
|
5. Product, schema and container/image versions.
|
|
6. File inventory with sizes and SHA-256 hashes.
|
|
7. Backup job metadata and timestamps.
|
|
8. A signed integrity receipt authenticating the exact manifest.
|
|
9. Encryption metadata when operator-managed encryption is enabled.
|
|
|
|
Do not include runtime session secrets or plaintext API tokens.
|
|
|
|
## Consistency strategy
|
|
|
|
Preferred v1 approach:
|
|
|
|
- create a PostgreSQL consistent logical dump;
|
|
- snapshot/copy immutable content-addressed attachment blobs;
|
|
- capture attachment metadata and blob manifest from the same defined backup window;
|
|
- validate that every active DB blob reference exists in the manifest;
|
|
- record files added after the cutoff for the next backup.
|
|
|
|
Because committed blobs are immutable, exact transaction coordination is simpler than for
|
|
mutable files. Deletions use retention, preventing a blob from disappearing during backup.
|
|
|
|
## Backup workflow
|
|
|
|
1. Acquire a backup job/lease.
|
|
2. Record application/schema version and cutoff.
|
|
3. Run PostgreSQL dump with failure-on-error.
|
|
4. Inventory referenced attachment hashes.
|
|
5. copy/link/archive required blobs without following unsafe paths;
|
|
6. create manifest;
|
|
7. verify dump readability and every file hash;
|
|
8. authenticate the manifest hash with HMAC-SHA-256;
|
|
9. optionally encrypt with an operator-supplied mechanism;
|
|
10. atomically publish completed backup;
|
|
11. apply retention only after a verified new backup exists;
|
|
12. record audit and administrator-visible result.
|
|
|
|
A failed backup never replaces the last known-good backup.
|
|
|
|
New backup directories and files use private Unix permissions (0700 and 0600).
|
|
Permission failures stop creation before publication. On Windows, provision and
|
|
verify restrictive NTFS ACLs for backups, keys and restore targets before use;
|
|
Unix mode checks do not enforce Windows ACLs. HMAC authenticates backups but
|
|
does not encrypt their contents, so archive/share access must remain restricted.
|
|
|
|
## Backup authenticity key and recovery
|
|
|
|
The `.integrity.json` receipt beside a backup is not trusted by itself. The application
|
|
authenticates its backup ID, exact manifest hash, completion time and verification result
|
|
with HMAC-SHA-256. The random 32-byte key is stored at
|
|
`KnowledgeHub:Paths:Keys/backup-integrity.key`; in the supplied Compose deployment this
|
|
is `/var/lib/knowledge-hub/keys/backup-integrity.key`. This location is outside
|
|
`KnowledgeHub:Paths:Backups`, so someone who can rewrite a backup and its receipt cannot
|
|
create a valid signature. The application creates the key when integrity evidence is
|
|
first written and restricts it to the service account on Unix hosts.
|
|
|
|
Treat this key as essential recovery material. Copy the exact key file into an encrypted,
|
|
access-controlled escrow location separate from the backup storage, and include it in
|
|
the operator's disaster-recovery procedure. To restore on a replacement host, recover
|
|
the key to the configured Keys directory before verifying or restoring backups. Never
|
|
put the key in a backup directory or in the same backup archive as the signed receipts.
|
|
Protect access to the escrow copy as carefully as access to the backups.
|
|
|
|
If the key is missing or unreadable, backup integrity is reported as unknown and restore
|
|
is refused. Do not replace a lost key with a newly generated one and expect existing
|
|
backups to validate: a new key authenticates only backups created after it. Recover the
|
|
original key from escrow first. If no copy exists, the existing backups cannot be
|
|
cryptographically authenticated by the application and must not be treated as verified.
|
|
Backups made before authenticated receipts were introduced also remain unverified; create
|
|
a new backup with this version before relying on application-verified restore.
|
|
|
|
## Retention defaults
|
|
|
|
Configurable starting policy:
|
|
|
|
- 7 daily;
|
|
- 4 weekly;
|
|
- 6 monthly;
|
|
- at least one manually protected backup.
|
|
|
|
Retention must account for available storage and never delete an in-progress restore
|
|
source.
|
|
|
|
## Restore workflow
|
|
|
|
Restore is deliberately separate from normal application startup.
|
|
|
|
1. Select backup. Copy read-only media to a writable recovery directory first,
|
|
preserving the manifest, signed receipts and file bytes. Recover the exact
|
|
`backup-integrity.key` from protected external escrow into the configured Keys
|
|
directory; never take it from the backup archive. The application acquires an
|
|
exclusive storage lease there to prevent retention from removing a restore source,
|
|
and writes its restore receipt there. Keep the original media unchanged.
|
|
2. Use the backup's recorded application image to create a clean PostgreSQL
|
|
database at exactly the manifest's schema version. Create an existing, empty
|
|
target attachments directory on writable storage with room for the full payload.
|
|
3. Copy the manifest and every listed database and attachment file into a private
|
|
staging directory inside the target attachments volume. Verify the HMAC against
|
|
the staged manifest bytes and verify each staged file's size and SHA-256 before
|
|
opening or changing the target database. Restore reads only from this verified
|
|
snapshot, so source-volume changes after verification cannot change restored bytes.
|
|
4. Restore database. The restore service refuses a target whose latest applied
|
|
migration differs from the authenticated manifest, before inserting any rows.
|
|
5. Restore attachment blobs by hash.
|
|
6. Verify the restored data with the recorded application version.
|
|
7. Upgrade through supported migrations using the intended newer image, then
|
|
repeat integrity checks before promotion.
|
|
8. Run integrity checks:
|
|
- FK/schema;
|
|
- user/membership;
|
|
- page/revision counts;
|
|
- current content hashes;
|
|
- link counts;
|
|
- project counts;
|
|
- attachment references vs blobs;
|
|
- import/audit consistency.
|
|
9. Run authenticated smoke tests.
|
|
10. Produce restore report.
|
|
11. Promote target only through an explicit operator step.
|
|
|
|
Never test restore by overwriting the only production instance.
|
|
|
|
Pass target PostgreSQL credentials through `--target-connection-file=<path>`, which
|
|
reads the connection string from a protected file. On Unix, the file must be readable
|
|
only by the restore account (mode `0600` or `0400`); the restore command rejects files
|
|
with group or other access. On Windows, apply an ACL that grants access only to the
|
|
restore operator/service account. Do not put the connection string in command-line
|
|
arguments. Restore errors deliberately omit provider exception details because they
|
|
can contain connection data.
|
|
|
|
The real-backup drill also requires the exact integrity key recovered from protected
|
|
external escrow. Pass only its file path through
|
|
`KNOWLEDGE_HUB_BACKUP_INTEGRITY_KEY_FILE`; the script copies the key with mode `0600`
|
|
into its temporary working directory, uses that directory as the restore Keys path,
|
|
and removes the temporary copy during cleanup. The key contents are never placed in an
|
|
argument or printed by the script.
|
|
|
|
## Backup encryption
|
|
|
|
The application may integrate an operator-supplied encryption command or library after
|
|
threat review. Requirements:
|
|
|
|
- key is not stored only inside the backup;
|
|
- command arguments do not expose secrets in process listings where avoidable;
|
|
- restore can verify authentication/integrity before extraction;
|
|
- failure is explicit;
|
|
- documentation includes key-loss consequences.
|
|
|
|
Unencrypted backups must be clearly labeled and protected by host/storage controls.
|
|
|
|
## Disaster recovery targets
|
|
|
|
Initial targets, to be confirmed on real hardware:
|
|
|
|
- RPO: last successful scheduled daily backup by default.
|
|
- RTO: documented single-host restore, target under 2 hours for typical personal dataset,
|
|
excluding transfer of very large attachments.
|
|
|
|
Do not claim an RTO until a measured production-like rehearsal exists.
|
|
|
|
## Tests
|
|
|
|
### Trusted-intake pilot recovery
|
|
|
|
Before enabling intake, capture a verified backup and restore it into an isolated,
|
|
empty target. Record the source commit, image digest, migration range, backup ID,
|
|
table counts and attachment hashes. A rollback of intake starts by setting
|
|
`IngestionProposal:Enabled=false` and revoking the separate client's service
|
|
account. Keep proposals, source references, attachment usages and audit history;
|
|
disabling the feature is not a retention-cleanup instruction. Published attachments
|
|
remain referenced by their pages. Restore a pre-release backup only after an
|
|
isolated restore proves it is necessary and the operator has assessed newer writes.
|
|
|
|
- create backup with representative data;
|
|
- verify manifest;
|
|
- reject a forged manifest and matching unsigned/invalid integrity receipt;
|
|
- corrupt dump and blob tests;
|
|
- clean-target restore;
|
|
- compare logical counts and hashes;
|
|
- restore older supported schema then migrate;
|
|
- missing integrity key or encryption key failure;
|
|
- interrupted backup cleanup;
|
|
- retention policy;
|
|
- no secret inclusion;
|
|
- large attachment set.
|