Publish curated Knowledge Hub source from 2ed833f290b439a7bc91e5fd19b042c269e3bee9

docs/18-backup-restore.md

@@ -16,7 +16,8 @@ A full backup set contains:
 5. Product, schema and container/image versions.
 6. File inventory with sizes and SHA-256 hashes.
 7. Backup job metadata and timestamps.
-8. Encryption metadata when operator-managed encryption is enabled.
+8. A signed integrity receipt authenticating the exact manifest.
+9. Encryption metadata when operator-managed encryption is enabled.
 
 Do not include runtime session secrets or plaintext API tokens.
 
@@ -42,13 +43,46 @@ mutable files. Deletions use retention, preventing a blob from disappearing duri
 5. copy/link/archive required blobs without following unsafe paths;
 6. create manifest;
 7. verify dump readability and every file hash;
-8. optionally encrypt with an operator-supplied mechanism;
-9. atomically publish completed backup;
-10. apply retention only after a verified new backup exists;
-11. record audit and administrator-visible result.
+8. authenticate the manifest hash with HMAC-SHA-256;
+9. optionally encrypt with an operator-supplied mechanism;
+10. atomically publish completed backup;
+11. apply retention only after a verified new backup exists;
+12. record audit and administrator-visible result.
 
 A failed backup never replaces the last known-good backup.
 
+New backup directories and files use private Unix permissions (0700 and 0600).
+Permission failures stop creation before publication. On Windows, provision and
+verify restrictive NTFS ACLs for backups, keys and restore targets before use;
+Unix mode checks do not enforce Windows ACLs. HMAC authenticates backups but
+does not encrypt their contents, so archive/share access must remain restricted.
+
+## Backup authenticity key and recovery
+
+The `.integrity.json` receipt beside a backup is not trusted by itself. The application
+authenticates its backup ID, exact manifest hash, completion time and verification result
+with HMAC-SHA-256. The random 32-byte key is stored at
+`KnowledgeHub:Paths:Keys/backup-integrity.key`; in the supplied Compose deployment this
+is `/var/lib/knowledge-hub/keys/backup-integrity.key`. This location is outside
+`KnowledgeHub:Paths:Backups`, so someone who can rewrite a backup and its receipt cannot
+create a valid signature. The application creates the key when integrity evidence is
+first written and restricts it to the service account on Unix hosts.
+
+Treat this key as essential recovery material. Copy the exact key file into an encrypted,
+access-controlled escrow location separate from the backup storage, and include it in
+the operator's disaster-recovery procedure. To restore on a replacement host, recover
+the key to the configured Keys directory before verifying or restoring backups. Never
+put the key in a backup directory or in the same backup archive as the signed receipts.
+Protect access to the escrow copy as carefully as access to the backups.
+
+If the key is missing or unreadable, backup integrity is reported as unknown and restore
+is refused. Do not replace a lost key with a newly generated one and expect existing
+backups to validate: a new key authenticates only backups created after it. Recover the
+original key from escrow first. If no copy exists, the existing backups cannot be
+cryptographically authenticated by the application and must not be treated as verified.
+Backups made before authenticated receipts were introduced also remain unverified; create
+a new backup with this version before relying on application-verified restore.
+
 ## Retention defaults
 
 Configurable starting policy:
@@ -65,13 +99,26 @@ source.
 
 Restore is deliberately separate from normal application startup.
 
-1. Select backup.
-2. Verify manifest, hashes, version compatibility and encryption.
-3. Create clean PostgreSQL database and empty target directories.
-4. Restore database.
+1. Select backup. Copy read-only media to a writable recovery directory first,
+   preserving the manifest, signed receipts and file bytes. Recover the exact
+   `backup-integrity.key` from protected external escrow into the configured Keys
+   directory; never take it from the backup archive. The application acquires an
+   exclusive storage lease there to prevent retention from removing a restore source,
+   and writes its restore receipt there. Keep the original media unchanged.
+2. Use the backup's recorded application image to create a clean PostgreSQL
+   database at exactly the manifest's schema version. Create an existing, empty
+   target attachments directory on writable storage with room for the full payload.
+3. Copy the manifest and every listed database and attachment file into a private
+   staging directory inside the target attachments volume. Verify the HMAC against
+   the staged manifest bytes and verify each staged file's size and SHA-256 before
+   opening or changing the target database. Restore reads only from this verified
+   snapshot, so source-volume changes after verification cannot change restored bytes.
+4. Restore database. The restore service refuses a target whose latest applied
+   migration differs from the authenticated manifest, before inserting any rows.
 5. Restore attachment blobs by hash.
-6. Start the recorded/current compatible application in maintenance mode.
-7. Apply supported migrations.
+6. Verify the restored data with the recorded application version.
+7. Upgrade through supported migrations using the intended newer image, then
+   repeat integrity checks before promotion.
 8. Run integrity checks:
    - FK/schema;
    - user/membership;
@@ -87,6 +134,21 @@ Restore is deliberately separate from normal application startup.
 
 Never test restore by overwriting the only production instance.
 
+Pass target PostgreSQL credentials through `--target-connection-file=<path>`, which
+reads the connection string from a protected file. On Unix, the file must be readable
+only by the restore account (mode `0600` or `0400`); the restore command rejects files
+with group or other access. On Windows, apply an ACL that grants access only to the
+restore operator/service account. Do not put the connection string in command-line
+arguments. Restore errors deliberately omit provider exception details because they
+can contain connection data.
+
+The real-backup drill also requires the exact integrity key recovered from protected
+external escrow. Pass only its file path through
+`KNOWLEDGE_HUB_BACKUP_INTEGRITY_KEY_FILE`; the script copies the key with mode `0600`
+into its temporary working directory, uses that directory as the restore Keys path,
+and removes the temporary copy during cleanup. The key contents are never placed in an
+argument or printed by the script.
+
 ## Backup encryption
 
 The application may integrate an operator-supplied encryption command or library after
@@ -125,11 +187,12 @@ isolated restore proves it is necessary and the operator has assessed newer writ
 
 - create backup with representative data;
 - verify manifest;
+- reject a forged manifest and matching unsigned/invalid integrity receipt;
 - corrupt dump and blob tests;
 - clean-target restore;
 - compare logical counts and hashes;
 - restore older supported schema then migrate;
-- missing key/encrypted backup failure;
+- missing integrity key or encryption key failure;
 - interrupted backup cleanup;
 - retention policy;
 - no secret inclusion;