Publish curated Knowledge Hub source from 2ed833f290b439a7bc91e5fd19b042c269e3bee9

docs/background-jobs.md

@@ -0,0 +1,43 @@
+# Durable background jobs
+
+Course tagging and manual backup creation run as PostgreSQL-backed jobs. The admin
+workspace shows persisted progress and supports cancellation. SSE sends the same
+safe snapshot as the polling endpoint and closes on a terminal state; the browser
+falls back to polling after an interrupted stream.
+
+Only an enabled workspace administrator can create or execute a job. Mutation
+endpoints require cookie authentication and CSRF validation. The worker rechecks
+the actor's membership before work and before each course page. An idempotency key
+is scoped to the workspace and actor; reusing it with a different payload returns
+409. Retrying the same payload returns the existing job, including at capacity.
+The job row and its `job.queued` audit event commit together. Cancellation requests
+and their `job.cancel_requested` audit event also commit together; repeating a
+running-job cancellation returns the current request without adding another audit
+event.
+
+The database serializes admission per workspace. Defaults under `KnowledgeHub:Jobs`
+are `MaxPendingPerWorkspace=10`, `MaxPendingPerActor=3`, and
+`MaxCreatedPerDay=100` (a rolling 24-hour window). Capacity rejection returns 429
+before inserting a job or page snapshot. HTTP creation also has a shared budget
+of ten requests per effective remote IP per minute. Only configured trusted
+proxies can affect that address. These limits bound distinct-key floods across
+application restarts and replicas as well as ordinary bursts.
+
+Course tagging snapshots at most `MaxCoursePages=100000` page IDs in PostgreSQL.
+The worker updates one current page at a time with its ETag, preserves content,
+skips existing tags, and records safe per-page failures. A lease is renewed while
+work is active; expired leases can be reclaimed with bounded retries. Default
+worker concurrency is two. Cancellation stops future items and preserves the
+already completed outcomes.
+
+Terminal jobs retain their counters, bounded errors, result, actor and audit trail.
+Their page-ID recovery snapshots are removed in the same transaction as the final
+state. This limits large item-table growth without discarding the job history.
+The daily admission budget bounds growth of that compact history. Source data is
+never removed by this maintenance.
+
+The job `Version` is a monotonic optimistic-concurrency token for persisted worker
+updates and SSE event IDs. If a cancellation update wins a race with a tracked worker
+save, the worker retries against the current version while keeping the cancellation
+request. SSE always emits a terminal event and closes when it observes a terminal
+job, including when that snapshot has the same version as the last progress event.