# Durable background jobs

Course tagging and manual backup creation run as PostgreSQL-backed jobs. The admin
workspace shows persisted progress and supports cancellation. SSE sends the same
safe snapshot as the polling endpoint and closes on a terminal state; the browser
falls back to polling after an interrupted stream.

Only an enabled workspace administrator can create or execute a job. Mutation
endpoints require cookie authentication and CSRF validation. The worker rechecks
the actor's membership before work and before each course page. An idempotency key
is scoped to the workspace and actor; reusing it with a different payload returns
409. Retrying the same payload returns the existing job, including at capacity.
The job row and its `job.queued` audit event commit together. Cancellation requests
and their `job.cancel_requested` audit event also commit together; repeating a
running-job cancellation returns the current request without adding another audit
event.

The database serializes admission per workspace. Defaults under `KnowledgeHub:Jobs`
are `MaxPendingPerWorkspace=10`, `MaxPendingPerActor=3`, and
`MaxCreatedPerDay=100` (a rolling 24-hour window). Capacity rejection returns 429
before inserting a job or page snapshot. HTTP creation also has a shared budget
of ten requests per effective remote IP per minute. Only configured trusted
proxies can affect that address. These limits bound distinct-key floods across
application restarts and replicas as well as ordinary bursts.

Course tagging snapshots at most `MaxCoursePages=100000` page IDs in PostgreSQL.
The worker updates one current page at a time with its ETag, preserves content,
skips existing tags, and records safe per-page failures. A lease is renewed while
work is active; expired leases can be reclaimed with bounded retries. Default
worker concurrency is two. Cancellation stops future items and preserves the
already completed outcomes.

Terminal jobs retain their counters, bounded errors, result, actor and audit trail.
Their page-ID recovery snapshots are removed in the same transaction as the final
state. This limits large item-table growth without discarding the job history.
The daily admission budget bounds growth of that compact history. Source data is
never removed by this maintenance.

The job `Version` is a monotonic optimistic-concurrency token for persisted worker
updates and SSE event IDs. If a cancellation update wins a race with a tracked worker
save, the worker retries against the current version while keeping the cancellation
request. SSE always emits a terminal event and closes when it observes a terminal
job, including when that snapshot has the same version as the last progress event.