46 lines
1.6 KiB
YAML
46 lines
1.6 KiB
YAML
# Job queue-age SLO alert.
|
|
#
|
|
# Required by docs/09_OPERATIONS/OBSERVABILITY.md's "Alerts" list ("...
|
|
# oude leases, queue-age SLO, repeated terminal jobs, ...").
|
|
#
|
|
# Fires when a job waits too long between being enqueued and being claimed
|
|
# by a worker, signalling the worker pool is falling behind or stuck.
|
|
#
|
|
# Tool-agnostic on purpose (no Prometheus/Grafana/Alertmanager assumed) --
|
|
# adapt `signal` to whichever metrics pipeline a given deployment already
|
|
# runs.
|
|
|
|
alert: JobQueueAgeSloBreached
|
|
severity: warning
|
|
description: >-
|
|
A job's age at the moment a worker claimed it exceeded the configured
|
|
SLO threshold, meaning jobs are waiting too long in the queue before
|
|
being picked up.
|
|
|
|
signal:
|
|
type: metric
|
|
metric: ragcore_job_age_seconds
|
|
# Recorded by src/ragcore/platform/telemetry/metrics.py::
|
|
# TelemetryMetrics.record_job_claimed -- see
|
|
# ops/dashboards/operations_overview.yaml's "Job age at claim" panel.
|
|
aggregation: p95
|
|
comparison: greater_than
|
|
threshold_seconds: 300
|
|
|
|
for: 5m
|
|
|
|
runbook:
|
|
- >-
|
|
Check ragcore_job_queue_depth alongside this alert -- a simultaneously
|
|
rising queue depth points at worker capacity being outpaced by job
|
|
volume; a flat or falling depth with high age instead suggests a
|
|
specific stuck job or a worker that stopped claiming work.
|
|
- >-
|
|
Confirm workers are actually running and their lease-recovery loop is
|
|
healthy (see the paired "oude leases" / stale-lease alert also
|
|
required by OBSERVABILITY.md).
|
|
- >-
|
|
If job volume genuinely exceeds capacity, scale the worker pool; if a
|
|
specific job type is slow, investigate that job's own processing time
|
|
rather than the queue mechanism itself.
|