Initial curated RAGcore source from c16667f0c9397e2bfc8dd0255b114fba23c8d05e

ops/alerts/provider_circuit_open.yaml

@@ -0,0 +1,56 @@
+# Provider circuit-open alert.
+#
+# Required by docs/09_OPERATIONS/OBSERVABILITY.md's "Alerts" list
+# ("... security denial spike, disk pressure en provider circuit open").
+#
+# Fires when a model provider's circuit breaker
+# (src/ragcore/infrastructure/models/resilience.py::CircuitBreaker) opens:
+# consecutive provider failures crossed the configured threshold, and
+# further calls are being short-circuited for the cooldown window instead
+# of retried against a provider that is currently failing.
+#
+# This file is intentionally tool-agnostic (no Prometheus/Loki/Alertmanager
+# assumed) -- adapt `signal` to whichever log or metrics pipeline a given
+# deployment already runs. tests/integration/resilience/
+# test_alert_matches_circuit_event.py proves the `signal.event` value below
+# stays in sync with the literal event name the code actually logs.
+
+alert: ProviderCircuitOpen
+severity: warning
+description: >
+  A model provider (embedding, reranker or generation) has failed enough
+  consecutive calls to trip its circuit breaker open. Calls to that
+  provider are being rejected immediately for the breaker's cooldown
+  window rather than retried against a provider that is currently down.
+
+signal:
+  type: log_event
+  # Structured JSON log line (see OBSERVABILITY.md's "Logs" contract:
+  # trace ID, safe resource IDs, stable event name -- never prompt,
+  # document or query content).
+  event: provider_circuit_open
+  fields:
+    provider: string    # the CircuitBreaker's own name, e.g. "ollama-generation"
+    from_state: string  # always "closed" or "half_open" for this alert
+    to_state: string     # always "open" for this alert
+
+for: 0s  # fire immediately; the event only occurs on a state transition, never continuously
+
+runbook:
+  - >-
+    Check the affected provider's own health/availability directly (process
+    up, model loaded, network reachable) using the `provider` field to
+    identify which one tripped.
+  - >-
+    Confirm unaffected surfaces -- retrieval (`/v1/search`, `/v1/context`)
+    does not depend on any generation provider and continues to serve
+    normally regardless of this alert.
+  - >-
+    No manual reset is required -- the circuit automatically half-opens
+    after its cooldown window (see the paired `provider_circuit_half_open`
+    event) and closes again on the next successful call
+    (`provider_circuit_closed`).
+  - >-
+    If `provider_circuit_open` keeps recurring across several cooldown
+    cycles for the same provider, treat it as a genuine outage and escalate
+    per the provider's own on-call runbook.

ops/alerts/queue_age_slo.yaml

@@ -0,0 +1,45 @@
+# Job queue-age SLO alert.
+#
+# Required by docs/09_OPERATIONS/OBSERVABILITY.md's "Alerts" list ("...
+# oude leases, queue-age SLO, repeated terminal jobs, ...").
+#
+# Fires when a job waits too long between being enqueued and being claimed
+# by a worker, signalling the worker pool is falling behind or stuck.
+#
+# Tool-agnostic on purpose (no Prometheus/Grafana/Alertmanager assumed) --
+# adapt `signal` to whichever metrics pipeline a given deployment already
+# runs.
+
+alert: JobQueueAgeSloBreached
+severity: warning
+description: >-
+  A job's age at the moment a worker claimed it exceeded the configured
+  SLO threshold, meaning jobs are waiting too long in the queue before
+  being picked up.
+
+signal:
+  type: metric
+  metric: ragcore_job_age_seconds
+  # Recorded by src/ragcore/platform/telemetry/metrics.py::
+  # TelemetryMetrics.record_job_claimed -- see
+  # ops/dashboards/operations_overview.yaml's "Job age at claim" panel.
+  aggregation: p95
+  comparison: greater_than
+  threshold_seconds: 300
+
+for: 5m
+
+runbook:
+  - >-
+    Check ragcore_job_queue_depth alongside this alert -- a simultaneously
+    rising queue depth points at worker capacity being outpaced by job
+    volume; a flat or falling depth with high age instead suggests a
+    specific stuck job or a worker that stopped claiming work.
+  - >-
+    Confirm workers are actually running and their lease-recovery loop is
+    healthy (see the paired "oude leases" / stale-lease alert also
+    required by OBSERVABILITY.md).
+  - >-
+    If job volume genuinely exceeds capacity, scale the worker pool; if a
+    specific job type is slow, investigate that job's own processing time
+    rather than the queue mechanism itself.