Initial curated RAGcore source from c16667f0c9397e2bfc8dd0255b114fba23c8d05e

ops/alerts/provider_circuit_open.yaml

@@ -0,0 +1,56 @@
+# Provider circuit-open alert.
+#
+# Required by docs/09_OPERATIONS/OBSERVABILITY.md's "Alerts" list
+# ("... security denial spike, disk pressure en provider circuit open").
+#
+# Fires when a model provider's circuit breaker
+# (src/ragcore/infrastructure/models/resilience.py::CircuitBreaker) opens:
+# consecutive provider failures crossed the configured threshold, and
+# further calls are being short-circuited for the cooldown window instead
+# of retried against a provider that is currently failing.
+#
+# This file is intentionally tool-agnostic (no Prometheus/Loki/Alertmanager
+# assumed) -- adapt `signal` to whichever log or metrics pipeline a given
+# deployment already runs. tests/integration/resilience/
+# test_alert_matches_circuit_event.py proves the `signal.event` value below
+# stays in sync with the literal event name the code actually logs.
+
+alert: ProviderCircuitOpen
+severity: warning
+description: >
+  A model provider (embedding, reranker or generation) has failed enough
+  consecutive calls to trip its circuit breaker open. Calls to that
+  provider are being rejected immediately for the breaker's cooldown
+  window rather than retried against a provider that is currently down.
+
+signal:
+  type: log_event
+  # Structured JSON log line (see OBSERVABILITY.md's "Logs" contract:
+  # trace ID, safe resource IDs, stable event name -- never prompt,
+  # document or query content).
+  event: provider_circuit_open
+  fields:
+    provider: string    # the CircuitBreaker's own name, e.g. "ollama-generation"
+    from_state: string  # always "closed" or "half_open" for this alert
+    to_state: string     # always "open" for this alert
+
+for: 0s  # fire immediately; the event only occurs on a state transition, never continuously
+
+runbook:
+  - >-
+    Check the affected provider's own health/availability directly (process
+    up, model loaded, network reachable) using the `provider` field to
+    identify which one tripped.
+  - >-
+    Confirm unaffected surfaces -- retrieval (`/v1/search`, `/v1/context`)
+    does not depend on any generation provider and continues to serve
+    normally regardless of this alert.
+  - >-
+    No manual reset is required -- the circuit automatically half-opens
+    after its cooldown window (see the paired `provider_circuit_half_open`
+    event) and closes again on the next successful call
+    (`provider_circuit_closed`).
+  - >-
+    If `provider_circuit_open` keeps recurring across several cooldown
+    cycles for the same provider, treat it as a genuine outage and escalate
+    per the provider's own on-call runbook.