# Provider circuit-open alert.
#
# Required by docs/09_OPERATIONS/OBSERVABILITY.md's "Alerts" list
# ("... security denial spike, disk pressure en provider circuit open").
#
# Fires when a model provider's circuit breaker
# (src/ragcore/infrastructure/models/resilience.py::CircuitBreaker) opens:
# consecutive provider failures crossed the configured threshold, and
# further calls are being short-circuited for the cooldown window instead
# of retried against a provider that is currently failing.
#
# This file is intentionally tool-agnostic (no Prometheus/Loki/Alertmanager
# assumed) -- adapt `signal` to whichever log or metrics pipeline a given
# deployment already runs. tests/integration/resilience/
# test_alert_matches_circuit_event.py proves the `signal.event` value below
# stays in sync with the literal event name the code actually logs.

alert: ProviderCircuitOpen
severity: warning
description: >
  A model provider (embedding, reranker or generation) has failed enough
  consecutive calls to trip its circuit breaker open. Calls to that
  provider are being rejected immediately for the breaker's cooldown
  window rather than retried against a provider that is currently down.

signal:
  type: log_event
  # Structured JSON log line (see OBSERVABILITY.md's "Logs" contract:
  # trace ID, safe resource IDs, stable event name -- never prompt,
  # document or query content).
  event: provider_circuit_open
  fields:
    provider: string    # the CircuitBreaker's own name, e.g. "ollama-generation"
    from_state: string  # always "closed" or "half_open" for this alert
    to_state: string     # always "open" for this alert

for: 0s  # fire immediately; the event only occurs on a state transition, never continuously

runbook:
  - >-
    Check the affected provider's own health/availability directly (process
    up, model loaded, network reachable) using the `provider` field to
    identify which one tripped.
  - >-
    Confirm unaffected surfaces -- retrieval (`/v1/search`, `/v1/context`)
    does not depend on any generation provider and continues to serve
    normally regardless of this alert.
  - >-
    No manual reset is required -- the circuit automatically half-opens
    after its cooldown window (see the paired `provider_circuit_half_open`
    event) and closes again on the next successful call
    (`provider_circuit_closed`).
  - >-
    If `provider_circuit_open` keeps recurring across several cooldown
    cycles for the same provider, treat it as a genuine outage and escalate
    per the provider's own on-call runbook.