57 lines
2.5 KiB
YAML
57 lines
2.5 KiB
YAML
# Provider circuit-open alert.
|
|
#
|
|
# Required by docs/09_OPERATIONS/OBSERVABILITY.md's "Alerts" list
|
|
# ("... security denial spike, disk pressure en provider circuit open").
|
|
#
|
|
# Fires when a model provider's circuit breaker
|
|
# (src/ragcore/infrastructure/models/resilience.py::CircuitBreaker) opens:
|
|
# consecutive provider failures crossed the configured threshold, and
|
|
# further calls are being short-circuited for the cooldown window instead
|
|
# of retried against a provider that is currently failing.
|
|
#
|
|
# This file is intentionally tool-agnostic (no Prometheus/Loki/Alertmanager
|
|
# assumed) -- adapt `signal` to whichever log or metrics pipeline a given
|
|
# deployment already runs. tests/integration/resilience/
|
|
# test_alert_matches_circuit_event.py proves the `signal.event` value below
|
|
# stays in sync with the literal event name the code actually logs.
|
|
|
|
alert: ProviderCircuitOpen
|
|
severity: warning
|
|
description: >
|
|
A model provider (embedding, reranker or generation) has failed enough
|
|
consecutive calls to trip its circuit breaker open. Calls to that
|
|
provider are being rejected immediately for the breaker's cooldown
|
|
window rather than retried against a provider that is currently down.
|
|
|
|
signal:
|
|
type: log_event
|
|
# Structured JSON log line (see OBSERVABILITY.md's "Logs" contract:
|
|
# trace ID, safe resource IDs, stable event name -- never prompt,
|
|
# document or query content).
|
|
event: provider_circuit_open
|
|
fields:
|
|
provider: string # the CircuitBreaker's own name, e.g. "ollama-generation"
|
|
from_state: string # always "closed" or "half_open" for this alert
|
|
to_state: string # always "open" for this alert
|
|
|
|
for: 0s # fire immediately; the event only occurs on a state transition, never continuously
|
|
|
|
runbook:
|
|
- >-
|
|
Check the affected provider's own health/availability directly (process
|
|
up, model loaded, network reachable) using the `provider` field to
|
|
identify which one tripped.
|
|
- >-
|
|
Confirm unaffected surfaces -- retrieval (`/v1/search`, `/v1/context`)
|
|
does not depend on any generation provider and continues to serve
|
|
normally regardless of this alert.
|
|
- >-
|
|
No manual reset is required -- the circuit automatically half-opens
|
|
after its cooldown window (see the paired `provider_circuit_half_open`
|
|
event) and closes again on the next successful call
|
|
(`provider_circuit_closed`).
|
|
- >-
|
|
If `provider_circuit_open` keeps recurring across several cooldown
|
|
cycles for the same provider, treat it as a genuine outage and escalate
|
|
per the provider's own on-call runbook.
|