# Health checking

EDRI has a built-in health-checking subsystem that monitors every Manager
process, aggregates their status, and exposes it on an HTTP endpoint. It is
cross-cutting: the {doc}`/router` drives and aggregates the checks, while
each {doc}`/manager` reports its own status and metrics on every round-trip.
The {doc}`/switch`, which lives outside the router/manager/event topology,
exposes its own analogous read-only status endpoint separately.

---

## How it works

* A periodic task is scheduled via the **Scheduler Manager** (using the scheduler’s `Set` event).
* The scheduled task emits a router `HealthCheck` event at a fixed interval. Every Manager is subscribed to it, so it doubles as a heartbeat to each Manager's pipe.
* Managers respond with status information, which the Router's **HealthChecker** aggregates.
* If a Manager stops reporting for multiple intervals, the HealthChecker escalates its recorded status to `WARNING`, then `ERROR`. This escalation **does not** trigger a restart on its own — it only updates the status surfaced to operators.
* A restart happens instead when delivering an event to a Manager fails with a broken pipe (a dead process). Because the periodic `HealthCheck` is routed to every Manager, it is normally what detects the dead pipe. The Router then drops the subscription and restarts the component from its definition, with `from_time` set to the moment of failure so missed events are replayed (see {ref}`delivery-guarantees`). A restart only happens if the component definition is available to the HealthChecker.

---

## Handler exceptions are captured, not fatal

When a `@request` / `@response` handler raises, the Manager does **not** crash. The resolver loop catches the exception, records it (the event name, the event payload, the exception, a formatted traceback, and the timestamp when it was caught), and keeps processing other events. The captured exceptions are attached to that Manager's **next** `HealthCheck` reply and flip its reported status to `WARNING`, then the buffer is cleared.

This is how handler errors become observable without scraping logs: they surface in the status endpoint below, per component, until the next clean health cycle.

---

(health-status-endpoint)=
## Status endpoints

EDRI exposes three HTTP endpoints for health monitoring, each serving a different purpose:

| Endpoint | Purpose | Behaviour |
|----------|---------|-----------|
| `GET /edri/live` | **Liveness** — confirms the Router event loop is running. Does not depend on component readiness. | Always returns **HTTP 200** when the Router event loop is running. Use for Kubernetes ``livenessProbe`` and ``startupProbe``. |
| `GET /edri/ready` | **Readiness** — all components are ready to accept traffic. | Returns **HTTP 200** when all components report ready. Returns **HTTP 503** if any component is warming up, draining, or has an unreachable dependency. |
| `GET /edri/health-check-status` | **Diagnostic** — detailed per-component health, vitals, exceptions. | Returns **HTTP 200** (or 503 when any component is not ready) with full diagnostic data. |

Both `/edri/ready` and `/edri/health-check-status` expose the same per-component records described below. No application code is needed for any of these endpoints.

### Diagnostic endpoint

The aggregated health of all monitored components is exposed as a built-in API resource:

* `GET /edri/health-check-status` (event `EdriHealth`, resource `health-check-status`)
* Available as JSON or, via {doc}`/api_content_negotiation`, rendered HTML (template `health_check_status.j2`).

Each record describes one monitored component:

| Field | Meaning |
|---|---|
| `name` | Component name. |
| `timestamp` | When its last health reply was recorded. |
| `status` | `OK` / `WARNING` / `DEGRADED` / `ERROR` (serialized as the `Status` enum value). `WARNING`/`ERROR` are escalated by the Router from silence; `DEGRADED` is only ever self-reported by a component's `health()` probe. |
| `ready` | Whether the component reports itself ready to accept traffic (a healthy component can still be `ready=false` while warming up or draining). |
| `reason` | Human-readable explanation for a non-OK status or non-ready state; `null` when there is nothing to say. |
| `exceptions` | Handler exceptions caught since the previous reply. Each entry includes the event name, the exception message, a formatted traceback, and the timestamp when it was captured (``captured_at``, ISO-8601). |
| `health_rtt_seconds` | Round-trip latency of the last health check, measured by the Router (dispatch → component → back). Rises before a struggling component goes silent, so it is a degradation signal the component cannot report about itself. |
| `vitals` | Operational metrics that ride along on the periodic health round-trip (below). `null` until the component has answered at least once. |

### Vitals

Some vitals are point-in-time snapshots; some are deltas accumulated over the interval since the previous check (divide by `EDRI_HEALTH_CHECK_TIMEOUT` for a per-second rate). All process-level metrics are best-effort and report `null` when unreadable on the platform.

| Vital | Kind | Meaning |
|---|---|---|
| `workers_running` | snapshot | Workers currently alive (busy slots). |
| `workers_pending` | snapshot | Work queued because all slots are full — the component's backpressure signal. |
| `workers_limit` | snapshot | The `max_workers` ceiling; `-1` means unbounded. |
| `workers_oldest_age_seconds` | snapshot | Age of the longest-running worker (`null` when idle) — how a deadlock surfaces while the main loop still answers. |
| `processed` | delta | Events dispatched to handlers since the previous check (÷ interval = throughput). |
| `failed` | delta | Of those, how many left a `FAILED` response (÷ interval = error rate). |
| `cpu_seconds` | delta | CPU time (user + system) consumed since the previous check (÷ interval = CPU utilization). |
| `rss_bytes` / `rss_peak_bytes` | snapshot | Resident memory now / high-water mark, in bytes. |
| `threads` | snapshot | Live threads — a growing count flags a thread leak. |
| `open_fds` | snapshot | Open file descriptors — an fd-leak detector. |
| `pid` | snapshot | Component process PID, for correlation with external tools. |
| `uptime_seconds` | snapshot | Seconds since the process entered its run loop — a small value on a long-lived component points at a crash-loop. |
| `custom` | — | Application-defined metrics added via `collect_custom_vitals()` (below); empty `{}` by default. |

---

## Reporting custom health (`health()`)

A Manager assesses its own health via an overridable `health()` probe that returns a `HealthReport(status, ready, reason)`. The default derives `status` purely from caught exceptions (any → `WARNING`) and always reports `ready=True`. Override it to add domain knowledge — typically checking the dependencies the component needs:

```python
from edri.dataclass.health_checker import HealthReport, Status

def health(self) -> HealthReport:
    if not self._database.is_connected():
        return HealthReport(Status.DEGRADED, ready=False, reason="database unreachable")
    return super().health()
```

The probe runs inline on the component's main loop on every health round-trip, so keep it cheap and non-blocking. Caught exceptions are reported separately regardless, so an override should escalate `status`, not suppress exceptions.

## Adding custom vitals (`collect_custom_vitals()`)

Beyond the built-in vitals, a Manager can attach its own metrics by overriding `collect_custom_vitals()`. They ride along on the same periodic health round-trip and appear under each record's `vitals.custom`:

```python
from typing import Any

def collect_custom_vitals(self) -> dict[str, Any]:
    return {
        "queue_depth": self._inbox.qsize(),
        "cache_entries": len(self.cache),
        "last_sync_age_seconds": time() - self._last_sync,
    }
```

The keys are reported verbatim (not camel-cased) under `vitals.custom` in the JSON response and each renders as its own line in the HTML page. Like `health()`, the probe runs inline on the main loop on every check, so keep it cheap and non-blocking. Return JSON-serializable values so they survive both the pipe round-trip and the endpoint's JSON encoding (`datetime`, `Enum`, `UUID` and `bytes` are encoded automatically). If the override raises, it is caught and logged and the custom block degrades to empty — it never wipes out the standard vitals.

---

## Configuration

| Environment Variable | Default | Description |
|---|---|---|
| `EDRI_HEALTH_CHECK_TIMEOUT` | `10` | Interval (seconds) between health checks. |
| `EDRI_HEALTH_CHECK_FAILURE_LIMIT` | `3` | How many missed intervals are tolerated before escalating status. |
| `EDRI_HEALTH_CHECK_LOG` | `true` | When `false`, the periodic `HealthCheck` event is not traced in the debug log. Missed-check warnings/errors and the endpoints below are unaffected. |

See {doc}`/env` for the full list.

---

## Related

* {doc}`/router` — how the HealthChecker schedules checks and restarts dead components
* {doc}`/manager` — the component side: `health()` and `collect_custom_vitals()` hooks
* {doc}`/switch` — the Switch's analogous read-only status endpoint