Health checking¶

EDRI has a built-in health-checking subsystem that monitors every Manager process, aggregates their status, and exposes it on an HTTP endpoint. It is cross-cutting: the Router drives and aggregates the checks, while each Manager reports its own status and metrics on every round-trip. The Switch, which lives outside the router/manager/event topology, exposes its own analogous read-only status endpoint separately.

How it works¶

A periodic task is scheduled via the Scheduler Manager (using the scheduler’s Set event).
The scheduled task emits a router HealthCheck event at a fixed interval. Every Manager is subscribed to it, so it doubles as a heartbeat to each Manager's pipe.
Managers respond with status information, which the Router's HealthChecker aggregates.
If a Manager stops reporting for multiple intervals, the HealthChecker escalates its recorded status to WARNING, then ERROR. This escalation does not trigger a restart on its own — it only updates the status surfaced to operators.
A restart happens instead when delivering an event to a Manager fails with a broken pipe (a dead process). Because the periodic HealthCheck is routed to every Manager, it is normally what detects the dead pipe. The Router then drops the subscription and restarts the component from its definition, with from_time set to the moment of failure so missed events are replayed (see Delivery guarantees). A restart only happens if the component definition is available to the HealthChecker.

Handler exceptions are captured, not fatal¶

When a @request / @response handler raises, the Manager does not crash. The resolver loop catches the exception, records it (the event name, the event payload, the exception, a formatted traceback, and the timestamp when it was caught), and keeps processing other events. The captured exceptions are attached to that Manager's next HealthCheck reply and flip its reported status to WARNING, then the buffer is cleared.

This is how handler errors become observable without scraping logs: they surface in the status endpoint below, per component, until the next clean health cycle.

Status endpoints¶

EDRI exposes three HTTP endpoints for health monitoring, each serving a different purpose:

Endpoint	Purpose	Behaviour
`GET /edri/live`	Liveness — confirms the Router event loop is running. Does not depend on component readiness.	Always returns HTTP 200 when the Router event loop is running. Use for Kubernetes `livenessProbe` and `startupProbe`.
`GET /edri/ready`	Readiness — all components are ready to accept traffic.	Returns HTTP 200 when all components report ready. Returns HTTP 503 if any component is warming up, draining, or has an unreachable dependency.
`GET /edri/health-check-status`	Diagnostic — detailed per-component health, vitals, exceptions.	Returns HTTP 200 (or 503 when any component is not ready) with full diagnostic data.

Both /edri/ready and /edri/health-check-status expose the same per-component records described below. No application code is needed for any of these endpoints.

Diagnostic endpoint¶

The aggregated health of all monitored components is exposed as a built-in API resource:

GET /edri/health-check-status (event EdriHealth, resource health-check-status)
Available as JSON or, via Content Negotiation, rendered HTML (template health_check_status.j2).

Each record describes one monitored component:

Field	Meaning
`name`	Component name.
`timestamp`	When its last health reply was recorded.
`status`	`OK` / `WARNING` / `DEGRADED` / `ERROR` (serialized as the `Status` enum value). `WARNING`/`ERROR` are escalated by the Router from silence; `DEGRADED` is only ever self-reported by a component's `health()` probe.
`ready`	Whether the component reports itself ready to accept traffic (a healthy component can still be `ready=false` while warming up or draining).
`reason`	Human-readable explanation for a non-OK status or non-ready state; `null` when there is nothing to say.
`exceptions`	Handler exceptions caught since the previous reply. Each entry includes the event name, the exception message, a formatted traceback, and the timestamp when it was captured (`captured_at`, ISO-8601).
`health_rtt_seconds`	Round-trip latency of the last health check, measured by the Router (dispatch → component → back). Rises before a struggling component goes silent, so it is a degradation signal the component cannot report about itself.
`vitals`	Operational metrics that ride along on the periodic health round-trip (below). `null` until the component has answered at least once.

Vitals¶

Some vitals are point-in-time snapshots; some are deltas accumulated over the interval since the previous check (divide by EDRI_HEALTH_CHECK_TIMEOUT for a per-second rate). All process-level metrics are best-effort and report null when unreadable on the platform.

Vital	Kind	Meaning
`workers_running`	snapshot	Workers currently alive (busy slots).
`workers_pending`	snapshot	Work queued because all slots are full — the component's backpressure signal.
`workers_limit`	snapshot	The `max_workers` ceiling; `-1` means unbounded.
`workers_oldest_age_seconds`	snapshot	Age of the longest-running worker (`null` when idle) — how a deadlock surfaces while the main loop still answers.
`processed`	delta	Events dispatched to handlers since the previous check (÷ interval = throughput).
`failed`	delta	Of those, how many left a `FAILED` response (÷ interval = error rate).
`cpu_seconds`	delta	CPU time (user + system) consumed since the previous check (÷ interval = CPU utilization).
`rss_bytes` / `rss_peak_bytes`	snapshot	Resident memory now / high-water mark, in bytes.
`threads`	snapshot	Live threads — a growing count flags a thread leak.
`open_fds`	snapshot	Open file descriptors — an fd-leak detector.
`pid`	snapshot	Component process PID, for correlation with external tools.
`uptime_seconds`	snapshot	Seconds since the process entered its run loop — a small value on a long-lived component points at a crash-loop.
`custom`	—	Application-defined metrics added via `collect_custom_vitals()` (below); empty `{}` by default.

Reporting custom health (`health()`)¶

A Manager assesses its own health via an overridable health() probe that returns a HealthReport(status, ready, reason). The default derives status purely from caught exceptions (any → WARNING) and always reports ready=True. Override it to add domain knowledge — typically checking the dependencies the component needs:

from edri.dataclass.health_checker import HealthReport, Status

def health(self) -> HealthReport:
    if not self._database.is_connected():
        return HealthReport(Status.DEGRADED, ready=False, reason="database unreachable")
    return super().health()

The probe runs inline on the component's main loop on every health round-trip, so keep it cheap and non-blocking. Caught exceptions are reported separately regardless, so an override should escalate status, not suppress exceptions.

Adding custom vitals (`collect_custom_vitals()`)¶

Beyond the built-in vitals, a Manager can attach its own metrics by overriding collect_custom_vitals(). They ride along on the same periodic health round-trip and appear under each record's vitals.custom:

from typing import Any

def collect_custom_vitals(self) -> dict[str, Any]:
    return {
        "queue_depth": self._inbox.qsize(),
        "cache_entries": len(self.cache),
        "last_sync_age_seconds": time() - self._last_sync,
    }

The keys are reported verbatim (not camel-cased) under vitals.custom in the JSON response and each renders as its own line in the HTML page. Like health(), the probe runs inline on the main loop on every check, so keep it cheap and non-blocking. Return JSON-serializable values so they survive both the pipe round-trip and the endpoint's JSON encoding (datetime, Enum, UUID and bytes are encoded automatically). If the override raises, it is caught and logged and the custom block degrades to empty — it never wipes out the standard vitals.

Configuration¶

Environment Variable	Default	Description
`EDRI_HEALTH_CHECK_TIMEOUT`	`10`	Interval (seconds) between health checks.
`EDRI_HEALTH_CHECK_FAILURE_LIMIT`	`3`	How many missed intervals are tolerated before escalating status.
`EDRI_HEALTH_CHECK_LOG`	`true`	When `false`, the periodic `HealthCheck` event is not traced in the debug log. Missed-check warnings/errors and the endpoints below are unaffected.

See Environment Variables for the full list.