# Performance & scaling

This page gives a feel for EDRI's throughput, latency and how it scales — plus a
reproducible harness so you can measure **your** workload on **your** hardware.

```{warning}
**Indicative numbers — read the shape, not the absolutes.** Every figure below
is from a single manual run; absolute throughput varies with hardware, load and
Python version. What transfers is the *scaling behavior* and *where the limits
are*. Re-run the [`benchmarks/`](#reproducing-these-measurements) suite to get
numbers for your own setup.

**Measurement environment:** 24-core Linux box · CPython 3.14 ·
`ENVIRONMENT=production` · clients over loopback · a single EDRI instance.
```

---

## What bounds EDRI's performance

EDRI trades raw single-endpoint throughput for isolation and explicit
concurrency (see {doc}`/architecture`):

* The **Router**, the **Broker** and each **Manager** are **single-threaded**
  processes. One request crosses several of them.
* Events move between processes as **pickled** objects through
  `multiprocessing` queues/pipes — every hop is a serialize + copy.
* Parallelism is **explicit**: a Manager handles one event at a time; CPU work
  must be pushed to **Workers** (threads or processes).

So the levers are: keep per-event work small, parallelize with Workers
(processes for CPU), split load across **multiple Managers**, and keep large
payloads off the pickled path (use {ref}`sharedmemorypipe`).

---

## For context: a single-process ASGI framework

```{warning}
**Not an apples-to-apples framework benchmark.** [FastAPI](https://fastapi.tiangolo.com/)
here is a single in-process ASGI app with no distributed routing — a bare
`async def` handler returning a dict. EDRI's `/noop` crosses **four
processes** (Listener → Broker → Router → Manager and back), pickling the
event at every hop, in exchange for the isolation and distributed routing
described above. The gap below is the visible cost of that architecture, not
a claim that EDRI is slow at what it's actually for. Reproduce with
`python benchmarks/compare_fastapi.py` (needs `pip install fastapi`,
otherwise not an EDRI dependency).
```

Same trivial `/noop` endpoint, same client-side load generator
(`http_load`), same hardware, two runs:

| Concurrency | EDRI (1 worker) | FastAPI (1 worker) | EDRI (4 workers) | FastAPI (4 workers) |
|---|---|---|---|---|
| 1 | ~1040–1115 | ~9400–10000 | ~1150–1165 | ~9480–9660 |
| 8 | ~3050–3390 | ~12200–13400 | ~4080–4130 | ~12700–13260 |
| 32 | ~3625–3790 | ~12150–12860 | ~4450–4475 | ~13040–13200 |

FastAPI is **~3–9×** faster here, and the gap is widest at concurrency 1
(~9×) and narrows toward higher concurrency (~3×): EDRI's fixed per-request
cost is the four-hop, four-process round trip itself, which dominates when
there's little concurrent work to pipeline behind it; FastAPI's is a single
in-process function call with no IPC at all. Neither number moves much with
`workers`, in either framework, at this concurrency — both are still far
from saturating this box's cores.

The honest takeaway: if the whole job is "answer this HTTP request with no
cross-process work," a single-process framework will always win, structurally.
EDRI's cost buys the things in the section above — process isolation per
Manager, a Worker model for CPU/I/O offload, distributed event routing, a
pluggable Broker/Router — that a single in-process handler doesn't need and
doesn't provide. Whether that trade is worth ~3–9× on a trivial endpoint
depends entirely on whether your application needs what it buys.

---

## HTTP request throughput & latency, WebSocket vs REST

A trivial round-trip (handler just sets one response field) over a persistent
**WebSocket** connection versus **REST** (`/noop`), same client machine, three
runs to check stability:

| Concurrency | WebSocket msgs/s | REST req/s |
|---|---|---|
| 1 | ~1115–1265 | ~1145–1200 |
| 8 | ~3350–3535 | ~2915–3100 |
| 32 | ~3220–3460 | ~3360–3485 |

REST tracks WebSocket closely: a WS client registers once on connect and then
just streams; REST checks a client's return pipe out of a pool, reuses it for
the request, and returns it once the response has drained — so neither path
pays a per-request registration cost. Take the exact ordering with a grain of
salt, though: the two load generators aren't quite apples-to-apples (`ws_load`
drives all connections from one `asyncio` event loop; `http_load` uses a real
OS thread per connection) — the headline is that neither path carries a
per-request registration cost, not that one is *architecturally* faster than
the other.

For chatty, low-latency interaction (live updates, RPC-style calls) WS still
avoids HTTP framing/header overhead per message, so it remains the better fit
even though the throughput gap has closed.

---

## gRPC vs WebSocket vs REST round-trip

Same trivial round-trip, same `Noop` event, now also over {doc}`/grpc` —
a unary call (one call per request, like REST) and a persistent bidi
`Connect` stream (like WebSocket), same client machine, three runs:

| Concurrency | WebSocket msgs/s | REST req/s | gRPC unary calls/s | gRPC bidi msgs/s |
|---|---|---|---|---|
| 1 | ~1178–1227 | ~1057–1138 | ~959–1021 | ~1106–1215 |
| 8 | ~2881–3009 | ~2526–2755 | ~2513–2523 | ~2504–2559 |
| 32 | ~3564–3613 | ~2843–2863 | ~2779–2840 | ~3856–3901 |

The shape matches the WS/REST comparison above, one level down: gRPC unary
tracks REST closely (both check a pipe out of a pool per call, no
per-request registration), and gRPC bidi tracks WebSocket closely (both
register once and then just stream) — with unary/bidi each landing a little
below their REST/WS counterpart. That gap is protobuf encode/decode plus one
extra `asyncio` hop per call inside `GRPCListener` (see {doc}`/grpc`'s Data
flow), on top of the same four-process round trip every transport pays.
`grpc.aio`'s own client-side overhead is part of that gap too, not isolated
here — this measures the whole call, not just the server side.

---

## Payload size

Same endpoint, response body of varying size, `ab -c 8`:

| Response size | Throughput | Mean latency |
|---|---|---|
| 1 KB | ~2635 req/s | 3.0 ms |
| 100 KB | ~1850 req/s | 4.3 ms |
| 1 MB | ~340 req/s | 23.6 ms |

Small and medium payloads are cheap; at 1 MB the per-event
serialize-and-copy across the pipeline dominates and throughput drops ~8×
from the 1 KB baseline. For
**large or streamed** responses, don't return a big pickled field — stream via
`File` + {ref}`sharedmemorypipe`, which moves the bytes
out-of-band with backpressure and keeps them off the broker.

> Large concurrent responses don't wedge the API: the broker's
> {ref}`non-blocking-outbound-send-sender-pool` keeps one slow
> client from blocking the others, so a few concurrent 1 MB responses can't
> stall the whole pipeline for everyone else.

---

## Core event-routing throughput

Independent of HTTP: request → response round-trips through the real Router
(each round-trip = two Router dispatches + two pickled pipe hops — a request
handler attaches a response and the same Manager's response handler receives
it back):

| Round-trips | Throughput |
|---|---|
| 1 000 | ~14 000–16 000 events/s |
| 10 000 | ~16 000–18 000 events/s |
| 50 000 | ~11 000–14 000 events/s |

So the Router sustains on the order of **~12–18k full event round-trips per
second** here (three runs each; low-n figures carry more relative noise since
the whole request completes in well under a second). This is the ceiling the
API path sits on top of; the HTTP numbers are lower because each request also
pays Listener parsing, the Broker, two extra process hops and JSON
serialization.

---

## Worker scaling (CPU-bound)

A CPU-bound handler (~128 ms of compute) offloaded to a Worker, `ab -c 8`, while
varying the worker type and `max_workers`:

| Worker type | `max_workers` | Throughput |
|---|---|---|
| `WorkerThread` | 1 | ~6.9 req/s |
| `WorkerThread` | 8 | ~7.0 req/s |
| `WorkerProcess` | 1 | ~7.0 req/s |
| `WorkerProcess` | 8 | ~36 req/s |

The headline: **threads do not scale CPU-bound work** — the GIL serializes them,
so `max_workers=8` is no faster than `1`. **Processes do** — ~5.2× here with
8 workers (not a full 8× because each task pays process-spawn overhead, and the
box was otherwise busy). Use `WorkerProcess` for CPU-bound work and size
`max_workers` to your cores; use `WorkerThread` for I/O-bound work (no GIL
contention while blocked, much cheaper than a process). See
{doc}`/worker`.

---

## Manager scaling

The same workload — a handler that burns ~2 ms of CPU **inline** (so the
single-threaded Manager is the bottleneck) — spread across N independent
Managers, each its own event type, driven at total concurrency 32:

| Managers | Aggregate throughput | p95 |
|---|---|---|
| 1 | ~450 req/s | 106 ms |
| 2 | ~730 req/s | 68 ms |
| 4 | ~990 req/s | 45 ms |

A single Manager caps at ~450 req/s (≈ 1 / 2 ms, serial), and adding Managers
scales aggregate throughput (~2.2× from 1→4) while cutting tail latency. It is
not perfectly linear — the shared Listener/Broker/Router pipeline is also in the
path — but this is the concrete payoff of the "split work across more Managers"
advice: each Manager is its own process and runs in true parallel.

---

(multi-worker-listener-uvicorn-workers)=
## Multi-worker Listener (uvicorn workers)

The Listener normally runs as a single process, so its own CPU-bound work —
HTTP/ASGI parsing, event pickling, JSON encoding — is capped at one core no
matter how much concurrency you throw at it. Set `workers` via
`EDRI_ASGI_OPTIONS` (see {doc}`/env` for details) to run N worker processes
instead, each a spawned OS process sharing the same listening socket and the
same Broker queue (the response-pipe pool that removes per-request
registration overhead — see below — already covers every worker, since it's
per-process state):

```sh
EDRI_ASGI_OPTIONS='{"workers": 4}'
```

WebSocket and REST `/noop` throughput, 1 worker vs. 4, two runs to check
stability:

| Concurrency | WS: 1 worker | WS: 4 workers | REST: 1 worker | REST: 4 workers |
|---|---|---|---|---|
| 1 | ~960–1010 | ~1070 | ~1095–1125 | ~1060–1090 |
| 8 | ~2660–2725 | ~3560–3790 | ~3005–3050 | ~3755–3815 |
| 32 | ~3035–3090 | ~4220–4240 | ~3660–3765 | ~4560–4675 |

Roughly **+25–35%** at concurrency 8/32 for both protocols. WebSocket moves
too even though a WS connection is pinned to whichever worker accepted it —
the gain is from spreading many independent connections' framing/encoding
work across cores, not from any one connection getting faster.

**Sizing:** re-measure with [`benchmarks/bench.py`](#reproducing-these-measurements)
on your own hardware and workload before picking a number — the right count
depends on how many cores the Listener's own parsing/encoding work can
actually use before something else in the pipeline (the Broker, still a
single process) becomes the next bottleneck. This is exactly the kind of
number the warning at the top of this page is about.

### Why the response-pipe pool matters here

Registering a client with the Broker — a `Register` round-trip, socket
setup, and an fd hand-off to pass the return pipe across the process
boundary — is real `multiprocessing` connection/socket work, not free. The
Broker is a single process regardless of how many Listener workers there
are, so without pooling, a multi-worker Listener would just mean more
requests paying that per-request registration cost against the same single
process — capping the gains above well below what multi-worker alone could
otherwise deliver.

The response-pipe pool avoids it: register a client once, hand its pipe back
to the pool once the response has been drained, and reuse it for the next
request instead of paying the registration cost again. That's why
multi-worker and the pool compound rather than either one hitting a ceiling
alone — multi-worker parallelizes the Listener's own CPU work across cores,
pooling keeps the Broker doing only real request-routing work.

**Trade-offs:**
* Requests are load-balanced across workers by the kernel accepting on a
  shared socket (no `SO_REUSEPORT` tuning), which is not perfectly even under
  low concurrency — fine in aggregate, worth knowing if you see per-worker
  vitals differ.
* Middleware instances must be picklable (they get sent to each spawned
  worker along with the rest of the app) — the same constraint any
  `multiprocessing`-shared object already has elsewhere in EDRI.
* This only applies to the API edge. The Broker, Router, and each Manager are
  still single processes — combine with **Manager scaling** (above) to raise
  the ceiling further down the pipeline too.

---

## Middleware overhead

Throughput of `/noop` (concurrency 16) with a chain of N pass-through
middlewares (each runs on both request and response):

| Middlewares | Throughput | p50 |
|---|---|---|
| 0 | ~3940 req/s | 3.8 ms |
| 1 | ~3530 req/s | 4.0 ms |
| 4 | ~3235 req/s | 4.4 ms |
| 8 | ~2735 req/s | 5.2 ms |

A small, roughly linear cost per middleware — p50 moves from 3.8 ms to 5.2 ms
across 8 middlewares, ~0.18 ms each. Still small in absolute terms: layer
middlewares (auth, logging, timing, …) reasonably freely; the cost that
matters is whatever work *your* middleware does, not the chain mechanism
itself.

---

## Serialization cost

Every cross-process hop pickles the event. Round-trip `pickle.dumps` +
`pickle.loads` of an event by payload size:

| Payload | dumps+loads |
|---|---|
| empty | ~5 µs |
| 1 KB | ~5 µs |
| 100 KB | ~12 µs |
| 1 MB | ~290 µs |

Pickle itself is cheap; it is the **number of hops × payload** that adds up (an
API request is pickled ~4 times end to end). This is the concrete reason large
payloads belong on `SharedMemoryPipe` rather than in a response field.

---

## Free-threaded CPython (no-GIL)

CPython 3.13+ offers a free-threaded build (`3.14t`; install with
`uv python install 3.14t` or an official free-threaded installer). The full
test suite, including the Switch integration tests (multiprocessing,
subprocess, TCP, pickle), passes on it unchanged.

`posix_ipc` (used by `SharedMemoryPipe`, see {ref}`sharedmemorypipe`) has not
declared free-threading support, so importing it re-enables the GIL for the
whole process — silently, with only a `RuntimeWarning`. A plain routing app
stays GIL-free; any app that uses `File`/`SharedMemoryPipe` loses
free-threading process-wide the moment that module loads.

EDRI's own architecture is multiprocessing end to end — each Manager is a
separate process — so removing the GIL does not by itself speed anything up;
no CPU-bound Python currently runs on threads inside one process.
Free-threading is an enabler, not a free win. It is not covered by CI: the
test matrix runs `3.12`/`3.13`/`3.14`/`3.15-rc`, no `t` variant.

---

## CPython experimental JIT

CPython 3.13+ ships an experimental copy-and-patch JIT, enabled at runtime with
`PYTHON_JIT=1` (the interpreter must be built with the JIT included — the
official 3.14 builds are, disabled by default). The full test suite passes with
it enabled.

It does not speed up EDRI's own pipeline: routing, pickling and JSON encoding
are attribute- and I/O-bound, not the tight numeric loops the JIT targets. On
the `managers` benchmark (handler burns CPU inline) it lifts aggregate
throughput ~10% and trims p95; on the `ws`/REST path it stays within noise. So
it only helps when Manager handlers do heavy CPU-bound Python — measure your own
workload before enabling it in production.

---

## Scaling guidance

| Symptom | Lever |
|---|---|
| A single Manager saturates (throughput ≈ 1 / handler time, latency climbing) | Split handlers across **more Managers**; run multiple app instances behind a load balancer |
| Listener itself is the bottleneck (HTTP parsing/pickling pegs one core) | [Multi-worker Listener](#multi-worker-listener-uvicorn-workers): `EDRI_ASGI_OPTIONS='{"workers": 4}'` — re-measure to size it |
| CPU-bound handler is slow / serializes | `WorkerProcess` with `max_workers` ≈ cores |
| Many blocking I/O calls in a handler | `WorkerThread` (cheap, GIL released while blocked) |
| Large responses / downloads | `File` + {ref}`sharedmemorypipe` (out-of-band, backpressured) |
| Slow clients stalling the API | Already handled by the broker {ref}`non-blocking-outbound-send-sender-pool`; size `EDRI_API_SENDER_THREADS` to expected concurrent-slow-client count |
| Need to see where time goes per request | A response {doc}`/middleware` that publishes `event._timing` as a `Server-Timing` header — per-stage latency in DevTools/APM |

---

## Reproducing these measurements

Everything above comes from the committed suite under `benchmarks/`:

* `benchmarks/app.py` — an EDRI server exposing the endpoints used here (trivial
  handler, sized payload, CPU worker, Manager↔Manager round-trip counter, busy
  per-manager endpoints, a WebSocket round-trip, a deliberately slow consumer),
  all shaped by environment variables (`BENCH_WORKER`, `BENCH_MAXW`,
  `BENCH_MANAGERS`, `BENCH_MIDDLEWARES`, `BENCH_GRPC`, `BENCH_SLOW`, …).
* `benchmarks/bench.py` — a self-contained runner (stdlib threads for HTTP, the
  `websockets` library for WS, `grpc.aio` for gRPC — no external load tool) that
  launches the server in the right shape per scenario and prints the tables above.
* `benchmarks/fastapi_app.py` + `benchmarks/compare_fastapi.py` — the
  single-process reference point above. Needs `pip install fastapi`
  separately; not an EDRI dependency.

```bash
python benchmarks/bench.py                 # all scenarios
python benchmarks/bench.py ws              # one: ws | managers | middleware | grpc | slow
python benchmarks/bench.py grpc            # gRPC vs WebSocket vs REST (needs the grpc extra)
python benchmarks/bench.py slow            # latency under a consumer that won't drain
python benchmarks/compare_fastapi.py       # EDRI vs FastAPI reference point
```

The `grpc` scenario needs `pip install edri[grpc]` (or `grpcio`/`grpcio-reflection`
directly) — not installed by default, the same way `fastapi` isn't for the
comparison above.

The HTTP-throughput, payload, core-routing and worker-scaling numbers in the
earlier sections were taken the same way; drive `benchmarks/app.py` directly if
you want to vary those by hand:

```bash
ENVIRONMENT=production python benchmarks/app.py        # then, against it:
curl http://127.0.0.1:8080/route/10000                 # core events/s
# worker scaling: restart with BENCH_WORKER=thread|process, BENCH_MAXW=1|8
```