Performance & scaling

This page gives a feel for EDRI's throughput, latency and how it scales — plus a reproducible harness so you can measure your workload on your hardware.

Warning

Indicative numbers — read the shape, not the absolutes. Every figure below is from a single manual run; absolute throughput varies with hardware, load and Python version. What transfers is the scaling behavior and where the limits are. Re-run the benchmarks/ suite to get numbers for your own setup.

Measurement environment: 24-core Linux box · CPython 3.14 · ENVIRONMENT=production · clients over loopback · a single EDRI instance.

What bounds EDRI's performance

EDRI trades raw single-endpoint throughput for isolation and explicit concurrency (see Architecture Overview):

  • The Router, the Broker and each Manager are single-threaded processes. One request crosses several of them.

  • Events move between processes as pickled objects through multiprocessing queues/pipes — every hop is a serialize + copy.

  • Parallelism is explicit: a Manager handles one event at a time; CPU work must be pushed to Workers (threads or processes).

So the levers are: keep per-event work small, parallelize with Workers (processes for CPU), split load across multiple Managers, and keep large payloads off the pickled path (use SharedMemoryPipe in Workers).

For context: a single-process ASGI framework

Warning

Not an apples-to-apples framework benchmark. FastAPI here is a single in-process ASGI app with no distributed routing — a bare async def handler returning a dict. EDRI's /noop crosses four processes (Listener → Broker → Router → Manager and back), pickling the event at every hop, in exchange for the isolation and distributed routing described above. The gap below is the visible cost of that architecture, not a claim that EDRI is slow at what it's actually for. Reproduce with python benchmarks/compare_fastapi.py (needs pip install fastapi, otherwise not an EDRI dependency).

Same trivial /noop endpoint, same client-side load generator (http_load), same hardware, two runs:

Concurrency

EDRI (1 worker)

FastAPI (1 worker)

EDRI (4 workers)

FastAPI (4 workers)

1

~1040–1115

~9400–10000

~1150–1165

~9480–9660

8

~3050–3390

~12200–13400

~4080–4130

~12700–13260

32

~3625–3790

~12150–12860

~4450–4475

~13040–13200

FastAPI is ~3–9× faster here, and the gap is widest at concurrency 1 (~9×) and narrows toward higher concurrency (~3×): EDRI's fixed per-request cost is the four-hop, four-process round trip itself, which dominates when there's little concurrent work to pipeline behind it; FastAPI's is a single in-process function call with no IPC at all. Neither number moves much with workers, in either framework, at this concurrency — both are still far from saturating this box's cores.

The honest takeaway: if the whole job is "answer this HTTP request with no cross-process work," a single-process framework will always win, structurally. EDRI's cost buys the things in the section above — process isolation per Manager, a Worker model for CPU/I/O offload, distributed event routing, a pluggable Broker/Router — that a single in-process handler doesn't need and doesn't provide. Whether that trade is worth ~3–9× on a trivial endpoint depends entirely on whether your application needs what it buys.

HTTP request throughput & latency, WebSocket vs REST

A trivial round-trip (handler just sets one response field) over a persistent WebSocket connection versus REST (/noop), same client machine, three runs to check stability:

Concurrency

WebSocket msgs/s

REST req/s

1

~1115–1265

~1145–1200

8

~3350–3535

~2915–3100

32

~3220–3460

~3360–3485

REST tracks WebSocket closely: a WS client registers once on connect and then just streams; REST checks a client's return pipe out of a pool, reuses it for the request, and returns it once the response has drained — so neither path pays a per-request registration cost. Take the exact ordering with a grain of salt, though: the two load generators aren't quite apples-to-apples (ws_load drives all connections from one asyncio event loop; http_load uses a real OS thread per connection) — the headline is that neither path carries a per-request registration cost, not that one is architecturally faster than the other.

For chatty, low-latency interaction (live updates, RPC-style calls) WS still avoids HTTP framing/header overhead per message, so it remains the better fit even though the throughput gap has closed.

gRPC vs WebSocket vs REST round-trip

Same trivial round-trip, same Noop event, now also over gRPC — a unary call (one call per request, like REST) and a persistent bidi Connect stream (like WebSocket), same client machine, three runs:

Concurrency

WebSocket msgs/s

REST req/s

gRPC unary calls/s

gRPC bidi msgs/s

1

~1178–1227

~1057–1138

~959–1021

~1106–1215

8

~2881–3009

~2526–2755

~2513–2523

~2504–2559

32

~3564–3613

~2843–2863

~2779–2840

~3856–3901

The shape matches the WS/REST comparison above, one level down: gRPC unary tracks REST closely (both check a pipe out of a pool per call, no per-request registration), and gRPC bidi tracks WebSocket closely (both register once and then just stream) — with unary/bidi each landing a little below their REST/WS counterpart. That gap is protobuf encode/decode plus one extra asyncio hop per call inside GRPCListener (see gRPC's Data flow), on top of the same four-process round trip every transport pays. grpc.aio's own client-side overhead is part of that gap too, not isolated here — this measures the whole call, not just the server side.

Payload size

Same endpoint, response body of varying size, ab -c 8:

Response size

Throughput

Mean latency

1 KB

~2635 req/s

3.0 ms

100 KB

~1850 req/s

4.3 ms

1 MB

~340 req/s

23.6 ms

Small and medium payloads are cheap; at 1 MB the per-event serialize-and-copy across the pipeline dominates and throughput drops ~8× from the 1 KB baseline. For large or streamed responses, don't return a big pickled field — stream via File + SharedMemoryPipe in Workers, which moves the bytes out-of-band with backpressure and keeps them off the broker.

Large concurrent responses don't wedge the API: the broker's Non-blocking outbound send (sender pool) keeps one slow client from blocking the others, so a few concurrent 1 MB responses can't stall the whole pipeline for everyone else.

Core event-routing throughput

Independent of HTTP: request → response round-trips through the real Router (each round-trip = two Router dispatches + two pickled pipe hops — a request handler attaches a response and the same Manager's response handler receives it back):

Round-trips

Throughput

1 000

~14 000–16 000 events/s

10 000

~16 000–18 000 events/s

50 000

~11 000–14 000 events/s

So the Router sustains on the order of ~12–18k full event round-trips per second here (three runs each; low-n figures carry more relative noise since the whole request completes in well under a second). This is the ceiling the API path sits on top of; the HTTP numbers are lower because each request also pays Listener parsing, the Broker, two extra process hops and JSON serialization.

Worker scaling (CPU-bound)

A CPU-bound handler (~128 ms of compute) offloaded to a Worker, ab -c 8, while varying the worker type and max_workers:

Worker type

max_workers

Throughput

WorkerThread

1

~6.9 req/s

WorkerThread

8

~7.0 req/s

WorkerProcess

1

~7.0 req/s

WorkerProcess

8

~36 req/s

The headline: threads do not scale CPU-bound work — the GIL serializes them, so max_workers=8 is no faster than 1. Processes do — ~5.2× here with 8 workers (not a full 8× because each task pays process-spawn overhead, and the box was otherwise busy). Use WorkerProcess for CPU-bound work and size max_workers to your cores; use WorkerThread for I/O-bound work (no GIL contention while blocked, much cheaper than a process). See Worker.

Manager scaling

The same workload — a handler that burns ~2 ms of CPU inline (so the single-threaded Manager is the bottleneck) — spread across N independent Managers, each its own event type, driven at total concurrency 32:

Managers

Aggregate throughput

p95

1

~450 req/s

106 ms

2

~730 req/s

68 ms

4

~990 req/s

45 ms

A single Manager caps at ~450 req/s (≈ 1 / 2 ms, serial), and adding Managers scales aggregate throughput (~2.2× from 1→4) while cutting tail latency. It is not perfectly linear — the shared Listener/Broker/Router pipeline is also in the path — but this is the concrete payoff of the "split work across more Managers" advice: each Manager is its own process and runs in true parallel.

Multi-worker Listener (uvicorn workers)

The Listener normally runs as a single process, so its own CPU-bound work — HTTP/ASGI parsing, event pickling, JSON encoding — is capped at one core no matter how much concurrency you throw at it. Set workers via EDRI_ASGI_OPTIONS (see Environment Variables for details) to run N worker processes instead, each a spawned OS process sharing the same listening socket and the same Broker queue (the response-pipe pool that removes per-request registration overhead — see below — already covers every worker, since it's per-process state):

EDRI_ASGI_OPTIONS='{"workers": 4}'

WebSocket and REST /noop throughput, 1 worker vs. 4, two runs to check stability:

Concurrency

WS: 1 worker

WS: 4 workers

REST: 1 worker

REST: 4 workers

1

~960–1010

~1070

~1095–1125

~1060–1090

8

~2660–2725

~3560–3790

~3005–3050

~3755–3815

32

~3035–3090

~4220–4240

~3660–3765

~4560–4675

Roughly +25–35% at concurrency 8/32 for both protocols. WebSocket moves too even though a WS connection is pinned to whichever worker accepted it — the gain is from spreading many independent connections' framing/encoding work across cores, not from any one connection getting faster.

Sizing: re-measure with benchmarks/bench.py on your own hardware and workload before picking a number — the right count depends on how many cores the Listener's own parsing/encoding work can actually use before something else in the pipeline (the Broker, still a single process) becomes the next bottleneck. This is exactly the kind of number the warning at the top of this page is about.

Why the response-pipe pool matters here

Registering a client with the Broker — a Register round-trip, socket setup, and an fd hand-off to pass the return pipe across the process boundary — is real multiprocessing connection/socket work, not free. The Broker is a single process regardless of how many Listener workers there are, so without pooling, a multi-worker Listener would just mean more requests paying that per-request registration cost against the same single process — capping the gains above well below what multi-worker alone could otherwise deliver.

The response-pipe pool avoids it: register a client once, hand its pipe back to the pool once the response has been drained, and reuse it for the next request instead of paying the registration cost again. That's why multi-worker and the pool compound rather than either one hitting a ceiling alone — multi-worker parallelizes the Listener's own CPU work across cores, pooling keeps the Broker doing only real request-routing work.

Trade-offs:

  • Requests are load-balanced across workers by the kernel accepting on a shared socket (no SO_REUSEPORT tuning), which is not perfectly even under low concurrency — fine in aggregate, worth knowing if you see per-worker vitals differ.

  • Middleware instances must be picklable (they get sent to each spawned worker along with the rest of the app) — the same constraint any multiprocessing-shared object already has elsewhere in EDRI.

  • This only applies to the API edge. The Broker, Router, and each Manager are still single processes — combine with Manager scaling (above) to raise the ceiling further down the pipeline too.

Middleware overhead

Throughput of /noop (concurrency 16) with a chain of N pass-through middlewares (each runs on both request and response):

Middlewares

Throughput

p50

0

~3940 req/s

3.8 ms

1

~3530 req/s

4.0 ms

4

~3235 req/s

4.4 ms

8

~2735 req/s

5.2 ms

A small, roughly linear cost per middleware — p50 moves from 3.8 ms to 5.2 ms across 8 middlewares, ~0.18 ms each. Still small in absolute terms: layer middlewares (auth, logging, timing, …) reasonably freely; the cost that matters is whatever work your middleware does, not the chain mechanism itself.

Serialization cost

Every cross-process hop pickles the event. Round-trip pickle.dumps + pickle.loads of an event by payload size:

Payload

dumps+loads

empty

~5 µs

1 KB

~5 µs

100 KB

~12 µs

1 MB

~290 µs

Pickle itself is cheap; it is the number of hops × payload that adds up (an API request is pickled ~4 times end to end). This is the concrete reason large payloads belong on SharedMemoryPipe rather than in a response field.

Free-threaded CPython (no-GIL)

CPython 3.13+ offers a free-threaded build (3.14t; install with uv python install 3.14t or an official free-threaded installer). The full test suite, including the Switch integration tests (multiprocessing, subprocess, TCP, pickle), passes on it unchanged.

posix_ipc (used by SharedMemoryPipe, see SharedMemoryPipe in Workers) has not declared free-threading support, so importing it re-enables the GIL for the whole process — silently, with only a RuntimeWarning. A plain routing app stays GIL-free; any app that uses File/SharedMemoryPipe loses free-threading process-wide the moment that module loads.

EDRI's own architecture is multiprocessing end to end — each Manager is a separate process — so removing the GIL does not by itself speed anything up; no CPU-bound Python currently runs on threads inside one process. Free-threading is an enabler, not a free win. It is not covered by CI: the test matrix runs 3.12/3.13/3.14/3.15-rc, no t variant.

CPython experimental JIT

CPython 3.13+ ships an experimental copy-and-patch JIT, enabled at runtime with PYTHON_JIT=1 (the interpreter must be built with the JIT included — the official 3.14 builds are, disabled by default). The full test suite passes with it enabled.

It does not speed up EDRI's own pipeline: routing, pickling and JSON encoding are attribute- and I/O-bound, not the tight numeric loops the JIT targets. On the managers benchmark (handler burns CPU inline) it lifts aggregate throughput ~10% and trims p95; on the ws/REST path it stays within noise. So it only helps when Manager handlers do heavy CPU-bound Python — measure your own workload before enabling it in production.

Scaling guidance

Symptom

Lever

A single Manager saturates (throughput ≈ 1 / handler time, latency climbing)

Split handlers across more Managers; run multiple app instances behind a load balancer

Listener itself is the bottleneck (HTTP parsing/pickling pegs one core)

Multi-worker Listener: EDRI_ASGI_OPTIONS='{"workers": 4}' — re-measure to size it

CPU-bound handler is slow / serializes

WorkerProcess with max_workers ≈ cores

Many blocking I/O calls in a handler

WorkerThread (cheap, GIL released while blocked)

Large responses / downloads

File + SharedMemoryPipe in Workers (out-of-band, backpressured)

Slow clients stalling the API

Already handled by the broker Non-blocking outbound send (sender pool); size EDRI_API_SENDER_THREADS to expected concurrent-slow-client count

Need to see where time goes per request

A response Middleware that publishes event._timing as a Server-Timing header — per-stage latency in DevTools/APM

Reproducing these measurements

Everything above comes from the committed suite under benchmarks/:

  • benchmarks/app.py — an EDRI server exposing the endpoints used here (trivial handler, sized payload, CPU worker, Manager↔Manager round-trip counter, busy per-manager endpoints, a WebSocket round-trip, a deliberately slow consumer), all shaped by environment variables (BENCH_WORKER, BENCH_MAXW, BENCH_MANAGERS, BENCH_MIDDLEWARES, BENCH_GRPC, BENCH_SLOW, …).

  • benchmarks/bench.py — a self-contained runner (stdlib threads for HTTP, the websockets library for WS, grpc.aio for gRPC — no external load tool) that launches the server in the right shape per scenario and prints the tables above.

  • benchmarks/fastapi_app.py + benchmarks/compare_fastapi.py — the single-process reference point above. Needs pip install fastapi separately; not an EDRI dependency.

python benchmarks/bench.py                 # all scenarios
python benchmarks/bench.py ws              # one: ws | managers | middleware | grpc | slow
python benchmarks/bench.py grpc            # gRPC vs WebSocket vs REST (needs the grpc extra)
python benchmarks/bench.py slow            # latency under a consumer that won't drain
python benchmarks/compare_fastapi.py       # EDRI vs FastAPI reference point

The grpc scenario needs pip install edri[grpc] (or grpcio/grpcio-reflection directly) — not installed by default, the same way fastapi isn't for the comparison above.

The HTTP-throughput, payload, core-routing and worker-scaling numbers in the earlier sections were taken the same way; drive benchmarks/app.py directly if you want to vary those by hand:

ENVIRONMENT=production python benchmarks/app.py        # then, against it:
curl http://127.0.0.1:8080/route/10000                 # core events/s
# worker scaling: restart with BENCH_WORKER=thread|process, BENCH_MAXW=1|8