# Switch

> **Security warning:** The Switch authenticates peers with a pre-shared key but does **not** encrypt traffic, and it deserializes Python pickle from every authenticated peer. Bound to anything other than loopback it requires `EDRI_SWITCH_SECRET` (or `EDRI_SWITCH_SECRET_FILE`); without one it starts on loopback only. Keep the port (default 8899) off the public internet and run it on a trusted, isolated network. See [Security](#security) below.

The **Switch** is a standalone TCP/IP server that acts as a central hub connecting multiple EDRI {doc}`/router` across a network. While the Router is the local dispatcher inside a single EDRI instance, the Switch enables **distributed event routing** across machines, containers, or data centers.

```mermaid
graph TB
    S((Switch))
    subgraph "Node 1"
        R1[Router A] --- C1[Connector A]
        C1 -- TCP/IP --> S
    end
    subgraph "Node 2"
        R2[Router B] --- C2[Connector B]
        C2 -- TCP/IP --> S
    end
    subgraph "Node 3"
        R3[Router C] --- C3[Connector C]
        C3 -- TCP/IP --> S
    end
```

The Switch itself is **not** a Router — it does not interpret event contents or maintain subscriptions. Payloads are opaque to it: it authenticates and re-frames each one but never deserializes it.

---

## Architecture

The Switch is composed of three internal components that run as daemon threads:

### Sender (per connection)

* Reads payloads from a per-connection `queue.Queue`.
* Wraps each one in an authenticated `EVENT` frame (see {ref}`switch-message-protocol`) and sends it over the TCP socket. Everything the Switch sends downstream is an `EVENT` frame — the Connector dispatches on the unpickled Python type, so demands and last-events blobs need no distinct type in this direction.
* If the send fails (remote side disconnected), the payload is placed back on the queue and the thread exits.
* The queue never loses messages — they are re-queued on send failure.

### Receiver (per connection)

* Reads authenticated frames from the TCP socket, verifying each frame's MAC before acting on it.
* Branches on the frame's `type` field:
  * `DEMANDS` and `LAST_EVENTS` — the payload is forwarded to **all other connected routers**.
  * `EVENT` — the 16-byte `target` field is the destination router UUID; the payload is forwarded only to that router.
* A frame that fails MAC verification, carries an out-of-order sequence number, or has an unexpected type drops the connection.
* On connection reset or socket closure, the thread exits gracefully.

### Forwarder (single thread)

* Reads `(router_id, forward_type, payload)` tuples from a central queue.
  * `ForwardType.SPECIFIC` — forward the payload only to `router_id`.
  * `ForwardType.OTHERS` — forward the payload to every connected router **except** `router_id`.
* This decouples network I/O from routing logic.

A separate daemon thread serves the read-only [status endpoint](#status-endpoint); it touches none of the forwarding path.

---

(switch-message-protocol)=
## Message protocol

Every frame in either direction is one 48-byte header followed by a payload:

| Offset | Size | Field   | Description                                                        |
|--------|------|---------|--------------------------------------------------------------------|
| 0      | 1    | version | Protocol version (`1`)                                             |
| 1      | 1    | type    | Frame type (see below)                                             |
| 2      | 2    | flags   | Reserved                                                           |
| 4      | 8    | seq     | Per-direction sequence number, gap-free, never reset               |
| 12     | 4    | length  | Payload length in bytes (big-endian unsigned)                      |
| 16     | 16   | target  | Destination router UUID on a Connector→Switch `EVENT`; zero otherwise |
| 32     | 16   | mac     | HMAC-SHA256 (truncated to 16 bytes) over `header[0:32] ‖ payload`  |
| 48     | …    | payload | Serialized event or demands data (raw bytes)                       |

The MAC covers every header byte except the MAC field itself, so type, sequence and target are authenticated, not just the body. It is keyed with the session key for that direction, derived during the handshake — not the pre-shared key itself. A frame whose MAC does not verify, or whose `seq` is not the next expected value, is rejected and the connection is dropped.

The payload cap is `EDRI_SWITCH_MAX_MESSAGE` (default 64 MiB); the `length` field is validated against it before the payload is read. Bulk data belongs on a {doc}`/shared_memory_pipe`, not the Switch.

The **source** router is not carried in each frame — it is the identity of the connection itself, established during the handshake. When the Switch forwards a payload to a destination router it builds a fresh `EVENT` frame with that connection's own keys and sequence numbers; frames are re-framed hop by hop, never passed through byte for byte.

### Frame types

| Type           | Direction              | Meaning                                                    |
|----------------|------------------------|------------------------------------------------------------|
| `HELLO_SERVER` | Switch → Connector     | First handshake frame; carries the server nonce            |
| `HELLO_CLIENT` | Connector → Switch     | Router UUID, client nonce, and the client's transcript MAC |
| `HELLO_ACCEPT` | Switch → Connector     | The server's transcript MAC; completes the handshake       |
| `EVENT`        | both                   | A shareable event; `target` names the destination router   |
| `DEMANDS`      | Connector → Switch     | Router announces its event-type subscriptions              |
| `LAST_EVENTS`  | Connector → Switch     | Router requests replay of events queued during an outage   |
| `ERROR`        | both                   | Reserved for a structured rejection reason                 |

The types are defined in `edri.protocol.frame.FrameType`.

---

## Connection lifecycle

### 1. Handshake

The Connector opens a TCP connection and runs a three-frame mutual handshake:

1. Switch → Connector: `HELLO_SERVER` with a random server nonce.
2. Connector → Switch: `HELLO_CLIENT` with the Router UUID, a client nonce, and a MAC over the transcript keyed by the pre-shared key.
3. Switch → Connector: `HELLO_ACCEPT` with the server's own transcript MAC.

Both sides prove possession of the pre-shared key, and the Router UUID is inside the MACed transcript, so an outsider cannot claim another router's identity. Each side then derives a pair of per-direction session keys from the two nonces and the pre-shared key; every frame after the handshake is MACed with those, never with the pre-shared key directly. A peer that fails the handshake has its connection dropped without a reply.

If a router with the same UUID is already connected, the newcomer — having proved key possession — **evicts** the incumbent: the old channel is closed and its Sender is stopped. A half-open socket left by a crashed node cannot be told from a healthy one, so this is what lets a restarted node reclaim its own identity. The new connection always starts with a fresh, empty queue.

### 2. Demand propagation

Immediately after the handshake, the Switch sends the **demands** (subscriptions) of every other connected router to the new router. This ensures the new router knows which event types other routers are interested in.

### 3. Ongoing communication

* Routers send shareable events to the Switch as `EVENT` frames addressed to specific remote routers.
* Routers broadcast their demands to all other routers on every change.
* The payload stays opaque to the Switch; it re-frames each one under the destination connection's keys and forwards it unread.

### 4. Disconnection and reconnection

When a TCP connection drops, the Switch keeps the `Connection` entry with its now-dead channel. Frames addressed to that router are still queued, but its Sender exits the first time it tries the dead socket, so the backlog is never delivered. When the router reconnects it gets a fresh connection with an empty queue, and that backlog is discarded.

The Switch therefore preserves **nothing** across a reconnect. Recovery is entirely the Connector's pull-based cache replay: on reconnect each side asks its peer for the ID of the last event it received and replays from the Router's in-memory cache everything after it, bounded by `EDRI_CACHE_TIMEOUT`. See {doc}`/connector` for that exchange.

---

## Running the Switch

### Standalone (Python)

```bash
python -m edri.switch.switch
```

This starts the Switch on `localhost:8899` by default, with no authentication — fine for local development. To accept connections from other hosts, set `EDRI_SWITCH_HOST` (e.g. `0.0.0.0`) **and** `EDRI_SWITCH_SECRET`; the Switch refuses to start on a non-loopback address without a secret.

### Docker

The project publishes a versioned Switch image for every release tag. Use the
version matching the EDRI package:

```bash
docker run -d --name edri-switch \
  -p 8899:8899 \
  -e EDRI_SWITCH_HOST=0.0.0.0 \
  -e EDRI_SWITCH_SECRET=<32+ byte key> \
  blastycz/edri:switch-<VERSION>
```

Prefer `EDRI_SWITCH_SECRET_FILE` with a mounted Docker or Kubernetes secret over
an inline `-e` value, which is visible in `docker inspect` and the pod spec.

> For the strongest reproducibility guarantee, pin the versioned image by its
> published SHA256 digest. The `switch-latest` tag points to the newest stable
> release; release candidates are available only through their explicit
> `switch-<VERSION>` tags. Avoid mutable tags in production.
>
> **Security:** The `-p 8899:8899` flag publishes the Switch port on the host's network.
> In a production environment, use ``-p 127.0.0.1:8899:8899`` to bind to loopback only,
> or omit the ``-p`` flag entirely and rely on Docker's internal networking
> (``--network host`` or a custom bridge network). The Switch port must **never** be
> exposed to the public internet. See [Security](#security) below.

### Docker (build locally)

```bash
docker build -f edri/switch/Dockerfile -t edri-switch .
docker run -d --name edri-switch -p 8899:8899 edri-switch
```

### Configuration

| Environment Variable | Default     | Description                            |
|----------------------|-------------|----------------------------------------|
| `EDRI_SWITCH_HOST`   | `localhost` | Address to bind the Switch to          |
| `EDRI_SWITCH_PORT`   | `8899`      | Port to bind the Switch to             |
| `EDRI_SWITCH_SECRET` | *(unset)*   | Pre-shared key the Switch and every Connector authenticate with; required for any non-loopback bind |
| `EDRI_SWITCH_SECRET_FILE` | *(unset)* | Path to a file holding one key per line (for key rotation); takes precedence over `EDRI_SWITCH_SECRET` |
| `EDRI_SWITCH_MAX_MESSAGE` | `67108864` | Largest accepted frame payload, in bytes (64 MiB) |
| `EDRI_SWITCH_STATUS_ENABLED` | `true` | Start the HTTP status endpoint (below) |
| `EDRI_SWITCH_STATUS_HOST` | `localhost` | Address to bind the status endpoint to |
| `EDRI_SWITCH_STATUS_PORT` | `8898`  | Port for the status endpoint           |

A secret must be at least 32 bytes. Generate one with
`python -c "import secrets; print(secrets.token_urlsafe(32))"`.

The Switch logs at `DEBUG` level by default.

---

(switch-status-endpoint)=
## Status endpoint

The Switch lives outside EDRI's router/manager/event topology — it forwards raw
bytes and never speaks the `Event` protocol — so the API
{ref}`health-status-endpoint` machinery cannot be
reused here. Instead it exposes the same *kind* of operational snapshot through a
thin, read-only stdlib HTTP server running in a daemon thread (zero extra
dependencies):

* `GET /status` (also served at `/`) → JSON. Any other path returns `404`.

It binds to `EDRI_SWITCH_STATUS_HOST:EDRI_SWITCH_STATUS_PORT` (default
`localhost:8898`) and can be turned off with `EDRI_SWITCH_STATUS_ENABLED=false`.
A taken status port is logged and skipped — it never stops the Switch from
forwarding. The handler only reads live structures, so it needs no locking and
cannot perturb routing; a momentarily stale count is the only cost.

```bash
curl http://localhost:8898/status
```

```json
{
  "connections": [
    {
      "router_id": "577e4ac2-6753-48b1-ac08-62097e5c5d7b",
      "peer": "127.0.0.1:35964",
      "queue_depth": 0,
      "has_demands": true,
      "demands_bytes": 142
    }
  ],
  "connections_total": 1,
  "forwarder_queue_depth": 0,
  "vitals": { "rss_bytes": 25477120, "cpu_seconds": 0.06, "threads": 4,
              "open_fds": 7, "pid": 2845344, "uptime_seconds": 1.94 }
}
```

| Field | Meaning |
|---|---|
| `connections[].router_id` | UUID of the connected Router. |
| `connections[].peer` | Remote `host:port` of the TCP connection; `null` once the socket is closed. |
| `connections[].queue_depth` | Messages queued for that Router — its outbound backpressure (a Router that stops draining). |
| `connections[].has_demands` / `demands_bytes` | Whether the Router has announced subscriptions yet, and their size. |
| `connections_total` | Number of connected Routers. |
| `forwarder_queue_depth` | Depth of the single central Forwarder queue — the Switch's overall backpressure signal. |
| `vitals` | Switch process metrics (`rss_bytes`, `rss_peak_bytes`, `cpu_seconds`, `threads`, `open_fds`, `pid`, `uptime_seconds`); best-effort, `null` when unreadable on the platform. |

---

## Resilience guarantees

| Scenario                    | Behavior                                                   |
|-----------------------------|------------------------------------------------------------|
| Router disconnects          | Frames addressed to it are queued but never delivered; that connection's Sender exits once it hits the dead socket |
| Router reconnects (same ID) | The old connection is evicted and a fresh, empty queue is created; the accumulated backlog is discarded |
| Switch crashes              | All in-memory queues are lost (no persistence)             |
| Network partition           | Events sent during the partition are lost by the Switch; recovery depends on the Connector's cache replay |

> **Important:** The Switch is a single-instance, in-memory component with no persistence and no built-in HA/failover, and it preserves nothing across a reconnect. The Router cache (`EDRI_CACHE_TIMEOUT`) is the only safety net for a Switch or link outage — the Connector replays missed events from it on reconnect (see {doc}`/connector`). For production resilience, monitor the Switch process and restart it promptly on failure.

---

(security)=
## Security

The Switch authenticates its peers, then forwards pickle-serialized payloads between them without inspecting them:

* **Pre-shared key authentication** — `EDRI_SWITCH_SECRET` (or `EDRI_SWITCH_SECRET_FILE`) is a cluster-wide key. The mutual handshake stops any peer without it from connecting or from impersonating another router. It does **not** authenticate members *against each other*: every node holds the same key. A Switch bound to any non-loopback address refuses to start without a secret; on loopback it starts without one and logs a warning. Generate one with `python -c "import secrets; print(secrets.token_urlsafe(32))"`.
* **No transport encryption** — frame payloads travel over TCP in the clear. The MAC authenticates each frame but does not hide it; a network-level attacker can still read every event payload. Use a VPN, WireGuard tunnel, mTLS proxy, or service mesh for sensitive payloads.
* **No payload validation** — the Switch does not deserialize or validate what it forwards. An authenticated but compromised peer can still send arbitrary pickle to another peer, which unpickles it — and unpickling is code execution, so one compromised key holder runs code on every other node. Connect only nodes you trust as much as each other, restrict who holds the key, and rotate it if it may have leaked (see `EDRI_SWITCH_SECRET_FILE` in {doc}`/env`, which keeps the old and new key valid at once).
* **Status endpoint** — the HTTP status endpoint (default port 8898) has no authentication of its own and exposes operational metadata (router IDs, peer addresses, queue depths). It refuses to start off-loopback unless a Switch secret is set, and even then it stays readable by anyone who can reach the port — restrict access to monitoring systems.

### Network policy (Kubernetes)

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: edri-switch-ingress
spec:
  podSelector:
    matchLabels:
      app: edri-switch
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: edri-app
      ports:
        - port: 8899
```

---

## Related

* {doc}`/connector` — how a Router connects to the Switch
* {doc}`/router` — local event dispatching, subscriptions, and shareable events
* {doc}`/event` — event model, including `shareable` flag
* {doc}`/env` — environment variables for Switch configuration