SharedMemoryPipe: streaming a ZIP of a folder¶

This example shows a working pattern for streaming a ZIP archive on the fly from a Worker to an API client using SharedMemoryPipe.

When to use this¶

You want to download a large folder as a ZIP.
You do not want to generate the ZIP file fully on disk first.
You want high throughput (Worker → client) using POSIX shared memory.

Note: HTTP Range resume is not supported for this streamed approach because the total ZIP size is not known up-front.

Event and response definitions¶

from http import HTTPMethod

from edri.api.dataclass import File
from edri.api.dataclass.api_event import api
from edri.dataclass.event import Event
from edri.dataclass.response import Response, response


@response
class FolderDownloadResponse(Response):
    file: File


@api(url="/", method=HTTPMethod.GET)
class FolderDownload(Event):
    path: str
    response: FolderDownloadResponse

Worker example¶

from pathlib import Path
from zipfile import ZIP_DEFLATED, ZipFile

from edri.abstract.worker import WorkerProcess
from edri.api.dataclass import File
from edri.utility.shared_memory_pipe import SharedMemoryPipe

from zzz_events import FolderDownload


class SharedMemoryZipWriter:
    """A write-only file object that forwards ZIP bytes to a pipe."""

    def __init__(self, pipe: SharedMemoryPipe) -> None:
        self._pipe = pipe

    def write(self, data: bytes) -> int:
        self._pipe.write(data)
        return len(data)

    def flush(self) -> None:
        pass

    def seekable(self) -> bool:
        return False


class ZipFolderWorker(WorkerProcess[FolderDownload]):

    def do(self) -> None:
        base = Path(self.event.path)

        with SharedMemoryPipe() as smp:
            # Tell the caller “a zip is coming”, and where to read it from.
            self.event.response.file = File(
                file_name="archive.zip",
                mime_type="application/zip",
                path=smp.reader(),
            )
            self.event_send(self.event)

            # ZipFile supports non-seekable output streams, so each ZIP record
            # is forwarded to shared memory as it is created.
            with ZipFile(
                SharedMemoryZipWriter(smp), "w", compression=ZIP_DEFLATED
            ) as archive:
                for path in base.rglob("*"):
                    if path.is_file():
                        # Stable ZIP paths (POSIX).
                        archive.write(path, path.relative_to(base).as_posix())

How it works¶

The worker creates a per-request SharedMemoryPipe.
It sets event.response.file to point to smp.reader() and sends the event.
The API layer begins downloading from the provided pipe reader.
The worker generates ZIP bytes incrementally and writes them into the pipe.

Requirements¶

FolderDownload must include a path field pointing to the folder to zip.
The API client must treat the response as a file download (File(...)).