Architecture Overview

SilentSwarm has one control-plane leader and multiple worker processes called fellows. The leader owns orchestration, registry state, APIs, streams, and the Ops UI. Fellows own training work, heartbeat reporting, log shipping, and tensor transport.

flowchart LR
    Operator[Operator / CI] --> CLI[swarm CLI]
    Operator --> UI[Ops UI]
    CLI --> Leader[Leader FastAPI]
    UI --> Leader
    Leader --> Registry[WorkerRegistry]
    Leader --> Jobs[JobManager]
    Leader --> Signals[SignalHub]
    Leader --> Metrics[Metrics and log streams]
    Signals -- WebSocket --> F0[Fellow 0]
    Signals -- WebSocket --> F1[Fellow 1]
    F0 -- activations --> F1
    F1 -- gradients --> F0
    F0 -- heartbeat/logs --> Leader
    F1 -- heartbeat/logs --> Leader

Design Goals

Goal	Current approach
Orchestration	FastAPI leader with jobs, workers, signals, metrics, logs, and Ops UI.
Pipeline training	Fellows execute assigned stages and exchange activation/gradient tensors.
Bandwidth research	Compression boundaries compare no compression, fixed quantization, and learned bottlenecks.
Operational access	Local CLI, Docker Compose frontend, remote bootstrap, and optional tunnel.
Observability	Heartbeats, worker log streams, job metrics streams, and run artifacts.

Two Planes

flowchart TB
    subgraph Control[Control plane]
        REST[REST /api/v1]
        WS[WebSocket /ws/signals/{fellow_id}]
        SSE[SSE logs and metrics]
    end
    subgraph Data[Data plane]
        ZMQ[ZeroMQ PUSH/PULL]
        Relay[HTTP relay fallback]
    end
    Leader[Leader :8080] --> Control
    Fellows[Fellows] --> Control
    Fellows --> Data

Pages

• runtime.md: leader, fellow, job, and training lifecycle.
• reference.md: transport, API routes, signals, and payloads.
• deployment.md: local, Docker, tunnel, and Kubernetes-oriented deployment notes.