Skip to content

Protocol and API Reference

This page is the compact reference for the control-plane API and data-plane transport. Use runtime.md first if you want the conceptual flow.

Control Plane

Purpose Route or command
Health GET /api/v1/health, GET /api/v1/ready
Jobs POST /api/v1/jobs, GET /api/v1/jobs, GET /api/v1/jobs/{job_id}
Job status POST /api/v1/jobs/{job_id}/status
Job metrics GET /api/v1/jobs/{job_id}/metrics/stream
Cancel job DELETE /api/v1/jobs/{job_id}
Checkpoint POST /api/v1/jobs/{job_id}/checkpoint
Workers GET /api/v1/workers, DELETE /api/v1/workers/{fellow_id}
Worker heartbeat POST /api/v1/workers/register, POST /api/v1/workers/heartbeat
Worker management POST /api/v1/workers/release, POST /api/v1/workers/update
Worker logs GET /api/v1/workers/{fellow_id}/logs/stream
Signals WS /ws/signals/{fellow_id}
Bootstrap GET /api/v1/bootstrap/manifest.json, GET /api/v1/bootstrap/fellow.sh
Ops UI GET /ops/join-v2

Control Signal Values

ControlSignal.signal supports:

  • START
  • PAUSE
  • CHECKPOINT
  • STOP
  • RECONFIGURE
  • UPDATE
  • PURGE

Registration And Heartbeat

sequenceDiagram
    participant Fellow
    participant Leader
    Fellow->>Leader: POST /api/v1/workers/register
    Leader-->>Fellow: RegisterResponse
    Fellow->>Leader: WS /ws/signals/{fellow_id}
    loop every heartbeat interval
        Fellow->>Leader: POST /api/v1/workers/heartbeat<br/>status, step, loss, bytes, live hardware
        Leader-->>Fellow: HeartbeatResponse
    end

Heartbeats include live GPU hardware telemetry when available. The leader uses that payload to refresh per-GPU free VRAM in /api/v1/cluster/nodes and /api/v1/cluster/devices, so the Ops UI shows current VRAM utilization instead of only the registration-time inventory.

Data Plane

Direct fellow-to-fellow traffic uses ZeroMQ multipart messages:

flowchart LR
    F0[Fellow i] -- activation PUSH :5557 --> F1[Fellow i+1]
    F1 -- gradient PUSH :5558 --> F0

Each packet carries metadata plus raw tensor bytes. The relay path uses the same multipart semantics over leader HTTP endpoints:

Relay route Purpose
POST /api/v1/relay/{job_id}/{channel}/push Push encoded frames into a channel.
GET /api/v1/relay/{job_id}/{channel}/pop Pop encoded frames, optionally blocking.

Transport selection is controlled with SWARM_PIPELINE_TRANSPORT=auto|tunnel|p2p.

Authentication Notes

  • The swarm CLI can send Authorization: Bearer <token> via --token or config.
  • Bootstrap downloads can require X-Bootstrap-Token when SWARM_BOOTSTRAP_REQUIRE_TOKEN=1.
  • Production deployments should enforce broader authentication at the leader or ingress layer.