Skip to content

Runtime Model

This page explains how the leader, fellows, jobs, metrics, and logs interact at runtime.

Leader Responsibilities

  • WorkerRegistry: tracks fellow registration, status, heartbeat, hardware, and network limits.
  • JobManager: owns job state, queue execution, and terminal transitions.
  • SignalHub: keeps WebSocket connections and sends START, CHECKPOINT, STOP, UPDATE, and PURGE signals.
  • JobMetricsAggregator: merges per-fellow metrics for swarm watch and the Ops UI.
  • WorkerLogBuffer and bootstrap log buffers: expose log history and SSE streams.
  • Bootstrap routes: serve manifest, fellow.sh, source archive, join sessions, and logs.

Fellow Responsibilities

  • Register with the leader and keep the signal WebSocket open.
  • Report heartbeat status, step, loss, bytes sent/received, and hardware metrics.
  • Run the assigned training stage and exchange activation/gradient frames.
  • Ship runtime logs back to the leader.
  • Process update and purge signals when safe.

Fellow Lifecycle

stateDiagram-v2
    [*] --> registering: runner starts
    registering --> ready: register + heartbeat accepted
    ready --> training: START signal
    training --> checkpointing: CHECKPOINT signal
    checkpointing --> ready: checkpoint complete
    training --> ready: job terminal
    ready --> registering: UPDATE signal
    ready --> offline: heartbeat stale
    training --> failed: exception or peer timeout
    offline --> registering: process restart

Job Lifecycle

stateDiagram-v2
    [*] --> PENDING: POST /api/v1/jobs
    PENDING --> RUNNING: enough ready fellows
    RUNNING --> COMPLETED: training finishes
    RUNNING --> FAILED: runtime failure
    RUNNING --> CANCELLED: DELETE /api/v1/jobs/{job_id}
    PENDING --> CANCELLED: DELETE /api/v1/jobs/{job_id}
    COMPLETED --> [*]
    FAILED --> [*]
    CANCELLED --> [*]

Training Data Flow

sequenceDiagram
    participant F0 as Stage 0 fellow
    participant Wire as ZMQ or relay
    participant F1 as Stage 1 fellow
    F0->>F0: run embedding and first chunk
    F0->>Wire: activation frames
    Wire->>F1: activation frames
    F1->>F1: run second chunk and loss
    F1->>Wire: gradient frames
    Wire->>F0: gradient frames
    F0->>F0: upstream backpropagation

The current distributed path supports two stages. It sends real TensorFlow activation and gradient tensors; direct transport uses ZeroMQ, and firewalled transport can use the leader HTTP relay.

Observability Flow

flowchart LR
    Fellow[Fellow runner] --> Heartbeat[Heartbeat metrics]
    Fellow --> Logs[FellowLogShipper]
    Heartbeat --> Leader[Leader]
    Logs --> Leader
    Leader --> Watch[swarm watch job_id]
    Leader --> UI[Ops UI]