Fellow Bootstrap Contract¶

This page documents the full bootstrap contract for leader-managed fellow onboarding.

Current Capabilities¶

Leader-managed onboarding provides:

a leader-served bootstrap manifest (manifest.json)
a leader-served shell bootstrap script (fellow.sh)
a leader-served source archive (source.tar.gz)
join sessions with optional token authentication
per-session log collection and SSE streaming
a web UI at /ops/join-v2 for creating join commands and monitoring connected fellows
live swarm graph with role lanes, stabilized interpolation, and per-fellow status in the /ops/join-v2 UI
non-interactive install and start flow with retry logic
macOS-tolerant dependency installation that lets a fellow join even when TensorFlow wheels are unavailable for the local Python version
ZMQ port overrides and idempotent re-runs
automatic fallback to the next free ZMQ port pair in non-interactive mode
SQLite-backed persistence for join sessions, bootstrap logs, and worker registry state across leader restarts

flowchart TD
  Operator[Operator or automation] --> Session[Create join session]
  Session --> Token[Receive session ID and token]
  Token --> Curl[Run leader-served curl command]
  Curl --> Manifest[Fetch manifest and bootstrap script]
  Manifest --> Archive[Download source archive]
  Archive --> Install[Create venv and install package]
  Install --> Start[Start fellow runner]
  Start --> Register[Register with leader]
  Register --> Ready[Fellow appears ready in Ops UI]
  Curl -. session logs .-> Logs[Bootstrap log stream]
  Logs -. SSE .-> Operator

Endpoints¶

Bootstrap Resources¶

Endpoint	Description
`GET /api/v1/bootstrap/manifest.json`	JSON description of the bootstrap contract
`GET /api/v1/bootstrap/fellow.sh`	Shell script that installs and starts a fellow
`GET /api/v1/bootstrap/source.tar.gz`	Gzip archive of the project source

When SWARM_BOOTSTRAP_REQUIRE_TOKEN=1 is set, all three endpoints require a valid X-Bootstrap-Token header.

Join Sessions¶

Endpoint	Description
`POST /api/v1/bootstrap/join-sessions`	Create a join session with a token and expiry
`GET /api/v1/bootstrap/join-sessions`	List all sessions
`GET /api/v1/bootstrap/join-sessions/{id}`	Get a session by ID
`PATCH /api/v1/bootstrap/join-sessions/{id}`	Update session status or host
`DELETE /api/v1/bootstrap/join-sessions/{id}`	Revoke a session

Bootstrap Logs¶

Endpoint	Description
`POST /api/v1/bootstrap/join-sessions/{id}/logs`	Append a log line
`GET /api/v1/bootstrap/join-sessions/{id}/logs`	Retrieve buffered log lines
`GET /api/v1/bootstrap/join-sessions/{id}/logs/stream`	SSE stream of log lines

Operations UI¶

Endpoint	Description
`GET /ops/join-v2`	Web UI that generates join commands and shows the swarm graph, training panel, logs, and connected fellows table
`GET /api/v1/ops/join-graph`	Snapshot payload for leader + worker graph rendering
`GET /api/v1/ops/join-graph/stream`	SSE stream of graph snapshot updates (1 s interval)

The /ops/join-v2 page is organized into three full-width vertical sections to use wide screens efficiently:

Add Fellow
Swarm Graph
Connected Fellows

Graph Readability Model¶

The /ops/join-v2 graph uses a lane-based layout to keep topology readable as the swarm grows:

leader is fixed on the left as a stable anchor
fellows are arranged on the right in a radial lane around a fixed center
node positions are interpolated between snapshots to avoid abrupt jumps

Node semantics:

labels show compact identity (#id + hostname) directly on the node
hostnames prefer the registered fellow host when the raw hardware hostname is only a container ID
full details (version, GPUs/VRAM, throughput, last seen) appear in tooltip and side panel
worker nodes use larger symbols and a custom hover card so state stays legible on wide operator screens
graph node details are refreshed from the same live worker inventory as the fellows table so status and VRAM stay aligned
status is encoded by both color and shape/ring
ready: circle with calm ring
training: rounded shape with pulsing highlight
offline: gray triangle (stale heartbeat)

Edge semantics:

default links are thin and subdued
links to actively training fellows are brighter and pulse subtly
leader-to-fellow links represent tunnel-capable control/data relay paths
fellow-to-fellow links are rendered only when direct P2P connectivity is available

Pipeline transport selection:

SWARM_PIPELINE_TRANSPORT=auto (default): tunnel fallback, auto-upgrade to direct P2P when fellow ports are reachable
SWARM_PIPELINE_TRANSPORT=tunnel: force leader relay tunnel
SWARM_PIPELINE_TRANSPORT=p2p: force direct fellow-to-fellow links

The graph header includes a small legend plus live counters for connected, training, and offline fellows.

flowchart LR
  subgraph Auto[SWARM_PIPELINE_TRANSPORT=auto]
    A[Fellow A] --> Probe{Peer ports reachable?}
    Probe -- yes --> P2P[Direct ZMQ P2P]
    Probe -- no --> Relay[Leader HTTP relay]
    Relay --> B[Fellow B]
    P2P --> B
  end

Sidepanel Actions¶

The graph sidepanel includes three actions per fellow:

Logs: opens bootstrap session logs via the existing join-session log stream
Restart: prepares a fresh join session and restart command for the selected fellow ID
Update: sends the existing POST /api/v1/workers/update signal to the selected fellow

Action feedback is rendered in a dedicated result area in the sidepanel and is kept stable across graph refreshes.

The Connected Fellows table uses the same action-button visual language (icon, color coding, and token-based styling) for logs, restart, and update actions.

Join UI State Persistence¶

The leader persists join sessions, bootstrap log entries, and worker registry entries to SQLite, so the join UI and worker inventory can recover their state after a restart.

default DB path: .swarm/join_state.sqlite3
override via env var: SWARM_JOIN_STATE_DB=/path/to/join_state.sqlite3

Persisted worker fields include fellow ID, host, ZMQ ports, hardware inventory, last known heartbeat/status, and current throughput counters. After a leader restart, workers remain visible immediately and subsequent heartbeats continue to update the restored entries.

Join Commands¶

Unsecured (local/dev)¶

curl -fsSL http://<leader>:8080/api/v1/bootstrap/fellow.sh | bash -s -- \
  --leader-url http://<leader>:8080 --dry-run

Remove --dry-run for a real install.

With a Join Session (recommended)¶

Create a session from the UI at /ops/join-v2 or via API:

curl -sX POST http://<leader>:8080/api/v1/bootstrap/join-sessions \
  -H "Content-Type: application/json" \
  -d '{"ttl_seconds": 3600}' | python3 -m json.tool

Use the returned session_id and token:

curl -fsSL http://<leader>:8080/api/v1/bootstrap/fellow.sh | bash -s -- \
  --leader-url http://<leader>:8080 \
  --session-id <SESSION_ID> \
  --token <TOKEN>

Token-Gated Mode¶

Set SWARM_BOOTSTRAP_REQUIRE_TOKEN=1 on the leader to require a valid token for all bootstrap requests. Without the env var the leader operates in local/dev mode (no token required).

Bootstrap Script Flags¶

Flag	Default	Description
`--leader-url URL`	(from script)	Leader base URL
`--fellow-id ID`	auto (hostname CRC32)	Explicit fellow ID
`--install-dir DIR`	`$HOME/.silent-swarm`	Installation root
`--session-id ID`	—	Join session ID for log callbacks
`--token TOKEN`	—	Session token for authenticated bootstrap
`--zmq-act-port PORT`	5557	ZMQ activation port
`--zmq-grad-port PORT`	5558	ZMQ gradient port
`--dry-run`	off	Print steps without installing
`--help`	—	Show help

Script Behavior¶

The script performs six steps on the remote host:

Fetch the bootstrap manifest from the leader.
Download the source archive from the leader.
Extract the archive into <install-dir>/app.
Create or reuse a Python virtual environment at <install-dir>/.venv.
Install dependencies via the virtualenv interpreter (python -m pip). Core dependencies are required; TensorFlow/Keras dependencies are best-effort on macOS. Core dependency installs retry up to 3 times.
Start python -m silent_swarm.fellow.runner in the background via nohup.

Additional behaviors:

Writes the process PID to <install-dir>/fellow.pid. Re-runs reuse the existing process if alive.
Writes runtime logs to <install-dir>/logs/fellow.out.
Writes runtime config to <install-dir>/fellow.env.
Installs a local helper command fellow (symlink at ~/.local/bin/fellow) for follow-up operations on the host.
Posts each log line back to the leader via the session log endpoint when --session-id is provided.
Updates the session status to completed or failed on exit.
The --token is passed as X-Bootstrap-Token header in all curl calls to the leader.
The shell wrapper prefers python3.13, python3.12, or python3.11 when present and falls back to any Python 3.11+ interpreter.
If the requested ZMQ ports are already occupied, the fellow runner automatically switches to the next free incremental pair in non-interactive bootstrap mode.
If python3 -m venv is unavailable, or it creates an environment without pip, the script tries ensurepip, then falls back to python3 -m virtualenv, and if needed tries python3 -m pip install --user virtualenv before failing.

Fellow Local CLI¶

After bootstrap, the host gets a local fellow command with a minimal operational UI:

fellow update – fetch latest bootstrap script from leader and re-install.
fellow jobs – show recent job-related events parsed from local fellow logs.
fellow logs --attach – attach/follow fellow.out (Ctrl+C to detach).

fellow update is interactive by default and asks:

Would you like to update <current_version> -> <leader_version> [y/N]

Use fellow update --yes for non-interactive execution.

Join Session Lifecycle¶

stateDiagram-v2
  [*] --> pending: session created
  pending --> active: first log line
  active --> completed: script exits 0
  active --> failed: script exits non-zero
  pending --> revoked: operator DELETE
  active --> revoked: operator DELETE
  pending --> expired: TTL reached
  active --> expired: TTL reached
  completed --> [*]
  failed --> [*]
  revoked --> [*]
  expired --> [*]

Remote Update via `swarm update`¶

The swarm update command pushes an UPDATE control signal to one or more connected fellows, instructing them to re-run the bootstrap install script from the leader.

Usage¶

# Update all connected fellows
swarm update

# Update a specific fellow
swarm update --fellow-id 7001

# Update multiple fellows
swarm update --fellow-id 7001 --fellow-id 7002

How It Works¶

The CLI posts to POST /api/v1/workers/update on the leader.
The leader broadcasts (or targets) a {"signal": "UPDATE", "payload": {"leader_url": "..."}} frame over the fellow's control WebSocket.

On receiving the signal, the fellow re-runs the bootstrap script:

curl -fsSL <leader_url>/api/v1/bootstrap/fellow.sh | bash -s -- --fellow-id <ID>

The bootstrap script replaces the installed source and restarts the runner process.

The signal is only processed when the fellow is in IDLE or PAUSED state. Fellows in TRAINING ignore the signal with a warning in their logs.

Response Shape¶

{"sent": [7001, 7002], "not_connected": [7003]}

sent — fellows that had an active WS connection and received the signal. not_connected — registered fellows that were not connected via WS at the time.

Security Notes¶

In local/dev mode (SWARM_BOOTSTRAP_REQUIRE_TOKEN unset) the bootstrap endpoints are openly accessible.
Set SWARM_BOOTSTRAP_REQUIRE_TOKEN=1 to require valid tokens in staging or shared environments.
Tokens are random 32-byte URL-safe strings generated with secrets.token_urlsafe.
Sessions expire automatically after the configured TTL (default 1 hour).
Expired and revoked tokens are rejected immediately.
Log endpoints and the PATCH endpoint do not require authentication (the caller needs to know the session ID).

Troubleshooting¶

python3 not found¶

Install Python 3.11+ via your OS package manager before running the script.

pip install fails¶

The script retries pip install up to 3 times with a 5-second delay. If it still fails, check network connectivity and PyPI reachability from the target host. Common causes: missing build tools (gcc, make) for compiled dependencies.

On macOS, TensorFlow/Keras installation is best-effort. If PyPI has no matching TensorFlow wheel for the selected Python version, bootstrap logs a warning and continues so the fellow can register with the leader. Training jobs on that host still need TensorFlow; install a compatible Python/TensorFlow pair and rerun the bootstrap command or fellow update --yes.

Virtual environment creation fails with `ensurepip is not available`¶

The bootstrap script first tries python3 -m venv. If that creates an environment without pip, it tries python -m ensurepip --upgrade, then falls back to python3 -m virtualenv, and finally tries to install virtualenv with python3 -m pip --user.

If you see No module named pip on the target host, that means the initial venv was created without bundled pip and all recovery paths failed.

If all fallbacks fail, install one of the OS packages below and rerun the bootstrap command:

Debian/Ubuntu: sudo apt install python3-venv
Alternative package on some distros: python3-virtualenv

Fellow process exits immediately¶

Check <install-dir>/logs/fellow.out for the error. Common causes: incorrect SWARM_LEADER_URL, leader not reachable, or ZMQ port conflict.

If the conflict is only on the default ports 5557/5558, the bootstrap path now auto-falls back to the next free pair, for example 5561/5562. You can still force explicit values with --zmq-act-port and --zmq-grad-port.

If the leader restarts and drops the signal WebSocket (for example close code 1012 service restart), the fellow now automatically reconnects with exponential backoff and re-registers on the leader.

Script hangs at step 2¶

The source archive is built on demand. On large repos this can take a few seconds. Ensure the leader host has sufficient disk space for a temporary in-memory gzip.

Session shows expired before bootstrap finishes¶

Increase ttl_seconds when creating the session. The default is 3600 s (1 hour).

Manifest Shape¶

{
  "format_version": "bootstrap-manifest-v1",
  "leader_url": "http://leader:8080",
  "leader_version": "0.1.0",
  "ready": true,
  "registered_workers": 0,
  "bootstrap_script_url": "http://leader:8080/api/v1/bootstrap/fellow.sh",
  "recommended_join_command": "curl -fsSL ... | bash -s -- --leader-url ...",
  "install": {
    "strategy": "leader_source_archive",
    "source_archive_url": "http://leader:8080/api/v1/bootstrap/source.tar.gz",
    ...
  },
  "script": { "endpoint": "...", "supported_flags": [...] }
}