Skip to content

Operations

This section documents runtime operations for SilentSwarm clusters, especially leader/fellow startup, remote onboarding, and troubleshooting.

flowchart LR
  Start[Start leader] --> Fellows[Attach fellows]
  Fellows --> Inspect[Inspect nodes and devices]
  Inspect --> Run[Run jobs or sweeps]
  Run --> Observe[Watch metrics and logs]
  Observe --> Manage[Update, release, or purge fellows]
  Manage --> Stop[Stop cluster]

Current Topics

  • Local and staging startup procedures.
  • Service health checks and observability.
  • Leader-served fellow onboarding through curl bootstrap.
  • Ops UI join flow and onboarding status.
  • Fellow release, update, and purge workflows.
  • Pipeline transport selection for tunnel and direct P2P modes.
  • Troubleshooting playbooks for bootstrap, leader connectivity, and port conflicts.

Documents

  • bootstrap-quickstart.md: common local, remote, token, and tunnel onboarding paths.
  • bootstrap.md: bootstrap manifest, shell contract, join flow, environment variables, and troubleshooting.

Local Cluster Lifecycle

swarm start-cluster --fellows 2
swarm nodes
swarm fellows
swarm stop-cluster

Local process logs are written under .swarm/runtime-logs/. Docker Compose users normally reach the leader through the nginx frontend at http://localhost:3000.

Docker and Tunnel Access

docker compose --profile local-fellow up -d --build
swarm --leader-url http://localhost:3000 nodes

For temporary public access to the Ops UI and bootstrap flow:

docker compose --profile local-fellow --profile tunnel up -d --build
docker compose --profile tunnel logs -f tunnel

The tunnel log prints a public /ops/join-v2 URL.

Remote Fellow Onboarding

Use the Ops UI at /ops/join-v2 or the bootstrap API documented in bootstrap.md. Join sessions can carry a token and stream logs back to the leader while the remote host installs and starts the fellow runner.

Runtime Management

  • swarm update [--fellow-id ID]: ask connected fellows to re-bootstrap from the leader.
  • swarm release --fellow-id ID: remove a fellow from the leader registry.
  • swarm release --fellow-id ID --purge-install: release and request local cleanup.
  • swarm pending-purges and swarm retry-purge: inspect and retry cleanup for fellows that were offline during purge delivery.
  • swarm watch <job_id>: stream live merged metrics from the leader.

Observability

  • Ops UI: /ops/join-v2 is a read-only join and monitoring surface. Selecting a fellow in the graph or table opens its runtime logs automatically and shows status, network speed bars, step/loss, VRAM used/free, and per-GPU telemetry or active GPU layer placement. Selecting the leader shows cluster performance, including total RAM, CPU cores, aggregate VRAM, network throughput, observed step time, steps/minute, and the active placement plan.
  • Worker logs: GET /api/v1/workers/{fellow_id}/logs/stream; the Ops UI hydrates logs from the REST snapshot first and keeps polling as a tunnel-safe fallback.
  • Fellow progress logs: each training progress post is logged at info level with job ID, fellow ID, step, loss, status code, HTTP latency, request/response bytes, and cumulative fellow byte counters. Fellows also update their local step/loss immediately so heartbeat metrics and the UI do not wait for completion.
  • Computation placement: leader dispatch ranks connected fellows by aggregate free GPU VRAM across eligible GPUs on each node. GPUs with less than placement_min_gpu_free_gb free memory are ignored; the default is 4.0 GB and can be set globally with SWARM_PLACEMENT_MIN_GPU_FREE_GB or per job config. With --partition weighted, node layer counts are weighted by aggregate eligible free VRAM, and each node placement contains a gpu_slots plan that splits the node's layer range across eligible GPUs by that GPU's currently free VRAM. Fellows make every active slot visible to TensorFlow and execute each local transformer layer inside the assigned GPU device scope, so the layer split is an actual intra-node multi-GPU execution plan rather than UI metadata. The final pipeline stage also places final_norm, the output head, loss, and gradient workspace on the local slot with the most free VRAM; it can still need more memory than its layer count alone suggests.
  • Pipeline stage ordering: by default, SWARM_PIPELINE_LAST_STAGE_PLACEMENT=largest_vram moves the strongest selected fellow to the final pipeline stage, because that stage owns final_norm, the output head, loss, and gradient workspace. Set pipeline_last_stage_placement=preserve in a job config to keep the raw placement order for diagnostics.
  • GPU runtime check: Linux fellows that should train on NVIDIA GPUs need the TensorFlow CUDA extras from tensorflow[and-cuda]. A fellow can still register GPU hardware from nvidia-smi while TensorFlow falls back to CPU if CUDA/cuDNN wheels are missing; check the fellow log for Created device ... GPU and avoid Skipping registering GPU devices before interpreting a stress run.
  • GPU memory cleanup: fellows start TensorFlow with TF_GPU_ALLOCATOR=cuda_malloc_async and TF_FORCE_GPU_ALLOW_GROWTH=true, then clear the Keras/TensorFlow runtime after every completed, failed, or cancelled training job. This prevents idle fellows from keeping large model graphs alive after a job ends. If a deployment is supervised by K8s, systemd, or another restart manager and must guarantee immediate CUDA context release, set SWARM_FELLOW_EXIT_AFTER_TRAINING=1 so the fellow exits after posting the final job status and is restarted by the supervisor.
  • Live VRAM in the Ops UI comes from heartbeat hardware telemetry. Registration-time GPU inventory is only the initial snapshot; while fellows are running, each heartbeat refreshes per-GPU free VRAM so node bars and GPU placement rows show current used/total memory.
  • Bootstrap logs: GET /api/v1/bootstrap/join-sessions/{id}/logs/stream.
  • Job metrics: GET /api/v1/jobs/{job_id}/metrics/stream.
  • Job loss history: GET /api/v1/jobs/{job_id}/metrics/history; the Ops UI uses this stored curve when a job is selected or the page is reloaded, and polls it as a fallback while live metrics are streaming.
  • Prometheus text metrics: GET /api/v1/metrics.

Transport Modes

  • SWARM_PIPELINE_TRANSPORT=auto: use direct P2P when ports are reachable and leader relay as fallback.
  • SWARM_PIPELINE_TRANSPORT=tunnel: force the leader relay.
  • SWARM_PIPELINE_TRANSPORT=p2p: force direct fellow-to-fellow links.

Useful Commands

swarm info
swarm start-cluster --fellows 2
swarm nodes
swarm fellows
swarm jobs
swarm watch <job_id>
swarm --leader-url http://localhost:3000 sweep --config exp/configs/exp_large_unlimited_compute_v1.json --partition weighted
swarm --leader-url http://localhost:3000 sweep --config exp/configs/exp_full_sweep_unlimited_v1.json
python exp/scripts/run_benchmark_suite.py --suite exp/configs/benchmarks/multinode_training_v1.json --leader-url http://localhost:3000 --dry-run
swarm update --fellow-id <id>
swarm release --fellow-id <id>
swarm stop-cluster

Benchmark Operations

Run benchmark suites only when the cluster is otherwise idle, because each scenario measures the whole placement and network path. Start with --dry-run to verify that the connected fellows satisfy the infra profile's fellow count, GPU count, and minimum free VRAM requirements. The runner stores artifacts below exp/runs/benchmarks/{suite_run_id}/.

The single_gpu_baseline and two_gpu_one_node scenarios use the job field max_gpus_per_worker so dispatch limits the number of eligible GPU slots on the selected fellow. Two-node scenarios leave that field unset so all eligible GPUs on each selected fellow can participate. The constrained-network scenario records its target profile with enforcement=external; apply real inter-node shaping outside SilentSwarm before running it if you need hard bandwidth or latency limits.

Distributed benchmark jobs export trained shard weights when the job payload contains artifact_paths.model_export_dir. The export keeps the training split visible in model_export/run_manifest.json and writes one weights.npz per stage/fellow. Jobs can also set shard_push_interval_steps to make each fellow export and upload its latest shard to the leader every N training steps. The default benchmark stress profile enables this every 20 steps with shard_push_to_leader=true.

The leader stores pushed shards in the job's artifact_paths.model_export_dir when that path is present, or below .swarm/model_exports/{job_id}/model_export otherwise. Each upload refreshes model_export/run_manifest.json and appends a receipt to job.config.latest_shard_pushes. This does not yet perform automatic resume, but it gives the leader the latest stage-local weights needed for a future K8s replacement fellow to reload a shard.

After a run, glue the shards into a single-device inference model:

python exp/scripts/glue_model_export.py \
  --export-dir exp/runs/benchmarks/<suite_run_id>/<scenario_id>/repeat-1/model_export

The command validates a dummy forward pass before writing model_export/full_model/inference_manifest.json. If full Keras serialization is not available for the local custom model class, it falls back to saving model.weights.h5 and records that in the inference manifest.

  • scripts/
  • docker/
  • src/silent_swarm/bootstrap/
  • src/silent_swarm/leader/
  • src/silent_swarm/fellow/
  • tests/