Skip to content

Experiments

This section documents the current SilentSwarm experiment workflow: active configuration files, cluster-backed execution, queued sweeps, and generated run artifacts.

Execution Model

  • Experiments must run against an active leader.
  • At least one fellow must be connected before execution starts.
  • Local standalone experiment execution is intentionally blocked.
  • If cluster_profile is defined, preflight validates requested topology against active fellows before training.
  • Run artifacts and reports include requested-versus-actual topology snapshots for analysis.
  • run_experiment.py uses sweep.dataset, sweep.tokenizer, and sweep.real_min_tokens from the config (e.g. wikitext103 runs).
flowchart LR
    Config[Experiment config] --> Loader[Config loader]
    Loader --> Grid[Expanded model profile runs]
    Grid --> Queue[Leader job queue]
    Queue --> Dispatch[Dispatcher waits for ready fellows]
    Dispatch --> Run[Sequential distributed training]
    Run --> Artifacts[Run artifacts in exp/runs]

Benchmark suites add one orchestration layer above this workflow:

flowchart TB
    Suite[Benchmark suite] --> Scenario[Scenario]
    Scenario --> Model[Model profile]
    Scenario --> Training[Training profile]
    Scenario --> Infra[Infra profile]
    Model --> Payload[Leader job payload]
    Training --> Payload
    Infra --> Payload
    Payload --> Dispatch[Leader placement]
    Dispatch --> Artifacts[Scenario artifacts]
    Dispatch --> Placement[placement_plan.json]
    Artifacts --> Manifest[model_export/run_manifest.json]

Active Config Layout

Top-level experiment files live in exp/configs/ and reference reusable model and training profiles:

  • exp_distributed_smoke_v1.json: short validation run for a connected cluster.
  • exp_full_sweep_cluster_network_v1.json: paper-oriented model/network sweep.
  • model/: model and compression profiles such as small_none.json, large_fixed_int8.json, and xlarge_learned_bottleneck.json.
  • training/: dataset, threshold, cluster, and network profiles.

The loader resolves training_config and model_configs into concrete job payloads before submission.

CLI-Only Sweep Workflow

# Optional: print human and LLM-oriented CLI guidance
swarm info
swarm llm-doc --output docs/llm_swarm_cli.md
swarm restart-cluster --fellows 2

# 1) Inspect / release fellows
swarm --leader-url http://127.0.0.1:8080 fellows --format json
swarm --leader-url http://127.0.0.1:8080 release --all

# 2) Attach fellows and apply network limits when needed
swarm --leader-url http://127.0.0.1:8080 set-speed --fellow-id 0 --tx-mbps 80 --rx-mbps 120
swarm --leader-url http://127.0.0.1:8080 set-speed --fellow-id 1 --tx-mbps 80 --rx-mbps 120

# 3) Run model profile sweep as sequential queue of jobs
swarm --leader-url http://127.0.0.1:8080 sweep --config exp/configs/exp_full_sweep_cluster_network_v1.json

The sweep command resolves model_configs and training_config, enqueues all expanded runs first, and then monitors terminal state for sequential execution.

Queue behavior (current):

  • swarm sweep enqueues all expanded model profile runs up front.
  • The leader executes queued jobs one-by-one (PENDING -> RUNNING -> terminal).
  • CLI output now includes periodic progress lines with current status and step per queued run.
  • Use swarm watch <job_id> to stream live per-fellow metrics for one job.

Dispatcher behavior:

  • A PENDING job is auto-transitioned to RUNNING once enough ready+connected fellows are present.
  • swarm nodes TX/RX counters reflect fellow-reported control-plane bytes (register/heartbeat/control-signal traffic).
  • On START, each fellow enters an async training loop and updates current_step/last_loss through heartbeats.
  • Repeated RUNNING status updates to /api/v1/jobs/{job_id}/status update last_step instead of being ignored.

Python Runner Workflow

exp/scripts/run_experiment.py runs the same cluster-backed experiment pipeline from Python and writes local artifacts. It requires a reachable leader and at least one active fellow.

python exp/scripts/run_experiment.py \
    --leader-url http://127.0.0.1:8080 \
    --config-path exp/configs/exp_full_sweep_cluster_network_v1.json

If network_profile.tc_profile is set in the resolved config, the runner applies exp/scripts/setup_netem.sh before training and calls exp/scripts/cleanup_netem.sh afterward. Use --skip-netem to disable that behavior for environments where tc is managed externally.

Benchmark Suite Workflow

Use benchmark suites when comparing cluster shapes rather than only model or compression variants. The default suite is exp/configs/benchmarks/multinode_training_v1.json and compares:

  • single GPU baseline,
  • two GPUs on one node,
  • two unconstrained nodes,
  • two nodes with a constrained-network profile.

Plan the suite first:

python exp/scripts/run_benchmark_suite.py \
    --suite exp/configs/benchmarks/multinode_training_v1.json \
    --leader-url http://localhost:3000 \
    --dry-run

Then execute it:

python exp/scripts/run_benchmark_suite.py \
    --suite exp/configs/benchmarks/multinode_training_v1.json \
    --leader-url http://localhost:3000 \
    --continue-on-fail

Each scenario defines model, training, infra, and artifact paths explicitly. The runner writes suite_plan.json, suite_manifest.json, per-scenario manifests, resolved configs, infra profiles, final job snapshots, and placement_plan.json after leader dispatch. Distributed fellows also export their trained local weights into model_export/shards/ when artifact_paths.model_export_dir is present in the job payload. If the training profile sets artifacts.shard_push_interval_steps, fellows refresh their local shard every N training steps and push the latest shard_manifest.json plus weights.npz back to the leader. This keeps the leader-side export directory current during the run instead of only after completion.

The model export is intentionally two-step:

  1. Training exports sharded weights plus manifests that preserve the exact fellow, GPU slot, and global layer placement.
  2. exp/scripts/glue_model_export.py reconstructs a full ExperimentModel from those shards and writes a single-device inference artifact under model_export/full_model/.
python exp/scripts/glue_model_export.py \
  --export-dir exp/runs/benchmarks/<suite_run_id>/<scenario_id>/repeat-1/model_export

This keeps multi-GPU training reproducible while still producing an artifact that can run inference on one GPU or CPU.

Run Artifacts

Each execution writes artifacts under exp/runs/<run_id>/:

File Purpose
manifest.json Run metadata, config source, leader URL, status, and commit when available.
config_resolved.json Concrete model, training, runtime, and selected profile data.
topology_snapshot.json Requested cluster profile, active fellows at start, and preflight result.
train_progress.ndjson Step-level progress and loss records.
metrics_summary.json Aggregated final loss, best loss, timing, throughput, and threshold status.

Benchmark suite runs additionally write suite_plan.json, suite_summary.json, infra_resolved.json, job_final.json, placement_plan.json, model_export/run_manifest.json, model_export/shards/*/shard_manifest.json, model_export/shards/*/weights.npz, and, after the glue step, model_export/full_model/inference_manifest.json.

Periodic shard pushes update the same model_export/shards/stage-*-fellow-*/ paths on the leader. The leader records recent push receipts in the job config under latest_shard_pushes, which is the operational hand-off point for future worker replacement and resume flows.

Paper-facing exports are curated under silentSwarmPaper/data/ and generated by the paper scripts, not by the live experiment runner.

Cluster Preflight

When a config includes cluster_profile, the runner compares the requested topology with active fellows returned by the leader. Count mismatches or exact placement mismatches fail before training starts. The requested profile and the actual topology are preserved in the run artifacts so results remain traceable.

Troubleshooting

  • If a sweep waits at PENDING, check that enough fellows are connected and ready.
  • If a job stays at RUNNING with last_step=0, release stale fellow IDs and restart the active fellows before retrying.
  • Use swarm jobs, swarm watch <job_id>, and the Ops UI loss curve to inspect progress.
  • Use swarm fellows --format json to confirm the leader sees the expected worker set.

Planning Context

Design notes for future topology-aware experiment config evolution are kept in the project's internal planning notes.

  • exp/configs/
  • exp/configs/model/
  • exp/configs/training/
  • exp/configs/infra/
  • exp/configs/benchmarks/
  • exp/scripts/
  • exp/runs/