Reproducible Campaigns

The experiments/ workspace holds everything a campaign produces and everything needed to reproduce it. The split is strict: recipes are tracked, outputs are local. Campaign configurations and runner scripts live in source control; every heavy output — generated benchmarks, run records, logs, checkpoints, evidence bundles, and the environment snapshot — stays on the machine that produced it and is regenerated on demand from the recipe.

🗂️ Layout

Output directories exist locally and are recreated automatically by the campaign runner when absent; nothing under them is ever staged.

🧱 Two Layers, One Source Of Truth

1. Reproducibility layer (granular). Every run is one JSON record at results/{campaign_id}/{solver_id}/{benchmark_id}/run_{idx}.json. The path itself is the index — campaign, then solver, then benchmark, then replicate — so any slice of the evidence is addressable without a database. Records carry first-class timing, convergence, solution, and solver-effort fields; nothing analytical hides in free-form notes.
2. Curation layer (tiered). evidence-bundles/{campaign_id}-{tier}/ holds the disclosure-tiered exports derived from the run records. Tier ids come from the fixed public-safe vocabulary — core, speed, quality, platform — and a bundle is always derived from the run-level records, never an independent source of truth.

🔁 Reproducibility Contract

• Every campaign declares one root seed; each run's seed derives deterministically from it and the run's position — same configuration, same seeds, same records, bit for bit.
• .checkpoints/{campaign_id}.json records the configuration hash. An unchanged configuration resumes from the checkpoint and skips completed runs; any configuration change invalidates the checkpoint outright.
• ENVIRONMENT.md snapshots host, operating system, Python, and the repository commit at execution time, so every result is traceable to the exact code and machine that produced it.
• Logs are append-only per execution.

These practices — per-run seed, configuration, and environment capture so a result can be regenerated from its recipe — follow recognized standards for documenting and reproducing computational experiments (ACM Artifact Review and Badging, Version 1.1, 2020; Gundersen & Kjensmo 2018).

The reproducibility manifest also justifies the 30-run floor with a derived power statement: at the floor, with a two-sided 0.05 level and 0.80 power, the paired design detects standardized effects of about 0.523 or larger under the ARE-adjusted Wilcoxon approximation (Noether 1987; framework: Campelo and Takahashi 2019). The sentence is recomputed from the floor at render time, so it can never drift from the floor it justifies. Two further protocol descriptors travel beside it: a budget-scaling entry recording how the evaluation budget grows with instance size (the size-scaled protocol of Vallada, Ruiz and Framiñan 2015, recorded by the lab fair-comparison manifest as an evaluation-scaled rule or as campaign-fixed), and a tuning-parity entry carrying the equal per-algorithm tuning budget every comparative campaign declares (LaTorre, Molina, Osaba, Poyatos, Del Ser and Herrera 2021); an undeclared parity renders honestly as undeclared.

🏃 How A Campaign Writes Into The Workspace

1. Author or adjust a campaign configuration under configs/ (smoke-pilot.json is the tracked starting point).

2. Run it through the campaign runner — programmatically as in the tutorials, via a tracked recipe script (uv run python experiments/scripts/run_smoke_pilot.py), or through the command-line interface:

   uv run dispatchatlas-lab run --config experiments/configs/smoke-pilot.json

3. The runner validates the configuration into a deterministic plan, writes results/{campaign_id}/plan.json and environment.json, then persists one content-hashed record per completed run plus any failed attempts.

4. Inspect run records under results/ and resume, replay, or re-run from the same configuration. Re-running the same configuration reproduces the same records bit for bit; a changed configuration invalidates the checkpoint rather than silently reusing it.

The campaign engine page documents validation, dry-run budgets, execution modes, retries, resume, and replay verification in full.

📦 From Run Records To Evidence Bundles

The export path turns a completed campaign directory into a curated bundle under evidence-bundles/{campaign_id}-{tier}/:

uv run dispatchatlas export `
  --campaign-dir .\experiments\results\smoke-pilot `
  --target-dir .\experiments `
  --authorized-output-root .\experiments `
  --tier core

tools/build_campaign_evidence.py drives the same path at scale, building benchmark catalogs, campaigns, analyses, and bundles for every tier from one recorded configuration. The evidence bundles page describes the tiers; analysis exports describes the statistical content.

🔒 Public-Safe Boundary

Public pages never read this workspace directly: the documentation site and the results portal consume only committed, disclosure-filtered exports produced from the curated layer. Private program ledgers in the workspace are ignored by source control and never reach a public surface.