Campaign Engine

dispatchatlas.lab owns reproducible campaign orchestration across benchmark providers and solver registries. It validates campaign configuration, estimates run cost, captures the runtime environment, persists checkpoints, classifies failures, and resumes incomplete runs without duplicating completed work.

Runnable examples: examples/run_experiment.py drives a smoke pilot, a stopping-rule ablation, and a seed-sensitivity sweep; examples/inspect_engine.py inspects host-aware worker sizing and the composable termination criteria.

Configuration

Campaigns declare benchmark selectors, solver ids, objectives, seed policy, stop criteria, disclosure labels, resource budgets, retry policy, execution mode, and an output repository.

from dispatchatlas.core import DisclosureLabel, TerminationPolicy
from dispatchatlas.lab import (
    CampaignConfig,
    ExecutionMode,
    OutputPolicy,
    ResourceBudget,
)
 
config = CampaignConfig(
    campaign_id="smoke-campaign",
    benchmark_ids=("dispatchatlas-smoke",),
    solver_ids=("earliest-start", "ndso-core"),
    objectives=("makespan",),
    root_seed=20260527,
    seed_namespace="docs.campaign.smoke",
    stop=TerminationPolicy(max_iterations=5),
    output=OutputPolicy("experiments"),
    resources=ResourceBudget(max_workers=2, max_concurrent_runs=2),
    execution_mode=ExecutionMode.BOUNDED,
    disclosure_labels=(DisclosureLabel.PUBLIC,),
)

OutputPolicy.root_dir is the experiments workspace root, not a per-campaign directory: every run lands under results/{campaign}/{solver}/{benchmark}/ inside that root.

Validation expands the selected benchmark problems and solver ids into deterministic run ids. Full-campaign plans can be dry-run for cost estimates, but execution stays blocked until statistical design approval is recorded. Seeded stochastic solvers can declare explicit per-solver seed replicates via solver_seed_replicates, while deterministic solvers stay at one recorded seed per problem and objective.

Execution

Use CampaignRunner with a benchmark provider, solver registry, and FileResultRepository. The default runner wires the bundled smoke benchmark provider and solver registry.

from pathlib import Path
 
from dispatchatlas.lab import default_campaign_runner
 
runner = default_campaign_runner(Path("experiments"))
plan = runner.validate(config)
budget = runner.dry_run(plan)
index = runner.run(plan)

Execution modes:

Campaign Kinds And Run-Count Policy

A campaign declares a kind. The catalog table below is generated from the experiment-design taxonomy, so its total is countable from the rows themselves.

The run-count policy enforces a statistical-power floor of at least thirty independent runs per (stochastic-solver, instance) cell at the pilot and full stages, following established guidance on the sample sizes needed for reliable comparison of randomized algorithms (Arcuri & Briand 2014); deterministic solvers run once. A smoke campaign flags a sub-floor count rather than rejecting it, so quick checks stay cheap without silently shipping an under-powered design.

Dual Stopping Protocols

A campaign reports each stochastic-solver result under both a fixed-budget protocol (iterations or wall-time) and a fixed-target protocol (terminate on a target objective). Declare both on one campaign through stopping_protocols; every cell is planned under each protocol while sharing one fixed seed, so the two reports are directly comparable. Each completed run records its protocol in its diagnostics under the stopping_protocol key.

from dispatchatlas.core import TerminationPolicy
from dispatchatlas.lab import StoppingProtocol, StoppingProtocolKind
 
protocols = (
    StoppingProtocol(
        name="fixed-budget",
        kind=StoppingProtocolKind.FIXED_BUDGET,
        stop=TerminationPolicy(max_iterations=200),
    ),
    StoppingProtocol(
        name="fixed-target",
        kind=StoppingProtocolKind.FIXED_TARGET,
        stop=TerminationPolicy(max_iterations=2000, target_objective=100.0),
    ),
)

Fair Comparison

A comparative campaign exercises every solver under identical termination, equal computational budgets, and one equal per-algorithm TuningBudget, with per-cell fixed seeds logged in the run manifest. Equalizing the tuning budget across solvers follows established benchmarking practice, which holds that unequal tuning effort confounds an otherwise fair comparison (Bartz-Beielstein et al. 2020). The campaign fails closed unless it compares at least two solvers and declares a tuning budget; the guarantee is recorded in the plan metadata under the fair_comparison* keys.

Safe-Max-Worker Topology

probe_capacity selects the largest worker count that stays inside the resource budget, caps each exact solver's internal thread pool so workers times threads never oversubscribes the host, computes the linear-algebra thread-pool pins that the process backend's worker initializer applies to prevent nested pools, and falls back to one deterministic worker for sequential and replay modes. Each run is owned by exactly one worker and persisted to its own run-id-keyed record, so aggregation is merge-only: merge_only_aggregation sorts runs by id and hashes their content-hash strings, producing a campaign hash that is bit-stable regardless of worker completion order.

Command-Line Interface

The dispatchatlas-lab command validates, costs, runs, resumes, and replays a campaign declared as a JSON configuration file:

dispatchatlas-lab validate --config examples/campaign-config.json
dispatchatlas-lab dry-run --config examples/campaign-config.json
dispatchatlas-lab run --config examples/campaign-config.json
dispatchatlas-lab resume --config examples/campaign-config.json
dispatchatlas-lab replay --config examples/campaign-config.json

Checkpoints And Results

The file-backed repository treats OutputPolicy.root_dir as the experiments workspace root and writes the hierarchical run-level layout:

• results/{campaign_id}/plan.json and results/{campaign_id}/environment.json
• results/{campaign_id}/{solver_id}/{benchmark_id}/run_{idx}.json — one record per completed run, where idx is the zero-based replicate index assigned from the run plan
• results/{campaign_id}/{solver_id}/{benchmark_id}/failures/run_{idx}.attempt-{N}.json — one record per failed attempt
• .checkpoints/{campaign_id}.json — the campaign checkpoint
• logs/{campaign_id}_{timestamp}.log — the per-campaign execution log
• ENVIRONMENT.md — a workspace-level Markdown snapshot of the execution environment (host, OS, Python, repository commit and branch, capture time, CPU count, machine), overwritten on every campaign initialization

Each run record is content-hashed over its deterministic scientific payload; wall-clock timing and resource measurements stored on the record are provenance and stay outside the hash, so replay verification is unaffected by run-to-run timing variation. Checkpoints list completed, failed, and pending run ids so a later resume() call skips completed work. Parallel campaigns persist completed result records as each worker finishes and write the final checkpoint at campaign end; resume still discovers completed records from disk before scheduling remaining work.

Environment Capture

Campaign environment stamps include operating system, Python version, platform tag, package versions, CPU count, machine/processor facts, current Git commit, configuration hash, and seed policy. Hostnames, credentials, and environment variables are not captured.

Failure Handling

Configuration, domain, unsupported-capability, and missing optional dependency errors are terminal. Other runtime exceptions are recoverable until the retry budget is exhausted. Every failed attempt is persisted before retry or resume.

Evidence Builder

Use tools/build_campaign_evidence.py to build candidate benchmark catalogs, full comparative campaigns, NDSO ablations, sensitivity checks, evidence bundles, and portal datasets from one recorded configuration:

uv run python tools/build_campaign_evidence.py `
  --output-root .\experiments `
  --problem-count-per-profile 30 `
  --stochastic-seeds 10 `
  --campaign-suffix local

The builder emits phase progress to stderr for catalog materialization, campaign planning, execution, analysis, bundle export, and report writing. The builder intentionally writes generated evidence outside Git-tracked source trees. Promote its outputs only through the release and disclosure gates.