Analysis Exports

dispatchatlas.analytica turns completed campaign repositories into statistical summaries, deterministic figures, evidence bundles, and portal-ready datasets. The package consumes stable JSON manifests and does not import the campaign runtime.

Runnable example: examples/analyze_results.py summarizes a result set with non-parametric statistics, effect sizes, confidence intervals, deterministic tables, and dual-format figures.

📥 Inputs

The analysis loader expects one campaign directory containing:

• checkpoint.json with completed, failed, and pending run ids.
• plan.json with run order and disclosure labels.
• environment.json with non-secret environment facts.
• results/*.json completed run rows.
• failures/*.json failed attempt rows when present.

Missing checkpoints, content hashes, objective values, or disclosure labels fail ingestion before analysis starts.

📊 Statistical Methods

A named-method registry declares every analysis method with its data assumptions, minimum-sample rule, failure mode, and lowest evidence tier. Each inferential method and quality indicator requires at least 30 independent runs per stochastic solver-instance cell; a sub-floor sample routes to the limitations surface rather than producing an invalid result.

For each objective, the summary layer reports solver counts, feasible counts, mean, median, standard deviation, bias-corrected and accelerated (BCa) bootstrap confidence intervals, the Wilcoxon signed-rank test with the Vargha-Delaney A12 and Cliff's delta effect sizes for pairwise comparison, the Friedman omnibus test with a Nemenyi critical-difference post-hoc over the multi-solver field, Holm or Hochberg correction over the pairwise family, and average ranks by benchmark problem. Multi-objective fronts additionally report the hypervolume (primary), IGD+, additive epsilon-indicator, and spread quality indicators.

The default objective is makespan with minimization semantics. The Wilcoxon test uses the exact null distribution for small tie-free samples and the tie-corrected normal approximation otherwise. Confidence intervals use deterministic bootstrap resampling with a recorded seed; they describe the observed campaign sample and are not full-campaign claims by themselves.

Beside the closed-form Bayesian sign test, bayesian_signed_rank_test draws a seeded Dirichlet-process posterior over the Walsh averages of the paired differences, so continuous metrics get the magnitude-aware Bayesian comparison of Benavoli, Corani, Demšar and Zaffalon (2017); the sign test stays the closed-form default. The multiple-comparison family adds Shaffer's static step-down procedure, which exploits the logical constraints of a complete all-pairs comparison family and fails closed on a partial one (Shaffer 1986), and Finner's step-down adjustment, a simpler power gain that accepts any family (Finner 1993); Holm remains the configured default. Relative percentage deviation joins the optimality-gap methods: arpd_rows scores every feasible run against a caller-supplied best-known reference value with explicit provenance, and ensure_equal_budgets verifies the size-scaled equal-budget protocol behind it — one identical budget per problem across solvers, budgets free to grow with instance size (Vallada, Ruiz and Framiñan 2015). Multi-objective hypervolume comparisons archive their shared reference point: multiobjective_indicator_report derives the point under a named rule — the margin-fraction default or the ishibuchi-h scaling — records the ideal, nadir, rule, parameter, and a fail-closed provenance vocabulary in a ReferencePointRecord, and ensure_shared_reference_points refuses to compare artifacts whose archived points differ (Ishibuchi, Imada, Setoguchi and Nojima 2018; the exact published offset stays marked APPROXIMATE pending the operator-reviewed full text). One hard rule guards every exported table: a p-value never ships without an effect size and an interval beside it. ensure_effect_sizes_beside_p_values checks the headers inside the table builders and again at bundle-write time, failing closed on any bare significance column (Carrasco, García, Rueda, Das and Herrera 2020).

🖼️ Figures And Tables

Evidence bundles include Markdown tables for solver summaries, pairwise comparisons, rankings, benchmark coverage, and infeasible rows. SVG figures carry accessible title and desc metadata for ranking bars, performance profiles, run-order objective trends, the Nemenyi critical-difference diagram, runtime-quality trade-offs, convergence trajectories and the convergence-variability band, the exploration-exploitation balance, the diversity-objective phase portrait, the global-versus-population-best (gbest/lbest) curve, runtime scalability by instance size, seed stability (coefficient of variation), reliability (feasible-run fraction), adaptability (cross-instance consistency), robustness (CVaR tail risk), the objective-distribution box-and-whisker, multi-objective quality indicators, the two-objective frontier, the search-space scatter, the oblique 3D search-space scatter, centroid trajectory, and animated population flipbook, and benchmark characterization. A figure manifest records each figure's role, provenance, and accessibility note. Figures that depend on diagnostics a campaign did not record (runtime, convergence trace, population diversity, the optional per-iteration population snapshot, instance scale) or on multiple seeds (seed stability) render a limitations notice. A supplement/limitations.md collects every method routed to limitations.

Re-running the same export against the same inputs produces the same JSON, table, and figure payloads.

The Dolan-Moré performance profile travels with its Moré-Wild companion: data_profile_svg and data_profile_tex plot the share of (solver, problem) cells solved to a target accuracy within a budget of evaluation groups — the evaluation-group adaptation of the simplex-gradient unit — from the same convergence traces the convergence figures consume, and a dataset without traces routes to the limitations figure. Performance profiles that compare more than two solvers carry an interpretation caveat in both output forms: a profile ranks each solver only against the per-problem best, so the relative order of non-best solvers is not implied (Gould and Scott 2016).

⚖️ Platform Comparison

The platform-comparison exports render rival-framework capability rows, a feature-richness coverage matrix, a distribution-distance bridge assessment, and artifact-inspection rows. Every row carries an evidence-source label (documented, measured, artifact-inspected, vendor-claim, or unsupported), and any unsupported row routes to a limitations-and-future-work surface. The export carries a standing guard line: the tables compare platform capabilities, features, and artifacts, and are not direct performance claims — performance evidence lives in the statistical analysis exports.

🥊 Head-To-Head Comparison Lane

The head-to-head lane compares one focal solver against a named baseline cohort using the same named non-parametric tests, orients every row toward the focal solver, and gates a beats-cohort claim on the focal solver holding the best average rank and winning every pairwise comparison at the corrected significance level with the effect size favouring it. Algorithmic-complexity and construct/internal/external threats-to-validity table builders accompany the lane for report consumption.

🔒 Disclosure Policy

Exports are filtered through executable disclosure policies:

• core permits core/public evidence only.
• speed permits core, speed, and public evidence.
• quality permits core, speed, quality, and public evidence.
• platform permits platform evidence while still blocking redacted source and stale-narrative terms.

Forbidden rows are excluded from evidence and portal datasets and written to a supplementary exclusions manifest with the reason.

🗂️ Bundle Layout

evidence-bundle/
  bundle.json
  tables/
    solver-summary.md
    pairwise-comparisons.md
    rankings.md
    benchmarks.md
    infeasible-rows.md
  figures/
    performance-profile.svg
    solver-rankings.svg
    run-order-trend.svg
    critical-difference.svg
    runtime-quality.svg
    convergence.svg
    convergence-band.svg
    characterization.svg
    exploration-exploitation.svg
    diversity-fitness-portrait.svg
    gbest-lbest.svg
    scalability.svg
    stability.svg
    reliability.svg
    adaptability.svg
    robustness.svg
    distribution.svg
    multiobjective-indicators.svg
    frontier.svg
    search-space-scatter.svg
    search-space-3d-scatter.svg
    search-space-trajectory.svg
    search-space-animation.svg
    figure-manifest.json
  supplement/
    exclusions.json
    limitations.md

bundle.json lists every generated file and hashes the canonical manifest payload, so every table, figure, and supplement stays inside one tracked, content-addressed manifest.

⌨️ Command

uv run dispatchatlas export `
  --campaign-dir .\experiments\results\smoke-pilot `
  --target-dir .\exports\smoke-pilot `
  --authorized-output-root .\exports `
  --tier core `
  --objective makespan `
  --portal

The command writes an evidence bundle plus an optional portal dataset. The evidence bundle includes tables, figures, and supplementary exclusions. The portal JSON includes searchable tags, disclosure labels, objective values, run ids, and hash-linked campaign metadata. --target-dir must resolve inside --authorized-output-root; the default authorized root is the current working directory.

Run uv run dispatchatlas export --help for option descriptions and examples. Runtime input errors print a recovery message by default; pass --debug before the subcommand when a Python traceback is needed for development.

📝 Report scaffolds

The analysis layer also turns a disclosure-filtered evidence bundle into a deterministic report scaffold keyed by evidence tier. A ReportScaffoldSpec declares the contribution claim, design rationale, and the claim gates; write_report_scaffold writes the evidence bundle plus a comprehensive section plan, a contribution claim, a claim-gate ledger, and a redaction report under report/. Every section anchors to a filtered evidence artifact, and the generated scaffold is scanned so a later-tier mechanism or a restricted internal marker fails the write closed.

Claim gates are fail-closed: a design-level claim opens unconditionally, while a comparative claim stays closed until a head-to-head comparison beats the cohort with every paired count at or above the statistical-power floor. A contribution_partition over the evidence tiers proves a zero-overlap contribution per tier, and redaction_report accepts a caller-supplied register so a non-public draft can be checked without that register ever entering shipped source.

uv run dispatchatlas report `
  --campaign-dir .\experiments\results\smoke-pilot `
  --target-dir .\scaffolds\smoke-pilot `
  --authorized-output-root .\scaffolds `
  --tier core