Skip to content

Analysis

Analysis datasets and settings, the statistical comparison of solvers, deviations and quality indicators, the method catalog, tables, the manifest, and the bundle.

dispatchatlas.analysis

Statistical analysis of campaign results.

Read a campaign directory, compare its solvers, and write every table, figure, and caveat to one reproducible bundle:

from dispatchatlas.analysis import AnalysisConfig, load_dataset, summarize, write_bundle

dataset = load_dataset("results/my-campaign")
summary = summarize(dataset, AnalysisConfig(correction="holm"))
write_bundle("results/my-campaign", "analysis/my-campaign")

The modules, from data to reports:

  • dataset: the planned runs of a campaign and what became of each; a run without a feasible result counts as a failure.
  • config: the settings of an analysis.
  • summary, comparison: ranks, omnibus and post-hoc tests, and paired comparisons with effect sizes and intervals; one solver against baselines.
  • deviation, indicators: relative percentage deviation and multi-objective quality indicators.
  • stats: the statistical kernels.
  • methods: the catalog of implemented methods.
  • tables, manifest, bundle: tables, the reproducibility manifest, and the bundle.

dispatchatlas.analysis.dataset

Campaign results as analysis data.

load_dataset() reads a campaign directory written by dispatchatlas.campaigns.run() through the readers of dispatchatlas.campaigns.store, and returns a Dataset that holds one Run for every run of the plan, in plan order, whether or not the run completed.

Every planned run has one of four statuses:

  • "succeeded": the run completed with a feasible schedule;
  • "infeasible": the run completed, but its schedule breaks a constraint, such as a hard deadline, or the solver returned no schedule;
  • "failed": every attempt of the run failed;
  • "missing": the run has no record yet, because the campaign is still running or was interrupted.

The analysis counts the last three as failures and never drops them: success rates divide by the planned runs, and value statistics score a failure as the worst possible value. Denominators are therefore fixed by the plan, and a solver cannot look better by failing on hard problems.

A failed run whose last attempt was killed at its hard limit is a killed run, and may hold the incumbent its search had reported when its time limit passed. It stays a failed run in every count; an analysis that asks for it scores the run by that incumbent instead of the worst value. What its search found after the time limit is never a score.

load_dataset() also checks the directory (check_campaign()) and keeps the report in Dataset.check, from which the analysis takes the findings it does not report itself.

RunStatus

RunStatus = Literal['succeeded', 'infeasible', 'failed', 'missing']

The outcome of a planned run, see the module documentation.

Run dataclass

Run(run_id: str, solver: str, problem: str, protocol: str, replicate: int, seed: int, status: RunStatus, record: RunRecord | None = None, failures: tuple[FailureRecord, ...] = ())

One planned run and what became of it.

Attributes:

Name Type Description
run_id str

Run id from the plan.

solver str

Solver name: the campaign label, or the solver id.

problem str

Problem id.

protocol str

Protocol name.

replicate int

Replicate number, from 0.

seed int

Seed the solver received.

status RunStatus

What became of the run.

record RunRecord | None

The run record, for a completed run.

failures tuple[FailureRecord, ...]

The failed attempts of the run, in attempt order. A run can fail and later complete, so a succeeded run may have failures too.

succeeded property

succeeded: bool

Whether the run completed with a feasible schedule.

killed property

killed: bool

Whether the run never completed and its last attempt was killed at its hard limit.

unstarted property

unstarted: bool

Whether the run never completed and its last attempt ended at the start-up limit.

Its worker did not begin the solve in time, so the failure says nothing about the solver.

incumbent property

incumbent: Incumbent | None

The feasible incumbent a killed run held when its time limit passed, or None.

The last candidate the run's search had reported within the time limit of its protocol, if the search found it feasible. A candidate found after the limit is never returned: a solver that stopped on time had no such seconds to search in. The values are the solver's own: no schedule exists to check them against.

late_incumbent property

late_incumbent: bool

Whether the search of a killed run reported an incumbent after its time limit.

No score uses such an incumbent, see incumbent.

outcome property

outcome: Outcome | None

The recorded outcome, or None for a run that did not complete.

provenance property

provenance: Provenance | None

How the last attempt of the run executed, or None for a run never attempted.

The attempt that completed the run, else its last failed attempt.

cpu_share property

cpu_share: float | None

Processor seconds of the completing attempt over its wall seconds, or None.

The processor seconds are those of the process that executed the attempt and, on POSIX systems, of the child processes it waited for. A share below 1 means that this process was not computing for part of the run: it waited for a processor, slept, or left the work to another process whose time the record does not hold, as a solver does on Windows when its backend runs as a program of its own. The share alone does not say which. A share above 1 means the solver ran threads or child processes side by side.

None for a run that did not complete, and when either time was not measured or the wall time is zero. The processor time of a process advances in steps, of about 16 milliseconds on Windows, so the share of a run shorter than a few tenths of a second is coarse.

objectives property

objectives: Mapping[str, float]

Recorded objective values, empty for a run that did not complete.

trace property

trace: tuple[TracePoint, ...]

Improvements of the best solution during the run, empty if not recorded.

value

value(objective: str) -> float | None

Return the recorded value of an objective, or None if there is none.

The value is returned whatever the status, so an infeasible run reports the value of its infeasible schedule. The analysis reads it only for succeeded runs.

points

points(objectives: Sequence[str]) -> tuple[tuple[float, ...], ...]

Return the objective vectors of the solutions the run found.

A multi-objective solver records a front; any other solver contributes the single vector of its best schedule.

Parameters:

Name Type Description Default
objectives Sequence[str]

The problem's objectives, in the problem's order, which is the order of recorded front vectors.

required

Returns:

Type Description
tuple[float, ...]

The vectors, in the recorded (natural) orientation. Empty unless

...

the run succeeded.

Dataset dataclass

Dataset(plan: Plan, runs: tuple[Run, ...], solvers: tuple[SolverEntry, ...], problems: tuple[PlannedProblem, ...], protocols: tuple[Protocol, ...], sessions: tuple[SessionRecord, ...] = (), check: CheckReport | None = None)

The runs of a campaign, with the plan that explains them.

Build one with load_dataset(), and narrow it with select().

Attributes:

Name Type Description
plan Plan

The campaign's plan, complete even after a selection.

runs tuple[Run, ...]

The selected runs, in plan order.

solvers tuple[SolverEntry, ...]

The selected solvers, in campaign order.

problems tuple[PlannedProblem, ...]

The selected problems, in plan order.

protocols tuple[Protocol, ...]

The selected protocols, in campaign order.

sessions tuple[SessionRecord, ...]

The runner sessions that produced the results.

check CheckReport | None

What the check of the campaign directory found when the dataset was loaded, for the whole directory whatever the selection. None for a dataset that was not loaded from a directory or was loaded without the check.

campaign property

campaign: str

Campaign id.

config_hash property

config_hash: str

Config hash of the campaign.

objectives property

objectives: tuple[str, ...]

Every objective the selected problems declare, in first-seen order.

solver

solver(name: str) -> SolverEntry

Return the selected solver with the given name.

Raises:

Type Description
ConfigError

If no selected solver has that name.

problem

problem(problem_id: str) -> PlannedProblem

Return the selected problem with the given id.

Raises:

Type Description
ConfigError

If no selected problem has that id.

protocol

protocol(name: str) -> Protocol

Return the selected protocol with the given name.

Raises:

Type Description
ConfigError

If no selected protocol has that name.

cell

cell(solver: str, problem: str, protocol: str) -> tuple[Run, ...]

Return the runs of one solver on one problem under one protocol, by replicate.

The tuple is empty when the plan excludes the pair, because the solver does not model a feature of the problem.

counts

counts() -> dict[RunStatus, int]

Return the number of selected runs of each status.

select

select(*, protocols: Iterable[str] | None = None, solvers: Iterable[str] | None = None, problems: Iterable[str] | None = None) -> Dataset

Return the dataset restricted to some protocols, solvers, and problems.

Parameters:

Name Type Description Default
protocols Iterable[str] | None

Protocol names to keep, or None to keep all.

None
solvers Iterable[str] | None

Solver names to keep, or None to keep all.

None
problems Iterable[str] | None

Problem ids to keep, or None to keep all.

None

Returns:

Type Description
Dataset

A dataset with the selected entries in their original order.

Raises:

Type Description
ConfigError

If a name is not in the dataset or is given twice.

load_dataset

load_dataset(directory: str | Path, *, check: bool = True) -> Dataset

Read a campaign directory into a dataset.

Parameters:

Name Type Description Default
directory str | Path

A campaign directory, <root>/<campaign id>, as dispatchatlas.campaigns.run() writes it.

required
check bool

Check the directory with check_campaign() and keep the report in Dataset.check. The check reads every record once more, before the dataset reads them, and its report holds none, so it adds to the time of the load and not to its peak memory. It does not derive the plan again from the stored campaign, which would need the campaign's instance files and solvers; dispatchatlas campaign check does. A blocking finding is logged as a warning.

True

Returns:

Type Description
Dataset

Every planned run with its status, record, and failed attempts, and

Dataset

the report of the check.

Raises:

Type Description
SerializationError

If the plan or a record is missing, malformed, or no longer matches its result hash.

dispatchatlas.analysis.config

The settings of an analysis.

AnalysisConfig holds every choice that changes an analysis result: what to analyse (protocol, objective, solvers, problems), how the per-problem values are formed, the significance and confidence levels, the bootstrap, the multiple-comparison correction, the omnibus test, the multi-objective and deviation references, and the focal solver compared with its baselines. The bundle manifest records all of it, so an analysis can be repeated from the manifest alone.

CellStatistic

CellStatistic = Literal['mean', 'median']

How the replicates of one solver on one problem become one value.

Omnibus

Omnibus = Literal['iman-davenport', 'friedman']

The omnibus test whose p-value decides whether the post-hoc test runs.

ReferenceRule

ReferenceRule = Literal['margin', 'ishibuchi']

How a hypervolume reference point is derived from the reference front.

TimeoutScore

TimeoutScore = Literal['worst', 'incumbent']

How a run that was killed at its hard limit is scored.

AnalysisConfig dataclass

AnalysisConfig(protocol: str | None = None, objective: str | None = None, sense: Sense | None = None, solvers: tuple[str, ...] | None = None, problems: tuple[str, ...] | None = None, alpha: float = 0.05, confidence: float = 0.95, resamples: int = 1000, seed: int = 0, interval: IntervalMethod = 'bca', correction: Correction = 'holm', omnibus: Omnibus = 'iman-davenport', statistic: CellStatistic = 'mean', replicate_floor: int | None = None, power: float = 0.8, references: Mapping[str, float] | None = None, reference_source: str | None = None, reference_points: Mapping[str, tuple[float, ...]] | None = None, reference_rule: ReferenceRule = 'margin', reference_parameter: float = 0.1, normalize: bool = True, focal: str | None = None, baselines: tuple[str, ...] | None = None, timeout_score: TimeoutScore = 'worst', overrun_tolerance: float = OVERRUN_MARGIN)

Settings of an analysis. Every field has a default.

Attributes:

Name Type Description
protocol str | None

Protocol to analyse. None analyses the campaign's only protocol; a bundle then analyses every protocol in turn.

objective str | None

Objective to compare solvers on. None takes the primary objective, which the analysed problems must share.

sense Sense | None

"min" or "max". None takes the sense the objective registry declares.

solvers tuple[str, ...] | None

Names of the solvers to analyse, or None for all.

problems tuple[str, ...] | None

Ids of the problems to analyse, or None for all.

alpha float

Significance level of the tests and corrections.

confidence float

Coverage of the confidence intervals.

resamples int

Bootstrap resamples per interval.

seed int

Seed of the bootstrap and of the power simulation.

interval IntervalMethod

Bootstrap interval, "bca" or "percentile".

correction Correction

Multiple-comparison correction of the pairwise tests.

omnibus Omnibus

Omnibus test that gates the Nemenyi post-hoc test.

statistic CellStatistic

How the replicates of a cell become one value, where a failed run counts as the worst value: "mean" makes any failure decisive, "median" needs failures in half of the runs.

replicate_floor int | None

Fewest replicates per cell that the design should have for stochastic solvers. Cells below it are reported as a limitation; nothing is excluded.

power float

Target power of the detectable-effect statement.

references Mapping[str, float] | None

Reference value, such as a best-known objective value, of each problem for the relative percentage deviation.

reference_source str | None

Where the reference values come from, shown with the deviation table.

reference_points Mapping[str, tuple[float, ...]] | None

Hypervolume reference point of a problem, in objective units and minimisation form, overriding the rule.

reference_rule ReferenceRule

Rule deriving the reference point from the reference front: "margin" puts it reference_parameter spans beyond the nadir, "ishibuchi" at ideal + (1 + 1/h) (nadir - ideal) with h = reference_parameter.

reference_parameter float

The margin (positive) or h (a positive integer).

normalize bool

Compute indicators on objectives scaled so that the reference front's ideal is 0 and its nadir is 1.

focal str | None

Name of a solver under study. A bundle then also compares it with each baseline, adjusting the p-values over these comparisons only (see compare_to_baselines()).

baselines tuple[str, ...] | None

Names of the baselines of focal, or None for every other analysed solver.

timeout_score TimeoutScore

How a run that never completed and was killed at its hard limit is scored. "worst" scores it as the worst value, like any other failure. "incumbent" scores it by the incumbent its search held when the time limit of its protocol passed, the last one it had reported by then, when that incumbent is feasible. An incumbent found after the time limit is never a score, so a run that overran is compared on the same seconds as one that stopped on time. The value is the solver's own, with no schedule to check it against. The limitations count these runs under either choice.

overrun_tolerance float

Share of the time limit by which the wall time of a completed run may pass the limit before the limitations name the run: 0.05 allows 5 % more wall time than the limit. The default is the share of the engine's note on a run, OVERRUN_MARGIN, and the engine's clock margin applies beside any share, so a wall time within CLOCK_MARGIN_SECONDS of the limit is never named. Nothing is excluded or rescored.

to_dict

to_dict() -> dict[str, Any]

Return every setting as a JSON-ready dictionary.

dispatchatlas.analysis.summary

The statistical comparison of the solvers of a campaign, under one protocol.

summarize() follows the non-parametric protocol for comparing algorithms over several problems (Demšar 2006; Derrac et al. 2011):

  1. Cells. The replicates of each solver on each problem become one score, the configured statistic (mean or median) of the planned runs, where a run that did not succeed counts as the worst value (infinity for a minimised objective). With timeout_score "incumbent", a run killed at its hard limit counts instead by the feasible incumbent it held when its time limit passed. Descriptive statistics and a bootstrap interval of the mean describe the succeeded runs.
  2. Reliability. Each solver's succeeded, infeasible, failed, and missing runs, over its planned runs.
  3. Ranks and omnibus test. On the complete blocks, the problems every analysed solver is planned on, the scores are ranked per problem (rank 1 best, ties sharing their mean rank). With at least three solvers and two blocks, the Friedman test and its Iman-Davenport form test whether the solvers differ, and Kendall's W, with a bootstrap interval over blocks, measures by how much.
  4. Post-hoc test. When the configured omnibus test rejects, the Nemenyi critical difference says which average ranks differ.
  5. Pairwise tests. Every pair of solvers is compared over the problems both are planned on with the Wilcoxon signed-rank test, and the p-values are adjusted over the family. Each comparison reports, beside its p-value, the paired win share with a bootstrap interval, the mean advantage with a bootstrap interval when no failure makes it infinite, and Cohen's d_z, as Carrasco et al. (2020) recommend.
  6. Limitations. What qualifies the results: failed or missing runs, with the missing ones by cell of solver and problem and the failed ones by solver and kind of error, runs whose worker did not begin the solve in time, runs killed at their hard limit and how they are scored, what the plan notes on the protocol, such as solvers declared deterministic that run once under a time limit, stop reasons that differ among the runs of the iterative solvers, completed runs whose wall time passed the time limit of the protocol, the processor share and the concurrency the runs of a protocol with a time limit executed under, problems left out of the ranks, too few shared problems for the paired test to reach significance, cells below a replicate floor, tests that could not run, and what the check of the campaign directory found beyond these (check_campaign()), such as an altered campaign file, cells with records for only some of their runs, traces that cannot be read against time, or sessions on other code.

Two scores tie when they are equal within TIE_TOLERANCE. A tied pair enters the Wilcoxon test as a zero difference, so the test and the win counts always agree on what is a tie. Every bootstrap uses the configured seed.

summarize

summarize(dataset: Dataset, config: AnalysisConfig | None = None) -> Summary

Compare the solvers of a dataset under one protocol.

Parameters:

Name Type Description Default
dataset Dataset

The campaign's runs, from load_dataset().

required
config AnalysisConfig | None

The analysis settings; the defaults analyse the only protocol on the primary objective.

None

Returns:

Type Description
Summary

Cells, solver summaries, omnibus and post-hoc tests, pairwise

Summary

comparisons, and limitations.

Raises:

Type Description
ConfigError

If the configuration names something the dataset lacks, leaves the protocol or objective ambiguous, or asks for Shaffer's correction when some pair of solvers shares no problem.

dispatchatlas.analysis.comparison

One solver against a set of baselines.

compare_to_baselines() answers the question a new method raises: is it better than each of the established ones? Every baseline is compared with the focal solver by the paired Wilcoxon signed-rank test of summarize(), and the p-values are adjusted over these k comparisons only, the many-to-one family, rather than over every pair of solvers (Demšar 2006, section 3.2.2). Every row is oriented towards the focal solver.

compare_to_baselines

compare_to_baselines(dataset: Dataset, focal: str, baselines: Sequence[str] | None = None, config: AnalysisConfig | None = None) -> BaselineComparison

Compare one solver with each baseline.

Parameters:

Name Type Description Default
dataset Dataset

The campaign's runs.

required
focal str

Name of the solver under study.

required
baselines Sequence[str] | None

Names of the baselines. None takes every other solver the configuration selects.

None
config AnalysisConfig | None

The analysis settings. Its solvers field, if set, must include the focal solver and the baselines.

None

Returns:

Type Description
BaselineComparison

One row per baseline, with the focal solver on the left, adjusted

BaselineComparison

with the configured correction over the baselines.

Raises:

Type Description
ConfigError

If the focal solver or a baseline is unknown, a baseline is the focal solver or is named twice, a baseline shares no problem with the focal solver, or Shaffer's correction is asked for, which applies to all pairwise comparisons only.

dispatchatlas.analysis.results

The results of an analysis: frozen records that tables and figures render.

Directional quantities favour the left solver when positive, or above 0.5 for the win share, whatever the objective's sense: an advantage is how much better the left solver's value is, in objective units.

Limitation dataclass

Limitation(topic: str, message: str)

A caveat that qualifies the results.

Attributes:

Name Type Description
topic str

What it concerns, for example "pairwise" or "replicates".

message str

One or two sentences stating the caveat.

CellSummary dataclass

CellSummary(solver: str, problem: str, planned: int, succeeded: int, score: float, mean: float | None, median: float | None, std: float | None, best: float | None, mean_low: float | None, mean_high: float | None)

The runs of one solver on one problem.

Attributes:

Name Type Description
solver str

Solver name.

problem str

Problem id.

planned int

Runs the plan holds for the cell.

succeeded int

Runs that completed with a feasible schedule.

score float

The value the comparisons use: the configured statistic of the planned runs, a failed run counting as the worst value, so it is infinite when failures decide it.

mean float | None

Mean value of the succeeded runs, or None without one.

median float | None

Median value of the succeeded runs.

std float | None

Sample standard deviation of the succeeded runs, with at least two.

best float | None

Best value of the succeeded runs.

mean_low float | None

Lower confidence bound of the mean.

mean_high float | None

Upper confidence bound of the mean.

SolverSummary dataclass

SolverSummary(solver: str, planned: int, succeeded: int, infeasible: int, failed: int, missing: int, problems: int, average_rank: float | None)

The reliability and average rank of one solver.

Attributes:

Name Type Description
solver str

Solver name.

planned int

Planned runs over the analysed problems.

succeeded int

Runs that completed with a feasible schedule.

infeasible int

Completed runs without a feasible schedule.

failed int

Runs whose attempts all failed.

missing int

Runs without a record yet.

problems int

Problems the plan pairs the solver with.

average_rank float | None

Mean rank over the complete blocks, rank 1 best, or None when no complete block exists.

success_rate property

success_rate: float

Share of the planned runs that succeeded; 0 without planned runs.

PairwiseComparison dataclass

PairwiseComparison(left: str, right: str, pairs: int, wins: int, losses: int, ties: int, win_share: float, win_share_low: float, win_share_high: float, median_advantage: float | None, mean_advantage: float | None, advantage_low: float | None, advantage_high: float | None, cohens_dz: float | None, statistic: float, p_value: float, adjusted_p_value: float, method: TestMethod, confidence: float)

A paired comparison of two solvers over the problems they share.

Each problem contributes one pair: the two solvers' cell scores. The Wilcoxon signed-rank test compares them, and the effect sizes and intervals beside the p-value say how large the difference is.

Attributes:

Name Type Description
left str

The solver the directional fields favour when positive.

right str

The other solver.

pairs int

Shared problems.

wins int

Problems the left solver wins.

losses int

Problems the left solver loses.

ties int

Problems where the scores tie.

win_share float

(wins + ties / 2) / pairs.

win_share_low float

Lower bootstrap bound of the win share.

win_share_high float

Upper bootstrap bound of the win share.

median_advantage float | None

Median over problems of how much better the left score is, in objective units, or None when undefined.

mean_advantage float | None

Mean advantage, or None when a failure makes an advantage infinite.

advantage_low float | None

Lower bootstrap bound of the mean advantage.

advantage_high float | None

Upper bootstrap bound of the mean advantage.

cohens_dz float | None

Mean advantage over its standard deviation, or None.

statistic float

Wilcoxon statistic min(W+, W-).

p_value float

Two-sided Wilcoxon p-value.

adjusted_p_value float

The p-value adjusted over the family of comparisons.

method TestMethod

How the p-value was obtained.

confidence float

Coverage of the intervals.

swapped

swapped() -> PairwiseComparison

Return the same comparison seen from the right solver.

OmnibusResult dataclass

OmnibusResult(blocks: int, groups: int, friedman_statistic: float, friedman_p_value: float, iman_davenport_statistic: float, iman_davenport_p_value: float, kendalls_w: float, kendalls_w_low: float, kendalls_w_high: float, confidence: float, test: str, alpha: float, significant: bool)

The Friedman and Iman-Davenport tests over the complete blocks.

Attributes:

Name Type Description
blocks int

Complete blocks: problems every analysed solver is planned on.

groups int

Solvers.

friedman_statistic float

Friedman's Q.

friedman_p_value float

Chi-square p-value of Q.

iman_davenport_statistic float

Iman and Davenport's F.

iman_davenport_p_value float

F p-value.

kendalls_w float

Kendall's coefficient of concordance, the effect size.

kendalls_w_low float

Lower bootstrap bound of W, resampling blocks.

kendalls_w_high float

Upper bootstrap bound of W.

confidence float

Coverage of the interval.

test str

The test whose p-value decides significance.

alpha float

Significance level.

significant bool

Whether that p-value is below alpha.

p_value property

p_value: float

The p-value of the deciding test.

CriticalDifference dataclass

CriticalDifference(value: float, alpha: float, groups: int, blocks: int, average_ranks: tuple[tuple[str, float], ...], significant_pairs: tuple[tuple[str, str], ...], cliques: tuple[tuple[str, ...], ...])

The Nemenyi post-hoc comparison of average ranks.

Attributes:

Name Type Description
value float

The critical difference.

alpha float

Significance level.

groups int

Solvers.

blocks int

Complete blocks.

average_ranks tuple[tuple[str, float], ...]

Each solver's average rank, best first.

significant_pairs tuple[tuple[str, str], ...]

Pairs, best first, whose ranks differ by more than the critical difference.

cliques tuple[tuple[str, ...], ...]

Maximal runs of solvers, in rank order, whose ranks lie within the critical difference of each other, the bars of a critical-difference diagram. Singletons are included.

Summary dataclass

Summary(campaign: str, protocol: str, objective: str, sense: Sense, config: AnalysisConfig, solvers: tuple[str, ...], problems: tuple[str, ...], blocks: tuple[str, ...], cells: tuple[CellSummary, ...], solver_summaries: tuple[SolverSummary, ...], pairwise: tuple[PairwiseComparison, ...], omnibus: OmnibusResult | None, critical_difference: CriticalDifference | None, limitations: tuple[Limitation, ...])

Every result of summarize() for one protocol.

Attributes:

Name Type Description
campaign str

Campaign id.

protocol str

The analysed protocol.

objective str

The analysed objective.

sense Sense

Its sense.

config AnalysisConfig

The configuration used.

solvers tuple[str, ...]

Analysed solvers, in campaign order.

problems tuple[str, ...]

Analysed problems, in plan order.

blocks tuple[str, ...]

The problems every analysed solver is planned on.

cells tuple[CellSummary, ...]

One summary per planned solver and problem.

solver_summaries tuple[SolverSummary, ...]

One summary per solver, in campaign order.

pairwise tuple[PairwiseComparison, ...]

One comparison per pair of solvers sharing a problem.

omnibus OmnibusResult | None

The omnibus tests, or None when they do not apply.

critical_difference CriticalDifference | None

The post-hoc test, or None.

limitations tuple[Limitation, ...]

The caveats that qualify these results.

comparison

comparison(left: str, right: str) -> PairwiseComparison

Return the comparison of two solvers, oriented towards left.

Raises:

Type Description
ConfigError

If the two solvers were not compared.

BaselineComparison dataclass

BaselineComparison(focal: str, baselines: tuple[str, ...], protocol: str, objective: str, sense: Sense, correction: Correction, alpha: float, rows: tuple[PairwiseComparison, ...], better_than_all: bool, limitations: tuple[Limitation, ...])

One solver compared with each of several baselines.

Attributes:

Name Type Description
focal str

The solver under study; every row has it on the left.

baselines tuple[str, ...]

The baselines, in campaign order.

protocol str

The analysed protocol.

objective str

The analysed objective.

sense Sense

Its sense.

correction Correction

The correction applied over the rows.

alpha float

Significance level.

rows tuple[PairwiseComparison, ...]

One comparison per baseline, adjusted over this family only.

better_than_all bool

Whether every row is significant at alpha after correction and has a win share above 0.5.

limitations tuple[Limitation, ...]

The caveats of the underlying summary.

dispatchatlas.analysis.deviation

Relative percentage deviation from reference values.

Scheduling benchmarks usually publish a best-known or optimal value per instance, and results are compared on the scale-free relative percentage deviation (RPD) from it (Vallada, Ruiz and Framinan 2015):

  • RPD = 100 (value - reference) / reference for a minimised objective and 100 (reference - value) / reference for a maximised one, so a positive RPD is always worse than the reference and a negative one improves on it;
  • the average RPD (ARPD) of a solver is the mean over all of its runs on all problems together.

Only succeeded runs have a value to deviate from. Each row reports how many of the planned runs that is, so an ARPD over few surviving runs is visible as such.

DeviationRow dataclass

DeviationRow(solver: str, problems: int, runs: int, planned: int, arpd: float | None, rpd_min: float | None, rpd_max: float | None)

The average relative percentage deviation of one solver.

Attributes:

Name Type Description
solver str

Solver name.

problems int

Problems with at least one succeeded run.

runs int

Succeeded runs, whose deviations are averaged.

planned int

Planned runs.

arpd float | None

Mean deviation over the succeeded runs, in percent, or None without one.

rpd_min float | None

Smallest deviation.

rpd_max float | None

Largest deviation.

relative_deviation

relative_deviation(value: float, reference: float, sense: Sense = 'min') -> float

Return the relative percentage deviation of a value from a reference.

Parameters:

Name Type Description Default
value float

The objective value.

required
reference float

The reference value, such as the best known, positive.

required
sense Sense

Whether the objective is minimised or maximised.

'min'

Returns:

Type Description
float

The deviation in percent, positive when worse than the reference.

Raises:

Type Description
ConfigError

If the reference is not positive or a value is not finite.

arpd

arpd(dataset: Dataset, config: AnalysisConfig) -> tuple[DeviationRow, ...]

Return the average relative percentage deviation of every solver.

Parameters:

Name Type Description Default
dataset Dataset

The campaign's runs.

required
config AnalysisConfig

The analysis settings; references must give a value for every analysed problem.

required

Returns:

Type Description
tuple[DeviationRow, ...]

One row per analysed solver, in campaign order.

Raises:

Type Description
ConfigError

If the configuration has no references or lacks one for an analysed problem, or the scope cannot be resolved.

dispatchatlas.analysis.indicators

Multi-objective quality indicators: hypervolume, IGD, and IGD+.

On every problem with at least two objectives, indicator_report() scores each run's approximation set, the front a multi-objective solver records or the single objective vector of any other solver's best schedule, then aggregates the runs of each solver:

  1. Objective vectors are put in minimisation form.
  2. The reference set of the problem is the non-dominated union of every run's set, over all analysed solvers. Its per-objective minimum is the ideal point and its maximum the nadir point.
  3. With normalize (the default), every objective is rescaled so that the ideal is 0 and the nadir 1; an objective on which the reference set does not vary is only shifted.
  4. The reference point is shared by every solver on the problem. It is the configured point, if any, or it is derived from the ideal and nadir: the "margin" rule places it m spans beyond the nadir (r = nadir + m (nadir - ideal), default m = 0.1), the "ishibuchi" rule at r = ideal + (1 + 1/h)(nadir - ideal) (Ishibuchi et al. 2018). On an objective without spread, both rules move |nadir| m, or m when the nadir is 0, beyond it (m = 1/h for the second rule).
  5. Each run gets the hypervolume of its set, the volume it dominates up to the reference point (Zitzler and Thiele 1999), computed exactly by dispatchatlas.core.pareto.hypervolume(); the IGD, the mean Euclidean distance from each reference-set point to the nearest point of the run's set (Coello Coello and Reyes Sierra 2004); and the IGD+, the same mean with the distance counting only the objectives in which the run's point is worse (Ishibuchi et al. 2015). Hypervolume is to be maximised, IGD and IGD+ minimised.
  6. A run that did not succeed has an empty set: hypervolume 0 and infinite IGD and IGD+. The configured statistic (mean or median) of the planned runs gives each solver's value on the problem, so failures count.

A reference point derived from the data moves when solvers are added or removed, so hypervolumes are comparable only between results that share their reference points; ensure_shared_reference_points() checks it.

ReferencePoint dataclass

ReferencePoint(problem: str, objectives: tuple[str, ...], ideal: Point, nadir: Point, point: Point, rule: Literal['fixed', 'margin', 'ishibuchi'], parameter: float | None, normalized: bool)

The frame in which one problem's indicators are computed.

Attributes:

Name Type Description
problem str

Problem id.

objectives tuple[str, ...]

The objectives, in the problem's order.

ideal Point

Per-objective minimum of the reference set, in minimisation form.

nadir Point

Per-objective maximum of the reference set, in minimisation form.

point Point

The hypervolume reference point, in objective units and minimisation form.

rule Literal['fixed', 'margin', 'ishibuchi']

"fixed" for a configured point, else the deriving rule.

parameter float | None

The rule's margin or h; None for a fixed point.

normalized bool

Whether the indicators use normalised objectives.

IndicatorRow dataclass

IndicatorRow(solver: str, problem: str, planned: int, succeeded: int, hypervolume: float, igd: float, igd_plus: float)

One solver's indicators on one problem, aggregated over its planned runs.

Attributes:

Name Type Description
solver str

Solver name.

problem str

Problem id.

planned int

Planned runs.

succeeded int

Runs with a feasible result.

hypervolume float

Hypervolume, higher is better.

igd float

Inverted generational distance, lower is better.

igd_plus float

IGD+, lower is better.

IndicatorReport dataclass

IndicatorReport(protocol: str, statistic: str, rows: tuple[IndicatorRow, ...], reference_points: tuple[ReferencePoint, ...], limitations: tuple[Limitation, ...])

The indicators of every solver on every multi-objective problem.

Attributes:

Name Type Description
protocol str

The analysed protocol.

statistic str

How runs were aggregated.

rows tuple[IndicatorRow, ...]

One row per planned solver and problem, by problem then solver.

reference_points tuple[ReferencePoint, ...]

The frame of each problem.

limitations tuple[Limitation, ...]

Problems that could not be scored, and why.

reference_point

reference_point(points: Sequence[Sequence[float]], rule: ReferenceRule = 'margin', parameter: float = 0.1) -> Point

Derive a hypervolume reference point from a set of minimisation vectors.

Parameters:

Name Type Description Default
points Sequence[Sequence[float]]

The reference front, one vector per point.

required
rule ReferenceRule

"margin" or "ishibuchi", see the module documentation.

'margin'
parameter float

The margin (positive) or h (a positive integer).

0.1

Returns:

Type Description
Point

The reference point.

Raises:

Type Description
ConfigError

If points is empty or ragged, or the parameter does not suit the rule.

igd

igd(front: Sequence[Sequence[float]], reference_set: Sequence[Sequence[float]]) -> float

Return the inverted generational distance of a front to a reference set.

The mean over reference points of the Euclidean distance to the nearest front point; infinite for an empty front.

Raises:

Type Description
ConfigError

If the reference set is empty or the vectors are ragged.

igd_plus

igd_plus(front: Sequence[Sequence[float]], reference_set: Sequence[Sequence[float]]) -> float

Return the IGD+ of a front to a reference set, in minimisation form.

Like igd(), but the distance from a reference point z to a front point a is sqrt(sum_k max(a_k - z_k, 0)^2): only the objectives in which a is worse count, which makes IGD+ weakly Pareto compliant.

Raises:

Type Description
ConfigError

If the reference set is empty or the vectors are ragged.

indicator_report

indicator_report(dataset: Dataset, config: AnalysisConfig | None = None) -> IndicatorReport

Return the indicators of every solver on every problem with several objectives.

Parameters:

Name Type Description Default
dataset Dataset

The campaign's runs.

required
config AnalysisConfig | None

The analysis settings: protocol, selection, aggregation statistic, reference points or rule, and normalisation.

None

Returns:

Type Description
IndicatorReport

The rows and the reference point of each problem. Problems with one

IndicatorReport

objective are skipped.

Raises:

Type Description
ConfigError

If the scope cannot be resolved, an objective is not registered, or a configured reference point does not fit its problem.

ensure_shared_reference_points

ensure_shared_reference_points(reports: Sequence[IndicatorReport]) -> None

Check that reports scoring the same problem used the same frame.

Hypervolumes measured against different reference points, or IGD values computed on differently normalised objectives, are on different scales.

Raises:

Type Description
ConfigError

If two reports score a problem against different reference points, rules, or normalisations.

dispatchatlas.analysis.methods

The catalog of the statistical methods DispatchAtlas implements.

Each Method names the function that implements it, the assumptions its result rests on, and the sources it follows, as keys of the bibliography in dispatchatlas.references. The analysis bundle lists the methods it used with their references, and the documentation renders the catalog.

MethodFamily

MethodFamily = Literal['test', 'post-hoc', 'correction', 'effect-size', 'interval', 'bayesian', 'power', 'deviation', 'indicator', 'reporting']

The role a method plays in an analysis.

Method dataclass

Method(id: str, name: str, family: MethodFamily, summary: str, assumptions: tuple[str, ...], references: tuple[str, ...], function: str)

One implemented statistical method.

Attributes:

Name Type Description
id str

Catalog key in kebab case.

name str

Human-readable name.

family MethodFamily

The role the method plays.

summary str

One sentence on what the method computes.

assumptions tuple[str, ...]

What the result rests on.

references tuple[str, ...]

Bibliography keys of the sources.

function str

Dotted path of the function that implements it.

implemented_methods

implemented_methods(family: MethodFamily | None = None) -> tuple[Method, ...]

Return the implemented methods, all of them or those of one family.

method

method(method_id: str) -> Method

Return the method with the given id.

Raises:

Type Description
ConfigError

If no method has that id.

dispatchatlas.analysis.tables

Tables of analysis results, rendered as Markdown, CSV, or LaTeX.

A Table holds typed columns and raw values; the renderers format them. Markdown and LaTeX show six significant digits (three for p-values), an em dash for a missing value, and intervals as [low, high]. CSV keeps full precision, writes an interval as two columns <key>_low and <key>_high, and names columns by their keys, one row per record.

Every renderer refuses a table that reports a p-value without an effect size and a confidence interval beside it, as Carrasco et al. (2020) recommend: check_p_values() enforces it from the column kinds.

ColumnKind

ColumnKind = Literal['text', 'count', 'number', 'p-value', 'effect', 'interval']

What a column holds, which decides its formatting and the p-value check.

Cell

Cell = str | int | float | bool | tuple[float | None, float | None] | None

A table value; intervals are (low, high) pairs.

Column dataclass

Column(key: str, header: str, kind: ColumnKind = 'text')

A table column.

Attributes:

Name Type Description
key str

Machine name, used as the CSV header.

header str

Human-readable header, used in Markdown and LaTeX.

kind ColumnKind

What the column holds.

Table dataclass

Table(name: str, title: str, columns: tuple[Column, ...], rows: tuple[tuple[Cell, ...], ...], notes: tuple[str, ...] = ())

A titled table of typed columns.

Attributes:

Name Type Description
name str

File stem, for example "pairwise".

title str

One-line caption.

columns tuple[Column, ...]

The columns.

rows tuple[tuple[Cell, ...], ...]

One value per column in every row.

notes tuple[str, ...]

Sentences printed under the table in Markdown and as comments in LaTeX.

check_p_values

check_p_values(table: Table) -> None

Refuse a table that reports a p-value without an effect size and an interval.

Raises:

Type Description
ConfigError

If the table has a p-value column but no effect-size column or no interval column.

to_markdown

to_markdown(table: Table) -> str

Render a table as GitHub-flavoured Markdown, with its title and notes.

to_csv

to_csv(table: Table) -> str

Render a table as CSV at full precision, headed by the column keys.

to_latex

to_latex(table: Table) -> str

Render a table as a LaTeX tabular with booktabs rules.

The title and notes become comments above and below, so the fragment can be placed in any float. It needs \usepackage{booktabs}.

runs_table

runs_table(dataset: Dataset, timeout_score: TimeoutScore = 'worst') -> Table

Return one row per run of the dataset: its facts and every objective value.

A failed run names the kind of its last failed attempt. The column scored_by says where the score of a run comes from: its result, the worst value, or, with timeout_score "incumbent", the incumbent a run killed at its hard limit held when its time limit passed. The objective columns of such a run hold the values of that incumbent, and its evaluations and elapsed columns say when its search found it, which is within the time limit.

The columns from time_limit to attempts compare a run with its budget and say under which load it executed. time_limit is the time limit of the run's protocol. wall_seconds is the wall time the engine measured for the last attempt of the run, the completing one or the last failed one, where elapsed is the solver's own report. overrun_seconds is the wall time minus the time limit, negative for a run that ended early. cpu_share is the processor seconds of the attempt over its wall seconds (Run.cpu_share): below 1 when the executing process was not computing for part of the run, whether it waited for a processor, slept, or left the work to another process, and above 1 when the solver ran threads. concurrent is the number of runs executing when the attempt started, itself included, cpu the processor its worker was pinned to, and attempts the number of the last attempt. A cell is empty when the protocol has no time limit or the record does not state the value.

For a run that did not complete, cpu_seconds, wall_seconds, concurrent, cpu, and attempts describe its last failed attempt, and overrun_seconds and cpu_share are empty: what ended that attempt is in the failure column, and its times say nothing about a budget.

Parameters:

Name Type Description Default
dataset Dataset

The runs.

required
timeout_score TimeoutScore

How the analysis the table belongs to scores a run that was killed at its hard limit.

'worst'

cells_table

cells_table(summary: Summary) -> Table

Return the per-problem results of every solver.

solvers_table

solvers_table(summary: Summary) -> Table

Return each solver's reliability and average rank, best rank first.

pairwise_table

pairwise_table(summary: Summary) -> Table

Return every pairwise comparison with its effect sizes and intervals.

baselines_table

baselines_table(comparison: BaselineComparison) -> Table

Return the comparisons of a focal solver with its baselines.

omnibus_table

omnibus_table(summary: Summary) -> Table

Return the omnibus tests with Kendall's W, and the post-hoc test as notes.

deviation_table

deviation_table(rows: Sequence[DeviationRow], objective: str, source: str | None = None) -> Table

Return the average relative percentage deviation of every solver.

indicators_table

indicators_table(report: IndicatorReport) -> Table

Return every solver's hypervolume, IGD, and IGD+ on every multi-objective problem.

dispatchatlas.analysis.manifest

The reproducibility manifest of an analysis bundle.

manifest.json records what a reader needs to repeat or to judge an analysis:

  • the campaign: id, config hash, seed, replicates, the digest of the analysed plan's runs, and run counts by status, and, when the campaign sets them, its seed namespace, its replication under a time limit, and the solvers' own replicate counts;
  • the full AnalysisConfig;
  • the DispatchAtlas version and the environment, as capture_environment() records it;
  • for each analysed protocol, the two sample sizes of the design: the replicates behind each per-problem value and the problems behind each paired test, with the smallest standardized effect the paired tests detect at the configured power, or the reason none is detectable;
  • whether figures were rendered;
  • the SHA-256 digest of every other file of the bundle.

The manifest holds no timestamp, so equal inputs give an identical file on the same machine and software.

MANIFEST_ARTIFACT module-attribute

MANIFEST_ARTIFACT = 'dispatchatlas.analysis-bundle'

The artifact field that identifies an analysis bundle manifest.

MANIFEST_VERSION module-attribute

MANIFEST_VERSION = 1

Version of the manifest format.

sha256

sha256(data: bytes | Path) -> str

Return the SHA-256 hex digest of bytes, or of a file's content.

analysis_record

analysis_record(dataset: Dataset, summary: Summary) -> dict[str, Any]

Describe the design behind one protocol's summary.

Parameters:

Name Type Description Default
dataset Dataset

The dataset the summary was computed from.

required
summary Summary

The summary.

required

Returns:

Type Description
dict[str, Any]

A JSON-ready dictionary: protocol, objective and sense, the analysed

dict[str, Any]

solvers and problems, the complete blocks, the replicates per cell,

dict[str, Any]

the problems per paired test, and the power statement.

build_manifest

build_manifest(dataset: Dataset, config: AnalysisConfig, analyses: list[dict[str, Any]], files: Mapping[str, str], figures: Mapping[str, Any]) -> dict[str, Any]

Return the manifest document of a bundle.

Parameters:

Name Type Description Default
dataset Dataset

The analysed dataset, before narrowing to a protocol.

required
config AnalysisConfig

The configuration of the analysis.

required
analyses list[dict[str, Any]]

One analysis_record() per analysed protocol.

required
files Mapping[str, str]

SHA-256 digest of every other file, keyed by path relative to the bundle, with / separators.

required
figures Mapping[str, Any]

Whether figures were rendered, and why not.

required

dispatchatlas.analysis.bundle

The analysis bundle: every table, figure, and caveat of a campaign in one directory.

write_bundle() analyses a campaign directory and writes:

<output>/
    index.md                  what the bundle holds: tables, figures, methods, and sources
    limitations.md            the caveats of every analysis
    runs.csv                  one row per analysed run, at full precision
    tables/<protocol>/<table>.md, .csv, .tex
    figures/<protocol>/...    SVG and TikZ figures, when the figure package is available
    manifest.json             configuration, versions, environment, and file digests

Each protocol is analysed separately (see summarize()). Tables with a p-value always carry an effect size and an interval beside it. The multi-objective indicator table appears when a problem has several objectives, the deviation table when the configuration gives reference values, and the baselines table, the focal solver against each baseline (see compare_to_baselines()), when it names a focal solver. Figures are drawn with the same settings as the tables, in the formats asked for, and the index lists each with its caption and the sources of its method.

Output is byte-reproducible: equal inputs, configuration, and software give identical files. Content files hold no timestamps or paths, JSON is written with sorted keys, and text with \n line endings; the environment appears in the manifest only.

MANIFEST_FILE module-attribute

MANIFEST_FILE = 'manifest.json'

Name of the manifest in a bundle directory.

FIGURE_FORMATS module-attribute

FIGURE_FORMATS: tuple[str, ...] = ('svg', 'tex')

The formats figures can be written in: SVG, and TikZ for LaTeX.

write_bundle

write_bundle(campaign_directory: str | Path, output_directory: str | Path, config: AnalysisConfig | None = None, *, figures: bool = True, formats: Sequence[str] = FIGURE_FORMATS) -> Path

Analyse a campaign and write the bundle.

Parameters:

Name Type Description Default
campaign_directory str | Path

A campaign directory written by dispatchatlas.campaigns.run().

required
output_directory str | Path

Where to write the bundle. It must be absent, empty, or hold an earlier bundle, whose files are replaced once the analysis has succeeded.

required
config AnalysisConfig | None

The analysis settings. Without a protocol, every protocol of the campaign is analysed. With a focal solver, the bundle holds the baselines table of every protocol.

None
figures bool

Render figures through dispatchatlas.analysis.figures.render_figures, with the settings of each protocol's analysis. When that package is not installed, the bundle is written without figures and the manifest says why.

True
formats Sequence[str]

Formats of the figures, from FIGURE_FORMATS. Their order does not matter.

FIGURE_FORMATS

Returns:

Type Description
Path

The bundle directory.

Raises:

Type Description
ConfigError

If the output directory holds other files, a format is unknown, or the configuration does not fit the campaign.

SerializationError

If the campaign directory cannot be read.