Analysis¶
Analysis datasets and settings, the statistical comparison of solvers, deviations and quality indicators, the method catalog, tables, the manifest, and the bundle.
dispatchatlas.analysis
¶
Statistical analysis of campaign results.
Read a campaign directory, compare its solvers, and write every table, figure, and caveat to one reproducible bundle:
from dispatchatlas.analysis import AnalysisConfig, load_dataset, summarize, write_bundle
dataset = load_dataset("results/my-campaign")
summary = summarize(dataset, AnalysisConfig(correction="holm"))
write_bundle("results/my-campaign", "analysis/my-campaign")
The modules, from data to reports:
dataset: the planned runs of a campaign and what became of each; a run without a feasible result counts as a failure.config: the settings of an analysis.summary,comparison: ranks, omnibus and post-hoc tests, and paired comparisons with effect sizes and intervals; one solver against baselines.deviation,indicators: relative percentage deviation and multi-objective quality indicators.stats: the statistical kernels.methods: the catalog of implemented methods.tables,manifest,bundle: tables, the reproducibility manifest, and the bundle.
dispatchatlas.analysis.dataset
¶
Campaign results as analysis data.
load_dataset() reads a campaign directory written by
dispatchatlas.campaigns.run() through the readers of
dispatchatlas.campaigns.store, and returns a Dataset that holds
one Run for every run of the plan, in plan order, whether or not the
run completed.
Every planned run has one of four statuses:
"succeeded": the run completed with a feasible schedule;"infeasible": the run completed, but its schedule breaks a constraint, such as a hard deadline, or the solver returned no schedule;"failed": every attempt of the run failed;"missing": the run has no record yet, because the campaign is still running or was interrupted.
The analysis counts the last three as failures and never drops them: success rates divide by the planned runs, and value statistics score a failure as the worst possible value. Denominators are therefore fixed by the plan, and a solver cannot look better by failing on hard problems.
A failed run whose last attempt was killed at its hard limit is a killed run, and may hold the incumbent its search had reported when its time limit passed. It stays a failed run in every count; an analysis that asks for it scores the run by that incumbent instead of the worst value. What its search found after the time limit is never a score.
load_dataset() also checks the directory
(check_campaign()) and keeps the
report in Dataset.check, from which the analysis takes the findings
it does not report itself.
RunStatus
¶
The outcome of a planned run, see the module documentation.
Run
dataclass
¶
Run(run_id: str, solver: str, problem: str, protocol: str, replicate: int, seed: int, status: RunStatus, record: RunRecord | None = None, failures: tuple[FailureRecord, ...] = ())
One planned run and what became of it.
Attributes:
| Name | Type | Description |
|---|---|---|
run_id |
str
|
Run id from the plan. |
solver |
str
|
Solver name: the campaign label, or the solver id. |
problem |
str
|
Problem id. |
protocol |
str
|
Protocol name. |
replicate |
int
|
Replicate number, from 0. |
seed |
int
|
Seed the solver received. |
status |
RunStatus
|
What became of the run. |
record |
RunRecord | None
|
The run record, for a completed run. |
failures |
tuple[FailureRecord, ...]
|
The failed attempts of the run, in attempt order. A run can fail and later complete, so a succeeded run may have failures too. |
killed
property
¶
Whether the run never completed and its last attempt was killed at its hard limit.
unstarted
property
¶
Whether the run never completed and its last attempt ended at the start-up limit.
Its worker did not begin the solve in time, so the failure says nothing about the solver.
incumbent
property
¶
incumbent: Incumbent | None
The feasible incumbent a killed run held when its time limit passed, or None.
The last candidate the run's search had reported within the time limit of its protocol, if the search found it feasible. A candidate found after the limit is never returned: a solver that stopped on time had no such seconds to search in. The values are the solver's own: no schedule exists to check them against.
late_incumbent
property
¶
Whether the search of a killed run reported an incumbent after its time limit.
No score uses such an incumbent, see incumbent.
outcome
property
¶
outcome: Outcome | None
The recorded outcome, or None for a run that did not complete.
provenance
property
¶
provenance: Provenance | None
How the last attempt of the run executed, or None for a run never attempted.
The attempt that completed the run, else its last failed attempt.
cpu_share
property
¶
Processor seconds of the completing attempt over its wall seconds, or None.
The processor seconds are those of the process that executed the attempt and, on POSIX systems, of the child processes it waited for. A share below 1 means that this process was not computing for part of the run: it waited for a processor, slept, or left the work to another process whose time the record does not hold, as a solver does on Windows when its backend runs as a program of its own. The share alone does not say which. A share above 1 means the solver ran threads or child processes side by side.
None for a run that did not complete, and when either time was
not measured or the wall time is zero. The processor time of a
process advances in steps, of about 16 milliseconds on Windows, so
the share of a run shorter than a few tenths of a second is coarse.
objectives
property
¶
Recorded objective values, empty for a run that did not complete.
trace
property
¶
trace: tuple[TracePoint, ...]
Improvements of the best solution during the run, empty if not recorded.
value
¶
Return the recorded value of an objective, or None if there is none.
The value is returned whatever the status, so an infeasible run reports the value of its infeasible schedule. The analysis reads it only for succeeded runs.
points
¶
Return the objective vectors of the solutions the run found.
A multi-objective solver records a front; any other solver contributes the single vector of its best schedule.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
objectives
|
Sequence[str]
|
The problem's objectives, in the problem's order, which is the order of recorded front vectors. |
required |
Returns:
| Type | Description |
|---|---|
tuple[float, ...]
|
The vectors, in the recorded (natural) orientation. Empty unless |
...
|
the run succeeded. |
Dataset
dataclass
¶
Dataset(plan: Plan, runs: tuple[Run, ...], solvers: tuple[SolverEntry, ...], problems: tuple[PlannedProblem, ...], protocols: tuple[Protocol, ...], sessions: tuple[SessionRecord, ...] = (), check: CheckReport | None = None)
The runs of a campaign, with the plan that explains them.
Build one with load_dataset(), and narrow it with select().
Attributes:
| Name | Type | Description |
|---|---|---|
plan |
Plan
|
The campaign's plan, complete even after a selection. |
runs |
tuple[Run, ...]
|
The selected runs, in plan order. |
solvers |
tuple[SolverEntry, ...]
|
The selected solvers, in campaign order. |
problems |
tuple[PlannedProblem, ...]
|
The selected problems, in plan order. |
protocols |
tuple[Protocol, ...]
|
The selected protocols, in campaign order. |
sessions |
tuple[SessionRecord, ...]
|
The runner sessions that produced the results. |
check |
CheckReport | None
|
What the check of the campaign directory found when the
dataset was loaded, for the whole directory whatever the
selection. |
objectives
property
¶
Every objective the selected problems declare, in first-seen order.
solver
¶
solver(name: str) -> SolverEntry
Return the selected solver with the given name.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If no selected solver has that name. |
problem
¶
problem(problem_id: str) -> PlannedProblem
Return the selected problem with the given id.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If no selected problem has that id. |
protocol
¶
protocol(name: str) -> Protocol
Return the selected protocol with the given name.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If no selected protocol has that name. |
cell
¶
cell(solver: str, problem: str, protocol: str) -> tuple[Run, ...]
Return the runs of one solver on one problem under one protocol, by replicate.
The tuple is empty when the plan excludes the pair, because the solver does not model a feature of the problem.
select
¶
select(*, protocols: Iterable[str] | None = None, solvers: Iterable[str] | None = None, problems: Iterable[str] | None = None) -> Dataset
Return the dataset restricted to some protocols, solvers, and problems.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
protocols
|
Iterable[str] | None
|
Protocol names to keep, or |
None
|
solvers
|
Iterable[str] | None
|
Solver names to keep, or |
None
|
problems
|
Iterable[str] | None
|
Problem ids to keep, or |
None
|
Returns:
| Type | Description |
|---|---|
Dataset
|
A dataset with the selected entries in their original order. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If a name is not in the dataset or is given twice. |
load_dataset
¶
load_dataset(directory: str | Path, *, check: bool = True) -> Dataset
Read a campaign directory into a dataset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
directory
|
str | Path
|
A campaign directory, |
required |
check
|
bool
|
Check the directory with
|
True
|
Returns:
| Type | Description |
|---|---|
Dataset
|
Every planned run with its status, record, and failed attempts, and |
Dataset
|
the report of the check. |
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the plan or a record is missing, malformed, or no longer matches its result hash. |
dispatchatlas.analysis.config
¶
The settings of an analysis.
AnalysisConfig holds every choice that changes an analysis result:
what to analyse (protocol, objective, solvers, problems), how the
per-problem values are formed, the significance and confidence levels, the
bootstrap, the multiple-comparison correction, the omnibus test, the
multi-objective and deviation references, and the focal solver compared with
its baselines. The bundle manifest records all of it, so an analysis can be
repeated from the manifest alone.
CellStatistic
¶
How the replicates of one solver on one problem become one value.
Omnibus
¶
The omnibus test whose p-value decides whether the post-hoc test runs.
ReferenceRule
¶
How a hypervolume reference point is derived from the reference front.
TimeoutScore
¶
How a run that was killed at its hard limit is scored.
AnalysisConfig
dataclass
¶
AnalysisConfig(protocol: str | None = None, objective: str | None = None, sense: Sense | None = None, solvers: tuple[str, ...] | None = None, problems: tuple[str, ...] | None = None, alpha: float = 0.05, confidence: float = 0.95, resamples: int = 1000, seed: int = 0, interval: IntervalMethod = 'bca', correction: Correction = 'holm', omnibus: Omnibus = 'iman-davenport', statistic: CellStatistic = 'mean', replicate_floor: int | None = None, power: float = 0.8, references: Mapping[str, float] | None = None, reference_source: str | None = None, reference_points: Mapping[str, tuple[float, ...]] | None = None, reference_rule: ReferenceRule = 'margin', reference_parameter: float = 0.1, normalize: bool = True, focal: str | None = None, baselines: tuple[str, ...] | None = None, timeout_score: TimeoutScore = 'worst', overrun_tolerance: float = OVERRUN_MARGIN)
Settings of an analysis. Every field has a default.
Attributes:
| Name | Type | Description |
|---|---|---|
protocol |
str | None
|
Protocol to analyse. |
objective |
str | None
|
Objective to compare solvers on. |
sense |
Sense | None
|
|
solvers |
tuple[str, ...] | None
|
Names of the solvers to analyse, or |
problems |
tuple[str, ...] | None
|
Ids of the problems to analyse, or |
alpha |
float
|
Significance level of the tests and corrections. |
confidence |
float
|
Coverage of the confidence intervals. |
resamples |
int
|
Bootstrap resamples per interval. |
seed |
int
|
Seed of the bootstrap and of the power simulation. |
interval |
IntervalMethod
|
Bootstrap interval, |
correction |
Correction
|
Multiple-comparison correction of the pairwise tests. |
omnibus |
Omnibus
|
Omnibus test that gates the Nemenyi post-hoc test. |
statistic |
CellStatistic
|
How the replicates of a cell become one value, where a
failed run counts as the worst value: |
replicate_floor |
int | None
|
Fewest replicates per cell that the design should have for stochastic solvers. Cells below it are reported as a limitation; nothing is excluded. |
power |
float
|
Target power of the detectable-effect statement. |
references |
Mapping[str, float] | None
|
Reference value, such as a best-known objective value, of each problem for the relative percentage deviation. |
reference_source |
str | None
|
Where the reference values come from, shown with the deviation table. |
reference_points |
Mapping[str, tuple[float, ...]] | None
|
Hypervolume reference point of a problem, in objective units and minimisation form, overriding the rule. |
reference_rule |
ReferenceRule
|
Rule deriving the reference point from the
reference front: |
reference_parameter |
float
|
The margin (positive) or |
normalize |
bool
|
Compute indicators on objectives scaled so that the reference front's ideal is 0 and its nadir is 1. |
focal |
str | None
|
Name of a solver under study. A bundle then also compares it
with each baseline, adjusting the p-values over these comparisons
only (see |
baselines |
tuple[str, ...] | None
|
Names of the baselines of |
timeout_score |
TimeoutScore
|
How a run that never completed and was killed at its
hard limit is scored. |
overrun_tolerance |
float
|
Share of the time limit by which the wall time of
a completed run may pass the limit before the limitations name
the run: 0.05 allows 5 % more wall time than the limit. The
default is the share of the engine's note on a run,
|
dispatchatlas.analysis.summary
¶
The statistical comparison of the solvers of a campaign, under one protocol.
summarize() follows the non-parametric protocol for comparing
algorithms over several problems (Demšar 2006; Derrac et al. 2011):
- Cells. The replicates of each solver on each problem become one
score, the configured statistic (mean or median) of the planned runs,
where a run that did not succeed counts as the worst value (infinity for
a minimised objective). With
timeout_score"incumbent", a run killed at its hard limit counts instead by the feasible incumbent it held when its time limit passed. Descriptive statistics and a bootstrap interval of the mean describe the succeeded runs. - Reliability. Each solver's succeeded, infeasible, failed, and missing runs, over its planned runs.
- Ranks and omnibus test. On the complete blocks, the problems every analysed solver is planned on, the scores are ranked per problem (rank 1 best, ties sharing their mean rank). With at least three solvers and two blocks, the Friedman test and its Iman-Davenport form test whether the solvers differ, and Kendall's W, with a bootstrap interval over blocks, measures by how much.
- Post-hoc test. When the configured omnibus test rejects, the Nemenyi critical difference says which average ranks differ.
- Pairwise tests. Every pair of solvers is compared over the problems both are planned on with the Wilcoxon signed-rank test, and the p-values are adjusted over the family. Each comparison reports, beside its p-value, the paired win share with a bootstrap interval, the mean advantage with a bootstrap interval when no failure makes it infinite, and Cohen's d_z, as Carrasco et al. (2020) recommend.
- Limitations. What qualifies the results: failed or missing runs,
with the missing ones by cell of solver and problem and
the failed ones by solver and kind of error,
runs whose worker did not begin the solve in time,
runs killed at their hard limit and how they are scored,
what the plan notes on the protocol, such as solvers declared
deterministic that run once under a time limit,
stop reasons that differ among the runs of the iterative solvers,
completed runs whose wall time passed the time limit of the protocol,
the processor share and the concurrency the runs of a protocol with a
time limit executed under,
problems left out of the ranks, too few shared problems for the paired
test to reach significance, cells below a replicate floor, tests that
could not run, and what the check of the campaign directory found beyond
these (
check_campaign()), such as an altered campaign file, cells with records for only some of their runs, traces that cannot be read against time, or sessions on other code.
Two scores tie when they are equal within
TIE_TOLERANCE. A tied pair enters
the Wilcoxon test as a zero difference, so the test and the win counts
always agree on what is a tie. Every bootstrap uses the configured seed.
summarize
¶
summarize(dataset: Dataset, config: AnalysisConfig | None = None) -> Summary
Compare the solvers of a dataset under one protocol.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset
|
Dataset
|
The campaign's runs, from
|
required |
config
|
AnalysisConfig | None
|
The analysis settings; the defaults analyse the only protocol on the primary objective. |
None
|
Returns:
| Type | Description |
|---|---|
Summary
|
Cells, solver summaries, omnibus and post-hoc tests, pairwise |
Summary
|
comparisons, and limitations. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the configuration names something the dataset lacks, leaves the protocol or objective ambiguous, or asks for Shaffer's correction when some pair of solvers shares no problem. |
dispatchatlas.analysis.comparison
¶
One solver against a set of baselines.
compare_to_baselines() answers the question a new method raises: is it
better than each of the established ones? Every baseline is compared with
the focal solver by the paired Wilcoxon signed-rank test of
summarize(), and the p-values are
adjusted over these k comparisons only, the many-to-one family, rather
than over every pair of solvers (Demšar 2006, section 3.2.2). Every row is
oriented towards the focal solver.
compare_to_baselines
¶
compare_to_baselines(dataset: Dataset, focal: str, baselines: Sequence[str] | None = None, config: AnalysisConfig | None = None) -> BaselineComparison
Compare one solver with each baseline.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset
|
Dataset
|
The campaign's runs. |
required |
focal
|
str
|
Name of the solver under study. |
required |
baselines
|
Sequence[str] | None
|
Names of the baselines. |
None
|
config
|
AnalysisConfig | None
|
The analysis settings. Its |
None
|
Returns:
| Type | Description |
|---|---|
BaselineComparison
|
One row per baseline, with the focal solver on the left, adjusted |
BaselineComparison
|
with the configured correction over the baselines. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the focal solver or a baseline is unknown, a baseline is the focal solver or is named twice, a baseline shares no problem with the focal solver, or Shaffer's correction is asked for, which applies to all pairwise comparisons only. |
dispatchatlas.analysis.results
¶
The results of an analysis: frozen records that tables and figures render.
Directional quantities favour the left solver when positive, or above 0.5 for the win share, whatever the objective's sense: an advantage is how much better the left solver's value is, in objective units.
Limitation
dataclass
¶
A caveat that qualifies the results.
Attributes:
| Name | Type | Description |
|---|---|---|
topic |
str
|
What it concerns, for example |
message |
str
|
One or two sentences stating the caveat. |
CellSummary
dataclass
¶
CellSummary(solver: str, problem: str, planned: int, succeeded: int, score: float, mean: float | None, median: float | None, std: float | None, best: float | None, mean_low: float | None, mean_high: float | None)
The runs of one solver on one problem.
Attributes:
| Name | Type | Description |
|---|---|---|
solver |
str
|
Solver name. |
problem |
str
|
Problem id. |
planned |
int
|
Runs the plan holds for the cell. |
succeeded |
int
|
Runs that completed with a feasible schedule. |
score |
float
|
The value the comparisons use: the configured statistic of the planned runs, a failed run counting as the worst value, so it is infinite when failures decide it. |
mean |
float | None
|
Mean value of the succeeded runs, or |
median |
float | None
|
Median value of the succeeded runs. |
std |
float | None
|
Sample standard deviation of the succeeded runs, with at least two. |
best |
float | None
|
Best value of the succeeded runs. |
mean_low |
float | None
|
Lower confidence bound of the mean. |
mean_high |
float | None
|
Upper confidence bound of the mean. |
SolverSummary
dataclass
¶
SolverSummary(solver: str, planned: int, succeeded: int, infeasible: int, failed: int, missing: int, problems: int, average_rank: float | None)
The reliability and average rank of one solver.
Attributes:
| Name | Type | Description |
|---|---|---|
solver |
str
|
Solver name. |
planned |
int
|
Planned runs over the analysed problems. |
succeeded |
int
|
Runs that completed with a feasible schedule. |
infeasible |
int
|
Completed runs without a feasible schedule. |
failed |
int
|
Runs whose attempts all failed. |
missing |
int
|
Runs without a record yet. |
problems |
int
|
Problems the plan pairs the solver with. |
average_rank |
float | None
|
Mean rank over the complete blocks, rank 1 best, or
|
success_rate
property
¶
Share of the planned runs that succeeded; 0 without planned runs.
PairwiseComparison
dataclass
¶
PairwiseComparison(left: str, right: str, pairs: int, wins: int, losses: int, ties: int, win_share: float, win_share_low: float, win_share_high: float, median_advantage: float | None, mean_advantage: float | None, advantage_low: float | None, advantage_high: float | None, cohens_dz: float | None, statistic: float, p_value: float, adjusted_p_value: float, method: TestMethod, confidence: float)
A paired comparison of two solvers over the problems they share.
Each problem contributes one pair: the two solvers' cell scores. The Wilcoxon signed-rank test compares them, and the effect sizes and intervals beside the p-value say how large the difference is.
Attributes:
| Name | Type | Description |
|---|---|---|
left |
str
|
The solver the directional fields favour when positive. |
right |
str
|
The other solver. |
pairs |
int
|
Shared problems. |
wins |
int
|
Problems the left solver wins. |
losses |
int
|
Problems the left solver loses. |
ties |
int
|
Problems where the scores tie. |
win_share |
float
|
|
win_share_low |
float
|
Lower bootstrap bound of the win share. |
win_share_high |
float
|
Upper bootstrap bound of the win share. |
median_advantage |
float | None
|
Median over problems of how much better the left
score is, in objective units, or |
mean_advantage |
float | None
|
Mean advantage, or |
advantage_low |
float | None
|
Lower bootstrap bound of the mean advantage. |
advantage_high |
float | None
|
Upper bootstrap bound of the mean advantage. |
cohens_dz |
float | None
|
Mean advantage over its standard deviation, or |
statistic |
float
|
Wilcoxon statistic |
p_value |
float
|
Two-sided Wilcoxon p-value. |
adjusted_p_value |
float
|
The p-value adjusted over the family of comparisons. |
method |
TestMethod
|
How the p-value was obtained. |
confidence |
float
|
Coverage of the intervals. |
OmnibusResult
dataclass
¶
OmnibusResult(blocks: int, groups: int, friedman_statistic: float, friedman_p_value: float, iman_davenport_statistic: float, iman_davenport_p_value: float, kendalls_w: float, kendalls_w_low: float, kendalls_w_high: float, confidence: float, test: str, alpha: float, significant: bool)
The Friedman and Iman-Davenport tests over the complete blocks.
Attributes:
| Name | Type | Description |
|---|---|---|
blocks |
int
|
Complete blocks: problems every analysed solver is planned on. |
groups |
int
|
Solvers. |
friedman_statistic |
float
|
Friedman's |
friedman_p_value |
float
|
Chi-square p-value of |
iman_davenport_statistic |
float
|
Iman and Davenport's |
iman_davenport_p_value |
float
|
F p-value. |
kendalls_w |
float
|
Kendall's coefficient of concordance, the effect size. |
kendalls_w_low |
float
|
Lower bootstrap bound of |
kendalls_w_high |
float
|
Upper bootstrap bound of |
confidence |
float
|
Coverage of the interval. |
test |
str
|
The test whose p-value decides significance. |
alpha |
float
|
Significance level. |
significant |
bool
|
Whether that p-value is below |
CriticalDifference
dataclass
¶
CriticalDifference(value: float, alpha: float, groups: int, blocks: int, average_ranks: tuple[tuple[str, float], ...], significant_pairs: tuple[tuple[str, str], ...], cliques: tuple[tuple[str, ...], ...])
The Nemenyi post-hoc comparison of average ranks.
Attributes:
| Name | Type | Description |
|---|---|---|
value |
float
|
The critical difference. |
alpha |
float
|
Significance level. |
groups |
int
|
Solvers. |
blocks |
int
|
Complete blocks. |
average_ranks |
tuple[tuple[str, float], ...]
|
Each solver's average rank, best first. |
significant_pairs |
tuple[tuple[str, str], ...]
|
Pairs, best first, whose ranks differ by more than the critical difference. |
cliques |
tuple[tuple[str, ...], ...]
|
Maximal runs of solvers, in rank order, whose ranks lie within the critical difference of each other, the bars of a critical-difference diagram. Singletons are included. |
Summary
dataclass
¶
Summary(campaign: str, protocol: str, objective: str, sense: Sense, config: AnalysisConfig, solvers: tuple[str, ...], problems: tuple[str, ...], blocks: tuple[str, ...], cells: tuple[CellSummary, ...], solver_summaries: tuple[SolverSummary, ...], pairwise: tuple[PairwiseComparison, ...], omnibus: OmnibusResult | None, critical_difference: CriticalDifference | None, limitations: tuple[Limitation, ...])
Every result of summarize() for one protocol.
Attributes:
| Name | Type | Description |
|---|---|---|
campaign |
str
|
Campaign id. |
protocol |
str
|
The analysed protocol. |
objective |
str
|
The analysed objective. |
sense |
Sense
|
Its sense. |
config |
AnalysisConfig
|
The configuration used. |
solvers |
tuple[str, ...]
|
Analysed solvers, in campaign order. |
problems |
tuple[str, ...]
|
Analysed problems, in plan order. |
blocks |
tuple[str, ...]
|
The problems every analysed solver is planned on. |
cells |
tuple[CellSummary, ...]
|
One summary per planned solver and problem. |
solver_summaries |
tuple[SolverSummary, ...]
|
One summary per solver, in campaign order. |
pairwise |
tuple[PairwiseComparison, ...]
|
One comparison per pair of solvers sharing a problem. |
omnibus |
OmnibusResult | None
|
The omnibus tests, or |
critical_difference |
CriticalDifference | None
|
The post-hoc test, or |
limitations |
tuple[Limitation, ...]
|
The caveats that qualify these results. |
comparison
¶
comparison(left: str, right: str) -> PairwiseComparison
Return the comparison of two solvers, oriented towards left.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the two solvers were not compared. |
BaselineComparison
dataclass
¶
BaselineComparison(focal: str, baselines: tuple[str, ...], protocol: str, objective: str, sense: Sense, correction: Correction, alpha: float, rows: tuple[PairwiseComparison, ...], better_than_all: bool, limitations: tuple[Limitation, ...])
One solver compared with each of several baselines.
Attributes:
| Name | Type | Description |
|---|---|---|
focal |
str
|
The solver under study; every row has it on the left. |
baselines |
tuple[str, ...]
|
The baselines, in campaign order. |
protocol |
str
|
The analysed protocol. |
objective |
str
|
The analysed objective. |
sense |
Sense
|
Its sense. |
correction |
Correction
|
The correction applied over the rows. |
alpha |
float
|
Significance level. |
rows |
tuple[PairwiseComparison, ...]
|
One comparison per baseline, adjusted over this family only. |
better_than_all |
bool
|
Whether every row is significant at |
limitations |
tuple[Limitation, ...]
|
The caveats of the underlying summary. |
dispatchatlas.analysis.deviation
¶
Relative percentage deviation from reference values.
Scheduling benchmarks usually publish a best-known or optimal value per instance, and results are compared on the scale-free relative percentage deviation (RPD) from it (Vallada, Ruiz and Framinan 2015):
RPD = 100 (value - reference) / referencefor a minimised objective and100 (reference - value) / referencefor a maximised one, so a positive RPD is always worse than the reference and a negative one improves on it;- the average RPD (ARPD) of a solver is the mean over all of its runs on all problems together.
Only succeeded runs have a value to deviate from. Each row reports how many of the planned runs that is, so an ARPD over few surviving runs is visible as such.
DeviationRow
dataclass
¶
DeviationRow(solver: str, problems: int, runs: int, planned: int, arpd: float | None, rpd_min: float | None, rpd_max: float | None)
The average relative percentage deviation of one solver.
Attributes:
| Name | Type | Description |
|---|---|---|
solver |
str
|
Solver name. |
problems |
int
|
Problems with at least one succeeded run. |
runs |
int
|
Succeeded runs, whose deviations are averaged. |
planned |
int
|
Planned runs. |
arpd |
float | None
|
Mean deviation over the succeeded runs, in percent, or
|
rpd_min |
float | None
|
Smallest deviation. |
rpd_max |
float | None
|
Largest deviation. |
relative_deviation
¶
relative_deviation(value: float, reference: float, sense: Sense = 'min') -> float
Return the relative percentage deviation of a value from a reference.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value
|
float
|
The objective value. |
required |
reference
|
float
|
The reference value, such as the best known, positive. |
required |
sense
|
Sense
|
Whether the objective is minimised or maximised. |
'min'
|
Returns:
| Type | Description |
|---|---|
float
|
The deviation in percent, positive when worse than the reference. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the reference is not positive or a value is not finite. |
arpd
¶
arpd(dataset: Dataset, config: AnalysisConfig) -> tuple[DeviationRow, ...]
Return the average relative percentage deviation of every solver.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset
|
Dataset
|
The campaign's runs. |
required |
config
|
AnalysisConfig
|
The analysis settings; |
required |
Returns:
| Type | Description |
|---|---|
tuple[DeviationRow, ...]
|
One row per analysed solver, in campaign order. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the configuration has no references or lacks one for an analysed problem, or the scope cannot be resolved. |
dispatchatlas.analysis.indicators
¶
Multi-objective quality indicators: hypervolume, IGD, and IGD+.
On every problem with at least two objectives, indicator_report() scores each
run's approximation set, the front a multi-objective solver records or the
single objective vector of any other solver's best schedule, then aggregates
the runs of each solver:
- Objective vectors are put in minimisation form.
- The reference set of the problem is the non-dominated union of every run's set, over all analysed solvers. Its per-objective minimum is the ideal point and its maximum the nadir point.
- With
normalize(the default), every objective is rescaled so that the ideal is 0 and the nadir 1; an objective on which the reference set does not vary is only shifted. - The reference point is shared by every solver on the problem. It is
the configured point, if any, or it is derived from the ideal and nadir:
the
"margin"rule places itmspans beyond the nadir (r = nadir + m (nadir - ideal), defaultm = 0.1), the"ishibuchi"rule atr = ideal + (1 + 1/h)(nadir - ideal)(Ishibuchi et al. 2018). On an objective without spread, both rules move|nadir| m, ormwhen the nadir is 0, beyond it (m = 1/hfor the second rule). - Each run gets the hypervolume of its set, the volume it dominates up
to the reference point (Zitzler and Thiele 1999), computed exactly by
dispatchatlas.core.pareto.hypervolume(); the IGD, the mean Euclidean distance from each reference-set point to the nearest point of the run's set (Coello Coello and Reyes Sierra 2004); and the IGD+, the same mean with the distance counting only the objectives in which the run's point is worse (Ishibuchi et al. 2015). Hypervolume is to be maximised, IGD and IGD+ minimised. - A run that did not succeed has an empty set: hypervolume 0 and infinite IGD and IGD+. The configured statistic (mean or median) of the planned runs gives each solver's value on the problem, so failures count.
A reference point derived from the data moves when solvers are added or
removed, so hypervolumes are comparable only between results that share
their reference points; ensure_shared_reference_points() checks it.
ReferencePoint
dataclass
¶
ReferencePoint(problem: str, objectives: tuple[str, ...], ideal: Point, nadir: Point, point: Point, rule: Literal['fixed', 'margin', 'ishibuchi'], parameter: float | None, normalized: bool)
The frame in which one problem's indicators are computed.
Attributes:
| Name | Type | Description |
|---|---|---|
problem |
str
|
Problem id. |
objectives |
tuple[str, ...]
|
The objectives, in the problem's order. |
ideal |
Point
|
Per-objective minimum of the reference set, in minimisation form. |
nadir |
Point
|
Per-objective maximum of the reference set, in minimisation form. |
point |
Point
|
The hypervolume reference point, in objective units and minimisation form. |
rule |
Literal['fixed', 'margin', 'ishibuchi']
|
|
parameter |
float | None
|
The rule's margin or |
normalized |
bool
|
Whether the indicators use normalised objectives. |
IndicatorRow
dataclass
¶
IndicatorRow(solver: str, problem: str, planned: int, succeeded: int, hypervolume: float, igd: float, igd_plus: float)
One solver's indicators on one problem, aggregated over its planned runs.
Attributes:
| Name | Type | Description |
|---|---|---|
solver |
str
|
Solver name. |
problem |
str
|
Problem id. |
planned |
int
|
Planned runs. |
succeeded |
int
|
Runs with a feasible result. |
hypervolume |
float
|
Hypervolume, higher is better. |
igd |
float
|
Inverted generational distance, lower is better. |
igd_plus |
float
|
IGD+, lower is better. |
IndicatorReport
dataclass
¶
IndicatorReport(protocol: str, statistic: str, rows: tuple[IndicatorRow, ...], reference_points: tuple[ReferencePoint, ...], limitations: tuple[Limitation, ...])
The indicators of every solver on every multi-objective problem.
Attributes:
| Name | Type | Description |
|---|---|---|
protocol |
str
|
The analysed protocol. |
statistic |
str
|
How runs were aggregated. |
rows |
tuple[IndicatorRow, ...]
|
One row per planned solver and problem, by problem then solver. |
reference_points |
tuple[ReferencePoint, ...]
|
The frame of each problem. |
limitations |
tuple[Limitation, ...]
|
Problems that could not be scored, and why. |
reference_point
¶
reference_point(points: Sequence[Sequence[float]], rule: ReferenceRule = 'margin', parameter: float = 0.1) -> Point
Derive a hypervolume reference point from a set of minimisation vectors.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
points
|
Sequence[Sequence[float]]
|
The reference front, one vector per point. |
required |
rule
|
ReferenceRule
|
|
'margin'
|
parameter
|
float
|
The margin (positive) or |
0.1
|
Returns:
| Type | Description |
|---|---|
Point
|
The reference point. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If |
igd
¶
Return the inverted generational distance of a front to a reference set.
The mean over reference points of the Euclidean distance to the nearest front point; infinite for an empty front.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the reference set is empty or the vectors are ragged. |
igd_plus
¶
Return the IGD+ of a front to a reference set, in minimisation form.
Like igd(), but the distance from a reference point z to a front
point a is sqrt(sum_k max(a_k - z_k, 0)^2): only the objectives
in which a is worse count, which makes IGD+ weakly Pareto compliant.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the reference set is empty or the vectors are ragged. |
indicator_report
¶
indicator_report(dataset: Dataset, config: AnalysisConfig | None = None) -> IndicatorReport
Return the indicators of every solver on every problem with several objectives.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset
|
Dataset
|
The campaign's runs. |
required |
config
|
AnalysisConfig | None
|
The analysis settings: protocol, selection, aggregation statistic, reference points or rule, and normalisation. |
None
|
Returns:
| Type | Description |
|---|---|
IndicatorReport
|
The rows and the reference point of each problem. Problems with one |
IndicatorReport
|
objective are skipped. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the scope cannot be resolved, an objective is not registered, or a configured reference point does not fit its problem. |
ensure_shared_reference_points
¶
ensure_shared_reference_points(reports: Sequence[IndicatorReport]) -> None
Check that reports scoring the same problem used the same frame.
Hypervolumes measured against different reference points, or IGD values computed on differently normalised objectives, are on different scales.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If two reports score a problem against different reference points, rules, or normalisations. |
dispatchatlas.analysis.methods
¶
The catalog of the statistical methods DispatchAtlas implements.
Each Method names the function that implements it, the assumptions
its result rests on, and the sources it follows, as keys of the bibliography
in dispatchatlas.references. The analysis bundle lists the methods it
used with their references, and the documentation renders the catalog.
MethodFamily
¶
MethodFamily = Literal['test', 'post-hoc', 'correction', 'effect-size', 'interval', 'bayesian', 'power', 'deviation', 'indicator', 'reporting']
The role a method plays in an analysis.
Method
dataclass
¶
Method(id: str, name: str, family: MethodFamily, summary: str, assumptions: tuple[str, ...], references: tuple[str, ...], function: str)
One implemented statistical method.
Attributes:
| Name | Type | Description |
|---|---|---|
id |
str
|
Catalog key in kebab case. |
name |
str
|
Human-readable name. |
family |
MethodFamily
|
The role the method plays. |
summary |
str
|
One sentence on what the method computes. |
assumptions |
tuple[str, ...]
|
What the result rests on. |
references |
tuple[str, ...]
|
Bibliography keys of the sources. |
function |
str
|
Dotted path of the function that implements it. |
implemented_methods
¶
implemented_methods(family: MethodFamily | None = None) -> tuple[Method, ...]
Return the implemented methods, all of them or those of one family.
dispatchatlas.analysis.tables
¶
Tables of analysis results, rendered as Markdown, CSV, or LaTeX.
A Table holds typed columns and raw values; the renderers format
them. Markdown and LaTeX show six significant digits (three for p-values),
an em dash for a missing value, and intervals as [low, high]. CSV keeps
full precision, writes an interval as two columns <key>_low and
<key>_high, and names columns by their keys, one row per record.
Every renderer refuses a table that reports a p-value without an effect size
and a confidence interval beside it, as Carrasco et al. (2020) recommend:
check_p_values() enforces it from the column kinds.
ColumnKind
¶
What a column holds, which decides its formatting and the p-value check.
Cell
¶
A table value; intervals are (low, high) pairs.
Column
dataclass
¶
Column(key: str, header: str, kind: ColumnKind = 'text')
A table column.
Attributes:
| Name | Type | Description |
|---|---|---|
key |
str
|
Machine name, used as the CSV header. |
header |
str
|
Human-readable header, used in Markdown and LaTeX. |
kind |
ColumnKind
|
What the column holds. |
Table
dataclass
¶
Table(name: str, title: str, columns: tuple[Column, ...], rows: tuple[tuple[Cell, ...], ...], notes: tuple[str, ...] = ())
A titled table of typed columns.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
File stem, for example |
title |
str
|
One-line caption. |
columns |
tuple[Column, ...]
|
The columns. |
rows |
tuple[tuple[Cell, ...], ...]
|
One value per column in every row. |
notes |
tuple[str, ...]
|
Sentences printed under the table in Markdown and as comments in LaTeX. |
check_p_values
¶
check_p_values(table: Table) -> None
Refuse a table that reports a p-value without an effect size and an interval.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the table has a p-value column but no effect-size column or no interval column. |
to_markdown
¶
to_markdown(table: Table) -> str
Render a table as GitHub-flavoured Markdown, with its title and notes.
to_csv
¶
to_csv(table: Table) -> str
Render a table as CSV at full precision, headed by the column keys.
to_latex
¶
to_latex(table: Table) -> str
Render a table as a LaTeX tabular with booktabs rules.
The title and notes become comments above and below, so the fragment can
be placed in any float. It needs \usepackage{booktabs}.
runs_table
¶
runs_table(dataset: Dataset, timeout_score: TimeoutScore = 'worst') -> Table
Return one row per run of the dataset: its facts and every objective value.
A failed run names the kind of its last failed attempt. The column
scored_by says where the score of a run comes from: its result,
the worst value, or, with timeout_score "incumbent", the
incumbent a run killed at its hard limit held when its time limit
passed. The objective columns of such a run hold the values of that
incumbent, and its evaluations and elapsed columns say when its
search found it, which is within the time limit.
The columns from time_limit to attempts compare a run with its
budget and say under which load it executed. time_limit is the time
limit of the run's protocol. wall_seconds is the wall time the
engine measured for the last attempt of the run, the completing one or
the last failed one, where elapsed is the solver's own report.
overrun_seconds is the wall time minus the time limit, negative for
a run that ended early. cpu_share is the processor seconds of the
attempt over its wall seconds (Run.cpu_share): below 1 when the
executing process was not computing for part of the run, whether it
waited for a processor, slept, or left the work to another process, and
above 1 when the solver ran threads. concurrent is the number of
runs executing when the attempt started, itself included, cpu the
processor its worker was pinned to, and attempts the number of the
last attempt. A cell is empty when the protocol has no time limit or
the record does not state the value.
For a run that did not complete, cpu_seconds, wall_seconds,
concurrent, cpu, and attempts describe its last failed
attempt, and overrun_seconds and cpu_share are empty: what ended
that attempt is in the failure column, and its times say nothing
about a budget.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset
|
Dataset
|
The runs. |
required |
timeout_score
|
TimeoutScore
|
How the analysis the table belongs to scores a run that was killed at its hard limit. |
'worst'
|
cells_table
¶
Return the per-problem results of every solver.
solvers_table
¶
Return each solver's reliability and average rank, best rank first.
pairwise_table
¶
Return every pairwise comparison with its effect sizes and intervals.
baselines_table
¶
baselines_table(comparison: BaselineComparison) -> Table
Return the comparisons of a focal solver with its baselines.
omnibus_table
¶
Return the omnibus tests with Kendall's W, and the post-hoc test as notes.
deviation_table
¶
deviation_table(rows: Sequence[DeviationRow], objective: str, source: str | None = None) -> Table
Return the average relative percentage deviation of every solver.
indicators_table
¶
indicators_table(report: IndicatorReport) -> Table
Return every solver's hypervolume, IGD, and IGD+ on every multi-objective problem.
dispatchatlas.analysis.manifest
¶
The reproducibility manifest of an analysis bundle.
manifest.json records what a reader needs to repeat or to judge an
analysis:
- the campaign: id, config hash, seed, replicates, the digest of the analysed plan's runs, and run counts by status, and, when the campaign sets them, its seed namespace, its replication under a time limit, and the solvers' own replicate counts;
- the full
AnalysisConfig; - the DispatchAtlas version and the environment, as
capture_environment()records it; - for each analysed protocol, the two sample sizes of the design: the replicates behind each per-problem value and the problems behind each paired test, with the smallest standardized effect the paired tests detect at the configured power, or the reason none is detectable;
- whether figures were rendered;
- the SHA-256 digest of every other file of the bundle.
The manifest holds no timestamp, so equal inputs give an identical file on the same machine and software.
MANIFEST_ARTIFACT
module-attribute
¶
The artifact field that identifies an analysis bundle manifest.
sha256
¶
Return the SHA-256 hex digest of bytes, or of a file's content.
analysis_record
¶
Describe the design behind one protocol's summary.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset
|
Dataset
|
The dataset the summary was computed from. |
required |
summary
|
Summary
|
The summary. |
required |
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
A JSON-ready dictionary: protocol, objective and sense, the analysed |
dict[str, Any]
|
solvers and problems, the complete blocks, the replicates per cell, |
dict[str, Any]
|
the problems per paired test, and the power statement. |
build_manifest
¶
build_manifest(dataset: Dataset, config: AnalysisConfig, analyses: list[dict[str, Any]], files: Mapping[str, str], figures: Mapping[str, Any]) -> dict[str, Any]
Return the manifest document of a bundle.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset
|
Dataset
|
The analysed dataset, before narrowing to a protocol. |
required |
config
|
AnalysisConfig
|
The configuration of the analysis. |
required |
analyses
|
list[dict[str, Any]]
|
One |
required |
files
|
Mapping[str, str]
|
SHA-256 digest of every other file, keyed by path relative to
the bundle, with |
required |
figures
|
Mapping[str, Any]
|
Whether figures were rendered, and why not. |
required |
dispatchatlas.analysis.bundle
¶
The analysis bundle: every table, figure, and caveat of a campaign in one directory.
write_bundle() analyses a campaign directory and writes:
<output>/
index.md what the bundle holds: tables, figures, methods, and sources
limitations.md the caveats of every analysis
runs.csv one row per analysed run, at full precision
tables/<protocol>/<table>.md, .csv, .tex
figures/<protocol>/... SVG and TikZ figures, when the figure package is available
manifest.json configuration, versions, environment, and file digests
Each protocol is analysed separately (see
summarize()). Tables with a p-value
always carry an effect size and an interval beside it. The multi-objective
indicator table appears when a problem has several objectives, the
deviation table when the configuration gives reference values, and the
baselines table, the focal solver against each baseline (see
compare_to_baselines()), when it
names a focal solver. Figures are drawn with the same settings as the
tables, in the formats asked for, and the index lists each with its caption
and the sources of its method.
Output is byte-reproducible: equal inputs, configuration, and software give
identical files. Content files hold no timestamps or paths, JSON is written
with sorted keys, and text with \n line endings; the environment appears
in the manifest only.
MANIFEST_FILE
module-attribute
¶
Name of the manifest in a bundle directory.
FIGURE_FORMATS
module-attribute
¶
The formats figures can be written in: SVG, and TikZ for LaTeX.
write_bundle
¶
write_bundle(campaign_directory: str | Path, output_directory: str | Path, config: AnalysisConfig | None = None, *, figures: bool = True, formats: Sequence[str] = FIGURE_FORMATS) -> Path
Analyse a campaign and write the bundle.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
campaign_directory
|
str | Path
|
A campaign directory written by
|
required |
output_directory
|
str | Path
|
Where to write the bundle. It must be absent, empty, or hold an earlier bundle, whose files are replaced once the analysis has succeeded. |
required |
config
|
AnalysisConfig | None
|
The analysis settings. Without a protocol, every protocol of the campaign is analysed. With a focal solver, the bundle holds the baselines table of every protocol. |
None
|
figures
|
bool
|
Render figures through
|
True
|
formats
|
Sequence[str]
|
Formats of the figures, from |
FIGURE_FORMATS
|
Returns:
| Type | Description |
|---|---|
Path
|
The bundle directory. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the output directory holds other files, a format is unknown, or the configuration does not fit the campaign. |
SerializationError
|
If the campaign directory cannot be read. |