Skip to content

Campaigns

Campaign configuration and files, planning, fairness, estimation, execution, records, the result store, and replay.

dispatchatlas.campaigns

Reproducible, resumable experiment campaigns.

A campaign runs every solver of a roster on every problem of its instance sources, under every stopping protocol, with seeds shared across solvers, and stores one JSON record per run:

from dispatchatlas.campaigns import load_campaign, read_runs, run

campaign = load_campaign("campaign.toml")
summary = run(campaign, "results")  # resumes when called again
records = read_runs(summary.directory)

The modules, in the order a campaign meets them:

dispatchatlas.campaigns.config

Campaign configuration: the instances, solvers, protocols, and settings of an experiment.

A Campaign states what determines an experiment's results and, separately, how to execute it:

  • instances: one or more instance sources, see InstanceSource;
  • solvers: registered solver ids with parameter values. A label tells two parameter settings of one solver apart;
  • protocols: named stop criteria. Every solver runs under every protocol with the same criteria, so each protocol is one fair comparison;
  • replicates: runs per stochastic solver, problem, and protocol. Deterministic solvers run once, unless replicate_timed gives them every replicate under a time limit or their entry states its own count;
  • seed: the root of every run seed, and seed_namespace, an optional name that separates the seeds of campaigns that share a root;
  • objectives: objectives that replace those the problems declare, if given;
  • execution and metadata: how to execute, and free annotations.

The config hash identifies an experiment. It is the SHA-256 digest of the canonical JSON of Campaign.identity(): the fields seed, instances, solvers, protocols, replicates, and objectives as written, so reordering a list also changes it, and each optional setting that is set to other than its default. Three things stay outside it. The campaign id only names where the results are stored. execution and metadata do not change what a run computes. And the locations of instance files and directories are left out of instances: a problem read from a file is identified by its content, whose digest enters the id of every run on it (see planning), so a campaign keeps its identity when its files move or live elsewhere on another machine.

SCHEMA_VERSION module-attribute

SCHEMA_VERSION = 1

Version of the campaign file format this release reads and writes.

DEFAULT_REPLICATES module-attribute

DEFAULT_REPLICATES = 30

Runs per stochastic solver, problem, and protocol when a campaign does not say.

DEFAULT_TIMEOUT_FACTOR module-attribute

DEFAULT_TIMEOUT_FACTOR = 1.0

Default multiple of a protocol's time limit after which a run is killed.

GRACE_SECONDS module-attribute

GRACE_SECONDS = 5.0

Default seconds a run may exceed timeout_factor * time_limit before it is killed.

The grace lets a solver that honours its limit finish the step it is in and return its result.

STARTUP_SECONDS module-attribute

STARTUP_SECONDS = 120.0

Default seconds a worker process has, from the hand-over of a run, to begin its solve.

They cover the start of the worker's interpreter, the loading of plugins, and the rebuilding of the solver and the problem, on a machine that may be busy.

CPU_RESERVE module-attribute

CPU_RESERVE = 1

Processors that workers = "auto" leaves to the supervising process and the system.

BACKENDS module-attribute

BACKENDS = ('sequential', 'process')

Execution backends: in the calling process, or in supervised worker processes.

THREAD_VARIABLES module-attribute

THREAD_VARIABLES = ('OMP_NUM_THREADS', 'MKL_NUM_THREADS', 'OPENBLAS_NUM_THREADS', 'BLIS_NUM_THREADS', 'VECLIB_MAXIMUM_THREADS', 'NUMEXPR_NUM_THREADS')

Environment variables that limit the thread pools of numerical libraries.

Every worker process starts with each of them set to the campaign's threads_per_worker.

InstanceSource

Where a campaign's problems come from.

FamilySource dataclass

FamilySource(family: str, seeds: tuple[int, ...], parameters: Mapping[str, Any] = dict())

Problems generated by a registered benchmark family, one per seed.

Attributes:

Name Type Description
family str

Id of a family in the benchmark registry, for example "flow-shop".

seeds tuple[int, ...]

Generator seeds. Each gives one problem, identified by the id the family assigns.

parameters Mapping[str, Any]

Family parameters; the others keep their defaults.

to_dict

to_dict() -> dict[str, Any]

Return the source as a JSON-ready dictionary.

identity

identity() -> dict[str, Any]

Return the fields of the source that the config hash covers.

arguments

arguments() -> dict[str, Any]

Return the parameters as keyword arguments: tables as dictionaries, lists as tuples.

SuiteSource dataclass

SuiteSource(suite: str, instances: tuple[str, ...], directory: Path | None = None)

Instances of a published suite from dispatchatlas.benchmarks.suites.

Attributes:

Name Type Description
suite str

Suite id, for example "taillard-flow-shop".

instances tuple[str, ...]

Instance names, for example ("ta001", "ta002").

directory Path | None

Directory holding the downloaded files of a file suite. Generated suites need none.

to_dict

to_dict() -> dict[str, Any]

Return the source as a JSON-ready dictionary.

identity

identity() -> dict[str, Any]

Return the fields of the source that the config hash covers: not the directory.

CatalogSource dataclass

CatalogSource(catalog: str, seed: int = 0)

Every instance of a built-in catalog, see build_catalog().

Attributes:

Name Type Description
catalog str

Catalog name, for example "smoke".

seed int

Root seed the catalog derives its instance seeds from.

to_dict

to_dict() -> dict[str, Any]

Return the source as a JSON-ready dictionary.

identity

identity() -> dict[str, Any]

Return the fields of the source that the config hash covers.

DirectorySource dataclass

DirectorySource(directory: Path)

Problems written by materialize().

Every file is checked against the SHA-256 digest its manifest records.

Attributes:

Name Type Description
directory Path

The directory holding manifest.json.

to_dict

to_dict() -> dict[str, Any]

Return the source as a JSON-ready dictionary.

identity

identity() -> dict[str, Any]

Return the fields of the source that the config hash covers: not the directory.

FileSource dataclass

FileSource(file: Path, format: str | None = None)

One instance file.

Attributes:

Name Type Description
file Path

A DispatchAtlas problem document (.json), or an instance in a published format read by load_instance().

format str | None

Format name for load_instance(), when the extension does not imply it.

to_dict

to_dict() -> dict[str, Any]

Return the source as a JSON-ready dictionary.

identity

identity() -> dict[str, Any]

Return the fields of the source that the config hash covers: not the path.

SolverSpec dataclass

SolverSpec(id: str, label: str | None = None, parameters: Mapping[str, Any] = dict(), replicates: int | None = None)

A registered solver with parameter values.

Attributes:

Name Type Description
id str

Id in the solver registry, for example "simulated-annealing".

label str | None

Name that tells this setting apart from other settings of the same solver. Defaults to the id.

parameters Mapping[str, Any]

Solver parameters; the others keep their defaults.

replicates int | None

Runs of this solver per problem and protocol, whatever the solver declares and whatever the protocol bounds. None leaves the count to the campaign: its replicates for a stochastic solver, and for one declared deterministic a single run, or every replicate under a time limit when the campaign sets replicate_timed.

name property

name: str

The label, or the id when there is none. Unique within a campaign.

to_dict

to_dict() -> dict[str, Any]

Return the specification as a JSON-ready dictionary.

replicates is part of it only when the specification sets one.

arguments

arguments() -> dict[str, Any]

Return the parameters as keyword arguments: tables as dictionaries, lists as tuples.

Solver instances built from them can be pickled, which the process backend needs.

Protocol dataclass

Protocol(name: str, stop: StopCriteria)

Named stop criteria that every solver of a campaign runs under.

Attributes:

Name Type Description
name str

Protocol name, used in result file names.

stop StopCriteria

The criteria. They must bound the run with a time, evaluation, or iteration limit, because iterative solvers never stop otherwise.

to_dict

to_dict() -> dict[str, Any]

Return the protocol as a JSON-ready dictionary.

Execution dataclass

Execution(backend: str = 'process', workers: int | Literal['auto'] = 1, timeout_factor: float | None = DEFAULT_TIMEOUT_FACTOR, grace: float = GRACE_SECONDS, retries: int = 0, record_schedules: bool = True, affinity: Literal['inherit', 'pin'] | tuple[int, ...] = 'inherit', threads_per_worker: int = 1, startup_limit: float = STARTUP_SECONDS, on_changed_code: Literal['warn', 'refuse'] = 'warn', on_changed_problem: Literal['rerun', 'refuse'] = 'rerun', log_level: Literal['inherit', 'info', 'debug'] = 'inherit')

How a campaign executes. None of these settings changes a run's result.

Attributes:

Name Type Description
backend str

"process" runs each run in one of workers supervised worker processes, which enforces hard time limits and survives a crashing solver. "sequential" runs in the calling process, one run at a time, relying on solvers to honour their time limits; it suits debugging and solvers that cannot be sent to another process.

workers int | Literal['auto']

Worker processes, or "auto" for the processors the calling process may use minus CPU_RESERVE. Must be 1 for the sequential backend.

timeout_factor float | None

A run whose protocol sets time_limit is killed timeout_factor * time_limit + grace seconds after its solve begins. At least 1. None disables hard limits.

grace float

Seconds added to the hard limit.

retries int

Further attempts for a run whose failure may be transient.

record_schedules bool

Store every run's best schedule in its record. Its digest is stored either way.

affinity Literal['inherit', 'pin'] | tuple[int, ...]

The processors the worker processes run on. "inherit" leaves every worker on the processors the calling process may use. "pin" confines worker k to the k-th of those processors, in index order, whichever cores they belong to. A list of processor indices confines worker k to its k-th entry. A pinned campaign needs the process backend and a processor of its own for every worker.

threads_per_worker int

Threads the numerical libraries of a worker process may use. Every variable of THREAD_VARIABLES is set to it in the environment a worker starts with, for any number of workers and whatever the calling environment holds. The sequential backend has no worker processes and leaves the environment of the calling process as it is, so it accepts only the default, which then states nothing about that process: set the variables before Python starts to limit such a run.

startup_limit float

Seconds a worker process has, from the hand-over of a run, to begin its solve. A worker that takes longer is killed, and the attempt is a start-up failure, which retries covers and which is no timeout of the solver. Positive.

on_changed_code Literal['warn', 'refuse']

What a session does when the code it runs on differs from the code of the last recorded session of the campaign directory: the Python version, the version of a recorded package, the library's distribution, revision, or source digest, a plugin distribution, or the git state of the library or of the working directory. "warn" logs a warning that names each difference and records them in the session; "refuse" ends the session before it writes anything. A difference of execution settings or of the machine is warned about and recorded under either value.

on_changed_problem Literal['rerun', 'refuse']

What a session does when the directory holds the record of a planned run made on another version of its problem. "rerun" runs the run again and moves the record aside when the new one is written; "refuse" ends the session before it moves or writes anything.

log_level Literal['inherit', 'info', 'debug']

Which records of the library's loggers the session log receives beside the engine's own, which it receives from the level INFO. "inherit" leaves the loggers as they are: the log receives what they let through, which is WARNING and higher unless the application lowered a level, and in a worker process unless something configured logging there. "info" and "debug" lower the library's loggers to that level for the session, in the calling process and in every worker, while the handlers of the application show what they showed before. A lowered level costs time inside a run: every record of a solver from that level on is made and handed over, so a solver that logs in its loop does less work in the same time limit.

max_attempts property

max_attempts: int

Attempts a run gets before a failure that may be transient settles it.

worker_count

worker_count() -> int

Return the number of workers, resolving "auto" on this machine.

worker_cpus

worker_cpus(workers: int | None = None) -> tuple[int, ...] | None

Return the processor each worker is pinned to, resolved on this machine.

Parameters:

Name Type Description Default
workers int | None

Number of workers. Defaults to worker_count().

None

Returns:

Type Description
tuple[int, ...] | None

One processor index per worker, in worker order, or None when

tuple[int, ...] | None

affinity is "inherit".

Raises:

Type Description
ConfigError

If the platform cannot pin a process, the workers outnumber the processors they are pinned to, or a listed processor is not one the calling process may use.

to_dict

to_dict() -> dict[str, Any]

Return the settings as a JSON-ready dictionary.

Campaign dataclass

Campaign(id: str, seed: int, instances: tuple[InstanceSource, ...], solvers: tuple[SolverSpec, ...], protocols: tuple[Protocol, ...], replicates: int = DEFAULT_REPLICATES, objectives: tuple[str, ...] | None = None, execution: Execution = Execution(), metadata: Mapping[str, Any] = dict(), base: Path | None = None, replicate_timed: bool = False, seed_namespace: str | None = None)

A reproducible experiment: problems, solvers, protocols, and replicates.

Attributes:

Name Type Description
id str

Campaign name. Results go to <root>/<id>/.

seed int

Root seed of every run seed.

instances tuple[InstanceSource, ...]

Where the problems come from.

solvers tuple[SolverSpec, ...]

The solvers to compare, with unique names.

protocols tuple[Protocol, ...]

Stop criteria every solver runs under, with unique names.

replicates int

Runs per stochastic solver, problem, and protocol.

objectives tuple[str, ...] | None

Objectives, primary first, that replace those every problem declares. None keeps the problems' own.

execution Execution

How to execute. Not part of the config hash.

metadata Mapping[str, Any]

Free annotations. Not part of the config hash.

base Path | None

Directory that relative paths in instance sources refer to; the directory of the campaign file when read by load_campaign(), else the working directory. Not part of the config hash.

replicate_timed bool

Whether a solver declared deterministic runs replicates times, like a stochastic one, under every protocol that sets a time limit. Where the clock ends such a run, its result depends on the speed and load of the machine, and a single run does not show by how much.

seed_namespace str | None

A name that separates the seeds of this campaign from those of every campaign with the same seed and another namespace, or none. None leaves every seed as the campaign seed alone gives it.

config_hash cached property

config_hash: str

SHA-256 hex digest of the canonical JSON of identity().

identity

identity() -> dict[str, Any]

Return the fields that determine the results, as JSON-ready values.

These are exactly the fields the config hash covers: seed, instances without file-system locations, solvers, protocols, replicates, and objectives, and the optional settings that are set: replicate_timed when it is true, seed_namespace when there is one, a solver's own replicates, and a protocol's schedule when it is not the default. A setting left at its default is absent from the identity.

solver

solver(name: str) -> SolverSpec

Return the solver with the given name.

Raises:

Type Description
ConfigError

If no solver has that name.

protocol

protocol(name: str) -> Protocol

Return the protocol with the given name.

Raises:

Type Description
ConfigError

If no protocol has that name.

resolve

resolve(path: Path) -> Path

Return path resolved against base when it is relative.

with_execution

with_execution(**changes: Any) -> Campaign

Return a copy with some execution settings changed.

The copy has the same config hash, so it continues the same results.

allowed_cpus

allowed_cpus() -> tuple[int, ...]

Return the indices of the processors this process may run on, sorted.

When the set cannot be read (see affinity_cpus()), every processor of the machine is returned. The result is never empty.

available_cpus

available_cpus() -> int

Return how many processors this process may run on, at least one.

This is the size of allowed_cpus(), so affinity masks, the cpuset of a container, and taskset limits are respected.

check_name

check_name(value: object, what: str) -> str

Return value if it can name a file or directory on every platform.

Campaign ids, solver names, protocol names, and problem ids name files in the result store, so they must start with a letter or digit and contain only letters, digits, ., _, and -.

Raises:

Type Description
ConfigError

If value is not such a string.

jsonable

jsonable(value: Any) -> Any

Return a frozen configuration value as plain JSON types.

Tables become dictionaries and sequences become lists, recursively.

dispatchatlas.campaigns.processors

The processors a process may run on, and placing a process on one of them.

An operating system can confine a process to some of the machine's logical processors: a scheduler affinity set on Linux (taskset, the cpuset of a container) or a process affinity mask on Windows. A new process starts with the set of the thread that creates it. allowed_cpus() reads the set of the calling process, so a worker count or a warning follows what the process may use and not what the machine has. confined() narrows the caller to one processor while it creates a process, which therefore runs there from its first instruction, and pin_process() confines a process that is already running.

  • Where os.sched_getaffinity exists (Linux and some other POSIX systems), the set is read and written with the scheduler calls. They act on one thread: a thread started later inherits the set, a thread that is already running keeps its own.
  • On Windows the set is read with GetProcessAffinityMask and written with SetProcessAffinityMask, which act on every thread of a process. One mask describes at most 64 processors, so on a machine with several processor groups the set cannot be read and pinning is refused.
  • Elsewhere (macOS) the system offers no such call: the set cannot be read and pinning is refused.

A set that cannot be read counts as every processor of the machine; affinity_cpus() tells a reading from that count. A limit on processor time, such as the CPU quota of a container, is not an affinity and is not seen.

Processors are named by the index the operating system gives them, from 0.

affinity_cpus

affinity_cpus() -> tuple[int, ...] | None

Return the processors the operating system confines this process to, sorted.

Returns:

Type Description
tuple[int, ...] | None

The processor indices, or None when the set cannot be read: the

tuple[int, ...] | None

platform has no affinity call, the call fails, or one Windows mask

tuple[int, ...] | None

cannot describe the process.

allowed_cpus

allowed_cpus() -> tuple[int, ...]

Return the indices of the processors this process may run on, sorted.

When the set cannot be read (see affinity_cpus()), every processor of the machine is returned. The result is never empty.

available_cpus

available_cpus() -> int

Return how many processors this process may run on, at least one.

This is the size of allowed_cpus(), so affinity masks, the cpuset of a container, and taskset limits are respected.

describe_cpus

describe_cpus(indices: Iterable[int]) -> str

Return processor indices as text, with runs collapsed: "0-3, 8".

pinning_refusal

pinning_refusal() -> str | None

Return why this platform cannot pin a process, or None when it can.

confined

confined(cpu: int) -> Iterator[None]

Confine the caller to one processor while it creates a process.

A process created inside the context starts on cpu, and so does every thread it starts, whenever it starts it. On Windows every thread of the calling process is confined for the duration; elsewhere only the calling thread is. The caller has its earlier processors back on exit.

Parameters:

Name Type Description Default
cpu int

Index of the processor, one of allowed_cpus().

required

Raises:

Type Description
ConfigError

If the platform cannot pin a process (see pinning_refusal()), the calling process may not use cpu, or the operating system refuses the request.

pin_process

pin_process(pid: int, cpu: int) -> None

Confine a running process to one processor.

On Windows every thread of the process moves. Elsewhere the main thread moves, with every thread started from then on; a thread that is already running stays where it is, so use confined() to place a process from its start.

Parameters:

Name Type Description Default
pid int

Id of the process, which the caller must be allowed to change: itself or one of its children.

required
cpu int

Index of the processor, one of allowed_cpus(): a process is never moved outside the set of the calling process.

required

Raises:

Type Description
ConfigError

If the platform cannot pin a process (see pinning_refusal()), the calling process may not use cpu, or the operating system refuses the request.

dispatchatlas.campaigns.io

Reading and writing campaign files.

Campaign files are TOML, read with the standard library's tomllib, or JSON. One strict reader serves both: every table rejects keys it does not know, every value must have its documented type (true is not a number and "false" is not a boolean), a JSON object may not repeat a key, and the document must declare schema_version = 1. Relative paths are resolved against the directory of the file.

Example:

schema_version = 1
id = "shop-baselines"
seed = 2024
replicates = 10

[[instances]]
family = "flow-shop"
parameters = { jobs = 10, machines = 5 }
seeds = [1, 2, 3]

[[instances]]
suite = "taillard-flow-shop"
instances = ["ta001", "ta002"]

[[solvers]]
id = "random-search"

[[solvers]]
id = "hill-climbing"
label = "hill-climbing-r50"
parameters = { restart_after = 50 }

[[protocols]]
name = "evaluations"
stop = { max_evaluations = 5000 }

[execution]
workers = "auto"
retries = 1

An instance source is one of {family, seeds, parameters?}, {suite, instances, directory?}, {catalog, seed?}, {directory}, or {file, format?}. A solver is {id, label?, parameters?, replicates?}. A protocol's stop table takes the fields of StopCriteria. The optional top-level keys are replicates, replicate_timed, seed_namespace, objectives, execution (the fields of Execution), and metadata. In JSON, null stands for an absent optional value.

load_campaign

load_campaign(path: str | Path) -> Campaign

Read a campaign file.

Parameters:

Name Type Description Default
path str | Path

A .toml or .json file.

required

Returns:

Type Description
Campaign

The campaign. Relative paths in its instance sources refer to the

Campaign

file's directory.

Raises:

Type Description
ConfigError

If the file cannot be read or is not a valid campaign.

loads_campaign

loads_campaign(text: str, *, format: str = 'toml', base: Path | None = None) -> Campaign

Parse a campaign from TOML or JSON text.

Parameters:

Name Type Description Default
text str

The document.

required
format str

"toml" or "json".

'toml'
base Path | None

Directory that relative paths refer to.

None

Raises:

Type Description
ConfigError

If the text is not a valid campaign document.

campaign_to_dict

campaign_to_dict(campaign: Campaign) -> dict[str, Any]

Return a campaign as a JSON-ready document that campaign_from_dict() reads back.

The directory that relative paths refer to, Campaign.base, is not part of the document.

campaign_from_dict

campaign_from_dict(document: Mapping[str, Any], *, base: Path | None = None) -> Campaign

Build a campaign from a parsed TOML or JSON document.

Parameters:

Name Type Description Default
document Mapping[str, Any]

The document, as campaign_to_dict() returns or a campaign file holds.

required
base Path | None

Directory that relative paths refer to.

None

Raises:

Type Description
ConfigError

If a key is unknown, a value has the wrong type, a required key is missing, or the values are inconsistent. The message names the offending entry, for example instances[1].seeds[0].

dispatchatlas.campaigns.planning

Planning: from a campaign to the runs it needs.

plan() loads every problem the campaign's sources name, checks every solver id and parameter set against the registry, and expands the grid problem x replicate x solver x protocol into PlannedRun items:

  • Replicates. A stochastic solver runs campaign.replicates times per problem and protocol, as replicates 0, 1, .... A deterministic one runs once, as replicate 0, unless the campaign sets replicate_timed: it then runs every replicate under each protocol that sets a time limit, where its result depends on the machine. A solver whose entry carries its own replicates runs that many times under every protocol, whatever it declares.
  • Seeds. Seeds are common random numbers: replicate r of problem p receives derive_seed(campaign.seed, "problem:" + p, r) under every solver and protocol. Results are therefore paired across solvers, and adding, removing, or reordering solvers never changes a seed. A campaign with a seed_namespace derives them from the root derive_seed(campaign.seed, "namespace:" + namespace) in place of its seed, so two campaigns that share a seed and differ in namespace share no run seed.
  • Run ids. "run-" and the first 16 hexadecimal digits of the SHA-256 digest of the canonical JSON of the config hash, the problem id and digest, the solver id, label, and parameters, the protocol name, and the replicate. Equal campaigns give equal run ids on every machine, and a problem whose content changes gets new ones.
  • Compatibility. A pair whose solver does not model a feature of the problem is excluded, and the plan records why (see compatibility).

The problem digest is that of the problem as solved, after any objective override. assess() verifies every plan before it is returned.

PlannedProblem dataclass

PlannedProblem(id: str, family: str | None, digest: str, source: int, tasks: int, resources: int, objectives: tuple[str, ...], features: tuple[str, ...])

A problem of a plan.

Attributes:

Name Type Description
id str

Problem id.

family str | None

Benchmark family, if known.

digest str

Digest of the problem as solved.

source int

Index of the instance source that produced it.

tasks int

Number of tasks.

resources int

Number of resources.

objectives tuple[str, ...]

Objectives it is solved for, primary first.

features tuple[str, ...]

Its structural features, sorted.

to_dict

to_dict() -> dict[str, Any]

Return the entry as a JSON-ready dictionary.

PlannedRun dataclass

PlannedRun(run_id: str, problem: str, solver: str, protocol: str, replicate: int, seed: int)

One run of a plan: a solver on a problem under a protocol.

Attributes:

Name Type Description
run_id str

Run id.

problem str

Problem id.

solver str

Solver name.

protocol str

Protocol name.

replicate int

Replicate number, from 0.

seed int

Seed the solver receives.

to_dict

to_dict() -> dict[str, Any]

Return the run as a JSON-ready dictionary.

Plan dataclass

Plan(campaign: str, config_hash: str, seed: int, replicates: int, problems: tuple[PlannedProblem, ...], solvers: tuple[SolverEntry, ...], protocols: tuple[Protocol, ...], runs: tuple[PlannedRun, ...], exclusions: tuple[Exclusion, ...] = (), caveats: tuple[Caveat, ...] = (), replicate_timed: bool = False, seed_namespace: str | None = None)

Every run a campaign needs, with the facts that explain it.

A plan is plain data: the store writes it to plan.json and read_plan() reads it back.

Attributes:

Name Type Description
campaign str

Campaign id.

config_hash str

Config hash of the campaign.

seed int

Campaign seed.

replicates int

Replicates of stochastic solvers.

problems tuple[PlannedProblem, ...]

The problems, in source order.

solvers tuple[SolverEntry, ...]

The solvers, in campaign order.

protocols tuple[Protocol, ...]

The protocols, in campaign order.

runs tuple[PlannedRun, ...]

The runs, ordered by problem, replicate, solver, and protocol.

exclusions tuple[Exclusion, ...]

Pairs that do not run, and why.

caveats tuple[Caveat, ...]

Pairings that run but need care in the analysis.

replicate_timed bool

Whether a solver declared deterministic runs every replicate under a protocol that sets a time limit.

seed_namespace str | None

The campaign's seed namespace, or None.

runs_digest cached property

runs_digest: str

SHA-256 hex digest of the canonical JSON of every run's id and seed, sorted by id.

Two plans have the same digest exactly when they hold the same runs with the same seeds, in whatever order. A run id covers the config hash, the problem and its content, the solver and its parameters, the protocol, and the replicate; the seed is derived by the library that plans. The digest does not cover the solver entries: the metadata a solver declares can differ between two plans of one digest, and Store.plan_change names such a difference.

run

run(run_id: str) -> PlannedRun

Return the run with the given id.

Raises:

Type Description
KeyError

If the plan has no such run.

has_run

has_run(run_id: str) -> bool

Return whether the plan contains a run with the given id.

problem

problem(problem_id: str) -> PlannedProblem

Return the problem with the given id.

Raises:

Type Description
KeyError

If the plan has no such problem.

solver

solver(name: str) -> SolverEntry

Return the solver with the given name.

Raises:

Type Description
KeyError

If the plan has no such solver.

protocol

protocol(name: str) -> Protocol

Return the protocol with the given name.

Raises:

Type Description
KeyError

If the plan has no such protocol.

replicates_of

replicates_of(solver: SolverEntry, protocol: Protocol | None = None) -> int

Return how many replicates a solver runs under a protocol.

The solver's own count when its entry carries one. Otherwise every replicate for a stochastic solver, and for one declared deterministic a single run, or every replicate when replicate_timed is set and the protocol sets a time limit.

Parameters:

Name Type Description Default
solver SolverEntry

A solver of the plan.

required
protocol Protocol | None

A protocol of the plan. None asks for the count under a protocol without a time limit.

None

to_dict

to_dict() -> dict[str, Any]

Return the plan as a JSON-ready document.

Besides the plan's fields, the document lists every protocol's parities, the fair-comparison assessment, and runs_digest, derived from the plan for readers of the file.

from_dict classmethod

from_dict(document: object) -> Plan

Build a plan from to_dict() output. Derived entries are ignored.

A document without replicate_timed or seed_namespace reads as a plan that sets neither.

Raises:

Type Description
SerializationError

If the document is not a plan of this schema version, or is malformed.

Prepared dataclass

Prepared(campaign: Campaign, plan: Plan, problems: Mapping[str, Problem], registry: SolverRegistry)

A plan together with what executing it needs.

Attributes:

Name Type Description
campaign Campaign

The campaign.

plan Plan

Its plan.

problems Mapping[str, Problem]

The problems as solved, by id.

registry SolverRegistry

The solver registry the plan was checked against.

run_seed

run_seed(root: int, problem_id: str, replicate: int, namespace: str | None = None) -> int

Return the seed of every run of one problem and replicate.

Parameters:

Name Type Description Default
root int

The campaign seed.

required
problem_id str

Id of the problem.

required
replicate int

Replicate number, from 0.

required
namespace str | None

The campaign's seed namespace, or None without one.

None

Returns:

Type Description
int

derive_seed(root, "problem:" + problem_id, replicate). With a

int

namespace, derive_seed(root, "namespace:" + namespace) takes the

int

place of root.

run_id

run_id(config_hash: str, problem_id: str, problem_hash: str, solver: SolverSpec, protocol: str, replicate: int) -> str

Return the id of one run.

Parameters:

Name Type Description Default
config_hash str

Config hash of the campaign.

required
problem_id str

Id of the problem.

required
problem_hash str

Digest of the problem as solved.

required
solver SolverSpec

The solver specification.

required
protocol str

Name of the protocol.

required
replicate int

Replicate number.

required

Returns:

Type Description
str

"run-" and 16 hexadecimal digits.

plan

plan(campaign: Campaign, *, registry: SolverRegistry | None = None, families: FamilyRegistry | None = None) -> Plan

Plan a campaign without running anything.

Parameters:

Name Type Description Default
campaign Campaign

The campaign.

required
registry SolverRegistry | None

Solver registry. Defaults to the built-in solvers and plugins.

None
families FamilyRegistry | None

Benchmark-family registry for family and catalog sources. Defaults to the built-in families and plugins.

None

Returns:

Type Description
Plan

The verified plan.

Raises:

Type Description
ConfigError

If a source cannot be loaded, a solver id or parameter is unknown, problem ids clash, or no run remains.

prepare

prepare(campaign: Campaign, *, registry: SolverRegistry | None = None, families: FamilyRegistry | None = None) -> Prepared

Plan a campaign and keep the loaded problems and the registry for execution.

Takes the same arguments and raises the same errors as plan().

check_compatibility

check_compatibility(campaign: Campaign, *, registry: SolverRegistry | None = None, families: FamilyRegistry | None = None) -> Compatibility

Load a campaign's problems and report which solvers each one excludes.

Unlike plan(), it reports rather than fails when every pair is excluded.

Parameters:

Name Type Description Default
campaign Campaign

The campaign.

required
registry SolverRegistry | None

Solver registry. Defaults to the built-in solvers and plugins.

None
families FamilyRegistry | None

Benchmark-family registry. Defaults to the built-in families and plugins.

None

Raises:

Type Description
ConfigError

If a source cannot be loaded or a solver id is unknown.

dispatchatlas.campaigns.compatibility

Which solvers run on which problems, and what to keep in mind when they do.

A solver declares the problem features it models in supports. A campaign never pairs a solver with a problem that has a feature outside that set: the cell is excluded and the plan records the exclusion with the missing features, so a comparison that is absent is always explained.

Caveats mark pairings that run but need care when results are compared:

  • a single-objective solver on a problem with several objectives optimises the primary objective only; the others describe its schedule;
  • on problems with more than three objectives most solutions become mutually non-dominated, so a solver whose selection relies on Pareto dominance loses selection pressure (Deb and Jain 2014).

MANY_OBJECTIVES module-attribute

MANY_OBJECTIVES = 4

Objective count from which dominance-based selection is flagged (Deb and Jain 2014).

Exclusion dataclass

Exclusion(problem: str, solver: str, features: tuple[str, ...])

A problem and solver that a campaign does not pair.

Attributes:

Name Type Description
problem str

Problem id.

solver str

Solver name.

features tuple[str, ...]

Features of the problem the solver does not model, sorted.

reason property

reason: str

Why the cell is excluded, as a sentence fragment.

to_dict

to_dict() -> dict[str, Any]

Return the exclusion as a JSON-ready dictionary.

from_dict classmethod

from_dict(document: object) -> Exclusion

Build an exclusion from to_dict() output.

Raises:

Type Description
SerializationError

If the document is malformed.

Caveat dataclass

Caveat(solver: str, problems: tuple[str, ...], message: str, reference: str | None = None)

A pairing that runs but needs care in the analysis.

Attributes:

Name Type Description
solver str

Solver name.

problems tuple[str, ...]

Ids of the problems concerned.

message str

What to keep in mind.

reference str | None

Bibliography key of the source, if any.

to_dict

to_dict() -> dict[str, Any]

Return the caveat as a JSON-ready dictionary.

from_dict classmethod

from_dict(document: object) -> Caveat

Build a caveat from to_dict() output.

Raises:

Type Description
SerializationError

If the document is malformed.

Compatibility dataclass

Compatibility(exclusions: tuple[Exclusion, ...] = (), caveats: tuple[Caveat, ...] = ())

The exclusions and caveats of a set of problems and solvers.

Attributes:

Name Type Description
exclusions tuple[Exclusion, ...]

Pairs that do not run, in problem order, then solver order.

caveats tuple[Caveat, ...]

Pairings that run but need care, in solver order.

excludes

excludes(problem: str, solver: str) -> bool

Return whether the pair of a problem id and a solver name is excluded.

to_dict

to_dict() -> dict[str, Any]

Return the assessment as a JSON-ready dictionary.

assess

assess(problems: Sequence[Problem], solvers: Sequence[tuple[str, SolverInfo]]) -> Compatibility

Decide which problem and solver pairs run, and which need caveats.

Parameters:

Name Type Description Default
problems Sequence[Problem]

The problems, with the objectives they will be solved for.

required
solvers Sequence[tuple[str, SolverInfo]]

Solver names with the metadata of the registered solver.

required

Returns:

Type Description
Compatibility

The exclusions and caveats.

dispatchatlas.campaigns.fairness

Fair comparison: the same budget, problems, and seeds for every solver, verified.

Solvers are compared fairly when, under each protocol, every one of them runs with identical stop criteria on the same problems with the same seeds. The planner builds runs that way; assess() checks it on the finished plan instead of assuming it, so the claim can also be re-checked from a stored plan.json:

  • every protocol bounds the run with a time, evaluation, or iteration limit;
  • every run refers to one of the plan's protocols, so all runs under a protocol share its stop criteria;
  • every solver has its full set of replicates under every protocol on every problem it is not excluded from (exclusions are recorded with their reason, see compatibility);
  • all runs of one problem and replicate share one seed, so results are paired across solvers and protocols.

A protocol's parities name the budgets it equalises, read off its stop criteria: equal-wall-clock (time_limit), equal-evaluations (max_evaluations), equal-iterations (max_iterations), and fixed-target (target). Wall-clock budgets depend on the speed and load of the machine, and runs they stop cannot be replayed; an iteration means a different amount of work in different solvers.

The assessment also carries notes: facts that break none of these rules but that a reader of a comparison needs (notes()). A note never refuses a plan. Both concern protocols with a time limit:

  • solvers whose threads parameters differ are noted, because equal wall-clock budgets then buy unequal processor time. Each solver counts with the value its plan entry records, which is the one its instance holds, the default included; a solver without such a parameter counts as one thread;
  • solvers declared deterministic that run once are noted when the campaign has more replicates, because the result of a run that the clock ends depends on the speed and load of the machine and one run does not show by how much. The campaign key replicate_timed gives them every replicate, and a solver whose entry states its own count is not noted. A stop criterion ends the run of an iterative solver, so such a solver is a note. A solver that is not iterative ends when its algorithm does: a constructive rule never reads the clock, and an exact solver can be ended by its time limit. The plan cannot tell the two apart, so such a solver is a conditional note (conditional_notes()), which holds only if the time limit ends one of its runs.

A plan says what should run. assess_outcomes() compares the records of the completed runs with it, protocol by protocol: for which reasons the runs of the iterative solvers ended, since runs of one protocol that ended for different reasons did not spend the same budget, which solvers' runs took more wall time than the protocol's time limit allows, and which of the solvers of a conditional note had a run that the time limit ended.

PARITIES module-attribute

PARITIES = (EQUAL_WALL_CLOCK, EQUAL_EVALUATIONS, EQUAL_ITERATIONS, FIXED_TARGET)

Every parity name, in report order.

Fairness dataclass

Fairness(solvers: tuple[str, ...], protocols: Mapping[str, tuple[str, ...]], paired: bool, issues: tuple[str, ...], notes: tuple[str, ...] = (), conditional_notes: tuple[str, ...] = ())

The fair-comparison assessment of a plan.

Attributes:

Name Type Description
solvers tuple[str, ...]

Names of the solvers compared.

protocols Mapping[str, tuple[str, ...]]

Parities of each protocol, by protocol name.

paired bool

Whether every problem and replicate has one seed.

issues tuple[str, ...]

Every violation found. Empty for a fair plan.

notes tuple[str, ...]

Facts that violate nothing but bear on a comparison. They leave a plan fair.

conditional_notes tuple[str, ...]

Facts that bear on a comparison only if a run ends a certain way, which the plan cannot tell (conditional_notes()). They leave a plan fair.

fair property

fair: bool

Whether no violation was found.

parities property

parities: tuple[str, ...]

Every parity some protocol realises, in PARITIES order.

to_dict

to_dict() -> dict[str, Any]

Return the assessment as a JSON-ready dictionary.

Overrun dataclass

Overrun(runs: int, over: int, worst: float)

The completed runs of one solver against the time limit of their protocol.

Attributes:

Name Type Description
runs int

Completed runs of the solver under the protocol.

over int

How many of them took more wall time than the time limit with the tolerance allows, by the rule of overran().

worst float

The longest wall time among those, as a multiple of the time limit.

ProtocolOutcomes dataclass

ProtocolOutcomes(protocol: str, stop_reasons: Mapping[str, Mapping[str, int]], overruns: Mapping[str, Overrun], unreplicated: Mapping[str, Mapping[str, int]] = dict())

How the completed runs of one protocol ended.

Attributes:

Name Type Description
protocol str

Name of the protocol.

stop_reasons Mapping[str, Mapping[str, int]]

For each iterative solver with a completed run, in plan order, how many of its runs ended for each stop reason. Solvers that are not iterative end when their algorithm does and are left out.

overruns Mapping[str, Overrun]

For each solver with a completed run over the time limit, in plan order, the count and extent. Empty for a protocol without a time limit.

unreplicated Mapping[str, Mapping[str, int]]

For each solver declared deterministic that is not iterative, runs once per problem under the protocol's time limit although the campaign has more replicates, and has a completed run that the time limit ended, in plan order, how many of its runs ended for each stop reason. These are the solvers of the plan's conditional note whose condition holds.

divergent property

divergent: bool

Whether the runs of the iterative solvers ended for more than one reason.

Runs that ended for different reasons, such as one at the time limit and one for lack of improvement, did not spend the same budget, in one solver or across several. A reached target or a proof of optimality is a legitimate end, so a divergence qualifies a comparison and does not refuse it.

to_dict

to_dict() -> dict[str, Any]

Return the outcomes as a JSON-ready dictionary.

Outcomes dataclass

Outcomes(protocols: tuple[ProtocolOutcomes, ...], overrun_tolerance: float, records: int)

The outcomes of a campaign's completed runs, compared with its plan.

Attributes:

Name Type Description
protocols tuple[ProtocolOutcomes, ...]

The outcomes under each protocol, in plan order.

overrun_tolerance float

Share of a time limit by which a run may exceed it without counting as over; the clock margin of overran() applies beside it.

records int

How many records counted: those of runs of the plan.

divergent property

divergent: tuple[str, ...]

Names of the protocols whose stop reasons diverge.

protocol

protocol(name: str) -> ProtocolOutcomes

Return the outcomes under the protocol with the given name.

Raises:

Type Description
KeyError

If the plan has no such protocol.

to_dict

to_dict() -> dict[str, Any]

Return the assessment as a JSON-ready dictionary.

parities

parities(stop: StopCriteria) -> tuple[str, ...]

Return the parities that stop criteria realise, in PARITIES order.

assess

assess(plan: Plan) -> Fairness

Check that a plan compares its solvers fairly.

Parameters:

Name Type Description Default
plan Plan

A plan, freshly built or read from a store.

required

Returns:

Type Description
Fairness

The assessment. Its issues name every violation, and its

Fairness

notes what else bears on the comparison.

notes

notes(plan: Plan, protocol: Protocol, solvers: Collection[str] | None = None) -> tuple[str, ...]

Return what bears on a comparison under one protocol without breaking a rule.

Parameters:

Name Type Description Default
plan Plan

The plan.

required
protocol Protocol

One of its protocols.

required
solvers Collection[str] | None

Names of the solvers compared. None compares every solver of the plan.

None

Returns:

Type Description
str

One sentence, without its capital and full stop, for each fact: the

...

compared solvers' thread counts differ under a time limit, and

tuple[str, ...]

iterative solvers declared deterministic run once under a time limit

tuple[str, ...]

although the campaign has more replicates. Empty for a protocol

tuple[str, ...]

without a time limit.

conditional_notes

conditional_notes(plan: Plan, protocol: Protocol, solvers: Collection[str] | None = None) -> tuple[str, ...]

Return what bears on a comparison under one protocol only if a run ends a certain way.

A solver declared deterministic that is not iterative ends when its algorithm does. A constructive rule never reads the clock, and an exact solver can be ended by its time limit; the plan cannot tell the two apart, so what it says about them holds only for a run that the time limit ends. assess_outcomes() finds those runs among the records.

Parameters:

Name Type Description Default
plan Plan

The plan.

required
protocol Protocol

One of its protocols.

required
solvers Collection[str] | None

Names of the solvers compared. None compares every solver of the plan.

None

Returns:

Type Description
str

One sentence, without its capital and full stop: solvers declared

...

deterministic that are not iterative run once under a time limit

tuple[str, ...]

although the campaign has more replicates. Empty when there is none,

tuple[str, ...]

and for a protocol without a time limit.

assess_outcomes

assess_outcomes(plan: Plan, records: Iterable[RunRecord], *, overrun_tolerance: float = OVERRUN_MARGIN) -> Outcomes

Compare the outcomes of a campaign's completed runs with its plan.

A plan can be fair and its runs still not comparable: under one protocol, a solver that stops for lack of improvement spends less than the budget another one uses to its time limit, and a solver that honours no deadline spends more. Neither shows in the plan.

Parameters:

Name Type Description Default
plan Plan

The plan.

required
records Iterable[RunRecord]

Records of completed runs. A record of a run that is not in the plan does not count.

required
overrun_tolerance float

Share of the time limit by which the wall time of a completed run may exceed it without counting as over. A run counts as over by the rule of the engine's note on it, overran(), so a wall time within the clock margin of the limit never does.

OVERRUN_MARGIN

Returns:

Type Description
Outcomes

For each protocol, the stop reasons of the iterative solvers' runs,

Outcomes

the solvers whose runs took more wall time than the time limit

Outcomes

allows, and the solvers declared deterministic that run once and

Outcomes

are not iterative whose runs the time limit ended.

Raises:

Type Description
ConfigError

If overrun_tolerance is not zero or a positive number.

dispatchatlas.campaigns.estimation

Estimates: how many runs a campaign needs, and how long they can take.

estimate() plans a campaign without running anything and bounds the duration of its solves from the protocols. The solve of a run whose protocol sets time_limit takes at most its hard limit when the process backend enforces one, and otherwise relies on the solver honouring the time limit. Every attempt may take that long, and a run gets up to 1 + retries attempts. The time a worker takes to begin a solve, which the start-up limit bounds, is not counted. A run whose protocol bounds only evaluations or iterations has no duration bound, so those runs are counted apart.

Estimate dataclass

Estimate(planned: int, completed: int, failed: int, remaining: int, problems: int, solvers: int, protocols: int, exclusions: int, by_protocol: Mapping[str, int], by_solver: Mapping[str, int], workers: int, timed: int, untimed: int, budget_seconds: float, worst_case_seconds: float)

The size of a campaign and bounds on its duration.

Attributes:

Name Type Description
planned int

Runs in the plan.

completed int

Runs an existing store already completed.

failed int

Runs that failures in an existing store settled.

remaining int

Runs that run() would attempt: neither completed nor settled.

problems int

Problems in the plan.

solvers int

Solvers in the plan.

protocols int

Protocols in the plan.

exclusions int

Problem and solver pairs that do not run.

by_protocol Mapping[str, int]

Remaining runs per protocol.

by_solver Mapping[str, int]

Remaining runs per solver.

workers int

Runs that execute at once.

timed int

Remaining runs whose protocol sets a time limit.

untimed int

Remaining runs without a time limit, whose duration has no bound.

budget_seconds float

Sum of the time limits of the timed runs.

worst_case_seconds float

Sum of the longest time the solve of every attempt of every timed run may take: its hard limit when enforced, else its time limit, times 1 + retries.

budget_wall_seconds property

budget_wall_seconds: float

budget_seconds spread evenly over the workers.

worst_case_wall_seconds property

worst_case_wall_seconds: float

worst_case_seconds spread evenly over the workers.

estimate

estimate(campaign: Campaign, *, root: str | Path | None = None, registry: SolverRegistry | None = None, families: FamilyRegistry | None = None, backend: str | None = None, workers: int | str | None = None) -> Estimate

Plan a campaign and bound the time its remaining runs can take.

Parameters:

Name Type Description Default
campaign Campaign

The campaign.

required
root str | Path | None

Output root. When given, runs that the campaign's directory already completed, or whose failures settled them, are left out.

None
registry SolverRegistry | None

Solver registry. Defaults to the built-in solvers and plugins.

None
families FamilyRegistry | None

Benchmark-family registry. Defaults to the built-in families and plugins.

None
backend str | None

Override of the campaign's execution backend.

None
workers int | str | None

Override of the campaign's worker count, or "auto".

None

Returns:

Type Description
Estimate

The estimate.

Raises:

Type Description
ConfigError

If the campaign cannot be planned.

dispatchatlas.campaigns.runner

Running a campaign: plan it, execute what remains, and record every attempt.

run() plans the campaign, opens its directory under the output root, and executes every run that is neither complete nor settled by failures. Calling it again after an interruption therefore resumes the campaign: records already written are kept when they belong to the same configuration and still match their result hash. A directory that holds results of another configuration is refused unless fresh asks to discard them.

Every attempt leaves a record the moment it ends: a run record for a completion, a failure record for a failure (see records). Attempt numbers continue across sessions. A failed run gets another attempt, at the end of the queue, until its failures settle it: its latest failure is terminal, or it has failed 1 + retries times. Each session writes sessions/<n>.json with the environment it runs in, when it starts and again when it ends.

A session that resumes a campaign compares its execution settings and its environment with those of the last session the directory recorded (session_changes()). It logs a warning that names each difference and stores them in its record, and a campaign whose on_changed_code is "refuse" refuses a session on other code instead. A file at the path of a planned run that is not the valid record of that run, such as a record made on another version of its problem, does not count, and its run is attempted again unless on_changed_problem is "refuse". The file stays where it is until the new record is written, and is then moved under superseded/; a record kept there that is the valid record of a planned run again, because its problem was restored, is moved back when a session starts.

Each session also writes logs/session-<n>.log, each line with the UTC time of its record. The log holds what the engine itself logs while run() works, from the level INFO, and what the other loggers of the library let through, in this process and in its worker processes: WARNING and higher unless the application lowered a level. The execution key log_level lowers those loggers for the session, at the cost that a solver which logs in its loop then pays inside its time limit. What the handlers of the application show does not change either way.

Progress is logged through dispatchatlas.core.logs: a line when the session starts, one at most every PROGRESS_INTERVAL seconds while it runs, and a summary at the end.

PROGRESS_INTERVAL module-attribute

PROGRESS_INTERVAL = 10.0

Seconds between progress lines while a session runs.

RunSummary dataclass

RunSummary(campaign: str, config_hash: str, directory: Path, session: int, planned: int, completed: int, failed: int, pending: int, attempted: int, succeeded: int, errors: int, elapsed: float)

What a session of run() did, and where the campaign stands.

Attributes:

Name Type Description
campaign str

Campaign id.

config_hash str

Config hash of the campaign.

directory Path

The campaign directory.

session int

Number of the session.

planned int

Runs in the plan.

completed int

Runs with a record, from every session.

failed int

Runs settled by failures, from every session.

pending int

Runs that still need an attempt.

attempted int

Attempts this session made.

succeeded int

Attempts this session that completed a run.

errors int

Attempts this session that failed.

elapsed float

Wall-clock seconds of the session.

done property

done: bool

Whether no run needs another attempt.

run

run(campaign: Campaign, root: str | Path, *, registry: SolverRegistry | None = None, families: FamilyRegistry | None = None, fresh: bool = False, retry_failed: bool = False, max_runs: int | None = None, backend: str | None = None, workers: int | str | None = None, affinity: str | Sequence[int] | None = None) -> RunSummary

Run a campaign, resuming from the results its directory already holds.

Parameters:

Name Type Description Default
campaign Campaign

The campaign.

required
root str | Path

Output root; the results go to root / campaign.id.

required
registry SolverRegistry | None

Solver registry. Defaults to the built-in solvers and plugins.

None
families FamilyRegistry | None

Benchmark-family registry. Defaults to the built-in families and plugins.

None
fresh bool

Discard every result in the campaign directory first.

False
retry_failed bool

Give every run that failures settled one more attempt.

False
max_runs int | None

Attempt at most this many pending runs in this session.

None
backend str | None

Override of the campaign's execution backend.

None
workers int | str | None

Override of the campaign's worker count, or "auto".

None
affinity str | Sequence[int] | None

Override of the campaign's affinity: "inherit", "pin", or processor indices.

None

Returns:

Type Description
RunSummary

The summary of the session.

Raises:

Type Description
ConfigError

If the campaign cannot be planned, its directory holds results of another configuration, another runner is using it, or runs are pending and its workers cannot be pinned as affinity asks: the platform cannot pin a process, the workers outnumber their processors, or a listed processor is not one this process may use. A campaign refused for its pinning loses no record, also with fresh, and a new one leaves no directory. Also if the campaign's on_changed_code is "refuse" and the code differs from that of the last recorded session, or its on_changed_problem is "refuse" and the directory holds a record made on another version of a planned run's problem; such a refusal moves no file and writes no session record. Also if a file that is not the valid record of a run cannot be moved under superseded/ when the new record of the run is to be written, for example because another program holds it open: the session then ends as failed, and the run is attempted again by the next.

KeyboardInterrupt

When interrupted. Records written so far are kept and the session is marked interrupted.

session_changes

session_changes(previous: SessionRecord, execution: Execution, workers: int, environment: Mapping[str, Any]) -> tuple[dict[str, Any], ...]

Return how a session differs from an earlier session of its campaign directory.

The execution settings are compared one by one, except those that only say what a session does about a difference, and the resolved worker count is compared when both sessions state the same workers. The environments are compared value by value. A value the earlier record does not state is not compared; a package, a plugin, or a git checkout that only one session has is a difference. The load of the machine is no part of the environment and is not compared.

Parameters:

Name Type Description Default
previous SessionRecord

The record of the earlier session.

required
execution Execution

The execution settings of the session.

required
workers int

Its number of workers, with "auto" resolved.

required
environment Mapping[str, Any]

Its environment, as capture_environment() records it.

required

Returns:

Type Description
dict[str, Any]

One document per difference, with field, a dotted path under

...

execution or environment or the name workers; kind,

tuple[dict[str, Any], ...]

which is "execution", "code" for the Python version, the

tuple[dict[str, Any], ...]

packages, the library, the plugins, and the git states, or

tuple[dict[str, Any], ...]

"machine" for the rest of the environment; and the values

tuple[dict[str, Any], ...]

earlier and now.

tasks_for

tasks_for(prepared: Prepared, runs: Sequence[PlannedRun], execution: Execution, attempt: Callable[[str], int]) -> list[Task]

Build the tasks that execute runs of a prepared plan.

Parameters:

Name Type Description Default
prepared Prepared

The prepared plan.

required
runs Sequence[PlannedRun]

Runs of its plan.

required
execution Execution

Execution settings, for hard limits and schedule recording.

required
attempt Callable[[str], int]

The attempt number of a run, given its id.

required

Returns:

Type Description
list[Task]

One task per run, each with a fresh solver instance.

overridden

overridden(base: Execution, backend: str | None, workers: int | str | None, affinity: str | Sequence[int] | None = None) -> Execution

Return execution settings with a caller's overrides applied.

Choosing the sequential backend, which has no worker processes, sets what only worker processes have to its default: affinity to "inherit", threads_per_worker to 1, and, without a worker count, workers to 1.

Parameters:

Name Type Description Default
base Execution

The campaign's settings.

required
backend str | None

Another backend, or None to keep the campaign's.

required
workers int | str | None

Another worker count or "auto", or None.

required
affinity str | Sequence[int] | None

Another affinity: "inherit", "pin", or processor indices; or None.

None

Raises:

Type Description
ConfigError

If the resulting settings are invalid.

dispatchatlas.campaigns.backends

Execution backends: where the runs of a campaign execute.

Both backends take Task items from a queue that the runner may extend while results arrive (a retry goes to its end), and yield each Completion or Failure as soon as it exists.

  • SequentialBackend runs one attempt after another in the calling process. Solvers must honour their own time limits, and a crash in native code ends the campaign. Nothing has to be picklable, which suits debugging and solvers defined interactively.
  • ProcessBackend runs attempts in worker processes started with the spawn method, so every platform behaves alike, and supervised by the calling process. Each worker runs one attempt at a time and keeps the problem of its last attempt, so a problem is sent to a worker only when the worker's previous attempt was on another one; runs are planned problem by problem, which makes that rare. The supervisor enforces two limits (see timeout). From the hand-over of a run its worker has the start-up limit to report that the solve begins; from the instant that report states, the run has its hard limit. The supervisor kills a worker that passes either, with every process the worker started, records a recoverable start-up failure or a terminal timeout, and hands the next run to a fresh worker. A hand-over never waits for a worker: a worker that is starting gets its run when it reports that it is ready, so the supervisor keeps reading reports and enforcing the limits of the other runs meanwhile. While a run with a hard limit executes, its worker forwards the incumbents its search reports, at most one every PROGRESS_INTERVAL seconds, and a timeout failure holds the last one and the last one found within the run's time limit. A worker that dies, from a crash in native code or the out-of-memory killer, costs only the attempt it was running, which is recorded as a recoverable failure. Every worker starts with the thread pools of numerical libraries limited: the variables of THREAD_VARIABLES are in the environment the worker process is created with, so no import in the worker comes before them. A worker can also be pinned to one processor: the supervisor creates it while confined to that processor itself, so the worker and every thread it starts run there, and the worker reports its processors when it is ready, which the supervisor checks. A backend that is given the digest of the library source its session started on has every worker report the digest of the source it starts on: a worker on other source is warned about, and every run it completes carries a note. A backend that is given a sink for log records has every worker send the records of WORKER_LOG_LEVEL or higher that the library's loggers let through in it, which a fresh worker does from WARNING; given a log level as well, the backend lowers the library's loggers of every worker to it, and what the handlers of a worker show stays as it was. The thread that logs never waits for the supervisor: a record goes to a bounded queue that another thread of the worker sends in batches, a record the queue has no room for is dropped and counted, and the supervisor adds the count to the log. The supervisor spends at most LOG_READ_SECONDS on the log records of one worker before it turns to the limits and the other workers, and log records do not postpone a kill.

Every result states how many runs were executing when its attempt was handed over (concurrent), the processor its worker is pinned to (cpu), and how long the attempt waited for its solve to begin (waited_seconds), from its hand-over, before a fresh worker exists, to the instant its worker states.

STARTUP_ATTEMPTS module-attribute

STARTUP_ATTEMPTS = 3

Consecutive worker processes that may exit before they start until the backend gives up.

PROGRESS_INTERVAL module-attribute

PROGRESS_INTERVAL = 0.25

Fewest seconds between two incumbents a worker forwards to the supervisor.

WORKER_LOG_LEVEL module-attribute

WORKER_LOG_LEVEL = logging.INFO

Lowest level of the log records a worker sends, unless the backend is given a lower one.

LOG_READ_SECONDS module-attribute

LOG_READ_SECONDS = 0.05

Longest the supervisor handles the log records of one worker in one reading of its pipe.

LogSink

LogSink = Callable[[str, float, int, str, str], None]

Receives a record a worker logged: the worker's name, the time of the record in seconds since the epoch, its level, the name of its logger, and its message.

Backend

Bases: Protocol

Where attempts execute.

name instance-attribute

name: str

"sequential" or "process".

workers instance-attribute

workers: int

Attempts that run at once.

run

run(queue: deque[Task]) -> Generator[Result]

Execute tasks from queue until it is empty, yielding each result.

SequentialBackend

Runs attempts one after another in the calling process.

run

run(queue: deque[Task]) -> Generator[Result]

Execute tasks from queue until it is empty, yielding each result.

ProcessBackend

ProcessBackend(workers: int, *, cpus: Sequence[int] | None = None, threads: int = 1, startup_limit: float = STARTUP_SECONDS, source: str | None = None, log: LogSink | None = None, log_level: int | None = None)

Runs attempts in supervised worker processes.

Parameters:

Name Type Description Default
workers int

Number of worker processes.

required
cpus Sequence[int] | None

The processor each worker is pinned to, in worker order, or None to leave the workers on the processors the calling process may use.

None
threads int

Threads the numerical libraries of a worker may use.

1
startup_limit float

Seconds a worker has, from the hand-over of a run, to report that the solve begins.

STARTUP_SECONDS
source str | None

The digest of the library source the session started on (source_digest()), which every worker compares the source it starts on with. None compares nothing.

None
log LogSink | None

Receives the records the workers log. None leaves them in the workers.

None
log_level int | None

The level the library's loggers of every worker are lowered to, so that log receives their records from that level. None leaves the loggers as they are, and log receives what they let through of WORKER_LOG_LEVEL or higher. Without log nothing is lowered.

None

Raises:

Type Description
ConfigError

If there is no worker, a worker has no thread, the start-up limit is not a positive, finite number, or there are fewer processors to pin to than workers.

run

run(queue: deque[Task]) -> Generator[Result]

Execute tasks from queue until it is empty, yielding each result.

Closing the iterator early, or an exception in the caller, kills the workers.

Raises:

Type Description
ConfigError

If worker processes keep exiting before they start, for example because a script starts a campaign without an if __name__ == "__main__": guard, or a worker cannot be created on its processor or reports other processors.

backend_for

backend_for(name: str, workers: int, *, cpus: Sequence[int] | None = None, threads: int = 1, startup_limit: float = STARTUP_SECONDS, source: str | None = None, log: LogSink | None = None, log_level: int | None = None) -> Backend

Return the backend of the given name.

Parameters:

Name Type Description Default
name str

"sequential" or "process".

required
workers int

Attempts that run at once.

required
cpus Sequence[int] | None

The processor each worker process is pinned to, if any.

None
threads int

Threads the numerical libraries of a worker process may use.

1
startup_limit float

Seconds a worker process has, from the hand-over of a run, to report that the solve begins.

STARTUP_SECONDS
source str | None

The digest of the library source the session started on, for the worker processes to compare theirs with.

None
log LogSink | None

Receives the records the worker processes log.

None
log_level int | None

The level the library's loggers of every worker process are lowered to, or None to leave them. The sequential backend has no worker process and uses none of the three.

None

Raises:

Type Description
ConfigError

If the name is unknown, or the sequential backend gets more than one worker or processors to pin to.

dispatchatlas.campaigns.execution

One attempt of one run: solve, check, measure, and describe the outcome.

attempt() is what every backend executes, in the calling process or in a worker process. It calls the solver, then checks the result without trusting it: check_schedule() decides whether the schedule is feasible, and evaluate() recomputes every objective from the schedule. It returns plain data, a Completion or a Failure, which crosses process boundaries.

Measurements cover the solve alone: wall time with time.perf_counter(); CPU time of the executing process with time.process_time(), which includes any threads the solver starts, plus, on POSIX systems, the CPU time of the child processes the solver started and waited for, from os.times(); and, on Linux, the peak resident memory of the process during the solve, read from /proc/self/status after resetting it through /proc/self/clear_refs. On Windows the CPU time of a child process, such as a backend that runs as a program of its own, is not in the measurement.

The instant the wall time counts from is also the origin of the solver's clock: attempt() calls the solver inside clock_origin(), so the time limit and the elapsed time of every solver cover everything the solver does from the start of the attempt, including what it does before its search starts, such as building an evaluator or a model. What a solver does once in a process, such as importing a backend, is done before that instant by warm_up(), so that it shortens the search of no run.

observe() adds what the time and the trace of the attempt show: a wall time beyond the time limit, an elapsed time the engine's clock contradicts, and a trace that cannot be drawn against time or evaluations or that ends at another value than the outcome. These notes and those of describe() are returned with the completion, logged, and stored in the run record.

classify() sorts errors into the categories of Category.

OVERRUN_MARGIN module-attribute

OVERRUN_MARGIN = 0.05

Share of its time limit by which the wall time of a run may pass the limit unnoted.

CLOCK_MARGIN_SECONDS module-attribute

CLOCK_MARGIN_SECONDS = 0.05

Seconds two times of a run may differ by unnoted, whatever their size.

The scheduler of the machine and the resolution of its clocks move the end of a run by a few hundredths of a second. The margin applies to the wall time against the time limit, and to the solver's elapsed time against the wall time.

CLOCK_MARGIN_SHARE module-attribute

CLOCK_MARGIN_SHARE = 0.05

Share of the engine's wall time by which a solver's elapsed time may differ from it unnoted.

Also the share of the time limit by which a run that stops for its time limit may end before the limit unnoted.

OBSERVATION_KINDS module-attribute

OBSERVATION_KINDS: tuple[str, ...] = ('overrun', 'early-stop', 'elapsed', 'trace-time', 'trace-evaluations', 'trace-value', 'no-trace')

The kinds of fact observations() finds in a completed attempt, in its order.

Result

Result = Completion | Failure

What an attempt produces.

Task dataclass

Task(run_id: str, attempt: int, problem: Problem, digest: str, solver: Solver, stop: StopCriteria, seed: int, hard_limit: float | None = None, record_schedule: bool = True)

One attempt of one run, ready to execute.

Attributes:

Name Type Description
run_id str

Run id.

attempt int

Attempt number, from 1.

problem Problem

The problem, as solved.

digest str

Digest of the problem.

solver Solver

A solver instance built with the campaign's parameters.

stop StopCriteria

Stop criteria of the run's protocol.

seed int

Seed of the run.

hard_limit float | None

Seconds after the start of the solve at which the process backend kills the run, or None.

record_schedule bool

Whether to return the schedule for the record.

Measurement dataclass

Measurement(started: str, finished: str, wall_seconds: float, cpu_seconds: float | None = None, peak_rss_bytes: int | None = None, worker: str = 'main', hard_limit: float | None = None, cpu: int | None = None, concurrent: int | None = None, waited_seconds: float | None = None, shared_clock: bool | None = None)

When, where, and at what cost an attempt executed.

Attributes:

Name Type Description
started str

UTC start time in ISO 8601.

finished str

UTC end time in ISO 8601.

wall_seconds float

Wall-clock seconds.

cpu_seconds float | None

CPU seconds of the executing process, if measured, and on POSIX systems also of the child processes it waited for during the attempt.

peak_rss_bytes int | None

Peak resident memory of the executing process, if measured.

worker str

Where the attempt executed: "main" or a worker slot.

hard_limit float | None

The hard limit armed for the attempt, if any.

cpu int | None

The processor the executing worker is pinned to, if any.

concurrent int | None

Runs executing when the attempt was handed to where it executes, itself included. None for an attempt that was never handed over.

waited_seconds float | None

Seconds from the hand-over of the attempt to its worker's slot until the instant the worker states as the start of the solve. None when the solve never began, and for the sequential backend.

shared_clock bool | None

True when the solver was called with its clocks starting at the instant wall_seconds counts from, as attempt() calls it. None for an attempt that did not reach that call or that the engine ended.

Completion dataclass

Completion(run_id: str, attempt: int, outcome: Outcome, result_hash: str, schedule: Schedule | None, measurement: Measurement, notes: tuple[str, ...] = ())

An attempt that returned a result.

Attributes:

Name Type Description
run_id str

Run id.

attempt int

Attempt number.

outcome Outcome

The checked outcome.

result_hash str

result_hash() of the outcome.

schedule Schedule | None

The schedule, when the campaign records schedules.

measurement Measurement

How the attempt executed.

notes tuple[str, ...]

What the engine notices about the attempt: the claims of the solver that the checks contradict (describe()), and what its time and trace show (observe()).

Failure dataclass

Failure(run_id: str, attempt: int, category: Category, error: RunError, measurement: Measurement, solve_started: bool | None = None, last_incumbent: Incumbent | None = None, limit_incumbent: Incumbent | None = None)

An attempt that ended without a result.

Attributes:

Name Type Description
run_id str

Run id.

attempt int

Attempt number.

category Category

Whether another attempt may succeed.

error RunError

What ended the attempt.

measurement Measurement

How the attempt executed.

solve_started bool | None

Whether the solve had begun when the attempt ended.

last_incumbent Incumbent | None

The last candidate the search of a run killed at its hard limit had reported, if any.

limit_incumbent Incumbent | None

The last candidate that search had reported within the time limit of the run, if any.

utc_now

utc_now() -> str

Return the current UTC time in ISO 8601 with microseconds and a Z suffix.

classify

classify(error: BaseException) -> Category

Return whether an attempt that raised error may succeed when repeated.

Typed DispatchAtlas errors are terminal: they state a fact about the configuration, the problem, or the installation that another attempt would meet again. Any other exception is recoverable.

warm_up

warm_up(solver: Solver) -> None

Do what a solver does once in a process before it can solve.

The first call of a solver in a process may pay a cost that no later call pays: the import of a backend, or of a module that loads on first use. Paid inside a run, that cost would shorten the search of the first run of every process and of no other. This function pays it outside any run: it imports numpy.random, which NumPy loads on first use, and calls the warm_up method of the solver when the solver defines one.

Parameters:

Name Type Description Default
solver Solver

The solver about to be called.

required

warm_up_task

warm_up_task(task: Task, worker: str = 'main') -> Failure | None

Warm up the solver of an attempt, before the clock of the attempt starts.

Parameters:

Name Type Description Default
task Task

The attempt.

required
worker str

Where it executes, for the measurement.

'main'

Returns:

Type Description
Failure | None

None, or the failure of the attempt when warm_up() raised.

attempt

attempt(task: Task, worker: str = 'main', origin: float | None = None) -> Result

Execute one attempt of a run.

The solver is warmed up first (warm_up_task()), outside the time of the attempt. It is then called inside clock_origin(), whose origin is the instant the wall time of the attempt counts from, so the solver's elapsed time and time limit count from the same instant. A completion carries the notes of describe() and observe().

Exceptions raised by the solver, by its warm-up, or by the checks of its result become a Failure. KeyboardInterrupt and SystemExit propagate.

Parameters:

Name Type Description Default
task Task

The attempt.

required
worker str

Where it executes, for the measurement.

'main'
origin float | None

The instant the attempt counts from, as time.perf_counter() counts, for a caller that has warmed up the solver with warm_up_task() and times the attempt from an instant of its own. By default the attempt warms up the solver and counts from the instant that is done.

None

Returns:

Type Description
Result

The completion or the failure.

describe

describe(problem: Problem, result: SolveResult) -> tuple[Outcome, Schedule | None, tuple[str, ...]]

Check a solver's result and describe it as an outcome.

Feasibility and objective values come from the schedule, not from the solver: the violation codes of check_schedule(), and every problem objective recomputed by evaluate(). Only a malformed schedule, one that misses, repeats, or invents tasks or uses a mode a task lacks, keeps the solver's values, since none can be computed from it.

Parameters:

Name Type Description Default
problem Problem

The problem solved.

required
result SolveResult

The solver's result.

required

Returns:

Type Description
Outcome

The outcome, the schedule with normalised numbers (or None), and

Schedule | None

notes on every claim of the solver the checks contradict.

Raises:

Type Description
SolverError

If the elapsed time, or a time or a value of the trace, is not a finite number. Such a result can be neither compared with the clock of the engine nor stored in a record.

overran

overran(wall_seconds: float, limit: float, margin: float = OVERRUN_MARGIN) -> bool

Return whether a wall time passes a time limit by more than the margins allow.

It does when it exceeds the limit by more than margin of the limit and by more than CLOCK_MARGIN_SECONDS. The engine's note on a run and the assessment of a campaign's outcomes (assess_outcomes()) both count an overrun by this rule.

Parameters:

Name Type Description Default
wall_seconds float

Wall-clock seconds of a solve, on the engine's clock.

required
limit float

The time limit of the run, in seconds.

required
margin float

Share of the limit by which the wall time may pass it.

OVERRUN_MARGIN

observations

observations(outcome: Outcome, stop: StopCriteria, wall_seconds: float, *, iterative: bool) -> tuple[tuple[str, str], ...]

Return what the time and the trace of a completed attempt show, each with its kind.

One fact is returned for each of these, under the kind named first:

  • overrun: the wall time the engine measured passes the time limit by more than OVERRUN_MARGIN of the limit and by more than CLOCK_MARGIN_SECONDS;
  • early-stop: the solver stopped for its time limit while that wall time is below the limit by more than CLOCK_MARGIN_SECONDS and by more than CLOCK_MARGIN_SHARE of the limit, as when a solver gives a search a share of its limit that counts from the start of the attempt;
  • elapsed: the elapsed time the solver reports differs from that wall time by more than CLOCK_MARGIN_SECONDS and by more than CLOCK_MARGIN_SHARE of it;
  • trace-time: the trace holds a negative time, times that decrease, or a time beyond the elapsed time the solver reports, one fact for each of the three;
  • trace-evaluations: the evaluation counts of the trace decrease;
  • trace-value: the last value of the trace differs from the value of the outcome;
  • no-trace: an iterative solver returned a schedule and no trace.

The facts follow from the outcome, the stop criteria, and the wall time alone, so they can be derived again from a run record.

Parameters:

Name Type Description Default
outcome Outcome

The checked outcome of the attempt.

required
stop StopCriteria

The stop criteria the solver received.

required
wall_seconds float

Wall-clock seconds of the solve, on the engine's clock.

required
iterative bool

Whether the solver runs until a stop criterion fires.

required

Returns:

Type Description
tuple[str, str]

Pairs of a kind, one of OBSERVATION_KINDS, and the fact in

...

words; empty when nothing is found.

observe

observe(outcome: Outcome, stop: StopCriteria, wall_seconds: float, *, iterative: bool) -> tuple[str, ...]

Return what the time and the trace of a completed attempt show, as notes.

The notes are the facts of observations(), in words and in its order.

Parameters:

Name Type Description Default
outcome Outcome

The checked outcome of the attempt.

required
stop StopCriteria

The stop criteria the solver received.

required
wall_seconds float

Wall-clock seconds of the solve, on the engine's clock.

required
iterative bool

Whether the solver runs until a stop criterion fires.

required

Returns:

Type Description
tuple[str, ...]

The notes, empty when nothing is found.

engine_failure

engine_failure(run_id: str, attempt_number: int, kind: ErrorKind, message: str, measurement: Measurement, *, solve_started: bool | None = None, last_incumbent: Incumbent | None = None, limit_incumbent: Incumbent | None = None) -> Failure

Return the failure of an attempt that the engine, not the solver, ended.

Timeouts and failed transfers are terminal; a worker that dies, and one that does not begin the solve within the start-up limit, are recoverable.

exception_error

exception_error(error: BaseException, kind: ErrorKind = EXCEPTION) -> RunError

Describe an exception: its qualified class, message, and traceback.

Classes of the builtins module keep their plain name. A traceback longer than 10,000 characters keeps its end, where the error is raised.

dispatchatlas.campaigns.timeout

Hard time limits: stopping a run that ignores its own.

Stop criteria are cooperative: a solver checks them itself, and one that does not, or that is stuck inside a native extension, keeps running. The process backend therefore gives every run whose protocol sets time_limit a hard limit of timeout_factor * time_limit + grace seconds, counted from the instant its worker states as the start of the solve. When the limit passes, the supervising process kills the worker and records a terminal timeout failure: the same instance, seed, and budget would overrun again. The failure holds the last incumbent the run's search had reported, and, apart from it, the last one the search had found within the time limit. A fresh worker takes the next run. Killing a process is the one interruption that reaches code inside a native extension.

Until its solve begins, a run is under the start-up limit instead: startup_limit seconds from its hand-over, in which the worker starts its interpreter and loads plugins, if it started for this run, and rebuilds the solver and the problem. The supervising process waits for none of that: a fresh worker gets its run when it reports that it is ready. A worker that does not begin in time is killed too, and the attempt is a recoverable start-up failure, not a timeout.

A kill reaches the processes a solver started. On POSIX systems each worker leads its own process group (detach()), and the kill reaches the whole group. On Windows the supervising process puts each worker in a job object of its own (contain()), which every process the worker starts joins, and the kill ends the job. Where a job object cannot be created or the worker cannot join it, a warning says so and the kill reaches the worker alone.

A worker does not outlive its supervising process either (end_with_parent()): it watches that process and ends itself, with the process group it leads, when the process is gone. On Windows the job object also ends with the supervising process, which holds its only handle.

hard_limit

hard_limit(stop: StopCriteria, execution: Execution) -> float | None

Return the hard limit of a run in seconds.

Parameters:

Name Type Description Default
stop StopCriteria

Stop criteria of the run's protocol.

required
execution Execution

The campaign's execution settings.

required

Returns:

Type Description
float | None

timeout_factor * time_limit + grace, or None when the

float | None

protocol sets no time limit, hard limits are disabled, or the backend

float | None

is sequential and cannot enforce them.

detach

detach() -> None

Make the calling process the leader of a new session and process group.

Called first thing in every worker, so that kill() reaches the processes a solver starts. Does nothing where sessions do not exist.

end_with_parent

end_with_parent() -> None

End the calling worker process when its supervising process is gone.

A worker whose supervisor was killed would otherwise compute until its run returns and it meets the closed pipe, beside the workers of a resumed session. A daemon thread waits for the process that started this one through multiprocessing and then ends this process at once. On POSIX systems it ends the process group with it, so that processes a solver started end too, but only a group this process leads: where detach() could not start one, the group is the supervising process's, and holds whatever launched the campaign. The thread needs the interpreter to act: a worker inside a native extension that keeps the interpreter lock ends when the extension returns. Does nothing in a process that multiprocessing did not start.

contain

contain(process: BaseProcess) -> int | None

Put a started worker process in a Windows job object of its own.

The job ends every process in it when it is terminated or when its last handle closes, and every process the worker starts joins it. The calling process holds the only handle, so the worker and what it started also end when the calling process dies.

Parameters:

Name Type Description Default
process BaseProcess

A started worker.

required

Returns:

Type Description
int | None

The handle of the job, for kill() and release(). None

int | None

on other systems, and where the job cannot be created or the worker

int | None

cannot join it, for example on a Windows version without nested jobs

int | None

when the calling process is in a job itself. A warning reports that

int | None

case, and kill() then reaches the worker alone.

kill

kill(process: BaseProcess, job: int | None = None, *, wait: float = 5.0) -> None

Kill a worker process and the processes it started, and reap it.

The processes it started are reached through the worker's process group on POSIX systems and through its job object on Windows.

Parameters:

Name Type Description Default
process BaseProcess

The worker.

required
job int | None

The worker's job object, as contain() returned it.

None
wait float

Seconds to wait for the process to be reaped.

5.0

release

release(job: int | None) -> None

Close the handle of a worker's job object, if it has one.

Processes still in the job end with it, so nothing a solver started outlives the worker's slot on Windows.

dispatchatlas.campaigns.records

The records a campaign writes: completed runs, failed attempts, and sessions.

Each record is a JSON document whose schema field names its type and version:

  • dispatchatlas.run/1 (RunRecord): one completed run. It names the run (campaign, config hash, problem, solver, protocol, replicate, seed), holds its Outcome, the result_hash of that outcome, the identity_hash of the fields that name the run, the best schedule as [task, mode, start, end] rows unless the campaign disables it, and the Provenance of the attempt that produced it.
  • dispatchatlas.failure/1 (FailureRecord): one failed attempt, with its error and Category, and, for a run killed at its hard limit, the last Incumbent its search had reported.
  • dispatchatlas.session/1 (SessionRecord): one invocation of the runner, with its execution settings, the environment it ran in, the load of the machine when it began and ended, how its settings and environment differ from those of the session before it, the digest of the plan it executed, and how that plan differs from the one the directory held (PlanChange).

The result hash is the SHA-256 digest of the canonical JSON of Outcome.hashed(): every outcome field except those that measure wall time (elapsed and the elapsed column of the trace) and extras, whose solver-specific facts carry no determinism guarantee. Nothing about the machine, the session, or the attempt enters it, so a replay on any machine compares results directly (see replaying). The schedule enters it through its digest, so a record with or without stored schedule rows hashes alike.

The identity hash is the SHA-256 digest of the canonical JSON of RunRecord.identity(): the digest of the problem, the id and the parameters of the solver, the protocol, the replicate, and the seed. A record whose outcome or identity was edited after it was written no longer matches its hashes, which RunRecord.verify() reports.

These classes convert records to and from documents. Where the files live is the business of store.

TRACE_COLUMNS module-attribute

TRACE_COLUMNS: tuple[str, ...] = ('elapsed', 'evaluations', 'iteration', 'value', 'feasible')

Column order of the rows of a recorded trace.

Category

Bases: StrEnum

Whether a failed attempt may succeed when tried again.

Typed DispatchAtlas errors (an invalid configuration, a feature the solver does not model, a missing optional dependency) and hard timeouts are terminal: the instance, seed, and budget are fixed, so another attempt would fail the same way. Any other exception, a worker process that dies, and a worker that does not begin the solve within the start-up limit are recoverable: memory pressure, an I/O error, or a busy machine may not recur.

ErrorKind

Bases: StrEnum

What ended a failed attempt.

EXCEPTION class-attribute instance-attribute

EXCEPTION = 'exception'

The solver, or the checks of its result, raised an exception.

TIMEOUT class-attribute instance-attribute

TIMEOUT = 'timeout'

The solve passed its hard time limit and its worker was killed.

STARTUP class-attribute instance-attribute

STARTUP = 'startup'

The worker did not begin the solve within the start-up limit and was killed.

WORKER_EXIT class-attribute instance-attribute

WORKER_EXIT = 'worker-exit'

The worker process died during the run.

TRANSFER class-attribute instance-attribute

TRANSFER = 'transfer'

The run could not be sent to, or loaded in, a worker process.

SessionStatus

Bases: StrEnum

How a session of the runner ended.

Outcome dataclass

Outcome(status: Status, stop_reason: StopReason, objective: str, value: float | None, objectives: Mapping[str, float], bound: float | None, gap: float | None, feasible: bool, violations: tuple[str, ...], evaluations: int, iterations: int, elapsed: float, trace: tuple[TracePoint, ...] = (), front: tuple[tuple[float, ...], ...] = (), schedule_digest: str | None = None, extras: Mapping[str, Scalar] = dict())

What a run found, as recorded.

Attributes:

Name Type Description
status Status

What the solver states about its schedule.

stop_reason StopReason

Why the solver stopped.

objective str

Name of the primary objective.

value float | None

The primary objective's value, or None without a schedule.

objectives Mapping[str, float]

Every problem objective's value, computed from the schedule by evaluate().

bound float | None

Proven bound on the primary objective, for exact solvers.

gap float | None

Relative distance between value and bound, as gap defines it.

feasible bool

Whether the schedule breaks no constraint, as checked by check_schedule().

violations tuple[str, ...]

Codes of the constraints the schedule breaks, sorted.

evaluations int

Schedules the solver evaluated.

iterations int

Main-loop iterations of the solver.

elapsed float

Wall-clock seconds the solver reports.

trace tuple[TracePoint, ...]

Improvements of the best solution during the run.

front tuple[tuple[float, ...], ...]

Objective vectors of the non-dominated solutions of a multi-objective solver, in the problem's objective order.

schedule_digest str | None

schedule_digest() of the schedule, or None without one.

extras Mapping[str, Scalar]

Solver-specific facts.

replayable property

replayable: bool

Whether the stop reason leaves the run reproducible by a replay.

A run stopped by its wall-clock limit is not: how far it got depends on the speed and load of the machine. The protocol decides too, see RunRecord.replayable.

to_dict

to_dict() -> dict[str, Any]

Return the outcome as a JSON-ready dictionary.

hashed

hashed() -> dict[str, Any]

Return the fields the result hash covers: all but wall times and extras.

result_hash

result_hash() -> str

Return the SHA-256 digest of the canonical JSON of hashed().

Raises:

Type Description
SerializationError

If a value is not finite.

from_dict classmethod

from_dict(document: object) -> Outcome

Build an outcome from to_dict() output.

Raises:

Type Description
SerializationError

If the document is malformed.

ProblemRef dataclass

ProblemRef(id: str, family: str | None, digest: str)

The problem a run solved.

Attributes:

Name Type Description
id str

Problem id.

family str | None

Benchmark family of the problem, if known.

digest str

problem_digest() of the problem as solved, after any objective override.

to_dict

to_dict() -> dict[str, Any]

Return the reference as a JSON-ready dictionary.

from_dict classmethod

from_dict(document: object) -> ProblemRef

Build a reference from to_dict() output.

SolverEntry dataclass

SolverEntry(name: str, id: str, label: str | None, parameters: Mapping[str, Any], family: str, stochastic: bool, iterative: bool, multi_objective: bool, threads: int | None = None, replicates: int | None = None)

A solver of a campaign, with the facts its metadata declares.

Attributes:

Name Type Description
name str

Label, or id when the campaign gives no label.

id str

Registered solver id.

label str | None

Campaign label, if any.

parameters Mapping[str, Any]

Parameter values set by the campaign.

family str

Solver family.

stochastic bool

Whether its results depend on the seed.

iterative bool

Whether it runs until a stop criterion fires.

multi_objective bool

Whether it optimises every objective at once.

threads int | None

The value of its threads parameter, the default when the campaign sets none. None for a solver without such a parameter and for an entry that does not state it.

replicates int | None

The replicate count the campaign gives this solver alone. None when the campaign gives none and for an entry that does not state it.

to_dict

to_dict() -> dict[str, Any]

Return the entry as a JSON-ready dictionary.

from_dict classmethod

from_dict(document: object) -> SolverEntry

Build an entry from to_dict() output.

Provenance dataclass

Provenance(attempt: int, session: int, started: str, finished: str, wall_seconds: float, cpu_seconds: float | None = None, peak_rss_bytes: int | None = None, worker: str = 'main', hard_limit: float | None = None, cpu: int | None = None, concurrent: int | None = None, waited_seconds: float | None = None, shared_clock: bool | None = None)

When, where, and at what cost one attempt of a run executed.

Attributes:

Name Type Description
attempt int

Attempt number, counted from 1 across sessions.

session int

Number of the runner session that executed it.

started str

UTC start time in ISO 8601.

finished str

UTC end time in ISO 8601.

wall_seconds float

Wall-clock seconds of the solve. For an attempt the engine ended, the seconds from the start of the solve, or from the hand-over when the solve never began, until the engine ended it.

cpu_seconds float | None

CPU seconds the executing process spent on the solve, including any threads the solver started and, on POSIX systems, the child processes the solver started and waited for. On Windows the time of child processes is not included. None when not measured.

peak_rss_bytes int | None

Peak resident memory of the executing process during the solve, where the platform can reset and read it (Linux). It includes the interpreter and the loaded libraries.

worker str

"main" for the sequential backend, else the worker slot.

hard_limit float | None

Seconds after the start of its solve at which the run would have been killed, or None when it had no hard limit.

cpu int | None

Index of the processor the worker was pinned to, or None when it ran on any processor the session's process may use.

concurrent int | None

Runs executing in the session when this one was handed to its worker: itself, the runs already executing, and the runs handed over with it. None when the record does not state it.

waited_seconds float | None

Seconds from the hand-over of the attempt to its worker's slot until the instant the worker states as the start of the solve. They hold the start of the worker's interpreter and the loading of plugins, for a worker that started for this attempt, the rebuilding of the solver and the problem, and the warm-up of the solver. None when the solve never began, for the sequential backend, and when the record does not state it.

shared_clock bool | None

Whether the clocks of the solver started at the instant wall_seconds counts from, so that the elapsed time of the outcome, the times of its trace, and the time limit count from the start of the attempt. None when the record does not state it, as for an attempt the engine ended.

to_dict

to_dict() -> dict[str, Any]

Return the provenance as a JSON-ready dictionary.

from_dict classmethod

from_dict(document: object) -> Provenance

Build provenance from to_dict() output.

The fields cpu, concurrent, waited_seconds, and shared_clock may be absent, and are then None.

RunRecord dataclass

RunRecord(run_id: str, campaign: str, config_hash: str, problem: ProblemRef, solver: SolverEntry, protocol: Protocol, replicate: int, seed: int, outcome: Outcome, result_hash: str, schedule: Schedule | None, provenance: Provenance, notes: tuple[str, ...] = (), identity_hash: str | None = None)

One completed run.

Attributes:

Name Type Description
run_id str

Run id from the plan.

campaign str

Campaign id.

config_hash str

Config hash of the campaign.

problem ProblemRef

The problem solved.

solver SolverEntry

The solver and its parameters.

protocol Protocol

The protocol and its stop criteria.

replicate int

Replicate number, from 0.

seed int

Seed the solver received.

outcome Outcome

What the run found.

result_hash str

Outcome.result_hash() at the time of the run.

schedule Schedule | None

The best schedule, unless the campaign did not record it.

provenance Provenance

How the attempt that completed the run executed.

notes tuple[str, ...]

What the engine noticed about the attempt: the claims of the solver that the checks of its result contradict, a wall time beyond the time limit, an elapsed time the engine's clock contradicts, and a trace whose times or evaluation counts are out of order or that ends at another value than the outcome. Empty when the engine noticed nothing, and when the record does not state them.

identity_hash str | None

identity_digest() at the time of the run. None for a record that does not state it, whose identity is then not checked.

replayable property

replayable: bool

Whether a replay can reproduce the run.

A run stopped by its wall-clock limit cannot, and neither can any run of an iterative solver under a protocol whose parameter schedules read the elapsed time, whatever stopped it. They do when the time limit is the protocol's only bound, and under any time limit when its schedule is "binding" (schedule_timed): a search then places itself on its schedule by the elapsed time (progress()), so what it does before it stops, at a target, at another limit, or for lack of improvement, depends on the speed and load of the machine. A solver that is not iterative follows no such schedule, and its run that completed under a time limit reproduces.

verify

verify() -> tuple[str, ...]

Return the integrity problems of the record, empty when it is intact.

The outcome must hash to result_hash, stored schedule rows must match the outcome's schedule digest, and the identity must hash to identity_hash when the record states one.

identity

identity() -> dict[str, Any]

Return the fields the identity hash covers.

They are the digest of the problem, the id and the parameters of the solver, the protocol with its stop criteria, the replicate, and the seed: what, with the library, determines the result of a run. The metadata a solver declares and the campaign's label for it are left out.

identity_digest

identity_digest() -> str

Return the SHA-256 digest of the canonical JSON of identity().

to_dict

to_dict() -> dict[str, Any]

Return the record as a JSON-ready document.

from_dict classmethod

from_dict(document: object) -> RunRecord

Build a record from to_dict() output.

The field notes may be absent, and is then empty, and so may identity_hash, which is then None.

Raises:

Type Description
SerializationError

If the document is not a run record of this schema version, or is malformed.

RunError dataclass

RunError(kind: ErrorKind, message: str, type: str | None = None, traceback: str | None = None)

The error that ended a failed attempt.

Attributes:

Name Type Description
kind ErrorKind

What ended the attempt.

message str

Human-readable description.

type str | None

Qualified exception class, for exceptions.

traceback str | None

Formatted traceback, for exceptions.

to_dict

to_dict() -> dict[str, Any]

Return the error as a JSON-ready dictionary.

from_dict classmethod

from_dict(document: object) -> RunError

Build an error from to_dict() output.

Incumbent dataclass

Incumbent(objective: str, value: float, objectives: Mapping[str, float], feasible: bool, evaluations: int, iteration: int, elapsed: float)

A candidate that the search of a killed run had reported as its best so far.

The values are the ones the solver's search computed. No schedule comes with them, so they are not checked as the result of a completed run is.

Attributes:

Name Type Description
objective str

The objective the search optimised.

value float

The candidate's value of that objective.

objectives Mapping[str, float]

Every objective of the problem for the candidate, by name.

feasible bool

Whether the search found the candidate feasible.

evaluations int

Candidates the search had evaluated when it found this one.

iteration int

Main-loop iteration in which the search found it.

elapsed float

Seconds from the start of the search until it found this one.

to_dict

to_dict() -> dict[str, Any]

Return the incumbent as a JSON-ready dictionary.

from_dict classmethod

from_dict(document: object) -> Incumbent

Build an incumbent from to_dict() output.

Raises:

Type Description
SerializationError

If the document is malformed or holds a value that is not finite.

FailureRecord dataclass

FailureRecord(run_id: str, campaign: str, config_hash: str, problem: ProblemRef, solver: SolverEntry, protocol: Protocol, replicate: int, seed: int, category: Category, error: RunError, provenance: Provenance, solve_started: bool | None = None, last_incumbent: Incumbent | None = None, limit_incumbent: Incumbent | None = None)

One failed attempt of a run.

Attributes:

Name Type Description
run_id str

Run id from the plan.

campaign str

Campaign id.

config_hash str

Config hash of the campaign.

problem ProblemRef

The problem.

solver SolverEntry

The solver and its parameters.

protocol Protocol

The protocol and its stop criteria.

replicate int

Replicate number, from 0.

seed int

Seed the solver received.

category Category

Whether another attempt may succeed.

error RunError

What ended the attempt.

provenance Provenance

How the attempt executed. CPU time and memory are only known when the attempt raised an exception.

solve_started bool | None

Whether the solve had begun when the attempt ended. False for an attempt that ended while its worker was starting or receiving the run. None when the record does not state it.

last_incumbent Incumbent | None

For a run killed at its hard limit, the last candidate its search had reported. None when it had reported none, because the solver does not run inside Search or had evaluated nothing, and for every other failure. A solver that searches past its time limit may have found it after the limit.

limit_incumbent Incumbent | None

For a run killed at its hard limit, the candidate its search held when the time limit of its protocol passed: the last one it reported within that many seconds of the start of the solve. Equal to last_incumbent when the search did not improve after the limit, and None when it had reported none by then.

to_dict

to_dict() -> dict[str, Any]

Return the record as a JSON-ready document.

from_dict classmethod

from_dict(document: object) -> FailureRecord

Build a record from to_dict() output.

The fields solve_started, last_incumbent, and limit_incumbent may be absent, and are then None.

Raises:

Type Description
SerializationError

If the document is not a failure record of this schema version, or is malformed.

PlanChange dataclass

PlanChange(left: tuple[Mapping[str, Any], ...] = (), entered: tuple[Mapping[str, Any], ...] = (), reseeded: tuple[Mapping[str, Any], ...] = (), solvers: tuple[Mapping[str, Any], ...] = ())

How a plan differs from the plan a directory holds under the same config hash.

A campaign file fixes the config hash, and the plan also follows what the file does not hold: the metadata the registered solvers declare, the content of the problems, and the library that plans. Each run is a document with run_id, problem, solver, protocol, replicate, and seed.

Attributes:

Name Type Description
left tuple[Mapping[str, Any], ...]

The runs of the earlier plan that the plan does not hold, in the order of the earlier plan.

entered tuple[Mapping[str, Any], ...]

The runs of the plan that the earlier plan did not hold, in plan order.

reseeded tuple[Mapping[str, Any], ...]

The runs both plans hold whose seed differs, in plan order, each with the seed of the plan and, as earlier_seed, the seed of the earlier plan. A record of such a run was made with the earlier seed and still counts as the run's record.

solvers tuple[Mapping[str, Any], ...]

The fields by which the solver entries differ, in plan order: each a document with solver (the solver's name), field, earlier, and now. A field the earlier entry does not state is not compared.

__bool__

__bool__() -> bool

Whether the plans differ in a run, a seed, or a solver entry.

SessionRecord dataclass

SessionRecord(session: int, campaign: str, config_hash: str, started: str, finished: str | None, status: SessionStatus, execution: Execution, workers: int, environment: Mapping[str, Any], counts: Mapping[str, int] = dict(), plan_digest: str | None = None, plan_left: tuple[Mapping[str, Any], ...] = (), plan_entered: tuple[Mapping[str, Any], ...] = (), plan_reseeded: tuple[Mapping[str, Any], ...] = (), plan_solvers: tuple[Mapping[str, Any], ...] = (), changes: tuple[Mapping[str, Any], ...] = (), load: Mapping[str, Any] = dict())

One invocation of the runner on a campaign.

Attributes:

Name Type Description
session int

Session number, from 1.

campaign str

Campaign id.

config_hash str

Config hash of the campaign.

started str

UTC start time in ISO 8601.

finished str | None

UTC end time, or None while running or after a crash.

status SessionStatus

How the session ended.

execution Execution

The execution settings in force.

workers int

Number of workers, with "auto" resolved.

environment Mapping[str, Any]

The environment, as capture_environment() records it.

counts Mapping[str, int]

Run counts at the end of the session: planned, completed, failed (settled by failures), pending, and this session's attempted, succeeded, and errors.

plan_digest str | None

Digest of the run ids and seeds of the plan the session executed, Plan.runs_digest. None for a record that does not state it.

plan_left tuple[Mapping[str, Any], ...]

The runs of the plan the directory held before the session, under the same config hash, that the session's plan does not hold, as PlanChange describes them.

plan_entered tuple[Mapping[str, Any], ...]

The runs of the session's plan that the earlier plan did not hold.

plan_reseeded tuple[Mapping[str, Any], ...]

The runs both plans hold whose seed differs.

plan_solvers tuple[Mapping[str, Any], ...]

The fields by which the solver entries of the two plans differ.

changes tuple[Mapping[str, Any], ...]

How the session differs from the last session the directory recorded before it: each a document with field, a dotted path such as execution.workers or environment.library.source_digest, the values earlier and now, and kind, which is "execution", "code", or "machine". Empty for a first session and for a record that does not state them.

load Mapping[str, Any]

The load of the machine, as sample_load() measures it, under start, taken before the first run of the session, and end, taken after its last. Empty for a record that does not state it.

to_dict

to_dict() -> dict[str, Any]

Return the record as a JSON-ready document.

from_dict classmethod

from_dict(document: object) -> SessionRecord

Build a record from to_dict() output.

A document without plan_digest, plan_left, plan_entered, plan_reseeded, or plan_solvers reads as a record that states no digest and no change of plan, and one without changes or load as a record that states neither.

Raises:

Type Description
SerializationError

If the document is not a session record of this schema version, or is malformed.

settled

settled(failures: Sequence[FailureRecord], max_attempts: int) -> bool

Return whether a run's failures settle it, so it gets no further attempt.

A run that never completed is settled when its latest failure is terminal, or when it has failed max_attempts times.

schedule_rows

schedule_rows(schedule: Schedule) -> list[list[Any]]

Return a schedule as [task, mode, start, end] rows in schedule order.

schedule_digest

schedule_digest(schedule: Schedule) -> str

Return the SHA-256 digest of the canonical JSON of schedule_rows().

protocol_from_dict

protocol_from_dict(document: object) -> Protocol

Build a protocol from the dictionary its to_dict method returns.

Raises:

Type Description
SerializationError

If the document is malformed.

dispatchatlas.campaigns.store

The result store: where a campaign's files live, and how to read them back.

Layout under an output root:

<root>/<campaign id>/
    campaign.json      the campaign document, its base directory, its config hash
    plan.json          the plan, with parities and the fairness assessment
    sessions/<n>.json  one session record per invocation of the runner, with the
                       digest of its plan and the runs that left or entered it
    runs/<solver>/<problem>/r<replicate:03d>-<protocol>.json
    failures/<solver>/<problem>/r<replicate:03d>-<protocol>.attempt-<n>.json
    superseded/runs/<solver>/<problem>/r<replicate:03d>-<protocol>.<run id>.json
    superseded/failures/<solver>/<problem>/r<replicate:03d>-<protocol>.attempt-<n>.<run id>.json
                       files moved aside when new records took their places
    logs/session-<n>.log  what each session logged, with UTC time stamps
    .lock              locked while a runner works on the campaign

<solver> is the solver's name, its label or else its id. Every document is written atomically, with sorted keys, two-space indentation, ASCII text, and a final newline, and a value that is not finite is refused.

The readers read_campaign(), read_plan(), read_runs(), read_failures(), and read_sessions() return typed records. They only return records of the plan in plan.json: a file left by an earlier plan, whose run id or config hash differs, is skipped with a warning. A run record raises an error when its content no longer matches its result hash or its identity hash, or when it states another problem, solver, protocol, replicate, or seed than the plan states for its run. read_runs() also warns about every other file under runs/, which read_strays() lists: records of runs the plan does not hold, files that cannot be read, and files that are no records.

Store is the writing side, used by the runner: it locks the directory, refuses to mix configurations, and settles which runs remain. It replaces no record. A file at the path of a planned run that is not the valid record of that run, such as a record of another version of its problem, stays where it is until the run is repeated: when the repeated run gets its first record, of its result or of a failed attempt, the file is moved under superseded/, with the run id it states before .json. The file of a failed attempt is moved the same way when a new attempt takes its path. So a session that repeats no run moves nothing, and a record counts again when its problem is restored. Nothing replaces a file under superseded/, and the only reader of that folder is the store itself, which moves a record back to its path when it is the valid record of a planned run that has none.

Strays dataclass

Strays(unplanned: tuple[Path, ...] = (), unreadable: tuple[Path, ...] = (), other: tuple[Path, ...] = ())

The files under runs/ that are at no path of a planned run.

Attributes:

Name Type Description
unplanned tuple[Path, ...]

Run records, such as those of runs that left the plan.

unreadable tuple[Path, ...]

Files named *.json that cannot be read as run records: one that is not UTF-8 text or not JSON, one that is no run record, and one the operating system does not let read.

other tuple[Path, ...]

Files that are no JSON documents, such as a temporary file an interrupted write left behind.

__bool__

__bool__() -> bool

Whether there is any such file.

Stale dataclass

Stale(run_id: str, problem: str, path: Path, found: str, reason: str, changed_problem: bool = False)

A file at the path of a planned run that is no valid record of the run.

Attributes:

Name Type Description
run_id str

Id of the planned run.

problem str

Id of the problem of the planned run.

path Path

The file: the one at the path of the run's record, or one at the path of a failed attempt of the run.

found str

The run id the file states, or run_id when it states none that can be part of a file name.

reason str

Why the file does not count as a record of the run.

changed_problem bool

Whether the file is a record made on another version of the problem of the run.

Kept dataclass

Kept(run_id: str, kept: Path, path: Path)

A file under superseded/ that is the valid record of a planned run without one.

Attributes:

Name Type Description
run_id str

Id of the planned run.

kept Path

The file under superseded/.

path Path

The path of the run's record, where the file belongs.

Ledger dataclass

Ledger(completed: frozenset[str] = frozenset(), failures: Mapping[str, tuple[FailureRecord, ...]] = dict(), stale: tuple[Stale, ...] = (), kept: tuple[Kept, ...] = (), unreadable: tuple[Path, ...] = ())

What a store already holds for a plan.

Attributes:

Name Type Description
completed frozenset[str]

Ids of the runs with a valid record.

failures Mapping[str, tuple[FailureRecord, ...]]

Failed attempts of the plan's runs, by run id, in attempt order.

stale tuple[Stale, ...]

The files at the paths of planned runs that do not count as their records: those at the paths of run records in plan order, then those at the paths of failed attempts in path order.

kept tuple[Kept, ...]

The valid records under superseded/ of planned runs without a valid record at their paths, in plan order.

unreadable tuple[Path, ...]

The files under failures/ that cannot be read as records of failed attempts, in path order. They count as no attempt of any run.

restored

restored() -> Ledger

Return the ledger as it stands once every kept record is back at its path.

The runs of the kept records are complete, and the stale files at their paths are no longer listed. Nothing is moved.

next_attempt

next_attempt(run_id: str) -> int

Return the number of the next attempt of a run: one past its last failure.

closed

closed(run_id: str, max_attempts: int) -> bool

Return whether failures settle a run that never completed.

See settled() for the rule.

pending

pending(plan: Plan, max_attempts: int, *, retry_failed: bool = False) -> tuple[PlannedRun, ...]

Return the runs of a plan that still need an attempt, in plan order.

Parameters:

Name Type Description Default
plan Plan

The plan.

required
max_attempts int

Attempts a run gets before a recoverable failure settles it.

required
retry_failed bool

Also return runs that failures settled.

False

Store

Store(directory: Path)

The writing side of a campaign directory.

Parameters:

Name Type Description Default
directory Path

The campaign directory, created when missing.

required

locked

locked() -> Iterator[None]

Hold the campaign's lock, so two runners never work on it at once.

The lock is an operating-system file lock, released when the process ends in any way. Where the platform offers none, nothing is locked.

Raises:

Type Description
ConfigError

If another process holds the lock.

open

open(campaign: Campaign, *, fresh: bool = False) -> None

Prepare the directory for a campaign and record the campaign in it.

Parameters:

Name Type Description Default
campaign Campaign

The campaign.

required
fresh bool

Discard every result the directory holds first.

False

Records are checked one by one when the runner reads them, so a campaign.json that cannot be read is simply replaced.

Raises:

Type Description
ConfigError

If the directory holds results of another configuration and fresh is false.

plan_change

plan_change(plan: Plan) -> PlanChange

Report how a plan differs from the plan the directory holds, and warn about it.

A plan can change while the campaign file, and so the config hash, stays the same: the plan also follows the metadata the registered solvers declare, the content of the problems, and the library that plans. When the directory holds a plan of the same config hash, a warning names the problems whose content changed, and one names the runs that left the plan, the runs that entered it, the runs whose seed changed, and the fields of the solver entries that changed. The records of runs that left stay in the directory and are no longer read. Nothing is written, so the comparison gives the same answer until write_plan() replaces the plan.

Returns:

Type Description
PlanChange

The differences. Empty when the directory holds no plan, one

PlanChange

that cannot be read, or one of another config hash.

write_plan

write_plan(plan: Plan) -> None

Write plan.json, replacing the plan the directory holds.

ledger

ledger(plan: Plan) -> Ledger

Read which runs of a plan are complete and which attempts failed.

A file at the path of a planned run that cannot be read, belongs to another plan, no longer matches its hashes, or states another problem, solver, protocol, replicate, or seed than the plan states for the run does not count: it is listed as stale and its run is attempted again. The file of a failed attempt at the path of a planned run that belongs to another plan is listed as stale too, and is no attempt of the run. A run without a valid record whose valid record lies under superseded/ is listed as kept. Nothing is moved or written here: reconcile() brings the kept records back, and write_run() and write_failure() move a stale file aside when a new record takes its place.

The library that plans derives the seeds, so a plan can give a run another seed than the plan the directory holds. A record made with the seed its run has in that plan counts, as one made with a seed a session recorded as replaced does.

holdings

holdings(plan: Plan) -> tuple[Ledger, dict[str, RunRecord]]

Read the ledger of a plan together with the valid records it counts.

Returns:

Type Description
Ledger

The ledger, as ledger() returns it, and the record of every

dict[str, RunRecord]

run it lists as completed, by run id and in plan order. Each

tuple[Ledger, dict[str, RunRecord]]

record is read and checked once.

refuse_changed_problems

refuse_changed_problems(ledger: Ledger) -> None

Refuse a directory that holds a record made on another version of a problem.

Raises:

Type Description
ConfigError

If a stale file of the ledger is the record of a planned run, or of a failed attempt of one, made on another version of its problem.

reconcile

reconcile(ledger: Ledger) -> Ledger

Bring back the kept records of a ledger, and warn about its stale files.

Each kept record is moved from superseded/ back to the path of its run, after the stale file at that path, if there is one, is moved aside. A move the operating system refuses is warned about, and the run is then attempted again. One warning names the stale files that remain: each stays where it is until a session repeats its run (write_run(), write_failure()).

Returns:

Type Description
Ledger

The ledger after the moves: the runs whose records are back

Ledger

are complete, and no record is listed as kept.

last_session

last_session() -> SessionRecord | None

Return the record of the latest session, or None when there is none.

A session record that cannot be read is skipped with a warning, and the one before it is returned.

log_path

log_path(session: int) -> Path

Return the path of the log file of a session.

next_session

next_session() -> int

Return the number of the next session: one past the highest recorded.

write_session

write_session(record: SessionRecord) -> None

Write a session record.

write_run

write_run(record: RunRecord) -> Path

Write the record of a completed run and return its path.

A file that is at the path is moved under superseded/ first.

Raises:

Type Description
ConfigError

If a file at the path cannot be moved aside. The record is then not written.

write_failure

write_failure(record: FailureRecord) -> Path

Write the record of a failed attempt and return its path.

A file that is at the path, such as the record of an attempt on another version of the problem, is moved under superseded/ first. So is a file at the path of the run's record: a run is attempted only while it has no valid record, so what stands there is none, and the run it stood for has now been repeated.

Raises:

Type Description
ConfigError

If such a file cannot be moved aside. The record is then not written.

campaign_directory

campaign_directory(root: str | Path, campaign_id: str) -> Path

Return the directory of a campaign under an output root.

run_path

run_path(directory: Path, solver: str, problem: str, replicate: int, protocol: str) -> Path

Return the path of the record of one run.

failure_path

failure_path(directory: Path, solver: str, problem: str, replicate: int, protocol: str, attempt: int) -> Path

Return the path of the record of one failed attempt.

recorded_config_hash

recorded_config_hash(directory: str | Path) -> str | None

Return the config hash a campaign directory records, or None if it has none.

A missing or unreadable campaign.json records none.

read_campaign

read_campaign(directory: str | Path) -> Campaign

Read the campaign a store was created from.

Relative paths in its sources refer to the directory recorded with it, so the campaign can be planned again from the store alone.

Raises:

Type Description
SerializationError

If campaign.json is missing or malformed, or the campaign no longer matches its recorded config hash.

read_plan

read_plan(directory: str | Path) -> Plan

Read the plan of a campaign.

Raises:

Type Description
SerializationError

If plan.json is missing or malformed.

read_runs

read_runs(directory: str | Path, *, verify: bool = True) -> tuple[RunRecord, ...]

Read every completed run of a campaign's plan, in plan order.

Parameters:

Name Type Description Default
directory str | Path

The campaign directory.

required
verify bool

Check that every record still matches its result hash, schedule digest, and identity hash, and that it states what the plan states for its run: the problem and its digest, the solver with its parameters, the protocol, the replicate, and the seed. The comparison with the plan needs no hash, so it also covers a record that states no identity hash. A seed the run had in an earlier plan of the directory is accepted, as the session records name it.

True

Every other file under runs/ is warned about, by kind and with the first names (read_strays()).

Raises:

Type Description
SerializationError

If the plan or a record is malformed, or, with verify, a record was modified after it was written.

read_strays

read_strays(directory: str | Path) -> Strays

List the files under runs/ that are at no path of a run of the plan.

No reader returns such a file, and no session replaces it.

Raises:

Type Description
SerializationError

If the plan is missing or malformed.

read_failures

read_failures(directory: str | Path) -> tuple[FailureRecord, ...]

Read every failed attempt of a campaign's plan, by plan order and attempt.

Raises:

Type Description
SerializationError

If the plan or a record is malformed.

read_sessions

read_sessions(directory: str | Path) -> tuple[SessionRecord, ...]

Read every session record of a campaign, by session number.

Raises:

Type Description
SerializationError

If a record is malformed.

dispatchatlas.campaigns.environment

The environment a campaign session runs in.

capture_environment() records, once per session, the facts that can change a measurement or explain a difference between sessions: the Python version and implementation, the operating system and machine, the processor model with its sockets and cores, the processors the process may use, the thread limits its environment sets for numerical libraries, the memory size, the versions of DispatchAtlas, NumPy, and the optional solver backends that are installed, where the DispatchAtlas distribution was installed from and at which revision, a digest of the DispatchAtlas source files, every installed plugin distribution with its version and revision, and the git state of the DispatchAtlas source and of the working directory when either is a git checkout. It records no host or user name.

sample_load() measures how busy the processors of the machine are over a short interval. A session takes one sample before its first run and one after its last, so its record shows a machine that other work shared.

LIBRARY module-attribute

LIBRARY = 'dispatchatlas'

Name of the distribution and of the package whose source a session identifies.

PACKAGES module-attribute

PACKAGES = (LIBRARY, 'numpy', 'ortools', 'pulp', 'highspy', 'pyscipopt', 'gurobipy')

Distributions whose versions a session records when they are installed.

PLUGIN_GROUPS module-attribute

PLUGIN_GROUPS = ('dispatchatlas.solvers', 'dispatchatlas.benchmarks')

Entry-point groups whose distributions a session records as plugins.

LOAD_SAMPLE_SECONDS module-attribute

LOAD_SAMPLE_SECONDS = 0.1

Seconds over which sample_load() measures the processors by default.

UNTRACKED_HASHED module-attribute

UNTRACKED_HASHED = 1000

Most untracked files whose content enters the digest of a git state.

UNTRACKED_BYTES module-attribute

UNTRACKED_BYTES = 1 << 24

Most bytes of untracked files that are read for the digest of a git state.

TreeComparison dataclass

TreeComparison(matched: int, missing: tuple[str, ...], extra: tuple[str, ...], differing: tuple[str, ...], normalized_newlines: bool)

How the files of two directory trees compare, by relative path and SHA-256 digest.

Attributes:

Name Type Description
matched int

Files present in both trees with equal digests.

missing tuple[str, ...]

Relative paths in the source tree but not in the installed one.

extra tuple[str, ...]

Relative paths in the installed tree but not in the source one.

differing tuple[str, ...]

Relative paths in both trees whose digests differ.

normalized_newlines bool

Whether CRLF was read as LF before hashing, so that identical means the same text rather than the same bytes.

identical property

identical: bool

Whether both trees hold the same files with the same content.

Byte identity only when normalized_newlines is False.

to_dict

to_dict() -> dict[str, Any]

Return the comparison as a JSON-able dictionary, identical included.

capture_environment

capture_environment(directory: str | Path | None = None, *, exclude: Sequence[str | Path] = ()) -> dict[str, Any]

Describe the machine and software a session runs on.

Parameters:

Name Type Description Default
directory str | Path | None

The working directory whose git state to record. Defaults to the current directory.

None
exclude Sequence[str | Path]

Directories left out of every git state: that of the working directory, of the DispatchAtlas source, and of the checkout of every editable plugin. The directory a campaign writes its results to is one, since its files change with every session.

()

Returns:

Type Description
dict[str, Any]

A JSON-ready dictionary with the keys python, platform,

dict[str, Any]

cpu, threads, memory_bytes, packages, library,

dict[str, Any]

plugins, and git.

dict[str, Any]

cpu holds the processor model, its sockets and physical

dict[str, Any]

cores, the logical processors of the machine, and the

dict[str, Any]

processors the process may use: their indices as allowed and

dict[str, Any]

their number as available. When the operating system does not

dict[str, Any]

tell which processors the process is confined to, allowed is

dict[str, Any]

None and available counts every processor of the machine.

dict[str, Any]

threads holds the value of every variable of

dict[str, Any]
dict[str, Any]

environment of the process, None when it is not set; worker

dict[str, Any]

processes start with the campaign's threads_per_worker in each

dict[str, Any]

of them instead.

dict[str, Any]

library describes the DispatchAtlas distribution as

dict[str, Any]

distribution_state() does, with three differences. Its

dict[str, Any]

revision is the commit of the source when the source files are

dict[str, Any]

tracked in a git checkout. source_digest is

dict[str, Any]

source_digest() of the package. And when the package that

dict[str, Any]

runs is not the one the installed distribution provides, for

dict[str, Any]

example a copy that comes first on the module search path, the

dict[str, Any]

installer's record does not describe it: origin is then

dict[str, Any]

"path", and version, editable, and, outside a git

dict[str, Any]

checkout, revision are None, while packages still

dict[str, Any]

states the version of the installed distribution. plugins maps

dict[str, Any]

the name of every distribution that provides an entry point of

dict[str, Any]
dict[str, Any]

git holds the git_state() of the DispatchAtlas source

dict[str, Any]

under dispatchatlas and of the working directory under

dict[str, Any]

working_directory. Facts that cannot be determined are None;

dict[str, Any]

packages that are not installed and directories that are not git

dict[str, Any]

checkouts are left out.

distribution_state

distribution_state(name: str, *, exclude: Sequence[str | Path] = ()) -> dict[str, Any] | None

Return the version of an installed distribution and where it was installed from.

The installer's record of a direct installation (direct_url.json in the distribution's metadata) tells the origin.

Parameters:

Name Type Description Default
name str

Name of the distribution.

required
exclude Sequence[str | Path]

Directories left out of the git state of an editable distribution's checkout.

()

Returns:

Type Description
dict[str, Any] | None

None when no such distribution is installed. Otherwise a

dict[str, Any] | None

dictionary with version; origin, which is "vcs" for an

dict[str, Any] | None

installation from a version-control URL, "archive" for one from

dict[str, Any] | None

an archive file or URL, "directory" for one from a local

dict[str, Any] | None

directory, and "index" when the installer recorded no direct

dict[str, Any] | None

origin, as for a package index; editable, whether the installed

dict[str, Any] | None

distribution runs the files of that local directory; and

dict[str, Any] | None

revision: the commit a "vcs" origin names, the hash of an

dict[str, Any] | None

"archive", and the commit of an editable directory that is a git

dict[str, Any] | None

checkout, whose git_state() is then under git. The

dict[str, Any] | None

revision of a copy built from a local directory is None: the

dict[str, Any] | None

record does not say which state of the directory was built.

plugin_states

plugin_states(*, exclude: Sequence[str | Path] = ()) -> dict[str, dict[str, Any]]

Return the distribution_state() of every installed plugin distribution, by name.

A plugin distribution is one that provides an entry point of PLUGIN_GROUPS.

Parameters:

Name Type Description Default
exclude Sequence[str | Path]

Directories left out of the git state of every editable plugin's checkout.

()

source_digest

source_digest(package: str | Path | None = None) -> str | None

Return a digest of the files of a package directory.

The SHA-256 digest covers the relative path and the content of every file under the directory, in path order, except compiled bytecode. Line endings are read as line feeds, so a checkout and an installed copy of the same files have one digest on every platform.

Parameters:

Name Type Description Default
package str | Path | None

The package directory. Defaults to the DispatchAtlas package this module belongs to.

None

Returns:

Type Description
str | None

The hexadecimal digest, or None when a file cannot be read.

compare_trees

compare_trees(installed: str | Path, source: str | Path, *, normalize_newlines: bool = True) -> TreeComparison

Compare an installed copy of a package with the source tree it should match.

Every file under each directory is compared by its path relative to the directory and the SHA-256 digest of its content; __pycache__ folders and compiled bytecode are left out, as in source_digest().

Parameters:

Name Type Description Default
installed str | Path

The installed package directory, for example from installed_package_dir().

required
source str | Path

The package directory of the source tree.

required
normalize_newlines bool

Read CRLF line endings as LF before hashing, so a checkout with converted line endings matches a copy built from the same commit. False compares bytes exactly.

True

Raises:

Type Description
OSError

If a directory or a file cannot be read.

installed_package_dir

installed_package_dir(name: str) -> Path | None

Return the directory the import system loads package name from, or None.

The package is located without being imported.

git_state

git_state(directory: Path, *, exclude: Sequence[str | Path] = (), within: str | Path | None = None) -> dict[str, Any] | None

Return the commit of the git checkout holding directory, and how it is modified.

Parameters:

Name Type Description Default
directory Path

A directory in the checkout.

required
exclude Sequence[str | Path]

Directories whose files are left out of the comparison with the commit. One outside the checkout changes nothing.

()
within str | Path | None

A folder of the checkout, absolute or relative to directory: only its files are compared with the commit, so a change elsewhere in the checkout leaves dirty false. The commit is the checkout's HEAD either way. A folder outside the checkout returns None.

None

Returns:

Type Description
dict[str, Any] | None

None when directory is not in a git checkout or git is not

dict[str, Any] | None

available. Otherwise a dictionary with commit; dirty,

dict[str, Any] | None

whether the checkout differs from the commit in a tracked file or

dict[str, Any] | None

holds an untracked file that git does not ignore; untracked,

dict[str, Any] | None

the number of such untracked files; and digest, a SHA-256

dict[str, Any] | None

digest of the uncommitted changes: the status of every changed

dict[str, Any] | None

path, the difference of the tracked files from the commit, and the

dict[str, Any] | None

content of the untracked files. Two checkouts of one commit with

dict[str, Any] | None

different uncommitted changes therefore have different digests. The

dict[str, Any] | None

digest of a checkout that is not dirty is None, and dirty,

dict[str, Any] | None

untracked, and digest are None when git does not answer.

The untracked files are read in path order, at most UNTRACKED_BYTES bytes in all: a file that would pass that total enters the digest by its path, its size, and its modification time instead of its content. More than UNTRACKED_HASHED untracked files enter by their paths only. The queries leave the checkout as it is: git writes neither its index nor any other file for them.

sample_load

sample_load(seconds: float = LOAD_SAMPLE_SECONDS) -> dict[str, Any]

Measure how busy the processors of the machine are over a short interval.

The processor times of the whole machine are read twice, seconds apart, so the sample counts the work of every process, this one included.

Parameters:

Name Type Description Default
seconds float

Length of the interval.

LOAD_SAMPLE_SECONDS

Returns:

Type Description
dict[str, Any]

A JSON-ready dictionary with at, the UTC time the sample began

dict[str, Any]

in ISO 8601; seconds; busy, the share of the processor time

dict[str, Any]

of the machine that was not idle over the interval, between 0 and

dict[str, Any]

1; and load_average, the one-minute load average of the system.

dict[str, Any]

busy is None where the platform does not give the processor

dict[str, Any]

times, and no interval then passes; load_average is None

dict[str, Any]

where the platform has none.

dispatchatlas.campaigns.checking

Checking a campaign directory before its results are used.

check_campaign() reads a campaign directory, writes nothing, and returns a CheckReport: a list of Finding objects, each with a code, a severity, a message, and the items it concerns. It answers what no reader of results asks: whether the stored campaign is the one that was run, whether the plan is the plan that campaign gives, whether every planned run has a valid record or failures that settle it, whether the budgets were kept, and whether the sessions agree on code and settings.

A finding has one of three severities:

  • blocking: the directory does not hold what it claims, so its results are not ready to be used: a missing or altered campaign file, a plan that is not the campaign's, a run without a record that is owed an attempt, a record that fails a hash, a seed used twice.
  • limitation: the results stand, with a caveat a report of them has to state: runs settled by failures, cells with fewer runs than planned, budgets that were passed, sessions on other code.
  • note: a fact worth knowing that qualifies nothing, such as an optional field that the records of the directory do not state.

CODES lists every code with its severity and meaning. Findings on the runs of one protocol name the protocol, so a reader of one protocol's results can leave out the others.

The check reads what the engine stored and derives the rest again: the ledger of the store (Store.holdings) says which records are valid, observations() derives the facts of a record's time and trace from the record alone, so records without stored notes are covered, assess_outcomes() compares the wall times and stop reasons with the plan, and session_changes() compares the sessions that executed the counted attempts.

SEVERITIES module-attribute

SEVERITIES: tuple[Severity, ...] = ('blocking', 'limitation', 'note')

The severities, from the heaviest.

CODES module-attribute

CODES: Mapping[str, tuple[Severity, str]] = {'recipe-missing': ('blocking', 'campaign.json is missing.'), 'recipe-unreadable': ('blocking', 'campaign.json cannot be read as a campaign.'), 'recipe-altered': ('blocking', 'campaign.json no longer matches its recorded config hash.'), 'plan-unreadable': ('blocking', 'plan.json is missing or cannot be read.'), 'plan-inconsistent': ('blocking', 'plan.json holds a run of a problem, solver, or protocol it does not list, or an id twice.'), 'files-inaccessible': ('blocking', 'The operating system does not let read a file of the directory, so its runs, files, records, and sessions were not checked.'), 'plan-foreign': ('blocking', 'plan.json states another config hash than campaign.json.'), 'plan-differs': ('blocking', 'plan.json is not the plan the stored campaign gives.'), 'plan-digest-differs': ('blocking', 'plan.json holds other runs or seeds than the plan the last session executed.'), 'runs-unattempted': ('blocking', 'Planned runs have no record and no failed attempt.'), 'runs-unsettled': ('blocking', "Planned runs failed and are owed another attempt, which the campaign's retries allow."), 'record-unreadable': ('blocking', "The file at a planned run's path is no run record."), 'record-foreign': ('blocking', "The record at a planned run's path belongs to another configuration or to another version of its problem."), 'record-hash-mismatch': ('blocking', 'A record fails its result hash, its schedule digest, or its identity hash.'), 'record-differs-from-plan': ('blocking', 'A record states another problem, solver, protocol, replicate, or seed than the plan.'), 'records-set-aside': ('blocking', 'The valid record of a planned run lies under superseded/ and not at its path.'), 'seed-reused': ('blocking', 'One seed serves two replicates of one problem.'), 'plan-not-derived': ('limitation', 'The plan could not be derived again from the stored campaign on this installation.'), 'plan-changed': ('limitation', 'The plan changed between sessions while the campaign file stayed the same.'), 'runs-failed': ('limitation', 'Runs never completed and are settled by their failures.'), 'cells-partial': ('limitation', 'A solver has records for some of the runs planned on a problem, not for all.'), 'solver-without-records': ('limitation', 'A solver with planned runs has no record.'), 'solver-excluded-everywhere': ('limitation', 'The plan excludes a solver from every problem, so it has no run.'), 'records-unplanned': ('limitation', 'Records under runs/ belong to runs the plan does not hold; no reader returns them.'), 'files-unreadable': ('limitation', 'Files under runs/, at no path of a planned run, cannot be read as run records.'), 'failures-unreadable': ('limitation', 'Files under failures/ cannot be read as records of failed attempts.'), 'trace-time-axis': ('limitation', 'Traces hold a negative time, times or evaluation counts that decrease, or a time beyond the elapsed time of their run.'), 'trace-contradicts-outcome': ('limitation', 'The last value of a trace differs from the value of its outcome.'), 'record-without-trace': ('limitation', 'Records of an iterative solver hold a schedule and no trace.'), 'budget-overrun': ('limitation', 'Completed runs took more wall time than their time limit and the tolerance allow.'), 'hard-limits-differ': ('limitation', 'The attempts under one protocol ran under different hard limits.'), 'stop-reasons-diverge': ('limitation', 'The runs of the iterative solvers under one protocol ended for different reasons.'), 'sessions-unreadable': ('limitation', 'A session record cannot be read.'), 'sessions-missing': ('limitation', 'Attempts name sessions the directory holds no record of.'), 'sessions-differ-code': ('limitation', 'The sessions that executed the counted attempts differ in the Python version, the packages, the library, or a git state.'), 'sessions-differ-plugins': ('limitation', 'The sessions that executed the counted attempts differ in their plugins.'), 'sessions-differ-execution': ('limitation', 'The sessions that executed the counted attempts differ in their execution settings.'), 'sessions-differ-processors': ('limitation', 'The sessions that executed the counted attempts of a protocol with a time limit differ in the processor, the processors they were allowed, or the thread settings.'), 'replicates-below-floor': ('limitation', 'Cells of replicated solvers hold fewer completed replicates than the floor.'), 'seeds-unpaired': ('limitation', 'One replicate of a problem ran with different seeds, so its runs are not paired.'), 'plan-not-compared': ('note', 'The plan was not derived again from the stored campaign, as the caller asked.'), 'plan-digest-unstated': ('note', 'No session record states the digest of its plan.'), 'plan-entries-differ': ('note', 'plan.json holds entries the library does not write for its plan: entries derived from the plan, or texts, that were changed or that another version wrote.'), 'pairs-excluded': ('note', 'The plan excludes pairs of problem and solver.'), 'files-stray': ('note', 'Files under runs/ are no JSON documents.'), 'failures-foreign': ('note', 'Files at the paths of failed attempts belong to another plan and count as no attempt.'), 'records-without-identity-hash': ('note', 'Records state no identity hash; they are compared with the plan field by field.'), 'records-noted': ('note', 'Records carry notes of the engine that the check does not derive from the record, such as a claim of the solver that the checks of its result contradict.'), 'clock-disagreement': ('note', "A solver's elapsed time differs from the engine's wall time, or a run stopped for its time limit before the limit."), 'sessions-differ-machine': ('note', 'The sessions that executed the counted attempts differ in their machine. What of it a time limit is measured on is the limitation sessions-differ-processors.'), 'sessions-fields-unstated': ('note', 'Session records do not state a part of the environment that another states, so that part is compared among the other sessions only.'), 'session-unfinished': ('note', 'The last session is not recorded as finished.')}

Every code a finding can have, with its severity and what it means.

Severity

Severity = Literal['blocking', 'limitation', 'note']

How much a finding weighs: it blocks the use of the results, qualifies them, or informs.

Finding dataclass

Finding(code: str, severity: Severity, message: str, items: tuple[str, ...] = (), protocol: str | None = None)

One thing a check of a campaign directory found.

Attributes:

Name Type Description
code str

What was found, a key of CODES.

severity Severity

How much it weighs.

message str

One or two sentences that state it.

items tuple[str, ...]

What it concerns, one entry each: cells, files, solvers, or fields, in the order of the plan or of the paths.

protocol str | None

The protocol whose runs it concerns, or None when it concerns the whole directory.

to_dict

to_dict() -> dict[str, Any]

Return the finding as a JSON-ready dictionary.

CheckReport dataclass

CheckReport(directory: Path, campaign: str | None, config_hash: str | None, findings: tuple[Finding, ...], counts: Mapping[str, int] = dict())

What a check of a campaign directory found.

Attributes:

Name Type Description
directory Path

The campaign directory.

campaign str | None

Campaign id of the plan, or None when the plan cannot be read.

config_hash str | None

Config hash of the plan, or else the one campaign.json records, or None.

findings tuple[Finding, ...]

The findings, the blocking ones first, then the limitations, then the notes.

counts Mapping[str, int]

The planned runs by what the directory holds for them: planned, completed (a valid record), failed (settled by failures), unsettled (failed and owed an attempt), unattempted, invalid (a file at the path that is no valid record), and set_aside (the valid record lies under superseded/). Empty when the check ended at the plan or at a file the operating system does not let read.

ok property

ok: bool

Whether no finding is blocking.

of

of(severity: Severity) -> tuple[Finding, ...]

Return the findings of one severity.

find

find(code: str) -> tuple[Finding, ...]

Return the findings with the given code, one per protocol it was found under.

to_dict

to_dict() -> dict[str, Any]

Return the report as a JSON-ready dictionary.

check_campaign

check_campaign(directory: str | Path, *, replicate_floor: int | None = None, overrun_tolerance: float = OVERRUN_MARGIN, replan: bool = True, registry: SolverRegistry | None = None, families: FamilyRegistry | None = None) -> CheckReport

Check a campaign directory and report what qualifies its results.

Nothing is written, moved, or run. The findings are, by what they concern:

  • the campaign file: missing, unreadable, or no longer matching its config hash;
  • the plan: of another config hash, different from the plan the stored campaign gives when it is derived again, different from the plan the last session executed, or changed between sessions;
  • the planned runs: without a record and without a failed attempt, with failures that do not settle them, settled by failures (by solver and kind of error), cells with records for only some of their runs, solvers without any record, and, apart from those, the pairs and the solvers the plan excludes;
  • the files: a file at a planned run's path that is no valid record of the run (unreadable, of another configuration or problem version, failing a hash, or differing from the plan), valid records lying under superseded/, records of runs the plan does not hold, and files that cannot be read or are no records;
  • the records: a trace with an unusable time axis or a last value that contradicts the outcome, an iterative solver's record without a trace, and wall times beyond the time limit, by solver with the largest ratio;
  • the protocols: hard limits that differ among the attempts and stop reasons that diverge among the iterative solvers;
  • the sessions: those that executed the counted attempts and differ in code, plugins, execution settings, or machine, each field that differs with its values and the sessions that have each; those that executed the attempts of a protocol with a time limit and differ in the processor, the processors they were allowed, or the thread settings; and those without a record;
  • the replication: cells below replicate_floor, a seed used for two replicates of one problem, and a replicate that ran with two seeds.

Records and session records may lack the fields that are optional in them. Where the check cannot make a comparison for that reason, it says so in a note: records without an identity hash, session records without the digest of their plan, and session records that state less of their environment than another's.

Parameters:

Name Type Description Default
directory str | Path

A campaign directory, <root>/<campaign id>.

required
replicate_floor int | None

The fewest completed replicates a cell of a replicated solver may hold without a finding. None sets no floor.

None
overrun_tolerance float

Share of a time limit by which the wall time of a completed run may pass it without counting as over, as in assess_outcomes().

OVERRUN_MARGIN
replan bool

Derive the plan again from the stored campaign and compare it with plan.json. This loads every problem of the campaign, so it needs the campaign's instance files and solvers.

True
registry SolverRegistry | None

Solver registry for deriving the plan. Defaults to the built-in solvers and plugins.

None
families FamilyRegistry | None

Benchmark-family registry for deriving the plan. Defaults to the built-in families and plugins.

None

Returns:

Type Description
CheckReport

The report. With a plan that cannot be read, or whose runs name

CheckReport

what it does not list, it holds the findings on the campaign file

CheckReport

and the plan only and no counts. When the operating system does not

CheckReport

let read a record, a file of a failed attempt, or a folder, as it

CheckReport

can while a session writes into the directory, the report says so

CheckReport

in a blocking finding and holds no counts either.

Raises:

Type Description
ConfigError

If directory is no directory, or replicate_floor is not a positive integer, or overrun_tolerance is not zero or a positive number.

dispatchatlas.campaigns.replaying

Replay: run completed runs again and check that they reproduce.

replay() re-executes runs of a campaign from their recorded seeds, with the same solver parameters and stop criteria, and compares result hashes. Before that it checks each record against its own result hash, so a record edited after it was written is caught even though its run reproduces. It never writes to the store.

A run stopped by its wall-clock limit is skipped: how far it got depends on the speed and load of the machine. Other runs reproduce exactly on any machine where the solver's arithmetic does; transcendental functions of the C library may differ in their last bit between platforms, which can steer a stochastic search elsewhere. Each mismatch lists the outcome fields that differ.

Verdict

Bases: StrEnum

What replaying one run showed.

MATCH class-attribute instance-attribute

MATCH = 'match'

The run reproduced its recorded result, and the record is intact.

MISMATCH class-attribute instance-attribute

MISMATCH = 'mismatch'

The result differs, or the record no longer matches its result hash.

SKIPPED class-attribute instance-attribute

SKIPPED = 'skipped'

The run cannot be replayed, or has no completed record.

FAILED class-attribute instance-attribute

FAILED = 'failed'

The replay attempt failed.

Difference dataclass

Difference(field: str, recorded: Any, replayed: Any)

One outcome field whose recorded and replayed values differ.

Attributes:

Name Type Description
field str

Name of the field.

recorded Any

The value in the record.

replayed Any

The value the replay produced.

ReplayEntry dataclass

ReplayEntry(run_id: str, solver: str, problem: str, protocol: str, replicate: int, verdict: Verdict, detail: str = '', differences: tuple[Difference, ...] = ())

The verdict on one run.

Attributes:

Name Type Description
run_id str

Run id.

solver str

Solver name.

problem str

Problem id.

protocol str

Protocol name.

replicate int

Replicate number.

verdict Verdict

What the replay showed.

detail str

Explanation of a verdict other than a match.

differences tuple[Difference, ...]

Differing outcome fields, for a mismatch.

ReplayReport dataclass

ReplayReport(campaign: str, config_hash: str, entries: tuple[ReplayEntry, ...])

The verdicts of a replay, in plan order.

Attributes:

Name Type Description
campaign str

Campaign id.

config_hash str

Config hash of the campaign.

entries tuple[ReplayEntry, ...]

One entry per selected run.

ok property

ok: bool

Whether no run mismatched or failed.

count

count(verdict: Verdict) -> int

Return how many runs got a verdict.

replay

replay(campaign: Campaign, root: str | Path, *, runs: Iterable[str] | None = None, registry: SolverRegistry | None = None, families: FamilyRegistry | None = None, backend: str | None = None, workers: int | str | None = None, affinity: str | Sequence[int] | None = None) -> ReplayReport

Re-execute completed runs and compare their results with the records.

Parameters:

Name Type Description Default
campaign Campaign

The campaign the results belong to.

required
root str | Path

Output root the campaign ran under.

required
runs Iterable[str] | None

Ids of the runs to replay. Defaults to every completed run.

None
registry SolverRegistry | None

Solver registry. Defaults to the built-in solvers and plugins.

None
families FamilyRegistry | None

Benchmark-family registry. Defaults to the built-in families and plugins.

None
backend str | None

Override of the campaign's execution backend.

None
workers int | str | None

Override of the campaign's worker count, or "auto".

None
affinity str | Sequence[int] | None

Override of the campaign's affinity: "inherit", "pin", or processor indices.

None

Returns:

Type Description
ReplayReport

The verdict on every selected run.

Raises:

Type Description
ConfigError

If the directory holds another configuration, has no results, a requested run id is not in the plan, or runs are to be replayed and the workers cannot be pinned as affinity asks.