Campaigns¶
Campaign configuration and files, planning, fairness, estimation, execution, records, the result store, and replay.
dispatchatlas.campaigns
¶
Reproducible, resumable experiment campaigns.
A campaign runs every solver of a roster on every problem of its instance sources, under every stopping protocol, with seeds shared across solvers, and stores one JSON record per run:
from dispatchatlas.campaigns import load_campaign, read_runs, run
campaign = load_campaign("campaign.toml")
summary = run(campaign, "results") # resumes when called again
records = read_runs(summary.directory)
The modules, in the order a campaign meets them:
configandio: the campaign and its strict TOML or JSON file, and the config hash;planning: problems, seeds, run ids, and the plan, checked bycompatibilityandfairness;estimation: run counts and time bounds;runner,backends,execution,timeout, andprocessors: execution with retries, hard time limits, resumption, and the processors the workers run on;records,store, andenvironment: what is written, where, and in which environment;checking: the check of a campaign directory before its results are used;replaying: verification that runs reproduce.
dispatchatlas.campaigns.config
¶
Campaign configuration: the instances, solvers, protocols, and settings of an experiment.
A Campaign states what determines an experiment's results and,
separately, how to execute it:
instances: one or more instance sources, seeInstanceSource;solvers: registered solver ids with parameter values. A label tells two parameter settings of one solver apart;protocols: named stop criteria. Every solver runs under every protocol with the same criteria, so each protocol is one fair comparison;replicates: runs per stochastic solver, problem, and protocol. Deterministic solvers run once, unlessreplicate_timedgives them every replicate under a time limit or their entry states its own count;seed: the root of every run seed, andseed_namespace, an optional name that separates the seeds of campaigns that share a root;objectives: objectives that replace those the problems declare, if given;executionandmetadata: how to execute, and free annotations.
The config hash identifies an experiment. It is the SHA-256 digest of the
canonical JSON of Campaign.identity(): the fields seed,
instances, solvers, protocols, replicates, and objectives
as written, so reordering a list also changes it, and each optional setting
that is set to other than its default. Three things stay outside
it. The campaign id only names where the results are stored.
execution and metadata do not change what a run computes. And the
locations of instance files and directories are left out of instances:
a problem read from a file is identified by its content, whose digest enters
the id of every run on it (see planning), so
a campaign keeps its identity when its files move or live elsewhere on
another machine.
SCHEMA_VERSION
module-attribute
¶
Version of the campaign file format this release reads and writes.
DEFAULT_REPLICATES
module-attribute
¶
Runs per stochastic solver, problem, and protocol when a campaign does not say.
DEFAULT_TIMEOUT_FACTOR
module-attribute
¶
Default multiple of a protocol's time limit after which a run is killed.
GRACE_SECONDS
module-attribute
¶
Default seconds a run may exceed timeout_factor * time_limit before it is killed.
The grace lets a solver that honours its limit finish the step it is in and return its result.
STARTUP_SECONDS
module-attribute
¶
Default seconds a worker process has, from the hand-over of a run, to begin its solve.
They cover the start of the worker's interpreter, the loading of plugins, and the rebuilding of the solver and the problem, on a machine that may be busy.
CPU_RESERVE
module-attribute
¶
Processors that workers = "auto" leaves to the supervising process and the system.
BACKENDS
module-attribute
¶
Execution backends: in the calling process, or in supervised worker processes.
THREAD_VARIABLES
module-attribute
¶
THREAD_VARIABLES = ('OMP_NUM_THREADS', 'MKL_NUM_THREADS', 'OPENBLAS_NUM_THREADS', 'BLIS_NUM_THREADS', 'VECLIB_MAXIMUM_THREADS', 'NUMEXPR_NUM_THREADS')
Environment variables that limit the thread pools of numerical libraries.
Every worker process starts with each of them set to the campaign's
threads_per_worker.
InstanceSource
¶
InstanceSource = FamilySource | SuiteSource | CatalogSource | DirectorySource | FileSource
Where a campaign's problems come from.
FamilySource
dataclass
¶
Problems generated by a registered benchmark family, one per seed.
Attributes:
| Name | Type | Description |
|---|---|---|
family |
str
|
Id of a family in the benchmark registry, for example
|
seeds |
tuple[int, ...]
|
Generator seeds. Each gives one problem, identified by the id the family assigns. |
parameters |
Mapping[str, Any]
|
Family parameters; the others keep their defaults. |
SuiteSource
dataclass
¶
Instances of a published suite from dispatchatlas.benchmarks.suites.
Attributes:
| Name | Type | Description |
|---|---|---|
suite |
str
|
Suite id, for example |
instances |
tuple[str, ...]
|
Instance names, for example |
directory |
Path | None
|
Directory holding the downloaded files of a file suite. Generated suites need none. |
CatalogSource
dataclass
¶
Every instance of a built-in catalog, see build_catalog().
Attributes:
| Name | Type | Description |
|---|---|---|
catalog |
str
|
Catalog name, for example |
seed |
int
|
Root seed the catalog derives its instance seeds from. |
DirectorySource
dataclass
¶
Problems written by materialize().
Every file is checked against the SHA-256 digest its manifest records.
Attributes:
| Name | Type | Description |
|---|---|---|
directory |
Path
|
The directory holding |
FileSource
dataclass
¶
One instance file.
Attributes:
| Name | Type | Description |
|---|---|---|
file |
Path
|
A DispatchAtlas problem document ( |
format |
str | None
|
Format name for |
SolverSpec
dataclass
¶
SolverSpec(id: str, label: str | None = None, parameters: Mapping[str, Any] = dict(), replicates: int | None = None)
A registered solver with parameter values.
Attributes:
| Name | Type | Description |
|---|---|---|
id |
str
|
Id in the solver registry, for example |
label |
str | None
|
Name that tells this setting apart from other settings of the same solver. Defaults to the id. |
parameters |
Mapping[str, Any]
|
Solver parameters; the others keep their defaults. |
replicates |
int | None
|
Runs of this solver per problem and protocol, whatever
the solver declares and whatever the protocol bounds. |
to_dict
¶
Return the specification as a JSON-ready dictionary.
replicates is part of it only when the specification sets one.
arguments
¶
Return the parameters as keyword arguments: tables as dictionaries, lists as tuples.
Solver instances built from them can be pickled, which the process backend needs.
Protocol
dataclass
¶
Protocol(name: str, stop: StopCriteria)
Named stop criteria that every solver of a campaign runs under.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
Protocol name, used in result file names. |
stop |
StopCriteria
|
The criteria. They must bound the run with a time, evaluation, or iteration limit, because iterative solvers never stop otherwise. |
Execution
dataclass
¶
Execution(backend: str = 'process', workers: int | Literal['auto'] = 1, timeout_factor: float | None = DEFAULT_TIMEOUT_FACTOR, grace: float = GRACE_SECONDS, retries: int = 0, record_schedules: bool = True, affinity: Literal['inherit', 'pin'] | tuple[int, ...] = 'inherit', threads_per_worker: int = 1, startup_limit: float = STARTUP_SECONDS, on_changed_code: Literal['warn', 'refuse'] = 'warn', on_changed_problem: Literal['rerun', 'refuse'] = 'rerun', log_level: Literal['inherit', 'info', 'debug'] = 'inherit')
How a campaign executes. None of these settings changes a run's result.
Attributes:
| Name | Type | Description |
|---|---|---|
backend |
str
|
|
workers |
int | Literal['auto']
|
Worker processes, or |
timeout_factor |
float | None
|
A run whose protocol sets |
grace |
float
|
Seconds added to the hard limit. |
retries |
int
|
Further attempts for a run whose failure may be transient. |
record_schedules |
bool
|
Store every run's best schedule in its record. Its digest is stored either way. |
affinity |
Literal['inherit', 'pin'] | tuple[int, ...]
|
The processors the worker processes run on. |
threads_per_worker |
int
|
Threads the numerical libraries of a worker
process may use. Every variable of |
startup_limit |
float
|
Seconds a worker process has, from the hand-over of a
run, to begin its solve. A worker that takes longer is killed,
and the attempt is a start-up failure, which |
on_changed_code |
Literal['warn', 'refuse']
|
What a session does when the code it runs on differs
from the code of the last recorded session of the campaign
directory: the Python version, the version of a recorded
package, the library's distribution, revision, or source
digest, a plugin distribution, or the git state of the
library or of the working directory. |
on_changed_problem |
Literal['rerun', 'refuse']
|
What a session does when the directory holds
the record of a planned run made on another version of its
problem. |
log_level |
Literal['inherit', 'info', 'debug']
|
Which records of the library's loggers the session log
receives beside the engine's own, which it receives from the
level |
max_attempts
property
¶
Attempts a run gets before a failure that may be transient settles it.
worker_count
¶
Return the number of workers, resolving "auto" on this machine.
worker_cpus
¶
Return the processor each worker is pinned to, resolved on this machine.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
workers
|
int | None
|
Number of workers. Defaults to |
None
|
Returns:
| Type | Description |
|---|---|
tuple[int, ...] | None
|
One processor index per worker, in worker order, or |
tuple[int, ...] | None
|
|
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the platform cannot pin a process, the workers outnumber the processors they are pinned to, or a listed processor is not one the calling process may use. |
Campaign
dataclass
¶
Campaign(id: str, seed: int, instances: tuple[InstanceSource, ...], solvers: tuple[SolverSpec, ...], protocols: tuple[Protocol, ...], replicates: int = DEFAULT_REPLICATES, objectives: tuple[str, ...] | None = None, execution: Execution = Execution(), metadata: Mapping[str, Any] = dict(), base: Path | None = None, replicate_timed: bool = False, seed_namespace: str | None = None)
A reproducible experiment: problems, solvers, protocols, and replicates.
Attributes:
| Name | Type | Description |
|---|---|---|
id |
str
|
Campaign name. Results go to |
seed |
int
|
Root seed of every run seed. |
instances |
tuple[InstanceSource, ...]
|
Where the problems come from. |
solvers |
tuple[SolverSpec, ...]
|
The solvers to compare, with unique names. |
protocols |
tuple[Protocol, ...]
|
Stop criteria every solver runs under, with unique names. |
replicates |
int
|
Runs per stochastic solver, problem, and protocol. |
objectives |
tuple[str, ...] | None
|
Objectives, primary first, that replace those every
problem declares. |
execution |
Execution
|
How to execute. Not part of the config hash. |
metadata |
Mapping[str, Any]
|
Free annotations. Not part of the config hash. |
base |
Path | None
|
Directory that relative paths in instance sources refer to;
the directory of the campaign file when read by
|
replicate_timed |
bool
|
Whether a solver declared deterministic runs
|
seed_namespace |
str | None
|
A name that separates the seeds of this campaign
from those of every campaign with the same |
config_hash
cached
property
¶
SHA-256 hex digest of the canonical JSON of identity().
identity
¶
Return the fields that determine the results, as JSON-ready values.
These are exactly the fields the config hash covers: seed,
instances without file-system locations, solvers,
protocols, replicates, and objectives, and the optional
settings that are set: replicate_timed when it is true,
seed_namespace when there is one, a solver's own replicates,
and a protocol's schedule when it is not the default. A setting
left at its default is absent from the identity.
solver
¶
solver(name: str) -> SolverSpec
Return the solver with the given name.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If no solver has that name. |
protocol
¶
protocol(name: str) -> Protocol
Return the protocol with the given name.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If no protocol has that name. |
allowed_cpus
¶
Return the indices of the processors this process may run on, sorted.
When the set cannot be read (see affinity_cpus()), every processor
of the machine is returned. The result is never empty.
available_cpus
¶
Return how many processors this process may run on, at least one.
This is the size of allowed_cpus(), so affinity masks, the cpuset
of a container, and taskset limits are respected.
check_name
¶
Return value if it can name a file or directory on every platform.
Campaign ids, solver names, protocol names, and problem ids name files in
the result store, so they must start with a letter or digit and contain
only letters, digits, ., _, and -.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If |
jsonable
¶
Return a frozen configuration value as plain JSON types.
Tables become dictionaries and sequences become lists, recursively.
dispatchatlas.campaigns.processors
¶
The processors a process may run on, and placing a process on one of them.
An operating system can confine a process to some of the machine's logical
processors: a scheduler affinity set on Linux (taskset, the cpuset of a
container) or a process affinity mask on Windows. A new process starts with
the set of the thread that creates it. allowed_cpus() reads the set of
the calling process, so a worker count or a warning follows what the process
may use and not what the machine has. confined() narrows the caller to
one processor while it creates a process, which therefore runs there from its
first instruction, and pin_process() confines a process that is already
running.
- Where
os.sched_getaffinityexists (Linux and some other POSIX systems), the set is read and written with the scheduler calls. They act on one thread: a thread started later inherits the set, a thread that is already running keeps its own. - On Windows the set is read with
GetProcessAffinityMaskand written withSetProcessAffinityMask, which act on every thread of a process. One mask describes at most 64 processors, so on a machine with several processor groups the set cannot be read and pinning is refused. - Elsewhere (macOS) the system offers no such call: the set cannot be read and pinning is refused.
A set that cannot be read counts as every processor of the machine;
affinity_cpus() tells a reading from that count. A limit on processor
time, such as the CPU quota of a container, is not an affinity and is not
seen.
Processors are named by the index the operating system gives them, from 0.
affinity_cpus
¶
Return the processors the operating system confines this process to, sorted.
Returns:
| Type | Description |
|---|---|
tuple[int, ...] | None
|
The processor indices, or |
tuple[int, ...] | None
|
platform has no affinity call, the call fails, or one Windows mask |
tuple[int, ...] | None
|
cannot describe the process. |
allowed_cpus
¶
Return the indices of the processors this process may run on, sorted.
When the set cannot be read (see affinity_cpus()), every processor
of the machine is returned. The result is never empty.
available_cpus
¶
Return how many processors this process may run on, at least one.
This is the size of allowed_cpus(), so affinity masks, the cpuset
of a container, and taskset limits are respected.
describe_cpus
¶
Return processor indices as text, with runs collapsed: "0-3, 8".
pinning_refusal
¶
Return why this platform cannot pin a process, or None when it can.
confined
¶
Confine the caller to one processor while it creates a process.
A process created inside the context starts on cpu, and so does every
thread it starts, whenever it starts it. On Windows every thread of the
calling process is confined for the duration; elsewhere only the calling
thread is. The caller has its earlier processors back on exit.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cpu
|
int
|
Index of the processor, one of |
required |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the platform cannot pin a process (see
|
pin_process
¶
Confine a running process to one processor.
On Windows every thread of the process moves. Elsewhere the main thread
moves, with every thread started from then on; a thread that is already
running stays where it is, so use confined() to place a process
from its start.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pid
|
int
|
Id of the process, which the caller must be allowed to change: itself or one of its children. |
required |
cpu
|
int
|
Index of the processor, one of |
required |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the platform cannot pin a process (see
|
dispatchatlas.campaigns.io
¶
Reading and writing campaign files.
Campaign files are TOML, read with the standard library's tomllib, or
JSON. One strict reader serves both: every table rejects keys it does not
know, every value must have its documented type (true is not a number and
"false" is not a boolean), a JSON object may not repeat a key, and the
document must declare schema_version = 1. Relative paths are resolved
against the directory of the file.
Example:
schema_version = 1
id = "shop-baselines"
seed = 2024
replicates = 10
[[instances]]
family = "flow-shop"
parameters = { jobs = 10, machines = 5 }
seeds = [1, 2, 3]
[[instances]]
suite = "taillard-flow-shop"
instances = ["ta001", "ta002"]
[[solvers]]
id = "random-search"
[[solvers]]
id = "hill-climbing"
label = "hill-climbing-r50"
parameters = { restart_after = 50 }
[[protocols]]
name = "evaluations"
stop = { max_evaluations = 5000 }
[execution]
workers = "auto"
retries = 1
An instance source is one of {family, seeds, parameters?},
{suite, instances, directory?}, {catalog, seed?}, {directory},
or {file, format?}. A solver is {id, label?, parameters?, replicates?}.
A protocol's stop table takes the fields of
StopCriteria. The optional top-level keys
are replicates, replicate_timed, seed_namespace, objectives,
execution (the fields of
Execution), and metadata. In
JSON, null stands for an absent optional value.
load_campaign
¶
load_campaign(path: str | Path) -> Campaign
Read a campaign file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
A |
required |
Returns:
| Type | Description |
|---|---|
Campaign
|
The campaign. Relative paths in its instance sources refer to the |
Campaign
|
file's directory. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the file cannot be read or is not a valid campaign. |
loads_campaign
¶
loads_campaign(text: str, *, format: str = 'toml', base: Path | None = None) -> Campaign
Parse a campaign from TOML or JSON text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
str
|
The document. |
required |
format
|
str
|
|
'toml'
|
base
|
Path | None
|
Directory that relative paths refer to. |
None
|
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the text is not a valid campaign document. |
campaign_to_dict
¶
campaign_to_dict(campaign: Campaign) -> dict[str, Any]
Return a campaign as a JSON-ready document that campaign_from_dict() reads back.
The directory that relative paths refer to, Campaign.base, is not
part of the document.
campaign_from_dict
¶
campaign_from_dict(document: Mapping[str, Any], *, base: Path | None = None) -> Campaign
Build a campaign from a parsed TOML or JSON document.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
document
|
Mapping[str, Any]
|
The document, as |
required |
base
|
Path | None
|
Directory that relative paths refer to. |
None
|
Raises:
| Type | Description |
|---|---|
ConfigError
|
If a key is unknown, a value has the wrong type, a
required key is missing, or the values are inconsistent. The
message names the offending entry, for example
|
dispatchatlas.campaigns.planning
¶
Planning: from a campaign to the runs it needs.
plan() loads every problem the campaign's sources name, checks every
solver id and parameter set against the registry, and expands the grid
problem x replicate x solver x protocol into PlannedRun items:
- Replicates. A stochastic solver runs
campaign.replicatestimes per problem and protocol, as replicates0, 1, .... A deterministic one runs once, as replicate 0, unless the campaign setsreplicate_timed: it then runs every replicate under each protocol that sets a time limit, where its result depends on the machine. A solver whose entry carries its ownreplicatesruns that many times under every protocol, whatever it declares. - Seeds. Seeds are common random numbers: replicate
rof problempreceivesderive_seed(campaign.seed, "problem:" + p, r)under every solver and protocol. Results are therefore paired across solvers, and adding, removing, or reordering solvers never changes a seed. A campaign with aseed_namespacederives them from the rootderive_seed(campaign.seed, "namespace:" + namespace)in place of its seed, so two campaigns that share a seed and differ in namespace share no run seed. - Run ids.
"run-"and the first 16 hexadecimal digits of the SHA-256 digest of the canonical JSON of the config hash, the problem id and digest, the solver id, label, and parameters, the protocol name, and the replicate. Equal campaigns give equal run ids on every machine, and a problem whose content changes gets new ones. - Compatibility. A pair whose solver does not model a feature of the
problem is excluded, and the plan records why (see
compatibility).
The problem digest is that of the problem as solved, after any objective
override. assess() verifies every plan
before it is returned.
PlannedProblem
dataclass
¶
PlannedProblem(id: str, family: str | None, digest: str, source: int, tasks: int, resources: int, objectives: tuple[str, ...], features: tuple[str, ...])
A problem of a plan.
Attributes:
| Name | Type | Description |
|---|---|---|
id |
str
|
Problem id. |
family |
str | None
|
Benchmark family, if known. |
digest |
str
|
Digest of the problem as solved. |
source |
int
|
Index of the instance source that produced it. |
tasks |
int
|
Number of tasks. |
resources |
int
|
Number of resources. |
objectives |
tuple[str, ...]
|
Objectives it is solved for, primary first. |
features |
tuple[str, ...]
|
Its structural features, sorted. |
PlannedRun
dataclass
¶
One run of a plan: a solver on a problem under a protocol.
Attributes:
| Name | Type | Description |
|---|---|---|
run_id |
str
|
Run id. |
problem |
str
|
Problem id. |
solver |
str
|
Solver name. |
protocol |
str
|
Protocol name. |
replicate |
int
|
Replicate number, from 0. |
seed |
int
|
Seed the solver receives. |
Plan
dataclass
¶
Plan(campaign: str, config_hash: str, seed: int, replicates: int, problems: tuple[PlannedProblem, ...], solvers: tuple[SolverEntry, ...], protocols: tuple[Protocol, ...], runs: tuple[PlannedRun, ...], exclusions: tuple[Exclusion, ...] = (), caveats: tuple[Caveat, ...] = (), replicate_timed: bool = False, seed_namespace: str | None = None)
Every run a campaign needs, with the facts that explain it.
A plan is plain data: the store writes it to plan.json and
read_plan() reads it back.
Attributes:
| Name | Type | Description |
|---|---|---|
campaign |
str
|
Campaign id. |
config_hash |
str
|
Config hash of the campaign. |
seed |
int
|
Campaign seed. |
replicates |
int
|
Replicates of stochastic solvers. |
problems |
tuple[PlannedProblem, ...]
|
The problems, in source order. |
solvers |
tuple[SolverEntry, ...]
|
The solvers, in campaign order. |
protocols |
tuple[Protocol, ...]
|
The protocols, in campaign order. |
runs |
tuple[PlannedRun, ...]
|
The runs, ordered by problem, replicate, solver, and protocol. |
exclusions |
tuple[Exclusion, ...]
|
Pairs that do not run, and why. |
caveats |
tuple[Caveat, ...]
|
Pairings that run but need care in the analysis. |
replicate_timed |
bool
|
Whether a solver declared deterministic runs every replicate under a protocol that sets a time limit. |
seed_namespace |
str | None
|
The campaign's seed namespace, or |
runs_digest
cached
property
¶
SHA-256 hex digest of the canonical JSON of every run's id and seed, sorted by id.
Two plans have the same digest exactly when they hold the same runs
with the same seeds, in whatever order. A run id covers the config
hash, the problem and its content, the solver and its parameters, the
protocol, and the replicate; the seed is derived by the library that
plans. The digest does not cover the solver entries: the metadata a
solver declares can differ between two plans of one digest, and
Store.plan_change names such a
difference.
run
¶
run(run_id: str) -> PlannedRun
Return the run with the given id.
Raises:
| Type | Description |
|---|---|
KeyError
|
If the plan has no such run. |
problem
¶
problem(problem_id: str) -> PlannedProblem
Return the problem with the given id.
Raises:
| Type | Description |
|---|---|
KeyError
|
If the plan has no such problem. |
solver
¶
solver(name: str) -> SolverEntry
Return the solver with the given name.
Raises:
| Type | Description |
|---|---|
KeyError
|
If the plan has no such solver. |
protocol
¶
protocol(name: str) -> Protocol
Return the protocol with the given name.
Raises:
| Type | Description |
|---|---|
KeyError
|
If the plan has no such protocol. |
replicates_of
¶
replicates_of(solver: SolverEntry, protocol: Protocol | None = None) -> int
Return how many replicates a solver runs under a protocol.
The solver's own count when its entry carries one. Otherwise every
replicate for a stochastic solver, and for one declared deterministic
a single run, or every replicate when replicate_timed is set
and the protocol sets a time limit.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
solver
|
SolverEntry
|
A solver of the plan. |
required |
protocol
|
Protocol | None
|
A protocol of the plan. |
None
|
to_dict
¶
Return the plan as a JSON-ready document.
Besides the plan's fields, the document lists every protocol's
parities, the fair-comparison assessment, and runs_digest,
derived from the plan for readers of the file.
from_dict
classmethod
¶
from_dict(document: object) -> Plan
Build a plan from to_dict() output. Derived entries are ignored.
A document without replicate_timed or seed_namespace reads as
a plan that sets neither.
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the document is not a plan of this schema version, or is malformed. |
Prepared
dataclass
¶
Prepared(campaign: Campaign, plan: Plan, problems: Mapping[str, Problem], registry: SolverRegistry)
A plan together with what executing it needs.
Attributes:
| Name | Type | Description |
|---|---|---|
campaign |
Campaign
|
The campaign. |
plan |
Plan
|
Its plan. |
problems |
Mapping[str, Problem]
|
The problems as solved, by id. |
registry |
SolverRegistry
|
The solver registry the plan was checked against. |
run_seed
¶
Return the seed of every run of one problem and replicate.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
root
|
int
|
The campaign seed. |
required |
problem_id
|
str
|
Id of the problem. |
required |
replicate
|
int
|
Replicate number, from 0. |
required |
namespace
|
str | None
|
The campaign's seed namespace, or |
None
|
Returns:
| Type | Description |
|---|---|
int
|
|
int
|
namespace, |
int
|
place of |
run_id
¶
run_id(config_hash: str, problem_id: str, problem_hash: str, solver: SolverSpec, protocol: str, replicate: int) -> str
Return the id of one run.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config_hash
|
str
|
Config hash of the campaign. |
required |
problem_id
|
str
|
Id of the problem. |
required |
problem_hash
|
str
|
Digest of the problem as solved. |
required |
solver
|
SolverSpec
|
The solver specification. |
required |
protocol
|
str
|
Name of the protocol. |
required |
replicate
|
int
|
Replicate number. |
required |
Returns:
| Type | Description |
|---|---|
str
|
|
plan
¶
plan(campaign: Campaign, *, registry: SolverRegistry | None = None, families: FamilyRegistry | None = None) -> Plan
Plan a campaign without running anything.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
campaign
|
Campaign
|
The campaign. |
required |
registry
|
SolverRegistry | None
|
Solver registry. Defaults to the built-in solvers and plugins. |
None
|
families
|
FamilyRegistry | None
|
Benchmark-family registry for family and catalog sources. Defaults to the built-in families and plugins. |
None
|
Returns:
| Type | Description |
|---|---|
Plan
|
The verified plan. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If a source cannot be loaded, a solver id or parameter is unknown, problem ids clash, or no run remains. |
prepare
¶
prepare(campaign: Campaign, *, registry: SolverRegistry | None = None, families: FamilyRegistry | None = None) -> Prepared
Plan a campaign and keep the loaded problems and the registry for execution.
Takes the same arguments and raises the same errors as plan().
check_compatibility
¶
check_compatibility(campaign: Campaign, *, registry: SolverRegistry | None = None, families: FamilyRegistry | None = None) -> Compatibility
Load a campaign's problems and report which solvers each one excludes.
Unlike plan(), it reports rather than fails when every pair is
excluded.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
campaign
|
Campaign
|
The campaign. |
required |
registry
|
SolverRegistry | None
|
Solver registry. Defaults to the built-in solvers and plugins. |
None
|
families
|
FamilyRegistry | None
|
Benchmark-family registry. Defaults to the built-in families and plugins. |
None
|
Raises:
| Type | Description |
|---|---|
ConfigError
|
If a source cannot be loaded or a solver id is unknown. |
dispatchatlas.campaigns.compatibility
¶
Which solvers run on which problems, and what to keep in mind when they do.
A solver declares the problem features it models in
supports. A campaign never pairs
a solver with a problem that has a feature outside that set: the cell is
excluded and the plan records the exclusion with the missing features, so a
comparison that is absent is always explained.
Caveats mark pairings that run but need care when results are compared:
- a single-objective solver on a problem with several objectives optimises the primary objective only; the others describe its schedule;
- on problems with more than three objectives most solutions become mutually non-dominated, so a solver whose selection relies on Pareto dominance loses selection pressure (Deb and Jain 2014).
MANY_OBJECTIVES
module-attribute
¶
Objective count from which dominance-based selection is flagged (Deb and Jain 2014).
Exclusion
dataclass
¶
A problem and solver that a campaign does not pair.
Attributes:
| Name | Type | Description |
|---|---|---|
problem |
str
|
Problem id. |
solver |
str
|
Solver name. |
features |
tuple[str, ...]
|
Features of the problem the solver does not model, sorted. |
from_dict
classmethod
¶
from_dict(document: object) -> Exclusion
Build an exclusion from to_dict() output.
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the document is malformed. |
Caveat
dataclass
¶
A pairing that runs but needs care in the analysis.
Attributes:
| Name | Type | Description |
|---|---|---|
solver |
str
|
Solver name. |
problems |
tuple[str, ...]
|
Ids of the problems concerned. |
message |
str
|
What to keep in mind. |
reference |
str | None
|
Bibliography key of the source, if any. |
from_dict
classmethod
¶
from_dict(document: object) -> Caveat
Build a caveat from to_dict() output.
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the document is malformed. |
Compatibility
dataclass
¶
assess
¶
assess(problems: Sequence[Problem], solvers: Sequence[tuple[str, SolverInfo]]) -> Compatibility
Decide which problem and solver pairs run, and which need caveats.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
problems
|
Sequence[Problem]
|
The problems, with the objectives they will be solved for. |
required |
solvers
|
Sequence[tuple[str, SolverInfo]]
|
Solver names with the metadata of the registered solver. |
required |
Returns:
| Type | Description |
|---|---|
Compatibility
|
The exclusions and caveats. |
dispatchatlas.campaigns.fairness
¶
Fair comparison: the same budget, problems, and seeds for every solver, verified.
Solvers are compared fairly when, under each protocol, every one of them runs
with identical stop criteria on the same problems with the same seeds. The
planner builds runs that way; assess() checks it on the finished plan
instead of assuming it, so the claim can also be re-checked from a stored
plan.json:
- every protocol bounds the run with a time, evaluation, or iteration limit;
- every run refers to one of the plan's protocols, so all runs under a protocol share its stop criteria;
- every solver has its full set of replicates under every protocol on every
problem it is not excluded from (exclusions are recorded with their reason,
see
compatibility); - all runs of one problem and replicate share one seed, so results are paired across solvers and protocols.
A protocol's parities name the budgets it equalises, read off its stop
criteria: equal-wall-clock (time_limit), equal-evaluations
(max_evaluations), equal-iterations (max_iterations), and
fixed-target (target). Wall-clock budgets depend on the speed and load
of the machine, and runs they stop cannot be replayed; an iteration means a
different amount of work in different solvers.
The assessment also carries notes: facts that break none of these rules
but that a reader of a comparison needs (notes()). A note never refuses
a plan. Both concern protocols with a time limit:
- solvers whose
threadsparameters differ are noted, because equal wall-clock budgets then buy unequal processor time. Each solver counts with the value its plan entry records, which is the one its instance holds, the default included; a solver without such a parameter counts as one thread; - solvers declared deterministic that run once are noted when the campaign
has more replicates, because the result of a run that the clock ends
depends on the speed and load of the machine and one run does not show by
how much. The campaign key
replicate_timedgives them every replicate, and a solver whose entry states its own count is not noted. A stop criterion ends the run of an iterative solver, so such a solver is a note. A solver that is not iterative ends when its algorithm does: a constructive rule never reads the clock, and an exact solver can be ended by its time limit. The plan cannot tell the two apart, so such a solver is a conditional note (conditional_notes()), which holds only if the time limit ends one of its runs.
A plan says what should run. assess_outcomes() compares the records of
the completed runs with it, protocol by protocol: for which reasons the runs
of the iterative solvers ended, since runs of one protocol that ended for
different reasons did not spend the same budget, which solvers' runs took
more wall time than the protocol's time limit allows, and which of the
solvers of a conditional note had a run that the time limit ended.
PARITIES
module-attribute
¶
Every parity name, in report order.
Fairness
dataclass
¶
Fairness(solvers: tuple[str, ...], protocols: Mapping[str, tuple[str, ...]], paired: bool, issues: tuple[str, ...], notes: tuple[str, ...] = (), conditional_notes: tuple[str, ...] = ())
The fair-comparison assessment of a plan.
Attributes:
| Name | Type | Description |
|---|---|---|
solvers |
tuple[str, ...]
|
Names of the solvers compared. |
protocols |
Mapping[str, tuple[str, ...]]
|
Parities of each protocol, by protocol name. |
paired |
bool
|
Whether every problem and replicate has one seed. |
issues |
tuple[str, ...]
|
Every violation found. Empty for a fair plan. |
notes |
tuple[str, ...]
|
Facts that violate nothing but bear on a comparison. They leave a plan fair. |
conditional_notes |
tuple[str, ...]
|
Facts that bear on a comparison only if a run
ends a certain way, which the plan cannot tell
( |
Overrun
dataclass
¶
The completed runs of one solver against the time limit of their protocol.
Attributes:
| Name | Type | Description |
|---|---|---|
runs |
int
|
Completed runs of the solver under the protocol. |
over |
int
|
How many of them took more wall time than the time limit with
the tolerance allows, by the rule of
|
worst |
float
|
The longest wall time among those, as a multiple of the time limit. |
ProtocolOutcomes
dataclass
¶
ProtocolOutcomes(protocol: str, stop_reasons: Mapping[str, Mapping[str, int]], overruns: Mapping[str, Overrun], unreplicated: Mapping[str, Mapping[str, int]] = dict())
How the completed runs of one protocol ended.
Attributes:
| Name | Type | Description |
|---|---|---|
protocol |
str
|
Name of the protocol. |
stop_reasons |
Mapping[str, Mapping[str, int]]
|
For each iterative solver with a completed run, in plan order, how many of its runs ended for each stop reason. Solvers that are not iterative end when their algorithm does and are left out. |
overruns |
Mapping[str, Overrun]
|
For each solver with a completed run over the time limit, in plan order, the count and extent. Empty for a protocol without a time limit. |
unreplicated |
Mapping[str, Mapping[str, int]]
|
For each solver declared deterministic that is not iterative, runs once per problem under the protocol's time limit although the campaign has more replicates, and has a completed run that the time limit ended, in plan order, how many of its runs ended for each stop reason. These are the solvers of the plan's conditional note whose condition holds. |
divergent
property
¶
Whether the runs of the iterative solvers ended for more than one reason.
Runs that ended for different reasons, such as one at the time limit and one for lack of improvement, did not spend the same budget, in one solver or across several. A reached target or a proof of optimality is a legitimate end, so a divergence qualifies a comparison and does not refuse it.
Outcomes
dataclass
¶
Outcomes(protocols: tuple[ProtocolOutcomes, ...], overrun_tolerance: float, records: int)
The outcomes of a campaign's completed runs, compared with its plan.
Attributes:
| Name | Type | Description |
|---|---|---|
protocols |
tuple[ProtocolOutcomes, ...]
|
The outcomes under each protocol, in plan order. |
overrun_tolerance |
float
|
Share of a time limit by which a run may exceed
it without counting as over; the clock margin of
|
records |
int
|
How many records counted: those of runs of the plan. |
protocol
¶
protocol(name: str) -> ProtocolOutcomes
Return the outcomes under the protocol with the given name.
Raises:
| Type | Description |
|---|---|
KeyError
|
If the plan has no such protocol. |
parities
¶
parities(stop: StopCriteria) -> tuple[str, ...]
Return the parities that stop criteria realise, in PARITIES order.
assess
¶
notes
¶
Return what bears on a comparison under one protocol without breaking a rule.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
plan
|
Plan
|
The plan. |
required |
protocol
|
Protocol
|
One of its protocols. |
required |
solvers
|
Collection[str] | None
|
Names of the solvers compared. |
None
|
Returns:
| Type | Description |
|---|---|
str
|
One sentence, without its capital and full stop, for each fact: the |
...
|
compared solvers' thread counts differ under a time limit, and |
tuple[str, ...]
|
iterative solvers declared deterministic run once under a time limit |
tuple[str, ...]
|
although the campaign has more replicates. Empty for a protocol |
tuple[str, ...]
|
without a time limit. |
conditional_notes
¶
conditional_notes(plan: Plan, protocol: Protocol, solvers: Collection[str] | None = None) -> tuple[str, ...]
Return what bears on a comparison under one protocol only if a run ends a certain way.
A solver declared deterministic that is not iterative ends when its
algorithm does. A constructive rule never reads the clock, and an exact
solver can be ended by its time limit; the plan cannot tell the two
apart, so what it says about them holds only for a run that the time
limit ends. assess_outcomes() finds those runs among the records.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
plan
|
Plan
|
The plan. |
required |
protocol
|
Protocol
|
One of its protocols. |
required |
solvers
|
Collection[str] | None
|
Names of the solvers compared. |
None
|
Returns:
| Type | Description |
|---|---|
str
|
One sentence, without its capital and full stop: solvers declared |
...
|
deterministic that are not iterative run once under a time limit |
tuple[str, ...]
|
although the campaign has more replicates. Empty when there is none, |
tuple[str, ...]
|
and for a protocol without a time limit. |
assess_outcomes
¶
assess_outcomes(plan: Plan, records: Iterable[RunRecord], *, overrun_tolerance: float = OVERRUN_MARGIN) -> Outcomes
Compare the outcomes of a campaign's completed runs with its plan.
A plan can be fair and its runs still not comparable: under one protocol, a solver that stops for lack of improvement spends less than the budget another one uses to its time limit, and a solver that honours no deadline spends more. Neither shows in the plan.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
plan
|
Plan
|
The plan. |
required |
records
|
Iterable[RunRecord]
|
Records of completed runs. A record of a run that is not in the plan does not count. |
required |
overrun_tolerance
|
float
|
Share of the time limit by which the wall time of
a completed run may exceed it without counting as over. A run
counts as over by the rule of the engine's note on it,
|
OVERRUN_MARGIN
|
Returns:
| Type | Description |
|---|---|
Outcomes
|
For each protocol, the stop reasons of the iterative solvers' runs, |
Outcomes
|
the solvers whose runs took more wall time than the time limit |
Outcomes
|
allows, and the solvers declared deterministic that run once and |
Outcomes
|
are not iterative whose runs the time limit ended. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If |
dispatchatlas.campaigns.estimation
¶
Estimates: how many runs a campaign needs, and how long they can take.
estimate() plans a campaign without running anything and bounds the
duration of its solves from the protocols. The solve of a run whose protocol
sets time_limit takes at most its hard limit when the process backend
enforces one, and otherwise relies on the solver honouring the time limit.
Every attempt may take that long, and a run gets up to 1 + retries
attempts. The time a worker takes to begin a solve, which the start-up limit
bounds, is not counted. A run whose protocol bounds only evaluations or
iterations has no duration bound, so those runs are counted apart.
Estimate
dataclass
¶
Estimate(planned: int, completed: int, failed: int, remaining: int, problems: int, solvers: int, protocols: int, exclusions: int, by_protocol: Mapping[str, int], by_solver: Mapping[str, int], workers: int, timed: int, untimed: int, budget_seconds: float, worst_case_seconds: float)
The size of a campaign and bounds on its duration.
Attributes:
| Name | Type | Description |
|---|---|---|
planned |
int
|
Runs in the plan. |
completed |
int
|
Runs an existing store already completed. |
failed |
int
|
Runs that failures in an existing store settled. |
remaining |
int
|
Runs that |
problems |
int
|
Problems in the plan. |
solvers |
int
|
Solvers in the plan. |
protocols |
int
|
Protocols in the plan. |
exclusions |
int
|
Problem and solver pairs that do not run. |
by_protocol |
Mapping[str, int]
|
Remaining runs per protocol. |
by_solver |
Mapping[str, int]
|
Remaining runs per solver. |
workers |
int
|
Runs that execute at once. |
timed |
int
|
Remaining runs whose protocol sets a time limit. |
untimed |
int
|
Remaining runs without a time limit, whose duration has no bound. |
budget_seconds |
float
|
Sum of the time limits of the timed runs. |
worst_case_seconds |
float
|
Sum of the longest time the solve of every
attempt of every timed run may take: its hard limit when
enforced, else its time limit, times |
estimate
¶
estimate(campaign: Campaign, *, root: str | Path | None = None, registry: SolverRegistry | None = None, families: FamilyRegistry | None = None, backend: str | None = None, workers: int | str | None = None) -> Estimate
Plan a campaign and bound the time its remaining runs can take.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
campaign
|
Campaign
|
The campaign. |
required |
root
|
str | Path | None
|
Output root. When given, runs that the campaign's directory already completed, or whose failures settled them, are left out. |
None
|
registry
|
SolverRegistry | None
|
Solver registry. Defaults to the built-in solvers and plugins. |
None
|
families
|
FamilyRegistry | None
|
Benchmark-family registry. Defaults to the built-in families and plugins. |
None
|
backend
|
str | None
|
Override of the campaign's execution backend. |
None
|
workers
|
int | str | None
|
Override of the campaign's worker count, or |
None
|
Returns:
| Type | Description |
|---|---|
Estimate
|
The estimate. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the campaign cannot be planned. |
dispatchatlas.campaigns.runner
¶
Running a campaign: plan it, execute what remains, and record every attempt.
run() plans the campaign, opens its directory under the output root, and
executes every run that is neither complete nor settled by failures. Calling
it again after an interruption therefore resumes the campaign: records already
written are kept when they belong to the same configuration and still match
their result hash. A directory that holds results of another configuration is
refused unless fresh asks to discard them.
Every attempt leaves a record the moment it ends: a run record for a
completion, a failure record for a failure (see
records). Attempt numbers continue across
sessions. A failed run gets another attempt, at the end of the queue, until
its failures settle it: its latest failure is terminal, or it has failed
1 + retries times. Each session writes sessions/<n>.json with the
environment it runs in, when it starts and again when it ends.
A session that resumes a campaign compares its execution settings and its
environment with those of the last session the directory recorded
(session_changes()). It logs a warning that names each difference and
stores them in its record, and a campaign whose on_changed_code is
"refuse" refuses a session on other code instead. A file at the path of a
planned run that is not the valid record of that run, such as a record made
on another version of its problem, does not count, and its run is attempted
again unless on_changed_problem is "refuse". The file stays where it
is until the new record is written, and is then moved under superseded/;
a record kept there that is the valid record of a planned run again, because
its problem was restored, is moved back when a session starts.
Each session also writes logs/session-<n>.log, each line with the UTC
time of its record. The log holds what the engine itself logs while
run() works, from the level INFO, and what the other loggers of the
library let through, in this process and in its worker processes: WARNING
and higher unless the application lowered a level. The execution key
log_level lowers those loggers for the session, at the cost that a
solver which logs in its loop then pays inside its time limit. What the
handlers of the application show does not change either way.
Progress is logged through dispatchatlas.core.logs: a line when the
session starts, one at most every PROGRESS_INTERVAL seconds while it
runs, and a summary at the end.
PROGRESS_INTERVAL
module-attribute
¶
Seconds between progress lines while a session runs.
RunSummary
dataclass
¶
RunSummary(campaign: str, config_hash: str, directory: Path, session: int, planned: int, completed: int, failed: int, pending: int, attempted: int, succeeded: int, errors: int, elapsed: float)
What a session of run() did, and where the campaign stands.
Attributes:
| Name | Type | Description |
|---|---|---|
campaign |
str
|
Campaign id. |
config_hash |
str
|
Config hash of the campaign. |
directory |
Path
|
The campaign directory. |
session |
int
|
Number of the session. |
planned |
int
|
Runs in the plan. |
completed |
int
|
Runs with a record, from every session. |
failed |
int
|
Runs settled by failures, from every session. |
pending |
int
|
Runs that still need an attempt. |
attempted |
int
|
Attempts this session made. |
succeeded |
int
|
Attempts this session that completed a run. |
errors |
int
|
Attempts this session that failed. |
elapsed |
float
|
Wall-clock seconds of the session. |
run
¶
run(campaign: Campaign, root: str | Path, *, registry: SolverRegistry | None = None, families: FamilyRegistry | None = None, fresh: bool = False, retry_failed: bool = False, max_runs: int | None = None, backend: str | None = None, workers: int | str | None = None, affinity: str | Sequence[int] | None = None) -> RunSummary
Run a campaign, resuming from the results its directory already holds.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
campaign
|
Campaign
|
The campaign. |
required |
root
|
str | Path
|
Output root; the results go to |
required |
registry
|
SolverRegistry | None
|
Solver registry. Defaults to the built-in solvers and plugins. |
None
|
families
|
FamilyRegistry | None
|
Benchmark-family registry. Defaults to the built-in families and plugins. |
None
|
fresh
|
bool
|
Discard every result in the campaign directory first. |
False
|
retry_failed
|
bool
|
Give every run that failures settled one more attempt. |
False
|
max_runs
|
int | None
|
Attempt at most this many pending runs in this session. |
None
|
backend
|
str | None
|
Override of the campaign's execution backend. |
None
|
workers
|
int | str | None
|
Override of the campaign's worker count, or |
None
|
affinity
|
str | Sequence[int] | None
|
Override of the campaign's |
None
|
Returns:
| Type | Description |
|---|---|
RunSummary
|
The summary of the session. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the campaign cannot be planned, its directory holds
results of another configuration, another runner is using it, or
runs are pending and its workers cannot be pinned as |
KeyboardInterrupt
|
When interrupted. Records written so far are kept and the session is marked interrupted. |
session_changes
¶
session_changes(previous: SessionRecord, execution: Execution, workers: int, environment: Mapping[str, Any]) -> tuple[dict[str, Any], ...]
Return how a session differs from an earlier session of its campaign directory.
The execution settings are compared one by one, except those that only
say what a session does about a difference, and the resolved worker
count is compared when both sessions state the same workers. The
environments are compared value by value. A value the earlier record
does not state is not compared; a package, a plugin, or a git checkout
that only one session has is a difference. The load of the machine is no
part of the environment and is not compared.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
previous
|
SessionRecord
|
The record of the earlier session. |
required |
execution
|
Execution
|
The execution settings of the session. |
required |
workers
|
int
|
Its number of workers, with |
required |
environment
|
Mapping[str, Any]
|
Its environment, as
|
required |
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
One document per difference, with |
...
|
|
tuple[dict[str, Any], ...]
|
which is |
tuple[dict[str, Any], ...]
|
packages, the library, the plugins, and the git states, or |
tuple[dict[str, Any], ...]
|
|
tuple[dict[str, Any], ...]
|
|
tasks_for
¶
tasks_for(prepared: Prepared, runs: Sequence[PlannedRun], execution: Execution, attempt: Callable[[str], int]) -> list[Task]
Build the tasks that execute runs of a prepared plan.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
prepared
|
Prepared
|
The prepared plan. |
required |
runs
|
Sequence[PlannedRun]
|
Runs of its plan. |
required |
execution
|
Execution
|
Execution settings, for hard limits and schedule recording. |
required |
attempt
|
Callable[[str], int]
|
The attempt number of a run, given its id. |
required |
Returns:
| Type | Description |
|---|---|
list[Task]
|
One task per run, each with a fresh solver instance. |
overridden
¶
overridden(base: Execution, backend: str | None, workers: int | str | None, affinity: str | Sequence[int] | None = None) -> Execution
Return execution settings with a caller's overrides applied.
Choosing the sequential backend, which has no worker processes, sets what
only worker processes have to its default: affinity to "inherit",
threads_per_worker to 1, and, without a worker count, workers
to 1.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
base
|
Execution
|
The campaign's settings. |
required |
backend
|
str | None
|
Another backend, or |
required |
workers
|
int | str | None
|
Another worker count or |
required |
affinity
|
str | Sequence[int] | None
|
Another |
None
|
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the resulting settings are invalid. |
dispatchatlas.campaigns.backends
¶
Execution backends: where the runs of a campaign execute.
Both backends take Task items from
a queue that the runner may extend while results arrive (a retry goes to its
end), and yield each Completion
or Failure as soon as it exists.
SequentialBackendruns one attempt after another in the calling process. Solvers must honour their own time limits, and a crash in native code ends the campaign. Nothing has to be picklable, which suits debugging and solvers defined interactively.ProcessBackendruns attempts in worker processes started with thespawnmethod, so every platform behaves alike, and supervised by the calling process. Each worker runs one attempt at a time and keeps the problem of its last attempt, so a problem is sent to a worker only when the worker's previous attempt was on another one; runs are planned problem by problem, which makes that rare. The supervisor enforces two limits (seetimeout). From the hand-over of a run its worker has the start-up limit to report that the solve begins; from the instant that report states, the run has its hard limit. The supervisor kills a worker that passes either, with every process the worker started, records a recoverable start-up failure or a terminal timeout, and hands the next run to a fresh worker. A hand-over never waits for a worker: a worker that is starting gets its run when it reports that it is ready, so the supervisor keeps reading reports and enforcing the limits of the other runs meanwhile. While a run with a hard limit executes, its worker forwards the incumbents its search reports, at most one everyPROGRESS_INTERVALseconds, and a timeout failure holds the last one and the last one found within the run's time limit. A worker that dies, from a crash in native code or the out-of-memory killer, costs only the attempt it was running, which is recorded as a recoverable failure. Every worker starts with the thread pools of numerical libraries limited: the variables ofTHREAD_VARIABLESare in the environment the worker process is created with, so no import in the worker comes before them. A worker can also be pinned to one processor: the supervisor creates it while confined to that processor itself, so the worker and every thread it starts run there, and the worker reports its processors when it is ready, which the supervisor checks. A backend that is given the digest of the library source its session started on has every worker report the digest of the source it starts on: a worker on other source is warned about, and every run it completes carries a note. A backend that is given a sink for log records has every worker send the records ofWORKER_LOG_LEVELor higher that the library's loggers let through in it, which a fresh worker does fromWARNING; given a log level as well, the backend lowers the library's loggers of every worker to it, and what the handlers of a worker show stays as it was. The thread that logs never waits for the supervisor: a record goes to a bounded queue that another thread of the worker sends in batches, a record the queue has no room for is dropped and counted, and the supervisor adds the count to the log. The supervisor spends at mostLOG_READ_SECONDSon the log records of one worker before it turns to the limits and the other workers, and log records do not postpone a kill.
Every result states how many runs were executing when its attempt was handed
over (concurrent), the processor its worker is pinned to (cpu), and
how long the attempt waited for its solve to begin (waited_seconds), from
its hand-over, before a fresh worker exists, to the instant its worker states.
STARTUP_ATTEMPTS
module-attribute
¶
Consecutive worker processes that may exit before they start until the backend gives up.
PROGRESS_INTERVAL
module-attribute
¶
Fewest seconds between two incumbents a worker forwards to the supervisor.
WORKER_LOG_LEVEL
module-attribute
¶
Lowest level of the log records a worker sends, unless the backend is given a lower one.
LOG_READ_SECONDS
module-attribute
¶
Longest the supervisor handles the log records of one worker in one reading of its pipe.
LogSink
¶
Receives a record a worker logged: the worker's name, the time of the record in seconds since the epoch, its level, the name of its logger, and its message.
Backend
¶
SequentialBackend
¶
ProcessBackend
¶
ProcessBackend(workers: int, *, cpus: Sequence[int] | None = None, threads: int = 1, startup_limit: float = STARTUP_SECONDS, source: str | None = None, log: LogSink | None = None, log_level: int | None = None)
Runs attempts in supervised worker processes.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
workers
|
int
|
Number of worker processes. |
required |
cpus
|
Sequence[int] | None
|
The processor each worker is pinned to, in worker order, or
|
None
|
threads
|
int
|
Threads the numerical libraries of a worker may use. |
1
|
startup_limit
|
float
|
Seconds a worker has, from the hand-over of a run, to report that the solve begins. |
STARTUP_SECONDS
|
source
|
str | None
|
The digest of the library source the session started on
( |
None
|
log
|
LogSink | None
|
Receives the records the workers log. |
None
|
log_level
|
int | None
|
The level the library's loggers of every worker are
lowered to, so that |
None
|
Raises:
| Type | Description |
|---|---|
ConfigError
|
If there is no worker, a worker has no thread, the start-up limit is not a positive, finite number, or there are fewer processors to pin to than workers. |
run
¶
Execute tasks from queue until it is empty, yielding each result.
Closing the iterator early, or an exception in the caller, kills the workers.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If worker processes keep exiting before they start,
for example because a script starts a campaign without an
|
backend_for
¶
backend_for(name: str, workers: int, *, cpus: Sequence[int] | None = None, threads: int = 1, startup_limit: float = STARTUP_SECONDS, source: str | None = None, log: LogSink | None = None, log_level: int | None = None) -> Backend
Return the backend of the given name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
|
required |
workers
|
int
|
Attempts that run at once. |
required |
cpus
|
Sequence[int] | None
|
The processor each worker process is pinned to, if any. |
None
|
threads
|
int
|
Threads the numerical libraries of a worker process may use. |
1
|
startup_limit
|
float
|
Seconds a worker process has, from the hand-over of a run, to report that the solve begins. |
STARTUP_SECONDS
|
source
|
str | None
|
The digest of the library source the session started on, for the worker processes to compare theirs with. |
None
|
log
|
LogSink | None
|
Receives the records the worker processes log. |
None
|
log_level
|
int | None
|
The level the library's loggers of every worker process
are lowered to, or |
None
|
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the name is unknown, or the sequential backend gets more than one worker or processors to pin to. |
dispatchatlas.campaigns.execution
¶
One attempt of one run: solve, check, measure, and describe the outcome.
attempt() is what every backend executes, in the calling process or in
a worker process. It calls the solver, then checks the result without trusting
it: check_schedule() decides whether the
schedule is feasible, and evaluate()
recomputes every objective from the schedule. It returns plain data, a
Completion or a Failure, which crosses process boundaries.
Measurements cover the solve alone: wall time with time.perf_counter();
CPU time of the executing process with time.process_time(), which
includes any threads the solver starts, plus, on POSIX systems, the CPU time
of the child processes the solver started and waited for, from
os.times(); and, on Linux, the peak resident memory of the process
during the solve, read from /proc/self/status after resetting it through
/proc/self/clear_refs. On Windows the CPU time of a child process, such
as a backend that runs as a program of its own, is not in the measurement.
The instant the wall time counts from is also the origin of the solver's
clock: attempt() calls the solver inside
clock_origin(), so the time limit and the
elapsed time of every solver cover everything the solver does from the start
of the attempt, including what it does before its search starts, such as
building an evaluator or a model. What a solver does once in a process, such
as importing a backend, is done before that instant by warm_up(), so
that it shortens the search of no run.
observe() adds what the time and the trace of the attempt show: a wall
time beyond the time limit, an elapsed time the engine's clock contradicts,
and a trace that cannot be drawn against time or evaluations or that ends at
another value than the outcome. These notes and those of describe()
are returned with the completion, logged, and stored in the run record.
classify() sorts errors into the categories of
Category.
OVERRUN_MARGIN
module-attribute
¶
Share of its time limit by which the wall time of a run may pass the limit unnoted.
CLOCK_MARGIN_SECONDS
module-attribute
¶
Seconds two times of a run may differ by unnoted, whatever their size.
The scheduler of the machine and the resolution of its clocks move the end of a run by a few hundredths of a second. The margin applies to the wall time against the time limit, and to the solver's elapsed time against the wall time.
CLOCK_MARGIN_SHARE
module-attribute
¶
Share of the engine's wall time by which a solver's elapsed time may differ from it unnoted.
Also the share of the time limit by which a run that stops for its time limit may end before the limit unnoted.
OBSERVATION_KINDS
module-attribute
¶
OBSERVATION_KINDS: tuple[str, ...] = ('overrun', 'early-stop', 'elapsed', 'trace-time', 'trace-evaluations', 'trace-value', 'no-trace')
The kinds of fact observations() finds in a completed attempt, in its order.
Task
dataclass
¶
Task(run_id: str, attempt: int, problem: Problem, digest: str, solver: Solver, stop: StopCriteria, seed: int, hard_limit: float | None = None, record_schedule: bool = True)
One attempt of one run, ready to execute.
Attributes:
| Name | Type | Description |
|---|---|---|
run_id |
str
|
Run id. |
attempt |
int
|
Attempt number, from 1. |
problem |
Problem
|
The problem, as solved. |
digest |
str
|
Digest of the problem. |
solver |
Solver
|
A solver instance built with the campaign's parameters. |
stop |
StopCriteria
|
Stop criteria of the run's protocol. |
seed |
int
|
Seed of the run. |
hard_limit |
float | None
|
Seconds after the start of the solve at which the process
backend kills the run, or |
record_schedule |
bool
|
Whether to return the schedule for the record. |
Measurement
dataclass
¶
Measurement(started: str, finished: str, wall_seconds: float, cpu_seconds: float | None = None, peak_rss_bytes: int | None = None, worker: str = 'main', hard_limit: float | None = None, cpu: int | None = None, concurrent: int | None = None, waited_seconds: float | None = None, shared_clock: bool | None = None)
When, where, and at what cost an attempt executed.
Attributes:
| Name | Type | Description |
|---|---|---|
started |
str
|
UTC start time in ISO 8601. |
finished |
str
|
UTC end time in ISO 8601. |
wall_seconds |
float
|
Wall-clock seconds. |
cpu_seconds |
float | None
|
CPU seconds of the executing process, if measured, and on POSIX systems also of the child processes it waited for during the attempt. |
peak_rss_bytes |
int | None
|
Peak resident memory of the executing process, if measured. |
worker |
str
|
Where the attempt executed: |
hard_limit |
float | None
|
The hard limit armed for the attempt, if any. |
cpu |
int | None
|
The processor the executing worker is pinned to, if any. |
concurrent |
int | None
|
Runs executing when the attempt was handed to where it
executes, itself included. |
waited_seconds |
float | None
|
Seconds from the hand-over of the attempt to its
worker's slot until the instant the worker states as the start
of the solve. |
shared_clock |
bool | None
|
|
Completion
dataclass
¶
Completion(run_id: str, attempt: int, outcome: Outcome, result_hash: str, schedule: Schedule | None, measurement: Measurement, notes: tuple[str, ...] = ())
An attempt that returned a result.
Attributes:
| Name | Type | Description |
|---|---|---|
run_id |
str
|
Run id. |
attempt |
int
|
Attempt number. |
outcome |
Outcome
|
The checked outcome. |
result_hash |
str
|
|
schedule |
Schedule | None
|
The schedule, when the campaign records schedules. |
measurement |
Measurement
|
How the attempt executed. |
notes |
tuple[str, ...]
|
What the engine notices about the attempt: the claims of the
solver that the checks contradict ( |
Failure
dataclass
¶
Failure(run_id: str, attempt: int, category: Category, error: RunError, measurement: Measurement, solve_started: bool | None = None, last_incumbent: Incumbent | None = None, limit_incumbent: Incumbent | None = None)
An attempt that ended without a result.
Attributes:
| Name | Type | Description |
|---|---|---|
run_id |
str
|
Run id. |
attempt |
int
|
Attempt number. |
category |
Category
|
Whether another attempt may succeed. |
error |
RunError
|
What ended the attempt. |
measurement |
Measurement
|
How the attempt executed. |
solve_started |
bool | None
|
Whether the solve had begun when the attempt ended. |
last_incumbent |
Incumbent | None
|
The last candidate the search of a run killed at its hard limit had reported, if any. |
limit_incumbent |
Incumbent | None
|
The last candidate that search had reported within the time limit of the run, if any. |
utc_now
¶
Return the current UTC time in ISO 8601 with microseconds and a Z suffix.
classify
¶
classify(error: BaseException) -> Category
Return whether an attempt that raised error may succeed when repeated.
Typed DispatchAtlas errors are terminal: they state a fact about the configuration, the problem, or the installation that another attempt would meet again. Any other exception is recoverable.
warm_up
¶
warm_up(solver: Solver) -> None
Do what a solver does once in a process before it can solve.
The first call of a solver in a process may pay a cost that no later
call pays: the import of a backend, or of a module that loads on first
use. Paid inside a run, that cost would shorten the search of the first
run of every process and of no other. This function pays it outside any
run: it imports numpy.random, which NumPy loads on first use, and
calls the warm_up method of the solver when the solver defines one.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
solver
|
Solver
|
The solver about to be called. |
required |
warm_up_task
¶
Warm up the solver of an attempt, before the clock of the attempt starts.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task
|
Task
|
The attempt. |
required |
worker
|
str
|
Where it executes, for the measurement. |
'main'
|
Returns:
| Type | Description |
|---|---|
Failure | None
|
|
attempt
¶
Execute one attempt of a run.
The solver is warmed up first (warm_up_task()), outside the time of
the attempt. It is then called inside
clock_origin(), whose origin is the
instant the wall time of the attempt counts from, so the solver's
elapsed time and time limit count from the same instant. A completion
carries the notes of describe() and observe().
Exceptions raised by the solver, by its warm-up, or by the checks of
its result become a Failure. KeyboardInterrupt and
SystemExit propagate.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task
|
Task
|
The attempt. |
required |
worker
|
str
|
Where it executes, for the measurement. |
'main'
|
origin
|
float | None
|
The instant the attempt counts from, as
|
None
|
Returns:
| Type | Description |
|---|---|
Result
|
The completion or the failure. |
describe
¶
describe(problem: Problem, result: SolveResult) -> tuple[Outcome, Schedule | None, tuple[str, ...]]
Check a solver's result and describe it as an outcome.
Feasibility and objective values come from the schedule, not from the
solver: the violation codes of
check_schedule(), and every problem
objective recomputed by evaluate().
Only a malformed schedule, one that misses, repeats, or invents tasks or
uses a mode a task lacks, keeps the solver's values, since none can be
computed from it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
problem
|
Problem
|
The problem solved. |
required |
result
|
SolveResult
|
The solver's result. |
required |
Returns:
| Type | Description |
|---|---|
Outcome
|
The outcome, the schedule with normalised numbers (or |
Schedule | None
|
notes on every claim of the solver the checks contradict. |
Raises:
| Type | Description |
|---|---|
SolverError
|
If the elapsed time, or a time or a value of the trace, is not a finite number. Such a result can be neither compared with the clock of the engine nor stored in a record. |
overran
¶
overran(wall_seconds: float, limit: float, margin: float = OVERRUN_MARGIN) -> bool
Return whether a wall time passes a time limit by more than the margins allow.
It does when it exceeds the limit by more than margin of the limit
and by more than CLOCK_MARGIN_SECONDS. The engine's note on a run
and the assessment of a campaign's outcomes
(assess_outcomes()) both count an
overrun by this rule.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
wall_seconds
|
float
|
Wall-clock seconds of a solve, on the engine's clock. |
required |
limit
|
float
|
The time limit of the run, in seconds. |
required |
margin
|
float
|
Share of the limit by which the wall time may pass it. |
OVERRUN_MARGIN
|
observations
¶
observations(outcome: Outcome, stop: StopCriteria, wall_seconds: float, *, iterative: bool) -> tuple[tuple[str, str], ...]
Return what the time and the trace of a completed attempt show, each with its kind.
One fact is returned for each of these, under the kind named first:
overrun: the wall time the engine measured passes the time limit by more thanOVERRUN_MARGINof the limit and by more thanCLOCK_MARGIN_SECONDS;early-stop: the solver stopped for its time limit while that wall time is below the limit by more thanCLOCK_MARGIN_SECONDSand by more thanCLOCK_MARGIN_SHAREof the limit, as when a solver gives a search a share of its limit that counts from the start of the attempt;elapsed: the elapsed time the solver reports differs from that wall time by more thanCLOCK_MARGIN_SECONDSand by more thanCLOCK_MARGIN_SHAREof it;trace-time: the trace holds a negative time, times that decrease, or a time beyond the elapsed time the solver reports, one fact for each of the three;trace-evaluations: the evaluation counts of the trace decrease;trace-value: the last value of the trace differs from the value of the outcome;no-trace: an iterative solver returned a schedule and no trace.
The facts follow from the outcome, the stop criteria, and the wall time alone, so they can be derived again from a run record.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
outcome
|
Outcome
|
The checked outcome of the attempt. |
required |
stop
|
StopCriteria
|
The stop criteria the solver received. |
required |
wall_seconds
|
float
|
Wall-clock seconds of the solve, on the engine's clock. |
required |
iterative
|
bool
|
Whether the solver runs until a stop criterion fires. |
required |
Returns:
| Type | Description |
|---|---|
tuple[str, str]
|
Pairs of a kind, one of |
...
|
words; empty when nothing is found. |
observe
¶
observe(outcome: Outcome, stop: StopCriteria, wall_seconds: float, *, iterative: bool) -> tuple[str, ...]
Return what the time and the trace of a completed attempt show, as notes.
The notes are the facts of observations(), in words and in its
order.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
outcome
|
Outcome
|
The checked outcome of the attempt. |
required |
stop
|
StopCriteria
|
The stop criteria the solver received. |
required |
wall_seconds
|
float
|
Wall-clock seconds of the solve, on the engine's clock. |
required |
iterative
|
bool
|
Whether the solver runs until a stop criterion fires. |
required |
Returns:
| Type | Description |
|---|---|
tuple[str, ...]
|
The notes, empty when nothing is found. |
engine_failure
¶
engine_failure(run_id: str, attempt_number: int, kind: ErrorKind, message: str, measurement: Measurement, *, solve_started: bool | None = None, last_incumbent: Incumbent | None = None, limit_incumbent: Incumbent | None = None) -> Failure
Return the failure of an attempt that the engine, not the solver, ended.
Timeouts and failed transfers are terminal; a worker that dies, and one that does not begin the solve within the start-up limit, are recoverable.
exception_error
¶
Describe an exception: its qualified class, message, and traceback.
Classes of the builtins module keep their plain name. A traceback
longer than 10,000 characters keeps its end, where the error is raised.
dispatchatlas.campaigns.timeout
¶
Hard time limits: stopping a run that ignores its own.
Stop criteria are cooperative: a solver checks them itself, and one that
does not, or that is stuck inside a native extension, keeps running. The
process backend therefore gives every run whose protocol sets time_limit
a hard limit of timeout_factor * time_limit + grace seconds, counted from
the instant its worker states as the start of the solve. When the limit
passes, the supervising process kills the worker and records a terminal
timeout failure: the same instance, seed, and budget would overrun again. The
failure holds the last incumbent the run's search had reported, and, apart
from it, the last one the search had found within the time limit. A fresh
worker takes the next run. Killing a process is the one interruption that
reaches code inside a native extension.
Until its solve begins, a run is under the start-up limit instead:
startup_limit seconds from its hand-over, in which the worker starts its
interpreter and loads plugins, if it started for this run, and rebuilds the
solver and the problem. The supervising process waits for none of that: a
fresh worker gets its run when it reports that it is ready. A worker that
does not begin in time is killed too, and the attempt is a recoverable
start-up failure, not a timeout.
A kill reaches the processes a solver started. On POSIX systems each worker
leads its own process group (detach()), and the kill reaches the whole
group. On Windows the supervising process puts each worker in a job object
of its own (contain()), which every process the worker starts joins,
and the kill ends the job. Where a job object cannot be created or the
worker cannot join it, a warning says so and the kill reaches the worker
alone.
A worker does not outlive its supervising process either
(end_with_parent()): it watches that process and ends itself, with the
process group it leads, when the process is gone. On Windows the job object
also ends with the supervising process, which holds its only handle.
hard_limit
¶
hard_limit(stop: StopCriteria, execution: Execution) -> float | None
Return the hard limit of a run in seconds.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
stop
|
StopCriteria
|
Stop criteria of the run's protocol. |
required |
execution
|
Execution
|
The campaign's execution settings. |
required |
Returns:
| Type | Description |
|---|---|
float | None
|
|
float | None
|
protocol sets no time limit, hard limits are disabled, or the backend |
float | None
|
is sequential and cannot enforce them. |
detach
¶
Make the calling process the leader of a new session and process group.
Called first thing in every worker, so that kill() reaches the
processes a solver starts. Does nothing where sessions do not exist.
end_with_parent
¶
End the calling worker process when its supervising process is gone.
A worker whose supervisor was killed would otherwise compute until its
run returns and it meets the closed pipe, beside the workers of a resumed
session. A daemon thread waits for the process that started this one
through multiprocessing and then ends this process at once. On POSIX
systems it ends the process group with it, so that processes a solver
started end too, but only a group this process leads: where
detach() could not start one, the group is the supervising
process's, and holds whatever launched the campaign. The thread needs
the interpreter to act: a worker inside a native extension that keeps
the interpreter lock ends when the extension returns. Does nothing in a
process that multiprocessing did not start.
contain
¶
Put a started worker process in a Windows job object of its own.
The job ends every process in it when it is terminated or when its last handle closes, and every process the worker starts joins it. The calling process holds the only handle, so the worker and what it started also end when the calling process dies.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
process
|
BaseProcess
|
A started worker. |
required |
Returns:
| Type | Description |
|---|---|
int | None
|
|
int | None
|
on other systems, and where the job cannot be created or the worker |
int | None
|
cannot join it, for example on a Windows version without nested jobs |
int | None
|
when the calling process is in a job itself. A warning reports that |
int | None
|
case, and |
kill
¶
Kill a worker process and the processes it started, and reap it.
The processes it started are reached through the worker's process group on POSIX systems and through its job object on Windows.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
process
|
BaseProcess
|
The worker. |
required |
job
|
int | None
|
The worker's job object, as |
None
|
wait
|
float
|
Seconds to wait for the process to be reaped. |
5.0
|
release
¶
Close the handle of a worker's job object, if it has one.
Processes still in the job end with it, so nothing a solver started outlives the worker's slot on Windows.
dispatchatlas.campaigns.records
¶
The records a campaign writes: completed runs, failed attempts, and sessions.
Each record is a JSON document whose schema field names its type and
version:
dispatchatlas.run/1(RunRecord): one completed run. It names the run (campaign, config hash, problem, solver, protocol, replicate, seed), holds itsOutcome, theresult_hashof that outcome, theidentity_hashof the fields that name the run, the best schedule as[task, mode, start, end]rows unless the campaign disables it, and theProvenanceof the attempt that produced it.dispatchatlas.failure/1(FailureRecord): one failed attempt, with its error andCategory, and, for a run killed at its hard limit, the lastIncumbentits search had reported.dispatchatlas.session/1(SessionRecord): one invocation of the runner, with its execution settings, the environment it ran in, the load of the machine when it began and ended, how its settings and environment differ from those of the session before it, the digest of the plan it executed, and how that plan differs from the one the directory held (PlanChange).
The result hash is the SHA-256 digest of the canonical JSON of
Outcome.hashed(): every outcome field except those that measure wall
time (elapsed and the elapsed column of the trace) and extras, whose
solver-specific facts carry no determinism guarantee. Nothing about the
machine, the session, or the attempt enters it, so a replay on any machine
compares results directly (see replaying). The
schedule enters it through its digest, so a record with or without stored
schedule rows hashes alike.
The identity hash is the SHA-256 digest of the canonical JSON of
RunRecord.identity(): the digest of the problem, the id and the
parameters of the solver, the protocol, the replicate, and the seed. A record
whose outcome or identity was edited after it was written no longer matches
its hashes, which RunRecord.verify() reports.
These classes convert records to and from documents. Where the files live is
the business of store.
TRACE_COLUMNS
module-attribute
¶
Column order of the rows of a recorded trace.
Category
¶
Bases: StrEnum
Whether a failed attempt may succeed when tried again.
Typed DispatchAtlas errors (an invalid configuration, a feature the solver does not model, a missing optional dependency) and hard timeouts are terminal: the instance, seed, and budget are fixed, so another attempt would fail the same way. Any other exception, a worker process that dies, and a worker that does not begin the solve within the start-up limit are recoverable: memory pressure, an I/O error, or a busy machine may not recur.
ErrorKind
¶
Bases: StrEnum
What ended a failed attempt.
EXCEPTION
class-attribute
instance-attribute
¶
The solver, or the checks of its result, raised an exception.
TIMEOUT
class-attribute
instance-attribute
¶
The solve passed its hard time limit and its worker was killed.
STARTUP
class-attribute
instance-attribute
¶
The worker did not begin the solve within the start-up limit and was killed.
WORKER_EXIT
class-attribute
instance-attribute
¶
The worker process died during the run.
TRANSFER
class-attribute
instance-attribute
¶
The run could not be sent to, or loaded in, a worker process.
SessionStatus
¶
Bases: StrEnum
How a session of the runner ended.
Outcome
dataclass
¶
Outcome(status: Status, stop_reason: StopReason, objective: str, value: float | None, objectives: Mapping[str, float], bound: float | None, gap: float | None, feasible: bool, violations: tuple[str, ...], evaluations: int, iterations: int, elapsed: float, trace: tuple[TracePoint, ...] = (), front: tuple[tuple[float, ...], ...] = (), schedule_digest: str | None = None, extras: Mapping[str, Scalar] = dict())
What a run found, as recorded.
Attributes:
| Name | Type | Description |
|---|---|---|
status |
Status
|
What the solver states about its schedule. |
stop_reason |
StopReason
|
Why the solver stopped. |
objective |
str
|
Name of the primary objective. |
value |
float | None
|
The primary objective's value, or |
objectives |
Mapping[str, float]
|
Every problem objective's value, computed from the
schedule by |
bound |
float | None
|
Proven bound on the primary objective, for exact solvers. |
gap |
float | None
|
Relative distance between value and bound, as
|
feasible |
bool
|
Whether the schedule breaks no constraint, as checked by
|
violations |
tuple[str, ...]
|
Codes of the constraints the schedule breaks, sorted. |
evaluations |
int
|
Schedules the solver evaluated. |
iterations |
int
|
Main-loop iterations of the solver. |
elapsed |
float
|
Wall-clock seconds the solver reports. |
trace |
tuple[TracePoint, ...]
|
Improvements of the best solution during the run. |
front |
tuple[tuple[float, ...], ...]
|
Objective vectors of the non-dominated solutions of a multi-objective solver, in the problem's objective order. |
schedule_digest |
str | None
|
|
extras |
Mapping[str, Scalar]
|
Solver-specific facts. |
replayable
property
¶
Whether the stop reason leaves the run reproducible by a replay.
A run stopped by its wall-clock limit is not: how far it got depends
on the speed and load of the machine. The protocol decides too, see
RunRecord.replayable.
hashed
¶
Return the fields the result hash covers: all but wall times and extras.
result_hash
¶
Return the SHA-256 digest of the canonical JSON of hashed().
Raises:
| Type | Description |
|---|---|
SerializationError
|
If a value is not finite. |
from_dict
classmethod
¶
from_dict(document: object) -> Outcome
Build an outcome from to_dict() output.
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the document is malformed. |
ProblemRef
dataclass
¶
The problem a run solved.
Attributes:
| Name | Type | Description |
|---|---|---|
id |
str
|
Problem id. |
family |
str | None
|
Benchmark family of the problem, if known. |
digest |
str
|
|
from_dict
classmethod
¶
from_dict(document: object) -> ProblemRef
Build a reference from to_dict() output.
SolverEntry
dataclass
¶
SolverEntry(name: str, id: str, label: str | None, parameters: Mapping[str, Any], family: str, stochastic: bool, iterative: bool, multi_objective: bool, threads: int | None = None, replicates: int | None = None)
A solver of a campaign, with the facts its metadata declares.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
Label, or id when the campaign gives no label. |
id |
str
|
Registered solver id. |
label |
str | None
|
Campaign label, if any. |
parameters |
Mapping[str, Any]
|
Parameter values set by the campaign. |
family |
str
|
Solver family. |
stochastic |
bool
|
Whether its results depend on the seed. |
iterative |
bool
|
Whether it runs until a stop criterion fires. |
multi_objective |
bool
|
Whether it optimises every objective at once. |
threads |
int | None
|
The value of its |
replicates |
int | None
|
The replicate count the campaign gives this solver
alone. |
from_dict
classmethod
¶
from_dict(document: object) -> SolverEntry
Build an entry from to_dict() output.
Provenance
dataclass
¶
Provenance(attempt: int, session: int, started: str, finished: str, wall_seconds: float, cpu_seconds: float | None = None, peak_rss_bytes: int | None = None, worker: str = 'main', hard_limit: float | None = None, cpu: int | None = None, concurrent: int | None = None, waited_seconds: float | None = None, shared_clock: bool | None = None)
When, where, and at what cost one attempt of a run executed.
Attributes:
| Name | Type | Description |
|---|---|---|
attempt |
int
|
Attempt number, counted from 1 across sessions. |
session |
int
|
Number of the runner session that executed it. |
started |
str
|
UTC start time in ISO 8601. |
finished |
str
|
UTC end time in ISO 8601. |
wall_seconds |
float
|
Wall-clock seconds of the solve. For an attempt the engine ended, the seconds from the start of the solve, or from the hand-over when the solve never began, until the engine ended it. |
cpu_seconds |
float | None
|
CPU seconds the executing process spent on the solve,
including any threads the solver started and, on POSIX systems,
the child processes the solver started and waited for. On
Windows the time of child processes is not included. |
peak_rss_bytes |
int | None
|
Peak resident memory of the executing process during the solve, where the platform can reset and read it (Linux). It includes the interpreter and the loaded libraries. |
worker |
str
|
|
hard_limit |
float | None
|
Seconds after the start of its solve at which the run
would have been killed, or |
cpu |
int | None
|
Index of the processor the worker was pinned to, or |
concurrent |
int | None
|
Runs executing in the session when this one was handed
to its worker: itself, the runs already executing, and the runs
handed over with it. |
waited_seconds |
float | None
|
Seconds from the hand-over of the attempt to its
worker's slot until the instant the worker states as the start
of the solve. They hold the start of the worker's interpreter and the
loading of plugins, for a worker that started for this attempt,
the rebuilding of the solver and the problem, and the
warm-up of the solver. |
shared_clock |
bool | None
|
Whether the clocks of the solver started at the
instant |
from_dict
classmethod
¶
from_dict(document: object) -> Provenance
Build provenance from to_dict() output.
The fields cpu, concurrent, waited_seconds, and
shared_clock may be absent, and are then None.
RunRecord
dataclass
¶
RunRecord(run_id: str, campaign: str, config_hash: str, problem: ProblemRef, solver: SolverEntry, protocol: Protocol, replicate: int, seed: int, outcome: Outcome, result_hash: str, schedule: Schedule | None, provenance: Provenance, notes: tuple[str, ...] = (), identity_hash: str | None = None)
One completed run.
Attributes:
| Name | Type | Description |
|---|---|---|
run_id |
str
|
Run id from the plan. |
campaign |
str
|
Campaign id. |
config_hash |
str
|
Config hash of the campaign. |
problem |
ProblemRef
|
The problem solved. |
solver |
SolverEntry
|
The solver and its parameters. |
protocol |
Protocol
|
The protocol and its stop criteria. |
replicate |
int
|
Replicate number, from 0. |
seed |
int
|
Seed the solver received. |
outcome |
Outcome
|
What the run found. |
result_hash |
str
|
|
schedule |
Schedule | None
|
The best schedule, unless the campaign did not record it. |
provenance |
Provenance
|
How the attempt that completed the run executed. |
notes |
tuple[str, ...]
|
What the engine noticed about the attempt: the claims of the solver that the checks of its result contradict, a wall time beyond the time limit, an elapsed time the engine's clock contradicts, and a trace whose times or evaluation counts are out of order or that ends at another value than the outcome. Empty when the engine noticed nothing, and when the record does not state them. |
identity_hash |
str | None
|
|
replayable
property
¶
Whether a replay can reproduce the run.
A run stopped by its wall-clock limit cannot, and neither can any run
of an iterative solver under a protocol whose parameter schedules
read the elapsed time, whatever stopped it. They do when the time
limit is the protocol's only bound, and under any time limit when its
schedule is "binding"
(schedule_timed): a
search then places itself on its schedule by the elapsed time
(progress()), so what
it does before it stops, at a target, at another limit, or for lack
of improvement, depends on the speed and load of the machine. A
solver that is not iterative follows no such schedule, and its run
that completed under a time limit reproduces.
verify
¶
Return the integrity problems of the record, empty when it is intact.
The outcome must hash to result_hash, stored schedule rows
must match the outcome's schedule digest, and the identity must
hash to identity_hash when the record states one.
identity
¶
Return the fields the identity hash covers.
They are the digest of the problem, the id and the parameters of the solver, the protocol with its stop criteria, the replicate, and the seed: what, with the library, determines the result of a run. The metadata a solver declares and the campaign's label for it are left out.
identity_digest
¶
Return the SHA-256 digest of the canonical JSON of identity().
from_dict
classmethod
¶
from_dict(document: object) -> RunRecord
Build a record from to_dict() output.
The field notes may be absent, and is then empty, and so may
identity_hash, which is then None.
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the document is not a run record of this schema version, or is malformed. |
RunError
dataclass
¶
RunError(kind: ErrorKind, message: str, type: str | None = None, traceback: str | None = None)
The error that ended a failed attempt.
Attributes:
| Name | Type | Description |
|---|---|---|
kind |
ErrorKind
|
What ended the attempt. |
message |
str
|
Human-readable description. |
type |
str | None
|
Qualified exception class, for exceptions. |
traceback |
str | None
|
Formatted traceback, for exceptions. |
Incumbent
dataclass
¶
Incumbent(objective: str, value: float, objectives: Mapping[str, float], feasible: bool, evaluations: int, iteration: int, elapsed: float)
A candidate that the search of a killed run had reported as its best so far.
The values are the ones the solver's search computed. No schedule comes with them, so they are not checked as the result of a completed run is.
Attributes:
| Name | Type | Description |
|---|---|---|
objective |
str
|
The objective the search optimised. |
value |
float
|
The candidate's value of that objective. |
objectives |
Mapping[str, float]
|
Every objective of the problem for the candidate, by name. |
feasible |
bool
|
Whether the search found the candidate feasible. |
evaluations |
int
|
Candidates the search had evaluated when it found this one. |
iteration |
int
|
Main-loop iteration in which the search found it. |
elapsed |
float
|
Seconds from the start of the search until it found this one. |
from_dict
classmethod
¶
from_dict(document: object) -> Incumbent
Build an incumbent from to_dict() output.
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the document is malformed or holds a value that is not finite. |
FailureRecord
dataclass
¶
FailureRecord(run_id: str, campaign: str, config_hash: str, problem: ProblemRef, solver: SolverEntry, protocol: Protocol, replicate: int, seed: int, category: Category, error: RunError, provenance: Provenance, solve_started: bool | None = None, last_incumbent: Incumbent | None = None, limit_incumbent: Incumbent | None = None)
One failed attempt of a run.
Attributes:
| Name | Type | Description |
|---|---|---|
run_id |
str
|
Run id from the plan. |
campaign |
str
|
Campaign id. |
config_hash |
str
|
Config hash of the campaign. |
problem |
ProblemRef
|
The problem. |
solver |
SolverEntry
|
The solver and its parameters. |
protocol |
Protocol
|
The protocol and its stop criteria. |
replicate |
int
|
Replicate number, from 0. |
seed |
int
|
Seed the solver received. |
category |
Category
|
Whether another attempt may succeed. |
error |
RunError
|
What ended the attempt. |
provenance |
Provenance
|
How the attempt executed. CPU time and memory are only known when the attempt raised an exception. |
solve_started |
bool | None
|
Whether the solve had begun when the attempt ended.
|
last_incumbent |
Incumbent | None
|
For a run killed at its hard limit, the last
candidate its search had reported. |
limit_incumbent |
Incumbent | None
|
For a run killed at its hard limit, the candidate
its search held when the time limit of its protocol passed: the
last one it reported within that many seconds of the start of the
solve. Equal to |
from_dict
classmethod
¶
from_dict(document: object) -> FailureRecord
Build a record from to_dict() output.
The fields solve_started, last_incumbent, and
limit_incumbent may be absent, and are then None.
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the document is not a failure record of this schema version, or is malformed. |
PlanChange
dataclass
¶
PlanChange(left: tuple[Mapping[str, Any], ...] = (), entered: tuple[Mapping[str, Any], ...] = (), reseeded: tuple[Mapping[str, Any], ...] = (), solvers: tuple[Mapping[str, Any], ...] = ())
How a plan differs from the plan a directory holds under the same config hash.
A campaign file fixes the config hash, and the plan also follows what the
file does not hold: the metadata the registered solvers declare, the
content of the problems, and the library that plans. Each run is a
document with run_id, problem, solver, protocol,
replicate, and seed.
Attributes:
| Name | Type | Description |
|---|---|---|
left |
tuple[Mapping[str, Any], ...]
|
The runs of the earlier plan that the plan does not hold, in the order of the earlier plan. |
entered |
tuple[Mapping[str, Any], ...]
|
The runs of the plan that the earlier plan did not hold, in plan order. |
reseeded |
tuple[Mapping[str, Any], ...]
|
The runs both plans hold whose seed differs, in plan
order, each with the seed of the plan and, as |
solvers |
tuple[Mapping[str, Any], ...]
|
The fields by which the solver entries differ, in plan
order: each a document with |
SessionRecord
dataclass
¶
SessionRecord(session: int, campaign: str, config_hash: str, started: str, finished: str | None, status: SessionStatus, execution: Execution, workers: int, environment: Mapping[str, Any], counts: Mapping[str, int] = dict(), plan_digest: str | None = None, plan_left: tuple[Mapping[str, Any], ...] = (), plan_entered: tuple[Mapping[str, Any], ...] = (), plan_reseeded: tuple[Mapping[str, Any], ...] = (), plan_solvers: tuple[Mapping[str, Any], ...] = (), changes: tuple[Mapping[str, Any], ...] = (), load: Mapping[str, Any] = dict())
One invocation of the runner on a campaign.
Attributes:
| Name | Type | Description |
|---|---|---|
session |
int
|
Session number, from 1. |
campaign |
str
|
Campaign id. |
config_hash |
str
|
Config hash of the campaign. |
started |
str
|
UTC start time in ISO 8601. |
finished |
str | None
|
UTC end time, or |
status |
SessionStatus
|
How the session ended. |
execution |
Execution
|
The execution settings in force. |
workers |
int
|
Number of workers, with |
environment |
Mapping[str, Any]
|
The environment, as
|
counts |
Mapping[str, int]
|
Run counts at the end of the session: |
plan_digest |
str | None
|
Digest of the run ids and seeds of the plan the session
executed, |
plan_left |
tuple[Mapping[str, Any], ...]
|
The runs of the plan the directory held before the
session, under the same config hash, that the session's plan
does not hold, as |
plan_entered |
tuple[Mapping[str, Any], ...]
|
The runs of the session's plan that the earlier plan did not hold. |
plan_reseeded |
tuple[Mapping[str, Any], ...]
|
The runs both plans hold whose seed differs. |
plan_solvers |
tuple[Mapping[str, Any], ...]
|
The fields by which the solver entries of the two plans differ. |
changes |
tuple[Mapping[str, Any], ...]
|
How the session differs from the last session the
directory recorded before it: each a document with |
load |
Mapping[str, Any]
|
The load of the machine, as
|
from_dict
classmethod
¶
from_dict(document: object) -> SessionRecord
Build a record from to_dict() output.
A document without plan_digest, plan_left, plan_entered,
plan_reseeded, or plan_solvers reads as a record that states
no digest and no change of plan, and one without changes or
load as a record that states neither.
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the document is not a session record of this schema version, or is malformed. |
settled
¶
settled(failures: Sequence[FailureRecord], max_attempts: int) -> bool
Return whether a run's failures settle it, so it gets no further attempt.
A run that never completed is settled when its latest failure is
terminal, or when it has failed max_attempts times.
schedule_rows
¶
schedule_rows(schedule: Schedule) -> list[list[Any]]
Return a schedule as [task, mode, start, end] rows in schedule order.
schedule_digest
¶
schedule_digest(schedule: Schedule) -> str
Return the SHA-256 digest of the canonical JSON of schedule_rows().
protocol_from_dict
¶
protocol_from_dict(document: object) -> Protocol
Build a protocol from the dictionary its to_dict method returns.
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the document is malformed. |
dispatchatlas.campaigns.store
¶
The result store: where a campaign's files live, and how to read them back.
Layout under an output root:
<root>/<campaign id>/
campaign.json the campaign document, its base directory, its config hash
plan.json the plan, with parities and the fairness assessment
sessions/<n>.json one session record per invocation of the runner, with the
digest of its plan and the runs that left or entered it
runs/<solver>/<problem>/r<replicate:03d>-<protocol>.json
failures/<solver>/<problem>/r<replicate:03d>-<protocol>.attempt-<n>.json
superseded/runs/<solver>/<problem>/r<replicate:03d>-<protocol>.<run id>.json
superseded/failures/<solver>/<problem>/r<replicate:03d>-<protocol>.attempt-<n>.<run id>.json
files moved aside when new records took their places
logs/session-<n>.log what each session logged, with UTC time stamps
.lock locked while a runner works on the campaign
<solver> is the solver's name, its label or else its id. Every document
is written atomically, with sorted keys, two-space indentation, ASCII text,
and a final newline, and a value that is not finite is refused.
The readers read_campaign(), read_plan(), read_runs(),
read_failures(), and read_sessions() return typed records. They
only return records of the plan in plan.json: a file left by an earlier
plan, whose run id or config hash differs, is skipped with a warning. A run
record raises an error when its content no longer matches its result hash or
its identity hash, or when it states another problem, solver, protocol,
replicate, or seed than the plan states for its run. read_runs() also
warns about every other file under runs/, which read_strays() lists:
records of runs the plan does not hold, files that cannot be read, and files
that are no records.
Store is the writing side, used by the runner: it locks the
directory, refuses to mix configurations, and settles which runs remain. It
replaces no record. A file at the path of a planned run that is not the valid
record of that run, such as a record of another version of its problem, stays
where it is until the run is repeated: when the repeated run gets its first
record, of its result or of a failed attempt, the file is moved under
superseded/, with the run id it states before .json. The file of a
failed attempt is moved the same way when a new attempt takes its path. So a
session that repeats no run moves nothing, and a record counts again when its
problem is restored.
Nothing replaces a file under superseded/, and the only reader of that
folder is the store itself, which moves a record back to its path when it is
the valid record of a planned run that has none.
Strays
dataclass
¶
Strays(unplanned: tuple[Path, ...] = (), unreadable: tuple[Path, ...] = (), other: tuple[Path, ...] = ())
The files under runs/ that are at no path of a planned run.
Attributes:
| Name | Type | Description |
|---|---|---|
unplanned |
tuple[Path, ...]
|
Run records, such as those of runs that left the plan. |
unreadable |
tuple[Path, ...]
|
Files named |
other |
tuple[Path, ...]
|
Files that are no JSON documents, such as a temporary file an interrupted write left behind. |
Stale
dataclass
¶
Stale(run_id: str, problem: str, path: Path, found: str, reason: str, changed_problem: bool = False)
A file at the path of a planned run that is no valid record of the run.
Attributes:
| Name | Type | Description |
|---|---|---|
run_id |
str
|
Id of the planned run. |
problem |
str
|
Id of the problem of the planned run. |
path |
Path
|
The file: the one at the path of the run's record, or one at the path of a failed attempt of the run. |
found |
str
|
The run id the file states, or |
reason |
str
|
Why the file does not count as a record of the run. |
changed_problem |
bool
|
Whether the file is a record made on another version of the problem of the run. |
Kept
dataclass
¶
A file under superseded/ that is the valid record of a planned run without one.
Attributes:
| Name | Type | Description |
|---|---|---|
run_id |
str
|
Id of the planned run. |
kept |
Path
|
The file under |
path |
Path
|
The path of the run's record, where the file belongs. |
Ledger
dataclass
¶
Ledger(completed: frozenset[str] = frozenset(), failures: Mapping[str, tuple[FailureRecord, ...]] = dict(), stale: tuple[Stale, ...] = (), kept: tuple[Kept, ...] = (), unreadable: tuple[Path, ...] = ())
What a store already holds for a plan.
Attributes:
| Name | Type | Description |
|---|---|---|
completed |
frozenset[str]
|
Ids of the runs with a valid record. |
failures |
Mapping[str, tuple[FailureRecord, ...]]
|
Failed attempts of the plan's runs, by run id, in attempt order. |
stale |
tuple[Stale, ...]
|
The files at the paths of planned runs that do not count as their records: those at the paths of run records in plan order, then those at the paths of failed attempts in path order. |
kept |
tuple[Kept, ...]
|
The valid records under |
unreadable |
tuple[Path, ...]
|
The files under |
restored
¶
restored() -> Ledger
Return the ledger as it stands once every kept record is back at its path.
The runs of the kept records are complete, and the stale files at their paths are no longer listed. Nothing is moved.
next_attempt
¶
Return the number of the next attempt of a run: one past its last failure.
pending
¶
pending(plan: Plan, max_attempts: int, *, retry_failed: bool = False) -> tuple[PlannedRun, ...]
Return the runs of a plan that still need an attempt, in plan order.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
plan
|
Plan
|
The plan. |
required |
max_attempts
|
int
|
Attempts a run gets before a recoverable failure settles it. |
required |
retry_failed
|
bool
|
Also return runs that failures settled. |
False
|
Store
¶
The writing side of a campaign directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
directory
|
Path
|
The campaign directory, created when missing. |
required |
locked
¶
Hold the campaign's lock, so two runners never work on it at once.
The lock is an operating-system file lock, released when the process ends in any way. Where the platform offers none, nothing is locked.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If another process holds the lock. |
open
¶
open(campaign: Campaign, *, fresh: bool = False) -> None
Prepare the directory for a campaign and record the campaign in it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
campaign
|
Campaign
|
The campaign. |
required |
fresh
|
bool
|
Discard every result the directory holds first. |
False
|
Records are checked one by one when the runner reads them, so a
campaign.json that cannot be read is simply replaced.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the directory holds results of another
configuration and |
plan_change
¶
plan_change(plan: Plan) -> PlanChange
Report how a plan differs from the plan the directory holds, and warn about it.
A plan can change while the campaign file, and so the config hash,
stays the same: the plan also follows the metadata the registered
solvers declare, the content of the problems, and the library that
plans. When the directory holds a plan of the same config hash, a
warning names the problems whose content changed, and one names the
runs that left the plan, the runs that entered it, the runs whose
seed changed, and the fields of the solver entries that changed. The
records of runs that left stay in the directory and are no longer
read. Nothing is written, so the comparison gives the same answer
until write_plan() replaces the plan.
Returns:
| Type | Description |
|---|---|
PlanChange
|
The differences. Empty when the directory holds no plan, one |
PlanChange
|
that cannot be read, or one of another config hash. |
write_plan
¶
write_plan(plan: Plan) -> None
Write plan.json, replacing the plan the directory holds.
ledger
¶
Read which runs of a plan are complete and which attempts failed.
A file at the path of a planned run that cannot be read, belongs to
another plan, no longer matches its hashes, or states another
problem, solver, protocol, replicate, or seed than the plan states
for the run does not count: it is listed as stale and its run is
attempted again. The file of a failed attempt at the path of a
planned run that belongs to another plan is listed as stale too, and
is no attempt of the run. A run without a valid record whose valid
record lies under superseded/ is listed as kept. Nothing is
moved or written here: reconcile() brings the kept records
back, and write_run() and write_failure() move a stale
file aside when a new record takes its place.
The library that plans derives the seeds, so a plan can give a run another seed than the plan the directory holds. A record made with the seed its run has in that plan counts, as one made with a seed a session recorded as replaced does.
holdings
¶
Read the ledger of a plan together with the valid records it counts.
Returns:
| Type | Description |
|---|---|
Ledger
|
The ledger, as |
dict[str, RunRecord]
|
run it lists as completed, by run id and in plan order. Each |
tuple[Ledger, dict[str, RunRecord]]
|
record is read and checked once. |
refuse_changed_problems
¶
refuse_changed_problems(ledger: Ledger) -> None
Refuse a directory that holds a record made on another version of a problem.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If a stale file of the ledger is the record of a planned run, or of a failed attempt of one, made on another version of its problem. |
reconcile
¶
Bring back the kept records of a ledger, and warn about its stale files.
Each kept record is moved from superseded/ back to the path of
its run, after the stale file at that path, if there is one, is
moved aside. A move the operating system refuses is warned about,
and the run is then attempted again. One warning names the stale
files that remain: each stays where it is until a session repeats
its run (write_run(), write_failure()).
Returns:
| Type | Description |
|---|---|
Ledger
|
The ledger after the moves: the runs whose records are back |
Ledger
|
are complete, and no record is listed as kept. |
last_session
¶
last_session() -> SessionRecord | None
Return the record of the latest session, or None when there is none.
A session record that cannot be read is skipped with a warning, and the one before it is returned.
next_session
¶
Return the number of the next session: one past the highest recorded.
write_run
¶
write_run(record: RunRecord) -> Path
Write the record of a completed run and return its path.
A file that is at the path is moved under superseded/ first.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If a file at the path cannot be moved aside. The record is then not written. |
write_failure
¶
write_failure(record: FailureRecord) -> Path
Write the record of a failed attempt and return its path.
A file that is at the path, such as the record of an attempt on
another version of the problem, is moved under superseded/ first.
So is a file at the path of the run's record: a run is attempted
only while it has no valid record, so what stands there is none,
and the run it stood for has now been repeated.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If such a file cannot be moved aside. The record is then not written. |
campaign_directory
¶
Return the directory of a campaign under an output root.
run_path
¶
Return the path of the record of one run.
failure_path
¶
failure_path(directory: Path, solver: str, problem: str, replicate: int, protocol: str, attempt: int) -> Path
Return the path of the record of one failed attempt.
recorded_config_hash
¶
Return the config hash a campaign directory records, or None if it has none.
A missing or unreadable campaign.json records none.
read_campaign
¶
read_campaign(directory: str | Path) -> Campaign
Read the campaign a store was created from.
Relative paths in its sources refer to the directory recorded with it, so the campaign can be planned again from the store alone.
Raises:
| Type | Description |
|---|---|
SerializationError
|
If |
read_plan
¶
read_plan(directory: str | Path) -> Plan
Read the plan of a campaign.
Raises:
| Type | Description |
|---|---|
SerializationError
|
If |
read_runs
¶
read_runs(directory: str | Path, *, verify: bool = True) -> tuple[RunRecord, ...]
Read every completed run of a campaign's plan, in plan order.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
directory
|
str | Path
|
The campaign directory. |
required |
verify
|
bool
|
Check that every record still matches its result hash, schedule digest, and identity hash, and that it states what the plan states for its run: the problem and its digest, the solver with its parameters, the protocol, the replicate, and the seed. The comparison with the plan needs no hash, so it also covers a record that states no identity hash. A seed the run had in an earlier plan of the directory is accepted, as the session records name it. |
True
|
Every other file under runs/ is warned about, by kind and with the
first names (read_strays()).
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the plan or a record is malformed, or, with
|
read_strays
¶
read_strays(directory: str | Path) -> Strays
List the files under runs/ that are at no path of a run of the plan.
No reader returns such a file, and no session replaces it.
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the plan is missing or malformed. |
read_failures
¶
read_failures(directory: str | Path) -> tuple[FailureRecord, ...]
Read every failed attempt of a campaign's plan, by plan order and attempt.
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the plan or a record is malformed. |
read_sessions
¶
read_sessions(directory: str | Path) -> tuple[SessionRecord, ...]
Read every session record of a campaign, by session number.
Raises:
| Type | Description |
|---|---|
SerializationError
|
If a record is malformed. |
dispatchatlas.campaigns.environment
¶
The environment a campaign session runs in.
capture_environment() records, once per session, the facts that can
change a measurement or explain a difference between sessions: the Python
version and implementation, the operating system and machine, the processor
model with its sockets and cores, the processors the process may use, the
thread limits its environment sets for numerical libraries, the memory size,
the versions of DispatchAtlas, NumPy, and the optional solver backends that
are installed, where the DispatchAtlas distribution was installed from and at
which revision, a digest of the DispatchAtlas source files, every installed
plugin distribution with its version and revision, and the git state of the
DispatchAtlas source and of the working directory when either is a git
checkout. It records no host or user name.
sample_load() measures how busy the processors of the machine are over
a short interval. A session takes one sample before its first run and one
after its last, so its record shows a machine that other work shared.
LIBRARY
module-attribute
¶
Name of the distribution and of the package whose source a session identifies.
PACKAGES
module-attribute
¶
PACKAGES = (LIBRARY, 'numpy', 'ortools', 'pulp', 'highspy', 'pyscipopt', 'gurobipy')
Distributions whose versions a session records when they are installed.
PLUGIN_GROUPS
module-attribute
¶
Entry-point groups whose distributions a session records as plugins.
LOAD_SAMPLE_SECONDS
module-attribute
¶
Seconds over which sample_load() measures the processors by default.
UNTRACKED_HASHED
module-attribute
¶
Most untracked files whose content enters the digest of a git state.
UNTRACKED_BYTES
module-attribute
¶
Most bytes of untracked files that are read for the digest of a git state.
TreeComparison
dataclass
¶
TreeComparison(matched: int, missing: tuple[str, ...], extra: tuple[str, ...], differing: tuple[str, ...], normalized_newlines: bool)
How the files of two directory trees compare, by relative path and SHA-256 digest.
Attributes:
| Name | Type | Description |
|---|---|---|
matched |
int
|
Files present in both trees with equal digests. |
missing |
tuple[str, ...]
|
Relative paths in the source tree but not in the installed one. |
extra |
tuple[str, ...]
|
Relative paths in the installed tree but not in the source one. |
differing |
tuple[str, ...]
|
Relative paths in both trees whose digests differ. |
normalized_newlines |
bool
|
Whether CRLF was read as LF before hashing, so
that |
capture_environment
¶
capture_environment(directory: str | Path | None = None, *, exclude: Sequence[str | Path] = ()) -> dict[str, Any]
Describe the machine and software a session runs on.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
directory
|
str | Path | None
|
The working directory whose git state to record. Defaults to the current directory. |
None
|
exclude
|
Sequence[str | Path]
|
Directories left out of every git state: that of the working directory, of the DispatchAtlas source, and of the checkout of every editable plugin. The directory a campaign writes its results to is one, since its files change with every session. |
()
|
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
A JSON-ready dictionary with the keys |
dict[str, Any]
|
|
dict[str, Any]
|
|
dict[str, Any]
|
|
dict[str, Any]
|
|
dict[str, Any]
|
processors the process may use: their indices as |
dict[str, Any]
|
their number as |
dict[str, Any]
|
tell which processors the process is confined to, |
dict[str, Any]
|
|
dict[str, Any]
|
|
dict[str, Any]
|
|
dict[str, Any]
|
environment of the process, |
dict[str, Any]
|
processes start with the campaign's |
dict[str, Any]
|
of them instead. |
dict[str, Any]
|
|
dict[str, Any]
|
|
dict[str, Any]
|
|
dict[str, Any]
|
tracked in a git checkout. |
dict[str, Any]
|
|
dict[str, Any]
|
runs is not the one the installed distribution provides, for |
dict[str, Any]
|
example a copy that comes first on the module search path, the |
dict[str, Any]
|
installer's record does not describe it: |
dict[str, Any]
|
|
dict[str, Any]
|
checkout, |
dict[str, Any]
|
states the version of the installed distribution. |
dict[str, Any]
|
the name of every distribution that provides an entry point of |
dict[str, Any]
|
|
dict[str, Any]
|
|
dict[str, Any]
|
under |
dict[str, Any]
|
|
dict[str, Any]
|
packages that are not installed and directories that are not git |
dict[str, Any]
|
checkouts are left out. |
distribution_state
¶
Return the version of an installed distribution and where it was installed from.
The installer's record of a direct installation (direct_url.json in
the distribution's metadata) tells the origin.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
Name of the distribution. |
required |
exclude
|
Sequence[str | Path]
|
Directories left out of the git state of an editable distribution's checkout. |
()
|
Returns:
| Type | Description |
|---|---|
dict[str, Any] | None
|
|
dict[str, Any] | None
|
dictionary with |
dict[str, Any] | None
|
installation from a version-control URL, |
dict[str, Any] | None
|
an archive file or URL, |
dict[str, Any] | None
|
directory, and |
dict[str, Any] | None
|
origin, as for a package index; |
dict[str, Any] | None
|
distribution runs the files of that local directory; and |
dict[str, Any] | None
|
|
dict[str, Any] | None
|
|
dict[str, Any] | None
|
checkout, whose |
dict[str, Any] | None
|
revision of a copy built from a local directory is |
dict[str, Any] | None
|
record does not say which state of the directory was built. |
plugin_states
¶
Return the distribution_state() of every installed plugin distribution, by name.
A plugin distribution is one that provides an entry point of
PLUGIN_GROUPS.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
exclude
|
Sequence[str | Path]
|
Directories left out of the git state of every editable plugin's checkout. |
()
|
source_digest
¶
Return a digest of the files of a package directory.
The SHA-256 digest covers the relative path and the content of every file under the directory, in path order, except compiled bytecode. Line endings are read as line feeds, so a checkout and an installed copy of the same files have one digest on every platform.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
package
|
str | Path | None
|
The package directory. Defaults to the DispatchAtlas package this module belongs to. |
None
|
Returns:
| Type | Description |
|---|---|
str | None
|
The hexadecimal digest, or |
compare_trees
¶
compare_trees(installed: str | Path, source: str | Path, *, normalize_newlines: bool = True) -> TreeComparison
Compare an installed copy of a package with the source tree it should match.
Every file under each directory is compared by its path relative to the
directory and the SHA-256 digest of its content; __pycache__
folders and compiled bytecode are left out, as in
source_digest().
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
installed
|
str | Path
|
The installed package directory, for example from
|
required |
source
|
str | Path
|
The package directory of the source tree. |
required |
normalize_newlines
|
bool
|
Read CRLF line endings as LF before hashing, so
a checkout with converted line endings matches a copy built from
the same commit. |
True
|
Raises:
| Type | Description |
|---|---|
OSError
|
If a directory or a file cannot be read. |
installed_package_dir
¶
Return the directory the import system loads package name from, or None.
The package is located without being imported.
git_state
¶
git_state(directory: Path, *, exclude: Sequence[str | Path] = (), within: str | Path | None = None) -> dict[str, Any] | None
Return the commit of the git checkout holding directory, and how it is modified.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
directory
|
Path
|
A directory in the checkout. |
required |
exclude
|
Sequence[str | Path]
|
Directories whose files are left out of the comparison with the commit. One outside the checkout changes nothing. |
()
|
within
|
str | Path | None
|
A folder of the checkout, absolute or relative to
|
None
|
Returns:
| Type | Description |
|---|---|
dict[str, Any] | None
|
|
dict[str, Any] | None
|
available. Otherwise a dictionary with |
dict[str, Any] | None
|
whether the checkout differs from the commit in a tracked file or |
dict[str, Any] | None
|
holds an untracked file that git does not ignore; |
dict[str, Any] | None
|
the number of such untracked files; and |
dict[str, Any] | None
|
digest of the uncommitted changes: the status of every changed |
dict[str, Any] | None
|
path, the difference of the tracked files from the commit, and the |
dict[str, Any] | None
|
content of the untracked files. Two checkouts of one commit with |
dict[str, Any] | None
|
different uncommitted changes therefore have different digests. The |
dict[str, Any] | None
|
digest of a checkout that is not dirty is |
dict[str, Any] | None
|
|
The untracked files are read in path order, at most
UNTRACKED_BYTES bytes in all: a file that would pass that total
enters the digest by its path, its size, and its modification time
instead of its content. More than UNTRACKED_HASHED untracked
files enter by their paths only. The queries leave the checkout as it
is: git writes neither its index nor any other file for them.
sample_load
¶
sample_load(seconds: float = LOAD_SAMPLE_SECONDS) -> dict[str, Any]
Measure how busy the processors of the machine are over a short interval.
The processor times of the whole machine are read twice, seconds
apart, so the sample counts the work of every process, this one
included.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
seconds
|
float
|
Length of the interval. |
LOAD_SAMPLE_SECONDS
|
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
A JSON-ready dictionary with |
dict[str, Any]
|
in ISO 8601; |
dict[str, Any]
|
of the machine that was not idle over the interval, between 0 and |
dict[str, Any]
|
1; and |
dict[str, Any]
|
|
dict[str, Any]
|
times, and no interval then passes; |
dict[str, Any]
|
where the platform has none. |
dispatchatlas.campaigns.checking
¶
Checking a campaign directory before its results are used.
check_campaign() reads a campaign directory, writes nothing, and returns
a CheckReport: a list of Finding objects, each with a code,
a severity, a message, and the items it concerns. It answers what no reader
of results asks: whether the stored campaign is the one that was run, whether
the plan is the plan that campaign gives, whether every planned run has a
valid record or failures that settle it, whether the budgets were kept, and
whether the sessions agree on code and settings.
A finding has one of three severities:
blocking: the directory does not hold what it claims, so its results are not ready to be used: a missing or altered campaign file, a plan that is not the campaign's, a run without a record that is owed an attempt, a record that fails a hash, a seed used twice.limitation: the results stand, with a caveat a report of them has to state: runs settled by failures, cells with fewer runs than planned, budgets that were passed, sessions on other code.note: a fact worth knowing that qualifies nothing, such as an optional field that the records of the directory do not state.
CODES lists every code with its severity and meaning. Findings on the
runs of one protocol name the protocol, so a reader of one protocol's results
can leave out the others.
The check reads what the engine stored and derives the rest again: the
ledger of the store (Store.holdings) says which records are valid,
observations() derives the facts of a
record's time and trace from the record alone, so records without stored
notes are covered,
assess_outcomes() compares the wall
times and stop reasons with the plan, and
session_changes() compares the sessions
that executed the counted attempts.
SEVERITIES
module-attribute
¶
SEVERITIES: tuple[Severity, ...] = ('blocking', 'limitation', 'note')
The severities, from the heaviest.
CODES
module-attribute
¶
CODES: Mapping[str, tuple[Severity, str]] = {'recipe-missing': ('blocking', 'campaign.json is missing.'), 'recipe-unreadable': ('blocking', 'campaign.json cannot be read as a campaign.'), 'recipe-altered': ('blocking', 'campaign.json no longer matches its recorded config hash.'), 'plan-unreadable': ('blocking', 'plan.json is missing or cannot be read.'), 'plan-inconsistent': ('blocking', 'plan.json holds a run of a problem, solver, or protocol it does not list, or an id twice.'), 'files-inaccessible': ('blocking', 'The operating system does not let read a file of the directory, so its runs, files, records, and sessions were not checked.'), 'plan-foreign': ('blocking', 'plan.json states another config hash than campaign.json.'), 'plan-differs': ('blocking', 'plan.json is not the plan the stored campaign gives.'), 'plan-digest-differs': ('blocking', 'plan.json holds other runs or seeds than the plan the last session executed.'), 'runs-unattempted': ('blocking', 'Planned runs have no record and no failed attempt.'), 'runs-unsettled': ('blocking', "Planned runs failed and are owed another attempt, which the campaign's retries allow."), 'record-unreadable': ('blocking', "The file at a planned run's path is no run record."), 'record-foreign': ('blocking', "The record at a planned run's path belongs to another configuration or to another version of its problem."), 'record-hash-mismatch': ('blocking', 'A record fails its result hash, its schedule digest, or its identity hash.'), 'record-differs-from-plan': ('blocking', 'A record states another problem, solver, protocol, replicate, or seed than the plan.'), 'records-set-aside': ('blocking', 'The valid record of a planned run lies under superseded/ and not at its path.'), 'seed-reused': ('blocking', 'One seed serves two replicates of one problem.'), 'plan-not-derived': ('limitation', 'The plan could not be derived again from the stored campaign on this installation.'), 'plan-changed': ('limitation', 'The plan changed between sessions while the campaign file stayed the same.'), 'runs-failed': ('limitation', 'Runs never completed and are settled by their failures.'), 'cells-partial': ('limitation', 'A solver has records for some of the runs planned on a problem, not for all.'), 'solver-without-records': ('limitation', 'A solver with planned runs has no record.'), 'solver-excluded-everywhere': ('limitation', 'The plan excludes a solver from every problem, so it has no run.'), 'records-unplanned': ('limitation', 'Records under runs/ belong to runs the plan does not hold; no reader returns them.'), 'files-unreadable': ('limitation', 'Files under runs/, at no path of a planned run, cannot be read as run records.'), 'failures-unreadable': ('limitation', 'Files under failures/ cannot be read as records of failed attempts.'), 'trace-time-axis': ('limitation', 'Traces hold a negative time, times or evaluation counts that decrease, or a time beyond the elapsed time of their run.'), 'trace-contradicts-outcome': ('limitation', 'The last value of a trace differs from the value of its outcome.'), 'record-without-trace': ('limitation', 'Records of an iterative solver hold a schedule and no trace.'), 'budget-overrun': ('limitation', 'Completed runs took more wall time than their time limit and the tolerance allow.'), 'hard-limits-differ': ('limitation', 'The attempts under one protocol ran under different hard limits.'), 'stop-reasons-diverge': ('limitation', 'The runs of the iterative solvers under one protocol ended for different reasons.'), 'sessions-unreadable': ('limitation', 'A session record cannot be read.'), 'sessions-missing': ('limitation', 'Attempts name sessions the directory holds no record of.'), 'sessions-differ-code': ('limitation', 'The sessions that executed the counted attempts differ in the Python version, the packages, the library, or a git state.'), 'sessions-differ-plugins': ('limitation', 'The sessions that executed the counted attempts differ in their plugins.'), 'sessions-differ-execution': ('limitation', 'The sessions that executed the counted attempts differ in their execution settings.'), 'sessions-differ-processors': ('limitation', 'The sessions that executed the counted attempts of a protocol with a time limit differ in the processor, the processors they were allowed, or the thread settings.'), 'replicates-below-floor': ('limitation', 'Cells of replicated solvers hold fewer completed replicates than the floor.'), 'seeds-unpaired': ('limitation', 'One replicate of a problem ran with different seeds, so its runs are not paired.'), 'plan-not-compared': ('note', 'The plan was not derived again from the stored campaign, as the caller asked.'), 'plan-digest-unstated': ('note', 'No session record states the digest of its plan.'), 'plan-entries-differ': ('note', 'plan.json holds entries the library does not write for its plan: entries derived from the plan, or texts, that were changed or that another version wrote.'), 'pairs-excluded': ('note', 'The plan excludes pairs of problem and solver.'), 'files-stray': ('note', 'Files under runs/ are no JSON documents.'), 'failures-foreign': ('note', 'Files at the paths of failed attempts belong to another plan and count as no attempt.'), 'records-without-identity-hash': ('note', 'Records state no identity hash; they are compared with the plan field by field.'), 'records-noted': ('note', 'Records carry notes of the engine that the check does not derive from the record, such as a claim of the solver that the checks of its result contradict.'), 'clock-disagreement': ('note', "A solver's elapsed time differs from the engine's wall time, or a run stopped for its time limit before the limit."), 'sessions-differ-machine': ('note', 'The sessions that executed the counted attempts differ in their machine. What of it a time limit is measured on is the limitation sessions-differ-processors.'), 'sessions-fields-unstated': ('note', 'Session records do not state a part of the environment that another states, so that part is compared among the other sessions only.'), 'session-unfinished': ('note', 'The last session is not recorded as finished.')}
Every code a finding can have, with its severity and what it means.
Severity
¶
How much a finding weighs: it blocks the use of the results, qualifies them, or informs.
Finding
dataclass
¶
Finding(code: str, severity: Severity, message: str, items: tuple[str, ...] = (), protocol: str | None = None)
One thing a check of a campaign directory found.
Attributes:
| Name | Type | Description |
|---|---|---|
code |
str
|
What was found, a key of |
severity |
Severity
|
How much it weighs. |
message |
str
|
One or two sentences that state it. |
items |
tuple[str, ...]
|
What it concerns, one entry each: cells, files, solvers, or fields, in the order of the plan or of the paths. |
protocol |
str | None
|
The protocol whose runs it concerns, or |
CheckReport
dataclass
¶
CheckReport(directory: Path, campaign: str | None, config_hash: str | None, findings: tuple[Finding, ...], counts: Mapping[str, int] = dict())
What a check of a campaign directory found.
Attributes:
| Name | Type | Description |
|---|---|---|
directory |
Path
|
The campaign directory. |
campaign |
str | None
|
Campaign id of the plan, or |
config_hash |
str | None
|
Config hash of the plan, or else the one
|
findings |
tuple[Finding, ...]
|
The findings, the blocking ones first, then the limitations, then the notes. |
counts |
Mapping[str, int]
|
The planned runs by what the directory holds for them:
|
check_campaign
¶
check_campaign(directory: str | Path, *, replicate_floor: int | None = None, overrun_tolerance: float = OVERRUN_MARGIN, replan: bool = True, registry: SolverRegistry | None = None, families: FamilyRegistry | None = None) -> CheckReport
Check a campaign directory and report what qualifies its results.
Nothing is written, moved, or run. The findings are, by what they concern:
- the campaign file: missing, unreadable, or no longer matching its config hash;
- the plan: of another config hash, different from the plan the stored campaign gives when it is derived again, different from the plan the last session executed, or changed between sessions;
- the planned runs: without a record and without a failed attempt, with failures that do not settle them, settled by failures (by solver and kind of error), cells with records for only some of their runs, solvers without any record, and, apart from those, the pairs and the solvers the plan excludes;
- the files: a file at a planned run's path that is no valid record of
the run (unreadable, of another configuration or problem version,
failing a hash, or differing from the plan), valid records lying
under
superseded/, records of runs the plan does not hold, and files that cannot be read or are no records; - the records: a trace with an unusable time axis or a last value that contradicts the outcome, an iterative solver's record without a trace, and wall times beyond the time limit, by solver with the largest ratio;
- the protocols: hard limits that differ among the attempts and stop reasons that diverge among the iterative solvers;
- the sessions: those that executed the counted attempts and differ in code, plugins, execution settings, or machine, each field that differs with its values and the sessions that have each; those that executed the attempts of a protocol with a time limit and differ in the processor, the processors they were allowed, or the thread settings; and those without a record;
- the replication: cells below
replicate_floor, a seed used for two replicates of one problem, and a replicate that ran with two seeds.
Records and session records may lack the fields that are optional in them. Where the check cannot make a comparison for that reason, it says so in a note: records without an identity hash, session records without the digest of their plan, and session records that state less of their environment than another's.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
directory
|
str | Path
|
A campaign directory, |
required |
replicate_floor
|
int | None
|
The fewest completed replicates a cell of a
replicated solver may hold without a finding. |
None
|
overrun_tolerance
|
float
|
Share of a time limit by which the wall time of a
completed run may pass it without counting as over, as in
|
OVERRUN_MARGIN
|
replan
|
bool
|
Derive the plan again from the stored campaign and compare
it with |
True
|
registry
|
SolverRegistry | None
|
Solver registry for deriving the plan. Defaults to the built-in solvers and plugins. |
None
|
families
|
FamilyRegistry | None
|
Benchmark-family registry for deriving the plan. Defaults to the built-in families and plugins. |
None
|
Returns:
| Type | Description |
|---|---|
CheckReport
|
The report. With a plan that cannot be read, or whose runs name |
CheckReport
|
what it does not list, it holds the findings on the campaign file |
CheckReport
|
and the plan only and no counts. When the operating system does not |
CheckReport
|
let read a record, a file of a failed attempt, or a folder, as it |
CheckReport
|
can while a session writes into the directory, the report says so |
CheckReport
|
in a blocking finding and holds no counts either. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If |
dispatchatlas.campaigns.replaying
¶
Replay: run completed runs again and check that they reproduce.
replay() re-executes runs of a campaign from their recorded seeds, with
the same solver parameters and stop criteria, and compares result hashes.
Before that it checks each record against its own result hash, so a record
edited after it was written is caught even though its run reproduces. It
never writes to the store.
A run stopped by its wall-clock limit is skipped: how far it got depends on the speed and load of the machine. Other runs reproduce exactly on any machine where the solver's arithmetic does; transcendental functions of the C library may differ in their last bit between platforms, which can steer a stochastic search elsewhere. Each mismatch lists the outcome fields that differ.
Verdict
¶
Bases: StrEnum
What replaying one run showed.
MATCH
class-attribute
instance-attribute
¶
The run reproduced its recorded result, and the record is intact.
MISMATCH
class-attribute
instance-attribute
¶
The result differs, or the record no longer matches its result hash.
SKIPPED
class-attribute
instance-attribute
¶
The run cannot be replayed, or has no completed record.
Difference
dataclass
¶
One outcome field whose recorded and replayed values differ.
Attributes:
| Name | Type | Description |
|---|---|---|
field |
str
|
Name of the field. |
recorded |
Any
|
The value in the record. |
replayed |
Any
|
The value the replay produced. |
ReplayEntry
dataclass
¶
ReplayEntry(run_id: str, solver: str, problem: str, protocol: str, replicate: int, verdict: Verdict, detail: str = '', differences: tuple[Difference, ...] = ())
The verdict on one run.
Attributes:
| Name | Type | Description |
|---|---|---|
run_id |
str
|
Run id. |
solver |
str
|
Solver name. |
problem |
str
|
Problem id. |
protocol |
str
|
Protocol name. |
replicate |
int
|
Replicate number. |
verdict |
Verdict
|
What the replay showed. |
detail |
str
|
Explanation of a verdict other than a match. |
differences |
tuple[Difference, ...]
|
Differing outcome fields, for a mismatch. |
ReplayReport
dataclass
¶
ReplayReport(campaign: str, config_hash: str, entries: tuple[ReplayEntry, ...])
The verdicts of a replay, in plan order.
Attributes:
| Name | Type | Description |
|---|---|---|
campaign |
str
|
Campaign id. |
config_hash |
str
|
Config hash of the campaign. |
entries |
tuple[ReplayEntry, ...]
|
One entry per selected run. |
replay
¶
replay(campaign: Campaign, root: str | Path, *, runs: Iterable[str] | None = None, registry: SolverRegistry | None = None, families: FamilyRegistry | None = None, backend: str | None = None, workers: int | str | None = None, affinity: str | Sequence[int] | None = None) -> ReplayReport
Re-execute completed runs and compare their results with the records.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
campaign
|
Campaign
|
The campaign the results belong to. |
required |
root
|
str | Path
|
Output root the campaign ran under. |
required |
runs
|
Iterable[str] | None
|
Ids of the runs to replay. Defaults to every completed run. |
None
|
registry
|
SolverRegistry | None
|
Solver registry. Defaults to the built-in solvers and plugins. |
None
|
families
|
FamilyRegistry | None
|
Benchmark-family registry. Defaults to the built-in families and plugins. |
None
|
backend
|
str | None
|
Override of the campaign's execution backend. |
None
|
workers
|
int | str | None
|
Override of the campaign's worker count, or |
None
|
affinity
|
str | Sequence[int] | None
|
Override of the campaign's |
None
|
Returns:
| Type | Description |
|---|---|
ReplayReport
|
The verdict on every selected run. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the directory holds another configuration, has no
results, a requested run id is not in the plan, or runs are to
be replayed and the workers cannot be pinned as |