Campaigns¶
A campaign is a reproducible experiment: every solver of a roster runs on every problem of its instance sources, under every stopping protocol, with a number of replicates. It is described in a TOML or JSON file, planned and checked before anything runs, executed in worker processes, recorded one JSON file per run, resumed after an interruption, checked as a directory before its results are used, and replayed to verify that its runs reproduce.
dispatchatlas campaign validate campaign.toml --check-compatibility
dispatchatlas campaign estimate campaign.toml
dispatchatlas campaign run campaign.toml --output results --workers auto
dispatchatlas campaign check results/<campaign id>
dispatchatlas campaign replay results/<campaign id>
dispatchatlas analyze results/<campaign id> --output analysis
The campaign file¶
A complete file, with every key explained. It is
examples/campaigns/reference.toml in the repository, and it validates as it
is:
# A campaign file that uses every key of the format, each one explained.
# Check it without running anything:
#
# dispatchatlas campaign validate examples/campaigns/reference.toml --check-compatibility
#
# The reader is strict: an unknown key, a value of the wrong type (true is not
# a number, "false" is not a boolean), or a missing required key is an error
# that names the entry, for example instances[1].seeds[0].
# ---------------------------------------------------------------- identity
schema_version = 1 # required: the file format version this release reads
id = "reference" # required: results go to <output>/reference/; letters,
# digits, '.', '_', '-'; not part of the config hash
seed = 20240601 # required: root of every run seed
replicates = 10 # runs per stochastic solver, problem, and protocol
# (default 30); deterministic solvers run once
objectives = ["makespan"] # optional: replaces the objectives every problem
# declares, primary first, here those of the smoke
# catalog's continuum problems, which declare three
# ---------------------------------------------------------------- instances
# One or more sources. Every problem id must be unique across them. Relative
# paths are relative to this file, and are left out of the config hash:
# a problem read from a file is identified by the digest of its content.
[[instances]] # problems generated by a registered benchmark family
family = "job-shop"
parameters = { jobs = 6, machines = 4 } # the other parameters keep their defaults
seeds = [1, 2] # one problem per generator seed
[[instances]] # instances of a published suite
suite = "taillard-open-shop"
instances = ["tai4x4-1", "tai4x4-2"]
# directory = "downloads" # file suites (OR-Library, Brandimarte, PSPLIB) read the
# files you downloaded from the suite's source here;
# generated suites (Taillard's) need none
[[instances]] # every instance of a built-in catalog
catalog = "smoke"
seed = 0 # root seed of the catalog's instance seeds (default 0)
[[instances]] # problems written by `dispatchatlas catalogs build`
directory = "instances" # or by materialize(); each file is checked against
# the SHA-256 digest in the directory's manifest.json
[[instances]] # one instance file
file = "problem.json" # a problem document, or a file in a published format
# format = "orlib-job-shop" # when the extension does not imply the format:
# orlib-job-shop, orlib-flow-shop, fjs, or psplib
# ---------------------------------------------------------------- solvers
# Registered solver ids with parameter values. Two settings of one solver need
# labels, which then name them in results and tables.
[[solvers]]
id = "neh"
[[solvers]]
id = "simulated-annealing" # every parameter at its default
[[solvers]]
id = "simulated-annealing"
label = "simulated-annealing-slow-cooling"
parameters = { cooling = 0.99 }
[[solvers]]
id = "tabu-search"
parameters = { tenure = 10, candidates = 30 }
[[solvers]]
id = "min-min" # does not model permutation problems, so the plan
# excludes it from the smoke catalog's flow shops and
# records why
[[solvers]]
id = "random-search" # the baseline every search should beat
# ---------------------------------------------------------------- protocols
# Named stop criteria. Every solver runs under every protocol with the same
# criteria, so each protocol is one fair comparison. Each needs a time,
# evaluation, or iteration limit; the first criterion met ends a run.
[[protocols]]
name = "evaluations-5000"
stop = { max_evaluations = 5000 } # equal numbers of evaluated schedules
[[protocols]]
name = "time-1s"
stop = { time_limit = 1.0, patience = 500 }
# equal wall-clock seconds, and a stop after 500
# iterations without improvement; also available:
# max_iterations, and target (stop once the primary
# objective reaches this value)
# ---------------------------------------------------------------- execution
# How to run. None of these settings changes a result, and none enters the
# config hash, so they may change between sessions of one campaign.
[execution]
backend = "process" # "process": supervised worker processes, with hard
# time limits; "sequential": in the calling process
workers = "auto" # worker processes, or "auto" for the processors
# minus one; must be 1 for the sequential backend
timeout_factor = 1.5 # kill a run timeout_factor * time_limit + grace
# seconds after its solve begins; at least 1
# (default 1.0)
grace = 5.0 # seconds added to the hard limit (default 5.0)
startup_limit = 120.0 # seconds a worker has, from the hand-over of a run,
# to begin its solve; a slower start is a start-up
# failure, which retries cover (default 120.0)
retries = 1 # further attempts after a failure that may be
# transient (default 0)
record_schedules = true # store each run's best schedule in its record; its
# digest is stored either way (default true)
affinity = "inherit" # "inherit": the workers run on every processor the
# process may use; "pin": worker k runs on the k-th
# of them; or a list of processor indices, one per
# worker, such as [2, 4, 6] (default "inherit")
threads_per_worker = 1 # threads the numerical libraries of a worker may
# use (default 1, the only value the sequential
# backend accepts)
on_changed_code = "warn" # a session on other code than the last recorded
# one: "warn" names each difference and records
# it; "refuse" ends the session (default "warn")
on_changed_problem = "rerun"
# a record made on another version of its
# problem: "rerun" runs its run again and keeps
# the record under superseded/; "refuse" ends
# the session (default "rerun")
log_level = "inherit" # what the session log holds beside the engine's
# own lines: "inherit": what the library's loggers
# let through, warnings unless the application
# lowered a level; "info" or "debug": everything
# from that level, which costs a solver that logs
# in its loop time inside its time limit
# (default "inherit")
# ---------------------------------------------------------------- metadata
# Free annotations, kept with the results; not part of the config hash.
[metadata]
purpose = "shows every key of the campaign format"
The reader is strict. An unknown key, a value of the wrong type, or a missing
required key is an error that names the entry, for example
instances[1].seeds[0] must be an integer, got the string '3'. The file must
declare schema_version = 1. Relative paths refer to the file's directory. In
JSON, null stands for an absent optional value.
Top-level keys¶
| Key | Required | Meaning |
|---|---|---|
schema_version |
yes | the file format version, 1 |
id |
yes | the campaign name; results go to <output>/<id>/ |
seed |
yes | the root of every run seed, a non-negative integer |
instances |
yes | one or more instance sources |
solvers |
yes | one or more solvers |
protocols |
yes | one or more stopping protocols |
replicates |
no | runs per stochastic solver, problem, and protocol; default 30 |
replicate_timed |
no | true gives a solver declared deterministic every replicate under a protocol that sets a time limit; default false |
seed_namespace |
no | a name that separates the run seeds of this campaign from those of campaigns with the same seed; default none |
objectives |
no | objectives, primary first, that replace those every problem declares |
execution |
no | how to run, see Execution |
metadata |
no | free annotations kept with the results |
Names that become file names, the campaign id, solver names, protocol names,
and problem ids, start with a letter or digit and hold only letters, digits,
., _, and -. Solver and protocol names must be unique, ignoring case.
Instance sources¶
| Source | Keys | Problems |
|---|---|---|
| family | family, seeds, parameters |
one problem per seed of a registered family |
| suite | suite, instances, directory |
named instances of a published suite; file suites read directory |
| catalog | catalog, seed |
every instance of a built-in catalog |
| directory | directory |
the problems a catalog build or materialize wrote, each checked against its digest |
| file | file, format |
one problem document or instance file |
Every problem id must be unique across the sources. Parameters of a family source are the family's own; the others keep their defaults.
Solvers and protocols¶
A solver entry has an id from the solver registry, optional parameters,
an optional label, and an optional replicates. The label names the entry
in results and tables, so two settings of one solver can compete in one
campaign. Parameters are checked against the solver when the campaign is
planned. replicates, an integer of at least 1, is the number of runs of
this solver per problem and protocol, in place of the count the campaign
would give it (see Planning).
A protocol has a name and a stop table with the fields of StopCriteria:
time_limit, max_evaluations, max_iterations, target, patience, and
schedule. Every protocol must set a time, evaluation, or iteration limit.
schedule ends no run. It states which clock a solver's parameter schedule
follows, such as an exploration rate that falls over the run. Such a solver
asks how far through its budget the run is, and the answer is the largest
spent fraction among the limits the setting counts:
schedule |
Limits counted | Consequence |
|---|---|---|
"deterministic", the default |
the evaluation and iteration limits; the time limit only when it is the only limit | under an evaluation or iteration limit the schedule is the same on every machine and replays exactly, and it stops short of its end when the time limit ends the run first |
"binding" |
every limit that is set | the schedule reaches its end at whichever limit ends the run, and under a time limit its position at a given evaluation depends on the speed and load of the machine, so no run of an iterative solver replays |
With the time limit as the only limit, both settings follow the elapsed time.
A solver that follows a schedule records in the extras of its outcome
schedule_clock, the limit that places the end of the run on the schedule
(evaluations, iterations, or time), and schedule_progress, that place
between 0 and 1. A value below 1 says the schedule did not reach its end.
The config hash¶
The config hash identifies an experiment: the SHA-256 digest of the
canonical JSON of the fields that determine the results, seed, instances,
solvers, protocols, replicates, and objectives, as written, so
reordering a list changes it too. The optional settings replicate_timed,
seed_namespace, a solver's replicates, and a protocol's schedule enter
it only when they are set to other than their default, so a campaign file
that sets none of them has the config hash, the seeds, and the run ids it has
without them. The campaign id only names the directory,
and execution and metadata never change a result, so they stay outside
the hash. The locations of instance files and directories stay outside it
too: a problem read from a file is identified by the digest of its content,
which enters the id of every run on it. A campaign therefore keeps its
identity when its files move, or live elsewhere on another machine.
Planning¶
campaign validate reads the file, loads every problem, checks every solver
id and parameter, and expands the grid problem × solver × protocol ×
replicate into planned runs, without running any. With --runs it lists every
planned run with its id, problem, solver, protocol, replicate, and seed, the
ones campaign run then writes to plan.json; with --json as well, runs
is that list in place of the number of runs.
- Replicates. A stochastic solver runs
replicatestimes per problem and protocol, as replicates 0, 1, and so on. A solver declared deterministic runs once, as replicate 0. Where a time limit ends its run, though, the result depends on the speed and load of the machine, and one run does not show by how much. Withreplicate_timed = true, such a solver runs every replicate under each protocol that sets a time limit, and still once under the others. A solver that ignores the time limit then repeats one result. Without the key, the plan notes the iterative solvers it concerns, and states the others in a conditional note (see Fair comparison). - A solver's own count. A solver entry with
replicatesruns that many times under every protocol, whatever it declares. It is the way to give several runs to a solver that is declared deterministic and is not under its parameters, such as an exact solver with several threads, and to give one run to a solver that ignores the clock in a campaign withreplicate_timed. - Run ids.
run-and the first 16 hexadecimal digits of the SHA-256 of the config hash, the problem id and digest, the solver id, label, and parameters, the protocol name, and the replicate. Equal campaigns give equal run ids on every machine, and a problem whose content changes gets new ones. - Compatibility. A solver is never paired with a problem that has a
feature the solver does not model. The pair is excluded, and the plan
records why.
--check-compatibilitylists every exclusion and every caveat, such as a single-objective solver on a problem with several objectives, which optimises the primary objective only.
Seeds and paired comparisons¶
Seeds are common random numbers. Replicate r of problem p receives
under every solver and every protocol. The results are therefore paired: the first replicate of every solver on a problem shares one seed, the second another, and so on. Adding, removing, or reordering solvers never changes a seed, so a campaign can grow a solver without rerunning the others.
Two campaigns with the same seed give the same seed to a problem they
share. To keep their random streams apart, as for a tuning study and the
comparison that follows it, give each a seed_namespace. The root of the
derivation is then
in place of seed, so two campaigns that differ only in namespace share no
run seed, and each stays paired within itself. The namespace is part of the
config hash. Without one, every seed is the one above.
Fair comparison¶
Solvers are compared fairly when, under each protocol, every one of them runs
with identical stop criteria, on the same problems, with the same seeds. The
planner builds runs that way, and the plan is then checked rather than assumed
to be fair (Fairness, recorded in plan.json):
- every protocol bounds its runs with a time, evaluation, or iteration limit;
- every run refers to one of the plan's protocols, so all runs under a protocol share its stop criteria;
- every solver has its full set of replicates under every protocol on every problem it is not excluded from, and no run is planned twice;
- all runs of one problem and replicate share one seed.
Each protocol states the budgets it equalises, its parities, read off its
stop criteria: equal-wall-clock (time_limit), equal-evaluations
(max_evaluations), equal-iterations (max_iterations), and fixed-target
(target). Equal evaluations is the budget to prefer for comparing
algorithms: it does not depend on the machine, and runs stopped by it replay
exactly. Wall-clock budgets depend on the speed and load of the machine, and
runs they stop cannot be replayed; neither can any run of an iterative
solver under a protocol whose only bound is its time limit, or whose
schedule is "binding" beside a time limit. An iteration means a different
amount of work in different solvers.
The assessment also carries notes: facts that break none of these rules
and that a reader of the comparison needs. The planner logs each as a
warning, campaign validate prints them, plan.json stores them, and the
analysis reports those of the analysed protocol as limitations. Both concern
protocols with a time limit: solvers whose threads differ (see
Execution), and iterative solvers declared deterministic that
run once although the campaign has more replicates, unless the solver entry
states its own replicates.
A solver declared deterministic that is not iterative ends when its algorithm
does. A constructive rule such as neh never reads the clock, and an exact
solver such as milp can be ended by its time limit; the plan cannot tell
the two apart. Such a solver that runs once under a time limit is therefore a
conditional note (conditional_notes in plan.json): it holds only if
the time limit ends one of the solver's runs. The planner logs it as
information, campaign validate prints it beside the notes, and the analysis
reports it as a limitation only for the solvers whose completed runs the time
limit ended.
A fair plan does not make the runs comparable by itself. Under one protocol, a
solver that stops for lack of improvement spends less than the budget another
uses up to its time limit, and a solver that honours no deadline spends more.
dispatchatlas.campaigns.fairness.assess_outcomes(plan, records) compares the
records of the completed runs with the plan, protocol by protocol: how many
runs of each iterative solver ended for each stop reason, whether those
reasons diverge, which solvers' runs took more wall time than the time limit
allows, and which solvers of a conditional note had a run that the time limit
ended. A run counts as over its time limit by the rule of the engine's note
on a run: its wall time passes the limit by more than the tolerance, 5 % of
the limit by default, and by more than 0.05 seconds. A divergence qualifies a
comparison and does not refuse it, since a reached target or a proof of
optimality is a legitimate end. The analysis reports these as limitations
(see Analyzing campaigns).
Estimating¶
campaign estimate counts the runs, by protocol and by solver, and bounds how
long their solves can take. The solve of a run whose protocol sets
time_limit takes at most its hard limit, and every attempt may take that
long; the time a worker needs to begin a solve is not counted. A run bounded
only by evaluations or iterations has no duration bound, so those runs are
counted apart. With --output, runs already completed there are left out.
Execution¶
The [execution] table, or the options of campaign run, choose how to run.
None of these settings changes a result, so they may differ between sessions
of one campaign.
- Backends.
process, the default, runs each attempt in one ofworkersworker processes, started with thespawnmethod so that every platform behaves alike, and supervised by the calling process.sequentialruns one attempt after another in the calling process; it suits debugging and solvers that cannot be sent to another process, but a solver that ignores its time limit, or a crash in native code, stops the campaign. A Python script that runs a campaign with the process backend must do so underif __name__ == "__main__":, since every worker imports the main module. - Processors. The engine counts the processors its process is allowed to
use, not those of the machine: the scheduler affinity on Linux, which
tasksetand the cpuset of a container set, and the process affinity mask on Windows. A limit on processor time is not an affinity: under the CPU quota of a container (docker run --cpus, a Kubernetes CPU limit) every processor of the host is still counted, so setworkersby hand there. Where the set cannot be read, on macOS and on a Windows machine whose processors form several processor groups, every processor of the machine is counted, the session recordscpu.allowedas null, and a campaign with a time limit logs a warning.workers = "auto"uses the count minus one.affinitychooses where the workers run."inherit", the default, leaves every worker on all the allowed processors."pin"confines workerkto thek-th allowed processor, in index order. It does not look at cores: on a machine with simultaneous multithreading that numbers the two hardware threads of a core side by side, two pinned workers share one core. A list such asaffinity = [2, 4, 6]confines workerkto itsk-th entry, so name one logical processor of each core there. A pinned worker is created on its processor, where every thread it starts also runs, and it reports its processors to the supervisor, which stops the campaign when they are not the one it was given. A pinned campaign with runs to attempt is refused, before it discards a record or creates its directory, when the workers outnumber the processors, when a listed processor is not an allowed one, and on a system that cannot pin a process: macOS, and a Windows machine whose processors form several processor groups. The sequential backend accepts only"inherit". - Threads. Every worker process starts with the thread pools of
numerical libraries limited to
threads_per_workerthreads, 1 by default. The engine setsOMP_NUM_THREADS,MKL_NUM_THREADS,OPENBLAS_NUM_THREADS,BLIS_NUM_THREADS,VECLIB_MAXIMUM_THREADS, andNUMEXPR_NUM_THREADSin the environment each worker is created with, so the limit holds for any number of workers, whatever the launching environment sets, and also when the launching script imports NumPy first. The sequential backend has no worker processes and leaves the environment of the calling process alone, so it accepts only the defaultthreads_per_worker; under a time limit the runner names every one of these variables that is unset, since a numerical library may then use every processor, and they take effect when set before Python starts. A solver's ownthreadsparameter is separate. When a protocol sets a time limit and the workers, each with the larger ofthreads_per_workerand the largestthreadsamong the solvers, need more processors than are allowed, the runner warns that the budgets differ between runs. A pinned worker has one processor, so the runner warns whenever that thread count is above 1. The plan records thethreadsevery solver holds in its solver entry, the solver's default when the campaign sets none. When a protocol with a time limit compares solvers whosethreadsdiffer, the plan keeps a note of it in its fairness assessment and stays fair. - Hard time limits. Stop criteria are cooperative, so the process
backend gives every run whose protocol sets
time_limita hard limit oftimeout_factor * time_limit + graceseconds, counted from the instant its worker states as the start of the solve. What comes before that instant is not charged to the run: the start of the worker's interpreter and the loading of plugins, for a worker that started for this run, the rebuilding of the solver and the problem, and the warm-up of the solver. The provenance of the attempt records it aswaited_seconds, counted from the hand-over, before a fresh worker exists. When the hard limit passes, the supervisor kills the worker and records a timeout, and a fresh worker takes the next run. The supervisor never waits for a worker to start: a starting worker gets its run when it reports that it is ready, so the limits of the other runs are enforced meanwhile. - One clock per run. The engine fixes the origin of a run's clock at the
start of each attempt, and every solver counts its elapsed time and its
time limit from that origin. It is one instant: the wall time of the
attempt counts from it, and under the process backend it is the instant
the worker states as the start of the solve, from which the hard limit
counts. The
elapsedof an outcome and the times of its trace are therefore seconds since the start of the attempt, for every solver alike: they include what a solver does in every run before its search starts, such as building its evaluator or its model, and the time limit covers that work too. A solver called directly, outside a campaign, counts from the instant it creates its clock, asdispatchatlas.core.clockdescribes; a solver from a plugin that keeps a clock of its own starts it atclock_start()to follow the same origin. The provenance of an attempt states that its solver was timed this way (shared_clock). - What a solver does once in a process. The first call of a solver in a
process may pay what no later call pays, such as the import of a backend.
Inside the time limit, that cost would shorten the search of the first run
each worker executes, and of the first run of every worker that replaces
a killed one, so replicates of one solver would search for unequal times
according to the order of the runs and the number of workers. The engine
therefore warms up the solver before the start of each attempt: it loads
NumPy's random module and calls the solver's
warm_upmethod, whichcp-sat,milp, andrestricted-neighbourhood-searchuse to import their backend. The warm-up is outside the time limit and the wall time of every run; under the process backend it counts toward the start-up limit and is part ofwaited_seconds. A warm-up that raises fails the attempt as anexception. A solver from a plugin that pays such a cost insidesolve, without awarm_upmethod, is charged for it in the first run of each process; see Extending. - What the time limit cannot stop. A solver checks its time limit between evaluations and never before its first one, so a run passes the limit by the evaluation in progress and by any work between two checks, and a heuristic that builds one schedule, such as a priority rule or a mapping heuristic, checks no limit at all. The sequential backend has no hard limit, and under the process backend a run that ends late but before its hard limit is an ordinary success. The engine therefore compares the wall time of every completed attempt with the time limit and keeps a note when it passes the limit by more than 5 percent and by more than 0.05 seconds, and the analysis reports these runs.
- Start-up limit. From the hand-over of a run, its worker has
startup_limitseconds, 120 by default, to begin the solve, whatever the size of the problem it is sent. A worker that takes longer, or never starts, is killed, and the attempt is a start-up failure (startup): a kind of its own, recoverable, and no timeout of the solver. The limit applies to every run of the process backend, with or without a time limit. Withretries = 0a start-up failure settles its run like any other failure;--retry-failedgives the run another attempt, and the analysis counts these runs apart. - What a kill reaches. A kill ends the worker and every process the solver started: on POSIX systems through the worker's process group, on Windows through a job object that the supervisor creates for each worker. Where Windows refuses the job object, for example on a version without nested jobs when the supervisor is in a job itself, a warning says so and a kill reaches the worker alone. A worker also ends itself when its supervising process is gone, so no worker of a killed session computes beside a resumed one. A worker inside a native extension that keeps the interpreter lock ends when the extension returns; on Windows the job object ends it at once.
- What a killed run keeps. While a run with a hard limit executes, its
worker forwards the incumbents of a solver that runs inside
Search: every objective value, the feasibility, and the evaluations, iteration, and elapsed time at which the search found the incumbent. A timeout failure holds two of them.last_incumbentis the last one the run reported; a solver that searches past its time limit may have found it after the limit.limit_incumbentis what the run held when its time limit passed: the last one found withintime_limitseconds of the start of the solve. The two are equal when the search did not improve after the limit. The values are the solver's own: no schedule comes with them, so they are not checked as the result of a completed run is. An incumbent leaves the worker at once, from the thread of the search, when at least 0.25 seconds have passed since the last report, so it is kept whatever the solver does next. One found sooner after a report waits for the rest of the 0.25 seconds, and is missing from the record if the worker is killed before then, or if the solver, right after finding it, enters a call that keeps the interpreter lock until the kill. The reports of a run end with its solve: one that falls due as the search returns is not sent, and the result of the run follows at once. Reporting costs the search about half a microsecond per improvement and, at most once every 0.25 seconds, one computation of the incumbent's objective values and one message on the pipe, however often the solver improves; the measured run time of a search whose every evaluation improves is the same with and without it, within 2 percent. A solver that does not run insideSearchreports nothing, and its timeout failure holds no incumbent. The analysis scores a killed run as the worst value unless its settingtimeout_scoreasks for the incumbent, and then bylimit_incumbentalone. - Retries. A failure is terminal when another attempt would fail the
same way: a typed error of the library, such as an invalid configuration or
a missing optional dependency, or a hard timeout. Any other exception, a
worker that dies, and a worker that does not begin the solve within the
start-up limit are recoverable. A run gets up to
1 + retriesattempts, until a terminal failure or the last attempt settles it. - What each attempt checks. Every schedule a solver returns is checked
with
check_schedule, and every objective is computed again from the schedule, so a record never relies on the solver's own claims. The measurements cover the solve alone: wall time, CPU time, and on Linux the peak resident memory. Each attempt also records the processor its worker was pinned to, how many runs were executing when it was handed to its worker, itself included, and how long it then waited for its solve to begin. - Notes. What the engine notices about a completed attempt is logged as
a warning and stored in the run record, in
notes; the run stays an ordinary completed run. A note is kept when the solver reports an objective value its schedule does not give, calls an infeasible schedule feasible, or returns a malformed schedule; when the wall time passes the time limit by more than 5 percent of the limit and by more than 0.05 seconds, which the scheduler of a machine can add to any run; when the solver stops for its time limit while the wall time is below the limit by more than 5 percent of it and by more than 0.05 seconds, as a solver does that gives a search a share of its limit counted from the start of the attempt; when the elapsed time the solver reports differs from the wall time the engine measured by more than 0.05 seconds and by more than 5 percent of that wall time; when the trace holds a negative time, times that decrease, or a time more than a microsecond beyond the elapsed time the solver reports; when the evaluation counts of the trace decrease; when the last value of the trace differs from the value of the outcome; and when an iterative solver returns a schedule and no trace. A result whose elapsed time, or whose trace holds a time or a value, that is not a finite number cannot be compared or stored: the attempt fails with anexception, and the session continues.
Storage layout¶
<output>/<campaign id>/
campaign.json the campaign document, its base directory, and its config hash
plan.json the plan, with the parities, the fairness assessment, and its digest
sessions/<n>.json one record per invocation of the runner, with its environment
and the runs that left or entered the plan
runs/<solver>/<problem>/r<replicate:03d>-<protocol>.json
failures/<solver>/<problem>/r<replicate:03d>-<protocol>.attempt-<n>.json
superseded/runs/<solver>/<problem>/r<replicate:03d>-<protocol>.<run id>.json
superseded/failures/<solver>/<problem>/r<replicate:03d>-<protocol>.attempt-<n>.<run id>.json
files moved aside when new records took their places
logs/session-<n>.log what each session logged, with UTC time stamps
.lock held while a runner works on the campaign
Every document is written atomically, with sorted keys, two-space indentation, ASCII text, and a final newline.
- A run record (
dispatchatlas.run/1) names the run, its campaign and config hash, problem id and digest, solver, protocol, replicate, and seed; holds the outcome, with the status, stop reason, objective values, bound, feasibility and violation codes as checked, evaluations, iterations, elapsed time, trace, and front; the best schedule as[task, mode, start, end]rows, unlessrecord_schedulesis false; and the provenance of the attempt: session, attempt number, worker, times, CPU seconds, peak memory, hard limit, the processor the worker was pinned to (cpu, null when it was not pinned), the runs executing when the run was handed to its worker (concurrent), and the seconds from that hand-over to the start of the solve (waited_seconds, null for the sequential backend), which hold the start of a worker that started for the run and the warm-up of the solver, and whether the solver's clocks started at the instant the wall time counts from (shared_clock, true for an attempt whose solver returned or raised, null for one the engine ended or that never reached its solver). A record that lacks one of the four reads as null for it. The CPU seconds are those of the executing process, with the threads the solver started, and on POSIX systems those of the child processes it started and waited for; on Windows the time of a child process, such as a backend that runs as a program of its own, is not in them. The record also holds thenotesof the engine on the attempt, a list of sentences that is empty when the engine noticed nothing; a record without the field reads as one with no notes. Theelapsedof the outcome is the solver's report, in seconds since the start of the attempt, and thewall_secondsof the provenance is the engine's own measurement from the same instant until the solver returned. - Its result hash is the SHA-256 of the outcome without the fields that measure wall time, and without the solver-specific extras. Nothing about the machine or the session enters it, so a replay on any machine compares results directly, and a record edited after it was written fails the comparison with its hash.
- Its identity hash (
identity_hash) is the SHA-256 of what names the run: the digest of the problem, the id and the parameters of the solver, the protocol with its stop criteria, the replicate, and the seed. A record whose seed, or another of these fields, was edited fails the comparison as a record with an edited outcome does:read_runsrefuses it, and a resume keeps it aside and runs its run again. The times an attempt measured, the solver-specific extras, and the provenance are under neither hash. A record without the field is not checked for it. - A record is also compared with what the plan states for its run: the
id and the digest of the problem, the name, the id, and the parameters of
the solver, the protocol, the replicate, and the seed. The comparison
needs no hash, so it holds for a record that states no identity hash as
well.
read_runsrefuses a record that states another value, and a resume runs its run again. A seed that the run had in an earlier plan of the directory is accepted, as the session records name it inplan_reseeded; when a session record cannot be read, the earlier seeds are not known and the seed is not compared. - A failure record (
dispatchatlas.failure/1) holds the error, its kind (exception,timeout,startup,worker-exit, ortransfer), and whether it is terminal or recoverable. It says whether the solve had begun when the attempt ended (solve_started), and the record of a run killed at its hard limit holds the last incumbent its search had reported (last_incumbent, null when it had reported none) and the last one it had found within its time limit (limit_incumbent, null when it had found none by then). A record that lacks one of the three fields reads as null for it. Thewall_secondsof an attempt that the engine ended count from the start of the solve, or from the hand-over when the solve never began. - A session record (
dispatchatlas.session/1) holds the execution settings, the counts, and the environment: Python, the operating system and machine, the processor and memory, the indices of the processors the session was allowed to use (cpu.allowed, null when the system does not tell, in which casecpu.availablecounts the machine), the value of each thread variable in the launching environment (threads, null when unset), the versions of DispatchAtlas, NumPy, and the installed backends, and what identifies the code.librarystates where the DispatchAtlas distribution was installed from (origin:vcs,archive,directory, orindex), whether it iseditable, itsrevision, andsource_digest, a SHA-256 over the files of the package. The revision is the commit of a checkout, the commit of the version-control URL an installed copy came from, or the hash of an archive; it is null for a copy built from a local directory, whose record does not say which state was built, and the source digest identifies that copy. When the package that runs is not the one the installed distribution provides, such as a copy that comes first on the module search path, the installer's record does not describe it:originispath, the version is null, and the revision is that of the copy's checkout or null.pluginsstates the same facts for every distribution that provides a solver or benchmark entry point.gitstates, for the source and for the working directory when either is a checkout, thecommit, whether the checkout isdirty(a tracked file differs from the commit, or there is an untracked file that git does not ignore), the number ofuntrackedfiles, and adigestof the uncommitted changes, so two different sets of edits of one commit have different digests; an editable plugin states the same for its checkout. The directory of the campaign is left out of every one of these states, so results kept inside a checkout are no change of its code. Untracked files enter the digest by their content up to 16 MiB in all, in path order; a file past that total enters by its size and modification time, and more than 1000 untracked files enter by their paths only. The queries do not write to the checkout: git's index stays as it is. The processor is recorded by its model name, with itssocketsand physicalcoreswhere the system gives them. The record holds no host or user name. Underloadit holds two samples of the machine, one taken before the first run (start) and one after the last (end, for a session that finished): the share of the processor time of the whole machine that was busy over a tenth of a second (busy), and the one-minute load average where the system has one. A session with nothing to attempt takes no sample. Underchangesit holds how the session differs from the session before it (see Resuming). It also holdsplan_digest, the digest of the run ids and seeds of the plan the session executed, whichplan.jsonstates asruns_digest, andplan_left,plan_entered,plan_reseeded, andplan_solvers: how that plan differs from the one the directory held before under the same config hash (see Resuming). A record without these fields reads as one that states no digest, no change, and no load.
read_campaign, read_plan, read_runs, read_failures, and
read_sessions read these files back as typed records. read_runs returns
the records of the plan, and warns about every other file under runs/:
records of runs the plan does not hold, files that cannot be read (one that
is not UTF-8 text or not JSON, one that is no run record, one the system
does not let read), and files that are no records. read_strays returns
those files. No session replaces or removes them.
The log file of a session holds one record per line, with its UTC time to the millisecond, its level, and its logger:
- what the engine itself logs while the session runs, from the level
INFO: the start of the session, its progress, its warnings, and its summary; - what the other loggers of the library let through, among them those of
solvers and plugins that log through
dispatchatlas.core.logs.get_logger:WARNINGand higher unless the application lowered a level. A worker process starts with the default level, so it sendsWARNINGand higher, each line marked with the name of the worker.
The execution key log_level asks for more: "info" or "debug" lowers
the library's loggers to that level for the session, in the calling process
and in every worker; the default "inherit" leaves them as they are. The
file is written beside whatever the application shows. Under every setting
the lines on standard error are the same with and without it, in the calling
process and in the workers, for the library's loggers and for the
application's own. Only a handler that is attached while a session runs is
not held to that: it receives what the lowered loggers record.
Logging inside a run costs the run time. A record below the level of its
logger is not made, and the call that would log it costs next to nothing,
so a solver that logs details in its loop is not slowed under the default
setting. A record that is made costs its making, its formatting, and a
buffered write or, in a worker, a place in a queue: tens of microseconds
each. A solver that logs every iteration of a short loop therefore does a
fraction of its work in the same time limit once log_level lets those
records through, and its results under a time limit are not comparable
with those of a session that did not. The thread that logs in a worker
never waits for the calling process: the queue holds 10,000 records, a
record beyond them is dropped, and the log states how many were. The log
file is buffered, so the lines of a burst below WARNING reach the disk
with the next warning, with the next record after a second, or when the
session ends. It grows to about 64 MiB (2^26 characters) at most: later
records are counted, and its last line states how many were left out.
Resuming¶
Running campaign run again with the same file and output resumes the
campaign. A record already written is kept when it belongs to the same config
hash, still matches its result hash and its identity hash, and states what
the plan states for its run, and only the runs that are neither
completed nor settled by failures are attempted. An interrupted session
therefore loses at most the runs that were in progress.
--retry-failedgives every run that failures settled one more attempt.--max-runs Nattempts at mostNruns in this session.--workers,--backend, and--affinityoverride the file's execution settings; they do not change the config hash.--affinitytakesinherit,pin, or processor indices such as0,2,4.--backend sequentialimplies--affinity inheritand the defaultthreads_per_worker.campaign replaytakes the same three options, so a campaign pinned on one machine resumes and replays on another. A campaign with nothing left to attempt is not refused for its pinning.- A session compares its execution settings and its environment with those
of the last session the directory recorded. It logs one warning that
names each difference with both values, such as
execution.workers 1 -> 2orenvironment.library.source_digest, and its record stores all of them inchanges, each withfield,earlier,now, andkind:execution,code(the Python version, the packages, the library, the plugins, and the git states), ormachine(the rest of the environment). A value the earlier record does not state is not compared. With the execution keyon_changed_code = "refuse", a session whose code differs is refused before it writes anything; the default"warn"resumes. A difference of execution settings or of the machine is never refused. The directory of the campaign itself is left out of every git state, so it may lie inside the working directory, the library's checkout, or the checkout of an editable plugin. Keep other files that change between sessions, such as the results of other campaigns, out of those states, by an ignore rule or a place outside the checkouts. - A worker process that starts after a source file of the library changed runs on other code than the session recorded. The runner warns about such a worker, and every run it completes carries a note with both source digests.
- A file at the path of a planned run that is not the valid record of that
run does not count, and is never replaced. This holds for a record made
on another version of its problem, whose content changed under the same
id, for a record that no longer matches one of its hashes or the plan,
and for a file that cannot be read. The session names such files in a
warning when it starts and runs their runs again. Each file stays where
it is until its run is repeated, and moves to
superseded/when the repeated run gets its first record, of its result or of a failed attempt: underruns/and the same solver and problem folders, with the run id it states before.json. A session that repeats no run, because of--max-runsor an interruption, moves nothing, and when the problem is what it was again, its records count again without a rerun. The record of a failed attempt on another version of a problem is kept the same way, undersuperseded/failures/, when a new attempt takes its path. - A record under
superseded/that is the valid record of a planned run again, because the problem was restored after some of its runs were repeated, moves back to its path when a session starts, and the record that stood there moves aside in its place. Nothing else readssuperseded/, and nothing replaces a file in it.campaign estimatecounts such a run as complete and moves nothing. - A file that cannot be moved aside, for example because another program holds it open, ends the session as failed with an error that names the file. The new record is not written, the file stays, and the next session attempts the run again.
- With the execution key
on_changed_problem = "refuse", a directory that holds a record, or the record of a failed attempt, of another version of a problem is refused before anything moves; the default"rerun"runs the runs again.--freshalso discardssuperseded/andlogs/. - A directory that holds results of another configuration is refused;
--freshdiscards its records first. Give a changed experiment a newidto keep both. - The plan is built again in every session, and it follows more than the
campaign file: the metadata the registered solvers declare, the content of
the problems, and the library that plans. When the new plan differs from
the one the directory holds under the same config hash, the runner warns
with the number of runs that left the plan and that entered it and names
the first of each, and the session record stores all of them in
plan_leftandplan_entered, beside theplan_digestof the new plan. The records of runs that left stay in the directory and are no longer read. The warning also names the runs whose seed changed and the fields of the solver entries that changed, such as a solver'siterativeflag, and the session record stores them inplan_reseeded(each run with itsearlier_seed) andplan_solvers(each withsolver,field,earlier, andnow). The session record is written beforeplan.jsonis replaced, so a session that ends before its record leaves the comparison to the next session. Two sessions with equalplan_digestexecuted the same runs with the same seeds. The digest does not cover the solver entries.
Checking a campaign directory¶
campaign check reads a campaign directory before its results are used and
reports what qualifies them. It writes nothing and runs nothing.
dispatchatlas campaign check results/<campaign id>
dispatchatlas campaign check results/<campaign id> --replicate-floor 10 --json
The readers of results work from plan.json and pass over what does not fit
it, so a directory can look complete and not be. The check asks what they do
not:
- The campaign file. Whether
campaign.jsonis there and still matches the config hash recorded with it. - The plan. Whether
plan.jsonlists the problem, the solver, and the protocol of each of its runs and names nothing twice; a plan that does not ends the check. Whether it has the campaign's config hash, and whether it is the plan that the stored campaign gives when it is planned again on this installation. Every field is compared, and each difference names its field: the campaign id, the seed and its namespace, the replicates, each field of each problem, solver, and protocol, the problem, solver, protocol, replicate, and seed of each run, the order of each list, the excluded pairs with their features, and the caveats. Whether it holds the runs and seeds the last session executed (plan_digest), and whether the plan changed between sessions. The entries that the library derives from a plan and writes for readers of the file, such asruns_digest, theparitiesof a protocol,fairness, and thereasonof an exclusion, are compared with what the file's own plan gives. The library derives them again wherever it reads a plan, so a difference there, and another wording of a caveat, is a note. - The planned runs. Every run needs a valid record or failures that
settle it. A run with neither was never attempted, and one whose failures
the campaign's
retriesdo not settle is owed another attempt: both block. Runs settled by failures are counted by solver and by the kind of error of their last attempt. A cell, one solver on one problem under one protocol, that has records for some of its planned runs only is named, and so is a solver without any record. The pairs of problem and solver that the plan excludes have no run and are not missing: they are listed apart, and a solver excluded from every problem is named. - The files. A file at a planned run's path that is no valid record of
the run: one that cannot be read, a record of another configuration or of
another version of its problem, a record that fails its result hash, its
schedule digest, or its identity hash, and a record that states another
problem, solver, protocol, replicate, or seed than the plan. A valid record
that lies under
superseded/. Records of runs the plan does not hold, files that cannot be read, and files that are no records. - The records. A trace that cannot be read against time, because it holds a negative time, times or evaluation counts that decrease, or a time beyond the elapsed time of its run; a trace whose last value differs from the value of the outcome; and a record of an iterative solver with a schedule and no trace. The check derives these facts from each record, so it covers records that hold no notes of the engine.
- The budgets. Under each protocol, the solvers whose completed runs
took more wall time than the time limit and the tolerance allow, with the
largest ratio (
--overrun-tolerance, by default the margin of the engine's note on a run); hard limits that differ among the attempts; and stop reasons that differ among the runs of the iterative solvers. - The sessions. The sessions that executed the attempts the directory
counts are compared with each other. Each field that differs is named
once, with its values and the sessions that have each. Differences in the
Python version, the packages, the library, or a git state, in the plugins,
in the execution settings, and in the machine are four findings. A field
is compared among the sessions whose records state it, and a note names
the sessions whose records state less than another's
(
sessions-fields-unstated), such as a record without the library, the plugins, the thread settings, or the allowed processors. A difference in the machine is a note, except for what a time limit is measured on: when the sessions that executed the attempts of a protocol with a time limit differ in the processor, the processors they were allowed, or the thread settings, that is a limitation of the protocol (sessions-differ-processors). A session that executed nothing that counts is not compared. An attempt of a session without a record is reported. - The replication. With
--replicate-floor N, the cells of replicated solvers that hold fewer thanNcompleted replicates. A seed that serves two replicates of one problem blocks, and a replicate of a problem that ran with two seeds, as after a plan gave its runs other seeds, is a limitation.
Every finding has a code, a severity, a message, the items it concerns, and, when it concerns the runs of one protocol, that protocol. The severities are:
- blocking: the directory does not hold what it claims, so its results are not ready to be used;
- limitation: the results stand, with a caveat that a report of them has to state;
- note: a fact that qualifies nothing, such as an optional field that
the records of the directory do not state. Such a directory is checked as
far as its files allow: records without an
identity_hashare compared with the plan field by field, and without aplan_digestin the session records the plan file is not compared with the last session.
The exit code is 0 when no finding is blocking, 1 when a finding is blocking
or the path is no directory, and 2 for a command line that is not valid,
such as a negative --overrun-tolerance, which is refused before anything
is read. The text form names the first six items of a finding; --json
prints the whole report, with ok, the planned runs by what the directory
holds for them (counts), the number of findings of each severity
(severities), and every finding with all of its items. The counts are
empty when the check ended at the plan or at a file that could not be read.
Planning the campaign again loads every problem, so it needs the campaign's
instance files and solvers; --no-replan leaves that comparison out and
says so in a note.
A directory can be checked while a session of the campaign writes into it.
The report is then of that moment: a run that is executing counts as
unattempted, the temporary file of a record being written can show as
files-stray, and a file that the operating system does not let read while
it is written ends the check with the blocking finding files-inaccessible,
which names the file. Check the directory again once the session has ended.
| Code | Severity | Meaning |
|---|---|---|
recipe-missing |
blocking | campaign.json is missing. |
recipe-unreadable |
blocking | campaign.json cannot be read as a campaign. |
recipe-altered |
blocking | campaign.json no longer matches its recorded config hash. |
plan-unreadable |
blocking | plan.json is missing or cannot be read. |
plan-inconsistent |
blocking | plan.json holds a run of a problem, solver, or protocol it does not list, or an id twice. |
files-inaccessible |
blocking | The operating system does not let read a file of the directory, so its runs, files, records, and sessions were not checked. |
plan-foreign |
blocking | plan.json states another config hash than campaign.json. |
plan-differs |
blocking | plan.json is not the plan the stored campaign gives. |
plan-digest-differs |
blocking | plan.json holds other runs or seeds than the plan the last session executed. |
runs-unattempted |
blocking | Planned runs have no record and no failed attempt. |
runs-unsettled |
blocking | Planned runs failed and are owed another attempt, which the campaign's retries allow. |
record-unreadable |
blocking | The file at a planned run's path is no run record. |
record-foreign |
blocking | The record at a planned run's path belongs to another configuration or to another version of its problem. |
record-hash-mismatch |
blocking | A record fails its result hash, its schedule digest, or its identity hash. |
record-differs-from-plan |
blocking | A record states another problem, solver, protocol, replicate, or seed than the plan. |
records-set-aside |
blocking | The valid record of a planned run lies under superseded/ and not at its path. |
seed-reused |
blocking | One seed serves two replicates of one problem. |
plan-not-derived |
limitation | The plan could not be derived again from the stored campaign on this installation. |
plan-changed |
limitation | The plan changed between sessions while the campaign file stayed the same. |
runs-failed |
limitation | Runs never completed and are settled by their failures. |
cells-partial |
limitation | A solver has records for some of the runs planned on a problem, not for all. |
solver-without-records |
limitation | A solver with planned runs has no record. |
solver-excluded-everywhere |
limitation | The plan excludes a solver from every problem, so it has no run. |
records-unplanned |
limitation | Records under runs/ belong to runs the plan does not hold; no reader returns them. |
files-unreadable |
limitation | Files under runs/, at no path of a planned run, cannot be read as run records. |
failures-unreadable |
limitation | Files under failures/ cannot be read as records of failed attempts. |
trace-time-axis |
limitation | Traces hold a negative time, times or evaluation counts that decrease, or a time beyond the elapsed time of their run. |
trace-contradicts-outcome |
limitation | The last value of a trace differs from the value of its outcome. |
record-without-trace |
limitation | Records of an iterative solver hold a schedule and no trace. |
budget-overrun |
limitation | Completed runs took more wall time than their time limit and the tolerance allow. |
hard-limits-differ |
limitation | The attempts under one protocol ran under different hard limits. |
stop-reasons-diverge |
limitation | The runs of the iterative solvers under one protocol ended for different reasons. |
sessions-unreadable |
limitation | A session record cannot be read. |
sessions-missing |
limitation | Attempts name sessions the directory holds no record of. |
sessions-differ-code |
limitation | The sessions that executed the counted attempts differ in the Python version, the packages, the library, or a git state. |
sessions-differ-plugins |
limitation | The sessions that executed the counted attempts differ in their plugins. |
sessions-differ-execution |
limitation | The sessions that executed the counted attempts differ in their execution settings. |
sessions-differ-processors |
limitation | The sessions that executed the counted attempts of a protocol with a time limit differ in the processor, the processors they were allowed, or the thread settings. |
replicates-below-floor |
limitation | Cells of replicated solvers hold fewer completed replicates than the floor. |
seeds-unpaired |
limitation | One replicate of a problem ran with different seeds, so its runs are not paired. |
plan-not-compared |
note | The plan was not derived again from the stored campaign, as the caller asked. |
plan-digest-unstated |
note | No session record states the digest of its plan. |
plan-entries-differ |
note | plan.json holds entries the library does not write for its plan: entries derived from the plan, or texts, that were changed or that another version wrote. |
pairs-excluded |
note | The plan excludes pairs of problem and solver. |
files-stray |
note | Files under runs/ are no JSON documents. |
failures-foreign |
note | Files at the paths of failed attempts belong to another plan and count as no attempt. |
records-without-identity-hash |
note | Records state no identity hash; they are compared with the plan field by field. |
records-noted |
note | Records carry notes of the engine that the check does not derive from the record, such as a claim of the solver that the checks of its result contradict. |
clock-disagreement |
note | A solver's elapsed time differs from the engine's wall time, or a run stopped for its time limit before the limit. |
sessions-differ-machine |
note | The sessions that executed the counted attempts differ in their machine. What of it a time limit is measured on is the limitation sessions-differ-processors. |
sessions-fields-unstated |
note | Session records do not state a part of the environment that another states, so that part is compared among the other sessions only. |
session-unfinished |
note | The last session is not recorded as finished. |
dispatchatlas analyze and load_dataset run the same check, without
planning the campaign again, and the analysis states the blocking findings
and the limitations for the analysed protocol among its own limitations. It
leaves out the five that it states itself, over the solvers and problems it
analyses: runs-unattempted, runs-failed, stop-reasons-diverge,
budget-overrun, and replicates-below-floor. It carries cells-partial
without the cells whose missing runs it names, and solver-without-records
without the solvers whose every run is among the failures it counts.
Replaying¶
campaign replay runs completed runs again from their recorded seeds, with
the same solver parameters and stop criteria, and compares result hashes,
without writing anything. It first checks every record against its own result
hash, so a record edited after it was written is caught even when its run
reproduces. Each run gets a verdict: match, mismatch with the outcome
fields that differ, skipped, or failed. Runs stopped by their wall-clock
limit are skipped, because how far they got depends on the machine. So is
every run of an iterative solver under a protocol whose only bound is its
time limit, whatever stopped it: without an evaluation or iteration limit a
search places itself on its schedule by the elapsed time, so a run that
stopped at a target or for lack of improvement also depended on the
machine. The same holds under any time limit when the protocol's schedule
is "binding". A solver that is not iterative, such as a priority rule, follows
no schedule, and its completed run is replayed under any protocol. --run
replays selected runs only, and the exit code is 1 when a run does not
reproduce.
From Python¶
Every command has a function behind it:
from dispatchatlas.campaigns import (
check_campaign,
estimate,
load_campaign,
plan,
read_runs,
replay,
run,
)
def main() -> None:
campaign = load_campaign("campaign.toml")
checked = plan(campaign) # what `campaign validate` shows
print(estimate(campaign).remaining) # what `campaign estimate` shows
summary = run(campaign, "results") # resumes when called again
findings = check_campaign(summary.directory) # what `campaign check` shows
for finding in findings.of("blocking"):
print(finding.code, finding.message, finding.items)
records = read_runs(summary.directory)
report = replay(campaign, "results")
print(report.ok)
if __name__ == "__main__":
main()
A campaign can also be built in Python from Campaign, FamilySource,
SuiteSource, SolverSpec, Protocol, and Execution, as the example
examples/campaign_analysis.py does, and written to a file with
campaign_to_dict.