Skip to content

Processes

Pinned child processes and the tools around them: the runner of calls, the child side of a call, the rows file, processor sets, process groups, process snapshots, memory, detached launch, and watching a run. The guide explains how they fit together.

dispatchatlas.processes

The operating system's view of a run: pinned child processes and the tools around them.

  • calls: run_calls(), which runs caller-defined functions, each once in a child process of its own on a processor of its own, and appends the rows they return to a rows file.
  • child: the child side of a call.
  • rows: the rows file, its writer, its strict reader, resume by keys, and the repair of a torn last line.
  • affinity: processor sets as text and masks, and a process pinning itself with a read-back.
  • groups: kill-on-close process groups (Windows job objects, POSIX sessions).
  • snapshots: the machine's processes with parent, executable, affinity mask, and start time.
  • memory: peak, available, and total memory.
  • launching: starting a command detached from its shell.
  • watching: a status line per interval about a running run.

The package sits between dispatchatlas.core and dispatchatlas.campaigns, which uses its affinity, group, and memory primitives. Its leaf modules use the standard library only.

Unlike the other packages, this one re-exports its names lazily: a child process started for every call runs python -m dispatchatlas.processes.child, which imports this package first, and importing the runner's dependencies there would add a substantial part of a second to every call. from dispatchatlas.processes import run_calls works as usual.

dispatchatlas.processes.calls

Running caller-defined functions, each once in a pinned child process of its own.

run_calls() runs a list of Call objects. Each call is one function, named "package.module:function", called with keyword arguments in a fresh child process started with the running interpreter (dispatchatlas.processes.child). The mapping it returns becomes one line of a rows file (RowWriter), written by the runner alone.

  • Processors. The run owns a set of processors; worker slot i owns processor cpus[i]. The runner pins itself to the whole set, creates each child while confined to the slot's processor, so the child (and a launcher executable in front of the interpreter) starts there, and the child then pins itself and reads its mask back: a mismatch is a failure, not a row.
  • Groups. Each slot has a kill-on-close process group (ProcessGroup) that the child also joins by name; whatever a call leaves running ends when the call ends, and every child ends when the runner dies.
  • Resume. The rows file is the record of what is done: calls whose key (read back through key_of) has a row are skipped. A torn last line, a key with two rows, a second runner on the same file, and a resume whose stamp, workers, or processors differ from the first invocation's are refused before anything starts.
  • Failures. A call that raises, returns something that is not a row, dies, or passes its time limit is recorded in the errors sidecar (<rows stem>.errors.jsonl), never in the rows file, and is retried on the next invocation unless skip_failed. After max_failures failures the run stops at once and raises CallsStoppedError.
  • Load. Every row records the number of slots, how many were busy when its call started, and the least that were busy during its life; ballast keeps idle slots busy with discarded repeat calls until the last real call ends.

Next to the rows file the runner writes <stem>.errors.jsonl, <stem>.ballast.jsonl (with ballast), <stem>.children.log (the children's standard output and error, tracebacks included), <stem>.runs.jsonl (one line per invocation: its stamp, workers, and processors), <stem>.lock (held while a runner uses the file), and <stem>.report.json.

ERROR_SCHEMA module-attribute

ERROR_SCHEMA = 'dispatchatlas.call-error/1'

The schema of an error record of the errors sidecar.

Call dataclass

Call(key: Key, target: str, arguments: Mapping[str, Any] = dict(), timeout_seconds: float | None = None, label: str | None = None)

One function to call once in a child process.

Attributes:

Name Type Description
key Key

What identifies the call: a string, an integer, or a tuple or list of keys. The row the call returns must give it back through the runner's key_of.

target str

"package.module:function", importable in the child.

arguments Mapping[str, Any]

Keyword arguments of the function; JSON-able.

timeout_seconds float | None

Wall-clock limit of the child, None for the runner's default.

label str | None

Free text for logs.

CallFacts dataclass

CallFacts(key: Any, attempt: int, cpu: int | None, affinity_inherited: int | None, affinity_mask: int | None, pid: int | None, group: str | None, threads: Mapping[str, str | None], peak_rss_bytes: int | None, started_utc: str | None, ended_utc: str | None, wall_seconds: float | None, workers: int, workers_live_start: int, workers_live_min: int, ballast: bool)

What the runner and the child recorded about one call.

Attributes:

Name Type Description
key Any

The call's key.

attempt int

1 for a first attempt, 1 + earlier error records otherwise.

cpu int | None

The slot's processor, None when the run does not pin.

affinity_inherited int | None

The child's mask before it pinned itself.

affinity_mask int | None

The child's mask read back after pinning.

pid int | None

The id of the process that ran the function.

group str | None

How the child joined its group ("job", "job (already)", "session").

threads Mapping[str, str | None]

The thread-limit variables the child saw.

peak_rss_bytes int | None

The child's peak resident set, where readable.

started_utc str | None

When the child began, in UTC.

ended_utc str | None

When the function returned, in UTC.

wall_seconds float | None

Seconds from the child's start to the function's end.

workers int

Worker slots of the run.

workers_live_start int

Busy slots once the runner had filled every slot it could when this call started.

workers_live_min int

The least busy slots during the call's life.

ballast bool

Whether this was a ballast call.

to_dict

to_dict() -> dict[str, Any]

Return the facts as a JSON-able dictionary.

CallError dataclass

CallError(key: Any, attempt: int, type: str, message: str, returncode: int | None, last_line: bool, facts: Mapping[str, Any] | None, cpu: int | None, started_utc: str, ended_utc: str, schema: str = ERROR_SCHEMA)

One failed attempt of a call, a line of the errors sidecar.

Attributes:

Name Type Description
key Any

The call's key.

attempt int

The attempt number.

type str

The error type: the class the function raised, a step of the child (PinningRefused, GroupRefused, TargetNotFound, RowNotMapping, RowNotSerializable, BadDocument), or the runner's (ProcessDied, Timeout, KeyMismatch, RowRefused, StartFailed, AttendFailed); without caller-named tokens.

message str

The message, without file paths or caller-named tokens.

returncode int | None

The child's exit code; negative for a POSIX signal.

last_line bool

Whether the child wrote its reply line.

facts Mapping[str, Any] | None

The call's facts, as far as known.

cpu int | None

The slot's processor.

started_utc str

When the runner started the child.

ended_utc str

When the runner saw it end.

schema str

to_dict

to_dict() -> dict[str, Any]

Return the record as a JSON-able dictionary.

from_dict classmethod

from_dict(document: Mapping[str, Any]) -> CallError

Read a record written by to_dict().

CallReport dataclass

CallReport(rows_written: int, done_before: int, skipped_failed: int, failures: int, ballast_runs: int, stopped: str | None, workers: int, cpus: tuple[int | None, ...], elapsed_seconds: float, rows: Path, errors: Path, ballast: Path | None)

What one invocation of run_calls() did.

Attributes:

Name Type Description
rows_written int

Rows appended by this invocation.

done_before int

Calls skipped because their key already had a row.

skipped_failed int

Calls skipped because of earlier error records.

failures int

Failed attempts of real calls in this invocation.

ballast_runs int

Ballast calls started.

stopped str | None

None when every call was attempted, else "max-failures" or "interrupted".

workers int

Worker slots.

cpus tuple[int | None, ...]

The slots' processors, None entries when unpinned.

elapsed_seconds float

Wall-clock seconds of the invocation.

rows Path

The rows file.

errors Path

The errors sidecar.

ballast Path | None

The ballast sidecar, None without ballast.

to_dict

to_dict() -> dict[str, Any]

Return the report as a JSON-able dictionary.

CallsStoppedError

CallsStoppedError(report: CallReport)

Bases: DispatchAtlasError

A run of calls stopped before every call was attempted.

Attributes:

Name Type Description
report

The report of the run, also written next to the rows file.

interleaved

interleaved(calls: Sequence[Call], *, seed: int, classes: Callable[[Call], int] | None = None) -> list[Call]

Return the calls in a seeded random order, optionally class by class.

Parameters:

Name Type Description Default
calls Sequence[Call]

The calls; not changed.

required
seed int

Seed of the shuffle.

required
classes Callable[[Call], int] | None

Gives each call a class; calls are ordered by class (lower first) and shuffled within each class.

None

run_calls

run_calls(calls: Sequence[Call], rows: str | Path, *, key_of: Callable[[Mapping[str, Any]], Any], cpus: Sequence[int] | int | str, workers: int | None = None, pin: bool = True, stamp: Mapping[str, Any] | None = None, finish: Callable[[dict[str, Any], CallFacts], Mapping[str, Any]] | None = None, max_failures: int = 3, skip_failed: bool = False, ballast: bool | Sequence[Call] = False, ballast_seed: int | None = None, allow_changed_run: bool = False, python: str | Path | None = None, entry: Sequence[str] = ('-m', 'dispatchatlas.processes.child'), environment: Mapping[str, str] | None = None, cwd: str | Path | None = None, clean_tokens: Sequence[str] = (), timeout_seconds: float | None = None, progress_every: int = 100, log: Logger | None = None) -> CallReport

Run every call not yet done, each in a child process, and append its row.

Parameters:

Name Type Description Default
calls Sequence[Call]

The calls, in the order to start them (see interleaved()). Keys must be distinct.

required
rows str | Path

The rows file; appended to, and read first for a resume.

required
key_of Callable[[Mapping[str, Any]], Any]

Gives a row's key; it must equal the call's key for the row the call returns (after finish and stamp).

required
cpus Sequence[int] | int | str

The run's processors: indices, a mask, or text for parse_cpus(). Each must be one the calling process may use. With pin=False only their number matters.

required
workers int | None

Worker slots, at most len(cpus); slot i owns cpus[i]. Defaults to len(cpus).

None
pin bool

Pin the runner to the whole set and each child to its slot's processor. Refused where the platform cannot pin.

True
stamp Mapping[str, Any] | None

Constant fields added to every row, such as provenance; a field the row already has is a RowRefused failure. A resume whose stamp differs from the first invocation's is refused unless allow_changed_run.

None
finish Callable[[dict[str, Any], CallFacts], Mapping[str, Any]] | None

Turns the function's row and the call's facts into the row written; by default the facts go under row["run"].

None
max_failures int

Failed attempts of real calls after which the run stops at once; 0 for no limit.

3
skip_failed bool

Skip calls with an earlier error record instead of retrying them.

False
ballast bool | Sequence[Call]

True to keep idle slots busy with repeats of calls in a seeded order, or the calls to repeat, in the order given; their rows are discarded and their facts go to the ballast sidecar. Ballast stops after three ballast calls in a row fail.

False
ballast_seed int | None

Seed of the order of ballast=True; by default one derived from the rows file's name.

None
allow_changed_run bool

Resume although the stamp, the number of workers, or the processors differ from those of the file's first invocation (recorded in <rows stem>.runs.jsonl, or, for a file without it, the stamp fields of its rows).

False
python str | Path | None

The interpreter of the children; sys.executable by default.

None
entry Sequence[str]

Arguments after the interpreter that run child_main().

('-m', 'dispatchatlas.processes.child')
environment Mapping[str, str] | None

Variables added to the children's environment, after the thread limits (each 1), PYTHONUNBUFFERED=1, and PYTHONIOENCODING=utf-8.

None
cwd str | Path | None

Working directory of the children.

None
clean_tokens Sequence[str]

Words removed from error messages, ignoring case.

()
timeout_seconds float | None

Default wall-clock limit of a child.

None
progress_every int

Log a progress line every this many rows.

100
log Logger | None

Logger for progress; this module's by default.

None

Returns:

Type Description
CallReport

The report, also written to <rows stem>.report.json.

A row holding a NaN, an infinity, or a value JSON has no form for is a RowNotSerializable failure (RowRefused when finish or stamp put it there), which names the field: standard JSON has no NaN, so write None or a string.

Raises:

Type Description
ConfigError

If the arguments are invalid or the platform cannot do what they ask (nothing is read or started then), if another runner holds the rows file, if the file holds a key twice, or if the run differs from the file's first invocation (see allow_changed_run); nothing is started then.

TornRowsError

If the rows file or the errors sidecar ends in a torn line; nothing is started then.

CallsStoppedError

When max_failures is reached; every child has ended and the report is written.

KeyboardInterrupt

After every child has ended and the report is written with stopped="interrupted".

dispatchatlas.processes.child

The child side of a call: one function called once in a process of its own.

python -m dispatchatlas.processes.child reads one JSON document from its standard input:

{"schema": "dispatchatlas.call/1", "key": ..., "target": "package.module:function",
 "arguments": {...}, "cpu": 7, "group": "<name>", "attempt": 1,
 "clean_tokens": [...], "parent": 1234}

First it keeps a private copy of its standard output descriptor for the reply and points descriptor 1, sys.stdout, and on Windows the standard output handle at the standard error stream for good, so nothing the function or anything it starts writes can corrupt the reply. Then, in this order, it joins its process group by name (join_group()), on POSIX systems ends itself if the runner parent is already gone and starts a thread that ends it when the runner dies, pins itself to cpu and reads its mask back (pin_self()), imports the target, and calls it with the arguments. The last line of its standard output is the reply, one JSON object: {"row": {...}, "facts": {...}} with exit code 0, or {"error": {"type": ..., "message": ...}, "facts": {...}} with exit code 1. A traceback goes to the standard error stream only. Each failing step has its own error type: BadDocument, GroupRefused, PinningRefused, TargetNotFound, RowNotMapping, RowNotSerializable, or the class of what the function raised.

facts holds pid, cpu, affinity_inherited (the mask the process started with), affinity_mask (the mask read back after pinning), group (how the group was joined), threads (the thread-limit variables of numerical libraries as the process saw them), peak_rss_bytes, started_utc, ended_utc, and wall_seconds.

The module imports the standard library and the leaf modules of this package only: a process started for every call should not pay for loading the library's solvers. For the same reason the error-naming rule of dispatchatlas.campaigns.execution.exception_error() is repeated here rather than imported.

SCHEMA module-attribute

SCHEMA = 'dispatchatlas.call/1'

The schema of the document a child reads.

THREAD_VARIABLES module-attribute

THREAD_VARIABLES = ('OMP_NUM_THREADS', 'MKL_NUM_THREADS', 'OPENBLAS_NUM_THREADS', 'BLIS_NUM_THREADS', 'VECLIB_MAXIMUM_THREADS', 'NUMEXPR_NUM_THREADS')

Environment variables that limit the thread pools of numerical libraries.

clean_message

clean_message(text: str, *, tokens: Iterable[str] = (), limit: int = _MESSAGE_LIMIT) -> str

Return an error message with file paths and named tokens removed.

Removed, in this order: everything from File " to the end of its line (a traceback frame), replaced by <frame>; a path in quotes, up to its closing quote, replaced by <path> inside the same quotes; Windows drive paths, UNC paths, and absolute POSIX paths of two or more segments, each replaced by <path>; then every token, ignoring case, replaced by <token>. Whitespace is collapsed and the result cut to limit characters.

An unquoted path may hold spaces, so it is taken to run to the last path separator on its line: text after a path on the same line can be removed with it. Names that must never appear are best passed as tokens, which are removed wherever they occur.

describe_error

describe_error(error: BaseException, *, tokens: Iterable[str] = ()) -> dict[str, str]

Return {"type", "message"} for an exception, both cleaned of tokens.

Classes of the builtins module keep their plain name; others are qualified by their module. An empty message becomes the type's name. The tokens are removed from the type too, since a module or class name can hold one.

locate_callable

locate_callable(target: str) -> Callable[..., Any]

Import "package.module:function" (or "package.module:Class.method").

Raises:

Type Description
ImportError

If the module cannot be imported.

AttributeError

If an attribute on the path is missing.

TypeError

If the target is malformed or not callable.

run_child

run_child(document: Mapping[str, Any], *, resolve: Callable[[str], Callable[..., Any]] | None = None) -> tuple[dict[str, Any], int]

Run one call described by document in the calling process.

Parameters:

Name Type Description Default
document Mapping[str, Any]

The call document (see the module docstring).

required
resolve Callable[[str], Callable[..., Any]] | None

Turns a target into a function; locate_callable() by default. A caller whose functions live in its own entry script passes its own.

None

Returns:

Type Description
tuple[dict[str, Any], int]

The reply object and the exit code: 0 for a row, 1 for an error.

child_main

child_main(argv: Sequence[str] | None = None, *, resolve: Callable[[str], Callable[..., Any]] | None = None) -> int

Read the document from standard input, run the call, write the reply line.

Parameters:

Name Type Description Default
argv Sequence[str] | None

Ignored apart from --help; the document comes on stdin.

None
resolve Callable[[str], Callable[..., Any]] | None None

Returns:

Type Description
int

The exit code: 0 for a row, 1 for an error reply.

The reply travels on a private copy of the standard output descriptor. Before anything else runs, descriptor 1 (and, on Windows, the process's standard output handle) is pointed at the standard error stream for the rest of the process's life, so whatever the function, an exit handler, a late thread, or a helper process that inherited it writes to standard output, before or after the reply, lands in the standard error stream. The function is meant to be the entry of a process of its own.

dispatchatlas.processes.rows

A rows file: one JSON object per line, appended by one writer, read back strictly.

RowWriter appends each row as one line, UTF-8 without a byte-order mark, ending in a line feed, in one write followed by a flush and an fsync, so a crash loses at most the row being written and leaves at most a torn last line. read_rows() refuses such a line, with its number and byte offset, rather than skip it; repair_rows() moves the torn bytes aside so the file can be resumed. done_keys() reads back the keys the rows hold, through a caller's key function, for a resume.

Keys are compared through canonical_key(), so a tuple key and the list that JSON gives back for it are one key.

Key

Key = str | int | tuple[Key, ...] | list[Key]

What identifies a call: a string, an integer, or a sequence of keys; JSON-able.

TornRowsError

TornRowsError(path: Path, line_number: int, byte_offset: int, reason: str)

Bases: ValueError

A rows file holds a line that is not one complete JSON object.

Attributes:

Name Type Description
path

The rows file.

line_number

The line, counted from 1.

byte_offset

Offset of the line's first byte in the file.

reason

What is wrong with the line.

RowWriter

RowWriter(path: str | Path, *, fsync: bool = True)

Appends rows to a file, one JSON object per line, with an fsync per row.

Parameters:

Name Type Description Default
path str | Path

The rows file; created when absent, appended to otherwise.

required
fsync bool

Call os.fsync after each row (default). False only flushes, for tests and scratch files.

True

A lock makes write() safe from several threads, but a file must have one writing object: two writers in two processes could interleave.

count property

count: int

Rows written by this writer.

write

write(row: Mapping[str, Any]) -> int

Append one row and return the number of rows written so far.

Raises:

Type Description
TypeError

If the row is not a mapping.

ValueError

If the row cannot be written as JSON (a value that is not JSON-able, a NaN or an infinity).

close

close() -> None

Close the file.

RepairReport dataclass

RepairReport(path: Path, aside: Path | None, removed_bytes: int, kept_lines: int)

What repair_rows() did.

Attributes:

Name Type Description
path Path

The rows file.

aside Path | None

The file holding the bytes removed, None when nothing was.

removed_bytes int

Bytes removed from the end of the rows file.

kept_lines int

Complete lines the rows file holds now.

canonical_key

canonical_key(key: Any) -> str

Return the canonical text of a key: compact JSON with sorted object keys.

Raises:

Type Description
ValueError

If the key cannot be written as JSON.

unwritable_field

unwritable_field(value: Any) -> str | None

Say which field of a row stops it being written as standard JSON, or None.

The row is walked depth first; the first problem found is described with its dotted path (x.y, items[2]): a NaN or an infinity, which standard JSON cannot hold, an object key that is not a string, or a value of a type JSON has no form for (a set, a NumPy scalar, a date).

read_rows

read_rows(path: str | Path) -> Iterator[dict[str, Any]]

Yield the rows of a rows file, in order.

Raises:

Type Description
TornRowsError

At the first line that is blank, is not a JSON object, or is the last line and lacks its line feed. Rows before it have been yielded.

OSError

If the file cannot be read.

done_keys

done_keys(path: str | Path, key_of: Callable[[Mapping[str, Any]], Any]) -> set[str]

Return the canonical keys of the rows of a file; empty when it does not exist.

Raises:

Type Description
TornRowsError

If a line is torn (see read_rows()); no key is returned then.

duplicate_keys

duplicate_keys(path: str | Path, key_of: Callable[[Mapping[str, Any]], Any]) -> list[str]

Return the canonical keys held by more than one row, in order of first sight.

A rows file written by one runner holds one row per key; a repeated key means two runners wrote it, or rows were copied in by hand. Empty when the file does not exist.

Raises:

Type Description
TornRowsError

If a line is torn (see read_rows()).

repair_rows

repair_rows(path: str | Path) -> RepairReport

Move the bytes after the last line feed of a rows file aside.

The bytes go to <path>.torn-<UTC time>.bin and the file is truncated after its last line feed. Kept lines are neither parsed nor rewritten, so a malformed line that is not last stays (and read_rows() still refuses it): that is another writer's damage, not a torn tail.

Raises:

Type Description
OSError

If the file cannot be read or written.

count_lines

count_lines(path: str | Path, *, offset: int = 0) -> tuple[int, int]

Return the line feeds of a file from byte offset, and the file's size.

A watcher counts a growing file incrementally by passing the size it got last time; a size below offset means the file was truncated. A missing file counts as empty.

dispatchatlas.processes.affinity

Processor sets as text and masks, and a process pinning itself with a read-back.

A process pinned from outside is not always the process that computes: where the python executable of an environment is a launcher that starts the real interpreter as its child, a mask set on the launcher's handle reaches the launcher alone. pin_self() therefore pins the calling process and reads its mask back, and refuses when the operating system reports anything other than what was asked.

  • On Windows the mask is set with SetProcessAffinityMask on the current process, which moves every thread, and read with GetProcessAffinityMask. One mask describes at most 64 processors, so on a machine with several processor groups nothing is pinned.
  • Where os.sched_setaffinity exists (Linux), the set of the calling thread is written, and every thread started later inherits it; a thread that is already running keeps its own, so pin first thing.
  • Elsewhere (macOS) the system has no such call and pinning is refused.

This module uses the standard library only, so a child process can import it in a few milliseconds. Its errors are RuntimeError and ValueError; dispatchatlas.campaigns.processors wraps the same primitives with the library's own errors.

PinReport dataclass

PinReport(wanted: int, inherited: int | None, read: int, cpus: tuple[int, ...])

What pin_self() set and read back.

Attributes:

Name Type Description
wanted int

The mask asked for.

inherited int | None

The mask the process had before the call, None when it could not be read. It shows whether a parent that created the process on a processor reached the process that computes.

read int

The mask read back after the call, equal to wanted.

cpus tuple[int, ...]

The processors of read, sorted.

parse_cpus

parse_cpus(text: str) -> tuple[int, ...]

Return the processors a text names, sorted and distinct.

Three forms are accepted: a hexadecimal mask ("0xF0F"), ranges and indices separated by commas ("4-7,12"), and a single index ("3").

Raises:

Type Description
ValueError

If the text is none of these forms, names a processor twice, has a range whose end precedes its start, or names no processor.

cpus_of_mask

cpus_of_mask(mask: int) -> tuple[int, ...]

Return the processors whose bits are set in mask, sorted.

mask_of_cpus

mask_of_cpus(cpus: Iterable[int]) -> int

Return the mask with the bits of cpus set.

current_mask

current_mask() -> int | None

Return the affinity mask of the calling process (Linux: thread), or None.

None means the platform has no affinity call, the call failed, or one Windows mask does not describe the process.

current_cpus

current_cpus() -> tuple[int, ...] | None

Return the processors of current_mask(), or None when it cannot be read.

pinning_refusal_text

pinning_refusal_text() -> str | None

Return why this platform cannot pin a process, or None when it can.

pin_self

pin_self(cpus: int | Iterable[int]) -> PinReport

Confine the calling process to processors, then read its mask back.

Parameters:

Name Type Description Default
cpus int | Iterable[int]

A processor index, or several. Each must be in the set the process has now: a process never widens its own set.

required

Returns:

Type Description
PinReport

The masks wanted, inherited, and read back.

Raises:

Type Description
RuntimeError

If the platform cannot pin (see pinning_refusal_text()), a processor is outside the current set, the operating system refuses, or the mask read back differs from the one asked for.

confined

confined(cpus: int | Iterable[int]) -> Iterator[None]

Confine the caller to processors for the duration of the context.

A process created inside the context starts on cpus. On Windows every thread of the calling process is confined for the duration; elsewhere only the calling thread is. The earlier set is restored on exit.

Raises:

Type Description
RuntimeError

If the set cannot be read or written, or a processor is outside it.

describe_cpus

describe_cpus(cpus: Iterable[int]) -> str

Return processor indices with runs collapsed: "0-3, 8".

dispatchatlas.processes.groups

Kill-on-close process groups: what a kill reaches, and what ends with the parent.

A ProcessGroup holds the processes of one worker slot so that one call ends all of them, and so that they end when the process that created the group dies.

  • On Windows the group is a named job object with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE. Each member starts suspended (ProcessGroup.popen_options()), and the parent assigns it to the job before resuming it (ProcessGroup.assign()), so a launcher executable cannot create the process that computes outside the job. Job membership is inherited, so everything a member starts is in the job. The child also joins the job by name (join_group()), which proves the membership from inside and covers a member the parent could not assign. The parent holds the only lasting handle: when it dies for any reason, the kernel ends every process in the job.
  • On POSIX systems each child is started in a session of its own (ProcessGroup.popen_options()), so a kill of its process group reaches what it started, and the child ends its group when its parent is gone (end_with_parent_process()).

This module uses the standard library only; errors are returned as text, or raised as RuntimeError on the child side.

ProcessGroup

ProcessGroup(name: str, platform: Literal['job', 'session', 'none'], handle: int | None = None, error: str | None = None)

One kill-on-close group of processes.

Create it with create(), start each member with popen_options(), then assign() it. terminate() ends every member; close() releases the group, and on Windows ends every process still in it.

Attributes:

Name Type Description
name

The group's name, which a child passes to join_group().

platform

"job" (a Windows job object), "session" (a POSIX session per member), or "none" (the job could not be created; see error).

handle

The job's handle on Windows, else None.

error

Why the group could not be created, else None.

create classmethod

create(name: str) -> ProcessGroup

Create a group; on Windows a named kill-on-close job object.

A Windows job of the same name that already exists is refused, so a group never holds another owner's processes.

popen_options

popen_options() -> dict[str, Any]

Return the keyword arguments of subprocess.Popen for a member.

Windows: a new console process group, so a Ctrl-C in the parent's console does not reach the member, and a suspended start: the process runs no instruction until assign() has put it in the job and resumed it, so everything it starts inherits the job. POSIX: a new session.

assign

assign(pid: int) -> str | None

Put a started process in the group; return the error, if any.

On Windows the process is resumed afterwards, whether or not it could join, since popen_options() starts it suspended. On POSIX systems the process must have been started with popen_options(); its id is recorded as a process group id.

assign_process

assign_process(process: Popen[Any]) -> str | None

Put a process started with popen_options() in the group; return the error.

Like assign(), but on Windows it uses the handle that subprocess.Popen already holds (with full access) instead of opening the process again by its id, so a process that cannot be opened is still assigned and, above all, resumed.

forget

forget(pid: int) -> None

Stop tracking a member that has ended (POSIX: its process group).

terminate

terminate(code: int = TERMINATED) -> None

End every process of the group; the group stays usable.

Windows: TerminateJobObject. POSIX: SIGKILL to the process group of every member still tracked.

close

close() -> None

Release the group. On Windows every process still in it ends.

group_name

group_name(owner_pid: int, slot: int) -> str

Return the name of the group of slot slot of process owner_pid.

On Windows the name is in the session's local namespace, so only processes of the same session can open it.

join_group

join_group(name: str) -> str

Join the group name from inside a member process.

Returns:

Type Description
str

"job" when the process joined the Windows job, ``"job

str

(already)"`` when it was in it (inherited from a launcher the parent

str

had assigned), and "session" on POSIX systems, where a member

str

leads its own session from its start and nothing is left to do.

Raises:

Type Description
RuntimeError

If the Windows job cannot be opened or joined.

end_with_parent_process

end_with_parent_process(runner: int | None = None) -> None

End the calling process and its process group when its parent is gone (POSIX).

Parameters:

Name Type Description Default
runner int | None

The id of the process that started this one, as that process recorded it. When the parent seen now is another process and runner no longer exists, the runner died before this call (during the interpreter's start), and the process ends at once. A parent seen now that differs while runner lives is a launcher in between, which is watched instead.

None

A daemon thread then polls the parent's id once a second; when it changes, the parent has died, and the thread kills the process group this process leads, or this process alone. On Windows it does nothing: the job object of the group ends with the parent's handle.

dispatchatlas.processes.snapshots

A snapshot of the machine's processes, without extra dependencies.

snapshot() lists every process with its parent, executable, affinity mask, and start time: on Windows through the Toolhelp snapshot and one limited query per process, on Linux from /proc. Elsewhere it returns an empty tuple. A field the caller may not read (a process of another user, a protected process) is None.

descendants() follows parent ids from a process, interpreters_on() finds Python interpreters that may run on given processors, and end_process() ends one process on request. Nothing here runs a shell command or a management query.

ProcessInfo dataclass

ProcessInfo(pid: int, parent_pid: int | None, name: str, executable: str | None, affinity_mask: int | None, started_utc: str | None)

One process of a snapshot().

Attributes:

Name Type Description
pid int

Process id.

parent_pid int | None

Id of the process that created it, None when unknown. The parent may have ended and its id been reused since.

name str

Executable file name (Windows) or command name (Linux).

executable str | None

Full path of the executable, None when unreadable.

affinity_mask int | None

Processors the process may run on, None when unreadable.

started_utc str | None

Creation time, ISO 8601 in UTC, None when unreadable.

snapshot

snapshot() -> tuple[ProcessInfo, ...]

Return every process of the machine, ordered by id; empty where unsupported.

descendants

descendants(pid: int, processes: Sequence[ProcessInfo] | None = None) -> tuple[ProcessInfo, ...]

Return the processes created by pid and by them, at any depth.

A process that names a parent but started before it holds a reused parent id, and is left out with what it created.

Parameters:

Name Type Description Default
pid int

The root process; it is not part of the result.

required
processes Sequence[ProcessInfo] | None

A snapshot to search; a fresh one by default.

None

interpreters_on

interpreters_on(cpus: Iterable[int], processes: Sequence[ProcessInfo] | None = None, *, exclude: Iterable[int] = ()) -> tuple[ProcessInfo, ...]

Return the Python interpreters whose affinity mask meets cpus.

An interpreter is a process named python, python3, python3.13, pythonw, with or without .exe. A process whose mask cannot be read is left out.

Parameters:

Name Type Description Default
cpus Iterable[int]

Processor indices.

required
processes Sequence[ProcessInfo] | None

A snapshot to search; a fresh one by default.

None
exclude Iterable[int]

Process ids to leave out, such as a runner and its children.

()

is_running

is_running(pid: int) -> bool

Return whether process pid exists and has not ended.

A process that has ended but whose parent has not reaped it yet (a POSIX zombie, or a Windows process whose handle is still open) is not running. A process the caller may not query counts as running.

end_process

end_process(pid: int) -> str | None

End process pid at once; return the error text, or None.

Windows: TerminateProcess with exit code 1. POSIX: SIGKILL.

dispatchatlas.processes.memory

The memory of the calling process and of the machine.

Each returns None where the platform does not tell. The module uses the standard library only.

peak_memory_bytes

peak_memory_bytes() -> int | None

Return the peak resident set of the calling process in bytes, or None.

available_memory_bytes

available_memory_bytes() -> int | None

Return the physical memory available to new work in bytes, or None.

total_memory_bytes

total_memory_bytes() -> int | None

Return the physical memory of the machine in bytes, or None.

dispatchatlas.processes.launching

Starting a command that must outlive the shell or session that starts it.

launch_detached() starts a command with no console window and no standard input, its output appended to a log file, in a process group of its own, and does not wait for it. On Windows it first asks to leave the job object of the calling process (CREATE_BREAKAWAY_FROM_JOB), which an agent session or a CI step may hold with kill-on-close; where the job forbids that, it starts the command inside the job and says so, since the command then ends with the job. On POSIX systems the command starts a new session.

From the command line:

python -m dispatchatlas.processes.launching --log run.log --cwd . [--env K=V ...] -- COMMAND ...

prints started pid N, breakaway yes|no and returns at once.

Launched dataclass

Launched(pid: int, breakaway: bool, log: Path)

A command launch_detached() started.

Attributes:

Name Type Description
pid int

Id of the started process.

breakaway bool

Whether it left the job object of the caller (Windows); False on Windows means it ends with that job. Always True on POSIX systems, where no job holds it.

log Path

The log file its output is appended to.

launch_detached

launch_detached(command: Sequence[str], *, log: str | Path, cwd: str | Path, environment: Mapping[str, str] | None = None) -> Launched

Start command detached from the caller and return without waiting.

The command gets the caller's environment with PYTHONUNBUFFERED=1 and PYTHONIOENCODING=utf-8 added, then environment; standard input from the null device; standard output and error appended to log, which is created with its folder when absent.

Raises:

Type Description
ValueError

If the command is empty.

OSError

If the command cannot be started.

main

main(argv: Sequence[str] | None = None) -> int

Run the command-line entry; see the module docstring.

dispatchatlas.processes.watching

Watching a long run from outside: one status line per interval, never a restart.

watch() polls a runner process and reports whether it is alive, how many processes it has started that are still running, how many other Python interpreters may run on the run's processors (a second run, a stray test, an orphan of an earlier run), how many lines its rows file holds, and how much memory the machine has available. It starts, restarts, and ends nothing: what to do about a status is the operator's decision.

From the command line:

python -m dispatchatlas.processes.watching --pid P --rows FILE --cpus MASK
    [--interval 60] [--status FILE] [--keep]

prints format_status() once per poll and appends it to --status, and stops after the first poll that finds the runner gone unless --keep. Where snapshot() is unsupported (macOS), the process counts are zero; whether the runner is alive is still probed directly (is_running()).

Status dataclass

Status(time_utc: str, runner_alive: bool, live_children: int, live_interpreters: int, foreign_interpreters: int, foreign_pids: tuple[int, ...], rows_lines: int, rows_bytes: int, available_memory_bytes: int | None)

One poll of watch().

Attributes:

Name Type Description
time_utc str

When the poll was taken, ISO 8601 in UTC.

runner_alive bool

Whether the runner process was running.

live_children int

Processes the runner started, at any depth, running: every launcher, interpreter, helper, and console host, so about two per busy slot with a Windows virtual-environment launcher.

live_interpreters int

Those of them named like a Python interpreter (launchers included).

foreign_interpreters int

Python interpreters on the watched processors that are neither the runner, its descendants, the watcher, nor an interpreter-named launcher above the runner or the watcher.

foreign_pids tuple[int, ...]

Their process ids.

rows_lines int

Complete lines of the rows file.

rows_bytes int

Size of the rows file.

available_memory_bytes int | None

Memory available on the machine, or None.

watch

watch(pid: int, *, rows: str | Path, cpus: Iterable[int], interval: float = 60.0, until: int | None = None, keep: bool = False) -> Iterator[Status]

Yield a Status of a runner every interval seconds.

Parameters:

Name Type Description Default
pid int

The runner's process id.

required
rows str | Path

Its rows file, counted incrementally.

required
cpus Iterable[int]

The processors the run owns.

required
interval float

Seconds between polls.

60.0
until int | None

Stop after this many polls; None for no limit.

None
keep bool

Keep polling after the runner is gone; by default the iterator stops after the first status that finds it gone.

False

format_status

format_status(status: Status) -> str

Return a status as one line, fields in a fixed order; RUNNER-DEAD first when gone.

main

main(argv: Sequence[str] | None = None) -> int

Run the command-line entry; see the module docstring.