Processes¶
Pinned child processes and the tools around them: the runner of calls, the child side of a call, the rows file, processor sets, process groups, process snapshots, memory, detached launch, and watching a run. The guide explains how they fit together.
dispatchatlas.processes
¶
The operating system's view of a run: pinned child processes and the tools around them.
calls:run_calls(), which runs caller-defined functions, each once in a child process of its own on a processor of its own, and appends the rows they return to a rows file.child: the child side of a call.rows: the rows file, its writer, its strict reader, resume by keys, and the repair of a torn last line.affinity: processor sets as text and masks, and a process pinning itself with a read-back.groups: kill-on-close process groups (Windows job objects, POSIX sessions).snapshots: the machine's processes with parent, executable, affinity mask, and start time.memory: peak, available, and total memory.launching: starting a command detached from its shell.watching: a status line per interval about a running run.
The package sits between dispatchatlas.core and
dispatchatlas.campaigns, which uses its affinity, group, and memory
primitives. Its leaf modules use the standard library only.
Unlike the other packages, this one re-exports its names lazily: a child
process started for every call runs python -m
dispatchatlas.processes.child, which imports this package first, and
importing the runner's dependencies there would add a substantial part of a
second to every call. from dispatchatlas.processes import run_calls
works as usual.
dispatchatlas.processes.calls
¶
Running caller-defined functions, each once in a pinned child process of its own.
run_calls() runs a list of Call objects. Each call is one
function, named "package.module:function", called with keyword
arguments in a fresh child process started with the running interpreter
(dispatchatlas.processes.child). The mapping it returns becomes one
line of a rows file (RowWriter),
written by the runner alone.
- Processors. The run owns a set of processors; worker slot
iowns processorcpus[i]. The runner pins itself to the whole set, creates each child while confined to the slot's processor, so the child (and a launcher executable in front of the interpreter) starts there, and the child then pins itself and reads its mask back: a mismatch is a failure, not a row. - Groups. Each slot has a kill-on-close process group
(
ProcessGroup) that the child also joins by name; whatever a call leaves running ends when the call ends, and every child ends when the runner dies. - Resume. The rows file is the record of what is done: calls whose key
(read back through
key_of) has a row are skipped. A torn last line, a key with two rows, a second runner on the same file, and a resume whose stamp, workers, or processors differ from the first invocation's are refused before anything starts. - Failures. A call that raises, returns something that is not a row,
dies, or passes its time limit is recorded in the errors sidecar
(
<rows stem>.errors.jsonl), never in the rows file, and is retried on the next invocation unlessskip_failed. Aftermax_failuresfailures the run stops at once and raisesCallsStoppedError. - Load. Every row records the number of slots, how many were busy when
its call started, and the least that were busy during its life;
ballastkeeps idle slots busy with discarded repeat calls until the last real call ends.
Next to the rows file the runner writes <stem>.errors.jsonl,
<stem>.ballast.jsonl (with ballast), <stem>.children.log (the
children's standard output and error, tracebacks included),
<stem>.runs.jsonl (one line per invocation: its stamp, workers, and
processors), <stem>.lock (held while a runner uses the file), and
<stem>.report.json.
ERROR_SCHEMA
module-attribute
¶
The schema of an error record of the errors sidecar.
Call
dataclass
¶
Call(key: Key, target: str, arguments: Mapping[str, Any] = dict(), timeout_seconds: float | None = None, label: str | None = None)
One function to call once in a child process.
Attributes:
| Name | Type | Description |
|---|---|---|
key |
Key
|
What identifies the call: a string, an integer, or a tuple or
list of keys. The row the call returns must give it back through
the runner's |
target |
str
|
|
arguments |
Mapping[str, Any]
|
Keyword arguments of the function; JSON-able. |
timeout_seconds |
float | None
|
Wall-clock limit of the child, |
label |
str | None
|
Free text for logs. |
CallFacts
dataclass
¶
CallFacts(key: Any, attempt: int, cpu: int | None, affinity_inherited: int | None, affinity_mask: int | None, pid: int | None, group: str | None, threads: Mapping[str, str | None], peak_rss_bytes: int | None, started_utc: str | None, ended_utc: str | None, wall_seconds: float | None, workers: int, workers_live_start: int, workers_live_min: int, ballast: bool)
What the runner and the child recorded about one call.
Attributes:
| Name | Type | Description |
|---|---|---|
key |
Any
|
The call's key. |
attempt |
int
|
1 for a first attempt, 1 + earlier error records otherwise. |
cpu |
int | None
|
The slot's processor, |
affinity_inherited |
int | None
|
The child's mask before it pinned itself. |
affinity_mask |
int | None
|
The child's mask read back after pinning. |
pid |
int | None
|
The id of the process that ran the function. |
group |
str | None
|
How the child joined its group ( |
threads |
Mapping[str, str | None]
|
The thread-limit variables the child saw. |
peak_rss_bytes |
int | None
|
The child's peak resident set, where readable. |
started_utc |
str | None
|
When the child began, in UTC. |
ended_utc |
str | None
|
When the function returned, in UTC. |
wall_seconds |
float | None
|
Seconds from the child's start to the function's end. |
workers |
int
|
Worker slots of the run. |
workers_live_start |
int
|
Busy slots once the runner had filled every slot it could when this call started. |
workers_live_min |
int
|
The least busy slots during the call's life. |
ballast |
bool
|
Whether this was a ballast call. |
CallError
dataclass
¶
CallError(key: Any, attempt: int, type: str, message: str, returncode: int | None, last_line: bool, facts: Mapping[str, Any] | None, cpu: int | None, started_utc: str, ended_utc: str, schema: str = ERROR_SCHEMA)
One failed attempt of a call, a line of the errors sidecar.
Attributes:
| Name | Type | Description |
|---|---|---|
key |
Any
|
The call's key. |
attempt |
int
|
The attempt number. |
type |
str
|
The error type: the class the function raised, a step of the
child ( |
message |
str
|
The message, without file paths or caller-named tokens. |
returncode |
int | None
|
The child's exit code; negative for a POSIX signal. |
last_line |
bool
|
Whether the child wrote its reply line. |
facts |
Mapping[str, Any] | None
|
The call's facts, as far as known. |
cpu |
int | None
|
The slot's processor. |
started_utc |
str
|
When the runner started the child. |
ended_utc |
str
|
When the runner saw it end. |
schema |
str
|
CallReport
dataclass
¶
CallReport(rows_written: int, done_before: int, skipped_failed: int, failures: int, ballast_runs: int, stopped: str | None, workers: int, cpus: tuple[int | None, ...], elapsed_seconds: float, rows: Path, errors: Path, ballast: Path | None)
What one invocation of run_calls() did.
Attributes:
| Name | Type | Description |
|---|---|---|
rows_written |
int
|
Rows appended by this invocation. |
done_before |
int
|
Calls skipped because their key already had a row. |
skipped_failed |
int
|
Calls skipped because of earlier error records. |
failures |
int
|
Failed attempts of real calls in this invocation. |
ballast_runs |
int
|
Ballast calls started. |
stopped |
str | None
|
|
workers |
int
|
Worker slots. |
cpus |
tuple[int | None, ...]
|
The slots' processors, |
elapsed_seconds |
float
|
Wall-clock seconds of the invocation. |
rows |
Path
|
The rows file. |
errors |
Path
|
The errors sidecar. |
ballast |
Path | None
|
The ballast sidecar, |
CallsStoppedError
¶
CallsStoppedError(report: CallReport)
Bases: DispatchAtlasError
A run of calls stopped before every call was attempted.
Attributes:
| Name | Type | Description |
|---|---|---|
report |
The report of the run, also written next to the rows file. |
interleaved
¶
interleaved(calls: Sequence[Call], *, seed: int, classes: Callable[[Call], int] | None = None) -> list[Call]
Return the calls in a seeded random order, optionally class by class.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
calls
|
Sequence[Call]
|
The calls; not changed. |
required |
seed
|
int
|
Seed of the shuffle. |
required |
classes
|
Callable[[Call], int] | None
|
Gives each call a class; calls are ordered by class (lower first) and shuffled within each class. |
None
|
run_calls
¶
run_calls(calls: Sequence[Call], rows: str | Path, *, key_of: Callable[[Mapping[str, Any]], Any], cpus: Sequence[int] | int | str, workers: int | None = None, pin: bool = True, stamp: Mapping[str, Any] | None = None, finish: Callable[[dict[str, Any], CallFacts], Mapping[str, Any]] | None = None, max_failures: int = 3, skip_failed: bool = False, ballast: bool | Sequence[Call] = False, ballast_seed: int | None = None, allow_changed_run: bool = False, python: str | Path | None = None, entry: Sequence[str] = ('-m', 'dispatchatlas.processes.child'), environment: Mapping[str, str] | None = None, cwd: str | Path | None = None, clean_tokens: Sequence[str] = (), timeout_seconds: float | None = None, progress_every: int = 100, log: Logger | None = None) -> CallReport
Run every call not yet done, each in a child process, and append its row.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
calls
|
Sequence[Call]
|
The calls, in the order to start them (see
|
required |
rows
|
str | Path
|
The rows file; appended to, and read first for a resume. |
required |
key_of
|
Callable[[Mapping[str, Any]], Any]
|
Gives a row's key; it must equal the call's key for the row
the call returns (after |
required |
cpus
|
Sequence[int] | int | str
|
The run's processors: indices, a mask, or text for
|
required |
workers
|
int | None
|
Worker slots, at most |
None
|
pin
|
bool
|
Pin the runner to the whole set and each child to its slot's processor. Refused where the platform cannot pin. |
True
|
stamp
|
Mapping[str, Any] | None
|
Constant fields added to every row, such as provenance; a
field the row already has is a |
None
|
finish
|
Callable[[dict[str, Any], CallFacts], Mapping[str, Any]] | None
|
Turns the function's row and the call's facts into the row
written; by default the facts go under |
None
|
max_failures
|
int
|
Failed attempts of real calls after which the run stops at once; 0 for no limit. |
3
|
skip_failed
|
bool
|
Skip calls with an earlier error record instead of retrying them. |
False
|
ballast
|
bool | Sequence[Call]
|
|
False
|
ballast_seed
|
int | None
|
Seed of the order of |
None
|
allow_changed_run
|
bool
|
Resume although the stamp, the number of workers,
or the processors differ from those of the file's first
invocation (recorded in |
False
|
python
|
str | Path | None
|
The interpreter of the children; |
None
|
entry
|
Sequence[str]
|
Arguments after the interpreter that run
|
('-m', 'dispatchatlas.processes.child')
|
environment
|
Mapping[str, str] | None
|
Variables added to the children's environment, after
the thread limits (each |
None
|
cwd
|
str | Path | None
|
Working directory of the children. |
None
|
clean_tokens
|
Sequence[str]
|
Words removed from error messages, ignoring case. |
()
|
timeout_seconds
|
float | None
|
Default wall-clock limit of a child. |
None
|
progress_every
|
int
|
Log a progress line every this many rows. |
100
|
log
|
Logger | None
|
Logger for progress; this module's by default. |
None
|
Returns:
| Type | Description |
|---|---|
CallReport
|
The report, also written to |
A row holding a NaN, an infinity, or a value JSON has no form for is a
RowNotSerializable failure (RowRefused when finish or
stamp put it there), which names the field: standard JSON has no
NaN, so write None or a string.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the arguments are invalid or the platform cannot
do what they ask (nothing is read or started then), if another
runner holds the rows file, if the file holds a key twice, or
if the run differs from the file's first invocation (see
|
TornRowsError
|
If the rows file or the errors sidecar ends in a torn line; nothing is started then. |
CallsStoppedError
|
When |
KeyboardInterrupt
|
After every child has ended and the report is
written with |
dispatchatlas.processes.child
¶
The child side of a call: one function called once in a process of its own.
python -m dispatchatlas.processes.child reads one JSON document from its
standard input:
{"schema": "dispatchatlas.call/1", "key": ..., "target": "package.module:function",
"arguments": {...}, "cpu": 7, "group": "<name>", "attempt": 1,
"clean_tokens": [...], "parent": 1234}
First it keeps a private copy of its standard output descriptor for the
reply and points descriptor 1, sys.stdout, and on Windows the standard
output handle at the standard error stream for good, so nothing the function
or anything it starts writes can corrupt the reply. Then, in this order, it
joins its process group by name
(join_group()), on POSIX systems ends
itself if the runner parent is already gone and starts a thread that ends
it when the runner dies, pins itself to cpu and reads its mask back
(pin_self()), imports the target, and
calls it with the arguments. The last line of its standard output is the
reply, one JSON object:
{"row": {...}, "facts": {...}} with exit code 0, or {"error": {"type":
..., "message": ...}, "facts": {...}} with exit code 1. A traceback goes
to the standard error stream only. Each failing step has its own error type:
BadDocument, GroupRefused, PinningRefused, TargetNotFound,
RowNotMapping, RowNotSerializable, or the class of what the function raised.
facts holds pid, cpu, affinity_inherited (the mask the
process started with), affinity_mask (the mask read back after pinning),
group (how the group was joined), threads (the thread-limit variables
of numerical libraries as the process saw them), peak_rss_bytes,
started_utc, ended_utc, and wall_seconds.
The module imports the standard library and the leaf modules of this
package only: a process started for every call should not pay for loading
the library's solvers. For the same reason the error-naming rule of
dispatchatlas.campaigns.execution.exception_error() is repeated here
rather than imported.
THREAD_VARIABLES
module-attribute
¶
THREAD_VARIABLES = ('OMP_NUM_THREADS', 'MKL_NUM_THREADS', 'OPENBLAS_NUM_THREADS', 'BLIS_NUM_THREADS', 'VECLIB_MAXIMUM_THREADS', 'NUMEXPR_NUM_THREADS')
Environment variables that limit the thread pools of numerical libraries.
clean_message
¶
Return an error message with file paths and named tokens removed.
Removed, in this order: everything from File " to the end of its
line (a traceback frame), replaced by <frame>; a path in quotes, up
to its closing quote, replaced by <path> inside the same quotes;
Windows drive paths, UNC paths, and absolute POSIX paths of two or more
segments, each replaced by <path>; then every token, ignoring case,
replaced by <token>. Whitespace is collapsed and the result cut to
limit characters.
An unquoted path may hold spaces, so it is taken to run to the last path
separator on its line: text after a path on the same line can be
removed with it. Names that must never appear are best passed as
tokens, which are removed wherever they occur.
describe_error
¶
Return {"type", "message"} for an exception, both cleaned of tokens.
Classes of the builtins module keep their plain name; others are
qualified by their module. An empty message becomes the type's name.
The tokens are removed from the type too, since a module or class name
can hold one.
locate_callable
¶
Import "package.module:function" (or "package.module:Class.method").
Raises:
| Type | Description |
|---|---|
ImportError
|
If the module cannot be imported. |
AttributeError
|
If an attribute on the path is missing. |
TypeError
|
If the target is malformed or not callable. |
run_child
¶
run_child(document: Mapping[str, Any], *, resolve: Callable[[str], Callable[..., Any]] | None = None) -> tuple[dict[str, Any], int]
Run one call described by document in the calling process.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
document
|
Mapping[str, Any]
|
The call document (see the module docstring). |
required |
resolve
|
Callable[[str], Callable[..., Any]] | None
|
Turns a target into a function; |
None
|
Returns:
| Type | Description |
|---|---|
tuple[dict[str, Any], int]
|
The reply object and the exit code: 0 for a row, 1 for an error. |
child_main
¶
child_main(argv: Sequence[str] | None = None, *, resolve: Callable[[str], Callable[..., Any]] | None = None) -> int
Read the document from standard input, run the call, write the reply line.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
argv
|
Sequence[str] | None
|
Ignored apart from |
None
|
resolve
|
Callable[[str], Callable[..., Any]] | None
|
See |
None
|
Returns:
| Type | Description |
|---|---|
int
|
The exit code: 0 for a row, 1 for an error reply. |
The reply travels on a private copy of the standard output descriptor. Before anything else runs, descriptor 1 (and, on Windows, the process's standard output handle) is pointed at the standard error stream for the rest of the process's life, so whatever the function, an exit handler, a late thread, or a helper process that inherited it writes to standard output, before or after the reply, lands in the standard error stream. The function is meant to be the entry of a process of its own.
dispatchatlas.processes.rows
¶
A rows file: one JSON object per line, appended by one writer, read back strictly.
RowWriter appends each row as one line, UTF-8 without a byte-order
mark, ending in a line feed, in one write followed by a flush and an
fsync, so a crash loses at most the row being written and leaves at most
a torn last line. read_rows() refuses such a line, with its number and
byte offset, rather than skip it; repair_rows() moves the torn bytes
aside so the file can be resumed. done_keys() reads back the keys the
rows hold, through a caller's key function, for a resume.
Keys are compared through canonical_key(), so a tuple key and the list
that JSON gives back for it are one key.
Key
¶
What identifies a call: a string, an integer, or a sequence of keys; JSON-able.
TornRowsError
¶
Bases: ValueError
A rows file holds a line that is not one complete JSON object.
Attributes:
| Name | Type | Description |
|---|---|---|
path |
The rows file. |
|
line_number |
The line, counted from 1. |
|
byte_offset |
Offset of the line's first byte in the file. |
|
reason |
What is wrong with the line. |
RowWriter
¶
Appends rows to a file, one JSON object per line, with an fsync per row.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
The rows file; created when absent, appended to otherwise. |
required |
fsync
|
bool
|
Call |
True
|
A lock makes write() safe from several threads, but a file must
have one writing object: two writers in two processes could interleave.
write
¶
Append one row and return the number of rows written so far.
Raises:
| Type | Description |
|---|---|
TypeError
|
If the row is not a mapping. |
ValueError
|
If the row cannot be written as JSON (a value that is not JSON-able, a NaN or an infinity). |
RepairReport
dataclass
¶
What repair_rows() did.
Attributes:
| Name | Type | Description |
|---|---|---|
path |
Path
|
The rows file. |
aside |
Path | None
|
The file holding the bytes removed, |
removed_bytes |
int
|
Bytes removed from the end of the rows file. |
kept_lines |
int
|
Complete lines the rows file holds now. |
canonical_key
¶
Return the canonical text of a key: compact JSON with sorted object keys.
Raises:
| Type | Description |
|---|---|
ValueError
|
If the key cannot be written as JSON. |
unwritable_field
¶
Say which field of a row stops it being written as standard JSON, or None.
The row is walked depth first; the first problem found is described with
its dotted path (x.y, items[2]): a NaN or an infinity, which
standard JSON cannot hold, an object key that is not a string, or a value
of a type JSON has no form for (a set, a NumPy scalar, a date).
read_rows
¶
Yield the rows of a rows file, in order.
Raises:
| Type | Description |
|---|---|
TornRowsError
|
At the first line that is blank, is not a JSON object, or is the last line and lacks its line feed. Rows before it have been yielded. |
OSError
|
If the file cannot be read. |
done_keys
¶
Return the canonical keys of the rows of a file; empty when it does not exist.
Raises:
| Type | Description |
|---|---|
TornRowsError
|
If a line is torn (see |
duplicate_keys
¶
Return the canonical keys held by more than one row, in order of first sight.
A rows file written by one runner holds one row per key; a repeated key means two runners wrote it, or rows were copied in by hand. Empty when the file does not exist.
Raises:
| Type | Description |
|---|---|
TornRowsError
|
If a line is torn (see |
repair_rows
¶
repair_rows(path: str | Path) -> RepairReport
Move the bytes after the last line feed of a rows file aside.
The bytes go to <path>.torn-<UTC time>.bin and the file is truncated
after its last line feed. Kept lines are neither parsed nor rewritten,
so a malformed line that is not last stays (and read_rows() still
refuses it): that is another writer's damage, not a torn tail.
Raises:
| Type | Description |
|---|---|
OSError
|
If the file cannot be read or written. |
count_lines
¶
Return the line feeds of a file from byte offset, and the file's size.
A watcher counts a growing file incrementally by passing the size it
got last time; a size below offset means the file was truncated. A
missing file counts as empty.
dispatchatlas.processes.affinity
¶
Processor sets as text and masks, and a process pinning itself with a read-back.
A process pinned from outside is not always the process that computes: where
the python executable of an environment is a launcher that starts the real
interpreter as its child, a mask set on the launcher's handle reaches the
launcher alone. pin_self() therefore pins the calling process and reads
its mask back, and refuses when the operating system reports anything other
than what was asked.
- On Windows the mask is set with
SetProcessAffinityMaskon the current process, which moves every thread, and read withGetProcessAffinityMask. One mask describes at most 64 processors, so on a machine with several processor groups nothing is pinned. - Where
os.sched_setaffinityexists (Linux), the set of the calling thread is written, and every thread started later inherits it; a thread that is already running keeps its own, so pin first thing. - Elsewhere (macOS) the system has no such call and pinning is refused.
This module uses the standard library only, so a child process can import it
in a few milliseconds. Its errors are RuntimeError and
ValueError; dispatchatlas.campaigns.processors wraps the same
primitives with the library's own errors.
PinReport
dataclass
¶
What pin_self() set and read back.
Attributes:
| Name | Type | Description |
|---|---|---|
wanted |
int
|
The mask asked for. |
inherited |
int | None
|
The mask the process had before the call, |
read |
int
|
The mask read back after the call, equal to |
cpus |
tuple[int, ...]
|
The processors of |
parse_cpus
¶
Return the processors a text names, sorted and distinct.
Three forms are accepted: a hexadecimal mask ("0xF0F"), ranges and
indices separated by commas ("4-7,12"), and a single index ("3").
Raises:
| Type | Description |
|---|---|
ValueError
|
If the text is none of these forms, names a processor twice, has a range whose end precedes its start, or names no processor. |
cpus_of_mask
¶
Return the processors whose bits are set in mask, sorted.
current_mask
¶
Return the affinity mask of the calling process (Linux: thread), or None.
None means the platform has no affinity call, the call failed, or one
Windows mask does not describe the process.
current_cpus
¶
Return the processors of current_mask(), or None when it cannot be read.
pinning_refusal_text
¶
Return why this platform cannot pin a process, or None when it can.
pin_self
¶
pin_self(cpus: int | Iterable[int]) -> PinReport
Confine the calling process to processors, then read its mask back.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cpus
|
int | Iterable[int]
|
A processor index, or several. Each must be in the set the process has now: a process never widens its own set. |
required |
Returns:
| Type | Description |
|---|---|
PinReport
|
The masks wanted, inherited, and read back. |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If the platform cannot pin (see
|
confined
¶
Confine the caller to processors for the duration of the context.
A process created inside the context starts on cpus. On Windows every
thread of the calling process is confined for the duration; elsewhere
only the calling thread is. The earlier set is restored on exit.
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If the set cannot be read or written, or a processor is outside it. |
describe_cpus
¶
Return processor indices with runs collapsed: "0-3, 8".
dispatchatlas.processes.groups
¶
Kill-on-close process groups: what a kill reaches, and what ends with the parent.
A ProcessGroup holds the processes of one worker slot so that one
call ends all of them, and so that they end when the process that created
the group dies.
- On Windows the group is a named job object with
JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE. Each member starts suspended (ProcessGroup.popen_options()), and the parent assigns it to the job before resuming it (ProcessGroup.assign()), so a launcher executable cannot create the process that computes outside the job. Job membership is inherited, so everything a member starts is in the job. The child also joins the job by name (join_group()), which proves the membership from inside and covers a member the parent could not assign. The parent holds the only lasting handle: when it dies for any reason, the kernel ends every process in the job. - On POSIX systems each child is started in a session of its own
(
ProcessGroup.popen_options()), so a kill of its process group reaches what it started, and the child ends its group when its parent is gone (end_with_parent_process()).
This module uses the standard library only; errors are returned as text, or
raised as RuntimeError on the child side.
ProcessGroup
¶
ProcessGroup(name: str, platform: Literal['job', 'session', 'none'], handle: int | None = None, error: str | None = None)
One kill-on-close group of processes.
Create it with create(), start each member with
popen_options(), then assign() it. terminate() ends
every member; close() releases the group, and on Windows ends every
process still in it.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
The group's name, which a child passes to |
|
platform |
|
|
handle |
The job's handle on Windows, else |
|
error |
Why the group could not be created, else |
create
classmethod
¶
create(name: str) -> ProcessGroup
Create a group; on Windows a named kill-on-close job object.
A Windows job of the same name that already exists is refused, so a group never holds another owner's processes.
popen_options
¶
Return the keyword arguments of subprocess.Popen for a member.
Windows: a new console process group, so a Ctrl-C in the parent's
console does not reach the member, and a suspended start: the
process runs no instruction until assign() has put it in the
job and resumed it, so everything it starts inherits the job.
POSIX: a new session.
assign
¶
Put a started process in the group; return the error, if any.
On Windows the process is resumed afterwards, whether or not it
could join, since popen_options() starts it suspended. On POSIX
systems the process must have been started with popen_options();
its id is recorded as a process group id.
assign_process
¶
Put a process started with popen_options() in the group; return the error.
Like assign(), but on Windows it uses the handle that
subprocess.Popen already holds (with full access) instead of
opening the process again by its id, so a process that cannot be
opened is still assigned and, above all, resumed.
terminate
¶
End every process of the group; the group stays usable.
Windows: TerminateJobObject. POSIX: SIGKILL to the process
group of every member still tracked.
group_name
¶
Return the name of the group of slot slot of process owner_pid.
On Windows the name is in the session's local namespace, so only processes of the same session can open it.
join_group
¶
Join the group name from inside a member process.
Returns:
| Type | Description |
|---|---|
str
|
|
str
|
(already)"`` when it was in it (inherited from a launcher the parent |
str
|
had assigned), and |
str
|
leads its own session from its start and nothing is left to do. |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If the Windows job cannot be opened or joined. |
end_with_parent_process
¶
End the calling process and its process group when its parent is gone (POSIX).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
runner
|
int | None
|
The id of the process that started this one, as that process
recorded it. When the parent seen now is another process and
|
None
|
A daemon thread then polls the parent's id once a second; when it changes, the parent has died, and the thread kills the process group this process leads, or this process alone. On Windows it does nothing: the job object of the group ends with the parent's handle.
dispatchatlas.processes.snapshots
¶
A snapshot of the machine's processes, without extra dependencies.
snapshot() lists every process with its parent, executable, affinity
mask, and start time: on Windows through the Toolhelp snapshot and one
limited query per process, on Linux from /proc. Elsewhere it returns an
empty tuple. A field the caller may not read (a process of another user, a
protected process) is None.
descendants() follows parent ids from a process, interpreters_on()
finds Python interpreters that may run on given processors, and
end_process() ends one process on request. Nothing here runs a shell
command or a management query.
ProcessInfo
dataclass
¶
ProcessInfo(pid: int, parent_pid: int | None, name: str, executable: str | None, affinity_mask: int | None, started_utc: str | None)
One process of a snapshot().
Attributes:
| Name | Type | Description |
|---|---|---|
pid |
int
|
Process id. |
parent_pid |
int | None
|
Id of the process that created it, |
name |
str
|
Executable file name (Windows) or command name (Linux). |
executable |
str | None
|
Full path of the executable, |
affinity_mask |
int | None
|
Processors the process may run on, |
started_utc |
str | None
|
Creation time, ISO 8601 in UTC, |
snapshot
¶
snapshot() -> tuple[ProcessInfo, ...]
Return every process of the machine, ordered by id; empty where unsupported.
descendants
¶
descendants(pid: int, processes: Sequence[ProcessInfo] | None = None) -> tuple[ProcessInfo, ...]
Return the processes created by pid and by them, at any depth.
A process that names a parent but started before it holds a reused parent id, and is left out with what it created.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pid
|
int
|
The root process; it is not part of the result. |
required |
processes
|
Sequence[ProcessInfo] | None
|
A snapshot to search; a fresh one by default. |
None
|
interpreters_on
¶
interpreters_on(cpus: Iterable[int], processes: Sequence[ProcessInfo] | None = None, *, exclude: Iterable[int] = ()) -> tuple[ProcessInfo, ...]
Return the Python interpreters whose affinity mask meets cpus.
An interpreter is a process named python, python3,
python3.13, pythonw, with or without .exe. A process whose
mask cannot be read is left out.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cpus
|
Iterable[int]
|
Processor indices. |
required |
processes
|
Sequence[ProcessInfo] | None
|
A snapshot to search; a fresh one by default. |
None
|
exclude
|
Iterable[int]
|
Process ids to leave out, such as a runner and its children. |
()
|
is_running
¶
Return whether process pid exists and has not ended.
A process that has ended but whose parent has not reaped it yet (a POSIX zombie, or a Windows process whose handle is still open) is not running. A process the caller may not query counts as running.
end_process
¶
End process pid at once; return the error text, or None.
Windows: TerminateProcess with exit code 1. POSIX: SIGKILL.
dispatchatlas.processes.memory
¶
The memory of the calling process and of the machine.
peak_memory_bytes(): the peak resident set of the calling process (WindowsPeakWorkingSetSize, LinuxVmHWM, macOSru_maxrss).available_memory_bytes(): the physical memory available to new work (WindowsullAvailPhys, LinuxMemAvailable).total_memory_bytes(): the physical memory of the machine.
Each returns None where the platform does not tell. The module uses the
standard library only.
dispatchatlas.processes.launching
¶
Starting a command that must outlive the shell or session that starts it.
launch_detached() starts a command with no console window and no
standard input, its output appended to a log file, in a process group of its
own, and does not wait for it. On Windows it first asks to leave the job
object of the calling process (CREATE_BREAKAWAY_FROM_JOB), which an agent
session or a CI step may hold with kill-on-close; where the job forbids
that, it starts the command inside the job and says so, since the command
then ends with the job. On POSIX systems the command starts a new session.
From the command line:
prints started pid N, breakaway yes|no and returns at once.
Launched
dataclass
¶
A command launch_detached() started.
Attributes:
| Name | Type | Description |
|---|---|---|
pid |
int
|
Id of the started process. |
breakaway |
bool
|
Whether it left the job object of the caller (Windows);
|
log |
Path
|
The log file its output is appended to. |
launch_detached
¶
launch_detached(command: Sequence[str], *, log: str | Path, cwd: str | Path, environment: Mapping[str, str] | None = None) -> Launched
Start command detached from the caller and return without waiting.
The command gets the caller's environment with PYTHONUNBUFFERED=1
and PYTHONIOENCODING=utf-8 added, then environment; standard
input from the null device; standard output and error appended to
log, which is created with its folder when absent.
Raises:
| Type | Description |
|---|---|
ValueError
|
If the command is empty. |
OSError
|
If the command cannot be started. |
main
¶
Run the command-line entry; see the module docstring.
dispatchatlas.processes.watching
¶
Watching a long run from outside: one status line per interval, never a restart.
watch() polls a runner process and reports whether it is alive, how
many processes it has started that are still running, how many other
Python interpreters may run on the run's processors (a second run, a stray
test, an orphan of an earlier run), how many lines its rows file holds, and
how much memory the machine has available. It starts, restarts, and ends
nothing: what to do about a status is the operator's decision.
From the command line:
python -m dispatchatlas.processes.watching --pid P --rows FILE --cpus MASK
[--interval 60] [--status FILE] [--keep]
prints format_status() once per poll and appends it to --status,
and stops after the first poll that finds the runner gone unless --keep.
Where snapshot() is unsupported
(macOS), the process counts are zero; whether the runner is alive is still
probed directly (is_running()).
Status
dataclass
¶
Status(time_utc: str, runner_alive: bool, live_children: int, live_interpreters: int, foreign_interpreters: int, foreign_pids: tuple[int, ...], rows_lines: int, rows_bytes: int, available_memory_bytes: int | None)
One poll of watch().
Attributes:
| Name | Type | Description |
|---|---|---|
time_utc |
str
|
When the poll was taken, ISO 8601 in UTC. |
runner_alive |
bool
|
Whether the runner process was running. |
live_children |
int
|
Processes the runner started, at any depth, running: every launcher, interpreter, helper, and console host, so about two per busy slot with a Windows virtual-environment launcher. |
live_interpreters |
int
|
Those of them named like a Python interpreter (launchers included). |
foreign_interpreters |
int
|
Python interpreters on the watched processors that are neither the runner, its descendants, the watcher, nor an interpreter-named launcher above the runner or the watcher. |
foreign_pids |
tuple[int, ...]
|
Their process ids. |
rows_lines |
int
|
Complete lines of the rows file. |
rows_bytes |
int
|
Size of the rows file. |
available_memory_bytes |
int | None
|
Memory available on the machine, or |
watch
¶
watch(pid: int, *, rows: str | Path, cpus: Iterable[int], interval: float = 60.0, until: int | None = None, keep: bool = False) -> Iterator[Status]
Yield a Status of a runner every interval seconds.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pid
|
int
|
The runner's process id. |
required |
rows
|
str | Path
|
Its rows file, counted incrementally. |
required |
cpus
|
Iterable[int]
|
The processors the run owns. |
required |
interval
|
float
|
Seconds between polls. |
60.0
|
until
|
int | None
|
Stop after this many polls; |
None
|
keep
|
bool
|
Keep polling after the runner is gone; by default the iterator stops after the first status that finds it gone. |
False
|
format_status
¶
format_status(status: Status) -> str
Return a status as one line, fields in a fixed order; RUNNER-DEAD first when gone.
main
¶
Run the command-line entry; see the module docstring.