Skip to content

Statistics

The statistical kernels: distributions, ranks, rank tests, effect sizes, bootstrap intervals, Bayesian tests, corrections, and power.

dispatchatlas.analysis.stats

Statistical kernels: plain numbers in, small frozen results out.

The kernels know nothing about campaigns. The rank tests, effect sizes, and corrections are pure Python and verified against reference values computed with SciPy; the bootstrap, the Bayesian tests, and the power simulation draw from numpy.random.default_rng(seed).

  • distributions: normal, chi-square, and F tails.
  • ranks: ties, midranks, and orientation.
  • nonparametric: Wilcoxon, sign, Mann-Whitney, Friedman, Iman-Davenport, and Nemenyi.
  • effect_sizes: A12, Cliff's delta, Cohen's d and d_z, and the paired win share.
  • bootstrap: BCa and percentile intervals.
  • bayesian: Bayesian sign and signed-rank tests.
  • corrections: multiple-comparison corrections.
  • power: power and sample size of the paired test.

dispatchatlas.analysis.stats.distributions

The normal, chi-square, and F distribution functions, in pure Python.

The rank tests read their p-values from these distributions: the normal for large-sample Wilcoxon and Mann-Whitney statistics, the chi-square for the Friedman statistic, and the F for the Iman-Davenport statistic.

  • The normal tails use math.erfc(), so a tail probability keeps its relative precision far into the tail instead of cancelling against one.
  • The normal quantile is P. J. Acklam's rational approximation, with a relative error below 1.15e-9 on the open unit interval. It serves critical values and confidence levels, never the inversion of a reported p-value.
  • The chi-square tail is the regularized upper incomplete gamma function Q(df/2, x/2): a power series below the transition x < a + 1 and the modified Lentz continued fraction above it (Press et al. 2007, section 6.2).
  • The F tail is the regularized incomplete beta function I_{d2/(d2 + d1 f)}(d2/2, d1/2), evaluated by the modified Lentz continued fraction on the side of its symmetry point where it converges (Press et al. 2007, section 6.4).

Both continued fractions and the series stop at relative convergence (1e-15 for the gamma function, 3e-14 for the beta function) and raise ArithmeticError rather than return a partial sum when an iteration cap is reached. The caps grow with the square root of the shape parameters, the rate at which the expansions converge near the distributions' centres.

normal_cdf

normal_cdf(z: float) -> float

Return the standard normal distribution function P(Z <= z).

normal_sf

normal_sf(z: float) -> float

Return the standard normal upper tail P(Z > z).

Unlike 1 - normal_cdf(z), it stays accurate when the tail is far below machine epsilon.

normal_ppf

normal_ppf(probability: float) -> float

Return the standard normal quantile by Acklam's rational approximation.

The central region [0.02425, 0.97575] uses a rational function of (p - 1/2)^2, and the tails a rational function of sqrt(-2 log p). The relative error is below 1.15e-9.

Parameters:

Name Type Description Default
probability float

A probability strictly between 0 and 1.

required

Returns:

Type Description
float

z such that normal_cdf(z) is probability.

Raises:

Type Description
ConfigError

If probability is not strictly between 0 and 1.

chi2_sf

chi2_sf(statistic: float, df: float) -> float

Return the chi-square upper tail P(X > statistic).

Parameters:

Name Type Description Default
statistic float

The observed statistic. Non-positive values give 1 and positive infinity gives 0.

required
df float

Degrees of freedom, positive.

required

Returns:

Type Description
float

The tail probability.

Raises:

Type Description
ConfigError

If df is not a positive finite number or the statistic is NaN.

ArithmeticError

If the incomplete gamma function fails to converge.

f_sf

f_sf(statistic: float, df_numerator: float, df_denominator: float) -> float

Return the F-distribution upper tail P(F > statistic).

Parameters:

Name Type Description Default
statistic float

The observed statistic. Non-positive values give 1 and positive infinity gives 0.

required
df_numerator float

Numerator degrees of freedom d1, positive.

required
df_denominator float

Denominator degrees of freedom d2, positive.

required

Returns:

Type Description
float

I_x(d2/2, d1/2) at x = d2 / (d2 + d1 statistic).

Raises:

Type Description
ConfigError

If a degree of freedom is not a positive finite number or the statistic is NaN.

ArithmeticError

If the incomplete beta function fails to converge.

dispatchatlas.analysis.stats.ranks

Ranks, ties, and the orientation of objective values.

Every rank-based statistic in this package opens the same way: replace values by their ranks, give tied values the mean of the positions they span (midranks), and orient the ranks so that rank 1 is the best value. Two values tie when they are equal within TIE_TOLERANCE, relatively or absolutely, so a difference left by floating-point summation is not read as a win. Infinite values are allowed and rank last or first: the analysis pipeline scores a failed run as the worst possible value.

TIE_TOLERANCE module-attribute

TIE_TOLERANCE = 1e-12

Two values tie when they differ by at most this much, relatively or absolutely.

Sense

Sense = Literal['min', 'max']

Whether smaller ("min") or larger ("max") values are better.

check_sense

check_sense(sense: object) -> Sense

Return sense if it is "min" or "max".

Raises:

Type Description
ConfigError

For any other value.

finite_or_infinite

finite_or_infinite(values: Iterable[float], what: str) -> list[float]

Return values as floats, refusing NaN.

Parameters:

Name Type Description Default
values Iterable[float]

The values.

required
what str

Name of the quantity, for the error message.

required

Raises:

Type Description
ConfigError

If a value is NaN or not a number.

ties

ties(left: float, right: float) -> bool

Return whether two values are equal within TIE_TOLERANCE.

Equal infinities tie; infinities of opposite sign do not.

strictly_better

strictly_better(left: float, right: float, sense: Sense) -> bool

Return whether left beats right under sense, ties excluded.

midranks

midranks(values: Sequence[float]) -> tuple[float, ...]

Return ascending ranks, from 1, with tied values sharing their mean rank.

Values are sorted, and each following value joins the current tie group while it ties with the group's first value, so groups never chain. The ranks are returned in the order of the input.

oriented_ranks

oriented_ranks(values: Sequence[float], sense: Sense) -> tuple[float, ...]

Return midranks oriented so that rank 1 is the best value under sense.

rank_block

rank_block(block: Mapping[str, float], sense: Sense) -> dict[str, float]

Rank the entries of one block, such as the solvers on one problem.

Parameters:

Name Type Description Default
block Mapping[str, float]

Value of each entry.

required
sense Sense

Whether smaller or larger values are better.

required

Returns:

Type Description
dict[str, float]

The oriented midrank of each entry, keyed like block.

dispatchatlas.analysis.stats.nonparametric

Rank tests: Wilcoxon, sign, Mann-Whitney, Friedman, Iman-Davenport, and Nemenyi.

The tests follow the non-parametric protocol for comparing algorithms over several problems (Demšar 2006; Derrac et al. 2011): a paired test for two algorithms, and for several algorithms an omnibus test over complete blocks followed by a post-hoc comparison of average ranks.

All tests are two-sided. Small samples use the exact null distribution: the signed-rank test with or without tied magnitudes, where ties make it the distribution conditional on the midranks, and the Mann-Whitney test without ties. Larger samples, and tied Mann-Whitney samples, use the normal approximation with the tie-corrected variance and a continuity correction of 0.5, whose tail comes from normal_sf() so that very small p-values keep their precision. Rank ties follow TIE_TOLERANCE, and so do the variance corrections: tie_sum() counts the groups that the midranks share.

WILCOXON_EXACT_LIMIT module-attribute

WILCOXON_EXACT_LIMIT = 50

Largest number of non-zero pairs for which the exact Wilcoxon null is used.

The subset-sum recursion costs about n * n (n + 1) additions, 130 thousand at the limit, and the counts of sign assignments stay below 2^53, so their conversion to floats is exact.

MANN_WHITNEY_EXACT_LIMIT module-attribute

MANN_WHITNEY_EXACT_LIMIT = 20

Largest size of either sample for which the exact Mann-Whitney null is used.

TestMethod

TestMethod = Literal['exact', 'exact-conditional', 'normal', 'degenerate']

How a p-value was obtained.

"exact" is the exact null distribution of untied ranks, "exact-conditional" the exact null distribution of the signed-rank statistic given the midranks of tied magnitudes, "normal" the normal approximation, and "degenerate" a test without data.

WilcoxonResult dataclass

WilcoxonResult(statistic: float, p_value: float, pairs: int, zeros: int, method: TestMethod)

A two-sided Wilcoxon signed-rank test.

Attributes:

Name Type Description
statistic float

min(W+, W-), the smaller signed-rank sum.

p_value float

Two-sided p-value.

pairs int

Number of non-zero differences the test ranked.

zeros int

Number of differences dropped as zero.

method TestMethod

How the p-value was obtained.

MannWhitneyResult dataclass

MannWhitneyResult(statistic: float, p_value: float, first_size: int, second_size: int, method: TestMethod)

A two-sided Mann-Whitney U test.

Attributes:

Name Type Description
statistic float

min(U1, U2).

p_value float

Two-sided p-value.

first_size int

Size of the first sample.

second_size int

Size of the second sample.

method TestMethod

How the p-value was obtained.

FriedmanResult dataclass

FriedmanResult(statistic: float, p_value: float, df: int, blocks: int, groups: int, average_ranks: tuple[tuple[str, float], ...])

A Friedman test over complete blocks.

Attributes:

Name Type Description
statistic float

The tie-corrected chi-square statistic Q.

p_value float

Upper-tail chi-square probability of Q.

df int

Degrees of freedom, the number of groups minus one.

blocks int

Number of blocks N, such as problems.

groups int

Number of groups k, such as solvers.

average_ranks tuple[tuple[str, float], ...]

Each group's mean rank over the blocks, rank 1 best, in the order of the first block's keys.

kendalls_w property

kendalls_w: float

Kendall's coefficient of concordance W = Q / (N (k - 1)), in [0, 1].

It is the effect size of the Friedman test: 0 when the rankings disagree completely, 1 when every block ranks the groups alike (Kendall and Babington Smith 1939).

ImanDavenportResult dataclass

ImanDavenportResult(statistic: float, p_value: float, df_numerator: int, df_denominator: int)

The Iman-Davenport F statistic derived from a Friedman result.

Attributes:

Name Type Description
statistic float

F = (N - 1) Q / (N (k - 1) - Q), infinite when every block ranks the groups alike.

p_value float

Upper-tail F probability of the statistic.

df_numerator int

k - 1.

df_denominator int

(k - 1)(N - 1).

wilcoxon_signed_rank

wilcoxon_signed_rank(differences: Iterable[float]) -> WilcoxonResult

Return the two-sided Wilcoxon signed-rank test of paired differences.

Differences within TIE_TOLERANCE of zero are dropped (Wilcoxon's treatment of zeros), the magnitudes of the others are midranked, and the statistic is the smaller of the positive and negative rank sums. Up to WILCOXON_EXACT_LIMIT pairs, the p-value comes from the exact null distribution of the statistic: under the null hypothesis each difference is positive or negative with equal probability, so every assignment of signs to the ranks is equally likely, and a subset-sum recursion counts the assignments by their rank sum. With tied magnitudes the ranks are the midranks, and the distribution is the one conditional on them (method "exact-conditional"), whose smallest p-value over n pairs is 2 / 2^n as without ties; a normal approximation can fall below that on a small tied sample. Above the limit the normal approximation uses the mean n(n+1)/4, the variance n(n+1)(2n+1)/24 - sum(t^3 - t)/48 over the tie groups, and a continuity correction of 0.5 (Wilcoxon 1945). Infinite differences are allowed and rank above every finite one.

Parameters:

Name Type Description Default
differences Iterable[float]

One difference per matched pair.

required

Returns:

Type Description
WilcoxonResult

The test, whose method names how the p-value was obtained; with

WilcoxonResult

no non-zero difference, a degenerate result with p-value 1.

Raises:

Type Description
ConfigError

If there are no differences or one is NaN.

sign_test

sign_test(wins: int, losses: int) -> float

Return the exact two-sided sign test p-value.

Under the null hypothesis each untied pair is a win with probability 1/2, so the p-value is twice the binomial tail of the smaller count, capped at 1 (Dixon and Mood 1946). Ties carry no direction and are left out by the caller.

Parameters:

Name Type Description Default
wins int

Pairs the first sample won.

required
losses int

Pairs the first sample lost.

required

Raises:

Type Description
ConfigError

If a count is negative or not an integer.

mann_whitney_u

mann_whitney_u(first: Iterable[float], second: Iterable[float]) -> MannWhitneyResult

Return the two-sided Mann-Whitney U test of two independent samples.

The pooled sample is midranked and U1 is the first sample's rank sum minus n1(n1+1)/2; the statistic is min(U1, n1 n2 - U1). When neither sample exceeds MANN_WHITNEY_EXACT_LIMIT and no values tie, the p-value comes from the exact null, which the recursion of Mann and Whitney (1947) counts. Otherwise the normal approximation uses the mean n1 n2 / 2, the variance n1 n2 / 12 * ((N + 1) - sum(t^3 - t) / (N (N - 1))), and a continuity correction of 0.5.

Parameters:

Name Type Description Default
first Iterable[float]

The first sample.

required
second Iterable[float]

The second sample.

required

Raises:

Type Description
ConfigError

If a sample is empty or holds NaN.

friedman_test

friedman_test(blocks: Sequence[Mapping[str, float]], sense: Sense = 'min') -> FriedmanResult

Return the Friedman test of several groups over complete blocks.

Each block, such as one problem, maps every group, such as a solver, to its value. Values are midranked within each block with rank 1 for the best, and the statistic Q = [12 / (N k (k+1)) sum_j R_j^2 - 3 N (k+1)] / [1 - sum(t^3 - t) / (N k (k^2 - 1))] divides the rank-sum dispersion by the tie correction; it is read against a chi-square distribution with k - 1 degrees of freedom (Friedman 1937). Infinite values are allowed, for example for failed runs.

Parameters:

Name Type Description Default
blocks Sequence[Mapping[str, float]]

One mapping per block, all with the same keys.

required
sense Sense

Whether smaller or larger values are better.

'min'

Raises:

Type Description
ConfigError

With fewer than two blocks or two groups, blocks that disagree on their keys, or a NaN value.

friedman_statistic

friedman_statistic(rank_sums: Sequence[float], blocks: int, ties: float = 0.0) -> float

Return the tie-corrected Friedman statistic from the rank sums of each group.

Parameters:

Name Type Description Default
rank_sums Sequence[float]

The sum over blocks of each group's rank.

required
blocks int

The number of blocks N.

required
ties float

sum(t^3 - t) over the tie groups of every block, see tie_sum().

0.0

Returns:

Type Description
float

Q. When the tie correction is not positive, which happens only

float

when every block ties every group, the uncorrected statistic.

iman_davenport

iman_davenport(friedman: FriedmanResult) -> ImanDavenportResult

Return the Iman-Davenport F statistic of a Friedman result.

F = (N - 1) Q / (N (k - 1) - Q) with k - 1 and (k - 1)(N - 1) degrees of freedom is less conservative than the chi-square reading of Q (Iman and Davenport 1980; Demšar 2006). When every block ranks the groups alike the denominator vanishes, and the statistic is infinite with p-value 0.

nemenyi_q

nemenyi_q(groups: int, alpha: float = 0.05) -> float

Return the Nemenyi critical value q_alpha for groups groups.

Tabulated for 2 to 12 groups at alpha 0.05 and 0.10 (Demšar 2006, Table 5(a)).

Raises:

Type Description
ConfigError

For another number of groups or another alpha.

nemenyi_critical_difference

nemenyi_critical_difference(groups: int, blocks: int, alpha: float = 0.05) -> float

Return the Nemenyi critical difference of average ranks.

Two groups differ significantly when their average ranks over the blocks differ by more than q_alpha sqrt(k (k + 1) / (6 N)) (Nemenyi 1963; Demšar 2006).

Raises:

Type Description
ConfigError

With fewer than one block, or when nemenyi_q() has no critical value.

tie_sum

tie_sum(values: Sequence[float], sense: Sense = 'min') -> float

Return sum(t^3 - t) over the groups t of tied values.

This is the tie term of the Wilcoxon, Mann-Whitney, and Friedman variance corrections. Values tie when they are equal within TIE_TOLERANCE: the groups are the ones that share a midrank in oriented_ranks(), so the term always agrees with the ranks the tests use.

Parameters:

Name Type Description Default
values Sequence[float]

The values of one sample or block.

required
sense Sense

Whether smaller or larger values are better, the orientation in which the ranks are formed.

'min'

Raises:

Type Description
ConfigError

If sense is neither "min" nor "max".

dispatchatlas.analysis.stats.effect_sizes

Effect sizes for paired and independent comparisons.

A p-value says whether a difference is detectable; an effect size says how large it is. Every function here is oriented so that a positive direction favours the left (first) sample under the objective sense: an A12 above 0.5, a positive Cliff's delta or Cohen's d, a win share above 0.5.

Two designs need different estimands:

  • Independent samples, such as the runs of two solvers on one problem: Vargha and Delaney's A12, Cliff's delta, and Cohen's d_s compare every left value with every right value.
  • Paired samples, such as one value per problem for each of two solvers: the paired win share (the share of pairs the left sample wins, ties counting one half) and Cohen's d_z (the mean paired difference over its standard deviation). The independent forms do not apply here: their cross pairs compare different problems, so they measure how spread out the problems are rather than how the samples differ.

The independent-sample effect sizes come with the magnitude labels their sources publish. The paired ones carry no label: the published cutoffs were calibrated for other estimands.

Magnitude

Magnitude = Literal['negligible', 'small', 'medium', 'large']

A conventional label for the size of an effect.

vargha_delaney_a12

vargha_delaney_a12(left: Iterable[float], right: Iterable[float], sense: Sense = 'min') -> float

Return Vargha and Delaney's A12 for two independent samples.

A12 is the probability that a random left value beats a random right value, ties counting one half (Vargha and Delaney 2000).

Parameters:

Name Type Description Default
left Iterable[float]

The first sample.

required
right Iterable[float]

The second sample.

required
sense Sense

Whether smaller or larger values are better.

'min'

Returns:

Type Description
float

A12 in [0, 1], above 0.5 when the left sample tends to be better.

Raises:

Type Description
ConfigError

If a sample is empty or holds NaN.

cliffs_delta

cliffs_delta(left: Iterable[float], right: Iterable[float], sense: Sense = 'min') -> float

Return Cliff's delta for two independent samples.

Delta is the share of cross pairs the left sample wins minus the share it loses (Cliff 1993); it equals 2 A12 - 1.

Returns:

Type Description
float

Delta in [-1, 1], positive when the left sample tends to be better.

Raises:

Type Description
ConfigError

If a sample is empty or holds NaN.

cohens_d

cohens_d(left: Iterable[float], right: Iterable[float], sense: Sense = 'min') -> float

Return Cohen's d_s for two independent samples.

The difference of the means divided by the pooled standard deviation sqrt((SS_left + SS_right) / (n_left + n_right - 2)) (Cohen 1988), without the small-sample correction of Hedges' g.

Returns:

Type Description
float

d_s, positive when the left mean is better; 0 when both samples are

float

constant.

Raises:

Type Description
ConfigError

If a sample has fewer than two values or a value is not finite.

cohens_dz

cohens_dz(differences: Iterable[float]) -> float | None

Return Cohen's d_z, the mean of paired differences over their standard deviation.

The standard deviation uses the n - 1 denominator. Orient the differences before calling, for example as the advantage of the left sample, so that a positive d_z favours it.

Returns:

Type Description
float | None

d_z, or None when it is undefined: fewer than two differences, a

float | None

difference that is not finite, or differences without spread.

Raises:

Type Description
ConfigError

If a difference is NaN.

paired_outcomes

paired_outcomes(left: Sequence[float], right: Sequence[float], sense: Sense = 'min') -> tuple[int, int, int]

Count the pairs the left sample wins, loses, and ties.

Two values tie when they are equal within TIE_TOLERANCE.

Returns:

Type Description
tuple[int, int, int]

(wins, losses, ties) of the left sample.

Raises:

Type Description
ConfigError

If the samples differ in length or hold NaN.

paired_win_share

paired_win_share(left: Sequence[float], right: Sequence[float], sense: Sense = 'min') -> float

Return the share of pairs the left sample wins, ties counting one half.

This is the sign-test counterpart of A12 for a paired design: 1 when the left sample wins every pair, 0.5 when wins and losses balance.

Raises:

Type Description
ConfigError

If the samples are empty, differ in length, or hold NaN.

a12_magnitude

a12_magnitude(a12: float) -> Magnitude

Return the Vargha-Delaney label of an A12: cutoffs 0.56, 0.64, and 0.71.

An A12 below 0.5 is labelled by its mirror image 1 - A12, which keeps the published cutoffs exact rather than subject to the rounding of A12 - 0.5.

cliffs_delta_magnitude

cliffs_delta_magnitude(delta: float) -> Magnitude

Return the label of a Cliff's delta: cutoffs 0.147, 0.33, and 0.474 (Romano et al. 2006).

cohens_d_magnitude

cohens_d_magnitude(d: float) -> Magnitude

Return Cohen's label of an independent-sample d_s: cutoffs 0.2, 0.5, and 0.8.

dispatchatlas.analysis.stats.bootstrap

Bootstrap confidence intervals: bias-corrected and accelerated, and percentile.

bootstrap_interval() resamples the observations with replacement, computes the statistic on every resample, and reads the interval from the sorted replicates with the linear-interpolation quantile quantile() (type 7 of Hyndman and Fan 1996).

The percentile interval takes the replicate quantiles at alpha/2 and 1 - alpha/2. The bias-corrected and accelerated (BCa) interval (Efron 1987) shifts those levels by the bias correction z0 = Phi^-1(#{replicates < estimate} / B), with a strict inequality, and by the acceleration a = sum(d^3) / (6 (sum(d^2))^(3/2)) from the leave-one-out jackknife deviations d, and reads the quantiles at Phi(z0 + (z0 + z) / (1 - a (z0 + z))) for z = Phi^-1(alpha/2) and Phi^-1(1 - alpha/2), clamped to [1e-9, 1 - 1e-9]. When every replicate lies on one side of the estimate, z0 is infinite and the interval falls back to the percentile one; the same happens when 1 - a (z0 + z) is not positive, where the correction is undefined. A sample of one value, or zero resamples, gives the degenerate interval at the estimate.

Resampling draws indices from numpy.random.default_rng(seed), one row of n indices per resample, so equal inputs and seed give equal intervals for a given NumPy version.

IntervalMethod

IntervalMethod = Literal['bca', 'percentile']

The bootstrap interval to compute.

Statistic

Statistic = Literal['mean', 'median'] | Callable[[NDArray[float64]], float]

A statistic: the mean, the median, or a function of the resampled rows.

BootstrapInterval dataclass

BootstrapInterval(estimate: float, low: float, high: float, level: float, resamples: int, method: Literal['bca', 'percentile', 'degenerate'])

A bootstrap confidence interval.

Attributes:

Name Type Description
estimate float

The statistic of the observed sample.

low float

Lower bound.

high float

Upper bound.

level float

Nominal coverage, for example 0.95.

resamples int

Number of resamples drawn.

method Literal['bca', 'percentile', 'degenerate']

The interval computed: "bca", "percentile" (asked for, or the fallback of BCa), or "degenerate".

quantile

quantile(ordered: Sequence[float], q: float) -> float

Return the q quantile of sorted values by linear interpolation.

The position (n - 1) q interpolates between the neighbouring order statistics, type 7 of Hyndman and Fan (1996) and NumPy's default.

Parameters:

Name Type Description Default
ordered Sequence[float]

Values sorted in ascending order.

required
q float

A probability in [0, 1].

required

Raises:

Type Description
ConfigError

If ordered is empty or q is outside [0, 1].

bootstrap_interval

bootstrap_interval(sample: Sequence[float] | Sequence[Sequence[float]] | NDArray[float64], *, statistic: Statistic = 'mean', level: float = 0.95, resamples: int = 1000, seed: int = 0, method: IntervalMethod = 'bca') -> BootstrapInterval

Return a bootstrap confidence interval of a statistic.

Parameters:

Name Type Description Default
sample Sequence[float] | Sequence[Sequence[float]] | NDArray[float64]

The observations: numbers, or rows of numbers when the statistic is a function of several columns. Resampling draws whole rows.

required
statistic Statistic

"mean", "median", or a function that maps an array of rows (one resample) to a number.

'mean'
level float

Nominal coverage, strictly between 0 and 1.

0.95
resamples int

Number of resamples, at least 0.

1000
seed int

Seed of the resampling generator, at least 0.

0
method IntervalMethod

"bca" or "percentile".

'bca'

Returns:

Type Description
BootstrapInterval

The interval and the estimate on the observed sample.

Raises:

Type Description
ConfigError

If the sample is empty or not finite, or a parameter is out of range.

dispatchatlas.analysis.stats.bayesian

Bayesian sign and signed-rank tests over paired differences.

Both tests of Benavoli, Corani, Demšar and Zaffalon (2017) put a posterior on the probability mass of three regions of a paired difference: below the region of practical equivalence (the rope, [-rope, rope], bounds included), inside it, and above it.

  • The sign test counts the differences in each region. The posterior of the three masses is Dirichlet(n_below, n_rope + prior, n_above): the prior's pseudo-observations sit in the rope.
  • The signed-rank test puts a Dirichlet process on the distribution of the differences, with a prior pseudo-observation at zero of weight prior. Each posterior draw weighs the atoms 0, d_1, ..., d_n by Dirichlet(prior, 1, ..., 1) weights w, and a region's mass is the total weight w_i w_j of the ordered pairs whose Walsh average (z_i + z_j) / 2 falls in it.

Each result reports two things per region. mean_* is the posterior expected mass. probability_* is the posterior probability that the region holds the largest of the three masses, the decision quantity of the authors' reference implementation baycomp, estimated from samples Monte Carlo draws of numpy.random.default_rng(seed). For the sign test the expected masses are exact; for the signed-rank test they are Monte Carlo averages of the same draws.

BayesianResult dataclass

BayesianResult(probability_below: float, probability_rope: float, probability_above: float, mean_below: float, mean_rope: float, mean_above: float, rope: float, prior: float, pairs: int, samples: int, seed: int)

Posterior facts about the three regions of a paired difference.

Attributes:

Name Type Description
probability_below float

Posterior probability that the region below -rope holds the largest mass.

probability_rope float

Same, for the rope.

probability_above float

Same, for the region above rope.

mean_below float

Posterior expected mass below -rope.

mean_rope float

Posterior expected mass of the rope.

mean_above float

Posterior expected mass above rope.

rope float

Half-width of the rope.

prior float

Strength of the prior.

pairs int

Number of differences.

samples int

Monte Carlo draws.

seed int

Seed of the draws.

bayesian_sign_test

bayesian_sign_test(differences: Iterable[float], *, rope: float = 0.0, prior: float = 1.0, samples: int = 2000, seed: int = 0) -> BayesianResult

Return the Bayesian sign test of paired differences.

Parameters:

Name Type Description Default
differences Iterable[float]

One difference per matched pair; infinities are allowed.

required
rope float

Half-width of the region of practical equivalence, at least 0.

0.0
prior float

Pseudo-observations the prior places in the rope, positive.

1.0
samples int

Monte Carlo draws for the probabilities, positive.

2000
seed int

Seed of the draws, at least 0.

0

Returns:

Type Description
BayesianResult

Exact posterior means and Monte Carlo probabilities.

Raises:

Type Description
ConfigError

If there is no difference, a difference is NaN, or a parameter is out of range.

bayesian_signed_rank_test

bayesian_signed_rank_test(differences: Iterable[float], *, rope: float = 0.0, prior: float = 0.5, samples: int = 2000, seed: int = 0) -> BayesianResult

Return the Bayesian signed-rank test of paired differences.

The cost grows as samples * (n + 1)^2.

Parameters:

Name Type Description Default
differences Iterable[float]

One finite difference per matched pair.

required
rope float

Half-width of the region of practical equivalence, at least 0.

0.0
prior float

Weight of the prior pseudo-observation at zero, positive.

0.5
samples int

Monte Carlo draws, positive.

2000
seed int

Seed of the draws, at least 0.

0

Returns:

Type Description
BayesianResult

Monte Carlo posterior means and probabilities.

Raises:

Type Description
ConfigError

If there is no difference, a difference is not finite, or a parameter is out of range.

dispatchatlas.analysis.stats.corrections

Multiple-comparison corrections of a family of p-values.

adjust_p_values() returns adjusted p-values: a hypothesis is rejected at family-wise level alpha when its adjusted p-value is at most alpha. With m hypotheses and the p-values sorted ascending as p_(1) <= ... <= p_(m):

  • Bonferroni (Dunn 1961): min(1, m p).
  • Holm (1979), step-down: the running maximum of min(1, (m - i + 1) p_(i)).
  • Hochberg (1988), step-up: from the largest p-value down, the running minimum of min(1, (m - i + 1) p_(i)).
  • Shaffer (1986), static step-down for all pairwise comparisons of k groups: Holm's multiplier is replaced by t_i, the largest number of hypotheses that can be true at once and is at most m - i + 1. The possible numbers of true hypotheses are S(k), with S(0) = S(1) = {0} and S(k) the union over j = 1..k of C(j, 2) + S(k - j).
  • Finner (1993), step-down: the running maximum of min(1, 1 - (1 - p_(i))^(m / i)).

Equal p-values are ordered by their keys, when given, and then by position, so the result never depends on the incidental order of the input.

Correction

Correction = Literal['none', 'bonferroni', 'holm', 'hochberg', 'shaffer', 'finner']

A multiple-comparison correction.

adjust_p_values

adjust_p_values(p_values: Sequence[float], method: Correction = 'holm', *, keys: Sequence[tuple[str, str]] | None = None, groups: int | None = None) -> tuple[float, ...]

Return the adjusted p-values of a family, in the order of the input.

Parameters:

Name Type Description Default
p_values Sequence[float]

One p-value per hypothesis, each in [0, 1].

required
method Correction

The correction.

'holm'
keys Sequence[tuple[str, str]] | None

For pairwise comparisons, the two group names of each hypothesis. They break ties between equal p-values and, for Shaffer's correction, give the number of groups.

None
groups int | None

For Shaffer's correction, the number of groups k; the family must hold all k (k - 1) / 2 pairs. Defaults to the number of distinct names in keys.

None

Raises:

Type Description
ConfigError

For an unknown method, a p-value outside [0, 1], keys that do not match the p-values, or a Shaffer family that is not complete.

shaffer_hypothesis_counts

shaffer_hypothesis_counts(groups: int) -> tuple[int, ...]

Return S(k): the numbers of pairwise hypotheses that can be true at once.

Splitting k groups into blocks of equal groups makes the pairs within each block true, so S(k) is the union over the size j of one block of C(j, 2) + S(k - j) (Shaffer 1986).

Raises:

Type Description
ConfigError

If groups is negative or not an integer.

dispatchatlas.analysis.stats.power

Power of the paired Wilcoxon signed-rank test, and the sample size it needs.

The pairwise comparisons test one paired difference per problem, so their sample size is the number of problems two solvers share, not the number of runs. Two questions follow, one for reading a campaign and one for planning it (Campelo and Takahashi 2019): which standardized effect can n pairs detect, and how many pairs does an effect need?

Both are answered for the test this package applies, not for an asymptotic stand-in: wilcoxon_power() simulates paired differences drawn from a normal distribution with mean effect and standard deviation 1, so the effect is the paired Cohen's d_z, and counts how often the two-sided test rejects at alpha. It uses the exact null distribution up to WILCOXON_EXACT_LIMIT pairs and the continuity-corrected normal approximation above, as wilcoxon_signed_rank() does. The draws come from numpy.random.default_rng(seed), so results are reproducible, and every effect is evaluated on the same draws, which keeps the simulated power monotone enough to search.

With n pairs, the smallest two-sided exact p-value is 2 / 2^n, so below attainable_floor() no data can reach significance at alpha, whatever the effect. detectable_effect_size() refuses such counts instead of reporting a detectable effect that does not exist.

PowerAssessment dataclass

PowerAssessment(pairs: int, alpha: float, power: float, effect: float, replicates: int, seed: int)

The smallest standardized effect a paired design detects.

Attributes:

Name Type Description
pairs int

Number of paired observations, such as shared problems.

alpha float

Two-sided significance level.

power float

Target probability of rejecting the null hypothesis.

effect float

Smallest paired Cohen's d_z detected with that power.

replicates int

Simulated samples per power estimate.

seed int

Seed of the simulation.

attainable_floor

attainable_floor(alpha: float = 0.05) -> int

Return the fewest pairs with which a two-sided Wilcoxon test can reject at alpha.

With n pairs the smallest attainable p-value is 2 / 2^n, reached when every difference has the same sign. At alpha = 0.05 the floor is 6 pairs (2 / 64 = 0.031); 5 pairs cannot go below 0.0625.

Raises:

Type Description
ConfigError

If alpha is not strictly between 0 and 1.

wilcoxon_power

wilcoxon_power(pairs: int, effect: float, *, alpha: float = 0.05, replicates: int = 10000, seed: int = 0) -> float

Return the simulated power of the two-sided Wilcoxon signed-rank test.

Parameters:

Name Type Description Default
pairs int

Number of paired differences, at least 1.

required
effect float

Mean of the normal differences, in standard deviations.

required
alpha float

Two-sided significance level.

0.05
replicates int

Simulated samples, at least 1.

10000
seed int

Seed of the simulation, at least 0.

0

Returns:

Type Description
float

The share of simulated samples the test rejects.

Raises:

Type Description
ConfigError

If an argument is out of range.

detectable_effect_size

detectable_effect_size(pairs: int, *, alpha: float = 0.05, power: float = 0.8, replicates: int = 10000, seed: int = 0) -> PowerAssessment

Return the smallest paired d_z that pairs pairs detect with the given power.

The effect is found by bisection on wilcoxon_power(), evaluated on one fixed set of simulated samples.

Parameters:

Name Type Description Default
pairs int

Number of paired observations.

required
alpha float

Two-sided significance level.

0.05
power float

Target power, strictly between 0 and 1.

0.8
replicates int

Simulated samples per power estimate.

10000
seed int

Seed of the simulation.

0

Raises:

Type Description
ConfigError

If pairs is below attainable_floor(), where no effect is detectable, or an argument is out of range.

required_paired_count

required_paired_count(effect: float, *, alpha: float = 0.05, power: float = 0.8, replicates: int = 10000, seed: int = 0) -> int

Return the number of pairs that detects a paired d_z with the given power.

Starting from attainable_floor(), the count doubles until the simulated power reaches the target, and bisection then finds the smallest count that reaches it.

Parameters:

Name Type Description Default
effect float

The paired Cohen's d_z to detect, positive.

required
alpha float

Two-sided significance level.

0.05
power float

Target power, strictly between 0 and 1.

0.8
replicates int

Simulated samples per power estimate.

10000
seed int

Seed of the simulation.

0

Raises:

Type Description
ConfigError

If an argument is out of range.