Statistics¶
The statistical kernels: distributions, ranks, rank tests, effect sizes, bootstrap intervals, Bayesian tests, corrections, and power.
dispatchatlas.analysis.stats
¶
Statistical kernels: plain numbers in, small frozen results out.
The kernels know nothing about campaigns. The rank tests, effect sizes, and
corrections are pure Python and verified against reference values computed
with SciPy; the bootstrap, the Bayesian tests, and the power simulation draw
from numpy.random.default_rng(seed).
distributions: normal, chi-square, and F tails.ranks: ties, midranks, and orientation.nonparametric: Wilcoxon, sign, Mann-Whitney, Friedman, Iman-Davenport, and Nemenyi.effect_sizes: A12, Cliff's delta, Cohen's d and d_z, and the paired win share.bootstrap: BCa and percentile intervals.bayesian: Bayesian sign and signed-rank tests.corrections: multiple-comparison corrections.power: power and sample size of the paired test.
dispatchatlas.analysis.stats.distributions
¶
The normal, chi-square, and F distribution functions, in pure Python.
The rank tests read their p-values from these distributions: the normal for large-sample Wilcoxon and Mann-Whitney statistics, the chi-square for the Friedman statistic, and the F for the Iman-Davenport statistic.
- The normal tails use
math.erfc(), so a tail probability keeps its relative precision far into the tail instead of cancelling against one. - The normal quantile is P. J. Acklam's rational approximation, with a relative error below 1.15e-9 on the open unit interval. It serves critical values and confidence levels, never the inversion of a reported p-value.
- The chi-square tail is the regularized upper incomplete gamma function
Q(df/2, x/2): a power series below the transitionx < a + 1and the modified Lentz continued fraction above it (Press et al. 2007, section 6.2). - The F tail is the regularized incomplete beta function
I_{d2/(d2 + d1 f)}(d2/2, d1/2), evaluated by the modified Lentz continued fraction on the side of its symmetry point where it converges (Press et al. 2007, section 6.4).
Both continued fractions and the series stop at relative convergence (1e-15
for the gamma function, 3e-14 for the beta function) and raise
ArithmeticError rather than return a partial sum when an iteration
cap is reached. The caps grow with the square root of the shape parameters,
the rate at which the expansions converge near the distributions' centres.
normal_cdf
¶
Return the standard normal distribution function P(Z <= z).
normal_sf
¶
Return the standard normal upper tail P(Z > z).
Unlike 1 - normal_cdf(z), it stays accurate when the tail is far
below machine epsilon.
normal_ppf
¶
Return the standard normal quantile by Acklam's rational approximation.
The central region [0.02425, 0.97575] uses a rational function of
(p - 1/2)^2, and the tails a rational function of
sqrt(-2 log p). The relative error is below 1.15e-9.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
probability
|
float
|
A probability strictly between 0 and 1. |
required |
Returns:
| Type | Description |
|---|---|
float
|
|
Raises:
| Type | Description |
|---|---|
ConfigError
|
If |
chi2_sf
¶
Return the chi-square upper tail P(X > statistic).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
statistic
|
float
|
The observed statistic. Non-positive values give 1 and positive infinity gives 0. |
required |
df
|
float
|
Degrees of freedom, positive. |
required |
Returns:
| Type | Description |
|---|---|
float
|
The tail probability. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If |
ArithmeticError
|
If the incomplete gamma function fails to converge. |
f_sf
¶
Return the F-distribution upper tail P(F > statistic).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
statistic
|
float
|
The observed statistic. Non-positive values give 1 and positive infinity gives 0. |
required |
df_numerator
|
float
|
Numerator degrees of freedom |
required |
df_denominator
|
float
|
Denominator degrees of freedom |
required |
Returns:
| Type | Description |
|---|---|
float
|
|
Raises:
| Type | Description |
|---|---|
ConfigError
|
If a degree of freedom is not a positive finite number or the statistic is NaN. |
ArithmeticError
|
If the incomplete beta function fails to converge. |
dispatchatlas.analysis.stats.ranks
¶
Ranks, ties, and the orientation of objective values.
Every rank-based statistic in this package opens the same way: replace values
by their ranks, give tied values the mean of the positions they span
(midranks), and orient the ranks so that rank 1 is the best value. Two values
tie when they are equal within TIE_TOLERANCE, relatively or
absolutely, so a difference left by floating-point summation is not read as a
win. Infinite values are allowed and rank last or first: the analysis
pipeline scores a failed run as the worst possible value.
TIE_TOLERANCE
module-attribute
¶
Two values tie when they differ by at most this much, relatively or absolutely.
finite_or_infinite
¶
Return values as floats, refusing NaN.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
values
|
Iterable[float]
|
The values. |
required |
what
|
str
|
Name of the quantity, for the error message. |
required |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If a value is NaN or not a number. |
ties
¶
Return whether two values are equal within TIE_TOLERANCE.
Equal infinities tie; infinities of opposite sign do not.
strictly_better
¶
strictly_better(left: float, right: float, sense: Sense) -> bool
Return whether left beats right under sense, ties excluded.
midranks
¶
Return ascending ranks, from 1, with tied values sharing their mean rank.
Values are sorted, and each following value joins the current tie group while it ties with the group's first value, so groups never chain. The ranks are returned in the order of the input.
oriented_ranks
¶
oriented_ranks(values: Sequence[float], sense: Sense) -> tuple[float, ...]
Return midranks oriented so that rank 1 is the best value under sense.
rank_block
¶
rank_block(block: Mapping[str, float], sense: Sense) -> dict[str, float]
Rank the entries of one block, such as the solvers on one problem.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
block
|
Mapping[str, float]
|
Value of each entry. |
required |
sense
|
Sense
|
Whether smaller or larger values are better. |
required |
Returns:
| Type | Description |
|---|---|
dict[str, float]
|
The oriented midrank of each entry, keyed like |
dispatchatlas.analysis.stats.nonparametric
¶
Rank tests: Wilcoxon, sign, Mann-Whitney, Friedman, Iman-Davenport, and Nemenyi.
The tests follow the non-parametric protocol for comparing algorithms over several problems (Demšar 2006; Derrac et al. 2011): a paired test for two algorithms, and for several algorithms an omnibus test over complete blocks followed by a post-hoc comparison of average ranks.
All tests are two-sided. Small samples use the exact null distribution: the
signed-rank test with or without tied magnitudes, where ties make it the
distribution conditional on the midranks, and the Mann-Whitney test without
ties. Larger samples, and tied Mann-Whitney samples, use the normal
approximation with the tie-corrected variance and a continuity correction of
0.5, whose tail comes from
normal_sf() so that very
small p-values keep their precision. Rank ties follow
TIE_TOLERANCE, and so do the
variance corrections: tie_sum() counts the groups that the midranks
share.
WILCOXON_EXACT_LIMIT
module-attribute
¶
Largest number of non-zero pairs for which the exact Wilcoxon null is used.
The subset-sum recursion costs about n * n (n + 1) additions, 130 thousand
at the limit, and the counts of sign assignments stay below 2^53, so their
conversion to floats is exact.
MANN_WHITNEY_EXACT_LIMIT
module-attribute
¶
Largest size of either sample for which the exact Mann-Whitney null is used.
TestMethod
¶
How a p-value was obtained.
"exact" is the exact null distribution of untied ranks,
"exact-conditional" the exact null distribution of the signed-rank
statistic given the midranks of tied magnitudes, "normal" the normal
approximation, and "degenerate" a test without data.
WilcoxonResult
dataclass
¶
WilcoxonResult(statistic: float, p_value: float, pairs: int, zeros: int, method: TestMethod)
A two-sided Wilcoxon signed-rank test.
Attributes:
| Name | Type | Description |
|---|---|---|
statistic |
float
|
|
p_value |
float
|
Two-sided p-value. |
pairs |
int
|
Number of non-zero differences the test ranked. |
zeros |
int
|
Number of differences dropped as zero. |
method |
TestMethod
|
How the p-value was obtained. |
MannWhitneyResult
dataclass
¶
MannWhitneyResult(statistic: float, p_value: float, first_size: int, second_size: int, method: TestMethod)
A two-sided Mann-Whitney U test.
Attributes:
| Name | Type | Description |
|---|---|---|
statistic |
float
|
|
p_value |
float
|
Two-sided p-value. |
first_size |
int
|
Size of the first sample. |
second_size |
int
|
Size of the second sample. |
method |
TestMethod
|
How the p-value was obtained. |
FriedmanResult
dataclass
¶
FriedmanResult(statistic: float, p_value: float, df: int, blocks: int, groups: int, average_ranks: tuple[tuple[str, float], ...])
A Friedman test over complete blocks.
Attributes:
| Name | Type | Description |
|---|---|---|
statistic |
float
|
The tie-corrected chi-square statistic |
p_value |
float
|
Upper-tail chi-square probability of |
df |
int
|
Degrees of freedom, the number of groups minus one. |
blocks |
int
|
Number of blocks |
groups |
int
|
Number of groups |
average_ranks |
tuple[tuple[str, float], ...]
|
Each group's mean rank over the blocks, rank 1 best, in the order of the first block's keys. |
kendalls_w
property
¶
Kendall's coefficient of concordance W = Q / (N (k - 1)), in [0, 1].
It is the effect size of the Friedman test: 0 when the rankings disagree completely, 1 when every block ranks the groups alike (Kendall and Babington Smith 1939).
ImanDavenportResult
dataclass
¶
The Iman-Davenport F statistic derived from a Friedman result.
Attributes:
| Name | Type | Description |
|---|---|---|
statistic |
float
|
|
p_value |
float
|
Upper-tail F probability of the statistic. |
df_numerator |
int
|
|
df_denominator |
int
|
|
wilcoxon_signed_rank
¶
wilcoxon_signed_rank(differences: Iterable[float]) -> WilcoxonResult
Return the two-sided Wilcoxon signed-rank test of paired differences.
Differences within TIE_TOLERANCE of zero are dropped (Wilcoxon's
treatment of zeros), the magnitudes of the others are midranked, and the
statistic is the smaller of the positive and negative rank sums. Up to
WILCOXON_EXACT_LIMIT pairs, the p-value comes from the exact
null distribution of the statistic: under the null hypothesis each
difference is positive or negative with equal probability, so every
assignment of signs to the ranks is equally likely, and a subset-sum
recursion counts the assignments by their rank sum. With tied
magnitudes the ranks are the midranks, and the distribution is the one
conditional on them (method "exact-conditional"), whose smallest
p-value over n pairs is 2 / 2^n as without ties; a normal
approximation can fall below that on a small tied sample. Above the
limit the normal
approximation uses the mean n(n+1)/4, the variance
n(n+1)(2n+1)/24 - sum(t^3 - t)/48 over the tie groups, and a
continuity correction of 0.5 (Wilcoxon 1945). Infinite differences are
allowed and rank above every finite one.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
differences
|
Iterable[float]
|
One difference per matched pair. |
required |
Returns:
| Type | Description |
|---|---|
WilcoxonResult
|
The test, whose |
WilcoxonResult
|
no non-zero difference, a degenerate result with p-value 1. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If there are no differences or one is NaN. |
sign_test
¶
Return the exact two-sided sign test p-value.
Under the null hypothesis each untied pair is a win with probability 1/2, so the p-value is twice the binomial tail of the smaller count, capped at 1 (Dixon and Mood 1946). Ties carry no direction and are left out by the caller.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
wins
|
int
|
Pairs the first sample won. |
required |
losses
|
int
|
Pairs the first sample lost. |
required |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If a count is negative or not an integer. |
mann_whitney_u
¶
mann_whitney_u(first: Iterable[float], second: Iterable[float]) -> MannWhitneyResult
Return the two-sided Mann-Whitney U test of two independent samples.
The pooled sample is midranked and U1 is the first sample's rank sum
minus n1(n1+1)/2; the statistic is min(U1, n1 n2 - U1). When
neither sample exceeds MANN_WHITNEY_EXACT_LIMIT and no values tie,
the p-value comes from the exact null, which the recursion of Mann and
Whitney (1947) counts. Otherwise the normal approximation uses the mean
n1 n2 / 2, the variance
n1 n2 / 12 * ((N + 1) - sum(t^3 - t) / (N (N - 1))), and a continuity
correction of 0.5.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
first
|
Iterable[float]
|
The first sample. |
required |
second
|
Iterable[float]
|
The second sample. |
required |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If a sample is empty or holds NaN. |
friedman_test
¶
friedman_test(blocks: Sequence[Mapping[str, float]], sense: Sense = 'min') -> FriedmanResult
Return the Friedman test of several groups over complete blocks.
Each block, such as one problem, maps every group, such as a solver, to
its value. Values are midranked within each block with rank 1 for the
best, and the statistic
Q = [12 / (N k (k+1)) sum_j R_j^2 - 3 N (k+1)] / [1 - sum(t^3 - t) / (N k (k^2 - 1))]
divides the rank-sum dispersion by the tie correction; it is read against
a chi-square distribution with k - 1 degrees of freedom (Friedman
1937). Infinite values are allowed, for example for failed runs.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
blocks
|
Sequence[Mapping[str, float]]
|
One mapping per block, all with the same keys. |
required |
sense
|
Sense
|
Whether smaller or larger values are better. |
'min'
|
Raises:
| Type | Description |
|---|---|
ConfigError
|
With fewer than two blocks or two groups, blocks that disagree on their keys, or a NaN value. |
friedman_statistic
¶
Return the tie-corrected Friedman statistic from the rank sums of each group.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
rank_sums
|
Sequence[float]
|
The sum over blocks of each group's rank. |
required |
blocks
|
int
|
The number of blocks |
required |
ties
|
float
|
|
0.0
|
Returns:
| Type | Description |
|---|---|
float
|
|
float
|
when every block ties every group, the uncorrected statistic. |
iman_davenport
¶
iman_davenport(friedman: FriedmanResult) -> ImanDavenportResult
Return the Iman-Davenport F statistic of a Friedman result.
F = (N - 1) Q / (N (k - 1) - Q) with k - 1 and (k - 1)(N - 1)
degrees of freedom is less conservative than the chi-square reading of
Q (Iman and Davenport 1980; Demšar 2006). When every block ranks the
groups alike the denominator vanishes, and the statistic is infinite with
p-value 0.
nemenyi_q
¶
Return the Nemenyi critical value q_alpha for groups groups.
Tabulated for 2 to 12 groups at alpha 0.05 and 0.10 (Demšar 2006,
Table 5(a)).
Raises:
| Type | Description |
|---|---|
ConfigError
|
For another number of groups or another |
nemenyi_critical_difference
¶
Return the Nemenyi critical difference of average ranks.
Two groups differ significantly when their average ranks over the
blocks differ by more than q_alpha sqrt(k (k + 1) / (6 N))
(Nemenyi 1963; Demšar 2006).
Raises:
| Type | Description |
|---|---|
ConfigError
|
With fewer than one block, or when |
tie_sum
¶
tie_sum(values: Sequence[float], sense: Sense = 'min') -> float
Return sum(t^3 - t) over the groups t of tied values.
This is the tie term of the Wilcoxon, Mann-Whitney, and Friedman
variance corrections. Values tie when they are equal within
TIE_TOLERANCE: the groups are
the ones that share a midrank in
oriented_ranks(), so the term
always agrees with the ranks the tests use.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
values
|
Sequence[float]
|
The values of one sample or block. |
required |
sense
|
Sense
|
Whether smaller or larger values are better, the orientation in which the ranks are formed. |
'min'
|
Raises:
| Type | Description |
|---|---|
ConfigError
|
If |
dispatchatlas.analysis.stats.effect_sizes
¶
Effect sizes for paired and independent comparisons.
A p-value says whether a difference is detectable; an effect size says how large it is. Every function here is oriented so that a positive direction favours the left (first) sample under the objective sense: an A12 above 0.5, a positive Cliff's delta or Cohen's d, a win share above 0.5.
Two designs need different estimands:
- Independent samples, such as the runs of two solvers on one problem: Vargha and Delaney's A12, Cliff's delta, and Cohen's d_s compare every left value with every right value.
- Paired samples, such as one value per problem for each of two solvers: the paired win share (the share of pairs the left sample wins, ties counting one half) and Cohen's d_z (the mean paired difference over its standard deviation). The independent forms do not apply here: their cross pairs compare different problems, so they measure how spread out the problems are rather than how the samples differ.
The independent-sample effect sizes come with the magnitude labels their sources publish. The paired ones carry no label: the published cutoffs were calibrated for other estimands.
Magnitude
¶
A conventional label for the size of an effect.
vargha_delaney_a12
¶
vargha_delaney_a12(left: Iterable[float], right: Iterable[float], sense: Sense = 'min') -> float
Return Vargha and Delaney's A12 for two independent samples.
A12 is the probability that a random left value beats a random right value, ties counting one half (Vargha and Delaney 2000).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
left
|
Iterable[float]
|
The first sample. |
required |
right
|
Iterable[float]
|
The second sample. |
required |
sense
|
Sense
|
Whether smaller or larger values are better. |
'min'
|
Returns:
| Type | Description |
|---|---|
float
|
A12 in |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If a sample is empty or holds NaN. |
cliffs_delta
¶
cliffs_delta(left: Iterable[float], right: Iterable[float], sense: Sense = 'min') -> float
Return Cliff's delta for two independent samples.
Delta is the share of cross pairs the left sample wins minus the share it
loses (Cliff 1993); it equals 2 A12 - 1.
Returns:
| Type | Description |
|---|---|
float
|
Delta in |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If a sample is empty or holds NaN. |
cohens_d
¶
cohens_d(left: Iterable[float], right: Iterable[float], sense: Sense = 'min') -> float
Return Cohen's d_s for two independent samples.
The difference of the means divided by the pooled standard deviation
sqrt((SS_left + SS_right) / (n_left + n_right - 2)) (Cohen 1988),
without the small-sample correction of Hedges' g.
Returns:
| Type | Description |
|---|---|
float
|
d_s, positive when the left mean is better; 0 when both samples are |
float
|
constant. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If a sample has fewer than two values or a value is not finite. |
cohens_dz
¶
Return Cohen's d_z, the mean of paired differences over their standard deviation.
The standard deviation uses the n - 1 denominator. Orient the
differences before calling, for example as the advantage of the left
sample, so that a positive d_z favours it.
Returns:
| Type | Description |
|---|---|
float | None
|
d_z, or |
float | None
|
difference that is not finite, or differences without spread. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If a difference is NaN. |
paired_outcomes
¶
paired_outcomes(left: Sequence[float], right: Sequence[float], sense: Sense = 'min') -> tuple[int, int, int]
Count the pairs the left sample wins, loses, and ties.
Two values tie when they are equal within
TIE_TOLERANCE.
Returns:
| Type | Description |
|---|---|
tuple[int, int, int]
|
|
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the samples differ in length or hold NaN. |
paired_win_share
¶
paired_win_share(left: Sequence[float], right: Sequence[float], sense: Sense = 'min') -> float
Return the share of pairs the left sample wins, ties counting one half.
This is the sign-test counterpart of A12 for a paired design: 1 when the left sample wins every pair, 0.5 when wins and losses balance.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the samples are empty, differ in length, or hold NaN. |
a12_magnitude
¶
a12_magnitude(a12: float) -> Magnitude
Return the Vargha-Delaney label of an A12: cutoffs 0.56, 0.64, and 0.71.
An A12 below 0.5 is labelled by its mirror image 1 - A12, which keeps
the published cutoffs exact rather than subject to the rounding of
A12 - 0.5.
dispatchatlas.analysis.stats.bootstrap
¶
Bootstrap confidence intervals: bias-corrected and accelerated, and percentile.
bootstrap_interval() resamples the observations with replacement,
computes the statistic on every resample, and reads the interval from the
sorted replicates with the linear-interpolation quantile quantile()
(type 7 of Hyndman and Fan 1996).
The percentile interval takes the replicate quantiles at alpha/2 and
1 - alpha/2. The bias-corrected and accelerated (BCa) interval (Efron
1987) shifts those levels by the bias correction z0 = Phi^-1(#{replicates <
estimate} / B), with a strict inequality, and by the acceleration
a = sum(d^3) / (6 (sum(d^2))^(3/2)) from the leave-one-out jackknife
deviations d, and reads the quantiles at
Phi(z0 + (z0 + z) / (1 - a (z0 + z))) for z = Phi^-1(alpha/2) and
Phi^-1(1 - alpha/2), clamped to [1e-9, 1 - 1e-9]. When every replicate
lies on one side of the estimate, z0 is infinite and the interval falls
back to the percentile one; the same happens when 1 - a (z0 + z) is not
positive, where the correction is undefined. A sample of one value, or zero
resamples, gives the degenerate interval at the estimate.
Resampling draws indices from numpy.random.default_rng(seed), one row of
n indices per resample, so equal inputs and seed give equal intervals for
a given NumPy version.
Statistic
¶
A statistic: the mean, the median, or a function of the resampled rows.
BootstrapInterval
dataclass
¶
BootstrapInterval(estimate: float, low: float, high: float, level: float, resamples: int, method: Literal['bca', 'percentile', 'degenerate'])
A bootstrap confidence interval.
Attributes:
| Name | Type | Description |
|---|---|---|
estimate |
float
|
The statistic of the observed sample. |
low |
float
|
Lower bound. |
high |
float
|
Upper bound. |
level |
float
|
Nominal coverage, for example 0.95. |
resamples |
int
|
Number of resamples drawn. |
method |
Literal['bca', 'percentile', 'degenerate']
|
The interval computed: |
quantile
¶
Return the q quantile of sorted values by linear interpolation.
The position (n - 1) q interpolates between the neighbouring order
statistics, type 7 of Hyndman and Fan (1996) and NumPy's default.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ordered
|
Sequence[float]
|
Values sorted in ascending order. |
required |
q
|
float
|
A probability in |
required |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If |
bootstrap_interval
¶
bootstrap_interval(sample: Sequence[float] | Sequence[Sequence[float]] | NDArray[float64], *, statistic: Statistic = 'mean', level: float = 0.95, resamples: int = 1000, seed: int = 0, method: IntervalMethod = 'bca') -> BootstrapInterval
Return a bootstrap confidence interval of a statistic.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample
|
Sequence[float] | Sequence[Sequence[float]] | NDArray[float64]
|
The observations: numbers, or rows of numbers when the statistic is a function of several columns. Resampling draws whole rows. |
required |
statistic
|
Statistic
|
|
'mean'
|
level
|
float
|
Nominal coverage, strictly between 0 and 1. |
0.95
|
resamples
|
int
|
Number of resamples, at least 0. |
1000
|
seed
|
int
|
Seed of the resampling generator, at least 0. |
0
|
method
|
IntervalMethod
|
|
'bca'
|
Returns:
| Type | Description |
|---|---|
BootstrapInterval
|
The interval and the estimate on the observed sample. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If the sample is empty or not finite, or a parameter is out of range. |
dispatchatlas.analysis.stats.bayesian
¶
Bayesian sign and signed-rank tests over paired differences.
Both tests of Benavoli, Corani, Demšar and Zaffalon (2017) put a posterior on
the probability mass of three regions of a paired difference: below the
region of practical equivalence (the rope, [-rope, rope], bounds
included), inside it, and above it.
- The sign test counts the differences in each region. The posterior of
the three masses is
Dirichlet(n_below, n_rope + prior, n_above): the prior's pseudo-observations sit in the rope. - The signed-rank test puts a Dirichlet process on the distribution of
the differences, with a prior pseudo-observation at zero of weight
prior. Each posterior draw weighs the atoms0, d_1, ..., d_nbyDirichlet(prior, 1, ..., 1)weightsw, and a region's mass is the total weightw_i w_jof the ordered pairs whose Walsh average(z_i + z_j) / 2falls in it.
Each result reports two things per region. mean_* is the posterior
expected mass. probability_* is the posterior probability that the region
holds the largest of the three masses, the decision quantity of the authors'
reference implementation baycomp, estimated from samples Monte Carlo
draws of numpy.random.default_rng(seed). For the sign test the expected
masses are exact; for the signed-rank test they are Monte Carlo averages of
the same draws.
BayesianResult
dataclass
¶
BayesianResult(probability_below: float, probability_rope: float, probability_above: float, mean_below: float, mean_rope: float, mean_above: float, rope: float, prior: float, pairs: int, samples: int, seed: int)
Posterior facts about the three regions of a paired difference.
Attributes:
| Name | Type | Description |
|---|---|---|
probability_below |
float
|
Posterior probability that the region below
|
probability_rope |
float
|
Same, for the rope. |
probability_above |
float
|
Same, for the region above |
mean_below |
float
|
Posterior expected mass below |
mean_rope |
float
|
Posterior expected mass of the rope. |
mean_above |
float
|
Posterior expected mass above |
rope |
float
|
Half-width of the rope. |
prior |
float
|
Strength of the prior. |
pairs |
int
|
Number of differences. |
samples |
int
|
Monte Carlo draws. |
seed |
int
|
Seed of the draws. |
bayesian_sign_test
¶
bayesian_sign_test(differences: Iterable[float], *, rope: float = 0.0, prior: float = 1.0, samples: int = 2000, seed: int = 0) -> BayesianResult
Return the Bayesian sign test of paired differences.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
differences
|
Iterable[float]
|
One difference per matched pair; infinities are allowed. |
required |
rope
|
float
|
Half-width of the region of practical equivalence, at least 0. |
0.0
|
prior
|
float
|
Pseudo-observations the prior places in the rope, positive. |
1.0
|
samples
|
int
|
Monte Carlo draws for the probabilities, positive. |
2000
|
seed
|
int
|
Seed of the draws, at least 0. |
0
|
Returns:
| Type | Description |
|---|---|
BayesianResult
|
Exact posterior means and Monte Carlo probabilities. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If there is no difference, a difference is NaN, or a parameter is out of range. |
bayesian_signed_rank_test
¶
bayesian_signed_rank_test(differences: Iterable[float], *, rope: float = 0.0, prior: float = 0.5, samples: int = 2000, seed: int = 0) -> BayesianResult
Return the Bayesian signed-rank test of paired differences.
The cost grows as samples * (n + 1)^2.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
differences
|
Iterable[float]
|
One finite difference per matched pair. |
required |
rope
|
float
|
Half-width of the region of practical equivalence, at least 0. |
0.0
|
prior
|
float
|
Weight of the prior pseudo-observation at zero, positive. |
0.5
|
samples
|
int
|
Monte Carlo draws, positive. |
2000
|
seed
|
int
|
Seed of the draws, at least 0. |
0
|
Returns:
| Type | Description |
|---|---|
BayesianResult
|
Monte Carlo posterior means and probabilities. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If there is no difference, a difference is not finite, or a parameter is out of range. |
dispatchatlas.analysis.stats.corrections
¶
Multiple-comparison corrections of a family of p-values.
adjust_p_values() returns adjusted p-values: a hypothesis is rejected
at family-wise level alpha when its adjusted p-value is at most alpha.
With m hypotheses and the p-values sorted ascending as
p_(1) <= ... <= p_(m):
- Bonferroni (Dunn 1961):
min(1, m p). - Holm (1979), step-down: the running maximum of
min(1, (m - i + 1) p_(i)). - Hochberg (1988), step-up: from the largest p-value down, the running
minimum of
min(1, (m - i + 1) p_(i)). - Shaffer (1986), static step-down for all pairwise comparisons of
kgroups: Holm's multiplier is replaced byt_i, the largest number of hypotheses that can be true at once and is at mostm - i + 1. The possible numbers of true hypotheses areS(k), withS(0) = S(1) = {0}andS(k)the union overj = 1..kofC(j, 2) + S(k - j). - Finner (1993), step-down: the running maximum of
min(1, 1 - (1 - p_(i))^(m / i)).
Equal p-values are ordered by their keys, when given, and then by position, so the result never depends on the incidental order of the input.
Correction
¶
A multiple-comparison correction.
adjust_p_values
¶
adjust_p_values(p_values: Sequence[float], method: Correction = 'holm', *, keys: Sequence[tuple[str, str]] | None = None, groups: int | None = None) -> tuple[float, ...]
Return the adjusted p-values of a family, in the order of the input.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
p_values
|
Sequence[float]
|
One p-value per hypothesis, each in |
required |
method
|
Correction
|
The correction. |
'holm'
|
keys
|
Sequence[tuple[str, str]] | None
|
For pairwise comparisons, the two group names of each hypothesis. They break ties between equal p-values and, for Shaffer's correction, give the number of groups. |
None
|
groups
|
int | None
|
For Shaffer's correction, the number of groups |
None
|
Raises:
| Type | Description |
|---|---|
ConfigError
|
For an unknown method, a p-value outside |
shaffer_hypothesis_counts
¶
Return S(k): the numbers of pairwise hypotheses that can be true at once.
Splitting k groups into blocks of equal groups makes the pairs within
each block true, so S(k) is the union over the size j of one block
of C(j, 2) + S(k - j) (Shaffer 1986).
Raises:
| Type | Description |
|---|---|
ConfigError
|
If |
dispatchatlas.analysis.stats.power
¶
Power of the paired Wilcoxon signed-rank test, and the sample size it needs.
The pairwise comparisons test one paired difference per problem, so their
sample size is the number of problems two solvers share, not the number of
runs. Two questions follow, one for reading a campaign and one for planning
it (Campelo and Takahashi 2019): which standardized effect can n pairs
detect, and how many pairs does an effect need?
Both are answered for the test this package applies, not for an asymptotic
stand-in: wilcoxon_power() simulates paired differences drawn from a
normal distribution with mean effect and standard deviation 1, so the
effect is the paired Cohen's d_z, and counts how often the two-sided test
rejects at alpha. It uses the exact null distribution up to
WILCOXON_EXACT_LIMIT
pairs and the continuity-corrected normal approximation above, as
wilcoxon_signed_rank() does.
The draws come from numpy.random.default_rng(seed), so results are
reproducible, and every effect is evaluated on the same draws, which keeps
the simulated power monotone enough to search.
With n pairs, the smallest two-sided exact p-value is 2 / 2^n, so
below attainable_floor() no data can reach significance at alpha,
whatever the effect. detectable_effect_size() refuses such counts
instead of reporting a detectable effect that does not exist.
PowerAssessment
dataclass
¶
The smallest standardized effect a paired design detects.
Attributes:
| Name | Type | Description |
|---|---|---|
pairs |
int
|
Number of paired observations, such as shared problems. |
alpha |
float
|
Two-sided significance level. |
power |
float
|
Target probability of rejecting the null hypothesis. |
effect |
float
|
Smallest paired Cohen's d_z detected with that power. |
replicates |
int
|
Simulated samples per power estimate. |
seed |
int
|
Seed of the simulation. |
attainable_floor
¶
Return the fewest pairs with which a two-sided Wilcoxon test can reject at alpha.
With n pairs the smallest attainable p-value is 2 / 2^n, reached
when every difference has the same sign. At alpha = 0.05 the floor
is 6 pairs (2 / 64 = 0.031); 5 pairs cannot go below 0.0625.
Raises:
| Type | Description |
|---|---|
ConfigError
|
If |
wilcoxon_power
¶
wilcoxon_power(pairs: int, effect: float, *, alpha: float = 0.05, replicates: int = 10000, seed: int = 0) -> float
Return the simulated power of the two-sided Wilcoxon signed-rank test.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pairs
|
int
|
Number of paired differences, at least 1. |
required |
effect
|
float
|
Mean of the normal differences, in standard deviations. |
required |
alpha
|
float
|
Two-sided significance level. |
0.05
|
replicates
|
int
|
Simulated samples, at least 1. |
10000
|
seed
|
int
|
Seed of the simulation, at least 0. |
0
|
Returns:
| Type | Description |
|---|---|
float
|
The share of simulated samples the test rejects. |
Raises:
| Type | Description |
|---|---|
ConfigError
|
If an argument is out of range. |
detectable_effect_size
¶
detectable_effect_size(pairs: int, *, alpha: float = 0.05, power: float = 0.8, replicates: int = 10000, seed: int = 0) -> PowerAssessment
Return the smallest paired d_z that pairs pairs detect with the given power.
The effect is found by bisection on wilcoxon_power(), evaluated on
one fixed set of simulated samples.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pairs
|
int
|
Number of paired observations. |
required |
alpha
|
float
|
Two-sided significance level. |
0.05
|
power
|
float
|
Target power, strictly between 0 and 1. |
0.8
|
replicates
|
int
|
Simulated samples per power estimate. |
10000
|
seed
|
int
|
Seed of the simulation. |
0
|
Raises:
| Type | Description |
|---|---|
ConfigError
|
If |
required_paired_count
¶
required_paired_count(effect: float, *, alpha: float = 0.05, power: float = 0.8, replicates: int = 10000, seed: int = 0) -> int
Return the number of pairs that detects a paired d_z with the given power.
Starting from attainable_floor(), the count doubles until the
simulated power reaches the target, and bisection then finds the
smallest count that reaches it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
effect
|
float
|
The paired Cohen's d_z to detect, positive. |
required |
alpha
|
float
|
Two-sided significance level. |
0.05
|
power
|
float
|
Target power, strictly between 0 and 1. |
0.8
|
replicates
|
int
|
Simulated samples per power estimate. |
10000
|
seed
|
int
|
Seed of the simulation. |
0
|
Raises:
| Type | Description |
|---|---|
ConfigError
|
If an argument is out of range. |