methods
the statistical toolkit, organized by the question it answers.
how reliable is your scoring?
G-theory variance decomposition (Cronbach, Gleser, Nanda & Rajaratnam 1972). decomposes score variance into person, item, rater, and interaction effects. tells you how much of your eval signal is the agent versus the judge versus the prompt versus noise.
Crate: salib-estimators (G-theory module)
do your judges agree?
inter-rater reliability. Cohen's κ, Fleiss' κ, ICC, Krippendorff's α, Gwet's AC1, Dawid-Skene latent-class model, and preference-leakage diagnostics.
Dawid-Skene matters for LLM-as-judge setups: it estimates each rater's confusion matrix and recovers the latent true label, even when ground truth is unavailable.
Crate: irr
which tasks are doing work?
item response theory (Lord & Novick 1968). fits a probabilistic model to each item's difficulty and discrimination. items with near-zero discrimination don't separate strong from weak models. items at extreme difficulty contribute nothing when the model is far from their threshold.
mojave-calibrate fits 1PL, 2PL, and 4PL IRT models via GPU-accelerated Bayesian estimation (py-irt / Pyro), and multidimensional IRT via deepirtools (IWAVE).
Package: mojave-calibrate (Python)
what's driving your scores?
global sensitivity analysis. Sobol' indices quantify how much of the output variance is attributable to each input factor. Shapley effects handle correlated inputs via coalitional game theory.
the full salib library is available: Saltelli, Morris, FAST, RBD-FAST, Borgonovo δ, PAWN, DGSM, PCE surrogates, HDMR, active subspaces.
Crates: salib-estimators, salib-shapley
can you stop early?
anytime-valid inference. sequential probability ratio tests (Wald 1945), the mSPRT with Beta mixing (Johari et al. 2022), e-values and e-processes (Ramdas et al. 2020+), confidence sequences.
you can peek at running eval results continuously without inflating your error rate. when GPU hours cost money, knowing you can stop at 14% of the benchmark matters.
Crate: seq-anytime-valid
did anything change?
statistical process control. Shewhart charts (1931), CUSUM (Page 1954), FIR CUSUM, EWMA, combined Shewhart-CUSUM, and the e-detector for anytime-valid change detection.
establish a baseline on your eval, then every commit either holds the line or triggers a control chart signal. catches both sudden failures and slow drift.
Crate: spc-charts
are some tasks redundant?
factor analysis. exploratory factor analysis discovers latent structure; confirmatory factor analysis (CFA) tests it. does your 50-item eval measure one thing or five?
mojave-calibrate fits factor models via deepirtools (deep generative IRT) and structural equation models via semopy.
Package: mojave-calibrate (Python)
is the eval gameable?
computerized adaptive testing + anti-gaming. randomized item selection from calibrated pools. adaptive testing selects the most informative next item given what's been observed, cutting eval cost without sacrificing precision.
Crate: eval-design
audit trail
tamper-evident hash chain and Ed25519 signing for every run card. COSE_Sign1 attestation. if you can't verify the provenance of a measurement, you can't trust it.
Crates: audit-chain, audit-sign