antimeme

4-gate validation

every statistical estimator in mojave must pass all four gates before release. no exceptions. the whole point of a measurement science library is that its own measurements are trustworthy.

gate 1: textbook reproductions

reproduce known results from canonical papers using golden datasets.

  • tolerances: rtol=1e-6, atol=1e-8 for closed-form, rtol=1e-3 for iterative (EM/REML), rtol=1e-2 for stochastic (bootstrap)
  • source: original papers (Cronbach 1951, Brennan 2001, Bland-Altman 1986, etc.)
  • each reproduction is a named test with a citation

gate 2: reference implementation cross-checks

agreement with authoritative reference implementations, primarily R.

  • R packages: psych, mirt, gtheory, irr, irrCAC, gsDesign, rpact, lme4
  • pinned reference versions in CI (Docker image with R + packages at specific versions)
  • cross-checks via subprocess — not rpy2
  • quarterly version matrix run against latest reference package versions
  • drift alarm: any delta > 2× tolerance opens an issue automatically

gate 3: property-based tests

invariant and identity tests that hold regardless of input data.

  • permutation invariance (α under item reorder, Krippendorff under rater-label permutation)
  • degenerate cases recover known forms (3PL with c=0 is 2PL, group-sequential K=1 is fixed-sample)
  • boundary conditions (perfect agreement → α=1, chance-level → α=0)
  • algebraic identities (Spearman-Brown, G-theory under perfect-rater design = Cronbach α)

gate 4: Monte Carlo calibration

per-release statistical validation under simulation.

  • bootstrap coverage: observed 95% CI coverage ∈ [0.93, 0.97] over 1000 simulated datasets
  • IRT recovery: RMSE(parameters) vs N at known ground truth
  • SPRT Type I: 100k reps under H0, observed rate = nominal α to MC error
  • group-sequential power: observed power matches reference implementation
  • IRR prevalence sweep: κ, AC1, α vs base rate at known agreement level

edge cases

every public API must handle adversarial inputs explicitly.

  • empty data (n=0) → explicit error, never silent NaN
  • single observation → documented degenerate behavior
  • all-identical responses → method-specific documented behavior
  • NaN/Inf → explicit policy (reject/propagate/impute), never silent
  • wrong dtype → reject at boundary, not deep in numerics
  • massive inputs (107 rows) → no OOM, streaming path tested

behavioral specs (TCK)

71 Gherkin feature files, 398 scenarios. every crate has a corresponding tck/ directory with .feature specs that define expected behavior before implementation begins. the spec is written first, tests are wired up (red), then the implementation makes them pass (green).

the TCK is the contract. if the spec says it, the code does it. if the code does something the spec doesn't say, that's a bug.