mojave

Measurement science for AI evaluation.

source · MIT OR Apache-2.0

what it answers

question	method	crate / package
how reliable is your scoring?	G-theory variance decomposition	`salib-estimators`
do your judges agree?	IRR + latent-class diagnostics	`irr`
which tasks are doing work?	IRT item analysis	`mojave-calibrate`
what's driving your scores?	Sobol/Shapley sensitivity analysis	`salib-estimators`, `salib-shapley`
can you stop early?	anytime-valid inference, e-processes	`seq-anytime-valid`
did anything change?	SPC control charts, e-detector	`spc-charts`
are some tasks redundant?	factor models, CFA	`mojave-calibrate`
is the eval gameable?	randomized item selection, anti-gaming	`eval-design`

architecture

two layers, clean JSON boundary. Rust owns correctness and real-time decisions. Python owns offline model fitting (IRT, factor analysis, CFA/SEM). no PyO3, no FFI.

eval runner output (Inspect, HAL, lm-eval, custom)
        │
        ▼
   eval-ingest
        │
   ┌────┼────────────────────┐
   │    │                    │
   ▼    ▼                    ▼
 Rust engine    Python calibration    audit-chain
 salib-*  GSA   (mojave-calibrate)    tamper-evident
 irr      IRR   py-irt    IRT         provenance
 seq-*    seq   deepirtools factors   Ed25519 signing
 spc-*    SPC   semopy    CFA/SEM
 eval-design
   │            │
   │  ◀─ JSON ──┘
   ▼
 mojave-cli
 reports · signals · stop/continue decisions

Rust crates

sensitivity analysis (salib-*)

strict superset of Python SALib's method coverage, in Rust. full docs.

crate	what
`salib-core`	RNG, distributions, problem specs
`salib-samplers`	LHS, Sobol, Morris, FAST, Plackett-Burman, fractional-factorial
`salib-estimators`	Sobol (S1/S2/ST), Morris, FAST, RBD-FAST, DGSM, PAWN, Borgonovo, G-theory, ANOVA, HDMR
`salib-surrogate`	polynomial chaos expansion (full + sparse LARS)
`salib-shapley`	Shapley effects for categorical inputs
`salib-validation`	reference functions (Ishigami, Sobol G), frozen SALib CSV data

measurement engine

crate	what
`irr`	Cohen's/Fleiss' κ, ICC, Krippendorff's α, Gwet's AC, Dawid-Skene, preference-leakage
`seq-anytime-valid`	SPRT, group-sequential, mSPRT, e-values, confidence sequences
`spc-charts`	Shewhart, CUSUM, FIR CUSUM, EWMA, combined Shewhart-CUSUM, e-detector, ARL
`eval-design`	computerized adaptive testing, randomized item selection, anti-gaming
`perturbation-engine`	deterministic perturbation primitives for eval sensitivity analysis
`change-attribution`	git-commit-to-score-change attribution

infrastructure

crate	what
`eval-core`	foundational types — trial records, item metadata, score types
`eval-ingest`	pluggable ingestion from eval runner output formats
`eval-orchestrator`	pipeline orchestration — ingest through analysis through reporting
`mojave-cli`	unified CLI entry point
`audit-chain`	tamper-evident hash chain for audit provenance
`audit-sign`	Ed25519 signing and COSE_Sign1 attestation

Python: mojave-calibrate

offline calibration pipeline. fits IRT models, factor models, and CFA/SEM, emits mojave-compatible JSON consumed by the Rust engine.

mojave-calibrate irt --input responses.jsonl --output pool.json \
    --model-type 2pl --content-domain reasoning --device cuda

mojave-calibrate factors --input responses.csv --output factors.json \
    --latent-size 3 --model-type grm

mojave-calibrate cfa --input data.csv --output cfa.json \
    --model "f1 =~ x1 + x2 + x3"

module	wraps	what
`irt.py`	py-irt	GPU Bayesian IRT (1PL, 2PL, 4PL) via Pyro
`factors.py`	deepirtools	multidimensional IRT + factor models via IWAVE
`cfa.py`	semopy	confirmatory factor analysis / structural equation modeling

development

# Rust
cargo test --workspace --all-targets
cargo clippy --workspace --all-targets -- -D warnings

# Python
cd python
uv sync --group dev
uv run pytest -v

Pre-commit hook: ./scripts/install-hooks.sh