run cards
a run card is a metrological report for a single eval run. not a score. not a pass/fail. an accounting of what the measurement actually tells you and where the uncertainty lives.
what goes in a run card
- aggregate accuracy + Wilson CIs — pooled across all perturbation variants, with proper binomial confidence intervals
- perturbation stability histogram — for each item, what fraction of variants got it right? items that flip under cosmetic perturbation are instrument noise, not signal
- sensitive item count — items where 25–75% of variants give different answers. these items are measuring presentation, not knowledge
- temperature decomposition — accuracy broken out by inference temperature, isolating stochastic from structural sensitivity
- sequential stopping analysis — retrospective replay: if you'd been monitoring with a Bernoulli mSPRT, when could you have stopped and still had a valid answer?
- IRT item parameters — difficulty and discrimination for every item (when calibration data is available)
the run card is a parametric LaTeX document. values go in a config file; the template handles layout, histograms, and conditional sections. partially filled cards compile cleanly — missing fields render as em-dashes, not errors.
example: TruthfulQA run card
TruthfulQA is widely used as a benchmark for model honesty. here is what mojave finds when you actually measure the instrument: 17.6% of items flip under cosmetic perturbation, and the variant spread is 11 percentage points. a leaderboard delta that size could be noise.
compare the stability histogram with ARC Challenge's: ARC is sharply bimodal — items are either always right or always wrong, with almost nothing in between. TruthfulQA has a long smear through the middle bins. those middle-bin items are measuring something, but it isn't what the benchmark claims.
example: cross-eval metrological summary
one model, six benchmarks, same perturbation design. the cross-eval summary compares metrological quality across instruments — which benchmarks hold up under perturbation and which ones don't.
templates
run card templates live in templates/run-card/ in the
mojave repo.
three variants:
- single-run-card — one model, one benchmark, one run
- cross-run-summary — one benchmark, many models/runs aggregated
- cross-eval-summary — one model, many benchmarks compared
all three use the same parametric engine: \rcset{key}{value} in
a config file, \rc{key} in the template. empty or missing keys
render as em-dashes. partially filled cards compile and read sensibly.