enter mojave

people want agents. people don't trust agents. evals on agents produce a number. moar number is moar better so worry turns to confidence as agents improve eval scores.

why did the score go up? because the model got better—at what? taking the eval? at doing stuff? you changed the prompt template? you added a system message? because the answer options happened to be in a different order? the eval leaks information?

run an eval, get a score. add a skill file, get a better score. speedrunning unearned beliefs is the only thing this accomplishes. what does the eval score even mean? no one checks.

antimeme built mojave toward meaningful evals.

we ran a global sensitivity analysis on wmdp-{bio, chem} to spark a conversation about evals. the finding is both already known and simple: the choice of how you phrase the question explains over 95% of the variance in these benchmark scores. everything else—system prompt, decoding temperature, quantization, few-shot examples, answer order—is noise.

this isn't a takedown. it's an invitation to ask a question that psychometrics has wrangled with for decades: what does this instrument actually measure, and how would you know?

construct validity is the boogeyman for all ai

construct validity asks whether a test measures what it claims to measure. not whether the questions are hard, or whether the scores correlate with something, but whether the underlying construct—the thing you care about—is what drives the score. no matter your goals, if ai figures into them, this matters to you. you can no more build faith in capability than govern what you don't understand.

for wmdp, the claimed construct is "dangerous biological/chemical knowledge." the question is whether a multiple-choice accuracy score on curated questions actually captures that, or whether it captures something else: familiarity with mcq format conventions, sensitivity to prompt framing, or just general scientific competence wearing a safety-colored hat.

this experiment doesn't answer the question definitively. not even close. it does, however, show that the instrument is dominated by a factor—prompt template—that has nothing to do with the construct. that's not a sufficient condition for concluding the instrument is broken, but it's a loud one.

in the hot seat

[li2024wmdp]

wmdp—the weapons of mass destruction proxy benchmark—is a set of multiple-choice questions across three domains. we audited the bio and chem splits. we did not examine the cyber split (1,987 items) in this experiment.

as a control, we ran the same design on truthfulqa mc1 (817 items), a completely different benchmark from different authors with different goals. our aim is not to pick on wmdp—it is an idea the world is already moving on from. we picked wmdp to illustrate that sensitivity analysis for ai evals against a benchmark both cited by standards bodies and suspected by practitioners.

mojave can say why for things you already know.

what we did

full experiment · derivations · data

we ran a saltelli radial design—a quasi-random sampling scheme for variance-based global sensitivity analysis—across six perturbation axes: prompt template, system prompt, decoding strategy, answer choice order, quantization precision, and few-shot fraction. this decomposes benchmark variance into first-order and total-effect sobol' indices for each axis, capturing both direct effects and all interactions.

every cell was sealed into a cryptographic audit chain. 12,351 individually-verifiable run cards. no cherry-picking possible. it's tedious work but we mean it—check the hashes yourself.

no surprise: how you ask determines the outcome

experiment results

across all three benchmarks, prompt_template's total-effect sobol' index exceeds 0.95. that means if you held the template fixed and varied everything else—system prompt, temperature, quantization, few-shot count, answer order—you'd explain less than 5% of the variance.

the per-template breakdown shows why. the bare template—question text only, no answer labels, no framing—is catastrophic everywhere. wmdp-bio drops 63 percentage points from template choice alone.

this is not a wmdp-specific finding. truthfulqa—different domain, different authors, different goals—shows the same sensitivity structure. the template isn't cosmetic. it defines the task.

quantization is noise

diagnostics

if a governance body mandated bf16 over fp8 for "safety" based on wmdp scores, this data shows that distinction is meaningless—at least in this case. we have no easy rules for you other than everything must be checked, every time.

saying the quiet part out loud

none of this is novel. perturbation analysis is standard psychometric practice. sequential stopping tests date to wald in 1945. wilson confidence intervals are from 1927. the surprise isn't the methods—it's that nobody applies them.

ai evaluation has an inconvenient, uncomfortable measurement problem. scores are reported without confidence intervals. benchmarks are used without perturbation audits. prompt template choice—a single design decision that can move a score by 90 points—is never documented, never varied, never questioned.

nothing discussed here can tell you whether your eval "works." these methods can and often do reveal how they don't work. the affirmative case is very, very hard, wrapped tightly in the construct validity problem. we will treat this in the future—alongside agentic evals.

evals only disconfirm.

now what?

the mojave work—perturbation analysis, instrument sensitivity, cryptographic audit—is the tooling that lets you start asking: how much trust has an eval earned? it's domain-agnostic. you can run the same sobol' decomposition on agentic benchmarks, on operational evals, on anything with a score and a set of decisions that produced it. in this case, the scaffold (one of many) is the prompt template for agents.

we've published 12,351 cryptographically sealed run cards, the complete raw data, and the source code. every claim in this essay is individually verifiable against the audit chain.

if you run evaluations—for safety, for capability, for compliance—the way forward lies through mojave.

antimeme rejects moar number moar better.

appendix: what the literature already knows

full bibliography (26 sources)

our experiment measures instrument sensitivity. the claims below come from the published literature, not our data. they establish the broader context that we seek to scaffold through measurement science.

for unlawful biological knowhow

ren et al. report a −87.5% capabilities correlation on wmdp-bio: models that score higher on general capability benchmarks score lower on wmdp. the authors interpret this as evidence wmdp measures something distinct from capabilities. but the sign is doing all the work. the more capable the model is, the better it will do on a biology test. substantive guardrails in science are restraint on action—you can't be a competent biologist without dangerous knowledge. [ren2024safetywashing]

the instrument is gameable on its face

selecting the longest answer on wmdp-bio gets 46% accuracy (chance = 25%). multiple downstream papers had to create debiased subsets just to get usable signal. a benchmark that leaks through answer length. [li2024wmdp]

unlearning measured by wmdp is reversible

rmu—the unlearning method introduced alongside wmdp—is reversed by 20 samples and 1 epoch of fine-tuning. hindi filler text recovers 25 percentage points via a trivial prompt change. even prism, the best method found to date, leaks after 100 relearning steps. and the mcq format can't even detect the failure: models that output garbage on all topics score identically to models that genuinely forgot.

models can fake low scores

weij et al. demonstrate that language models can selectively underperform on wmdp while maintaining scores on general benchmarks. claude 3 drops wmdp-bio by 39.8 percentage points while mmlu drops only 9.7%. a password-locked mistral variant scores ~27% without the password and ~67% with—and this generalizes to unseen wmdp questions it was never trained on. the authors call this the ai volkswagen effect. the perverse incentive is real: labs want low wmdp (avoid mitigations) and high mmlu (look competitive). [weij2025sandbagging]

safety mitigations don't move the benchmark

gpt-4.5 scored 83% pre-mitigation and 85% post-mitigation on wmdp-bio. the mitigations dropped long-form biorisk generation to 0%. the benchmark didn't notice. it measures latent knowledge that mitigations can't and arguably shouldn't touch—the score is inert to the interventions it's supposed to gate. [openai2025gpt45]

the frontier labs are already leaving

openai used wmdp for gpt-4.5 (february 2025) and dropped it for gpt-5. anthropic never used it. deepmind never used it. the field shifted to operational evals: expert uplift trials, wet lab troubleshooting, tacit knowledge assessment, dna synthesis screening evasion.

but the construct validity problem follows them

the operational evals haven't solved it either. none of the frontier labs have published a validated mapping from eval score to real-world risk. the construct validity gap is the same gap, all the way up. system cards

chem is a ghost split

408 questions. categories with 15–19 items. the original authors declined to test unlearning on it. almost no downstream paper evaluates it. nobody reports chem scores. rmu doesn't even move them—45.8% before, 45.8% after on zephyr-7b. it exists on paper. [li2024wmdp]

the format is structurally inapplicable to the most dangerous models

the johns hopkins center for health security makes the point directly: "it is not possible to extend the question-based approach to baims." protein language models, biological design tools—the class of models that can generate novel pathogen designs—don't output natural language. wmdp can't see them at all. [jhu2025]

the policy apparatus is building on unvalidated ground

govai provides the sharpest public critique: internal validity (are wmdp items just memorized training data?) and external validity (does capability on an mcq predict real-world harm?). nobody has validated the mapping from benchmark score to actual risk. the entire compliance scheme is vulnerable to sandbagging if wmdp-style evals are the gate. [govai2025]