mojave-wmdp
construct validity audit of wmdp-bio, wmdp-chem, and truthfulqa mc1 via global sensitivity analysis (saltelli/sobol').
companion to enter mojave · tool: mojave · 2026-05-24
a saltelli radial design across six perturbation axes decomposes benchmark variance into first-order and total-effect sobol' indices. prompt template explains >95% of total variance on all three benchmarks. everything else is noise.
experiment design
perturbation matrix
| axis | levels | values |
|---|---|---|
| prompt_template | 5 | bare, cot, letter-only, lm-eval-default, verbose-rationale |
| system_prompt | 4 | helpful, domain-expert, none, safety-aware |
| decoding | 3 | greedy (T=0.0), T=0.7, T=1.0 |
| choice_order | 2 | original, shuffled |
| quantization | 2 | bf16, fp8 |
| n_shot_frac | continuous | [0, 0.05] |
parameters
| parameter | value |
|---|---|
| model | qwen/qwen2.5-7b-instruct |
| design | saltelli radial, N=512, k=6 |
| cells per eval | 4,096 |
| total cells | 12,288 |
| benchmarks | wmdp-bio (1,273 items), wmdp-chem (408 items), truthfulqa mc1 (817 items) |
| infrastructure | runpod gpu pods (a40/a6000), vllm, up to 46 concurrent |
| gsa engine | salib 0.1.1 (rust), 10,000 bootstrap resamples |
| artifacts | 12,351 cryptographically sealed pdf run cards |
headline: prompt template dominates everything
| eval | prompt_template STi | next-highest STi | mean accuracy |
|---|---|---|---|
| wmdp-bio | 0.9507 | quantization 0.1015 | 0.5967 |
| wmdp-chem | 0.9548 | n_shot_frac 0.0389 | 0.4153 |
| truthfulqa mc1 | 0.9682 | n_shot_frac 0.0299 | 0.5073 |
STi is the total-effect sobol' index—it captures both the direct effect and all interactions. a value of ~0.95 means that if you held prompt_template fixed and varied everything else, you'd explain only ~5% of the variance.
borgonovo δ (distribution-free confirmation)
| eval | prompt_template δ | next-highest δ |
|---|---|---|
| wmdp-bio | 0.5137 | n_shot_frac 0.1160 |
| wmdp-chem | 0.3974 | choice_order 0.1335 |
| truthfulqa mc1 | 0.3373 | n_shot_frac 0.1446 |
per-template accuracy
| template | bio | chem | truthfulqa |
|---|---|---|---|
| bare | 0.140 | 0.058 | 0.034 |
| cot | 0.700 | 0.528 | 0.625 |
| letter-only | 0.741 | 0.496 | 0.610 |
| lm-eval-default | 0.767 | 0.522 | 0.638 |
| verbose-rationale | 0.633 | 0.472 | 0.628 |
the bare template—question text only, no framing, no answer labels—is catastrophic everywhere. wmdp-bio drops 63pp from template choice alone. chem: 46pp. truthfulqa: 60pp.
diagnostics
sum-of-indices
| eval | Σ S1 | Σ STi | interpretation |
|---|---|---|---|
| wmdp-bio | 0.903 | 1.295 | moderate interactions (gap ~0.39) |
| wmdp-chem | 0.939 | 1.107 | mild interactions (gap ~0.17) |
| truthfulqa | 0.993 | 1.068 | near-additive (gap ~0.07) |
bf16 vs fp8 quantization
| eval | bf16 mean | fp8 mean | diff | t | p |
|---|---|---|---|---|---|
| wmdp-bio | 0.6004 | 0.5931 | +0.73pp | 0.972 | 0.331 |
| wmdp-chem | 0.4165 | 0.4141 | +0.23pp | 0.404 | 0.686 |
| truthfulqa | 0.5046 | 0.5101 | −0.55pp | −0.736 | 0.462 |
welch's t-test, two-tailed. none significant at α=0.05. quantization from bf16 to fp8 does not meaningfully affect accuracy.
caveats
- single model. qwen2.5-7b-instruct only. the methodology is the contribution; specific numbers are one data point.
- rescored, not raw. inspect's built-in scorer has artifacts on some templates.
rescore_fast.pyapplies corrected scoring. raw logs available. - no irt / item-level analysis. sensitivity analysis is at the cell level (per-run accuracy). item-level analysis is future work.
cryptographic integrity
every artifact carries:
- sha-256 companion data hash of its source json
- audit chain seal via
mojave audit seal—hash-linked append-only chain with sequence number and chain tip hash
12,351 individually-sealed pdf run cards. no cherry-picking possible. view audit log.