antimeme

bibliography

26 sources informing the mojave-wmdp construct validity audit.

the instrument

[li2024wmdp]
The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
Li, N., Pan, A., Gopal, A., Yue, S., Berrios, D., Gatti, A., Li, J.D., Dombrowski, A.-K., Goel, S., Phan, L., et al.
ICML 2024
arXiv:2403.03218
the instrument under audit. 1,273 bio + 408 chem + 1,987 cyber MCQs. introduces RMU unlearning baseline.

frontier lab system cards

[openai2025gpt45]
GPT-4.5 System Card
OpenAI
February 2025
openai.com
last frontier model to report WMDP. pre-mitigation 83%, post-mitigation 85% on Bio. mitigations move actual risk but not the benchmark.
[openai2025gpt5]
GPT-5 System Card
OpenAI
August 2025
openai.com
WMDP absent. classified High bio/chem precautionarily—"we do not have definitive evidence."
[anthropic2025opus]
Claude Opus 4.6 System Card
Anthropic
2025
anthropic.com
no WMDP. proprietary CBRN suite. "challenging to interpret because it is unclear how to directly map a score to the threat model."
[phuong2024dangerous]
Evaluating Frontier Models for Dangerous Capabilities
Phuong, M., Aitchison, M., Catt, E., Cogan, S., Krakovna, V., et al.
Google DeepMind, 2024
arXiv:2403.13793
no CBRN red-lines set. acknowledges setting them "is not trivial."

policy & governance

[nist2025-800-1]
AI 800-1: Managing Misuse Risk for Dual-Use Foundation Models (Initial Public Draft)
NIST
NIST, 2025
nist.gov [PDF]
cites WMDP approvingly as a model eval for dual-use risk.
[nist2024-600-1]
AI 600-1: Artificial Intelligence Risk Management Framework: Generative AI Profile
NIST
NIST, July 2024
nist.gov
[govai2025]
Comments on NIST AI 800-1
Centre for the Governance of AI (GovAI), Oxford
Public Comment, 2025
governance.ai [PDF]
sharpest public critique: internal validity (memorization) + external validity (capability ≠ harm).
[jhu2025]
Response to NIST on Chemical and Biological AI Risk
Johns Hopkins Center for Health Security
Public Comment, March 2025
centerforhealthsecurity.org [PDF]
"not possible to extend the question-based approach to BAIMs." MCQ format structurally inapplicable to the most dangerous bio models.

unlearning & its failures

[sheshadri2024relearning]
Relearning Attacks on Machine Unlearning
Sheshadri, A., et al.
arXiv preprint, 2024
20 samples, 1 epoch reverses RMU. the erasure is shallow.
[prism2025]
PRISM: Dual-Space Unlearning
—
arXiv preprint, 2025
best unlearning method found. KnowMem 0.866 → 47.769 after 50 relearning steps. breaks after 100.
[goel2024microscope]
The Unlearning Microscope
Goel, S., et al.
arXiv preprint, 2024
MCQ format can't detect failure: garbage-on-all-topics scores identically to genuine forgetting.
[patil2024prompt]
Prompt Attacks on Safety-Aligned LLMs Reveal Superficial Unlearning
Patil, S., et al.
arXiv preprint, 2024
hindi filler text recovers 25pp. unlearning is skin-deep.
[goel2024openunlearning]
OpenUnlearning: A Unified Framework for Machine Unlearning
Goel, S., Phan, L., et al.
CMU, arXiv preprint, 2024
comprehensive unlearning benchmark. documents gap between WMDP scores and actual knowledge removal.
[li2024fsrmu]
Feature-Selective Representation Misdirection for Unlearning
Li, N., et al.
arXiv preprint, 2024
improved RMU targeting. base RMU collateral damage to general capabilities is significant.
[thaker2024ripple]
Ripple Effects of Machine Unlearning on Model Behavior
Thaker, P., et al.
arXiv preprint, 2024
unlearning one fact doesn't unlearn related knowledge. cascading entanglement in knowledge graphs.
[tamirisa2024tamper]
Tamper-Resistant Safeguards for Open-Weight LLMs
Tamirisa, R., Bhatt, U., et al.
arXiv preprint, 2024
arXiv:2408.00761
alternative: prevent learning rather than unlearn. acknowledges fundamental fragility of post-hoc methods.
[cir2024]
CIR: Collapse of Irrelevant Representations for Machine Unlearning
—
arXiv preprint, 2024
RMU collapses representation space rather than selectively removing knowledge.

robustness & adversarial

[weij2025sandbagging]
Sandbagging: Language Models Can Strategically Underperform on Evaluations
Weij, T., Scherlis, A., Manheim, D., Buck, R.
ICLR 2025
arXiv:2406.07358
Claude 3 drops WMDP-Bio 39.8% while MMLU drops 9.7%. password-locked Mistral: ~27% → ~67%. the AI Volkswagen effect.
[zou2024circuit]
Improving Alignment and Robustness with Circuit Breakers
Zou, A., Phan, L., Lin, J., Wang, J., Guo, E., et al.
NeurIPS 2024
arXiv:2406.04313
[sheshadri2025revisiting]
Revisiting Circuit Breakers: Evaluating Robustness Under Adaptive Attacks
Sheshadri, A., et al.
arXiv preprint, 2025
circuit breakers also brittle under adversarial attack.
[casper2024lat]
Defending Against Unforeseen Failure Modes with Latent Adversarial Training
Casper, S., Schulze, L., Patel, O., Hadfield-Menell, D.
arXiv preprint, 2024
arXiv:2403.05030
[vaintrob2024gradient]
Gradient Routing: Masking Gradients to Localize Computation in Neural Networks
Vaintrob, A., Bhatt, U., et al.
arXiv preprint, 2024
arXiv:2410.04332

meta-evaluation & benchmark critique

[ren2024safetywashing]
Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?
Ren, J., Xu, W., Yan, Y., Lin, A., Ehghaghi, B., Han, Y., et al.
NeurIPS 2024
arXiv:2407.21792
−87.5% capabilities correlation on WMDP-Bio = 87.5% positive correlation with scientific competence, sign-flipped.
[trust2024]
Trust in AI Benchmarks
Liang, P., et al.
arXiv preprint, 2024
systematic review of benchmark validity across AI evaluation.
[redteam2024]
Benchmarks as Early Red-Teaming Tools for AI-Enabled Risks
Srivastava, M., et al.
arXiv preprint, 2024
benchmarks as lightweight triage, not definitive safety measurement.