derivations
mathematical foundations for the sobol'/saltelli variance decomposition used in mojave-wmdp. from anova decomposition through radial sampling to bootstrap confidence intervals.
1. anova decomposition (hoeffding-sobol')
for a square-integrable function f(x) on the unit hypercube [0,1]k, there exists a unique decomposition:
where each term has zero mean over its complement variables. this gives the variance decomposition:
in our experiment, f maps a 6-dimensional configuration (prompt_template, system_prompt, decoding, choice_order, quantization, n_shot_frac) to benchmark accuracy.
2. sobol' indices
first-order index Si
fraction of total variance explained by axis i alone:
total-effect index STi
fraction of total variance attributable to axis i, including all interactions:
where X₋ᵢ denotes all axes except i. equivalently, STi = Si + all interaction terms involving i.
interpretation
| condition | meaning |
|---|---|
| Si ≈ STi | axis i acts additively (no interactions) |
| STi >> Si | axis i participates in interactions |
| Σ Si ≈ 1 | model is additive |
| Σ STi > 1 | interactions present (double-counting shared variance) |
3. saltelli radial sampling
the saltelli (2002) scheme estimates both Si and STi from N(k+2) model evaluations rather than the N(2k+2) of the original sobol' (1993) scheme.
construction
- draw two independent N×k quasi-random matrices A and B (sobol' sequence)
- for each axis i, construct matrix AB(i): take A, replace column i with column i from B
- evaluate f on rows of A, B, and all k matrices AB(i)
estimators (jansen 1999 / saltelli 2010)
our parameters
| parameter | value | note |
|---|---|---|
| N | 512 | base sample size |
| k | 6 | perturbation axes |
| cells/eval | 4,096 | N × (k + 2) |
| total cells | 12,288 | 3 benchmarks × 4,096 |
4. handling mixed axes
five of six axes are categorical (finite levels). the saltelli design operates on [0,1]k, so categorical axes use equal-width bins:
| axis | levels | mapping |
|---|---|---|
| prompt_template | 5 | [0, 0.2) → bare, [0.2, 0.4) → cot, … |
| system_prompt | 4 | [0, 0.25) → helpful, … |
| decoding | 3 | [0, 0.33) → greedy, … |
| choice_order | 2 | [0, 0.5) → original, [0.5, 1) → shuffled |
| quantization | 2 | [0, 0.5) → bf16, [0.5, 1) → fp8 |
| n_shot_frac | continuous | linear: u → u × 0.05 |
this preserves the quasi-random sampling properties of the sobol' sequence while respecting the discrete nature of categorical axes.
5. borgonovo δ (distribution-free sensitivity)
the borgonovo (2007) δ importance measure is a moment-independent sensitivity index based on the L1 distance between the unconditional and conditional output distributions:
properties:
- δi ∈ [0, 1]
- δi = 0 iff Y and Xi are independent
- no assumptions about model linearity or output distribution shape
- captures effects invisible to variance-based indices (e.g., a factor that shifts shape but not variance)
in our implementation, δ is estimated via kernel density estimation on the conditional samples from the saltelli design.
6. bootstrap confidence intervals
all sobol' indices are reported with 95% confidence intervals computed via the percentile bootstrap method:
- resample the N base-sample rows with replacement (preserving the A/B/AB(i) structure)
- recompute all indices on the resampled data
- repeat 10,000 times
- report [2.5th percentile, 97.5th percentile] as the 95% CI
| parameter | value |
|---|---|
| resamples | 10,000 |
| ci method | percentile |
| ci level | 95% |
7. implementation
all gsa computation uses salib 0.1.1—a rust implementation of the saltelli/sobol'/borgonovo algorithms.
| component | tool |
|---|---|
| design generation | salib-rs (sobol' sequence + saltelli construction) |
| index estimation | salib-rs (jansen/saltelli estimators) |
| borgonovo δ | salib-rs (kde-based) |
| bootstrap | salib-rs (parallel resampling) |
| orchestration | mojave-gsa CLI binary |
source: github.com/antimeme-ai/mojave