antimeme / docs / salib

Benchmarks

Criterion benchmarks for every analysis method on the Ishigami function (d = 3), plus sampling.

Methodology

Criterion 0.5, 100 samples per benchmark, automatic warmup. Statistic: median. Machine: Apple Silicon.

Test function: Ishigami f(x) = \sin(x_1) + 7\sin^2(x_2) + 0.1 x_3^4 \sin(x_1) with x_i \in [-\pi, \pi].

What is timed: Methods marked [fn] evaluate the model function inside the timed loop (sampling + analysis are interleaved in the estimator). All other methods take pre-computed (X, Y) and time the analysis step only.

Analysis

Method N Time
Sobol (Saltelli 2010) [fn] 1024 28 µs
Sobol (Saltelli 2010) [fn] 4096 116 µs
Sobol (Saltelli 2010) [fn] 8192 239 µs
Sobol (Saltelli 2010) [fn] 16384 482 µs
Jansen [fn] 1024 24 µs
Jansen [fn] 4096 101 µs
Jansen [fn] 8192 220 µs
Janon [fn] 1024 28 µs
Janon [fn] 4096 114 µs
Janon [fn] 8192 260 µs
Sobol on Sobol’ G (8D) [fn] 1024 60 µs
Sobol on Sobol’ G (8D) [fn] 4096 249 µs
Sobol on Sobol’ G (8D) [fn] 8192 495 µs
FAST / eFAST [fn] 1025 57 µs
FAST / eFAST [fn] 4097 280 µs
FAST / eFAST [fn] 8193 642 µs
Morris [fn] 10 0.7 µs
Morris [fn] 20 1.0 µs
Morris [fn] 50 2.2 µs
RBD-FAST 1024 41 µs
RBD-FAST 4096 205 µs
RBD-FAST 8192 455 µs
Borgonovo \delta 1024 793 µs
Borgonovo \delta 4096 3.27 ms
Borgonovo \delta 8192 6.78 ms
PAWN 1024 108 µs
PAWN 4096 497 µs
PAWN 8192 1.15 ms
DGSM 1024 2.3 µs
DGSM 4096 11 µs
DGSM 8192 29 µs
Regression (SRC/SRRC/PCC/PRCC) 1024 307 µs
Regression (SRC/SRRC/PCC/PRCC) 4096 1.11 ms
Regression (SRC/SRRC/PCC/PRCC) 8192 1.78 ms

[fn] = benchmark includes Ishigami evaluation inside the timed loop. For these methods, the reported time is analysis + function evaluation. Ishigami is trivial (~3 ns/eval); for expensive models, function evaluation dominates and the analysis overhead shown here becomes negligible.

Morris N is trajectory count r; total evaluations are r \times (d + 1).

FAST N values are odd (required by the algorithm).

Sampling

Method N Time
Saltelli matrix 1024 10 µs
Saltelli matrix 4096 56 µs
Saltelli matrix 16384 249 µs
Morris trajectories 10 0.9 µs
Morris trajectories 50 3.7 µs
Morris trajectories 100 7.4 µs

Saltelli N is base sample size; the matrix has N \times (d + 2) rows. Morris N is trajectory count.

Reproducing

cargo bench --manifest-path crates/salib/Cargo.toml

# Regenerate this document from Criterion results
python benches/generate_benchmark_doc.py