Benchmark Methodology¶

This page documents how our benchmarks were conducted for reproducibility.

Hardware¶

Software Versions¶

Benchmark Configuration¶

Time-Bounded Benchmarks¶

• Budget: 5 seconds wall-clock time
• Warmup: 1 second (excluded from measurement)
• Iterations: 5 runs, median reported
• RNG Seed: 42 (deterministic)

Test Functions¶

Measurement Protocol¶

1. Isolation: Benchmarks run on dedicated hardware, no other processes
2. CPU Governor: Set to performance mode
3. Hyperthreading: Enabled
4. Metrics:
5. Trials completed per second
6. Time to best value
7. Final objective value

Reproduction Steps¶

# Clone and build
git clone https://github.com/novelbytelabs/ArqonHPO.git
cd ArqonHPO
cargo build --release

# Run benchmarks
cargo bench --bench optimizer_comparison

# Compare with Optuna
python benchmarks/optuna_comparison.py

Data Files¶

Raw benchmark data available in:

• docs/docs/benchmarks/benchmark_data.csv

Limitations¶

• Single machine: Results may vary on different hardware
• Python GIL: Optuna measurements include Python overhead
• Function cost: Benchmarks use instant ($\approx 0$ms) functions

For expensive functions (>1s per evaluation), Optuna's smarter sampling may outperform ArqonHPO's volume advantage.