gp_ei — Gaussian-process Bayesian optimization (Expected Improvement)

Role: optimization · kind: n4m_sampler_kind_t = N4M_SAMPLER_GP_EI · since: ABI 2.1 (F4)

Bayesian optimization with a Gaussian-process surrogate and the Expected Improvement acquisition. After n_startup_trials random trials, every ask fits an RBF GP on the completed, scored history over the continuous axes (int / float / log-int / log-float, in unit space), then returns the candidate that maximises EI over a random acquisition batch. This is the sample-efficient sampler for smooth, low-dimensional, expensive objectives — the regime where a single PLS/DL fit dominates the trial cost, so spending compute to pick the next point pays off.

The surrogate is deliberately simple and dependency-free:

  • Kernel: squared-exponential (RBF), unit signal variance on standardised targets, lengthscale from the median pairwise distance heuristic (no marginal-likelihood inner loop — robust and cheap).

  • Fit: K + 1e-6·I, dense Cholesky, α = K⁻¹y by forward/back substitution (from-scratch, in gp.cpp; fine for the trial counts NIRS finetuning uses).

  • Acquisition: EI with a small exploration margin ξ = 0.01, maximised by random search over 64 candidates per ask (direction-symmetric — MAXIMIZE is handled by negating the posterior mean).

Non-continuous axes (categorical / ordinal / sorted-tuple / conditionally inactive) are drawn independently by the shared decode — the GP models only the continuous subspace (Optuna’s independent-fallback convention). Warm-start (enqueue) is unsupported (N4M_ERR_UNSUPPORTED): a forced candidate is not a model proposal. Purely-categorical spaces degrade to random (use tpe there).

Usage (C ABI)

n4m_optimizer_options_t opts;
n4m_optimizer_options_init(&opts);
opts.sampler = N4M_SAMPLER_GP_EI;
opts.n_startup_trials = 8;   // random exploration before the GP takes over

Parity

  • Tier C (self-consistency + convergence): GP-fitting details (lengthscale policy, acquisition optimiser) differ across libraries, so there is no bit-exact external reference. The C++ test asserts convergence on a smooth 2-D objective in ~60 evaluations (best < 0.5; empirically < 0.03 across ten seeds — far more sample-efficient than random/CMA-ES on the same objective). Cross-binding identical at a fixed seed via the shared n4m_rng and native kernel/Cholesky.

References

  • Jones, Schonlau & Welch, Efficient Global Optimization of Expensive Black-Box Functions, J. Global Optimization 13 (1998), 455–492. jones1998ego

  • Rasmussen & Williams, Gaussian Processes for Machine Learning, MIT Press (2006). rasmussen2006gp