tpe — Tree-structured Parzen Estimator (sampler)

Role: optimization · kind: n4m_sampler_kind_t = N4M_SAMPLER_TPE · since: ABI 2.1 (F4)

Tree-structured Parzen Estimator — the Optuna-default sampler — in its univariate form. After n_startup_trials random trials, each parameter is modelled independently: the completed history is sorted by score and split into a good set (top γ = 0.25) and the rest. Two Parzen (kernel-density) models are built — l(x) over the good values and g(x) over the bad values — and the next value is chosen to maximise the density ratio l(x)/g(x), concentrating sampling where good trials cluster while avoiding the bad region:

  • numeric axes: history is mapped to unit space (unit_from_numeric, log-aware), KDE with a 1/√n-scaled Gaussian bandwidth; n_ei = 24 candidates are drawn from l and the best l/g is decoded back.

  • categorical / ordinal axes: Laplace-smoothed category frequencies in the good vs bad sets; the category maximising l/g is chosen.

TPE handles mixed and conditional spaces naturally (each active parameter is modelled on the trials in which it was active), which is where it beats CMA-ES (continuous-only) and the population samplers. It plugs into the base sampler’s per-parameter hooks, so constraints, conditions, forced values, sorted tuples, ask_batch, and warm-start all work as for the base sampler. The categorical proposal is sampled proportional to l/g (not argmax), so the base constraint-retry loop escapes an infeasible categorical combination. TPE activates only once ≥ max(n_startup_trials, 2) trials exist for an axis, and falls back to a uniform draw before that.

Stepped / integer axes are modelled in continuous unit space and snapped by numeric_from_unit at decode time (a minor l/g approximation on the grid); the decoded value is always on-grid.

Usage (C ABI)

n4m_optimizer_options_t opts;
n4m_optimizer_options_init(&opts);
opts.sampler = N4M_SAMPLER_TPE;
opts.n_startup_trials = 15;   // random exploration before TPE kicks in

Parity

  • Tier B-state: the RNG-free sub-decisions — the γ good/bad split and the l/g ranking over an injected candidate set — are compared against Optuna’s TPESampler; the public ask point is RNG-entangled (it draws n_ei candidates) and is not bit-matched. An optuna-compat flag + a decision-level fixture land with Track-Q. Convergence on a mixed continuous+categorical objective is verified in the C++ tests.

  • Not a bit-for-bit clone of Optuna’s TPE (its behaviour depends on undocumented, version-drifting constants); this is a clean-room univariate TPE.

References

  • Bergstra, Bardenet, Bengio & Kégl, Algorithms for Hyper-Parameter Optimization, NeurIPS (2011); Bergstra, Yamins & Cox, Making a Science of Model Search, ICML (2013). See _finetuning_bibliography.bib.