Finetuning implementation roadmap — native `libn4m` HPO, all languages¶

Status: implementation roadmap (companion to NATIVE_FINETUNING.md, which holds the why and the architecture). Revised twice: after an internal adversarial pass (corrections marked [review]) and after a Codex read-only review (corrections marked [codex], transcript in reviews/finetuning-roadmap/). Goal: a native finetuner in nirs4all-methods — every HPO algorithm from the strategy implemented once in C++/C-ABI, reachable from a nirs4all pipeline in every language (Python, R, MATLAB/Octave, WASM), with parity, docs and bibliography like every other method, while Optuna stays in Python as an optional sampler. End-user acceptance: in a pipeline, a model step’s finetune_params resolves to a native sampler (sampler:"random"|"sobol"|"cmaes"|"tpe"|…, optionally pruner:"asha"|"hyperband"|"median") and runs identically in Python/R/MATLAB/WASM; sampler:"optuna" keeps the Python engine. Same seed → bit-identical trial sequence across bindings.

0. How to read this roadmap¶

Unit of delivery. The finetuner is a new C-ABI category (optimization, header cpp/include/n4m/optimization.h, included by n4m.h) with one handle-based ask/tell surface and many samplers/pruners behind two enums. The ABI surface lands once (Phase F0); each later sampler is a Sampler/Pruner subclass + a parity fixture + a doc/bib page.
Sampler ⟂ pruner is a hard split. [review] A sampler proposes; a pruner/scheduler decides early-stopping over intermediate scores. They compose (sampler=tpe × pruner=median), like Optuna’s sampler=/pruner=. n4m_optimizer_create takes both via an options struct (§4-F0). Never flatten pruners into the sampler enum.
Enum values ARE an ABI-minor change. [codex] Adding a value to n4m_sampler_kind_t/n4m_pruner_kind_t exports no new symbol, but the repo’s own rule classes a new enum value as ABI MINOR (cpp/include/n4m/n4m_version.h:11). Therefore F0 reserves the full numeric range of every enum now; unimplemented values return N4M_ERR_UNSUPPORTED until their phase lands — so no later phase touches the ABI at all. (Bindings still add their language-side enum mirror + docs when a value is activated; that is not a libn4m ABI change.)
Definition of done per sampler/pruner — the 6-surface method-add checklist (.github/PULL_REQUEST_TEMPLATE/method-add.md) applies, minus new symbols after F0:
1. C++ core — a Sampler/Pruner subclass under cpp/src/core/optimization/<name>.{hpp,cpp}. [review] New core .cpp are not auto-globbed — list them explicitly in add_library(n4m_core OBJECT …) in cpp/src/CMakeLists.txt (only cpp/src/c_api/*.cpp is CONFIGURE_DEPENDS-globbed via n4m_targets.cmake, so c_api_optimization.cpp auto-compiles).
2. ABI — no new symbol; activate a reserved enum value. Enums are 4-byte (N4M_STATIC_ASSERT(sizeof(...)==4, …)).
3. Tests — cases in cpp/tests/test_optimization.cpp, compiled into the single n4m_tests binary by the hand-rolled zero-dependency harness (cpp/tests/harness.hpp + main.cpp — not doctest, despite older CLAUDE.md wording); register the file in cpp/tests/CMakeLists.txt. Cover the ask/tell contract, determinism-given-seed, a converging non-trivial objective, and a correct prune.
4. Reference & parity — needs a NEW schema, the prediction-oriented one does not fit. [codex] benchmarks/parity_timing/registry.py’s MethodSpec (:8668) and parity/scripts/per_method_parity.py compare prediction arrays (:179, :265) — they cannot hold a pruner decision fixture, a TPE/CMA state fixture, or an ask/tell trace. Add an HpoSpec registry schema + dedicated comparators (parity/comparator/{pruner_decisions,sampler_state,study_trace}.py) + fixtures (parity/fixtures/opt_<name>_v1.json) + an explicit CI job (the cross-binding runner is currently manual, not a gate — .github/workflows/parity-gate.yml:157). This machinery is itself an F0/F2 deliverable, not a reuse.
5. Catalog — catalog/methods/optimization.<name>.yaml (regenerated from catalog/methods.yaml via split_legacy_methods.py, which maps an arbitrary dotted method_id to a file — two- or three-segment both exist, e.g. catalog/methods.yaml:675). Specify namespace/leaf/c_surface + reference coverage; run catalog/scripts/validate.py --strict-abi and --check-references (a separate gate; the donor-category allow-list at validate.py:43 must gain optimization).
6. Bindings + docs — thin wrapper where in scope; docs/methods/<name>.md with bibliography; CHANGELOG.md; dashboard timing (benchmarks/cross_binding/donor_ops.py) if benchmarked.
Parity model reused, adapted for stochasticity (§3): cross-binding stays the tight 1e-12 gate (bindings vs C++ native, same seed → same study); reference parity is per-sampler (bit-exact only for deterministic math, state-level for TPE/CMA-ES, decision-level for pruners).
Optuna coexistence is a hard invariant. Nothing removes nirs4all.optimization.optuna. The native engine is added alongside and selected by a string; in R/MATLAB/WASM it is the only engine.

1. Target end-state (what “done” looks like)¶

pipeline step:  { "model": PLSRegression(),
                  "finetune_params": { "n_trials": 60, "sampler": "tpe", "pruner": "asha",
                                       "metric": "rmse", "eval_mode": "mean",
                                       "model_params": { "n_components": ("int", 1, 30) } } }

Python  → OptunaManager (default)  OR  native optimizer (routed by string)          ← both available
R       → n4m_tune(spec, X, y, sampler="tpe", pruner="asha", seed=…)                 ← native only
MATLAB  → n4m.finetune(X, Y, space, struct('Sampler','tpe','Pruner','asha', …))      ← native only
WASM    → await n4m.finetune({ searchSpace, sampler:"tpe", pruner:"asha", … })       ← native only

Every route: (a) accepts the same search-space DSL (or loudly rejects the unsupported subset, §3-D6); (b) produces a TrialResult stream with per-trial status ∈ {completed, pruned, failed}, score, duration, error; (c) is seedable, timeout-bounded, resumable, warm-startable; (d) at a fixed seed yields bit-identical trials across bindings (native samplers only). Each sampler/pruner has a doc page + citations and a parity verdict in CI.

2. The sampler & pruner set (single source of truth for scope)¶

2a. Samplers — `n4m_sampler_kind_t`¶

#	id (`optimization.<id>`)	Algorithm	Space kinds	Parity tier (§3)	Phase	Build
1	`random`	Uniform random search	all	Tier C	F1	reuse RNG
2	`sobol`	Sobol low-discrepancy	int/float/log	A unscrambled; B scrambled	F1	low-new
3	`lhs`	Latin Hypercube	int/float/log	Tier C	F1	low-new
4	`ternary`	Ternary unimodal-int search (ports `BinarySearchSampler`)	single unimodal int	Tier A	F1	low-new
5	`ga`	GA over typed candidates (generalize `ga_select`)	discrete/categorical/perm	Tier B-state	F3	reuse loop
6	`pso`	PSO over typed candidates (generalize `pso_select`)	binary/categorical	Tier B-state	F3	reuse loop
7	`cmaes`	CMA-ES (+ mandatory independent-random fallback for categorical/conditional dims)	continuous/ordinal	Tier B-state	F4	med-new
8	`tpe`	Tree-structured Parzen Estimator (conditional/tree handling is core)	mixed + conditional	Tier B-state	F4	high-new
9	`gp_ei`	GP + Expected Improvement	small int/float	Tier B — optional / cuttable	F4	high-new

grid is NOT a libn4m sampler [review]. Exhaustive enumeration is dag-ml’s enumerate_variants (the parity-locked single source of truth); a small finite native space is covered by random/sobol over categoricals.

Categorical values & string lifetime [codex]. Optuna categories may be bool|int|float|str. The ABI takes typed categorical payloads (a tagged value list), not only const char*; returned labels are core-owned UTF-8 copies valid until the search space is destroyed (never host-freed, per rule 3), and the integer index is the canonical identity for cross-binding parity. String-only would silently break bool/int categories round-tripping through Optuna’s DSL.

Warm-start / enqueue. Every sampler accepts host-injected candidates via n4m_optimizer_enqueue(...) (Optuna’s enqueue_trial/force_params): evaluate the published/current-best preprocessing chain first; seed from a prior study for nirs4all-papers reproducibility. Reserved in F0.

2b. Pruners / schedulers — `n4m_pruner_kind_t` [review]¶

id	Algorithm	Fidelity axis	Parity tier	Phase
`none`	no early stopping	—	—	F0
`median`	median/percentile stopping (Vizier)	any intermediate-score stream	Tier B-decision	F2
`asha`	asynchronous successive halving	see §2c	Tier B-decision	F2
`hyperband`	bracket scheduler over ASHA (owns (n,r) allocation, pairs with a base sampler)	see §2c	Tier B-decision	F2
`racing`	Hoeffding/Bernstein racing (fold-safe early-stop, §2c)	CV folds	Tier B-decision	F2

Pruner verdicts are pure functions of the intermediate-score history + schedule (no RNG) — the one cleanly decision-testable case.

2c. Fidelity axis — the load-bearing correction [review + codex]¶

Successive-halving/Hyperband/ASHA assume rank-preservation across fidelities. fidelity = fraction of CV folds violates it: folds are exchangeable, so a fold-subset is a higher-variance estimator of the identical target, not a lower-fidelity proxy — with “smaller k” it systematically prefers smaller n_components (fewer training samples ⇒ fewer stable latent variables), so ASHA can prune the config that wins at full data (rank inversion). Fold count also gives ≤5× dynamic range, far short of what Hyperband’s guarantees assume.

The native fidelity axes, in priority:

PLS n_components learning curve (primary) — with a HARD scope caveat [codex]. The claim “CV-RMSE at 1..K is a free by-product of one fit” is true only for the exact PLS1 / NIPALS / regression-deflation sweep path (cpp/src/core/sweep.cpp:2052, sweep.hpp:3, q == 1), and even there it is one max-K prefix fit per CV fold (cpp/tests/test_sweep.cpp:525), not one global fit. The generic component-CV path loops k=1..K calling CV each time (cpp/src/core/model_selection.cpp:69) — no free prefix. F2 must therefore either (a) scope the learning-curve fidelity to the eligible PLS1/NIPALS route, or (b) first build a real prefix-CV engine for SIMPLS/multi-target. State which per method.
Subsample fraction (general). Increasing sample fraction; honest low-fidelity, caveated for very small n.
Epochs (DL, host-side, Flavor C). Host reports per-epoch validation as rungs.
CV folds → racing, not ASHA. The correct fold-based early-stop is statistical racing (Hoeffding/Bernstein; Maron & Moore). Document the rank-preservation assumption in parity/tolerances.md.

tell_intermediate(trial, step, value) is axis-agnostic; step is the rung (a n_components count / subsample rung / epoch).

2d. `auto` policy¶

Finite categorical → dag-ml grid. Single unimodal int → ternary. Combinatorial → ga/pso. Continuous, budget≥~30 → cmaes (independent fallback for non-continuous dims), else gp_ei (if built). Mixed/conditional budget≥~20 → tpe. First ~10 trials sobol/random. Pruner defaults to asha on the n_components axis where the learning curve is eligible, else median, else none.

3. Decisions that MUST be locked in F0 (they change the frozen ABI)¶

The F0 ABI is a one-shot freeze, so the entire shape — options, results, status, batch, metric, constraints — must be settled now or a later phase forces a second ABI-minor break.

D1. Sampler ⟂ pruner + grid-is-dag-ml + Hyperband owns bracket scheduling. (§0, §2)
D2. Fidelity axis = n_components learning-curve (scoped to eligible PLS1/NIPALS), subsample, epochs; racing for folds; the tell_intermediate step semantics. (§2c)
D3. Async pruning (ASHA/median) so tell_intermediate → out_should_prune is sound per-tell.
D4. Result/status model [codex, Blocker]. tell(score) alone cannot represent a failed trial, a pruned terminal state, a timeout/cancellation, an error string, or wall-clock. Freeze: n4m_trial_status_t {RUNNING, COMPLETED, PRUNED, FAILED}; a terminal n4m_optimizer_tell_result(opt, trial_id, status, score, const char* error) (with tell(score) = tell_result(COMPLETED, score, NULL)); per-trial duration; and the TrialResult stream carrying all of it (matches Optuna’s TrialSummary/FinetuneResult — nirs4all/optimization/optuna.py:281,445).
D5. Batch / parallel ask [codex, Major]. Studio is async/streamable; a scalar ask with no imputation makes parallel asks propose duplicate candidates. Freeze either n4m_optimizer_ask_batch(opt, n, out[]) or an active-trial-aware scalar ask with an explicit constant-liar strategy set in the options struct. Reproducibility is guaranteed for sequential tell-order; parallel documents the liar policy.
D6. Metric / task / direction ABI [codex, Major]. A bare int32_t metric is regression-only. nirs4all resolves direction from metric and task type, covering regression and classification (nirs4all/core/metrics.py:8,119; optuna.py:578). Freeze an objective/metric surface with: a metric enum spanning both task types, a direction (auto-from-metric or explicit), multi-target aggregation, and multi-objective explicitly reserved/unsupported (a reserved vector-objective shape, returning N4M_ERR_UNSUPPORTED until built).
D7. Native-routable DSL subset [codex, Major]. The Optuna DSL also has sorted_tuple and static scalar pass-through (optuna.py:1146,1299). F0 either implements these param kinds or the native path loudly rejects any finetune_params key it can’t honor (sorted_tuple, fold_strategy:"individual" beyond host-looping, eval_mode if not native, unknown keys) — never silently drops — so sampler:"tpe" and sampler:"optuna" optimize the same space and objective.
D8. SearchSpace constraint richness [codex, Major]. dag-ml’s generation constraints are mutex/requires/exclude plus pick/arrange/count and multi-parent conditions (nirs4all/pipeline/dagml/detect.py:263). A single-value set_condition(child, parent, parent_equals) is too weak → Flavor A (grid) and Flavor B (native) would evaluate different spaces. Freeze a generic constraint ABI (n4m_search_space_add_constraint(kind, refs[], n) with kind ∈ {mutex_group, requires, exclude, condition_in, condition_not_in} + multi-parent), co-designed JSON-identical with the dag-ml SearchSpace contract, with explicit N4M_ERR_UNSUPPORTED for anything it can’t express.
D9. Forward-compatible options struct [codex]. n4m_optimizer_create(ctx, space, const n4m_optimizer_options_t* opts, out) where n4m_optimizer_options_t { size_t struct_size; sampler_kind; pruner_kind; direction; eval_mode; metric; liar_kind; uint64_t seed; double timeout_seconds; int32_t n_startup_trials; /* reserved[] */ }. The struct_size prologue lets fields be added later without an ABI break.
D10. Resume mechanism = replay-based (record ask/tell log; create+replay; needs D5’s order) or a versioned little-endian n4m_optimizer_save/load blob (reserve those symbols in F0 if chosen). No handle-serialization precedent exists in libn4m.
D11. n4m_finetune_estimator is a thin driver, not a second control loop [codex]. It reuses the same ask/tell primitives, the identical TrialResult + best-param schema, and a single validation_plan (no nested CV, no selection, no leakage enforcement — those belong to dag-ml). Documented as single-level convenience so it cannot diverge from the dag-ml Tuner path; alternatively deferred until the dag-ml contract is frozen.

Parity tiers (unchanged intent, restated): Tier A = deterministic math bit-exact (ternary; unscrambled Sobol only, vs scipy.stats.qmc.Sobol(scramble=False, bits=30) with pinned direction file + index convention — the useful scrambled Sobol is Tier-B behavioral). Tier B-decision = pruners on canned history (RNG-free verdicts, clean). Tier B-state = TPE/CMA-ES assert state (split index / distribution update), never the RNG-entangled sample. Tier C = self-consistency + regret (random, lhs). Cross-binding is Tier-A-tight (1e-12) for every native sampler.

5. Bibliography (per sampler — canonical citations for the doc pages)¶

Collect in docs/methods/_finetuning_bibliography.bib. [review] sobol venue corrected (SIAM J. Sci. Comput., not ACM TOMS); keep both Joe & Kuo papers distinct.

Sampler / pruner	Primary reference(s)	Reference impl (donor)
`random`	Bergstra & Bengio, Random Search for Hyper-Parameter Optimization, JMLR 13 (2012), 281–305	self / nirs4all
`sobol`	Sobol′, USSR Comput. Math. & Math. Phys. 7 (1967), 86–112; Joe & Kuo, Constructing Sobol Sequences with Better Two-Dimensional Projections, SIAM J. Sci. Comput. 30 (2008), 2635–2654 (scipy’s direction numbers); cf. Joe & Kuo, Remark on Algorithm 659, ACM TOMS 29 (2003), 49–57	`scipy.stats.qmc.Sobol`
`lhs`	McKay, Beckman & Conover, Technometrics 21 (1979), 239–245	`scipy.stats.qmc.LatinHypercube`
`ternary`	ports nirs4all `BinarySearchSampler` (unimodal ternary search)	nirs4all (self-consistency)
`median`	Golovin et al., Google Vizier, KDD (2017) — median stopping rule	Optuna `MedianPruner`
`asha`	Jamieson & Talwalkar, Non-stochastic Best Arm Identification and HPO, AISTATS (2016); Karnin, Koren & Somekh, ICML (2013); Li et al. (ASHA), MLSys (2020)	Optuna `SuccessiveHalvingPruner`
`hyperband`	Li, Jamieson, DeSalvo, Rostamizadeh & Talwalkar, Hyperband, JMLR 18 (2018) / ICLR (2017)	Optuna `HyperbandPruner`
`racing`	Maron & Moore, Hoeffding Races, NeurIPS (1993); Mnih, Szepesvári & Audibert, Empirical Bernstein Stopping, ICML (2008)	nirs4all (self-consistency)
`cmaes`	Hansen & Ostermeier, Evol. Comput. 9 (2001), 159–195; Hansen, CMA-ES Tutorial (2016), arXiv:1604.00772	`pycma`
`tpe`	Bergstra, Bardenet, Bengio & Kégl, NeurIPS (2011); Bergstra, Yamins & Cox, ICML (2013)	Optuna `TPESampler`
`gp_ei`	Jones, Schonlau & Welch, J. Glob. Optim. 13 (1998), 455–492; Snoek, Larochelle & Adams, NeurIPS (2012)	`scikit-optimize`
`ga`	Leardi & Lupiáñez González, Chemom. Intell. Lab. Syst. 41 (1998), 195–207	R `genalg` / nirs4all (`ga_select` donor)
`pso`	Kennedy & Eberhart, A discrete binary version of the particle swarm algorithm, IEEE SMC (1997), 4104–4108	`pyswarms` BinaryPSO (`pso_select` donor)

6. Two-phase execution & agent parallelism¶

The work is split into two execution phases to (a) let the finetuner stabilize inside nirs4all-methods before anything downstream depends on it, and (b) not disturb a concurrent agent working in dag-ml / nirs4all-core. The F-phases above (F0–F7) map onto these two execution phases.

6.1 Is the two-phase constraint well-founded? — Yes, and it is the natural architecture¶

Phase 1 (methods) is 100% self-contained. F0–F4 + the methods-side bindings + the HPO parity machinery all live in nirs4all-methods. Zero edits to dag-ml, dag-ml-data, or nirs4all-core.
Cross-language finetuning of native models already works at the end of Phase 1 — without dag-ml. The methods bindings (R/WASM/MATLAB/Python) expose the optimizer directly, so a binding user can finetune a native PLS/estimator in any language before Phase 2 starts. dag-ml is needed only for pipeline-orchestrated finetuning (native OOF/leakage/selection/lineage), cross-language pipeline reachability, and host-model (Flavor C) coordination.
The one coupling is neutralized. F0’s SearchSpace constraint ABI (D8) is designed against the already-implemented, parity-locked dag-ml generation vocabulary (generation.rs, detect.py:263) — not against unwritten F5a work — so Phase 1 freezes it unilaterally, with no coordination with the dag-ml agent. Phase 2’s F5a then maps its JSON contract onto the already-frozen ABI.
Phase 2 depends on Phase 1, never the reverse — exactly the requested ordering (Phase 1 stable & validated → Phase 2).

So the constraint is not a compromise; it is how the layering already wants to run.

6.2 The phase boundary¶

Execution phase	Repos touched	F-phases	Deliverable
Phase 1 — engine	`nirs4all-methods` only	F0, F1, F2, F3, F4 + methods bindings + HPO parity/CI	A stable, validated, parity-gated native finetuner in `libn4m`, idiomatic and bit-reproducible in every binding, callable directly (incl. `n4m_finetune_estimator` from R/WASM/MATLAB/Python).
Phase 1-bridge — Python consumer (optional, dag-ml-free, safe to run any time after Phase 1’s Python binding)	`nirs4all` (Python) — not dag-ml/core	the dag-ml-free half of F5c + F6-Python	A `NativeOptimizerManager` behind the controller `_execute_finetune` dispatch + `N4MSearchCV` + best-param normalization, reusing nirs4all’s Python-engine fold/eval/refit machinery (the same path `OptunaManager` uses today). Result: `sampler:"tpe"` works in a nirs4all Python pipeline. Touches neither dag-ml nor nirs4all-core, so it does not disturb the other agent.
Phase 2 — orchestration (after Phase 1 is stable; coordinate with / hand to the dag-ml agent)	`dag-ml`, `dag-ml-data`, `nirs4all-core` (+ the dag-ml-backend half of nirs4all)	F5a, F5b, core re-export, dag-ml-backend routing, Flavor C, F7	Native finetuning through the dag-ml engine: cross-language pipeline reachability, native OOF/leakage/selection/lineage, host-model tuning, Studio persistence/streaming.

Note on F5c. Codex framed F5c as cross-repo because routing finetune through the dag-ml backend needs the dag-ml Tuner. But the nirs4all Python engine can drive the native optimizer directly (host-drives-the-loop), bypassing dag-ml — that half of F5c is Phase 1-bridge and dag-ml-free. The dag-ml-backend half of F5c stays in Phase 2.

6.3 Phase 1 dependency DAG — maximum agent parallelism¶

F0 is the only serialization point (it freezes the ABI everything compiles against — see FINETUNING_F0_PR.md). Once it merges, Phase 1 forks into independent tracks against the frozen optimization.h:

                          ┌──────────────────────────── F0 (ONE PR — ABI freeze + random/none slice) ───────────────────────────┐
                          │  serial gate: header + enums + options + ask/tell + finetune_estimator + ABI snapshots + parity CI  │
                          └───────────────────────────────────────────────┬───────────────────────────────────────────────────┘
   after F0 merges, all tracks run CONCURRENTLY (arrows = intra-track order only):
   ── Track S (samplers) ······  sobol  ∥  lhs  ∥  ternary                         [3 agents, fully independent]
   ── Track P (pruners) ·······  [fidelity-engine agent] → median ∥ asha ∥ hyperband ∥ racing   [1 → 4 agents]
   ── Track G (population) ·····  [RNG-consolidation agent] → ga ∥ pso             [1 → 2 agents; isolated re-freeze]
   ── Track M (model-based) ····  cmaes  ∥  tpe  ∥  (gp_ei?)                        [2–3 agents, independent]
   ── Track Q (parity/CI) ·····  HpoSpec + comparators + cross-binding CI gate     [1 agent, START FIRST — every DoD gates on it]
   ── Track B (bindings) ······  python → ( R ∥ MATLAB/Octave ∥ WASM ) → cross-binding parity gate   [1 → 3 agents]

Concurrency peaks at ~10–14 agents. Immediately after F0: sobol, lhs, ternary, the fidelity-engine, the RNG-consolidation, cmaes, tpe, the parity machinery, and the Python binding can all start at once (~9). As the two sub-dependencies clear (fidelity engine → 4 pruners; RNG consolidation → ga/pso; Python binding → R/MATLAB/WASM), the fan-out grows.

Intra-Phase-1 ordering rules (the only real constraints):

Everything waits on F0 (the ABI freeze). Nothing else in Phase 1 touches the header/version/snapshots.
Track Q starts first. Each sampler/pruner’s merge gate needs the HpoSpec + comparators + CI job. Implementation can proceed against local tests in parallel, but land Q early.
Pruners wait on the fidelity engine (the eligible-PLS1 n_components learning-curve producer, §2c-1) — one shared sub-task, then median/asha/hyperband/racing parallelize.
ga/pso wait on the RNG-consolidation sub-task (isolated because it may re-freeze GA/PSO fixtures — keep it off the critical path of the other tracks).
R/MATLAB/WASM bindings wait on the Python binding proving the wrapper pattern (Python is also the cross-binding parity reference); they then parallelize and can wrap samplers incrementally as each lands.
The cross-binding parity gate (tail of Track B) needs Track Q + the bindings + the samplers under test.

Phase 1 is “done, stable & validated” when: all Phase-1 samplers/pruners pass their tier gate, cross-binding parity is a green CI gate at 1e-12, and n4m_finetune_estimator is validated end-to-end. Only then does Phase 2 begin — handed to (or coordinated with) the dag-ml/core agent, against the now-frozen ABI.

6.4 Phase 2 sequencing (for later; owned by the dag-ml/core agent)¶

Phase-1 frozen ABI ──► F5a (dag-ml-data SearchSpace + constraints, maps onto frozen ABI)
                         └─► F5b (dag-ml Tuner exec: ask/tell in nested CV, Flavor B/C)
                               └─► nirs4all-core re-export  ∥  dag-ml-backend routing in nirs4all
                                     └─► F7 (Studio persistence/streaming; persistence ABI already shipped in F0)

F5a/F5b are independently reviewable; core re-export and dag-ml-backend routing parallelize once F5b lands.

7. Per-phase acceptance gates (summary)¶

Always: cmake --build --preset dev-release + ctest --preset dev-release; clang-format/clang-tidy; git diff --check.
F0 (ABI blocker): ABI symbol diff clean vs regenerated expected_symbols_{linux,macos,windows}.txt; docs/abi/changes_log.md; bump_version.sh --check; SONAME/linkage; complete surface reserved (options/result/status/batch/metric/constraints/enqueue/since_id/save-load-if-chosen) so no second blocker PR; the HPO parity CI job exists (even if only random is wired).
Per sampler/pruner: n4m_tests case incl. non-trivial convergence / correct prune; parity fixture frozen; verdict at its tier (A ≤1e-12; B-decision verdict match; B-state distribution/split match; C self-consistency + regret); validate.py --strict-abi --check-references green; docs/methods/<name>.md + bib; CHANGELOG.md.
F5a/b/c: validate_contracts.py SearchSpace + constraints byte-identical; dag-ml cargo fmt/clippy/test; nirs4all pipeline acceptance (native drives a real pipeline; best-params refit identical; Optuna path unchanged).
F6: cross-binding parity at 1e-12 as a CI gate; per-binding smoke; Optuna-coexistence test.
F7: Studio live-progress + resume + warm-start; dashboard timing refreshed.

8. Risks & mitigations¶

PLS learning-curve fidelity over-scoped [codex]. Free prefix-CV exists only for the PLS1/NIPALS sweep route and is per-fold, not global; generic CV re-fits per k. Mitigation: scope F2 to the eligible route or build a prefix-CV engine for SIMPLS/multi-target first. (§2c-1)
Fold-fraction fidelity is ML-unsound. Mitigation: n_components learning-curve / subsample / racing; document in tolerances.md. (§2c)
F0 result/status/metric/constraint under-freeze → a second ABI-minor break [codex]. Mitigation: freeze the full options/result/status/batch/metric/constraint surface + reserve all enum values now (D4–D9); struct_size prologue for forward-compat.
Two control loops diverge (n4m_finetune_estimator vs dag-ml Tuner) [codex]. Mitigation: the native entry is a thin single-level driver over the same primitives + schema; nested CV / selection / leakage stay in dag-ml (D11).
HPO parity has nowhere to live [codex]. Mitigation: HpoSpec + dedicated comparators + a real CI job — designed in F0, not retrofitted onto the prediction-only MethodSpec.
Cross-repo sequencing hidden by a single “F5” [codex]. Mitigation: F5a/F5b/F5c each with cross-repo gates; acceptance spans methods + dag-ml + nirs4all.
Sampler/pruner conflation / DSL divergence. Mitigation: split enums (D1); native-routable subset with loud rejects (D7); constraint richness (D8).
Parallel ask duplicates trials. Mitigation: ask_batch / active-trial-aware ask + constant-liar (D5).
Parity over-claims (Sobol/TPE). Mitigation: unscrambled-only Sobol Tier-A; TPE/CMA-ES assert state; pruners decision-level. (§3)
RNG consolidation moving GA/PSO numbers. Mitigation: in F3, not the ABI PR; re-freeze with a note.
CMA-ES on mixed spaces / GP-EI scope creep. Mitigation: mandatory independent fallback; GP-EI explicitly optional.
Categorical string lifetime / typed values [codex]. Mitigation: core-owned label copies, canonical index, typed categorical payloads.

9. What this roadmap does NOT do (scope guard)¶

Does not remove or reduce Optuna in Python — Optuna stays a first-class default-in-Python sampler.
Does not put DL objective evaluation in libn4m — torch/TF training stays host-side (Flavor C: native ask, host evaluation with epoch-fidelity, native tell).
Does not chase bit-parity with Optuna’s stochastic samples — cross-binding bit-parity (native RNG) is the guarantee; reference parity is per-tier.
Does not re-implement grid enumeration in libn4m — that is dag-ml’s parity-locked oracle.
Does not support multi-objective in v1 — reserved as N4M_ERR_UNSUPPORTED (D6).
Does not add a mandatory dependency — every sampler is against the from-scratch core.

Companion reconciliation. NATIVE_FINETUNING.md §4.2/§4.4/§5/§6 have been annotated/corrected to match this roadmap (sampler ⟂ pruner, grid-is-dag-ml, n_components-not-fold fidelity, pruner/eval_mode controls, n4m.model_selection not n4m.sklearn). This roadmap is authoritative where they still differ.

nirs4all-methods

Navigation

Finetuning implementation roadmap — native `libn4m` HPO, all languages¶

0. How to read this roadmap¶

1. Target end-state (what “done” looks like)¶

2. The sampler & pruner set (single source of truth for scope)¶

2a. Samplers — `n4m_sampler_kind_t`¶

2b. Pruners / schedulers — `n4m_pruner_kind_t` [review]¶

2c. Fidelity axis — the load-bearing correction [review + codex]¶

2d. `auto` policy¶

3. Decisions that MUST be locked in F0 (they change the frozen ABI)¶

4. Phase plan¶

Phase F0 — ABI foundations + decisions (the ONE ABI-blocker PR)¶

Phase F1 — deterministic & baseline samplers (Tier A/C)¶

Phase F2 — pruners & multi-fidelity (highest-value native win — on the right, scoped axis)¶

Phase F3 — population samplers + RNG consolidation¶

Phase F4 — model-based samplers¶

Phase F5 — cross-repo orchestration (reachable from the pipeline) — SPLIT into 3 checkpoints [codex]¶

Phase F6 — bindings + cross-binding gate [codex: paths corrected]¶

Phase F7 — Studio, persistence, streaming, dashboard¶

5. Bibliography (per sampler — canonical citations for the doc pages)¶

6. Two-phase execution & agent parallelism¶

6.1 Is the two-phase constraint well-founded? — Yes, and it is the natural architecture¶

6.2 The phase boundary¶

6.3 Phase 1 dependency DAG — maximum agent parallelism¶

6.4 Phase 2 sequencing (for later; owned by the dag-ml/core agent)¶

7. Per-phase acceptance gates (summary)¶

8. Risks & mitigations¶

9. What this roadmap does NOT do (scope guard)¶

Finetuning implementation roadmap — native libn4m HPO, all languages¶

0. How to read this roadmap¶

1. Target end-state (what “done” looks like)¶

2. The sampler & pruner set (single source of truth for scope)¶

2a. Samplers — n4m_sampler_kind_t¶

2b. Pruners / schedulers — n4m_pruner_kind_t [review]¶

2c. Fidelity axis — the load-bearing correction [review + codex]¶

2d. auto policy¶

3. Decisions that MUST be locked in F0 (they change the frozen ABI)¶

4. Phase plan¶

Phase F0 — ABI foundations + decisions (the ONE ABI-blocker PR)¶

Phase F1 — deterministic & baseline samplers (Tier A/C)¶

Phase F2 — pruners & multi-fidelity (highest-value native win — on the right, scoped axis)¶

Phase F3 — population samplers + RNG consolidation¶

Phase F4 — model-based samplers¶

Phase F5 — cross-repo orchestration (reachable from the pipeline) — SPLIT into 3 checkpoints [codex]¶

Phase F6 — bindings + cross-binding gate [codex: paths corrected]¶

Phase F7 — Studio, persistence, streaming, dashboard¶

5. Bibliography (per sampler — canonical citations for the doc pages)¶

6. Two-phase execution & agent parallelism¶

6.1 Is the two-phase constraint well-founded? — Yes, and it is the natural architecture¶

6.2 The phase boundary¶

6.3 Phase 1 dependency DAG — maximum agent parallelism¶

6.4 Phase 2 sequencing (for later; owned by the dag-ml/core agent)¶

7. Per-phase acceptance gates (summary)¶

8. Risks & mitigations¶

9. What this roadmap does NOT do (scope guard)¶

Finetuning implementation roadmap — native `libn4m` HPO, all languages¶

2a. Samplers — `n4m_sampler_kind_t`¶

2b. Pruners / schedulers — `n4m_pruner_kind_t` [review]¶

2d. `auto` policy¶