# Pipeline finetuning — tuning operators & sub-pipelines, not just scalars

**Status:** design + phased plan (companion to [`FINETUNING_ROADMAP.md`](FINETUNING_ROADMAP.md) and
[`NATIVE_FINETUNING.md`](NATIVE_FINETUNING.md)). Written after the Phase-1 native HPO
engine landed. **Nothing here is implemented yet** — it plans the step from
"finetune scalar params" to "finetune the pipeline structure itself".

## 1. The goal (what the user wants)

Finetune a **pipeline directly**, with the **operators themselves in the search
space** — going beyond primitive `int/float/categorical` to tune **sub-pipelines
or whole pipelines**, addressed in the Optuna / scikit-learn `object__attribute`
style:

```python
pipeline.finetune({
    "preproc_0": ["snv", "msc", ["sg", {"window": (5, 21, 2), "order": (2, 3)}]],
    "model":     ["pls", "ridge"],
    "model__pls__n_components": ("int", 1, 30),        # active only when model == pls
    "model__ridge__alpha":      ("float", 1e-4, 1e2, "log"),  # active only when model == ridge
}, n_trials=100, sampler="tpe", pruner="asha")
```

A list = a **choice of operator**; a tuple = a **numeric range**; nesting = a
**sub-pipeline**. `a__b__c` addresses a nested slot. Each trial should
**materialise a concrete pipeline**, be fit + CV-scored, and the best pipeline
refit — identically in Python / R / MATLAB / WASM.

## 2. What works TODAY (verified in the native substrate)

The native ask/tell optimizer (ABI 2.1) already has the pieces for a **flat
pipeline, one level of operator-choice per slot**:

| Capability | Native primitive | Evidence |
|---|---|---|
| "which operator" choice | `n4m_search_space_add_categorical` (str/int/float/bool) | `test_space_build` |
| attribute active only for a chosen operator | `N4M_CONSTRAINT_CONDITION_IN` / `_NOT_IN` (child active iff parent label ∈ set) | `test_conditional_activation` (`gamma` active iff `kernel=="rbf"`) |
| deactivating a slot cascades to its attributes | `#` name cascade — deactivating `X` deactivates all `X#*` | `apply_conditions` in `optimizer.cpp` |
| host reads what to build | `n4m_trial_is_active(name)` | `test_conditional_activation` |
| a sampler that groks conditional/tree spaces | `N4M_SAMPLER_TPE` (Tree-structured Parzen) | `test_tpe_converges_mixed` |

**So today, by hand, a host could already compile** a one-level space like the
example's `model` + `model__pls__n_components` + `model__ridge__alpha`: declare the
`model` categorical, then `n_components` with a `CONDITION_IN(model=="pls")` and
`alpha` with a `CONDITION_IN(model=="ridge")`, ask/tell, and read `is_active` to
build the chosen branch. `object__attribute` is just a host naming convention the
optimizer already tolerates (native cascade separator is `#`; a controller can
present `__` and translate).

## 3. What's missing (the gap between "works by hand" and the goal)

### 3a. The pipeline ⇄ search-space compiler + materialiser (the bulk — Phase 2)

Nobody turns a **pipeline-with-tunable-slots** into the flat conditional space and
back:

- **compile**: walk the pipeline spec → emit the categorical slot params + each
  operator's attribute params + the conditional-activation constraints + the
  hierarchical names. Recurse for sub-pipelines.
- **materialise**: given a proposed trial, read the active slots/attrs
  (`is_active` + getters) and rebuild a **concrete pipeline/graph** to fit.
- **eval loop**: host-drives-eval (already the ABI model) — for each trial,
  materialise → CV → `tell()` (+ optional `tell_intermediate` for pruning on the
  `n_components` fidelity axis) → refit the best.

Per the North Star this is **coordination/orchestration**, so it belongs in
**`dag-ml`** (graph compile/plan/materialise/refit — cross-language, so R/WASM/
MATLAB get pipeline finetuning too) plus a **`nirs4all` Python controller** for the
`object__attribute` UX. It is **not** numerical-kernel work.

### 3b. Native ABI gaps that make deep/branchy spaces robust (small — Phase 1.5)

The substrate handles one level cleanly but is fragile for the fully-general case:

- **Single parent per condition.** A child may be gated by exactly one parent
  (`optimization.h:43`; a 2nd parent → `N4M_ERR_UNSUPPORTED`). So "attr active when
  op==sg **and** mode==derivative" (multi-gate) is unexpressible.
- **Label-only cascade, not active-state-aware.** `apply_conditions` is a single
  pass keying off the parent's *label*, with the `#`-prefix carrying "parent
  inactive ⇒ children inactive". Correct for one level; for **deep nesting** (a
  child that is itself a parent) it relies entirely on careful `#`-naming and is
  not validated topologically.
- **No first-class "choice / branch sub-space".** Declaring "slot = one of {branch
  specs}" today means hand-wiring N conditions + a naming discipline. A first-class
  node would remove that boilerplate and let the sampler reason about the branch
  structure directly.
- **No variable-length structure.** "A pipeline of length 1–5, each step one of N
  operators" has no primitive. A **bounded** version flattens to K conditional
  slots (pure convention, no ABI); a genuinely dynamic/unbounded space is a bigger
  research item, out of scope for v1.

## 4. Design — `object__attribute` → flat conditional space

```
USER SPEC (nirs4all Python controller, Optuna-style)
    "model": ["pls", "ridge"]
    "model__pls__n_components": ("int", 1, 30)
    "model__ridge__alpha": ("float", 1e-4, 1e2, "log")
    "preproc_0": ["snv", "msc", ["sg", {"window": (5,21,2), "order": (2,3)}]]
        │
        ▼  COMPILE  (dag-ml, cross-language)
NATIVE SEARCH SPACE
    categorical  model {pls, ridge}
    categorical  preproc_0 {snv, msc, sg}
    int          model#pls#n_components [1,30]     COND_IN(model, "pls")
    log_float    model#ridge#alpha [1e-4,1e2]      COND_IN(model, "ridge")
    int          preproc_0#sg#window [5,21] step 2 COND_IN(preproc_0, "sg")
    int          preproc_0#sg#order  [2,3]         COND_IN(preproc_0, "sg")
        │
        ▼  ASK  (native optimizer, any binding)
TRIAL  {model="pls", model#pls#n_components=12, preproc_0="snv", … (sg#* inactive)}
        │
        ▼  MATERIALISE  (dag-ml)  — read is_active + getters
CONCRETE PIPELINE  [SNV] → [PLSRegression(n_components=12)]
        │
        ▼  EVAL (host loop)  materialise → CV → tell(score)  [+ tell_intermediate → prune]
        ▼  best trial → refit
```

The compiler is a pure function `pipeline_template → (search_space, name_map)`;
the materialiser is `(trial, template) → concrete_pipeline`. Both are deterministic
and live once in `dag-ml`, so every binding gets identical behaviour — the same
cross-binding guarantee the Track-Q golden traces already enforce for scalars.

## 5. Phased plan

### Phase 1.5 — native enablers (this repo, `nirs4all-methods`) — OPTIONAL, ABI-additive

Only needed for **deep/branchy** spaces; one-level pipelines work without them.
Recommended order (each is a small, Codex-reviewed, parity-gated block like the
Phase-1 samplers):

| ID | Enabler | ABI impact | Status |
|---|---|---|---|
| **E2** | Topological, active-state-aware conditional cascade in `apply_conditions` — a child is active iff parent is active **and** label matches, deactivate-only so declaration order is irrelevant. | Behavioural only (no symbol) | ✅ **done + Codex-reviewed** (merged to main). |
| **E1** | Multi-parent conditions (`CONDITION_ALL_IN`, or allow repeated parents with AND semantics). | Enum value (ABI minor, reserved slot) | ⬜ deferred — not needed by the current nirs4all integration (single-parent `when` covers it). |
| **E3** | First-class choice/branch node: `n4m_search_space_add_choice(slot, branch_labels…)` + branch-scoped param helpers. | New symbols (ABI minor) | ⬜ deferred — the host/controller compiles choices→conditions today (belongs in the orchestrator per the North Star). |
| **E4** | Bounded variable-length convention (K slots each "operator-or-none") documented + a helper. | Convention (no ABI) or one helper | ⬜ deferred. |

**Delivered on top of E2 (Phase 2, nirs4all Python):** the `finetune_params`
**`when`/`when_not` clause** — an attribute active only when a sibling categorical
matches a value — compiled into native conditional-activation constraints. This is
`object__attribute` conditional finetuning for a single model (e.g. SVR `gamma`
active only for `kernel ∈ {rbf, poly}`). See
`nirs4all/optimization/n4m_engine.py` and example `U08_native_finetuning.py`. Full
operator/sub-pipeline *materialisation* (choosing preprocessing operators, whole
sub-pipelines) remains the dag-ml compiler work (P1–P6 below).

Each ships with: C++ core + test, a `parity/hpo/` spec exercising the conditional
structure (extend `HpoSpec` with a `structural` cell + golden trace), doc + bib,
ABI snapshot regen if E1/E3 add a symbol. **These bump the ABI minor**, so they
need your green light (the standing rule: no ABI change without sign-off).

### Phase 2 — the actual pipeline finetuning (`dag-ml` + `nirs4all-core` + `nirs4all` Python)

The substance. **Touches other repos → blocked on your green light + Phase-2
start** (and an agent may be active in dag-ml/nirs4all-core — coordinate).

| ID | Deliverable | Repo |
|---|---|---|
| **P1** | Tunable-graph annotation: mark graph nodes/params tunable (choice / range / nested). | `dag-ml` |
| **P2** | `compile(template) → native search space` + `materialise(trial, template) → graph`. | `dag-ml` (calls the native search-space + optimizer via its C-ABI) |
| **P3** | Host-drives-eval loop over materialised graphs (CV, `tell`, optional pruning on `n_components`). | `dag-ml` |
| **P4** | `pipeline.finetune({...}, sampler=…, pruner=…)` — the `object__attribute` UX; wraps P1–P3 + the native optimizer binding; Optuna stays selectable. | `nirs4all` (Python controller) |
| **P5** | Idiomatic thin wrappers so R / MATLAB / WASM get the same `finetune(template, …)`. | `nirs4all-core` bindings |
| **P6** | Extend Track-Q: golden **pipeline-space** traces (same template + seed → same materialised pipeline sequence, every binding). | `nirs4all-methods` `parity/hpo/` (contract) + per-repo runners |

### Dependencies

```
E2 ─┬─► E1 ─► E3 ─► (E4)        [Phase 1.5, nirs4all-methods, ABI minor]
    └───────────────► P2 ──► P3 ──► P4 ──► P5
                 P1 ─┘                 └──► P6 (contract back in nirs4all-methods)
```

P2 can start against **today's** substrate (one-level spaces) and adopt E1/E3 as
they land — so Phase 2 is **not blocked** on Phase 1.5; the enablers upgrade it
from "flat pipelines" to "arbitrary nested sub-pipelines".

## 6. Verdict

- **Flat pipeline, one operator-choice per slot with gated attributes:** possible
  **now**, no native change — a host compiler over the existing conditional space.
- **Sub-pipelines / whole-pipeline / deep nesting in `object__attribute` form:**
  **not yet** — needs the Phase-2 compiler+materialiser+controller (the bulk) and,
  for robustness/ergonomics, the Phase-1.5 native enablers E1–E3.
- **None of it requires new numerical kernels** — the optimizer, samplers and
  pruners are done. The remaining work is *space construction* (small, native) and
  *orchestration* (large, dag-ml + Python), exactly the split the North Star
  predicts.