Tunable & explainable pipelines — cross-language synthesis

Status: decision + planning synthesis (spans dag-ml, nirs4all-methods, nirs4all). Companions: NATIVE_FINETUNING.md (why native HPO), FINETUNING_ROADMAP.md (the libn4m optimizer roadmap), PIPELINE_FINETUNING.md (the earlier object__attribute plan). This document supersedes the “how do we make it cross-language” parts of those.

Codex independent review: done (high reasoning effort, all three repos). It converges with this recommendation, with three material corrections now folded throughout and summarised in §7. Headline: “Adopt the estimator-facade approach. Do not build a tuner or SHAP engine inside dag-ml, and do not link dag-ml directly to libn4m. Compose them in the bindings.” — but “the ‘no dag-ml change at all’ part is too optimistic; a small cross-language execution-contract addition is unavoidable.”

1. Objectives (owner’s exact ask)

  1. dag-ml exposes a finetuning interface (as it does for Optuna today).

  2. nirs4all-methods provides a cross-language, reproducible set of finetuning methods for nirs4all-core in every language.

  3. The pipeline itself is tunable — by Optuna or the n4m methods — exactly the way an sklearn.Pipeline is tunable.

  4. A dag-ml pipeline is usable as a BaseEstimator-equivalent in multiple languages, so you can:

    • finetune at (sub)pipeline scale, and

    • run an explainer (SHAP) at pipeline scale — because today it is impossible to SHAP e.g. a Torch model buried in a complex pipeline.

Hard constraint: achieve this without changing the repos’ responsibilities and without launching a big new dag-ml build if it can be avoided.

2. The core insight — reify, don’t build a tuner

sklearn works because a Pipeline is a BaseEstimator: fit / predict / set_params / get_params. GridSearchCV, Optuna and SHAP all operate on it uniformly — none of them looks inside. They only need that interface.

So the whole feature reduces to one thing: give a dag-ml (sub-)pipeline the same fit/predict/set_params interface. Everything else is then reuse:

Goal

Met by

Needs a native dag-ml HPO loop?

tune the pipeline (Optuna)

Optuna drives the estimator as a black box

❌

tune the pipeline (n4m, any language)

the libn4m ask/tell optimizer drives the estimator

❌

finetune a sub-pipeline

reify the sub-DAG as an estimator, tune it

❌

SHAP at pipeline scale

a model-agnostic explainer wraps estimator.predict

❌

The last row is the key unlock: a model-agnostic explainer (SHAP KernelExplainer / PermutationExplainer) needs only predict(X). Wrap the pipeline’s predict and SHAP explains the whole pipeline — including a Torch model inside it — because it never sees the Torch model, only the pipeline’s outputs. That is exactly why Torch-in-a-complex-pipeline can’t be explained today, and reification fixes it for free.

dag-ml is the thing being tuned/explained, not the tuner/explainer.

3. Current status (verified in code)

Legend: ✅ exists · 🟡 partial · ❌ missing.

3a. nirs4all-methods (libn4m) — the finetuning kernels

  • ✅ Native ask/tell optimizer: 9 samplers (random, sobol[Tier-A vs scipy], lhs, ternary, ga, pso, cmaes, tpe, gp_ei) + 5 pruners (none, median, asha, hyperband, racing). Stable C-ABI (n4m_optimizer_*, n4m_search_space_*, n4m_trial_*), enum-value-only, cross-binding parity-tested (Track-Q golden traces + Sobol-vs-scipy + pruner-vs-reference). Objective 2 is essentially met.

  • ✅ Conditional/nested search space (CONDITION_IN/NOT_IN, active-aware cascade — E2).

  • ✅ Python binding (n4m.model_selection.optimizer).

  • 🟡 Delivery gaps (Codex): the optimizer is not yet in the shipping PyPI package (only the dev tree); the host-side space compiler follows map insertion order (n4m_engine.py:194) rather than an explicit ordered schema; save/load returns NOT_IMPLEMENTED; some options are advertised but inert. → backlog M1/M2/M4.

  • ❌ R / MATLAB-Octave / WASM bindings of the optimizer (thin marshallers; not yet).

3b. dag-ml — the coordinator / DAG

  • ✅ Host operator vtable RuntimeController::invoke(NodeTask) -> NodeResult (host fits/transforms/predicts operators; phases FitCv/Refit/Predict/Explain). A JSONL process adapter + a one-shot Python op_callback transport the same contract. This is the substrate a driver sits on.

  • ✅ Execution bundles + replay/predict; candidate SELECT + scoring C-ABI; nested (inner) CV per node (inner_cv: NestedCvSpec).

  • ✅ Per-node flat param override GenerationParamOverride{node_id, params}.

  • 🟡 NodeKind::Tuner DSL node — passthrough only: sampler/n_trials are inert; the only native “search” is Cartesian/Zip variant enumeration + CV-SELECT (grid, not adaptive). The tuner is compiled to an external host controller node.

  • ❌ No native ask/tell tuner (no sampler/trial loop) and no dagml_tune/fit/ predict/explain execute C-ABI symbol (68 symbols are control-plane: compile, plan, schedule, select, score, replay, contracts).

  • ❌ No link to nirs4all-methods (zero n4m_* references; roadmap “largely-blocked”).

  • ❌ No native SHAP/attribution — only an ExplanationBlock payload contract + an Explain phase gate; the reference CLI emits a hardcoded mock.

  • ❌ No general sub-DAG fit (NodeKind::Subgraph is a bare enum variant).

  • ❌ No nested object__attribute addressing at the graph level (overrides are flat).

3c. nirs4all (Python)

  • ✅ Optuna finetuning (mature) + a new N4MFinetuneManager (engine:"n4m") that drives the libn4m optimizer, incl. when/when_not conditional attributes and options operator choice (operators in the search space) for sklearn-Pipeline model steps. Matches Optuna’s optimum; 26 unit tests. But Python-only orchestration.

  • 🟡 The dag-ml backend (engine:"dag-ml", ADR-17) runs plain pipelines natively, but force-routes any pipeline with finetune_params/train_params to the legacy Python engine — so finetuning never touches dag-ml today.

  • ❌ No DagMlEstimator (fit/predict/set_params) facade; no pipeline-scale SHAP over it.

3d. Verdict against the objectives

#

Objective

Status

1

dag-ml exposes a finetuning interface

🟡 grid-only + external-controller; not an adaptive/drivable estimator yet

2

nirs4all-methods cross-language finetuning methods

✅ (optimizer done; non-Python bindings pending)

3

pipeline tunable by Optuna or n4m

🟡 Python only (sklearn-Pipeline model step); not cross-language, not the dag-ml pipeline

4

dag-ml pipeline as BaseEstimator + (sub)pipeline finetune + pipeline-scale SHAP

❌ not implemented

4. Target architecture (reification — no responsibility change)

                 ┌─────────────────────────────────────────────┐
   any binding   │  DagMlEstimator  (fit / predict /            │  ← NEW, thin, per-binding
   (py/R/wasm)   │                   set_params / get_params)   │     facade over existing
                 └───────┬───────────────────────────┬─────────┘     dag-ml execution
                         │ set_params(node__p=…)      │ predict(X)
                         ▼                            ▼
      ┌──────────── dag-ml (UNCHANGED responsibility) ────────────┐
      │ compile→plan→refit(fit_cv, OOF)→bundle ; predict on bundle │
      │ per-node override · host operator vtable · SELECT/score    │
      └────────────────────────────────────────────────────────────┘
                         ▲                            ▲
        ask config       │ tell(OOF score)            │ predict(X_perturbed)
                 ┌───────┴────────┐          ┌────────┴──────────┐
                 │ optimizer      │          │ SHAP / explainer  │  ← reuse; only needs predict
                 │ Optuna (py) OR │          │ (host-side, any   │
                 │ libn4m (any)   │          │  language)        │
                 └────────────────┘          └───────────────────┘

  • Finetuning at (sub)pipeline scale = propose config → set_params → fit(CV/OOF) → score → tell. The estimator is a black box; the optimizer drives it from outside. The HPO algorithm stays in n4m, the execution stays in dag-ml; only ~thin glue wires them.

  • Explainability at pipeline scale = a model-agnostic explainer wraps estimator.predict. No SHAP engine in dag-ml; works for Torch-in-pipeline.

Responsibilities are preserved:

  • dag-ml → still the coordinator/executor. The DagMlEstimator packages its existing execute; it is not a new responsibility (and can live in the bindings).

  • nirs4all-methods → still the numeric + optimizer kernels. Unchanged.

  • nirs4all (host controllers) → owns the estimator facade + finetune loop + SHAP wrapper. This is exactly the “Python-exclusive controllers” layer of the North Star.

5. Backlog (minimal-first; converged with Codex)

The one correction vs. the first draft: dag-ml does need a small execution- contract slice (D-items) — it cannot yet return reusable fitted state or real predictions from replay. This is an extraction/export job (the machinery exists — RuntimeController::invoke, NodeTask/NodeResult carrying preds+artifacts+lineage, the scheduler’s fit/refit, ExecutionBundle), not a new subsystem, and explicitly not a tuner or SHAP engine.

dag-ml — small, unavoidable (D)

ID

Item

Size

D1

Promote one shared training operation into core (compile→plan→FIT_CV→score/select→optional REFIT→capture artifacts/caches→bundle) returning a TrainingOutcome {effective plan, bundle, scores + terminal prediction blocks, lineage, prediction-cache payloads, artifact records}. The CLI (main.rs:~2951) and Python (in_process.rs:~520) each reproduce this privately today. Include a refit=false eval mode so HPO trials don’t needlessly refit.

M

D2

Expose training via the C ABI — stateless JSON-in/JSON-out over the existing vtables; no long-lived Rust estimator handle.

S

D3

Real replay output: dagml_replay_execute_json currently returns only counts (ReplayExecutionSummary, lib.rs:~4559); add a ReplayOutcome with terminal + aggregated predictions, explanations, lineage.

S

D4

Estimator output binding: {node_id, port, target/class, aggregation} (or v1: exactly one unambiguous terminal prediction sink).

S

D5

Canonical parameter addressing — structured {node_id, namespace∈{operator,fit,structural}, path:[…]} (NOT node__subparam as the wire format). Structural change = clone template → apply patches → recompile → invalidate fitted state.

S

nirs4all-methods — finish delivery (M)

ID

Item

Size

M1

Ship the optimizer in the actual PyPI package (today it’s only importable from the dev tree / a hacked venv).

S

M2

Ordered search-space schema: canonicalize ordered axes / categorical values / conditions with an explicit order — the current Python compiler follows mapping insertion order (n4m_engine.py:194), which is not a sound cross-language contract. Validate duplicate names, unknown condition parents, cycles. Implement-or-reject currently-inert options.

S

M3

≥1 non-Python optimizer wrapper (R or WASM) + cross-binding golden proposal traces driven by the same score tape (extends Track-Q).

M

M4 (defer)

save/load (currently NOT_IMPLEMENTED, c_api_optimization.cpp:480) — needed for resumable Studio jobs, not for a first release. Do not grow n4m_finetune_estimator into a host-callback runner.

S

nirs4all (host) — the estimator + one shared objective (N)

ID

Item

Size

N1

New DagMLPipelineEstimator (nirs4all/sklearn/dagml_pipeline.py) — sklearn-cloneable; stores template config pre-fit; get_params(deep=True) exposes explicit graph params; set_params invalidates plan_/bundle_/artifacts; fit→D1/D2, predict→D3; explicit scorer + predict_proba; accepts identities/groups/metadata/multi-source, not just ndarray X. Do NOT retrofit NIRSPipeline (predict-only; fit() raises, pipeline.py:226).

M

N2

Generic PipelineObjective used by both Optuna and n4m: ask → clone_unfitted → with_params → leakage-safe CV → tell → refit winner. Reuse N4MFinetuneManager’s space compiler, but separate optimizer-driving from terminal-model training (it currently calls private controller build/train/evaluate).

M

N3

Pipeline-scale SHAP over the fitted facade’s predict (model-agnostic Kernel/Permutation). Extending nirs4all.explain() to accept the estimator is nice UX, not required.

M

N4

Thin cross-language finetune glue in R / WASM / MATLAB — the tiny ask→with_params→cv→tell→refit loop wiring the libn4m optimizer to the local DagMLPipelineEstimator. Acceptable per §6 iff all bindings consume the same canonical SearchSpaceSpec, dag-ml eval op, score direction/aggregation, failure/pruning rules, and conformance tape.

M each

Deferred (explicitly NOT in the first feature)

  • Arbitrary embedded sub-DAG extraction (needs a small GraphSliceSpec compiling to a standalone GraphSpec; NodeKind::Subgraph is a stub). — see §6.

  • Structural search (operator replacement / branch activation / variable-length) — that’s template materialization + recompile, a separate feature from value tuning.

  • Parallel adaptive HPO, wall-clock timeouts, native study persistence.

Smallest credible milestone (Codex’s, matches this backlog):

  1. dag-ml: D1+D2+D3 (shared train op + C-ABI training + real replay output).

  2. nirs4all: N1 + N2 (trainable estimator + one CV objective for Optuna and n4m).

  3. nirs4all-methods: M1 + M3 (optimizer in shipping Python + one 2nd language + score-tape conformance).

  4. Demonstrate: complex branch/stacking fit→predict; sklearn clone/CV; sequential n4m HPO; Optuna HPO through the same objective; Kernel SHAP on raw features with a buried Torch model.

  5. Defer embedded slicing, structural search, parallel HPO, study persistence.

6. The one genuine decision — where the finetune loop lives

  • Option A (recommended, small): host-side thin glue (B5). The loop is wiring two already-cross-language C-ABI objects (optimizer + estimator). The HPO algorithm (n4m) and the execution (dag-ml) are shared; only the ~30-line loop is per-binding, and the search space is declared once (B2). This honours “no responsibility change + avoid big dag-ml work.”

  • Option B (bigger, later): the native dag-ml Tuner coordinator (B7). Removes even the per-language glue, but is the large dag-ml build the owner wants to avoid, and it nudges a responsibility (HPO search) into dag-ml that its current migration split deliberately keeps host-side.

Recommendation: ship Option A; keep Option B as an explicit, deferred escape hatch justified only by measured per-language duplication.

Codex reinforces Option A, strongly: “A native callback runner would create Python GIL, R unwinding, MATLAB lifetime and WASM async/reentrancy problems while moving host orchestration into the numerical-methods repository. Ask/tell is already the shared native helper.” The loop is tiny; the hard semantics (fold isolation, OOF, scoring, refit, artifact lifecycle) live behind the shared dag-ml evaluator contract, not in the loop. Per-language glue is acceptable iff all bindings consume the same canonical SearchSpaceSpec, dag-ml eval op, score direction/ aggregation rules, trial failure/pruning rules, and conformance score tape.

7. Codex independent review — verdict (folded in)

Verdict: adopt the estimator-facade approach. Converges with §2–§6. Do not build dagml_tune, a SHAP algorithm in dag-ml, a direct dag-ml→libn4m dependency, or a long-lived callback-owning Rust estimator. Compose in the bindings.

Codex’s per-goal scorecard:

Goal

Codex verdict

dag-ml exposes finetuning

Yes — as a parameterizable, CV-evaluable estimator interface, not an optimizer

n4m provides reproducible methods

Yes — the ask/tell ABI is the right shared state machine

Optuna or n4m tunes a pipeline

Yes — both drive the same estimator objective

Pipeline-scale SHAP

Yes — for model-agnostic SHAP over a deterministic, row-preserving predict

Arbitrary embedded sub-DAG estimator

Not yet — requires explicit graph slicing (GraphSliceSpec)

Structural pipeline search

Not solved by flat overrides — needs materialization/recompile

The one correction to this doc’s first draft (now applied in §4/§5): the “no dag-ml change at all” claim was too optimistic. dag-ml’s common ABI cannot yet (1) create reusable fitted pipeline state, or (2) return real predictions from replay (it returns counts). Both are small extraction/export jobs (D1–D3), because the Rust core already has RuntimeController::invoke, NodeTask/NodeResult (carrying preds/ explanations/artifacts/lineage), the scheduler’s fit/refit, and ExecutionBundle. It confirmed everything else is bindings/host work, and that the fitted state should be a serializable (template, effective ExecutionPlan, ExecutionBundle, output binding, host-owned artifacts) package — matching dag-ml’s existing ownership boundary.

Codex also flagged the exact place the current Python path falls short of goal 4: nirs4all/pipeline/explainer.py:113,199 captures and explains one inner model, not the complete pipeline — which pipeline-scale SHAP over estimator.predict fixes.

8. Risks & hard rules (sharpened by Codex)

Leakage / OOF — the load-bearing rule. Never fit(X,y) → score(X,y) → tell. Use fit(outer_train) → score(outer_validation), or a dag-ml CV evaluation that returns a leakage-safe OOF score. Final refit of the winner happens only after selection. The outer optimizer owns outer validation; any dag-ml internal CV / stacking / early-stopping must operate strictly inside outer-train. Preserve groups, repetitions, sample identities, source layout and metadata when subsetting. Trial caches + artifact namespaces must include (param fingerprint, data fingerprint, fold id, trial id, seed). Never reuse a fitted handle between candidates. A trial failure → tell_result(FAILED), not an artificial inf (unless that policy is explicitly standardized).

Parameter addressing. Separate graph/control params, operator constructor params, and fit-only kwargs. Topology/operator replacement = structural materialization + recompile, not a value patch. Never mutate an already-fitted plan or bundle. Do not parse raw node ids with __ — keep a reversible alias table. Expose only explicitly-declared graph/controller params; dag-ml cannot introspect hidden defaults inside opaque host objects. (A fixed graph with nested value params is cheap; searching over operator replacement / branch activation / variable length is the separate materialization feature — deferred.)

SHAP correctness & cost. Pipeline-scale Kernel/Permutation SHAP works only when predict is deterministic, row-preserving, target-free, fixed-output, side-effect-free. For a buried Torch model: eval mode, disable dropout & prediction-time augmentation, no-grad, fixed seeds / deterministic backend, and explain logits/probabilities, not hard labels. For spectra, independent-wavelength masking is off-manifold and brutally expensive — use grouped adjacent bands, a proper background distribution, batched predictions, and a bounded budget. Multi-output pipelines must explicitly select the target/class explained. Perturbed samples need fresh content-derived identities so prediction caches can’t return unperturbed rows.

Reproducibility — three distinct claims. (1) optimizer trajectory = same ordered space + seed + ask/tell event tape; (2) objective = additionally identical folds, metrics, data, pipeline behaviour; (3) artifact/result = additionally identical portable operators + numeric env. Start with sequential ask/tell, one coordinator thread, no wall-clock timeout — adaptive samplers diverge if tell-order or even one score changes; do not claim parallel HPO reproducible yet. Use an ordered vector schema with explicit categorical order — not language map iteration (today’s Python compiler follows insertion order, n4m_engine.py:194). A facade does not make Python artifacts portable: a Torch/joblib pipeline is pipeline-SHAP-able in Python; it crosses languages only when its operators use portable n4m artifacts (or another agreed portable format) — otherwise the contract is cross-language while the fitted host artifact stays language-owned, which is consistent with the existing responsibility split.

The pivot (D1–D3): the whole approach rests on dag-ml exposing reusable fitted state + real replay predictions. Confirmed small (extraction/export), not a subsystem.