Generic estimator roles (ABI 2.13) — design

Status: proposal revised after one adversarial Codex review (2026-09-26; findings folded into §2 and §7). Target: the next Methods release batch (1.0.22), which also carries the unreleased ABI 2.11/2.12 work.

1. Problem

Every n4m method must be usable from Python, R and JS/WASM through the same life cycle — create with parameters, fit, transform or predict on new data, export the fitted state, import it in another language — so that bindings and the nirs4all / DAG-ML controllers consume a few standard interfaces instead of one adapter per method. This is the sklearn BaseEstimator / TransformerMixin / RegressorMixin / ClassifierMixin contract, but owned by the C++ core and exposed through the C ABI.

Audit at ABI 2.12 (212 catalog entries):

Reachable through

Methods

Generic role with portable fitted state (N4MM model, N4MP pipeline, affine n4m_model_from_method_result)

28

Generic role without state or without serialization (2.11 splitters, 2.11 augmentations, 2.12 filters)

38

Per-method C functions only

the rest

Consequences today:

  • There is no native method manifest. Parameter names, defaults, required inputs and state formats are re-encoded by hand in each binding (R vectors and strcmp dispatch, Python _C_PREFIX tables and ~165 classes, JS strcmp chain). Defaults already drifted between bindings.

  • Numerics leaked into bindings: Python computes classifier decisions, DS/PDS application and several numpy Ridge/PLS fits; R has an affine fallback.

  • Many estimators are train-only (no out-of-sample predict), transformer state lives only in native handles (not picklable, not saveRDS-able), selectors have no apply step, classifiers have no native predict_proba.

  • N4MP re-implements MSC/EMSC/ASLS/OSC/EPO/detrend/Norris-Williams/wavelet instead of calling the per-method kernels; only SNV and Savitzky-Golay are checked equal.

2. Decisions

D0 — Every catalog entry is an estimator or a procedure

Each catalog entry declares kind: estimator or kind: procedure.

  • Estimators have a fitted state and a life cycle (D1–D6): transformers, regressors, classifiers, selectors, sample filters.

  • Procedures are one-shot computations with no reusable state: splitters, train-only augmentations, diagnostics, sweeps and HPO campaigns, metrics. They are called through one generic entry point, n4m_procedure_run(ctx, method_index, params, inputs, out_result), which validates named parameters and inputs against the manifest and returns a n4m_method_result_t whose layout is fixed by the procedure role (D5).

“Every method usable in Python, R and WASM” therefore means: every estimator through the estimator life cycle and every procedure through the procedure call, both driven by the same manifest. No entry is exempt.

D0b — Typed role interfaces, not one catch-all interface

The surface is a set of typed interfaces, in the scikit-learn / torch spirit: each role has its own operations and an input/output signature, and that signature is what types the nirs4all / DAG-ML controllers and therefore the Studio and Web nodes. A method implements one or more roles (PLS is both a regressor and a transformer); it never exposes operations of a role it does not declare.

Role

Analogy

Operations

Signature (ports)

DAG-ML node

transformer

TransformerMixin / nn.Module X→X

fit, transform, export

Data[n,p] (+Target if supervised) → Data[n,k]

transform

selector

SelectorMixin

fit, selected_indices, transform (column subset), export

Data[n,p] (+Target) → Data[n,k⊆p]

transform (shape changing)

regressor

RegressorMixin

fit, predict, export

Data[n,p] + Target[n,q] → Prediction[n,q]

model

classifier

ClassifierMixin

fit, predict_labels, decision_function, predict_proba (when defined), classes, export

Data + Labels[n] → Prediction labels[n] / scores[n,c]

model

sample filter

outlier detector (fit_predict mask)

fit, apply_mask, export

Data (+Target) → keep mask[n], train only

exclude

splitter (procedure)

BaseCrossValidator

split

Data (+Target, groups) → folds

split

augmenter (procedure)

train-only sampler

augment

Data (+Target) → Data’ (+Target’ for mixing methods), train only

augmentation

generic procedure

function

run

inputs → MethodResult

none (diagnostics, sweeps)

In C the stateful roles share one opaque n4m_estimator_t, because ownership, parameters and N4ME serialization are identical; the role is part of the method identity (roles mask), each operation checks it, and the manifest publishes the role, its signature and the DAG-ML node kind. Typing lives in the contract, not in the handle: bindings expose one base class per role (NativeTransformer, NativeSelector, NativeRegressor, NativeClassifier, NativeSampleFilter, NativeSplitter, NativeAugmenter), multi-role methods inherit several, and controller manifests are derived from the same role → node-kind mapping.

D1 — One estimator handle over a closed, compiled adapter table

Add one opaque handle, n4m_estimator_t, created from a method_id. Internally each supported method registers a C++ adapter implementing a small interface:

namespace n4m::estimator {
struct FitInputs;            // see D4
class Writer; class Reader;  // N4ME state blocks, see D6

class Adapter {
 public:
  virtual ~Adapter() = default;
  virtual n4m_status_t fit(n4m_context_t*, const FitInputs&) = 0;
  virtual n4m_status_t transform(n4m_context_t*, const View& X, MutView out) const;   // TRANSFORMER, SELECTOR
  virtual n4m_status_t predict(n4m_context_t*, const View& X, MutView out) const;     // REGRESSOR, CLASSIFIER scores
  virtual n4m_status_t predict_proba(n4m_context_t*, const View& X, MutView out) const;
  virtual n4m_status_t predict_labels(n4m_context_t*, const View& X, int64_t* out) const;
  virtual n4m_status_t selected_indices(Span<int64_t> out, int64_t* count) const;     // SELECTOR
  virtual n4m_status_t apply_mask(n4m_context_t*, const View& X, const View* Y, uint8_t* mask) const; // SAMPLE_FILTER
  virtual n4m_status_t output_shape(int32_t op, int64_t rows, int64_t* r, int64_t* c) const = 0;
  virtual n4m_status_t save_state(Writer&) const = 0;
  virtual n4m_status_t load_state(Reader&) = 0;
  virtual const n4m_method_result_t* fit_result() const;  // diagnostics, borrowed
};
}

Unimplemented operations return N4M_ERR_UNSUPPORTED. Adapters call the existing kernels (the *_state_* C engines, n4m_estimators_*_fit, n4m_model_fit); they never re-implement them.

Rejected alternatives:

  • Extend n4m_model_t (N4MM) to every method. N4MM is a PLS-shaped record (algorithm/solver/deflation, latent arrays). Forcing transformers, selectors, filters and splitters into it breaks its meaning and its inspection contract.

  • Extend n4m_pipeline_t (N4MP) to every method. N4MP is a transformer sequence with positional double parameters; it cannot express predictors, selectors, integer/array parameters or extra fit inputs.

  • Open plugin registration. Not needed: every method is compiled in, and a closed table keeps the ABI and the serialized formats verifiable.

N4MM and N4MP stay. They become state encodings embedded by the adapters that need them (D6), and n4m_model_* / n4m_pipeline_* keep working.

D2 — The catalog is the manifest; the library exposes it

The catalog entry (authored in catalog/methods.yaml, split into catalog/methods/<id>.yaml, schema catalog/schema/method.json) gains an estimator block. It is the authoring source; the compiled table is the only runtime authority: bindings generate their classes from a JSON dump produced by the built library (n4m_cli --manifest-json), never from the YAML directly.

estimator:
  kind: estimator                    # estimator|procedure (D0)
  roles: [transformer]               # estimators: transformer|regressor|classifier|selector|sample_filter
                                     # procedures: splitter|augmenter|generic
  params:
    - {name: n_components, type: int, default: 1, min: 1}
    - {name: scale, type: bool, default: true}
  inputs: {y: required}              # y: none|optional|required; groups, feature_groups,
                                     # sample_weight, blocks, axis, target_domain: none|optional|required
  state:                             # every field needed to transform/predict new rows
    format: n4me.osc.v1
    fields: [x_mean: f64[p], weights: f64[p,k], loadings: f64[p,k]]
    retains_training_rows: false

Capabilities are not declared by hand: they are derived from the adapter’s implemented operations and, for imported states, from the validated N4ME bytes (the N4MM inspection rule).

A generator (catalog/scripts/generate_estimator_manifest.py) emits a compiled table (cpp/src/estimator/generated/manifest.inc) with, for each method, its id, roles, parameter descriptors, input requirements, flags and the adapter factory. CI fails if the generated file is out of date, and validate_catalog.py checks that every estimator block names an adapter that exists.

Introspection ABI (library-owned static strings, never freed):

typedef struct n4m_method_info_v1_t {
    uint32_t struct_size;
    const char* method_id;      /* "preprocessing.orthogonalization.osc" */
    const char* fq_name;        /* "n4m.transform.orthogonalization.osc" */
    uint32_t roles;             /* N4M_ROLE_* mask */
    uint64_t capabilities;      /* N4M_CAP_* mask, see D5 */
    int32_t n_params;
    const char* state_format;   /* "n4me.osc.v1" */
} n4m_method_info_v1_t;

typedef struct n4m_param_info_v1_t {
    uint32_t struct_size;
    const char* name;
    int32_t type;               /* N4M_METHOD_PARAM_INT|DOUBLE|BOOL|ENUM|INT_ARRAY|DOUBLE_ARRAY */
    int32_t has_default;        /* 0 = required parameter */
    int64_t default_length;     /* 1 for scalars, n for array defaults */
    double min_value, max_value;/* NaN when unbounded */
    int32_t n_choices;          /* ENUM */
    const char* const* choices;
} n4m_param_info_v1_t;

N4M_API n4m_status_t n4m_method_count(int32_t* out);
N4M_API n4m_status_t n4m_method_find(const char* method_id, int32_t* out_index);
N4M_API n4m_status_t n4m_method_info_v1(int32_t index, n4m_method_info_v1_t* out);
N4M_API n4m_status_t n4m_method_param_info_v1(int32_t index, int32_t param, n4m_param_info_v1_t* out);
N4M_API n4m_status_t n4m_method_param_default_double(int32_t index, int32_t param, double* out, int64_t cap, int64_t* count);
N4M_API n4m_status_t n4m_method_param_default_int(int32_t index, int32_t param, int64_t* out, int64_t cap, int64_t* count);

Descriptor rules follow the existing inspection contract: the caller sets struct_size to the size it was compiled with; the library writes min(struct_size, sizeof(current)) bytes, zeroes that prefix on every error, and rejects a struct_size smaller than the v1 layout with N4M_ERR_INVALID_ARGUMENT.

Bindings generate their classes, R constructors and JS factories from this introspection (at build time, from the same generated JSON exported by the generator). They no longer carry method lists.

D3 — Named, typed parameters; defaults owned by the core

N4M_API n4m_status_t n4m_params_create(n4m_context_t*, int32_t method_index, n4m_params_t** out);
N4M_API n4m_status_t n4m_params_set_int(n4m_params_t*, const char* name, int64_t v);
N4M_API n4m_status_t n4m_params_set_double(n4m_params_t*, const char* name, double v);
N4M_API n4m_status_t n4m_params_set_bool(n4m_params_t*, const char* name, int v);
N4M_API n4m_status_t n4m_params_set_enum(n4m_params_t*, const char* name, const char* choice);
N4M_API n4m_status_t n4m_params_set_int_array(n4m_params_t*, const char* name, const int64_t* v, int64_t n);
N4M_API n4m_status_t n4m_params_set_double_array(n4m_params_t*, const char* name, const double* v, int64_t n);
N4M_API n4m_status_t n4m_params_get_*(...);   /* resolved value, default when unset */
N4M_API void n4m_params_destroy(n4m_params_t*);

Setters take no context; they return a precise status (N4M_ERR_INVALID_ARGUMENT for unknown name, type or bound) and the named diagnostic is reported by n4m_params_validate(ctx, params), which n4m_estimator_create and n4m_procedure_run also call, through the context error buffer. Unknown names, wrong types and out-of-bounds values therefore fail at set, with the message available at validation. Unset parameters take the manifest default, so a Level 1 recipe {method_id, params} yields the same estimator in every language.

D4 — One versioned fit-input struct

typedef struct n4m_fit_inputs_v1_t {
    uint32_t struct_size;
    const n4m_matrix_view_t* X;             /* required */
    const n4m_matrix_view_t* Y;             /* continuous targets */
    const int64_t* labels; int64_t n_labels; /* class IDs for classifiers */
    const double* sample_weight; int64_t n_sample_weight;
    const int64_t* groups; int64_t n_groups;              /* sample groups */
    const int32_t* feature_groups; int64_t n_feature_groups;
    const int32_t* block_sizes; int32_t n_blocks;        /* multiblock column partition */
    const double* axis; int64_t n_axis;                   /* wavelengths */
    const n4m_matrix_view_t* X_target;      /* target-domain / slave spectra */
    const int64_t* fold_ids; int64_t n_fold_ids; /* internal-CV folds (selectors, AOM) */
} n4m_fit_inputs_v1_t;

Seeds are hyperparameters (seed, noise_seed, …), as random_state is in scikit-learn and as the n4m references take them: they belong to the parameters, so a recipe {method_id, params} fixes every random draw. Since ABI 2.14 they are optional: the manifest publishes no default (null, has_default = 1 with default_length = 0), the facades default them to None / NULL / undefined, and “unset” means “not set by the caller”, so a host runtime can tell an explicit operator seed from one it may derive. An unset seed runs as seed 0 (the former default, results unchanged), and an N4ME state records the seed it ran with. Internal-CV methods take a cv parameter; when fold_ids are absent the core builds the canonical contiguous plan (fold size n/k, last fold absorbs the remainder).

Classifiers receive integer class IDs; the state stores the sorted unique IDs (n4m_estimator_classes) and hosts map IDs to names in their envelope. Kernels that take int32_t labels and a class count get them from the adapter (remapped to 0..k-1). Wavelength-dependent methods receive axis here and build their kernel at fit, instead of at construction.

fit checks the manifest input requirements and returns N4M_ERR_INVALID_ARGUMENT with a context message naming the missing input. Lengths are exact (ABI 2.14 made the rule explicit in every facade): Y has X->rows rows and at least one column, labels, sample weights, groups and fold ids have X->rows entries, feature groups and the axis X->cols; a mismatch is N4M_ERR_SHAPE_MISMATCH naming the input. Nothing is broadcast or reshaped, natively or in a facade: R never recycles a vector to the rows of X, Python and JS never infer the rows of a target from its element count (a (2, 20) target for 40 rows is refused, not reinterpreted). The facades check before building the views so the error names their own argument; the native check is the guard every binding shares. Views may be strided; adapters whose kernel needs contiguous row-major data get a private copy from one shared helper. This also lifts the contiguous-only restriction of the 2.11/2.12 roles. Every matrix a caller passes (X, Y, the target domain, and the new rows of every operation) is validated once, before dispatch, by the same check: a valid layout and float64 values. Another dtype is N4M_ERR_DTYPE_MISMATCH, never read as doubles; the procedures sharing the adapters get the same check.

D5 — Estimator life cycle and capabilities

typedef enum { N4M_ROLE_TRANSFORMER = 1, N4M_ROLE_REGRESSOR = 2, N4M_ROLE_CLASSIFIER = 4,
               N4M_ROLE_SELECTOR = 8, N4M_ROLE_SAMPLE_FILTER = 16 } n4m_role_t;
/* procedures (kind N4M_METHOD_PROCEDURE): N4M_ROLE_SPLITTER = 32, N4M_ROLE_AUGMENTER = 64,
   N4M_ROLE_GENERIC = 128, exactly one */

/* capabilities */
#define N4M_CAP_OOS_TRANSFORM       (1ull << 0)  /* transform new rows */
#define N4M_CAP_OOS_PREDICT         (1ull << 1)
#define N4M_CAP_PREDICT_PROBA       (1ull << 2)
#define N4M_CAP_SERIALIZABLE        (1ull << 3)
#define N4M_CAP_SUPERVISED          (1ull << 4)
#define N4M_CAP_SEEDED              (1ull << 5)
#define N4M_CAP_TRAIN_ONLY          (1ull << 6)  /* augmenters: never applied at predict */
#define N4M_CAP_RETAINS_TRAINING_ROWS (1ull << 7)/* state embeds training data */
#define N4M_CAP_AFFINE              (1ull << 8)  /* predict == X @ B + b */
#define N4M_CAP_DECISION_FUNCTION   (1ull << 9)

N4M_API n4m_status_t n4m_estimator_create(n4m_context_t*, const char* method_id,
                                          const n4m_params_t* params /* NULL = defaults */,
                                          n4m_estimator_t** out);
N4M_API n4m_status_t n4m_estimator_fit(n4m_context_t*, n4m_estimator_t*, const n4m_fit_inputs_v1_t*);
N4M_API n4m_status_t n4m_estimator_is_fitted(const n4m_estimator_t*, int* out);
N4M_API n4m_status_t n4m_estimator_output_shape(const n4m_estimator_t*, int32_t op,
                                                int64_t n_rows, int64_t* rows, int64_t* cols);
N4M_API n4m_status_t n4m_estimator_transform(n4m_context_t*, const n4m_estimator_t*,
                                             const n4m_matrix_view_t* X, n4m_matrix_view_t* out);
N4M_API n4m_status_t n4m_estimator_predict(...same shape as transform...);
N4M_API n4m_status_t n4m_estimator_decision_function(...); /* method-defined scores, e.g. QDA log-likelihoods */
N4M_API n4m_status_t n4m_estimator_predict_proba(...);     /* only when the method defines probabilities */
N4M_API n4m_status_t n4m_estimator_predict_labels(n4m_context_t*, const n4m_estimator_t*,
                                                  const n4m_matrix_view_t* X, int64_t* out, int64_t n);
N4M_API n4m_status_t n4m_estimator_classes(const n4m_estimator_t*, int64_t* out, int64_t cap, int64_t* count);
N4M_API n4m_status_t n4m_estimator_selected_indices(const n4m_estimator_t*, int64_t* out, int64_t cap, int64_t* count);
N4M_API n4m_status_t n4m_estimator_apply_mask(n4m_context_t*, const n4m_estimator_t*,
                                              const n4m_matrix_view_t* X, const n4m_matrix_view_t* Y,
                                              uint8_t* mask, n4m_filter_stats_t* stats);
N4M_API n4m_status_t n4m_estimator_fit_result(const n4m_estimator_t*, const n4m_method_result_t** borrowed);
N4M_API n4m_status_t n4m_estimator_info(const n4m_estimator_t*, int32_t* method_index, uint64_t* capabilities);
N4M_API n4m_status_t n4m_estimator_get_params(const n4m_estimator_t*, n4m_params_t** out_copy);
N4M_API void n4m_estimator_destroy(n4m_estimator_t*);

Classifier labels, decision scores and probabilities are computed natively; each classifier documents its score semantics in the manifest, and PREDICT_PROBA is set only where probabilities are defined.

fit_result is borrowed: valid until the next fit, load_state or destroy of the same estimator. Since ABI 2.14 a (re)fit fits a fresh state and replaces the previous one only on success: a failed refit leaves the estimator fitted with its previous state. Facades follow the same rule for the host metadata they keep beside the handle (class names, target shape, feature names): published with the new state, never before the native fit succeeded. All _v1 input structs carry struct_size set by the caller; unknown trailing fields are rejected, shorter known layouts accepted.

Procedures run through one call, driven by method index and named parameters (no handle, no no-op fit); the role fixes the result layout:

/* roles: N4M_ROLE_SPLITTER | N4M_ROLE_AUGMENTER | N4M_ROLE_GENERIC, exactly one */
N4M_API n4m_status_t n4m_procedure_run(n4m_context_t*, int32_t method_index, const n4m_params_t*,
                                       const n4m_fit_inputs_v1_t*, n4m_method_result_t** out);
/* splitters: every fold, zero-based rows of X, borrowed from the result */
N4M_API n4m_status_t n4m_method_result_get_n_folds(const n4m_method_result_t*, int32_t* out);
N4M_API n4m_status_t n4m_method_result_get_fold(const n4m_method_result_t*, int32_t fold,
                                                const int64_t** train, int64_t* n_train,
                                                const int64_t** test, int64_t* n_test);

  • Splitter: all folds in one call (a DAG-ML split node needs them all); n_folds is n_splits for k-fold kinds, 1 otherwise. Y-only splitters split the rows of X without reading it.

  • Augmenter: double matrix "X", same shape and row order as the input, train-only. Augmenters that mix rows (mixup, local mixup) require y and add double matrix "Y": the targets mixed with the same draw (the same partner row and weight as row i of "X"), with the shape of the input y. No other augmenter returns "Y"; the others refuse y.

  • Generic: the named outputs of the method’s C function; functions that return plain values (metrics, T², transfer metrics) are packed under their C names.

Rejected: the separate n4m_split_run (one fold per call) and n4m_augment_run (caller-allocated output) proposed earlier. They double the entry points for no capability: the result is read once, and a per-fold call recomputes the whole partition for every fold.

Inputs reuse n4m_fit_inputs_v1_t and its checks (a missing required or an unused input is refused by name). Diagnostics that need a second matrix take it as target_domain (PLS monitoring: the phase-2 rows; transfer metrics: the target set); diagnostics defined on a fitted PLS model fit it from X, y and the declared PLS parameters. Axis-dependent augmenters whose kernel works in the axis’ own units take axis as given (optional where the kernel falls back to the index grid). Kernels with nanometre constants (band positions of the temperature and moisture effects, the 1500 nm reference of particle-size scattering, the detector ranges of roll-off and the combined edge artifacts) require axis in nanometres, as their n4m references assume; the procedure refuses an axis that is not finite and strictly increasing (the kernels interpolate on it). Stray light depends on the channel count only and takes no axis.

The mixup kernels draw every row’s partner and weight once, then apply them to X and, when given, to y (an internal kernel change; the per-method n4m_augmentation_mixup_* / _local_mixup_* functions still return X only). Local mixup searches neighbours in X alone.

D6 — N4ME fitted-state format

"N4ME" | format u32 = 1 | writer ABI (3 × u32)
method_id (u32 length + UTF-8) | params (canonical, sorted by name, typed)
capabilities u64 | n_features_in u64 | n_outputs u64
state block: u32 tag + u64 length + adapter bytes   (repeated)
FNV-1a-64 over all preceding bytes

  • Adapters that already have a format embed it unchanged as a state block: affine and PLS regressors embed N4MM, N4MP-kind transformers embed their N4MP single-step payload. Every other adapter defines its own block, versioned by state_format in the manifest.

  • Import validates magic, versions, checksum, the method id against the table, the parameters against the manifest, every state field’s declared shape, and bounds, then calls load_state. Section lengths are u64; the default payload cap is 256 MiB, raised by the caller through n4m_context_set_max_state_bytes for large retained-row models (a GPR-PLS Cholesky factor alone is n² doubles: 64 MiB ≈ 2,900 training rows). The ABI mirrors N4MM/N4MP: n4m_estimator_export_size, _export_to_buffer, _import_from_buffer, and an allocation-free n4m_serialization_inspect_estimator_v1.

  • States that embed training rows (RETAINS_TRAINING_ROWS: kernel PLS, LW-PLS, GPR-PLS; LW-PLS also needs a new out-of-sample predict kernel, since today it only predicts its training rows) export only when the caller passes N4M_EXPORT_ALLOW_TRAINING_ROWS; otherwise export fails with a message. This keeps data retention an explicit decision. Policy (ABI 2.14, identical on every surface): n4m_estimator_contains_training_rows answers for a fitted or imported state (the manifest capability retains_training_rows answers per method); every facade export takes an explicit opt-in that defaults to off (Python to_n4me(allow_training_rows=True), R n4m_estimator_export(object, allow_training_rows = TRUE), JS toN4me({ allowTrainingRows: true }), Rust to_n4me(&ctx, true)). An N4ME export is the shareable artefact; in-process persistence of the live object (Python pickle, R saveRDS) is a local checkpoint and keeps the rows as the object does. Hosts writing shareable envelopes pass the opt-in only on an explicit user decision.

  • Parameters against state (ABI 2.14). The payload is protected against corruption by its checksum, not against edits (the checksum can be recomputed): import therefore also checks that the parameters describe the restored state. Each parameter of a method is

    • recorded: the state holds its value or a dimension it fixes (PLS/PCR n_components, solver and centring/scaling flags in the embedded N4MM, recursive PLS window_size, N-PLS mode_j x mode_k, kernel PLS effective gamma, PLS-DA heads n_components, top_k of the selectors that keep exactly top_k columns, the DS/SAPS fit_intercept layout, GLM family, OnPLS component counts, GPR-PLS n_components). Import refuses a different value with N4M_ERR_CORRUPT_BUFFER naming the parameter; the manifest flags these parameters (recorded, n4m_param_info_v1_t), and the C++ conformance suite mutates each of them in a real payload, recomputes the checksum and expects the refusal;

    • bounded: a kernel legitimately reduces the requested value to what the data allow, so the request is only an upper (or lower) bound on the effective value the state holds, and import checks the bound: PLS-QDA and OSC components (min(requested, rank)), flexible / wavelet PCA and SVD (at most an integer request), leverage PCA components (at most the request, else 50), selector sizes (min_features, min_selected, max_features, windows and intervals), the AOM calibration branch (one of branches); conditional rules check a recorded effect when the parameter has one (no centring leaves zero means, a given reference is stored as is, a crop keeps the axis inside its bounds, PDS stays banded);

    • replayed: the adapter rebuilds its behaviour from the parameter at load (transform kernels, kernel type, LW-PLS neighbours, filter thresholds), so the state cannot contradict it;

    • fit-only: it shaped the fit and left no trace (penalties, tolerances, iteration counts, seeds, the hyperparameters of affine predictors whose state is coefficients only); it is provenance and cannot be checked. Every adapter family implements Adapter::check_params; kernel state loaders check the rules that need the kernel’s own configuration.

N4ME is the Level 2 unit of portability: the same bytes predict identically in Python, R and WASM.

D7 — Existing symbols

  • ABI policy: minor releases are additive. Per-method C symbols (n4m_transform_*, n4m_estimators_*_fit, …) stay as the kernel-level API. Bindings stop using them for the generic path.

  • The ABI 2.11 n4m_splitter_run / n4m_augmentation_run (on main, in the R n4m 1.0.21.9006 source) stay; n4m_procedure_run calls them.

  • The ABI 2.12 filter-role symbols (n4m_sample_filter_*, n4m_feature_filter_*) were never distributed (verified 2026-09-26: they existed only on the unmerged branch of PR #37 and a local R library). They are replaced by the SAMPLE_FILTER / SELECTOR estimator roles and removed (header, symbol snapshots, binding facades); the change log records 2.12 as a superseded, unreleased draft. The composite filter is not carried over: composing filters is DAG-ML’s job (D8).

  • N4MP operator kinds are re-pointed at the per-method kernels, with an equivalence test per kind, so one implementation remains per method.

D8 — What stays out of n4m

Composition (pipelines, branches, stacking, CV/OOF, selection) belongs to DAG-ML. n4m provides estimators; DAG-ML controllers compose them. Model families that need a host framework (torch, sklearn RF, ranger) are not n4m estimators.

D9 — Role pipeline: the trained linear recipe is native (ABI 2.14)

The portable trained pipeline (a recipe of n4m:<method id> steps plus one N4ME state per fitted step, the nirs4all trained_pipeline.v8 envelope) was implemented five times — nirs4all Python, Core Python, Core WASM, Core Rust and the R product — and the copies diverged: contradictory recipe/state pairs and empty recipes accepted, multi-target Y dropped before supervised steps in WASM, column identity lost, training-row export policy different per surface (integration audit 2026-09-27, F03/F05/F06/F10). ABI 2.14 moves the semantics into one owner, n4m_role_pipeline_t; the envelope JSON stays a thin host wrapper (schema string, recipe tokens, base64, checksums, class names).

This is not composition in the D8 sense: a role pipeline is one linear, already-decided chain (no branches, no CV/OOF, no selection, no stacking). DAG-ML keeps composing; a fitted linear chain it wants to ship is exchanged as a role pipeline.

Recipe. n4m_role_pipeline_create(ctx, n, method_ids, params, &out): zero or more sample filters, zero or more transformers / selectors, exactly one regressor or classifier last. A multi-role method plays the role of its position (PLS: transformer inside, regressor last). Refused with the step named: empty recipe, unknown method, procedure, params of another method, missing required parameter, filter after a transformer, non-terminal regressor, missing terminal.

Fit routing (n4m_role_pipeline_fit, one n4m_fit_inputs_v1_t):

Input

Reaches

Y (n×q)

every step whose manifest requires y, all q columns (F06)

labels

steps declaring labels (the classifier); without Y, the non-terminal steps that require y get the class ids as one double column (what the host implementations did)

sample_weight, groups, fold_ids

steps declaring them

feature_groups, blocks, axis, X_target

steps declaring them, only while no transformer or selector has changed the columns (they describe the pipeline input columns); a later step that requires one is refused

Sample filters are fitted, applied to the training rows, and subset every row-aligned input (X, Y, labels, weights, groups, fold ids); they keep no state. An input no step uses is refused by name, as n4m_estimator_fit does. Optional y is not forwarded (unchanged from the host code). Every error is prefixed step <i> (<method id>). On failure the pipeline is unfitted.

Feature identity (F03). n4m_role_pipeline_set_feature_names (before fit or import; UTF-8, unique) stores the input column names; n4m_role_pipeline_check_features(ctx, p, n_columns, names) refuses a width mismatch and, when names are stored and given, any renamed or reordered column (the message says which). Arrays without names are positional. Facades call it before every operation with the caller’s column names.

Portable state (F05, F10). n4m_role_pipeline_export_state_* exports one N4ME per stateful step (filters excluded); a state embedding training rows needs N4M_EXPORT_ALLOW_TRAINING_ROWS, and n4m_role_pipeline_step_info_v1.contains_training_rows says which do. n4m_role_pipeline_import_states rebuilds a fitted pipeline from the recipe and the states and refuses: a state count different from the stateful steps, a state of another method, a parameter whose resolved value differs from the recipe’s (typed comparison after default resolution; for example CPPLS 2 vs 1 components), a state lacking the operation its role needs, and widths that do not chain.

Operations. transform (transformers/selectors only), predict, predict_labels, decision_function, predict_proba and classes through the terminal step, with the estimator refusals; step introspection (n4m_role_pipeline_step_info_v1: method, role played, state index, fitted widths, training rows). Zero new rows give an empty output of the right width whatever the steps: the width and output shape are checked, then no step runs.

Class labels and column names (facades). The core works on int64 class IDs; the label table and the column names are facade data, and every facade applies the same contract before any native call:

  • Without a label table the labels are the class IDs; integer labels must fit int64 (JS: safe integers), never wrapped (a uint64 2^63 is refused).

  • At fit, strings and non-integer numbers become the table of sorted unique labels, the IDs their positions. Missing (None / NA / null) and non-finite (NaN, ±Inf) labels are refused, and so are booleans.

  • An imported table (class_names, label_names, classNames) must be a non-empty list of unique strings or finite numbers, not both, and hold an entry for every class ID of the fitted state (n4m_role_pipeline_classes): 0 <= id < len(table). It may be longer than the set of IDs (labels whose rows a train-only filter removed keep their slot); it is refused on a pipeline whose final step is not a classifier.

  • Column names containing NUL are refused at fit, import and predict, before a C string would truncate them (R strings cannot hold NUL; Rust CString refuses it).

Facades: Python n4m.roles.RolePipeline, R n4m_role_pipeline(), JS RolePipeline, Rust n4m::roles::RolePipeline (it takes i64 class IDs and holds no label table). The shared fixture parity/fixtures/role_pipeline_negative.json holds the native negative cases (cases), the facade-level label_cases and name_cases of the contract above (matched by message, no native status), and one pipeline fitted in Python, replayed by the four suites.

3. Bindings and controllers

  • Python: one NativeEstimator base in n4m (BaseEstimator plus the mixin selected by role), with get_params/set_params from the manifest, fit/transform/predict/predict_proba, and __getstate__/__setstate__ through N4ME. Named classes (n4m.PLS, n4m.OSC, …) are generated thin subclasses; existing hand-written classes are replaced when their method is covered, and the numpy fallbacks listed in §1 are deleted.

  • R: one S3 class n4m_estimator (n4m_estimator("…", …), fit, predict, transform, n4m_export, n4m_import); N4ME bytes kept in the object so saveRDS/readRDS rehydrate without refitting. Named constructors generated from the manifest. The strcmp dispatch and the hard-coded kind vectors are deleted.

  • JS/WASM: one NativeEstimator class; EXPORTED_FUNCTIONS already come from the ABI snapshot. The fitModel name chain and coefficient-based prediction are deleted in favour of N4ME.

  • Controllers: DAG-ML controller manifests for n4m nodes are generated from the n4m manifest (roles → node kinds, params → schema), so the nirs4all Python, nirs4all R and Core/WASM controllers accept every method with no per-method code. Level 1 recipes are {method_id, params}; Level 2 artifacts carry N4ME bytes. The trained envelopes v1–v7 converge on one N4ME-based envelope.

4. Coverage plan

Slices, each merged to main without release, in the order of what they unlock:

Slice

Content

Methods

S0

Core: manifest generator + introspection, params, estimator handle, procedure call, N4ME, conformance harness; adapters for the 16 affine regressors and the N4MM PLS families

~24

S1

Promote weighted_pls, o2pls; AOM/sweep results through input_coefficients; 26 selectors (apply = column subset)

~37

S2

58 transformers (34 stateless, 24 fitted, each with a full state spec: several kernels expose only opaque state today and need accessors); N4MP re-pointed at the kernels

58

S3

Classifiers with native labels/decision/proba where defined; sample filters; splitters and 22 augmenters through the procedure dispatch

~45

S4

Non-affine regressors (GPR-PLS, PLS-GLM, kernel PLS; LW-PLS with a new predict kernel), transfer (DS, PDS, robust DS, slope/bias, DI-PLS), augmenters with paired Y (kernel change) or axis

~25

S5

Remaining procedures: diagnostics, sweeps, AOM/HPO campaigns, metrics through n4m_procedure_run

rest

The final claim is per entry: estimator or procedure, languages, operations, oracle result — never a bare 212/212.

Status (2026-09-27): 208 of 213 catalog entries reach the generic surface — 148 estimators (regressors incl. 16 AOM/POP ones, PLS-GLM and PLS-Cox, 4 classifiers, selectors, transformers, 4 sample filters) and 60 procedures (9 splitters, 39 augmenters, 12 generic); the generated per-entry record is docs/parity/estimator_roles_coverage.md. Python, R, JS/WASM and Rust facades are generated from or driven by the manifest. The shared fixture parity/fixtures/estimator_roles_n4me.json is replayed by the four bindings: estimator states at 1e-12 with byte-identical re-export, each binding’s own fit reproducing the Python fit, and every procedure’s default run. Each role class also matches its n4m reference (bitwise where the same kernel runs). PLS-Cox takes its survival response in the existing inputs: Y has two columns, time then event indicator (1 = event, 0 = censored), and predict returns one column, the risk score. Not covered: three AOM orchestrations (screen/refit, staged campaign, linear Ridge stack: DAG-ML composition), the Python-only moment stack and the composite filter (a composition of filters).

5. Tests

  • Conformance suite (C++), generated from the manifest: for every method, fit on a train split; transform/predict/proba on held-out rows; export → import → identical outputs (bitwise); parameter round trip; refusal of each missing required input; refusal of unsupported operations; seeded determinism. Procedures: run with the default parameters, same seed → same result, column-major X → same result, refusal of missing, unused and foreign inputs or parameters, and bitwise equality with the direct C entry point. Target-mixing augmenters return "Y" equal to "X" bitwise when y is X (same draw, same rows); nanometre augmenters refuse a decreasing, repeated or non-finite axis.

  • Kernel equivalence: estimator output equals the per-method C function output (and each N4MP kind equals its kernel).

  • Cross-binding matrix: one fixture set produced by the C++ suite (inputs, params, N4ME bytes, outputs); Python, R and JS each load the same bytes, predict, and must match within the documented tolerance; each binding also exports and the other two import. The matrix (method × language × operation) is published in docs/parity/ and is the only basis for coverage claims.

6. Decisions on the review questions

  1. Class labels: integer IDs in the core and in N4ME; hosts own the ID→name mapping in their envelope, under the shared label contract (D9).

  2. No JSON parser in the core; bindings generate typed setters from the manifest dump.

  3. The 2.12 filter symbols are removed as one unreleased ABI revision (headers, snapshots and change log together), after the non-distribution check above.

7. Review record

One adversarial Codex review (2026-09-26) raised: LW-PLS has no out-of-sample path; classifier inputs/probabilities were underspecified; procedures were exempted instead of covered; state serialization and size limits were underestimated; descriptor/lifetime rules were missing; array defaults, nested configs and paired-Y augmentation were not expressible; and 2.12 removal needed a release-boundary rule. All are addressed above (D0, D2–D7, §4).