Roadmap

⚠️ LEGACY (pre-rename). This is the original phase log from before the pls4all/p4a → nirs4all-methods/n4m rename (it still says pls4all, references Direction_Technique.md/Overview.md, and stops at phase-49 / v0.97). It is kept as a historical record of the shipped algorithm phases. The authoritative, current plan is REFACTOR_PLAN.md (phases A–F); do not treat anything below as current.

pls4all is built in deliberate phases. Each delivered phase has a self-contained note under roadmap/. The canonical technical spec is Direction_Technique.md. The full algorithm taxonomy is in Overview.md.

The project rule remains:

  • C++17 internal core.

  • Public stable C ABI only.

  • No mandatory runtime dependency beyond libc/libstdc++/libgcc.

  • Thin bindings over the C ABI.

  • Every numerical method must have a deterministic parity gate against the best available R or Python reference.

  • GitHub Actions are parked for now to save runner quota; the authoritative gate is the local parity/build/test/sanitizer run.

Current Checkpoint - 2026-05-16

Latest local tag: phase-49-vissa-pls (0.97.0+abi.1.14.0).

Last green local gate:

  • 97 deterministic parity fixtures.

  • 256 C++ ABI/core tests (unchanged at 256; Batch 1 promotes existing internal kernels to public ABI rather than adding new ones).

  • pls4all_cli --selfcheck.

  • pls4all_cli --bench smoke for every shipped PLS solver.

  • Python ctypes smoke: bindings/python/smoke_aom_pop.py exercises every AOM/POP fixture through the public C ABI.

  • Python model smoke: pls4all.Model.fit / predict / get_array succeeds.

  • ABI symbol diff against cpp/abi/expected_symbols_linux.txt: 159 symbols, all p4a_* prefixed.

  • ldd dependency audit: only libc/libstdc++/libgcc/libm/loader.

  • UBSAN.

  • ASAN+UBSAN.

  • Benchmarks: python benchmarks/run.py --check passes for every shipped suite (aom_global, pls_regression, matrix).

  • Parity gate (benchmarks/parity_timing/runner.py): 13 PASS (all external refs), 24 paper-only (smoke-only), 0 numpy-mirror.

Current git notes:

  • main is ahead of origin/main by ~25 local commits.

  • Untracked files intentionally left out of commits: Backlog.md, docs/_bench/.

  • AOM-PLS parity oracle is nirs4all/bench/AOM_v0/aompls, not the packaged nirs4all library surface.

Status Summary

Done

  • Phase 0 — ABI / build foundation, fixture schema, CLI selfcheck.

  • Phase 1 — Dependency-free NIPALS PLS regression with fit / predict / transform, fitted-array accessors, serialization.

  • Phase 3 (a–r) — Preprocessing pipeline (identity, center, autoscale, Pareto, SNV, MSC, EMSC, Detrend, Savitzky-Golay, Norris-Williams, ASLS, Haar wavelet, OSC, EPO), regression + classification metrics + calibration, splitters (k-fold, LOO, holdout, external, repeated, Monte-Carlo, Kennard-Stone, SPXY), cross-validation engine, VIP and selectivity ratio, component-prefix coefficients.

  • Phase 4 (a–s) — SIMPLS / SVD / PCR / linear kernel / wide-kernel / orthogonal-scores / power / randomized-SVD solvers; PLSCanonical (NIPALS + SVD); PLSSVD; PLS-DA; OPLS / OPLS-DA / multi-response shared predictive score; SIMPLS component-count CV; PLS-LDA; PLS-logistic; MB-PLS; LW-PLS.

  • Phase 5 (a–u) — Variable-selection family: rangers (VIP, coefficient, selectivity ratio), interval (moving-window, biPLS, siPLS), stability (Monte-Carlo, UVE, EMCUVE, randomization), wrappers (SPA, CARS, Random Frog, SCARS, GA, Shaving, REP, IPW, ST, BVE, T², WVC, WVC threshold).

  • Phase 6 (a–f) — AOM-PLS / POP-PLS: strict-linear operator family (identity, polynomial detrend, zero-padded Savitzky-Golay, finite difference, Norris-Williams, Whittaker, FCK); global AOM-SIMPLS CV selection; POP-PLS per-component covariance selection; public C ABI for AOM/POP including the p4a_validation_plan_t opaque handle.

  • Phase 7a — Benchmark foundation: benchmarks/ directory, Python ctypes driver, AOM-PLS global benchmark vs nirs4all/bench/AOM_v0/aompls oracle, gated accuracy CSV + informational timing CSV.

  • Phase 7f (partial shipped) — Native AOM robust-HPO compact/wide preprocessing screen through n4m_aom_robust_hpo_fit; broader source-free portfolio/gating objects remain Python/reference layer until separately hardened.

Phase 7 - Benchmark Foundation - shipped through 7e

  • Phase 7b: Python ctypes Model wrapper + NumPy zero-copy MatrixView constructor. Model.fit/predict/transform/get_array cover all 9 shipped PLS regression solvers. New benchmarks/runners/pls_regression.py runner (smoke gated; full matrix on demand).

  • Phase 7c: minimum-viable R package at bindings/r/pls4all/ with .Call gateway and fit/predict wrappers for the same solver list. No Rcpp dependency. Documentation for build / install / smoke. R toolchain not installed on this host — package builds against the C ABI on a future host; matrix runner skips R rows gracefully when Rscript is absent.

  • Phase 7d: pls4all_cli --bench subcommand that times native C++ fit + predict per algo on a deterministic synthetic dataset and prints a CSV-parseable row.

  • Phase 7e: benchmarks/runners/matrix.py orchestrator. For each (algo, n_samples, n_features) cell it times native C++ / pls4all- Python / pls4all-R / sklearn reference, writes a gated accuracy CSV and an informational timing CSV + summary markdown + metadata JSON. Smoke set is committed; full matrix (9 algos × 5 n × 3 p = 135 cells) is parameterized — re-run under varying CPU pinning when the host is free.

  • Phase 7f (partial shipped): AOM robust-HPO integration. The native compact/wide strict-linear screen is catalogued and wrapped in Python. The broader portfolio/gating layer remains reference-only until it gets native ABI, timing and catalog contracts.

Phase 8 - 15 - PLS extensions (§7, §11, §12, §13, §17, §18, §19)

Internal C++ kernels. Public C ABI exposure deferred to the binding tranche.

  • Phase 8 (§7): Sparse SIMPLS via soft-thresholding; cfg.sparsity_lambda, P4A_ALGO_SPARSE_PLS dispatch.

  • Phase 9 (§17): T² Hotelling, Q-residuals (SPE), DModX in cpp/src/core/pls_diagnostics.{hpp,cpp}.

  • Phase 10 (§18): one-SE rule on top of the existing component-count CV.

  • Phase 11 (§19): empirical percentile thresholds for T² and Q-residuals (pls_monitoring_fit / pls_monitoring_evaluate).

  • Phase 13 (§13): domain-invariant PLS (fit_di_pls).

  • Phase 14 (§11): just-in-time PLS — covered by the existing fit_predict_lw_pls (documented).

  • Phase 15 (§12): recursive PLS — moving-window SIMPLS refit (cpp/src/core/recursive_pls.{hpp,cpp}).

Phase 16 - 30 - Overview taxonomy completion

Closes every remaining Overview section. All internal C++ kernels.

  • Phase 16 (§8): O2PLS — bi-directional OPLS with predictive + X-orthogonal + Y-orthogonal components.

  • Phase 17 (§8): SO-PLS — sequential + orthogonalized PLS for B X-blocks predicting Y.

  • Phase 18 (§8): OnPLS — joint + per-block unique components.

  • Phase 19 (§8): ROSA — Response-Oriented Sequential Alternation (per-component best-block selection).

  • Phase 20 (§7): sPLS-DA via dummy encoding + sparse SIMPLS.

  • Phase 21 (§7): group sparse and fused sparse PLS variants.

  • Phase 22 (§9): N-PLS (Bro’s algorithm for 3-way tensors).

  • Phase 24 (§10.2): non-linear kernel PLS (RBF / polynomial / sigmoid) via Gram-matrix dual NIPALS.

  • Phase 25 (§1): CPPLS / powered PLS (continuous gamma).

  • Phase 26: weighted PLS, robust PLS (Huber IRLS), ridge PLS, continuum regression.

  • Phase 27: PLS-GLM, PLS-QDA, PLS-Cox.

  • Phase 28: PDS (window-LS) and DS calibration transfer, MIR-PLS, missing-aware NIPALS.

  • Phase 29 (§18): approximate-PRESS with leverage-inflated residual PRESS and component selection.

  • Phase 30 (§20): bagging-PLS, boosting-PLS, random-subspace PLS.

All shipped as internal kernels in cpp/src/core/{multiblock_extensions,tensor_pls,kernel_pls,extra_pls}.{hpp,cpp}.

Next (Phase 6 continuation)

  • Phase 6g — POP holdout / approximate-PRESS / one-SE variants; AOM-NIPALS materialized engine; AOM/POP covariance and adjoint fast paths; soft / sparse / superblock AOM selection fixtures; per-block and per-target AOM plans.

  • Phase 6h — Integrate the locally developed AOM-PLS and FCL-PLS work as first-class pls4all methods.

Later (binding + algorithm tracks)

  • Binding roadmap (after Phase 7c): MATLAB MEX, JS / WebAssembly, Julia, Java / Android, plus secondary targets (C#, Rust, Go, Swift).

  • Algorithm taxonomy backlog — see the Remaining Algorithm Taxonomy section below.

Next Agent Prompt

Continue from /home/delete/nirs4all/pls4all on main, currently tagged phase-49-vissa-pls (0.97.0+abi.1.14.0). Do not use GitHub Actions for now. Keep using the local gate: pinned fixture generator, dev-release build, C++ tests, CLI selfcheck (pls4all_cli --selfcheck), CLI bench smoke (pls4all_cli --bench --algo pls_simpls --samples 200 --features 100), Python smoke (bindings/python/smoke_aom_pop.py plus the version/context/config snippet in bindings/python/README.md plus pls4all.Model.fit/predict round-trip), ABI symbol diff, ldd, git diff --check, UBSAN and ASAN+UBSAN, and python benchmarks/run.py --check for the benchmark gate. Also run the new external-reference parity gate PYTHONPATH=bindings/python/src \ parity/python_generator/.venv/bin/python -m benchmarks.parity_timing.runner and re-commit benchmarks/results/parity_gate/. Leave untracked Backlog.md and docs/_bench/ alone. For AOM/POP parity, use /home/delete/nirs4all/nirs4all/bench/AOM_v0/aompls as the oracle.

Algorithm development is now complete for every Overview section. The next agent should:

  1. Public C ABI exposure for the new internal kernels shipped in pls4all::core:: over phases 6g – 30. Plan one MINOR ABI bump per batch (the §6g AOM/POP policy work, the §17/§18/§19 diagnostics and monitoring, the §8/§9/§10.2/§13 algorithm batches, and the §20 ensembles), each additive only. Update cpp/abi/expected_symbols_linux.txt and cpp/include/pls4all/p4a.h.

  2. Python ctypes bindings for the newly exposed surface; extend bindings/python/src/pls4all/* and the smoke driver.

  3. R .Call gateway for the same surface in bindings/r/pls4all/.

  4. MATLAB MEX, JS-WASM, Julia, Java — pick up the remaining binding targets once the Python / R coverage is solid.

  5. Benchmark matrix expansion: add new columns to benchmarks/runners/matrix.py for every shipped method. Run python benchmarks/run.py --benchmark matrix --scale full --repeats 5 under OMP_NUM_THREADS=1|5|10 when the host is free.

  6. Then start the Acceleration Roadmap (BLAS / OpenMP / CUDA).

For AOM/POP parity, use /home/delete/nirs4all/nirs4all/bench/AOM_v0/aompls as the oracle. The pinned parity/python_generator/ venv (sklearn 1.4.2) is still broken on this host’s Python 3.13; new fixture work should either restore that venv or migrate to a numpy-only reference path.

Shipped Core (phase log)

Phase 0 - ABI and Build Foundation - shipped

  • CMake project, presets, warning/sanitizer options.

  • Public p4a.h ABI with opaque handles, matrix views, enums, status codes and version/ABI compatibility functions.

  • Context/config/matrix/operator-bank/gating/pipeline/model/array lifecycle.

  • Shared and static libp4a; symbol snapshot gate.

  • CLI selfcheck.

  • Fixture schema and parity infrastructure.

  • Minimal ctypes Python lifecycle/config binding.

  • README skeletons for R, MATLAB, JS/WASM and Android.

Phase 1 - PLS CPU Reference - shipped

  • Dependency-free NIPALS PLS1/PLS2 regression.

  • Fit/predict/transform.

  • Center/scale learned on train and applied leak-free.

  • Fitted-array accessors.

  • Component-prefix coefficients.

  • Binary export/import with versioned serialization.

Phase 3 - Preprocessing, Validation and Metrics - shipped through 3r

  • Pipeline fit/transform.

  • Identity, center, autoscale, Pareto, SNV.

  • MSC, EMSC.

  • Polynomial detrend.

  • Savitzky-Golay smoothing and first/second derivatives.

  • ASLS baseline.

  • Norris-Williams derivatives.

  • Haar wavelet denoising.

  • OSC and EPO.

  • Regression metrics: RMSE, MAE, bias, R2/Q2, slope/intercept, RPD, RPIQ.

  • Splitters: k-fold, LOO, holdout, external folds, repeated k-fold, Monte-Carlo, Kennard-Stone, SPXY.

  • Cross-validation engine.

  • Binary and multiclass classification metrics plus calibration bins.

  • VIP and selectivity ratio.

  • Original-scale component-prefix coefficients.

Phase 4 - Advanced PLS Variants - shipped through 4s

  • SIMPLS.

  • SVD PLS.

  • PCR.

  • Linear kernel-algorithm PLS.

  • Wide-kernel PLS.

  • Orthogonal-scores PLS.

  • Power-method PLS.

  • Randomized-SVD PLS.

  • PLSCanonical with NIPALS/SVD.

  • PLSSVD direct cross-covariance scores.

  • PLS-DA.

  • OPLS and OPLS-DA.

  • Multiclass OPLS-DA/common predictive score model.

  • SIMPLS component-count CV.

  • PLS-LDA.

  • PLS-logistic.

  • MB-PLS.

  • LW-PLS.

Phase 5 - Variable Selection - shipped through 5u

  • VIP, coefficient-magnitude and selectivity-ratio rankers.

  • Moving-window / contiguous interval CV.

  • biPLS backward interval selection.

  • siPLS interval-combination search.

  • Monte-Carlo coefficient stability.

  • UVE with artificial variables.

  • SPA-PLS.

  • CARS-PLS.

  • Random Frog PLS.

  • SCARS-PLS.

  • GA-PLS.

  • Shaving-PLS.

  • REP-PLS.

  • IPW-PLS.

  • ST-PLS.

  • BVE-PLS.

  • T2-PLS.

  • WVC-PLS numeric selection.

  • WVC threshold/factor rules.

  • EMCUVE.

  • PLS randomization-test selection.

Phase 6 - AOM-PLS and POP-PLS - shipped through 6f

  • Phase 6a: internal soft/hard AOM preprocessing-bank transform primitive.

  • Phase 6b: internal global AOM-SIMPLS CV selection against nirs4all/bench/AOM_v0/aompls for identity/detrend.

  • Phase 6c: strict-linear AOM kernels for zero-padded Savitzky-Golay, finite difference and Norris-Williams, plus wider global-selection parity.

  • Phase 6d: strict-linear Whittaker and FCK AOM operators, with direct transform parity and global-selection parity.

  • Phase 6e: internal POP-PLS per-component SIMPLS covariance selector, with selected operator sequence, component candidate scores, prefix scores, full-fit predictions and bench-compatible CV scoring semantics.

  • Phase 6f: public C ABI for validation plans, AOM global and POP per- component selection (opaque result handles, typed accessors, hardened fold validation). Python ctypes smoke that drives every shipped AOM/POP fixture through the new surface.

Phase 7 - Benchmark Foundation - shipped through 7a

  • Phase 7a: benchmarks/ directory with a deterministic Python driver. First runner compares the public C ABI p4a_aom_global_select against nirs4all/bench/AOM_v0/aompls on 4 cases (identity-favoured and detrend-favoured). Numerical deltas are gated by python benchmarks/run.py --check; wall-clock timings are recorded separately and informational.

Active Track

Phase 6g - POP/AOM Policy Expansion

Goal: complete the policy variants already present in the bench oracle.

Deliverables:

  • POP holdout / approximate-PRESS / one-SE variants.

  • AOM-NIPALS materialized path.

  • AOM/POP covariance and adjoint fast paths where the strict-linear operator contract permits it.

  • Soft, sparse and superblock AOM selection fixtures.

  • Per-block and per-target AOM plans.

Reference:

  • nirs4all/bench/AOM_v0/aompls/selection.py

  • nirs4all/bench/AOM_v0/aompls/simpls.py

  • nirs4all/bench/AOM_v0/aompls/nipals.py

  • nirs4all/bench/AOM_v0/aompls/banks.py

Phase 6h - Local AOM/FCL Integration

Goal: integrate the locally developed AOM-PLS and FCL-PLS work as first-class pls4all methods.

Deliverables:

  • Identify the local source-of-truth implementations and freeze parity fixtures before porting.

  • Port FCL-PLS kernels behind internal C++ APIs.

  • Add fixtures for AOM/FCL edge cases, not only synthetic happy paths.

  • Document numerical conventions and deviations from the local prototypes.

Binding Roadmap

Phase 2 is intentionally delayed until the core stabilizes. It is now active in parallel with Phase 7 because the benchmark matrix needs every shipped language binding.

Python (Phase 7b — in progress)

  • Expand ctypes binding beyond lifecycle/config and AOM/POP.

  • NumPy zero-copy matrix views (p4a_matrix_view_t from numpy.ndarray.ctypes.data, ascontiguousarray fallback).

  • Fit/predict/transform wrappers for every shipped solver (NIPALS, orthogonal-scores, SIMPLS, kernel, wide-kernel, SVD, power, randomized-SVD, plus PCR).

  • sklearn-compatible BaseEstimator / RegressorMixin / TransformerMixin classes (deferred to a later Python tranche).

  • Fixture-driven Python parity tests.

R (Phase 7c — in progress)

  • .Call gateway over the C ABI.

  • R matrices to p4a_matrix_view_t without unnecessary copies when possible (via REAL() on a contiguous matrix).

  • Parity against R pls, ropls, mixOmics, plsVarSel where applicable.

  • CRAN-compatible package skeleton and smoke tests.

MATLAB

  • MEX gateway.

  • Class wrapper for model handles.

  • Model import/export.

  • Prediction-first examples.

JavaScript / WebAssembly

  • Emscripten build of the C ABI.

  • Predict-first npm package.

  • TypedArray matrix views.

  • Model load/predict smoke tests.

Julia

  • Thin ccall wrapper.

  • Matrix view helpers.

  • Parity fixture runner.

Java / Android

  • JNI bridge and Kotlin API.

  • Android AAR.

  • Predict-first model loading.

  • On-device smoke tests.

Other bindings to consider after these are stable: C#, Rust, Go and Swift.

Benchmark Roadmap

Benchmarks ship under benchmarks/ and follow a strict split:

  • Accuracy CSVs are committed and gated by python benchmarks/run.py --check. They contain only deterministic numerical deltas (operator match, component-count match, RMSE / coefficient deltas).

  • Timing CSVs are committed for traceability but NOT gated. They are platform-dependent, recorded with explicit host info (Python version, platform, processor, logical cores, environment variables such as OMP_NUM_THREADS / OPENBLAS_NUM_THREADS).

  • Summary markdown is regenerated on every run with a status table.

Shipped

  • 7a — AOM-PLS global selection vs nirs4all/bench/AOM_v0/aompls, 4 cases (9x6, 12x8, 16x10, 14x9), Python ctypes path.

Active

  • 7b — PLS regression benchmark matrix: 9 solvers (NIPALS / orthogonal-scores / SIMPLS / kernel-algorithm / wide-kernel / SVD / power / randomized-SVD / PCR) × 5 sample sizes (200, 500, 1000, 2000, 10000) × 3 feature counts (100, 1000, 10000), pls4all-Python vs scikit-learn PLSRegression. CPU-count parameterization through environment variables (run later under 1 / 5 / 10 cores when the host machine is free).

  • 7c — R-side numbers added to the same matrix via the minimum R binding shipped in the binding roadmap.

  • 7d — Native C++ baseline column via pls4all_cli --bench.

  • 7e — Orchestrator that runs the full matrix end-to-end, splits outputs per language and per core count, regenerates the summary markdown.

  • 7f (partial shipped) — Native AOM robust-HPO compact/wide screen shipped; broader portfolio/gating layer remains reference-only.

Later

  • POP per-component benchmark column.

  • AOM-PLS benchmark expanded to the wider operator bank (SG, Norris- Williams, Whittaker, FCK).

  • Preprocessing throughput vs NumPy / SciPy references.

  • Variable-selection runtime vs Python / R references.

  • LW-PLS and MB-PLS scaling.

  • Batch CV, bootstrap, CARS / MCUVE / AOM sweeps.

  • Memory footprint and dependency audit per build.

  • GPU / OpenMP / BLAS backend self-parity once the Acceleration Roadmap starts.

Benchmark outputs live under benchmarks/results/; only curated CSVs and summaries are committed.

Acceleration Roadmap

Acceleration remains optional and must not change the C ABI.

  • BLAS backend.

  • OpenMP for fold/bootstrap/operator scans.

  • Batch APIs for large validation and selection workloads.

  • CUDA backend in a separate optional shared library.

  • Later: WebGPU/WASM SIMD if the JS target needs it.

Every accelerated backend needs self-parity against reference CPU.

Remaining Algorithm Taxonomy

These are not yet first-class shipped methods and should be split into small reviewed phases with parity references:

  • CPPLS / powered PLS.

  • Weighted and sample-weighted PLS.

  • Robust PLS.

  • Ridge/regularized/penalized PLS.

  • Continuum regression.

  • MIR-PLS.

  • Sparse PLS and sparse PLS-DA.

  • PLS-QDA, PLS-GLM, PLS-Cox and survival PLS.

  • O2PLS, DOSC-PLS, OnPLS.

  • Missing-aware NIPALS.

  • Calibration transfer and domain adaptation: PDS, DS, di-PLS and related methods.

  • Multiway/tensor PLS: N-PLS, Tri-PLS, PARAFAC-PLS, Tucker-PLS.

  • Dynamic, recursive and online PLS.

  • Ensemble PLS methods: bagging, boosting and random-subspace PLS.

Release Discipline

For every phase:

  1. Add or update a focused roadmap/phase-*.md.

  2. Implement the smallest coherent tranche.

  3. Generate parity fixtures with the pinned generator venv.

  4. Build with /home/delete/.venv/bin/cmake --build --preset dev-release --parallel.

  5. Run /home/delete/.venv/bin/ctest --preset dev-release --output-on-failure.

  6. Run build/dev-release/cpp/tests/pls4all_tests.

  7. Run CLI selfcheck and Python smoke.

  8. Run ABI symbol diff and ldd dependency audit.

  9. Run git diff --check.

  10. Run UBSAN and ASAN+UBSAN local builds.

  11. For benchmark phases, run python benchmarks/run.py --check.

  12. Commit as release(phase-X): version-topic.

  13. Tag as phase-X-topic.

Do not push unless explicitly asked.