Fitted-model wire serialization

The C ABI implements three backward-readable fitted-model wire versions behind:

  • n4m_model_export_size

  • n4m_model_export_to_buffer

  • n4m_model_import_from_buffer

  • n4m_serialization_inspect

  • n4m_serialization_inspect_model_v1

  • n4m_serialization_inspect_pipeline_v1

The format is little-endian and starts with:

magic[4] = "N4MM"
u32 format_version = 1, 2 or 3
u32 writer_abi_major
u32 writer_abi_minor
u32 writer_abi_patch

The shared payload stores model metadata, PLS centring/scaling statistics, coefficients, latent matrices and optional training scores. A trailing FNV-1a 64-bit checksum covers every byte before the checksum field. Imports reject bad magic, truncated payloads, impossible dimensions, length mismatches and checksum failures with N4M_ERR_CORRUPT_BUFFER; unsupported format versions return N4M_ERR_VERSION_INCOMPATIBLE.

Format 1 remains byte-for-byte unchanged and is still emitted for every model without a pipeline that records no component resolution. Format 2 is emitted only for such a model fitted with the exact SNV(no params) -> Savitzky-Golay smooth(window, poly) pipeline. Format 3 (ABI 2.20) is emitted for every model that records a component resolution: the PLS regression solvers (also for PLS-DA) and PCR fitted from 1.3.7 on, with or without the pipeline. After the eleven format-1 vectors it appends:

u64 count = 1
f64 component_resolution   # finite, >= +0 (X-Y cross-covariance resolution)
u64 count = n_components
f64 unresolved[n_components]  # each +0 or 1
u32 pipeline_present       # 0 or 1
[the format-2 pipeline block below when pipeline_present = 1]
u64 fnv1a64 checksum

Imports and inspection reject other counts, values (negative, non-finite or -0 resolution; flags other than +0 and 1) and pipeline markers even when the checksum is recomputed. A model imported from format 1 or 2 has no recorded resolution: N4M_MODEL_COMPONENT_RESOLUTION and N4M_MODEL_UNRESOLVED_COMPONENTS then return N4M_ERR_UNSUPPORTED, and it re-exports as format 1 or 2. Readers before ABI 2.20 reject format 3 with N4M_ERR_VERSION_INCOMPATIBLE.

The format-2 pipeline block (also used by format 3) follows. When the SG operator omits parameters, model-pipeline parsing uses the public nirs4all defaults window=11 and poly=3; export always records them explicitly. In format 2 it follows the eleven format-1 vectors directly:

u32 operator_count = 2
u32 first_kind = N4M_OP_SNV
u32 first_param_count = 0
u32 second_kind = N4M_OP_SAVGOL_SMOOTH
u32 second_param_count = 4
f64 window                 # odd integer, 3..501
f64 polynomial_degree      # integer, 0 <= degree < window
f64 derivative_order = 0
f64 delta = 1
u32 semantic_profile = N4M_PIPELINE_SEMANTIC_PROFILE_NIRS4ALL_SNV_SAVGOL_V1
u32 snv_axis = 1
u32 snv_with_mean = 1
u32 snv_with_std = 1
u32 snv_ddof = 0
u32 savgol_mode = N4M_PP_SAVGOL_INTERP
f64 savgol_cval = +0
u64 fnv1a64 checksum

This fixed block is deliberately not a general pipeline envelope. Fit rejects different orders, operators and algorithms. Inspection/import reject altered tags, counts or parameters even when the non-cryptographic checksum is recomputed. The restored model owns the fitted stateless pipeline and reapplies it before prediction and latent transformation.

The ABI fields are provenance, not a strict load gate in either version. The current importer accepts a structurally valid payload written by a different ABI version and records a compatibility warning on the context. Callers must therefore not describe an ABI-major or newer-minor mismatch as a guaranteed rejection.

N4MM is a raw fitted-model payload. It has no canonical filename extension yet and is not the nirs4all .n4a pipeline bundle. The possible .n4am envelope remains a future contract and is not part of any N4MM version. The FNV-1a trailer detects corruption; it is not a content-address or a training-data fingerprint.

n4m_serialization_inspect_model_v1 is the authoritative, allocation-free inspection gate for a complete fitted-model payload. It validates the header, supported format, checksum, algorithm/solver/deflation recipe, dimensions, flags, all eleven length-delimited array sections, the format-3 component-resolution section, the pipeline block when present, and exact end-of-payload before filling n4m_serialized_model_info_v1_t. The output is zero on every failure. Its schema version is 1 and its capability bits are derived from the validated N4MM bytes:

  • native format-1 fitted models report PREDICT | TRANSFORM;

  • the bounded format-2 pipeline model reports PREDICT | TRANSFORM | PIPELINE;

  • format-3 models report PREDICT | TRANSFORM, plus PIPELINE when their pipeline flag is set;

  • N4M_ALGO_IMPORTED_LINEAR_PREDICTOR reports PREDICT | AFFINE and never reports TRANSFORM.

Unknown model recipes return N4M_ERR_UNSUPPORTED; future wire versions return N4M_ERR_VERSION_INCOMPATIBLE; malformed or checksum-invalid payloads return N4M_ERR_CORRUPT_BUFFER. External JSON, manifests and filename conventions are not capability authorities.

n4m_serialization_inspect_pipeline_v1 is the additive ABI-2.5 plan attestation surface. The caller passes both a pointer and its allocation size; the current n4m_serialized_pipeline_info_v1_t is 96 bytes. Format 1, and format 3 without the pipeline flag, return success with present=0. Format 2, and format 3 with the flag, return operator_count=2, ordered kinds N4M_OP_SNV then N4M_OP_SAVGOL_SMOOTH, the versioned user semantic profile, SNV axis/mean/std/ddof, SG window/polynomial/derivative/delta/mode/cval, and separate raw/model feature widths. Both widths equal n_features because the only admitted plan is dimension-preserving. The profile fixes row-wise SNV with population standard deviation (ddof=0), then SG interpolation at both edges (derivative=0, delta=1, cval=+0).

The plan fingerprint algorithm is N4M_PIPELINE_FINGERPRINT_FNV1A64_V1: FNV-1a 64 over the exact validated 84-byte little-endian block beginning at operator_count and ending at savgol_cval. It fingerprints the canonical plan, its semantic profile and parameters, not learned model arrays or training data. The canonical 5/2 plan has fingerprint 0x4ec584c6e32e3416. The inspector uses the same full decoder as the model descriptor, so checksum failure, invalid tags or counts, non-canonical negative zero, inconsistent dimensions and trailing bytes fail before any field is published. The writable output prefix is zero on error.

Optimizer checkpoint wire format

The optimizer uses a separate N4MOPT format behind n4m_optimizer_save / n4m_optimizer_load. Version 1 is a little-endian, self-contained study checkpoint:

magic[8] = "N4MOPT\r\n"
u32 format_version = 1
u32 header_size = 32
u64 total_size                 # exact; includes zero padding + checksum
u64 payload_size               # excludes padding + checksum
payload[payload_size]
zero padding to an 8-byte total
u64 fnv1a64(header || payload || padding)

The payload starts with independently hashed, length-delimited canonical encodings of the ordered SearchSpace and normalized n4m_optimizer_options_t. It then stores base state (SplitMix64 state, elapsed-time offsets, next ids/sequences, full trial lifecycle, intermediates, structured errors and enqueue FIFO) and a sampler-tagged state block. Sampler blocks contain only portable scalars/vectors: Sobol Gray-code state, GA population, PSO particle/best state, separable CMA-ES distribution/population, or GP-EI decoded proposals. LHS designs and ternary/TPE state are deterministically derived from the encoded options, space, RNG and history. Pruner decisions are likewise reconstructed from options and intermediate history.

The decoder is transactional and applies a 64 MiB envelope, per-string and per-collection limits, and remaining-buffer checks before allocation. It rejects unknown versions, non-zero padding, trailing/truncated data, invalid UTF-8, non-canonical option/space fingerprints, invalid lifecycle event graphs and sampler-state shape mismatches. It never reads or writes a pointer, size_t, C++ object representation, vtable, clock epoch, or host endianness-dependent scalar. The FNV checksum is corruption detection, not a security MAC.

The public save symbol predates byte dtypes. It therefore returns an owning N4M_DTYPE_I64 array whose underlying bytes are the aligned checkpoint; bindings must treat the storage as bytes, not numeric word values. Python presents those bytes directly through Optimizer.save() / Optimizer.load().

Before resuming, compare the loaded optimizer with a fresh expected optimizer using n4m_optimizer_configuration_matches(actual, expected, &matches) or Python loaded.configuration_matches(expected). The native comparison covers ordered space, category codecs, constraints/conditions and options; history, RNG, queues and elapsed/adaptive state are excluded. Both handles must be open, and the comparison leaves them unchanged. A trial budget is a caller policy, not an optimizer option, so callers may extend it separately.