Introduction

RustyML is a machine learning and deep learning library written entirely in Rust. It needs no external library to link against, no Python interpreter, and no FFI boundary to move arrays across. This guide is the hands-on companion to the library. It takes you from a fresh cargo new to training neural networks. It also covers tuning the parallel and serial thresholds inside the hot kernels, and shows what lands on disk when you serialize a model. This guide tracks RustyML 0.14. Every complete example compiles against that version with the full and show_progress features on, so what you read is what the compiler accepts.

What this guide is

The API reference on docs.rs is the source of truth for every signature, every trait bound, and every enum variant. This guide does not duplicate that reference. It explains the decisions a signature does not state. docs.rs lists the arguments to Adam::new. This guide explains which optimizer to choose. It explains why a per-call random_state overrides the global seed instead of the reverse. It also explains what happens when you load a saved network into a model whose layer shapes no longer match. The guide stays concrete, states an opinion where the evidence supports one, and names the pitfalls in specific estimators.

The classical-ML half of the crate follows one shape almost everywhere: construct, fit, predict. That is the same (&x, &y)-then-&x rhythm scikit-learn uses.

use rustyml::prelude::machine_learning::*;
use ndarray::array;

fn main() {
    // A tiny linear relationship: y = 3 * x
    let x = array![[1.0], [2.0], [3.0], [4.0]];
    let y = array![3.0, 6.0, 9.0, 12.0];

    // new(fit_intercept). The default solver is the exact closed form
    let mut model = LinearRegression::new(true);
    model.fit(&x, &y).unwrap();

    let predictions = model.predict(&x).unwrap();
    println!("predictions: {:?}", predictions);
}

The rest of this guide unpacks that pattern, one module at a time. Models share the Fit and Predict traits. Metrics always take (y_true, y_pred) in that order. A single set_global_seed call makes the whole crate deterministic.

Who it is for

This guide serves 2 kinds of readers. The first is a Rust developer who wants machine learning without leaving the Rust ecosystem. That reader needs no pip, no linked BLAS, and no unsafe FFI to audit. A cargo add command adds a crate that follows Rust’s rules on ownership, Send/Sync, and error handling. The second is an ML practitioner coming from Python, fluent in scikit-learn and Keras. That reader wants the same mental model in Rust. RustyML gives it: fit/predict methods and a Keras-style Sequential model built with .add(...) and .compile(...). Its confusion matrices and silhouette scores mean what they mean in scikit-learn. RustyML expresses these ideas as compiled, statically typed, parallel-by-default Rust. Where RustyML departs from scikit-learn or Keras, this guide states the difference and the reason for it.

You do not need prior experience with Rust numerical code. You should be comfortable reading Rust and running cargo. Data flows through ndarray arrays throughout the crate, so Working with ndarray covers that library before you meet it in every later chapter.

How the book is organized

The crate splits into 5 feature-gated modules: machine_learning, neural_network, utils, metrics, and math. It also has a domain-split prelude. The chapters follow that structure. Read the first chapter start to finish, then use the rest as reference material.

Chapters 2 and 3 cover the two large algorithm families and most of the crate’s surface area. Chapters 4 through 6 support those families. Preprocessing runs before a model, metrics run after it, and both rest on the same numerical primitives. Chapter 7 covers cross-cutting advanced topics. It explains how random_state resolves against the global seed, what the postcard-serialized bytes of a saved model contain, and how the runtime-tunable parallelism gates work. It also shows how to compile a build that pulls in only metrics or only math.

How to read it

Read Getting Started once, in order, from start to end. It is the only chapter that assumes you read the chapters before it. It installs the crate, sets up ndarray, and builds a complete model, so every later chapter can build on that shared base. Every chapter after Chapter 1 is reference material, and you can read those chapters in any order. Jump straight to Support Vector Machines, Optimizers, or Clustering Metrics as your problem demands. Follow the inline cross-links when one page depends on a concept from another page. If RustyML already compiles for you and a model already trains, Chapter 1 has done its job. Treat the rest of the guide as a manual: open the page you need.

Versions, feedback, and conventions

This edition tracks the current crate version, 0.14. RustyML is pre-1.0 and under active development. The API is stabilizing, but breaking changes can still land in minor releases. If a signature in this guide differs from what the compiler accepts, trust docs.rs for the exact version in use. Send bug reports, feature requests, and corrections to this guide to the GitHub repository. Issues and pull requests are welcome there.

This guide follows 2 conventions. First, a complete example is a self-contained program with a main function, a tiny inline dataset, and few iterations. Paste a complete example into a full-feature project and run it as written. A fragment or a signature is marked as such. Second, outside the leaf metrics and math modules, every fallible call returns RustymlResult<T>, an alias for Result<T, rustyml::error::Error> over a structured, matchable error enum. The examples reach for .unwrap() only to stay short. Error Handling shows what to write instead in real code.

1. Getting Started

RustyML is a machine learning and deep learning library written entirely in Rust: no BLAS to link, no Python runtime, no C++ bindings.

Prerequisites. You should be comfortable with cargo, Rust’s ownership and borrowing, and the basic supervised loop (train, predict, evaluate). No prior ndarray experience is required — 1.3 covers the part of it you need. Everything runs on Rust 1.89 or newer, with no system libraries to install.

1.1 What is RustyML: a short tour of what RustyML offers (classical ML estimators, a Keras-style neural-network stack, preprocessing, evaluation metrics), and an honest account of the trade-offs pure Rust brings.

1.2 Installation and Feature Flags: what each feature means and which ones you should turn on.

1.3 Working with ndarray: a short introduction to ndarray. Every matrix RustyML takes in or hands back is an ndarray type, so knowing how to use ndarray is unavoidable.

1.4 Your First End-to-End Model: building a complete model pipeline with RustyML, hands on.

1.5 The Prelude and Imports: how RustyML’s prelude modules let you pull names into scope quickly.

1.6 Error Handling: what RustyML’s error type is, and how you should handle failures.

1.1. What is RustyML

RustyML is a machine learning and deep learning library written entirely in Rust. It covers the full workflow a data-science project needs — data preprocessing, feature engineering, model training, and evaluation. It provides classical machine-learning estimators (linear models, decision trees, SVMs, clustering, dimensionality reduction, anomaly detection) as well as a Keras-style neural-network framework.

This guide documents version 0.14. The API is stabilizing, but minor releases can still introduce breaking changes, so pin a concrete version in Cargo.toml for production rather than tracking *. The authoritative API reference is at docs.rs/rustyml, and the source at github.com/SomeB1oody/RustyML.

1.1.1. Pure Rust, end to end

RustyML contains no C or C++ code: no BLAS to link, no LAPACK, no CUDA. That makes it highly portable, spares you from configuring a complicated environment by hand, and keeps you from hitting inscrutable errors at build time — which suits both production and newcomers. Most of the code is written in safe Rust, so its memory safety is guaranteed.

For performance, matrix multiplication goes through the pure-Rust gemmkit crate (reaching ndarray via the zero-copy gemmkit-ndarray adapter). It dispatches at run time to the widest SIMD the current CPU actually supports (AVX-512F, AVX2+FMA, NEON, wasm simd128, with a scalar fallback), and decides on its own whether a given product is worth threading and across how many workers — so you get excellent performance across very different hardware.

1.1.2. Parallelism

RustyML parallelizes its compute-heavy kernels with Rayon, but never blindly. Below a certain size the overhead of multithreading makes the parallel path slower than the serial one, so every kernel class has a calibrated size threshold and switches to the parallel path only once parallel is measurably faster. Those thresholds are not hardcoded constants: rustyml::tuning lets you override them at run time without a recompile, which matters most when you deploy the same binary to machines with very different core counts. See Performance Tuning and Parallelism for the details.

What the design buys you:

• Parallel reductions are deterministic: the blocked fold sums in a fixed order no matter how many threads run it, so results do not drift as you scale cores.
• Performance is predictable: no garbage-collection pauses, no JIT warmup, and no global interpreter lock serializing your threads.
• Nearly every randomized component honors a global seed (see Reproducibility and Random Seeds), so a run reproduces across machines. The dimensionality reducers’ iterative eigensolvers are deliberately left out, because they converge to the same result whatever the seed is.

1.1.3. Features and modules

RustyML is split into five modules, each controlled by a Cargo feature (prelude is shared). Naming features lets you compile only the parts you use. machine_learning, neural_network, utils, and metrics all enable math automatically.

The default feature turns everything on. There is also a separate show_progress feature that draws training progress bars; see Installation and Feature Flags.

1.1.4. An API modeled on scikit-learn and Keras

Classical estimators follow scikit-learn, exposing methods like fit and predict; the neural network follows Keras’s Sequential model with its add / compile / fit / predict flow — which makes it easier to pick up if you know the Python data-science ecosystem. What changes is that data is ndarray arrays rather than NumPy (see Working with ndarray), and fallible calls return Result instead of raising exceptions.

Here is a classical machine-learning example; you can see the API shape follows scikit-learn:

use rustyml::prelude::machine_learning::*;
use ndarray::array;

fn main() {
    // new(fit_intercept); the default solver is the exact closed form
    let mut model = LinearRegression::new(true);

    let x = array![[1.0, 2.0], [2.0, 3.0], [3.0, 4.0]];
    let y = array![6.0, 9.0, 12.0];

    model.fit(&x, &y).unwrap();
    let predictions = model.predict(&x).unwrap();
    println!("predictions: {:?}", predictions);
}

And here is neural-network code, whose architecture follows Keras:

use rustyml::prelude::neural_network::*;
use ndarray::Array;

fn main() {
    // 4 samples, 8 input features, 1 output
    let x = Array::ones((4, 8)).into_dyn();
    let y = Array::ones((4, 1)).into_dyn();

    let mut model = Sequential::new();
    model
        .add(Dense::new(8, 16, Activation::ReLU).unwrap())
        .add(Dense::new(16, 1, Activation::Linear).unwrap())
        .compile(
            Adam::new(0.001, 0.9, 0.999, 1e-8, 0.0).unwrap(),
            MeanSquaredError::new(),
        );

    model.summary(); // prints the architecture, just like Keras
    model.fit(&x, &y, 5).unwrap();

    let predictions = model.predict(&x).unwrap();
    println!("prediction shape: {:?}", predictions.shape());
}

The metrics keep the same design as scikit-learn too (every metric takes its arguments in (y_true, y_pred) order):

use rustyml::metrics::*;
use ndarray::array;

fn main() {
    let y_true = array![1.0, 0.0, 0.0, 1.0, 1.0];
    let y_pred = array![1.0, 0.0, 1.0, 1.0, 0.0];

    let cm = ConfusionMatrix::new(&y_true, &y_pred);
    println!("accuracy: {:.3}", cm.accuracy());
    println!("f1 score: {:.3}", cm.f1_score());
}

Unlike Python, RustyML’s error propagation wraps the result of a fallible call in Result<T, Error>, and you can match on it to handle each outcome separately (either it succeeded and gives you T, or it failed and gives you an Error). See Error Handling.

Hyperparameters are validated at the point they are supplied, and an illegal value is rejected on the spot. Configuration uses the builder pattern: an estimator names its essential hyperparameters in new, then layers optional settings through chained with_* methods, each validating what it receives (for example LinearRegression::new(true).with_regularization(..)?).

1.1.5. What pure Rust buys you

A RustyML program compiles to a single self-contained binary. There is no extra toolchain to install and no complicated environment to configure. Trained classical models and neural-network weights serialize to binary through save_to_path / load_from_path; see Model Persistence in Depth. And because there is no GC, no interpreter, and no warmup, latency is predictable.

1.1.6. Notes on scope

RustyML is CPU-only. There is no GPU or CUDA backend. The neural-network framework suits small-to-medium models and deep learning that sits close to classical ML; it is not for training large vision or language models. The classical machine_learning and utils estimators all take an f64 feature matrix, but the element type of what predict gives back varies by model, see the table in Working with ndarray. The neural-network stack works in f32, and its tensor type is Tensor = ArrayD<f32>. The framework does not build a dynamic autodiff graph the way PyTorch does.

1.2. Installation and Feature Flags

First you need a Rust toolchain installed locally — Rust 1.89 or newer, with cargo available. Check with:

rustc --version
cargo --version

The output should look something like:

rustc 1.96.0 (ac68faa20 2026-05-25)
cargo 1.96.0 (30a34c682 2026-05-25)

Note that RustyML turns on every feature by default. If you want to trim it down and enable only part of it, you can configure that by hand in Cargo.toml; that is covered further down, so there is nothing to worry about yet.

1.2.1. Adding RustyML to your project

If you have not created a project yet, make one with cargo:

cargo new my_project # replace my_project with whatever name you want

A successful run prints something like:

$ cargo new my_project
    Creating binary (application) `my_project` package
note: see more `Cargo.toml` keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

RustyML is easy to find on crates.io.

Open the project’s Cargo.toml and add this under [dependencies]:

[dependencies]
rustyml = "0.14"
ndarray = "0.17"

• The first line means “use version 0.14 of the rustyml crate with its default features” (that is, with everything enabled), so all of RustyML is available to you.
• The second line means “use version 0.17 of the ndarray crate”. You will almost always need ndarray as a direct dependency as well, because it is the crate RustyML depends on to carry your data.

Sometimes you do not need all of RustyML’s features, and you can pick a combination by adjusting the features field on that first dependency. A few common shapes:

# Omitting `features` means the default feature set (everything)
rustyml = "0.14"

# Just the neural-network framework (requires turning the default off)
rustyml = { version = "0.14", default-features = false, features = ["neural_network"] }

# Everything: machine_learning, neural_network, utils, metrics, math (equivalent to default)
rustyml = { version = "0.14", features = ["full"] }

# Everything, plus terminal progress bars during training
rustyml = { version = "0.14", features = ["full", "show_progress"] }

7.4 Minimal Builds and Modular Integration goes into trimming features in depth.

1.2.2. The feature matrix

Eight features control the build. default and full are bundles.

Some things the table cannot show:

• machine_learning, neural_network, utils, and metrics all enable math automatically. You never need to add math by hand alongside another module — only if the numeric module is all you want (see 6. Math Utilities).
• rustyml::error (the unified Error type, see 1.6 Error Handling) and rustyml::random (global-seed control, see 7.1 Reproducibility and Random Seeds) appear whenever any of machine_learning, neural_network, or utils is on — but not in a metrics-only or math-only build, since those two leaf modules neither return RustymlResult nor consume randomness.
• rustyml::tuning (runtime parallelism gates, see 7.3 Performance Tuning and Parallelism) is present no matter which feature you enable.
• The rustyml::prelude module is always compiled, so you can always import through it; see 1.5 The Prelude and Imports.

1.2.3. Which third-party crates each feature pulls in

The table below shows which third-party libraries each RustyML feature needs. You do not need any of this to use the crate; it is here so you can see the dependency picture.

The matrix-multiply backend lives in gemmkit rather than in RustyML. It exposes GEMMKIT_* environment variables for machine-specific tuning rather than general scheduling; see 7.3 Performance Tuning and Parallelism.

serde, postcard, and thiserror are the heavy dependencies. Dropping them (by staying on metrics and/or math) shrinks build time and the dependency count — for example:

rustyml = { version = "0.14", default-features = false, features = ["metrics"] }

1.2.4. A minimal end-to-end check

With the dependency written into Cargo.toml, put this in src/main.rs:

use rustyml::prelude::machine_learning::*;
use ndarray::array;

fn main() {
    // 3 samples, 2 features
    let x = array![[1.0, 2.0], [2.0, 3.0], [3.0, 4.0]];
    let y = array![6.0, 9.0, 12.0];

    // new(fit_intercept) -> Self; the default solver is exact OLS.
    let mut model = LinearRegression::new(true);
    model.fit(&x, &y).unwrap(); // train the model

    // predict with the trained model
    let predictions = model.predict(&x).unwrap();
    println!("predicted {} values", predictions.len());
}

Run cargo run; the expected output is:

predicted 3 values

1.2.5. Pairing with ndarray

RustyML uses the array types ndarray provides, both on the way in and on the way out. The classical machine-learning module takes an Array2<f64> feature matrix and an Array1<f64> target vector; the neural network uses Tensor, an alias for ArrayD<f32> (note that ML and utilities are f64 while the network is f32). RustyML 0.14 builds against ndarray 0.17, so make sure the ndarray version in your Cargo.toml is 0.17.

RustyML does not re-export ndarray’s constructors, so building the arrays you feed in (array!, Array2::from_shape_vec, Array::ones, and friends) means calling into the ndarray crate. That is why it has to be a dependency in your Cargo.toml. 1.3 Working with ndarray covers the array-building patterns the estimators expect.

1.2.6. Using show_progress for training progress bars

show_progress adds no types and unlocks no module. With it enabled, the terminal shows a progress bar with elapsed time, position, and the current loss, and prints “Training completed” when it finishes; the mini-batch loop (fit_with_batches) shows the average loss over the batches of the epoch it is part-way through. That bar is not how you get the loss out — fit and fit_with_batches return a History carrying one loss per epoch whether or not the feature is on. What it buys you is watching the number move while a long run is still going, instead of reading it once the call returns.

Add "show_progress" to rustyml’s features in Cargo.toml:

rustyml = { version = "0.14", features = ["full", "show_progress"] }

Run cargo run again and you will see the bar (there is so little data in this example that you may not catch it moving):

[00:00:00] ######################################## 1000/1000 | Cost: 0.001324 | Max iterations | Iterations: 1000
predicted 3 values

It is best turned on only for interactive use.

1.2.7. MSRV

RustyML’s minimum supported Rust version is 1.89. On an older toolchain the resolver refuses the crate. rustup update stable updates the toolchain.

Beyond that there is nothing to configure — which is exactly RustyML’s advantage.

1.3. Working with ndarray

Every matrix type in RustyML comes from ndarray. If you want the more advanced parts of ndarray — broadcasting rules, the various iterators, linear-algebra helpers — read the official ndarray documentation.

RustyML 0.14 depends on ndarray 0.17. Your own Cargo.toml needs to use the same ndarray version:

[dependencies]
ndarray = "0.17"
rustyml = { version = "0.14", features = ["full"] }

1.3.1. f64 versus f32

In RustyML the elements of a classical machine-learning matrix are f64, while the neural-network stack’s tensor elements are f32.

The classical estimators (see machine_learning) take a two-dimensional Array2<f64> feature matrix laid out as samples by features (one row per sample), plus a one-dimensional Array1<_> target vector. The target’s element type varies by model:

The neural-network stack uses Tensor, which is a type alias:

pub type Tensor = ArrayD<f32>;

Tensor differs from the classical machine-learning matrices in two ways:

• The element type is f32: half the precision, half the memory, and the width every deep-learning framework standardizes on, because neural networks are compute-bound and the extra throughput is worth a few bits of precision.
• The dimensionality is dynamic: ArrayD (equivalently Array<f32, IxDyn>) carries its rank at run time rather than encoding it in the type. A wrong rank surfaces at run time as an Err; see 1.3.7.

1.3.2. Building arrays

Here are the common ways to build the ndarray matrix types:

use ndarray::{array, Array, Array1, Array2};

fn main() {
    // `array!`: the shape is inferred from the nesting — this is a 3x2 `Array2<f64>`
    let m = array![[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]];
    assert_eq!(m.shape(), &[3, 2]);

    // `from_shape_vec`: (shape, flat row-major Vec)
    // The length must equal the product of the shape, or you get an Err
    let x = Array2::from_shape_vec((2, 3), vec![1.0, 2.0, 3.0, 4.0, 5.0, 6.0]).unwrap();
    assert_eq!(x.shape(), &[2, 3]);

    // `from_shape_fn`: give it a shape, plus a closure that produces each element
    let g = Array2::from_shape_fn((3, 2), |(i, j)| (i * 2 + j) as f64);
    assert_eq!(g[[2, 1]], 5.0);

    // `from_vec`: turn a `Vec<T>` into an `Array1<T>`
    let y = Array1::from_vec(vec![2.0, 4.0, 6.0]);
    // `from_iter`: fill an `Array1<T>` from an iterator
    let labels = Array1::from_iter(0..3i32);

    // Constant / all-zero / all-one fills, with the element type spelled out
    let zeros = Array2::<f64>::zeros((2, 2));
    let ones = Array2::<f64>::ones((2, 2));
    let filled = Array::from_elem((2, 2), 7.0f64);

    println!("{} {} {} {}", y.len(), labels.len(), zeros.sum(), ones.sum());
    println!("{}", filled.sum());
}

Most data arrives as nested Vecs, and that is when you reach for from_shape_vec. It reads the Vec in row-major order. Its return value is wrapped in a Result, and if you get an Err, the shape is almost certainly wrong.

Often you want to initialize a matrix randomly. The ndarray-rand crate provides Array::random for that, and you add it to your Cargo.toml:

[dependencies]
ndarray-rand = "0.16"
ndarray = "0.17"

Example:

use ndarray::Array;
use ndarray_rand::RandomExt; // the extension trait that gives ndarray's array types
                             // random construction — bring it into scope for `Array::random`
use ndarray_rand::rand_distr::Uniform; // samples uniformly between the given bounds

fn main() {
    let w = Array::random((4, 3), Uniform::new(-1.0, 1.0).unwrap());
    println!("{:?}", w.shape());
}

When you use RustyML, do not seed ndarray-rand’s generator by hand. RustyML has its own randomness management (set_global_seed, a per-model with_random_state, and Sequential::new_with_seed), so control randomness through RustyML’s API instead; see 7.1 Reproducibility and Random Seeds.

1.3.3. Inputs to the classical estimators

Here is the shape a typical supervised classical fit and predict signature takes:

pub fn fit<S1, S2>(&mut self, x: &ArrayBase<S1, Ix2>, y: &ArrayBase<S2, Ix1>) -> Result<&mut Self, Error>
where
    S1: Data<Elem = f64>,
    S2: Data<Elem = f64>,

Two things depart from it, both visible in the table in 1.3.1: LDA’s y is Data<Elem = i32> rather than f64, and the unsupervised estimators (KMeans, DBSCAN, MeanShift, PCA, KernelPCA) take the feature matrix alone, with no y argument at all.

ndarray has two kinds of borrow:

• a reference to an owned array (&Array)
• a view (ArrayView), which is what .view() and .slice(..) produce
• Both are zero-copy, and RustyML accepts either. The parameter is a reference, so write &array or &array.view(); a bare array.view() does not type-check.

You also have to keep the feature matrix two-dimensional (Ix2) and the target vector one-dimensional (Ix1).

use rustyml::machine_learning::LinearRegression;
use ndarray::{s, Array1, Array2};

fn main() {
    let x = Array2::from_shape_vec((4, 1), vec![1.0, 2.0, 3.0, 4.0]).unwrap();
    let y = Array1::from_vec(vec![2.0, 4.0, 6.0, 8.0]);

    let mut model = LinearRegression::new(true);

    // A reference to the owned array
    model.fit(&x, &y).unwrap();

    // A reference to a view
    model.fit(&x.view(), &y.view()).unwrap();

    // A reference to a slice: train on the first three rows only
    // This subview is a borrow into `x`
    let x_head = x.slice(s![0..3, ..]);
    let y_head = y.slice(s![0..3]);
    model.fit(&x_head, &y_head).unwrap();
}

What .view() and .slice(..) really buy you is the ability to feed a subset of an array — a range of rows, a strided window, a single column reshaped — straight into fit or predict without first materializing that subset as a new owned array. To train on rows 0..800 of a 1000-row matrix, x.slice(s![0..800, ..]) is all it takes.

1.3.4. Inputs to the neural-network stack

The neural layers want ArrayD<f32>, but you almost never construct one directly. The usual path is to build a fixed-dimension array and then convert it to the dynamic dimension IxDyn. That conversion is a thin wrapper; it does not touch the data:

use rustyml::neural_network::sequential::Sequential;
use rustyml::neural_network::layers::{Activation, Dense};
use rustyml::neural_network::optimizers::Adam;
use rustyml::neural_network::losses::MeanSquaredError;
use ndarray::Array2;

fn main() {
    // 4 samples, 2 features: build an `Array2<f32>` first, then convert to `ArrayD`
    let x = Array2::from_shape_vec(
        (4, 2),
        vec![0.0f32, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 1.0],
    )
    .unwrap()
    .into_dyn(); // convert to a dynamic-dimension tensor
    let y = Array2::from_shape_vec((4, 1), vec![0.0f32, 1.0, 1.0, 0.0])
        .unwrap()
        .into_dyn(); // convert to a dynamic-dimension tensor

    let mut model = Sequential::new();
    model
        .add(Dense::new(2, 4, Activation::ReLU).unwrap())
        .add(Dense::new(4, 1, Activation::Linear).unwrap());
    model.compile(Adam::new(0.001, 0.9, 0.999, 1e-8, 0.0).unwrap(), MeanSquaredError::new());

    model.fit(&x, &y, 5).unwrap();
    let preds = model.predict(&x).unwrap();
    println!("prediction tensor shape: {:?}", preds.shape());
}

Output:

prediction tensor shape: [4, 1]

1.3.5. Input shapes

RustyML’s input format is one row per sample, one column per feature.

The tensor shapes for neural-network layers match Keras’s default (batch, height, width, channels). The channel axis is always the trailing one and the batch axis is always the leading one, with the spatial axes in between — so a Keras user should feel at home. Here are the exact layouts:

For example:

• A batch of 8 single-channel 28×28 images is [8, 28, 28, 1]
• A bank of 3×3 filters producing 16 feature maps has weight shape [3, 3, 1, 16]

Here is a code example:

use rustyml::neural_network::layers::{Activation, Conv1D};
use rustyml::neural_network::traits::Layer;
use ndarray::Array;

fn main() {
    // filters=2, kernel=3, input_shape=[batch, length, channels], stride=1
    let mut conv = Conv1D::new(2, 3, vec![1, 6, 1], 1, Activation::Linear).unwrap();

    // Channels last: [batch=1, length=6, channels=1]
    let input = Array::from_shape_vec((1, 6, 1), vec![1.0f32, 2.0, 3.0, 4.0, 5.0, 6.0])
        .unwrap()
        .into_dyn();

    let output = conv.forward(&input).unwrap();
    println!("conv output shape: {:?}", output.shape()); // [1, 4, 2]
}

The constructor’s input_shape argument records the [batch, length, channels] the layer is configured for. Passing a tensor whose rank or channel count disagrees is rejected at forward time. For the per-layer detail, see 3.5 Convolutional Layers and 3.6 Pooling Layers.

1.3.6. Practical advice

Real data rarely arrives as an Array2. The most common shape is a Vec<Vec<f64>>, one inner Vec per parsed CSV row. The conversion is to flatten it into a single row-major Vec<T> and hand it to from_shape_vec along with the (rows, cols) you counted:

use ndarray::Array2;

fn main() {
    // As if parsed from CSV: 3 rows, 2 columns
    let rows: Vec<Vec<f64>> = vec![
        vec![1.0, 2.0],
        vec![3.0, 4.0],
        vec![5.0, 6.0],
    ];
    let n_rows = rows.len();
    let n_cols = rows[0].len();

    // `flatten()` flattens the nested vectors in row-major order
    let flat: Vec<f64> = rows.into_iter().flatten().collect();
    let x = Array2::from_shape_vec((n_rows, n_cols), flat).unwrap();

    assert_eq!(x.shape(), &[3, 2]);
    println!("{:?}", x.row(0)); // [1.0, 2.0]
}

If your rows are ragged, flatten still flattens the nested vectors, but the length will not equal n_rows * n_cols and from_shape_vec returns an Err.

Other tools you will use:

• The s! macro expresses ranges and strides
• concatenate joins arrays along an existing axis
• outer_iter (equivalently axis_iter(Axis(0))) walks each row as a one-dimensional view, which is how you iterate sample by sample

Code example:

use ndarray::{array, concatenate, s, Axis};

fn main() {
    let x = array![[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]];

    // Slice: all rows, first two columns (drop a feature)
    let first_two_cols = x.slice(s![.., 0..2]);
    assert_eq!(first_two_cols.shape(), &[3, 2]);

    // Slice: a single row as a submatrix
    let middle = x.slice(s![1..2, ..]);
    assert_eq!(middle.shape(), &[1, 3]);

    // Concatenate two matrices along axis 0 (append rows)
    let more = array![[10.0, 11.0, 12.0]];
    let stacked = concatenate(Axis(0), &[x.view(), more.view()]).unwrap();
    assert_eq!(stacked.shape(), &[4, 3]);

    // Iterate each row as an `ArrayView1`
    // The sample-by-sample idiom
    for (i, row) in x.outer_iter().enumerate() {
        println!("row {i} sums to {}", row.sum());
    }
}

Turning raw labels into the Array1 a model wants is usually a map over your source data collected into an array. When a model or a loss expects one-hot targets rather than integer class ids, use RustyML’s to_categorical function, which turns an Array1<i32> of class ids into an Array2<f64> one-hot matrix:

use rustyml::utils::to_categorical;
use ndarray::Array1;

fn main() {
    // Map string labels to integer class ids, then into an `Array1<i32>`
    let raw = ["cat", "dog", "cat", "bird"];
    let ids: Array1<i32> = raw
        .iter()
        .map(|s| match *s {
            "cat" => 0,
            "dog" => 1,
            _ => 2,
        })
        .collect::<Vec<_>>()
        .into();

    // One-hot encode: 4 samples, 3 classes -> a (4, 3) `Array2<f64>`
    let onehot = to_categorical(&ids, None).unwrap();
    assert_eq!(onehot.shape(), &[4, 3]);
    println!("{:?}", onehot.row(1)); // [0.0, 1.0, 0.0]
}

Label encoding, its inverse, and the mapping-preserving variants are covered in full in 4.3 Label Encoding. Preprocessing your feature matrix with standardize and normalize is in 4.2 Standardization and Normalization.

1.3.7. Shape-mismatch errors

Two different error systems catch shape problems. Errors that happen while you are building an array come from ndarray — from_shape_vec returns ndarray’s ShapeError when the buffer length disagrees with the shape. Errors that happen while RustyML runs come back as RustyML’s own Error enum, which subdivides them further:

• Error::DimensionMismatch { expected, found }: two scalar counts disagree, such as the feature count at predict time versus at fit time, or x.nrows() versus y.len()
• Error::ShapeMismatch { expected, found }: two whole tensor shapes disagree, such as a gradient in the neural-network stack not matching the activation it flows into
• Error::InvalidInput(_): the input’s rank is wrong, such as handing a 2-D tensor to a Conv1D that expects 3-D

The one you will hit most often is a feature-count mismatch: you train on p features and then call predict with a matrix that has a different number of columns. RustyML validates this and returns DimensionMismatch:

use rustyml::machine_learning::LinearRegression;
use rustyml::error::Error;
use ndarray::{Array1, Array2};

fn main() {
    // Train on 2 features
    let x = Array2::from_shape_vec((3, 2), vec![1.0, 2.0, 2.0, 3.0, 3.0, 4.0]).unwrap();
    let y = Array1::from_vec(vec![6.0, 9.0, 12.0]);
    let mut model = LinearRegression::new(true);
    model.fit(&x, &y).unwrap();

    // Predict with 3 features
    // The model returns an error
    let bad = Array2::from_shape_vec((1, 3), vec![4.0, 5.0, 6.0]).unwrap();
    match model.predict(&bad) {
        Err(Error::DimensionMismatch { expected, found }) => {
            println!("expected {expected} features, got {found}");
        }
        other => println!("unexpected: {other:?}"),
    }
}

Because Error is marked #[non_exhaustive], a match over it always needs a trailing _ => arm, which keeps your code compiling when new error variants are added. For the full error-handling story, see 1.6 Error Handling.

1.4. Your First End-to-End Model

1.4.1. The complete program

use ndarray::{Array1, Array2};
use rustyml::machine_learning::LogisticRegression;
use rustyml::metrics::{ConfusionMatrix, accuracy};
use rustyml::set_global_seed;
use rustyml::utils::StandardScaler;
use rustyml::utils::train_test_split::train_test_split;

fn main() {
    // Set the global seed for reproducibility
    set_global_seed(42);

    // Build a synthetic 2-class dataset
    let n_per_class = 30usize;
    let mut rows: Vec<f64> = Vec::with_capacity(2 * n_per_class * 2);
    let mut labels: Vec<f64> = Vec::with_capacity(2 * n_per_class);
    for i in 0..n_per_class {
        let a = (i % 6) as f64;
        let b = (i / 6) as f64;
        // Class 0 is centered near -0.6
        // Class 1 near +0.6
        rows.push(-0.6 + 0.3 * a);
        rows.push((-0.6 + 0.3 * b) * 100.0);
        labels.push(0.0);
        rows.push(0.6 + 0.3 * a);
        rows.push((0.6 + 0.3 * b) * 100.0);
        labels.push(1.0);
    }
    let n = labels.len();
    let x: Array2<f64> = Array2::from_shape_vec((n, 2), rows).unwrap();
    let y: Array1<f64> = Array1::from(labels);

    // Split into a training and a test set
    let (x_train, x_test, y_train, y_test) =
        train_test_split(x, y, Some(0.25), Some(42)).unwrap();

    // Standardize: fit the statistics on the training set only, then apply them to both splits
    let mut scaler = StandardScaler::new();
    let x_train_std = scaler.fit_transform(&x_train).unwrap();
    let x_test_std = scaler.transform(&x_test).unwrap();

    // Train
    let mut model = LogisticRegression::new(true, 0.5, 1000, 1e-6).unwrap();
    model.fit(&x_train_std, &y_train).unwrap();

    // Predict on the test set
    let preds: Array1<f64> = model.predict(&x_test_std).unwrap();

    // Evaluate
    let acc = accuracy(&y_test, &preds);
    let cm = ConfusionMatrix::new(&y_test, &preds);
    let (tp, fp, tn, fn_) = cm.get_counts();
    println!("test accuracy = {:.3}", acc);
    println!("TP={tp} FP={fp} TN={tn} FN={fn_}");
    println!("{}", cm.summary());
    println!("iterations run: {:?}", model.get_actual_iterations());

    // Save and load the model
    let path = "logreg_model.bin";
    model.save_to_path(path).unwrap();
    let loaded = LogisticRegression::load_from_path(path).unwrap();
    let preds_loaded = loaded.predict(&x_test_std).unwrap();
    println!("round-trip predictions identical: {}", preds == preds_loaded);
}

These println! calls produce a small report, laid out like this:

test accuracy = 0.XXX
TP=.. FP=.. TN=.. FN=..
Confusion Matrix:
+-----------------+--------------------+--------------------+
|                 | Predicted Positive | Predicted Negative |
+-----------------+--------------------+--------------------+
| Actual Positive | TP: ..             | FN: ..             |
| Actual Negative | FP: ..             | TN: ..             |
+-----------------+--------------------+--------------------+
... derived metrics ...
iterations run: Some(..)
round-trip predictions identical: true

1.4.2. Seeding for reproducibility

rustyml::set_global_seed takes a u64. It sets a thread-local seed, and every unseeded (random_state == None) component constructed afterwards on the same thread derives its RNG from it — the Keras-style global-seed model. See 7.1. Reproducibility and Random Seeds.

In this particular pipeline it changes none of the numbers. LogisticRegression::fit is plain full-batch gradient descent with no randomness at all (fitting twice on the same data gives bit-identical weights), and standardize is deterministic too. The only stochastic step is the shuffle inside train_test_split, and I pass it an explicit Some(42). Since a local seed, when set, outranks the global one, that step follows the local seed. Still, make a habit of calling set_global_seed at the top of your code.

1.4.3. The dataset and the label contract

The features are an Array2<f64> of shape (n_samples, n_features) and the labels are an Array1<f64>. The label element type is not incidental: LogisticRegression::fit requires y to be an f64 array whose values are exactly 0.0 or 1.0 (you can see the rule stated in its documentation), and anything else returns Error::InvalidInput. There is no LabelEncoder step folded into fit; if your labels are strings or arbitrary integers, you convert them yourself — see 4.3. Label Encoding.

1.4.4. Splitting the data

let (x_train, x_test, y_train, y_test) = train_test_split(x, y, Some(0.25), Some(42)).unwrap();

train_test_split consumes x and y, so clone them beforehand if you still need the originals afterwards. Passing None for test_size means 0.3 (a 70/30 split), and None for random_state means seed-from-entropy (or from the global seed, if one is set).

An empty dataset returns Error::EmptyInput, mismatched x/y lengths return Error::DimensionMismatch, a test_size outside (0, 1) is Error::InvalidParameter, and a dataset too small to split returns Error::InvalidInput. The function keeps at least one sample on each side: with ten rows and test_size = 0.99, the training side still has one row rather than zero.

On imbalanced data, a plain random split can push an entire class into the test set or into the training set. That is when train_test_split_stratified is the better choice, since it splits each class independently.

The split mechanics and the stratified variant are covered in 4.1. Train-Test Split.

1.4.5. Standardizing without leaking test information

Logistic regression is fit by gradient descent, and gradient descent cares about feature scale. The code above builds feature 2 a hundred times larger than feature 1, which lets feature 2 dominate the gradient, so each feature needs standardizing to zero mean and unit variance.

There are two ways to standardize:

• Call the function standardize(data, axis): a stateless, one-shot transform that computes the mean and standard deviation from the array you pass it, returns a new standardized array, and keeps nothing.
• Use the StandardScaler struct and its methods: it holds state, so fit computes the statistics and stores them, and transform applies the stored statistics to any array you give it later.

With the function, the common mistake is to call standardize once on x_train and again on x_test, which leaves the training and test sets on two different scales. Standardizing the whole matrix before splitting folds the test set’s mean and variance into the numbers the model trains on, which inflates your accuracy.

The correct approach is the StandardScaler struct and its methods — fit once on the training set, then transform everything:

use rustyml::utils::StandardScaler;

let mut scaler = StandardScaler::new();
let x_train_std = scaler.fit_transform(&x_train).unwrap(); // learns mean/std HERE, from train
let x_test_std = scaler.transform(&x_test).unwrap();       // applies the same statistics to test

Use fit/fit_transform on the training set only, and transform for everything else. The statistics are readable (scaler.get_mean(), get_scale()). The divisor is the population standard deviation (ddof = 0, matching scikit-learn’s StandardScaler).

The function is the right call when you do not need statistics to stay consistent, or when you need the Row/Global axes the struct does not cover:

use ndarray::array;
use rustyml::utils::standardize::{standardize, StandardizationAxis};

fn main() {
    let x = array![[1.0, 200.0], [3.0, 400.0], [5.0, 600.0]];
    let z = standardize(&x, StandardizationAxis::Column).unwrap();
    println!("standardized shape: {:?}", z.dim());
    println!("{:?}", z);
}

Output:

standardized shape: (3, 2)
[[-1.224744871391589, -1.224744871391589],
 [0.0, 0.0],
 [1.224744871391589, 1.224744871391589]], shape=[3, 2], strides=[2, 1], layout=Cc (0x5), const ndim=2

For the fuller tutorial, see 4.2. Standardization and Normalization.

1.4.6. Training the model

let mut model = LogisticRegression::new(true, 0.5, 1000, 1e-6).unwrap();
model.fit(&x_train_std, &y_train).unwrap();

LogisticRegression::new returns Result<Self, Error> because it validates its hyperparameters first and returns Error::InvalidParameter if they do not meet the requirements. You can also use LogisticRegression::default() for a model with default values (fit_intercept = true, learning_rate = 0.01, max_iter = 100, tol = 1e-4). Compared with those defaults, the learning rate and iteration budget passed to new here are both larger, because standardized features tolerate a bigger step. Regularization is off by default; enable it with .with_regularization(RegularizationType::L2(alpha)), whose return value is likewise wrapped in a Result (a negative or non-finite alpha returns an error). RustyML measures penalty strength the same way scikit-learn’s SGDClassifier/SGDRegressor does, while a LogisticRegression(C=c) corresponds to alpha = 1 / (c * n) for n training samples. The complete conversion table is in RegularizationType’s own documentation.

fit takes references to the feature matrix and the target vector. Its return value is wrapped in a Result to guard against:

• the 0.0/1.0 label check
• empty input (Error::EmptyInput)
• mismatched x/y lengths (Error::DimensionMismatch)
• non-finite feature values (Error::NonFinite). It also trips Error::NonFinite mid-loop if the weights or the loss overflow.

Iteration stops once the loss change between consecutive iterations falls below tol, or once the iteration count reaches max_iter. model.get_actual_iterations() tells you which of the two stopped it: if it equals max_iter, the model hit the iteration ceiling rather than converging, and you should raise max_iter or revisit the learning rate. For more on logistic regression, see 2.2. Logistic Regression.

1.4.7. Predicting

let preds: Array1<f64> = model.predict(&x_test_std).unwrap();

predict returns Result<Array1<f64>, Error>, each element a hard class label 0.0 or 1.0 (the sigmoid probability thresholded at 0.5). If you want the underlying probabilities rather than the labels, call predict_proba.

predict’s return value is likewise wrapped in a Result to guard against:

• calling it before fit, which is Error::NotFitted
• passing a feature count different from the one the model was trained on, which is Error::DimensionMismatch

1.4.8. Accuracy and the confusion matrix

let acc = accuracy(&y_test, &preds);
let cm = ConfusionMatrix::new(&y_test, &preds);
let (tp, fp, tn, fn_) = cm.get_counts();
println!("{}", cm.summary());

accuracy(y_true, y_pred) -> f64 computes accuracy, the fraction of matching labels. ConfusionMatrix::new(y_true, y_pred) builds the confusion matrix, and get_counts() returns (tp, fp, tn, fn). The confusion matrix takes hard labels only: every element of both arrays must be exactly 0.0 or 1.0, and anything else panics. If your labels use a different pair — the -1.0/+1.0 a margin classifier emits, say — use ConfusionMatrix::new_with_labels(y_true, y_pred, negative_label, positive_label), which corresponds to scikit-learn’s confusion_matrix(..., labels=[neg, pos]). Its methods are accuracy(), precision(), recall(), specificity(), f1_score(), balanced_accuracy(), and mcc(), and you can print summary() for the formatted table shown earlier. On imbalanced data, do not use raw accuracy — reach for balanced_accuracy() or mcc() instead.

For more classification metrics, see 5.2. Classification Metrics.

1.4.9. Saving and loading the model

model.save_to_path("logreg_model.bin").unwrap();
let loaded = LogisticRegression::load_from_path("logreg_model.bin").unwrap();

save_to_path(&self, path: &str) -> Result<(), Error> uses postcard to serialize the entire model — weights, intercept flag, learning rate, iteration count, regularization setting — into binary. load_from_path(path: &str) -> Result<Self, Error> reads it back. For the details, see 3.9. Saving and Loading Weights and 7.2. Model Persistence in Depth.

1.5. The Prelude and Imports

1.5.1. Three ways to bring names into scope

Glob the whole prelude

use rustyml::prelude::*;

This pulls everything into the current scope. It suits getting code written quickly, without having to look up where each item lives.

Glob a single prelude category

The prelude is split into four submodules, so you can import just what you need:

use rustyml::prelude::machine_learning::*; // classical estimators, traits, and shared enums
use rustyml::prelude::neural_network::*;   // Sequential, History, Tensor, layers, losses, optimizers
use rustyml::prelude::metrics::*;          // the evaluation-metric functions and types
use rustyml::prelude::utils::*;            // standardize, normalize, scalers, encoders, split

Use this when the file you are writing clearly belongs to a single domain.

Import by exact path

use rustyml::machine_learning::LinearRegression;
use rustyml::traits::{Fit, Predict};
use rustyml::metrics::r2_score;

Library code that has to be maintained long-term should prefer this style.

1.5.2. What the prelude re-exports

The prelude is a hand-picked list: it re-exports only the items you reach for often. The tables below show exactly what each prelude submodule re-exports.

Machine learning

Neural network

Note the exact casing on RMSprop (lowercase p). Activations come in two forms: one is the standalone layers ReLU/Softmax/Linear/Sigmoid/Tanh. The other is, in any layer that accepts an Activation enum, either picking a variant (Activation::ReLU, Activation::Softmax, Activation::Linear, Activation::Sigmoid, Activation::Tanh) or passing one of those standalone activation layers instead (they impl Layer and convert Into<Activation>).

For example, one of Dense::new’s parameters is activation: impl Into<Activation>, so you can pass either an Activation enum variant or a standalone activation layer: Dense::new(3, 8, Activation::ReLU) and Dense::new(3, 8, ReLU::new()) are equivalent.

Metrics

Unlike the error-propagation design in the rest of the crate, the metric functions panic outright on an error, which keeps the module lightweight; see 5. Model Evaluation. Their argument order is (y_true, y_pred), matching scikit-learn.

Utilities

1.5.3. Feature gates decide what the prelude contains

The rustyml::prelude module is always compiled, but each submodule only appears once the matching module feature is enabled. So use rustyml::prelude::* does not mean everything RustyML can do — it means everything the features you enabled can do. For what each feature contains, see 1.2 Installation and Feature Flags.

1.5.4. Using fully-qualified paths

To find where an item lives, docs.rs/rustyml is the place to look. Here is a lookup table for the main types:

1.6. Error Handling

1.6.1. One error type

RustyML has exactly one error type, rustyml::error::Error. Every fallible operation in the crate returns Result<T, rustyml::error::Error>, which also has an alias:

pub type RustymlResult<T> = std::result::Result<T, Error>;

Error is not in the prelude, so the error machinery is imported separately with use rustyml::error::Error; and friends. These are Error’s variants:

Note that DimensionMismatch compares scalar counts, such as a feature count or a vector length, while ShapeMismatch is about two whole tensor shapes disagreeing, which shows up mostly in the neural-network code.

Error is annotated #[non_exhaustive], which means a match over it must carry a wildcard _ => (or Err(e) =>) arm.

1.6.2. The domain sub-errors

Three of Error’s variants each wrap a smaller enum. Concerns that only apply to neural networks (layer state, weight shapes, compilation) and those that only apply to trees (classification versus regression) stay in their own enum.

NnError (at rustyml::neural_network::NnError) contains:

• ForwardPassNotRun(&'static str)
• WeightShape { name, expected, found }
• NotCompiled(&'static str)
• EmptyModel

Code example:

use rustyml::neural_network::sequential::Sequential;
use rustyml::neural_network::layers::Dense;
use rustyml::neural_network::layers::activation::ReLU;
use rustyml::neural_network::NnError;
use rustyml::error::Error;
use ndarray::Array;

fn main() {
    let mut model = Sequential::new();
    model.add(Dense::new(4, 2, ReLU::new()).unwrap());

    let x = Array::ones((3, 4)).into_dyn();
    let y = Array::ones((3, 2)).into_dyn();

    // compile() was never called, so no optimizer or loss is configured yet
    match model.fit(&x, &y, 1) {
        Ok(_) => unreachable!("training should not have started"),
        Err(Error::NeuralNetwork(NnError::NotCompiled(missing))) => {
            println!("compile the model first: `{missing}` is not specified");
        }
        Err(e) => println!("unexpected: {e}"),
    }
}

TreeError (at rustyml::machine_learning::TreeError) has these two variants:

• NotClassificationTree
• CorruptStructure(&'static str)

Code example:

use rustyml::machine_learning::{Algorithm, DecisionTree, TreeError};
use rustyml::error::Error;
use ndarray::array;

fn main() {
    // A regression tree (is_classifier = false) has no per-class probabilities
    let tree = DecisionTree::new(Algorithm::CART, false).unwrap();
    let x = array![[1.0, 2.0]];

    match tree.predict_proba(&x) {
        Err(Error::Tree(TreeError::NotClassificationTree)) => {
            println!("predict_proba is classification-only");
        }
        other => println!("unexpected: {other:?}"),
    }
}

IoError (at rustyml::error::IoError) has four variants:

• Std(std::io::Error) for filesystem failures
• Serialization(postcard::Error) for the binary format (RustyML serializes with postcard)
• ModelStructureMismatch(String) for when a loaded neural-network file does not match the target architecture (a different number of layers, a different layer type at some position, or a weight whose shape does not fit the target layer)
• UnsupportedModelFormat(String) for when the file is not a RustyML model file at all, or its on-disk format version is not the one this build writes

Code example:

use rustyml::machine_learning::LinearRegression;
use rustyml::error::{Error, IoError};

fn main() {
    match LinearRegression::load_from_path("model_that_does_not_exist.bin") {
        Ok(_) => unreachable!("the file should not exist"),
        Err(Error::Io(IoError::Std(io_err))) => {
            // io_err is the underlying std::io::Error (kind NotFound here).
            println!("filesystem error: {io_err}");
        }
        Err(Error::Io(IoError::Serialization(e))) => {
            println!("the file exists but is not a valid model: {e}");
        }
        Err(e) => println!("unexpected: {e}"),
    }
}

For the serialization format and versioning, see 7.2. Model Persistence in Depth.

1.6.3. Matching on specific variants

The everyday failure is calling predict before fit, which returns Error::NotFitted carrying its own name as a &'static str:

use rustyml::machine_learning::LinearRegression;
use rustyml::error::Error;
use ndarray::array;

fn main() {
    // Constructed, but never fitted
    let model = LinearRegression::new(true);
    let x = array![[1.0, 2.0], [3.0, 4.0]];

    match model.predict(&x) {
        Ok(preds) => println!("{preds:?}"),
        Err(Error::NotFitted(name)) => {
            println!("`{name}` was not fitted; call fit() first");
        }
        Err(Error::DimensionMismatch { expected, found }) => {
            println!("wrong feature count: model wants {expected}, got {found}");
        }
        // `Error` is `#[non_exhaustive]`, so the wildcard arm is mandatory
        Err(e) => println!("other error: {e}"),
    }
}

The DimensionMismatch arm is there to show the pattern; this particular call actually triggers NotFitted. But feed a fitted model a matrix with the wrong number of columns and you take the second arm, with expected set to the feature count seen at fit time and found set to the one you passed to predict.

1.6.4. Propagating with ?

The whole crate uses a single error type, so failures anywhere in a pipeline can be returned as Error with nothing beyond Result and ?:

use rustyml::machine_learning::{LinearRegression, RegularizationType};
use rustyml::error::RustymlResult;
use ndarray::{array, Array1, Array2};

fn train_and_predict(x: &Array2<f64>, y: &Array1<f64>) -> RustymlResult<Array1<f64>> {
    // Every ? below lifts a rustyml::error::Error out of a fallible call
    let mut model = LinearRegression::new(true)
        .with_regularization(RegularizationType::L2(0.01))?;       // maybe InvalidParameter
    model.fit(x, y)?;                                              // maybe EmptyInput / DimensionMismatch / NonFinite
    let preds = model.predict(x)?;                                 // maybe NotFitted / DimensionMismatch
    Ok(preds)
}

fn main() {
    let x = array![[1.0], [2.0], [3.0]];
    let y = Array1::from_vec(vec![2.0, 4.0, 6.0]);

    match train_and_predict(&x, &y) {
        Ok(preds) => println!("got {} predictions", preds.len()),
        Err(e) => eprintln!("pipeline failed: {e}"),
    }
}

When you do need to report a foreign error (from the standard library or another crate) while folding it into this scheme and keeping its cause chain, reach for the Context extension trait (which has to be imported into scope). It is implemented for any Result<T, E> whose E is Send + Sync + 'static and implements std::error::Error, so it composes with ?. context takes the message eagerly, while with_context takes a closure that runs only on the error path — prefer the closure form whenever building the message allocates (anything with format!), so the success path never runs it:

use rustyml::error::{Context, Error, RustymlResult};

fn parse_threshold(raw: &str) -> RustymlResult<f64> {
    // A std ParseFloatError, wrapped together with our context as Error::Computation,
    // with its source() chain preserved for downcasting later.
    let value: f64 = raw
        .parse()
        .with_context(|| format!("parsing threshold from {raw:?}"))?;
    Ok(value)
}

fn main() {
    match parse_threshold("not-a-number") {
        Ok(v) => println!("threshold = {v}"),
        Err(Error::Computation { context, source }) => {
            println!("{context}");
            if let Some(cause) = source {
                println!("  caused by: {cause}");
            }
        }
        Err(e) => println!("unexpected: {e}"),
    }
}

The foreign error becomes the source of an Error::Computation, reachable through the standard std::error::Error::source() chain and downcastable back to its original concrete type without losing any information.

1.6.5. Eager validation

RustyML’s error-handling design is that anything taking a hyperparameter validates it eagerly and returns Result, rather than panicking on illegal input.

use rustyml::machine_learning::LinearRegression;
use rustyml::machine_learning::linear_model::LeastSquaresSolver;
use rustyml::error::Error;

fn main() {
    // learning_rate must be positive and finite
    // 0.0 returns an error
    match LinearRegression::new(true).with_solver(LeastSquaresSolver::GradientDescent {
        learning_rate: 0.0,
        max_iter: 1000,
        tol: 1e-6,
    }) {
        Ok(_) => unreachable!("a zero learning rate must not be accepted"),
        Err(Error::InvalidParameter { name, reason }) => {
            // bad parameter `learning_rate`: must be positive and finite, got 0
            println!("bad parameter `{name}`: {reason}");
        }
        Err(e) => println!("unexpected: {e}"),
    }
}

A few places still panic outright:

• The functions in the metrics and math modules panic on an error instead of returning Result, which keeps those modules lightweight.
• Whether an ndarray operation outside RustyML returns Result or panics is ndarray’s decision, and RustyML has no say in it.

2. Classical Machine Learning

Classical machine learning covers everything in RustyML that is not a neural network: linear models, trees, kernel methods, clustering, and dimensionality reduction. These algorithms train fast, need little data, and produce models you can inspect. Reach for them first. Move to Chapter 3 only when a problem needs a deep network. Every estimator in this chapter lives under rustyml::machine_learning and shares one small contract. Construct it with new, which validates its arguments and returns Result. LinearRegression::new is the one exception: it cannot fail, so it returns Self directly. Train the model with fit, and run inference with predict, unless the model reduces dimensionality. The dimensionality-reduction transformers use transform and fit_transform instead of predict. Learn this rhythm once, and it carries across all 14 models in this chapter.

Read Chapter 1 before this chapter. Read Working with ndarray first, since every model consumes an Array2<f64> feature matrix. Read Error Handling too, since constructors and fit/predict all return the crate’s Result. Chapter 4 covers encoding and scaling your features. Chapter 5 covers the accuracy, silhouette, and R^2 scores you use to judge these models.

Every model in this chapter follows the shape below, with only the details changing:

use rustyml::machine_learning::LinearRegression;
use ndarray::array;

fn main() {
    // construct -> fit -> predict, the rhythm every estimator repeats
    let mut model = LinearRegression::new(true);
    let x = array![[1.0, 2.0], [2.0, 3.0], [3.0, 4.0]];
    let y = array![6.0, 9.0, 12.0];
    model.fit(&x, &y).unwrap();

    let preds = model.predict(&array![[4.0, 5.0]]).unwrap();
    println!("prediction: {:?}", preds);
}

Each section stands alone, so jump straight to the model you need. Reading in order moves from the simplest estimators to the most involved.

Supervised learning predicts a target from labeled examples. Linear Regression fits a continuous target, with optional L1/L2 regularization and a choice of a gradient-descent or a closed-form solver. It is the best place to learn the fit/predict loop. Logistic Regression reuses that gradient machinery for binary classification. K-Nearest Neighbors skips training and classifies by proximity, with a selectable distance metric and weighting scheme. Decision Trees split the feature space into readable if/else rules, using the ID3, C4.5, or CART algorithm, with pruning. Support Vector Machines covers 2 models: a kernelized SVC (SMO solver) for curved boundaries, and a fast LinearSVC for wide, high-dimensional data. Linear Discriminant Analysis classifies and reduces dimensions at the same time, by modeling each class as a Gaussian with a shared covariance.

Clustering groups unlabeled data. KMeans partitions points into a fixed number of clusters, using k-means++ initialization. It is fast, and the usual default choice. DBSCAN finds clusters of any shape by density, and labels outliers as noise. It needs no cluster count in advance. Mean Shift also finds the number of clusters on its own, by climbing a density surface. Score all 3 methods with the clustering metrics.

Dimensionality reduction compresses features while it keeps their structure. Principal Component Analysis is the linear default for decorrelation and compression. Kernel PCA extends it to nonlinear structure, through RBF, polynomial, and other kernels. t-SNE embeds high-dimensional data into 2 or 3 dimensions, for visualization only. It learns no reusable projection. It exposes fit_transform alone, with no out-of-sample transform.

Anomaly detection stands as its own family. Isolation Forest scores how easily each point isolates under random splits, and flags outliers without needing any labels.

2.1. Linear Regression

LinearRegression fits a linear map y_hat = X * w + b by least squares. RustyML’s default path is the exact closed form, like scikit-learn’s LinearRegression. The coefficients agree with Python’s results to about 1e-15. Batch gradient descent is available on demand, for cases the closed form cannot serve.

This split shapes the whole page. The default estimator has no learning rate, no iteration budget, and no convergence tolerance to set. The 3 settings that do exist belong to the iterative solver, and they travel with it. The source for this page is src/machine_learning/linear_model/linear_regression.rs and its integration tests.

2.1.1. How the model is trained

Two strategies minimize the same objective. LeastSquaresSolver picks between them.

The closed form (LeastSquaresSolver::Normal, the default) solves the ridge system in one shot. With no penalty, this is plain OLS. It runs an SVD least-squares solve on the augmented design [Xc; sqrt(lambda) * I]. This returns the minimum-norm solution even when X^T * X is singular, for example with perfectly collinear columns or more features than samples. When fit_intercept is set, the solver mean-centers the features and target first, so the intercept stays out of the penalty. It then recovers the intercept as mean(y) - mean(X) * w. The closed form has nothing to iterate. get_actual_iterations() returns Some(0) after a normal-solver fit. That 0 signals that the fit took no gradient steps.

Gradient descent (LeastSquaresSolver::GradientDescent) initializes the weight vector w to zeros and the intercept b to zero, then repeats a full-batch update. Each iteration computes the prediction and residual over the whole training set, the scalar cost, the gradients, and the parameter step.

The cost is the mean squared error, halved, plus an optional penalty. Let n be the sample count and e = X * w + b - y be the residual vector. One iteration computes cost = dot(e, e) / (2 * n) + penalty. The penalty term is 0 with no regularization. It is alpha * sum(|w_j|) for L1. It is (alpha / 2) * dot(w, w) for L2. The prediction adds the intercept b only when fit_intercept is set. Regularization never penalizes the intercept, because it shrinks slopes, not the bias.

The gradients follow directly. The weight gradient is grad_w = (X^T * e) / n, plus alpha * w for L2. The intercept gradient is grad_b = sum(e) / n, or 0 when fit_intercept is false. The update step is plain gradient descent with a fixed learning_rate. It sets w to w - learning_rate * grad_w and b to b - learning_rate * grad_b. L1 does not appear in the gradient. The solver applies it after the step, through a proximal operator (see 2.1.6). Gradient descent here has no momentum, no adaptive rate, and no line search. The step size you set on the solver is the step size used on every iteration. This is why feature scaling matters so much (see 2.1.5).

3 numerical guards run inside the loop. If the cost, any gradient component, or any updated parameter becomes NaN or infinite, fit aborts immediately with Error::NonFinite rather than returning a garbage model. A divergent learning rate trips this guard within a handful of iterations, instead of silently producing inf coefficients. The closed form has its own version of the same check on the solution it computes.

The solver computes the residual sum of squares and the intercept gradient’s sum with deterministic blocked folds. The matrix-vector products, X * w and X^T * e, run in parallel above an internal size gate. Neither solver carries any randomness. 2 runs on the same machine with the same data produce identical coefficients. The determinism test asserts bit-identical predictions across 2 independently constructed models. For the parallel-reduction machinery behind this, see 7.3. Performance Tuning and Parallelism.

2.1.2. Constructing a model

The constructor takes one argument and is infallible:

pub fn new(fit_intercept: bool) -> Self

There is nothing else to validate here, which is why new returns Self rather than Result. The iteration settings that used to sit on the estimator now live inside the solver variant that uses them. with_solver validates them instead.

LinearRegression::default() is the same as LinearRegression::new(true). It sets fit_intercept = true, selects LeastSquaresSolver::Normal, and applies no regularization. The two constructors always build the same algorithm now. default() is the analogue of Python’s LinearRegression(), and it is exact.

Two builder methods refine a constructed model. Both consume and return self, so they chain. Both return Result, because both validate what they receive:

pub fn with_solver(self, solver: LeastSquaresSolver) -> Result<Self, Error>
pub fn with_regularization(self, regularization: RegularizationType) -> Result<Self, Error>

with_solver checks the selected variant’s payload. A non-positive or non-finite learning_rate or tol, or a max_iter of 0, yields Error::InvalidParameter with the offending field’s name. LeastSquaresSolver::Normal has no payload, so selecting it can never fail. with_regularization validates the penalty coefficient alpha the same way. It rejects a negative or non-finite alpha on the spot. The regularization enum is re-exported at rustyml::machine_learning::RegularizationType:

pub enum RegularizationType {
    L1(f64), // Lasso: penalty alpha * sum(|w_j|)
    L2(f64), // Ridge: penalty (alpha / 2) * dot(w, w)
}

Every hyperparameter and fitted quantity is readable through a getter. The fitted ones return Option, None until fit has run:

There are no get_learning_rate, get_max_iterations, or get_tolerance accessors. Those numbers are fields of the solver variant. Read them back by matching on get_solver():

if let LeastSquaresSolver::GradientDescent { learning_rate, max_iter, tol } = model.get_solver() {
    println!("learning_rate = {learning_rate}, cap = {max_iter}, tol = {tol}");
}

There is deliberately no cost-history accessor either. The model stores the final iteration count, not the per-iteration cost curve. If you want to watch the cost descend live, build with the show_progress feature. Then fit renders a progress bar with the running cost and convergence counter. Otherwise get_actual_iterations() is the only window into what the optimizer did (see 2.1.7).

2.1.3. Fitting and predicting

The four data-facing methods are generic over ndarray’s storage type S: Data<Elem = f64>. This lets them accept owned arrays, views, and slices alike, without a copy:

pub fn fit<S1, S2>(&mut self, x: &ArrayBase<S1, Ix2>, y: &ArrayBase<S2, Ix1>) -> Result<&mut Self, Error>;
pub fn predict<S>(&self, x: &ArrayBase<S, Ix2>) -> Result<Array1<f64>, Error>;
pub fn fit_predict<S1, S2>(&mut self, x: &ArrayBase<S1, Ix2>, y: &ArrayBase<S2, Ix1>) -> Result<Array1<f64>, Error>;
pub fn score<S1, S2>(&self, x: &ArrayBase<S1, Ix2>, y: &ArrayBase<S2, Ix1>) -> Result<f64, Error>;

x is a feature matrix with one sample per row and one feature per column. y is the target vector. fit returns &mut Self for chaining. predict returns the prediction vector. fit_predict runs fit then predict on the same data. score returns the coefficient of determination, R^2 (1 - SS_res / SS_tot). A score of 1.0 is perfect. A score of 0.0 matches a model that always predicts the mean. A negative score is worse than the mean. The generic Fit and Predict traits expose the same operations for code that works across estimator types. Their trait methods just forward to these inherent methods.

fit rejects an empty x with Error::EmptyInput. It rejects a y whose length differs from x’s row count with Error::DimensionMismatch. It rejects any NaN or infinite entry in x with Error::NonFinite. predict and score also return Error::NotFitted when you call them before training. predict checks that the column count matches the training data, again with Error::DimensionMismatch. All of these come from the crate’s unified Error type. See 1.6. Error Handling.

A complete run on a small multivariate problem, y = 2 * x1 + 3 * x2 + 1. Nothing is configured beyond the intercept, so this is exact OLS:

use ndarray::array;
use rustyml::machine_learning::LinearRegression;

fn main() {
    // Six samples spanning the feature space.
    let x = array![
        [1.0, 1.0],
        [2.0, 1.0],
        [1.0, 2.0],
        [3.0, 2.0],
        [2.0, 3.0],
        [4.0, 1.0],
    ];
    let y = array![6.0, 8.0, 9.0, 13.0, 14.0, 12.0];

    // fit_intercept = true, closed-form solver, no regularization.
    let mut model = LinearRegression::new(true);
    model.fit(&x, &y).unwrap();

    // Coefficients land on [2.0, 3.0] and the intercept on 1.0, to machine precision.
    println!("coefficients = {:?}", model.get_coefficients().unwrap());
    println!("intercept    = {}", model.get_intercept().unwrap());
    println!("iterations   = {}", model.get_actual_iterations().unwrap()); // 0

    // Predict on unseen rows: (1,1) -> 6.0, (2,3) -> 14.0.
    let preds = model.predict(&array![[1.0, 1.0], [2.0, 3.0]]).unwrap();
    println!("predictions  = {:?}", preds);

    // R^2 on the training data, 1.0 for exactly-linear data.
    println!("R^2          = {}", model.score(&x, &y).unwrap());
}

The closed form lands on the least-squares optimum rather than approaching it. The recovered slopes match to machine precision. Gradient descent, by contrast, only converges toward that solution. The tests assert its slopes stay within 3e-3 of the true values.

2.1.4. Choosing a solver: the normal equation vs. gradient descent

LeastSquaresSolver is a payload-carrying enum. Each variant owns exactly the settings it uses. You cannot pass a learning rate to the closed form, and you cannot leave one unset for the iterative path.

pub enum LeastSquaresSolver {
    Normal, // the default
    GradientDescent { learning_rate: f64, max_iter: usize, tol: f64 },
}

LeastSquaresSolver::Normal is the right choice for small to medium dense problems. This is why it is the default. It is exact, needs no scaling, and has no hyperparameters to tune.

Prefer gradient descent in 3 cases. First, when the dataset is large enough that the closed-form factorization, roughly O(n * p^2), costs too much. Second, when you want L1, since the closed form has no L1 solution. Third, when you match a scikit-learn or Keras workflow that also uses iterative optimization.

The Normal solver supports only no regularization or L2. Pairing it with an L1 penalty makes fit return Error::InvalidInput, because Lasso has no closed form. Use gradient descent for L1.

Selecting gradient descent and configuring it are one expression, and the result is a Result, because the payload is validated:

use ndarray::array;
use rustyml::machine_learning::LinearRegression;
use rustyml::machine_learning::linear_model::LeastSquaresSolver;

fn main() {
    // y = 3*x0 - 2*x1 + 5, exactly.
    let x = array![
        [1.0, 1.0],
        [2.0, 0.0],
        [0.0, 3.0],
        [4.0, 2.0],
        [3.0, 1.0],
        [1.0, 4.0],
    ];
    let y = array![6.0, 11.0, -1.0, 13.0, 12.0, 0.0];

    // The exact solution, in one step.
    let mut exact = LinearRegression::new(true);
    exact.fit(&x, &y).unwrap();

    // The iterative solution: the settings travel with the variant that uses them.
    let mut iterative = LinearRegression::new(true)
        .with_solver(LeastSquaresSolver::GradientDescent {
            learning_rate: 0.01,
            max_iter: 10_000,
            tol: 1e-9,
        })
        .unwrap();
    iterative.fit(&x, &y).unwrap();

    println!("exact     = {:?}", exact.get_coefficients().unwrap());     // ~ [3.0, -2.0]
    println!("iterative = {:?}", iterative.get_coefficients().unwrap()); // approaches the same
    println!("iterations: exact = {:?}, iterative = {:?}",
        exact.get_actual_iterations(), iterative.get_actual_iterations());
}

The two solvers agree numerically when they solve the same objective. The gradient-descent cost divides the data term by 2 * n but scales the L2 penalty by only alpha / 2. This makes the equivalent ridge penalty on the raw sum of squares equal to lambda = n * alpha. That value is exactly what the normal solver uses. The integration tests confirm that the closed-form L2 solution matches what gradient descent converges to, for the same alpha.

2.1.5. Standardize your features first

This section is about gradient descent. The closed form sidesteps this whole concern, which is one more reason it is the default.

Gradient descent with a single global learning rate depends on how well your feature scales match. Suppose one column ranges over the thousands and another sits near unit magnitude. The loss surface then becomes a steep, narrow valley. A learning rate small enough to avoid overshooting on the steep axis crawls along the shallow one, so convergence takes far more iterations. A learning rate large enough to make progress on the shallow axis diverges on the steep one and trips the finiteness guard. Standardizing each column to zero mean and unit variance makes the curvature roughly isotropic. Then one learning rate serves every direction, and you can safely use a larger one.

StandardScaler does exactly this. It remembers the training mean and standard deviation, so it transforms the test set with the training statistics, not the test set’s own. See 4.2. Standardization and Normalization for the full train and test workflow, and for the stateless standardize free function. The coefficients you read back are then in standardized units, not raw feature units.

use ndarray::array;
use rustyml::machine_learning::LinearRegression;
use rustyml::machine_learning::linear_model::LeastSquaresSolver;
use rustyml::utils::StandardScaler;

fn main() {
    // Column 0 lives in the thousands. Column 1 is near unit scale.
    let x_raw = array![
        [1000.0, 1.0],
        [2000.0, 3.0],
        [3000.0, 2.0],
        [4000.0, 5.0],
        [5000.0, 4.0],
    ];
    let y = array![10.0, 23.0, 32.0, 45.0, 54.0];

    // Zero mean, unit variance per column. The scaler keeps the statistics for later batches.
    let mut scaler = StandardScaler::new();
    let x = scaler.fit_transform(&x_raw).unwrap();

    // Isotropic features tolerate a much larger step than the raw data would.
    let mut model = LinearRegression::new(true)
        .with_solver(LeastSquaresSolver::GradientDescent {
            learning_rate: 0.1,
            max_iter: 5_000,
            tol: 1e-9,
        })
        .unwrap();
    model.fit(&x, &y).unwrap();

    println!("iterations   = {}", model.get_actual_iterations().unwrap());
    println!("R^2          = {}", model.score(&x, &y).unwrap());
    // These slopes are in standardized units.
    println!("coefficients = {:?}", model.get_coefficients().unwrap());
}

As a rule of thumb for the step size, start around 0.1 on standardized data. Back off by a factor of 10 if fit returns Error::NonFinite or the iteration count pins to max_iter. On raw, small-magnitude integer features, a rate near 0.01 is usually safe.

2.1.6. Regularization: L1 vs L2

L2 (RegularizationType::L2(alpha), ridge) adds (alpha / 2) * dot(w, w) to the cost and alpha * w to the weight gradient. It shrinks every coefficient smoothly toward zero, without forcing any coefficient to exactly zero. This suits correlated features, where you want a stable, low-variance fit. Ridge spreads weight across collinear columns instead of letting one grow too large. Both solvers support L2.

L1 (RegularizationType::L1(alpha), lasso) adds alpha * sum(|w_j|) to the cost. The solver applies it with a proximal step. After each gradient step, it soft-thresholds every weight by learning_rate * alpha. Any weight the data cannot justify lands on exactly 0.0 and stays there. This method is called ISTA, and it is what makes L1 a feature selector. An older sub-gradient form added alpha * sign(w) to the gradient instead. That form only approaches zero over time, so it never produced true sparsity. Either way, the intercept stays unpenalized. L1 needs the gradient-descent solver, because no closed form exists for it.

Setting alpha correctly matters most when you port a model from Python. Every estimator that takes RegularizationType minimizes a mean data term plus an undivided penalty:

L1:  (1 / n) * sum(loss) + alpha * ||w||_1
L2:  (1 / n) * sum(loss) + alpha * 0.5 * ||w||^2

This matches scikit-learn’s SGDRegressor and SGDClassifier objective exactly, so alpha transfers 1:1 from either one. The closed-form estimators use their own conventions, and those conventions differ from each other:

Here n is the number of training samples. The Ridge row states the lambda = n * alpha relationship from 2.1.4, read in the other direction.

This snippet uses the exact solver, so the shrinkage is unambiguous. Ridge produces a strictly smaller coefficient norm than OLS on nearly-collinear data:

use ndarray::array;
use rustyml::machine_learning::{LinearRegression, RegularizationType};

fn main() {
    // Two nearly-collinear features make plain OLS coefficients large and unstable.
    let x = array![
        [1.0, 0.9],
        [2.0, 2.1],
        [3.0, 2.9],
        [4.0, 4.2],
        [5.0, 5.1],
    ];
    let y = array![1.0, 2.0, 3.0, 4.0, 5.0];

    // Both use the default closed-form solver.
    let mut ols = LinearRegression::new(true);
    ols.fit(&x, &y).unwrap();

    let mut ridge = LinearRegression::new(true)
        .with_regularization(RegularizationType::L2(1.0))
        .unwrap();
    ridge.fit(&x, &y).unwrap();

    let sq_norm =
        |m: &LinearRegression| m.get_coefficients().unwrap().iter().map(|c| c * c).sum::<f64>();
    println!("OLS   ||w||^2 = {}", sq_norm(&ols));
    println!("ridge ||w||^2 = {}", sq_norm(&ridge)); // strictly smaller
}

For L1, select gradient descent explicitly. The Normal solver rejects L1 with Error::InvalidInput. The crate’s own tests use 200 features, where column 0 carries the signal and the other 199 columns are pure noise. A small L1 penalty keeps the informative coefficient dominant and drives every noise coefficient to literal 0.0. Count the zeros with iter().filter(|c| **c == 0.0).count(). The tests assert exactly this behavior.

2.1.7. Convergence and diagnosing non-convergence

This section applies to LeastSquaresSolver::GradientDescent. The closed form has no notion of convergence. It either produces a solution or returns an error.

Convergence is not declared on a single small step. After each iteration, the optimizer compares the absolute change in cost against tol. A change below tol increments a counter. Training stops only after 3 consecutive iterations below tol. Any iteration whose cost change exceeds the threshold resets the counter to zero. This rule guards against a premature stop on a temporary plateau. The loop also stops unconditionally at max_iter.

After fit, compare get_actual_iterations() against the max_iter you configured. Read max_iter back by matching on get_solver(). If the actual count is strictly below the cap, the 3-in-a-row rule fired, and the model converged. If it equals the cap, the optimizer ran out of budget, and the fit may be underconverged.

The usual remedies, in order:

1. Standardize the features (see 2.1.5).
2. Raise max_iter.
3. Raise the learning rate for faster descent, but watch for divergence.
4. Loosen tol, if you do not need the last digit of precision.
5. Drop the with_solver call, and use the exact closed-form answer instead.

Divergence is the other failure mode. A learning rate that is too large sends the cost, and then the parameters, to infinity. The in-loop finiteness guard converts that into Error::NonFinite within a few iterations, instead of returning coefficients filled with inf. A NonFinite error from fit, on clean and finite data, almost always means the learning rate is too high, or the features need scaling. The model keeps no per-iteration cost history. To watch the descent directly, build with the show_progress feature. Then fit renders the running cost and the k/3 convergence counter on each iteration.

2.1.8. Inspecting the fitted parameters

The getters let you read the fitted model without running inference again. get_coefficients() returns Option<&Array1<f64>>, a borrow, because the model owns its weights. get_intercept() and get_actual_iterations() return owned Option values instead. The solver’s settings come back inside the enum. Read them with a match or an if let.

use ndarray::array;
use rustyml::machine_learning::LinearRegression;
use rustyml::machine_learning::linear_model::LeastSquaresSolver;

fn main() {
    let x = array![[1.0], [2.0], [3.0], [4.0], [5.0]];
    let y = array![3.0, 5.0, 7.0, 9.0, 11.0]; // y = 2x + 1

    let mut model = LinearRegression::new(true)
        .with_solver(LeastSquaresSolver::GradientDescent {
            learning_rate: 0.01,
            max_iter: 10_000,
            tol: 1e-10,
        })
        .unwrap();
    model.fit(&x, &y).unwrap();

    // Weights come back by reference, so iterate without moving them out of the model.
    let coefs = model.get_coefficients().unwrap();
    for (j, w) in coefs.iter().enumerate() {
        println!("w[{j}] = {w:.6}");
    }

    // Intercept and iteration count are owned values.
    println!("intercept       = {:.6}", model.get_intercept().unwrap());
    let ran = model.get_actual_iterations().unwrap();
    println!("iterations run  = {ran}");

    // The iteration settings live in the solver variant.
    match model.get_solver() {
        LeastSquaresSolver::Normal => println!("closed form, nothing to tune"),
        LeastSquaresSolver::GradientDescent { learning_rate, max_iter, tol } => {
            println!("learning_rate   = {learning_rate}");
            println!("tolerance       = {tol}");
            println!("max_iter        = {max_iter}");
            println!("converged early = {}", ran < max_iter);
        }
    }

    // Regularization is readable whether or not the model has been fitted.
    println!("regularization  = {:?}", model.get_regularization_type());
}

When fit_intercept is false, get_intercept() returns Some(0.0) after fitting, by contract. The intercept is fixed at zero, not merely small. Reading it back as exactly 0.0 confirms that no bias term was learned.

2.1.9. Saving and loading a model

A fitted model serializes to a compact postcard binary blob. The blob includes the coefficients, the intercept, the hyperparameters, and the training metadata. save_to_path(&self, path: &str) writes it. load_from_path(path: &str) -> Result<Self, Error> reads it back. Both methods report I/O and serialization failures as Error::Io. A round trip reproduces predictions exactly, because the raw f64 coefficients stay bit for bit the same.

use ndarray::array;
use rustyml::machine_learning::LinearRegression;

fn main() {
    let x = array![[1.0], [2.0], [3.0], [4.0], [5.0]];
    let y = array![3.0, 5.0, 7.0, 9.0, 11.0];

    let mut model = LinearRegression::new(true);
    model.fit(&x, &y).unwrap();
    let before = model.predict(&array![[6.0]]).unwrap();

    // Persist to disk, then reload into a fresh instance.
    model.save_to_path("lr_model.bin").unwrap();
    let loaded = LinearRegression::load_from_path("lr_model.bin").unwrap();
    let after = loaded.predict(&array![[6.0]]).unwrap();

    println!("before = {:?}, after = {:?}", before, after); // identical

    std::fs::remove_file("lr_model.bin").unwrap();
}

The serialized layout changed when the iteration settings moved into the solver variant. What used to be 3 loose fields is now 1 payload. A blob written by an older version will not load. Re-fit and re-save any persisted models.

The file extension carries no meaning. The format is always postcard binary, whether you name the file .bin, .dat, or anything else. LinearRegression also derives Clone and Debug. An in-process copy is a plain model.clone(), and {:?} prints the full struct. For full details on cross-model persistence, including neural-network weights, see 7.2. Model Persistence in Depth.

For the classification counterpart built on the same gradient-descent machinery, continue to 2.2. Logistic Regression. For more regression scores you can compute on a fitted model’s predictions, see 5.1. Regression Metrics.

2.2. Logistic Regression

LogisticRegression is RustyML’s linear classifier for binary problems. It shares most of its machinery with LinearRegression: a weight vector, an optional intercept, gradient descent, and optional L1 or L2 penalties. It replaces the squared-error objective with the logistic loss. It also passes the linear score through a sigmoid. The output then reads as a probability. This model corresponds to scikit-learn’s sklearn.linear_model.LogisticRegression, stripped to the essentials. RustyML supports one binary class boundary and plain full-batch gradient descent. It has no built-in multiclass support. It also expresses regularization strength directly, not as the inverse C.

2.2.1. What the model actually optimizes

Each sample gets a linear score z = w * x (plus a bias when the model fits an intercept). The sigmoid sigmoid(z) = 1 / (1 + e^-z) maps that score into (0, 1). RustyML reads that value as the probability of the positive class. Training minimizes the mean binary cross-entropy between those probabilities and the labels. The optimizer is plain full-batch gradient descent. Every iteration computes the gradient over the entire training set, (1/n) * X^T * (sigmoid(X * w) - y). It then takes one step of size learning_rate against that gradient.

This choice has 2 consequences in practice. First, there is no stochastic sampling and no random initialization. Weights start at exactly zero, and the update is deterministic. 2 fits on the same data and hyperparameters produce bit-identical weights (the test suite verifies this). Second, full-batch gradient descent is a first-order method with one global step size. It is far more sensitive to feature scaling and to learning_rate than the quasi-Newton solvers (lbfgs, liblinear) that scikit-learn defaults to. This is the main behavioral difference from those solvers. It drives the practical advice in 2.2.6.

RustyML evaluates the loss in the numerically stable log-sum-exp form, instead of taking the log of a sigmoid. The formula is max(z, 0) - z * y + ln(1 + e^-|z|). Large-magnitude logits therefore do not overflow the loss computation. The weights can still overflow, which section 2.2.6 covers as a real failure mode. The per-iteration logits, gradient, and loss run through RustyML’s parallel GEMV and deterministic reduction primitives above their size gates. Results are therefore reproducible on a given machine. See 7.3. Performance Tuning and Parallelism for more.

2.2.2. Constructing the model

2 constructors exist. LogisticRegression::default() gives a reasonable starting point. LogisticRegression::new(...) sets every hyperparameter and validates each one up front.

use rustyml::machine_learning::LogisticRegression;

fn main() {
    // Defaults: fit_intercept = true, lr = 0.01, max_iter = 100, tol = 1e-4, no penalty
    let _a = LogisticRegression::default();

    // Explicit: new(fit_intercept, learning_rate, max_iterations, tolerance)
    let _b = LogisticRegression::new(true, 0.1, 1000, 1e-6).unwrap();
}

new returns Result<Self, Error>. It rejects an invalid hyperparameter immediately with Error::InvalidParameter, instead of letting it fail later at fit time.

The defaults are deliberately conservative. learning_rate = 0.01 with only 100 iterations rarely converges, except on trivial, well-scaled data. Treat default() as a smoke test, not a production configuration. Most real fits need a larger learning_rate (after standardizing the data) and a max_iterations value in the thousands.

Every stored value has a getter. 2 of them are the model’s after-fit diagnostics:

get_actual_iterations() gives the convergence check. Training stops when the loss change falls below tolerance between iterations, or when it reaches max_iterations. If the returned count equals max_iterations, assume the model did not converge to the tolerance. Raise max_iterations, raise learning_rate, or standardize the inputs instead of trusting that boundary.

2.2.3. The label contract: strictly 0 and 1

fit takes a feature matrix x (rows are samples, columns are features) and a target vector y. Both must be f64 arrays with the same storage type. The label domain is exact. Every entry of y must be 0.0 or 1.0. RustyML rejects anything else, such as 0.5, 2.0, or -1.0, with Error::InvalidInput. This matches the label convention used by SVC and LinearSVC, which also predict 0.0 and 1.0. If your labels use a different encoding, map them to {0, 1} first. For string or categorical labels, encode them first with 4.3. Label Encoding. Pick which class is positive, the 1, on purpose. That choice defines what precision, recall, and the probability output mean.

use ndarray::array;
use rustyml::machine_learning::LogisticRegression;

fn main() {
    // Logical AND, encoded as {0.0, 1.0}
    let x_train = array![[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]];
    let y_train = array![0.0, 0.0, 0.0, 1.0];

    let mut model = LogisticRegression::new(true, 0.5, 500, 1e-7).unwrap();
    model.fit(&x_train, &y_train).unwrap();

    let preds = model.predict(&x_train).unwrap(); // Array1<f64> with values in {0.0, 1.0}
    println!("predictions: {:?}", preds);
    println!("iterations:  {:?}", model.get_actual_iterations());
}

When fit_intercept is true, the model prepends the bias as weight index 0. get_weights() then returns n_features + 1 values. When fit_intercept is false, it returns exactly n_features values. This indexing matters when you inspect the weights, because the regularizer treats the intercept specially (see below).

2.2.4. Predicting: hard labels versus probabilities

RustyML has 3 prediction entry points. The distinction between hard labels and probabilities is the part most people get wrong.

• predict(&x) -> Result<Array1<f64>, Error> returns hard class labels, 0.0 or 1.0. It thresholds the positive-class probability at 0.5.
• predict_proba(&x) -> Result<Array1<f64>, Error> returns the raw positive-class probability in (0, 1). It gives one value per sample, the output of the sigmoid.
• fit_predict(&mut self, x, y) runs fit, then predict on the same x, for a quick check on the training set.

predict is exactly predict_proba with a fixed >= 0.5 cutoff. The model has no threshold parameter. That fixed cutoff works for balanced problems, but the 0.5 boundary is a modeling choice, not a rule. When a false positive costs more or less than a false negative, or the classes are imbalanced, call predict_proba and set your own threshold.

use ndarray::array;
use rustyml::machine_learning::LogisticRegression;

fn main() {
    let x_train = array![[-3.0], [-2.0], [-1.0], [1.0], [2.0], [3.0]];
    let y_train = array![0.0, 0.0, 0.0, 1.0, 1.0, 1.0];

    let mut model = LogisticRegression::new(true, 0.3, 500, 1e-7).unwrap();
    model.fit(&x_train, &y_train).unwrap();

    let x_test = array![[-0.5], [0.5]];
    let proba = model.predict_proba(&x_test).unwrap(); // positive-class probability
    let default = model.predict(&x_test).unwrap();     // 0.5 threshold -> Array1<f64> of {0.0, 1.0}

    // A stricter operating point: only call positive when p >= 0.8
    let strict: Vec<i32> = proba.iter().map(|&p| if p >= 0.8 { 1 } else { 0 }).collect();

    println!("proba:   {:?}", proba);
    println!("default: {:?}", default);
    println!("strict:  {:?}", strict);
}

All 3 methods validate the input against the trained feature count, excluding the implicit bias column. Calling an unfitted model returns Error::NotFitted. A wrong number of columns returns Error::DimensionMismatch. A NaN or infinite entry returns Error::NonFinite. Pass features without a manual bias column. The model adds and removes that column internally to match how it trained.

2.2.5. Regularization

By default there is no penalty. This departs from scikit-learn, whose default is L2. Add a penalty with the builder method with_regularization. It consumes and returns the model, so it chains directly off new:

use ndarray::array;
use rustyml::machine_learning::{LogisticRegression, RegularizationType};

fn main() {
    let x = array![
        [-4.0, -3.0], [-3.0, -4.0], [-2.0, -1.0],
        [ 2.0,  1.0], [ 3.0,  4.0], [ 4.0,  3.0],
    ];
    let y = array![0.0, 0.0, 0.0, 1.0, 1.0, 1.0];

    let mut plain = LogisticRegression::new(true, 0.1, 2000, 1e-8).unwrap();
    plain.fit(&x, &y).unwrap();

    let mut ridge = LogisticRegression::new(true, 0.1, 2000, 1e-8)
        .unwrap()
        .with_regularization(RegularizationType::L2(5.0))
        .unwrap();
    ridge.fit(&x, &y).unwrap();

    // L2 norm of the feature weights, skipping the (unpenalized) bias at index 0
    let feature_norm = |m: &LogisticRegression| {
        m.get_weights().unwrap().iter().skip(1).map(|w| w * w).sum::<f64>()
    };
    println!("no penalty: {:.4}", feature_norm(&plain));
    println!("L2(5.0):    {:.4}", feature_norm(&ridge));
}

The f64 inside each variant is the penalty strength alpha. It must be non-negative and finite (alpha = 0 is accepted and equals no penalty). RegularizationType::L2(alpha) (ridge) adds alpha * 0.5 * ||w||^2 to the loss. It shrinks weights smoothly toward zero. RegularizationType::L1(alpha) (lasso) adds alpha * ||w||_1. It drives individual weights to exactly zero, giving a sparse model useful for feature selection.

That “exactly zero” result is literal. RustyML does not fold L1 into the gradient as alpha * sign(w). A sub-gradient step only approaches zero, so that approach gives no sparsity, however long it runs. Instead, the optimizer takes its ordinary gradient step and then applies a proximal step. Every feature weight is soft-thresholded by learning_rate * alpha, so a weight the data cannot support lands on 0.0 and stays there. This method is ISTA. It is what makes L1 an actual feature selector. Count the zeros with w.iter().skip(1).filter(|v| **v == 0.0).count().

3 implementation details change how you pick alpha:

• The intercept is never penalized. When the model fits an intercept, the penalty gradient starts at feature index 1, so the bias stays free to move. This is the standard, correct choice. The regularizer should not fight the model’s ability to shift its decision boundary. Skip index 0 when you compare weight norms, as the example does.
• The penalty is not divided by the sample count. The data term is the mean log-loss, but RustyML adds the penalty as the absolute value alpha * R(w). So alpha keeps a fixed meaning no matter how many rows the data has. Replicating a dataset leaves the regularized optimum unchanged (the test suite checks this invariance). Here, a larger alpha means more regularization, the opposite of scikit-learn’s inverse C.
• alpha transfers 1:1 from scikit-learn’s SGD estimators. It needs conversion from the rest. The objective above, mean data term plus undivided penalty, matches SGDClassifier exactly. An alpha tuned there moves over untouched. From LogisticRegression(C=c), use alpha = 1 / (c * n), where n is the number of training samples. The full conversion table lives on RegularizationType, shared with LinearRegression.

2.2.6. Standardization, class imbalance, and separable data

3 failure modes are common enough to name here.

Standardize your features. The optimizer is full-batch gradient descent with one global learning_rate. Features on very different scales therefore converge at very different rates. The step that suits a feature ranging over [0, 1] is far too small for one ranging over [0, 10000]. Training then crawls, and the loss plateaus before it reaches tolerance within max_iterations. Center and scale the data first with the tools in 4.2. Standardization and Normalization. This step gives the largest gain in both convergence speed and numerical stability. It also lets you use a useful learning_rate, such as 0.1 to 1.0, instead of the cautious default.

There is no class weighting. The model has no class_weight parameter, and predict fixes its threshold at 0.5. On imbalanced data, it can learn to predict only the majority class and still report a high accuracy. Use 2 defenses. Threshold predict_proba yourself, at an operating point chosen from a precision-recall or ROC analysis. Also evaluate with metrics that imbalance does not fool, such as balanced_accuracy, mcc, or roc_auc from 5.2. Classification Metrics, instead of raw accuracy.

Perfectly separable data makes the unregularized maximum likelihood estimate diverge. When a hyperplane separates the classes cleanly, the likelihood maximizes by pushing the weight norm toward infinity, since the probabilities saturate at 0 and 1. Unregularized gradient descent then keeps growing the weights the longer it runs. In practice, max_iterations and tolerance cut training off, and the classification stays correct. The weights, and so the probabilities, still become arbitrary in magnitude and poorly calibrated. A large learning_rate on large-magnitude separable inputs can also overflow the weight update completely. RustyML catches this with an in-loop Error::NonFinite guard, instead of silently returning NaN. Adding even a small L2 penalty bounds the optimum, keeps the probabilities meaningful, and removes the overflow risk. This is the main reason to keep a penalty on by default in production.

2.2.7. Polynomial features for non-linear boundaries

The decision boundary is linear in the feature space you give it. Non-linear problems therefore need a richer space. generate_polynomial_features(&x, degree) expands each row into all monomials up to degree. For 2 features at degree 2, this gives the 5 columns [x1, x2, x1^2, x1*x2, x2^2], with no constant column (the intercept already supplies that). Fit on the expansion, and predict on the same expansion.

use ndarray::array;
use rustyml::machine_learning::{LogisticRegression, generate_polynomial_features};

fn main() {
    // Inner ring = class 0, outer ring = class 1: not linearly separable in (x1, x2)
    let x = array![
        [ 1.0, 0.0], [0.0,  1.0], [-1.0, 0.0], [0.0, -1.0],
        [ 5.0, 0.0], [0.0,  5.0], [-5.0, 0.0], [0.0, -5.0],
    ];
    let y = array![0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0];

    // 2 features, degree 2 -> [x1, x2, x1^2, x1*x2, x2^2]
    let x_poly = generate_polynomial_features(&x, 2);
    assert_eq!(x_poly.ncols(), 5);

    let mut model = LogisticRegression::new(true, 0.01, 3000, 1e-7).unwrap();
    model.fit(&x_poly, &y).unwrap();

    // The x1^2 + x2^2 term makes the two rings linearly separable
    let preds = model.predict(&x_poly).unwrap();
    println!("{:?}", preds);
}

The column count grows combinatorially with both feature count and degree. 3 features at degree 3 already yields 19 columns. Use this tool only for a handful of features and a low degree. Reach for a kernel method, such as SVC, when the expansion gets large.

2.2.8. A worked example with evaluation

This example fits on a small 2-feature set, then evaluates it with the classification metrics from Chapter 5. predict already returns hard {0.0, 1.0} labels as an Array1<f64>. That is exactly what ConfusionMatrix::new requires. It thresholds nothing itself, so it rejects a probability vector instead of binarizing it silently. roc_auc works the other way. It wants a boolean truth vector alongside the continuous predict_proba scores, because ranking is the whole point.

use ndarray::{array, Array1};
use rustyml::machine_learning::LogisticRegression;
use rustyml::metrics::{ConfusionMatrix, accuracy, roc_auc};

fn main() {
    let x_train = array![
        [-2.0, -1.5], [-1.5, -2.0], [-1.0, -0.5], [-2.5, -1.0], [-0.5, -1.0],
        [ 2.0,  1.5], [ 1.5,  2.0], [ 1.0,  0.5], [ 2.5,  1.0], [ 0.5,  1.0],
    ];
    let y_train = array![0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0];

    let mut model = LogisticRegression::new(true, 0.5, 500, 1e-7).unwrap();
    model.fit(&x_train, &y_train).unwrap();

    // Already hard {0.0, 1.0} labels, which is what ConfusionMatrix::new requires
    let preds = model.predict(&x_train).unwrap();

    let cm = ConfusionMatrix::new(&y_train, &preds);
    println!("{}", cm.summary());
    println!("accuracy:  {:.3}", accuracy(&y_train, &preds));

    // ROC AUC ranks the probabilities, so it needs the raw scores, not the 0/1 labels
    let truth: Array1<bool> = y_train.mapv(|v| v > 0.5);
    let scores = model.predict_proba(&x_train).unwrap();
    println!("ROC AUC:   {:.3}", roc_auc(&truth, &scores));
}

ConfusionMatrix::summary() prints the counts alongside accuracy, balanced accuracy, precision, recall, specificity, F1, and MCC in one table. It gives a fast way to check a binary classifier. Feeding predict_proba into roc_auc, instead of the thresholded labels, makes the AUC a threshold-independent measure of ranking quality. It answers how well the model orders positives above negatives, regardless of where you later set the cutoff. The output above depends on the data. Here is the shape to expect:

Confusion Matrix:
+-----------------+--------------------+--------------------+
| ...             | Predicted Positive | Predicted Negative |
...
Performance Metrics:
- Accuracy:          <0.0..1.0>
- ...
accuracy:  <0.0..1.0>
ROC AUC:   <0.0..1.0>

2.2.9. Persistence and reproducibility

A trained model serializes to a compact postcard binary through save_to_path and load_from_path. This binary carries the weights, hyperparameters, and iteration count. A round-trip is byte-exact, so a loaded model’s predictions match the original’s predictions exactly.

use ndarray::array;
use rustyml::machine_learning::LogisticRegression;

fn main() {
    let x = array![[-2.0, 1.0], [-1.0, -1.0], [1.0, 1.0], [2.0, -1.0]];
    let y = array![0.0, 0.0, 1.0, 1.0];

    let mut model = LogisticRegression::new(true, 0.3, 500, 1e-7).unwrap();
    model.fit(&x, &y).unwrap();

    let path = "logreg_model.bin";
    model.save_to_path(path).unwrap();
    let loaded = LogisticRegression::load_from_path(path).unwrap();

    assert_eq!(model.predict(&x).unwrap(), loaded.predict(&x).unwrap());

    std::fs::remove_file(path).unwrap();
    println!("round-trip OK");
}

Fitting has no randomness anywhere, so reproducibility comes free. There is no seed to set, unlike the sampling-based models in this chapter. 2 fits on identical data and hyperparameters yield identical weights. This is what makes the persistence round-trip exact. See 7.1. Reproducibility and Random Seeds for where seeds do matter. See 7.2. Model Persistence in Depth for the serialization format and its cross-version limits. If you build with the show_progress feature, fit also renders a live progress bar with the running loss. This bar is a convenient way to watch for the non-convergence and divergence behaviors described above.

2.3. K-Nearest Neighbors

K-Nearest Neighbors (KNN) does almost no work at training time. Instead, it does most of its work at prediction time. RustyML implements it as KNN<T>, a generic classifier over any label type T that supports hashing and equality comparison.

The API resembles scikit-learn’s KNeighborsClassifier, with a few RustyML-specific differences. The label type is generic, not limited to integers. The distance metric is one DistanceCalculationMetric enum, shared with DBSCAN and the clustering metrics. Tie-breaking is deterministic, based on the order labels first appear. A separate entry point runs prediction in parallel.

2.3.1. Lazy Learning: Where the Cost Lives

KNN is the purest lazy learner in the crate. Its fit method does almost no learning. It validates the input, copies the training feature matrix, and encodes the labels into compact usize indices, which makes voting a cheap integer operation. Unlike logistic regression or a decision tree, KNN does not compress the training data into weights or a tree of splits. predict does all the real work.

This deferral has a real cost. There is no trained model to consult, so classifying one query means measuring its distance to every training row and keeping the k smallest. On the brute-force path, the distance stage costs O(n_train * n_test * d) for n_test queries against n_train training rows in d dimensions. Each query also needs an O(n_train) partial selection step to pull out the k nearest values. This step uses Quickselect through select_nth_unstable, not a full O(n_train log n_train) sort. Memory use is O(n_train * d), because the entire training set stays in memory for the life of the model. The training set is the model. You accept this tradeoff to get a non-parametric classifier with no training phase and no assumptions about the shape of the decision boundary.

Two consequences follow. First, prediction latency scales with the size of the training set. KNN that runs fast on a few thousand rows can become the bottleneck on a few hundred thousand. Second, geometry decides accuracy at query time. This is why the distance metric and feature scaling, covered later in this page, matter more here than in almost any other model.

2.3.2. Constructing a Classifier

The constructor takes only k. Every other setting has a default value. You set these values through chained builder methods:

// Core surface (from src/machine_learning/neighbors/knn.rs)
pub fn new(k: usize) -> Result<Self, Error>;                       // Err if k == 0
pub fn with_weighting_strategy(self, s: WeightingStrategy) -> Self;
pub fn with_metric(self, m: DistanceCalculationMetric) -> Result<Self, Error>; // validates Minkowski p

pub fn fit<S1, S2>(&mut self, x: &ArrayBase<S1, Ix2>, y: &ArrayBase<S2, Ix1>)
    -> Result<&mut Self, Error>;
pub fn predict<S>(&self, x: &ArrayBase<S, Ix2>) -> Result<Array1<T>, Error>;
pub fn predict_parallel<S>(&self, x: &ArrayBase<S, Ix2>) -> Result<Array1<T>, Error>; // T: Sync + Send
pub fn fit_predict<S1, S2>(&mut self, x: &..., y: &...) -> Result<Array1<T>, Error>;

new returns Error::InvalidParameter when k == 0. This is the one failure every caller can hit during construction. with_metric can also fail, because it validates the Minkowski order, covered in the next section. with_weighting_strategy cannot fail and returns Self directly. A fully specified builder chain therefore ends with ? or .unwrap() on the metric call. This matches the order used throughout the test suite.

The two enums that parameterize the model:

KNN::<T>::default() gives you k = 5, Uniform weighting, and Euclidean distance. These are the same defaults you get from calling new(5) and changing nothing else. Read the stored configuration back with get_k, get_weighting_strategy, get_metric, and get_x_train. get_x_train returns Option<&Array2<f64>>, which is None before you call fit.

The label type T is fully generic. Any type that is Clone + Hash + Eq works, so integer class codes, String labels, or your own enum all work. fit encodes the labels it sees into indices, in first-appearance order, and stores the reverse map. predict decodes the indices back to the original T. Feed in Array1<String>, and you get Array1<String> back. KNN is a classifier only. The crate has no KNN regressor. For neighbor-averaged regression, build it yourself on the distance primitives in Chapter 6.1.

Here is a full example. It uses both the sequential and parallel entry points:

use ndarray::array;
use rustyml::machine_learning::{DistanceCalculationMetric, KNN, WeightingStrategy};

fn main() {
    let x_train = array![
        [1.0, 2.0],
        [2.0, 3.0],
        [3.0, 4.0],
        [6.0, 6.0],
        [7.0, 7.0],
        [8.0, 8.0],
    ];
    let y_train = array![0, 0, 0, 1, 1, 1];

    let mut knn = KNN::new(3)
        .unwrap()
        .with_weighting_strategy(WeightingStrategy::Uniform)
        .with_metric(DistanceCalculationMetric::Euclidean)
        .unwrap();

    knn.fit(&x_train, &y_train).unwrap();

    let x_test = array![[1.5, 2.5], [7.5, 7.0]];
    let seq = knn.predict(&x_test).unwrap();
    let par = knn.predict_parallel(&x_test).unwrap();

    assert_eq!(seq, par); // deterministic: both paths agree exactly
    println!("k = {}", knn.get_k());
    println!("predictions: {:?}", seq);
}

fit checks for the mistakes that would otherwise cause a panic at predict time. It returns Error::EmptyInput for a zero-row x. It returns Error::NonFinite if x holds a NaN or an infinity. It returns Error::DimensionMismatch when y.len() differs from x.nrows(). It returns Error::InvalidInput when the training set has fewer samples than k. For example, you cannot ask for 5 neighbors from 3 points. predict and predict_parallel add Error::NotFitted when you call them before fit. They also return EmptyInput, DimensionMismatch for a query with the wrong feature count, and NonFinite for a query matrix with NaN or infinite values. See Error Handling for the full Error enum.

fit_predict fits the model, then predicts on the same training matrix. With k = 1, it returns the training labels unchanged. Each point’s nearest neighbor is itself, at distance zero. This makes fit_predict useful as a sanity check, but useless as an accuracy estimate. For a real estimate of generalization, hold out data with Train-Test Split and score it with the classification metrics.

2.3.3. Distance Metrics and the Minkowski Order

The metric decides what counts as nearest. RustyML exposes 3 metrics through one enum, shared across the library. Euclidean (L2) is the straight-line default. Manhattan (L1) sums the absolute differences between coordinates. Use Manhattan when features are independent axes measured in different units, or when you want robustness against a single outlying coordinate. Minkowski(p) generalizes both metrics. p = 1 reduces to Manhattan, and p = 2 reduces to Euclidean, exactly. The test suite asserts these equalities on shared data. Intermediate or larger values of p interpolate and extrapolate the shape of the unit ball.

with_metric validates the Minkowski order. It returns Error::InvalidParameter if p < 1 or p is not finite. This is a real constraint, not a style choice. Orders below 1 violate the triangle inequality, so the result is no longer a valid metric. Such an order would also break the pruning logic of the kd-tree index, described later in this page. The bare distance function minkowski_distance_row panics on p < 1. Routing through with_metric turns that panic into a recoverable Err you can handle. Minkowski(2.0) is legal, and numerically identical to Euclidean. Prefer the Euclidean variant when you want L2. Euclidean enables a matrix-multiply fast path, covered in section 2.3.7, that the general Minkowski code does not have.

use ndarray::array;
use rustyml::machine_learning::{DistanceCalculationMetric, KNN, WeightingStrategy};

fn main() {
    let x_train = array![[3.0, 0.0], [0.0, 4.0]];
    let y_train = array![0, 1];

    let mut knn = KNN::new(1)
        .unwrap()
        .with_weighting_strategy(WeightingStrategy::Uniform)
        .with_metric(DistanceCalculationMetric::Minkowski(3.0))
        .unwrap();
    knn.fit(&x_train, &y_train).unwrap();

    // Under L3: dist((0,3),(3,0)) = 54^(1/3) ~= 3.78 > dist((0,3),(0,4)) = 1
    let x_test = array![[0.0, 3.0]];
    println!("{:?}", knn.predict(&x_test).unwrap()); // nearest is (0,4) -> class 1
}

Chapter 6.1, Distance Metrics covers the metric abstraction in more depth, including the comparable-distance trick that lets the spatial index skip the final root.

2.3.4. Weighting Strategies and Tie-Breaking

After KNN finds the k neighbors, WeightingStrategy decides how their labels combine into one prediction.

Uniform is a plain majority vote. Each of the k neighbors contributes one vote to its class, and the class with the most votes wins. Distance weights each neighbor by 1.0 / distance, so a neighbor twice as close counts twice as much. Use distance weighting when k is large enough that the neighbor set reaches genuinely dissimilar points. The far points still vote, but their influence decays. Distance weighting also lowers the sensitivity of the result to the exact value of k.

Distance weighting has one edge case that the implementation handles explicitly. A query that coincides with a training point sits at distance zero, and 1.0 / 0.0 is infinity. To avoid that, the code checks for exact matches first. If any of the k neighbors sits at distance exactly 0.0, only those exact-match neighbors vote, by count, and KNN ignores the rest. This makes an exact hit behave like a lookup, which is almost always what you want.

use ndarray::array;
use rustyml::machine_learning::{DistanceCalculationMetric, KNN, WeightingStrategy};

fn main() {
    let x_train = array![[0.0, 0.0], [10.0, 0.0]];
    let y_train = array![0, 1];

    let mut knn = KNN::new(2)
        .unwrap()
        .with_weighting_strategy(WeightingStrategy::Distance)
        .with_metric(DistanceCalculationMetric::Euclidean)
        .unwrap();
    knn.fit(&x_train, &y_train).unwrap();

    // Both points are always in the k=2 set. The nearer one wins the weighted vote.
    let x_test = array![[1.0, 0.0], [9.0, 0.0]];
    println!("weighted: {:?}", knn.predict(&x_test).unwrap()); // [0, 1]

    // Exact match short-circuits the 1/0 problem: it votes by count, not weight.
    let x_exact = array![[0.0, 0.0]];
    println!("exact:    {:?}", knn.predict(&x_exact).unwrap()); // [0]
}

RustyML breaks ties with a deliberate, documented rule, not an arbitrary one. A tie happens when 2 classes have equal vote counts under Uniform, or equal summed weight under Distance. The winner is the class with the smallest encoded index. That index is not the smallest label value. It is the order in which fit first saw each label. For example, if your training targets show label 7 before label 3, 7 encodes to index 0 and wins ties against 3. Ties are broken deterministically and reproducibly, but the resolution depends on the order of the training rows. Reordering your data can flip a tied prediction. This same determinism is what lets predict and predict_parallel guarantee identical results.

2.3.5. Choosing k

k is the setting that changes behavior the most. It acts as a direct bias-variance dial. A small k, down to the extreme of k = 1, gives a low-bias, high-variance classifier. Its decision boundary hugs the data and follows every wrinkle, including mislabeled points and noise. A large k averages over a wider neighborhood. This lowers variance but raises bias. Push k far enough, and the model drifts toward always predicting the most common class overall. It then starts to erase small but real minority regions. A common starting point is a k near the square root of the training-set size. Tune it against a validation split. There is no substitute for measuring the result.

The classic advice to use an odd k for binary classification is about ties. RustyML’s tie-break is deterministic, so an even k never causes an error. A 50/50 split resolves by first-encounter order, but that resolution can feel arbitrary, and it depends on the order of your data. An odd k keeps binary votes from ever landing on a tie. Distance weighting also reduces this problem, because exact ties in summed real-valued weights are unlikely. Remember also the hard floor from section 2.3.2. fit rejects any k greater than the number of training samples.

This example makes the variance concrete. It places a single mislabeled point inside the class-0 region, then places a query right next to it. At k = 1, the noise wins. At k = 3 and k = 5, the surrounding genuine class-0 points outvote it:

use ndarray::array;
use rustyml::machine_learning::{DistanceCalculationMetric, KNN, WeightingStrategy};

fn main() {
    // Two clean clusters plus one mislabeled point at (2.5, 0): it sits inside
    // the class-0 region but carries the class-1 label.
    let x_train = array![
        [0.0, 0.0], [1.0, 0.0], [2.0, 0.0], [3.0, 0.0],     // class 0
        [10.0, 0.0], [11.0, 0.0], [12.0, 0.0], [13.0, 0.0], // class 1
        [2.5, 0.0],                                         // noise, class 1
    ];
    let y_train = array![0, 0, 0, 0, 1, 1, 1, 1, 1];

    let x_test = array![[2.4, 0.0]]; // right next to the noisy point

    for k in [1usize, 3, 5] {
        let mut knn = KNN::new(k)
            .unwrap()
            .with_weighting_strategy(WeightingStrategy::Uniform)
            .with_metric(DistanceCalculationMetric::Euclidean)
            .unwrap();
        knn.fit(&x_train, &y_train).unwrap();
        let pred = knn.predict(&x_test).unwrap();
        println!("k = {k}: prediction = {}", pred[0]);
    }
}

The prediction flips from the noisy label to the correct one as k grows:

k = 1: prediction = 1
k = 3: prediction = 0
k = 5: prediction = 0

This single-point sensitivity at k = 1 is the high-variance failure mode. Increasing k trades it for a smoother, more biased boundary.

2.3.6. Feature Scaling Is Not Optional

This mistake causes more failures than any other, so it gets its own section. KNN ranks neighbors by raw distance, and every metric here sums per-coordinate differences. Suppose one feature ranges over the thousands and another ranges over [0, 1]. The large-range feature then dominates the distance, and the small-range feature becomes invisible, regardless of which one actually carries the label. Unlike a linear model, which can learn a small coefficient for a large-scale feature, KNN has no coefficients to compensate. You must scale the features yourself before you call fit.

The example below encodes the label entirely in a small-range column. A large-range column carries no information about the label. On raw features, the large column decides the nearest neighbor, and the prediction is wrong. Standardizing with the training set’s per-column mean and standard deviation, applied to both train and query data, puts the informative column on equal footing. The prediction is then right:

use ndarray::{array, Axis};
use rustyml::machine_learning::KNN;

fn main() {
    // Column 0 spans the ~1000s and is uninformative. Column 1 in {0, 10} carries the label.
    let x_train = array![
        [1000.0, 0.0],  // class 0
        [3000.0, 0.0],  // class 0
        [1050.0, 10.0], // class 1
        [3050.0, 10.0], // class 1
    ];
    let y_train = array![0, 0, 1, 1];

    // Column 1 = 9.0 says class 1. Column 0 = 1010.0 is closest to a class-0 row.
    let x_test = array![[1010.0, 9.0]];

    let mut raw = KNN::new(1).unwrap();
    raw.fit(&x_train, &y_train).unwrap();
    let raw_pred = raw.predict(&x_test).unwrap();

    // Standardize with statistics computed on the TRAINING set, applied to both.
    let mean = x_train.mean_axis(Axis(0)).unwrap();
    let std = x_train.std_axis(Axis(0), 0.0);
    let x_train_s = (&x_train - &mean) / &std;
    let x_test_s = (&x_test - &mean) / &std;

    let mut scaled = KNN::new(1).unwrap();
    scaled.fit(&x_train_s, &y_train).unwrap();
    let scaled_pred = scaled.predict(&x_test_s).unwrap();

    println!("raw features:  {:?}", raw_pred);    // dominated by column 0 -> [0]
    println!("standardized:  {:?}", scaled_pred); // respects column 1  -> [1]
}

The example scales the data by hand, to stay self-contained. It still follows correct statistical practice. The mean and standard deviation come from the training data only, then apply to the query. The code never re-estimates them on the test set. In a real pipeline, use the crate’s standardize helper, or normalize, instead of writing this by hand. Fit the transform on the training set, then apply that same transform to new data. Fitting the transform again on the test set leaks information. Standardization to zero mean and unit variance is the usual choice. Min-max normalization is the alternative when you need features bounded to a fixed range.

2.3.7. Sequential vs Parallel Prediction, the kd-tree, and the Euclidean Fast Path

RustyML gives you 2 prediction entry points. predict runs sequentially and works for any label type. predict_parallel spreads the per-query work across a Rayon pool. Use predict_parallel for large query batches. It requires T: Sync + Send. Both methods build any shared index once, up front, on a single thread, before the queries run. predict_parallel then parallelizes over the test rows. The tie-break is deterministic, so the two paths return bit-identical label arrays. The test suite checks this across Uniform, Distance, and large-k configurations. You can develop with predict, then switch to predict_parallel for throughput, without changing a single result.

Under the hood, the search takes one of 2 routes. In low dimensions, at most 8 features, predict builds a kd-tree over the training data on first use, and caches it. The kd-tree gives average-case neighbor lookups that beat scanning every row. Above 8 features, the tree stops pruning effectively. This is the curse of dimensionality, where nearly every point becomes roughly equidistant from every other. Above that limit, the code falls back to a brute-force scan, and the full O(n_train * n_test * d) cost from section 2.3.1 applies. This 8-feature ceiling comes from calibration on one data shape, not from a universal law. Clustered data and different dataset sizes shift the actual crossover point. It remains the fixed threshold that the current implementation uses.

The brute-force Euclidean case gets a dedicated optimization. Squared Euclidean distance expands to ||x||^2 + ||t||^2 - 2 * x . t. The only per-pair term left is the dot product x . t, which is a matrix multiply. RustyML precomputes the training rows’ squared norms once and shares them across every query. It then computes the cross terms through the gemmkit matrix-multiply backend. This backend blocks the work to keep it cache-resident on large training sets. It also switches between a per-row GEMV swarm and a tiled GEMM, depending on whether the training matrix still fits in the shared L3 cache. Manhattan and Minkowski have no such algebraic shortcut. They fall back to a plain per-pair metric scan. This is another reason to prefer the Euclidean variant when you want L2. The kd-tree rebuilds lazily, and KNN drops it whenever you call fit again. A refitted model therefore never serves stale neighbors. See Performance Tuning and Parallelism for more on the parallelism gates and tuning.

2.3.8. Persistence

KNN<T> serializes with save_to_path and load_from_path when T is Serialize + Deserialize, as i32 and String are. Persistence writes the compact postcard binary format, regardless of the file extension you choose. It stores exactly what defines the model: k, the weighting strategy, the metric, the training matrix, and the label encoding. The kd-tree is not serialized. It is marked #[serde(skip)] and rebuilds lazily on the loaded model’s first predict call. A reloaded classifier therefore produces predictions identical to the original, with no extra work from you.

use ndarray::array;
use rustyml::machine_learning::{DistanceCalculationMetric, KNN};

fn main() {
    let x_train = array![
        [0.0, 0.0], [1.0, 0.0], [2.0, 0.0],
        [10.0, 0.0], [11.0, 0.0], [12.0, 0.0],
    ];
    let y_train = array![0, 0, 0, 1, 1, 1];

    let mut knn = KNN::new(3)
        .unwrap()
        .with_metric(DistanceCalculationMetric::Manhattan)
        .unwrap();
    knn.fit(&x_train, &y_train).unwrap();

    let path = "knn_model.bin";
    knn.save_to_path(path).unwrap();

    // Rebuilds its kd-tree lazily on first predict. k, metric, and labels are restored.
    let loaded = KNN::<i32>::load_from_path(path).unwrap();

    let x_test = array![[0.5, 0.0], [11.5, 0.0]];
    assert_eq!(
        knn.predict(&x_test).unwrap(),
        loaded.predict(&x_test).unwrap()
    );
    println!("round-trip predictions match");

    std::fs::remove_file(path).unwrap();
}

A KNN model carries its entire training set, so the serialized file grows with n_train * d. Persistence here saves your data plus a little metadata, not a handful of learned parameters. If model size matters to you, that alone is a reason to consider a parametric classifier instead. Model Persistence in Depth covers the format and its guarantees.

KNN<T> also implements the crate’s shared Fit and Predict traits. These traits are re-exported from machine_learning, and defined in crate::traits. Fit takes the training data as an (x, y) tuple. You will normally call the inherent fit, predict, and predict_parallel methods shown throughout this page. The traits exist so that generic code can treat every estimator the same way.

2.4. Decision Trees

A decision tree splits the feature space into axis-aligned boxes with a chain of if/else tests. Each box gets one constant prediction: a class label (or class distribution) for classification, or the mean target for regression. RustyML packs this into one DecisionTree type. You choose the algorithm with the Algorithm parameter: ID3, C45, or CART.

In scikit-learn, you pick DecisionTreeClassifier or DecisionTreeRegressor, then pick a criterion with a criterion= string. RustyML makes the algorithm the top-level choice instead. Each Algorithm variant bundles the impurity measure, the split-selection rule, and the categorical-split policy together.

A numeric feature always splits on a binary threshold. A sample with feature <= t goes left. The algorithm changes only how RustyML scores thresholds and whether categorical columns get multi-way branches.

DecisionTree lives under rustyml::machine_learning. It follows the new -> fit -> predict contract from Classical Machine Learning.

2.4.1. Choosing an algorithm: ID3, C4.5, CART

The Algorithm enum has exactly 3 variants. The differences are not cosmetic. Each variant changes what the tree can do and how it scores a split.

This table gives 2 consequences. First, only CART supports regression. DecisionTree::new(Algorithm::ID3, false) and DecisionTree::new(Algorithm::C45, false) fail right away with Error::InvalidInput, not at fit time. The constructor is the fast-fail gate for this check.

Second, the C4.5 gain ratio fixes a bias in ID3 toward high-cardinality features. Information gain favors a feature with many distinct values. In the extreme, a unique-per-row ID column scores a perfect gain but does not generalize at all. C4.5 divides the gain by the split’s intrinsic information to penalize a split that fans out into many thin branches. When the intrinsic information falls to zero (a single-branch partition), C4.5 rejects the split. Gain ratio is the safer default when a dataset mixes categorical columns of very different cardinality.

For plain numeric data with no categorical columns, all 3 algorithms reduce to the same greedy binary-threshold search. They tend to agree on the result. Use CART unless you need entropy-based scoring or multi-way categorical branches.

2.4.2. Classification: construct, fit, predict

The classifier path is new(algorithm, true). Labels must be non-negative integers encoded as f64. They must be dense from 0. RustyML infers the number of classes as max(label) + 1. A label set like {0, 5} allocates 6 classes, with 4 of them empty and unused. Encode your labels to 0..k-1 before you fit (see Label Encoding).

use rustyml::machine_learning::{Algorithm, DecisionTree};
use ndarray::array;

fn main() {
    // 2 features, 3 classes, split cleanly along feature 0
    let x = array![
        [0.0, 0.0],
        [0.1, 0.0],
        [0.2, 0.1],
        [10.0, 1.0],
        [10.1, 1.0],
        [20.0, 2.0],
        [20.1, 2.0],
    ];
    let y = array![0.0, 0.0, 0.0, 1.0, 1.0, 2.0, 2.0];

    let mut tree = DecisionTree::new(Algorithm::CART, true)
        .unwrap()
        .with_max_depth(5)
        .with_random_state(42);
    tree.fit(&x, &y).unwrap();

    let x_test = array![[0.05, 0.0], [10.2, 1.0], [20.2, 2.0]];
    let labels = tree.predict(&x_test).unwrap(); // Array1<f64>, 1 label per row
    let proba = tree.predict_proba(&x_test).unwrap(); // Array2<f64>

    println!("labels: {:?}", labels);
    println!("proba shape: {:?}", proba.shape()); // [3, 3] = (n_samples, n_classes)
    println!("n_classes: {:?}", tree.get_n_classes()); // Some(3)
}

predict returns an Array1<f64> of labels. predict_proba returns an Array2<f64>. Each row is the leaf’s empirical class distribution, so it sums to 1.0. The argmax of each row agrees with the matching predict label.

On a pure leaf, the distribution is one-hot. On an impure leaf, it holds the exact class frequencies of the training samples that reached it. An impure leaf appears only when you stop growth early (see the next section).

DecisionTree also has 2 single-sample methods: predict_one(&[f64]) -> f64 and predict_proba_one(&[f64]) -> Vec<f64>. fit_predict fits the tree, then predicts on the same training matrix in 1 call.

Calling predict_proba on a regression tree is a runtime error. It returns Error::Tree(TreeError::NotClassificationTree).

2.4.3. Regression with CART

Set is_classifier = false to switch the tree to MSE impurity. MSE impurity is the population variance of a node’s targets. Each leaf predicts the mean of its training targets.

This rule explains the staircase output of a regression tree. Predictions are piecewise-constant, with 1 plateau per leaf. A regression tree never extrapolates beyond the training range.

use rustyml::machine_learning::{Algorithm, DecisionTree};
use ndarray::array;

fn main() {
    // Step function: the target jumps from 1 to 10 somewhere between x = 2 and x = 10
    let x = array![[0.0], [1.0], [2.0], [10.0], [11.0], [12.0]];
    let y = array![1.0, 1.0, 1.0, 10.0, 10.0, 10.0];

    // Only CART supports regression
    let mut tree = DecisionTree::new(Algorithm::CART, false).unwrap();
    tree.fit(&x, &y).unwrap();

    // Each query lands in a leaf and gets that leaf's mean target
    let preds = tree.predict(&array![[1.5], [11.5]]).unwrap();
    println!("{:?}", preds); // [1.0, 10.0], the 2 leaf means
}

A non-CART regressor request fails at construction time. This check is the only place RustyML validates the algorithm and task combination eagerly:

// ID3 and C4.5 are classification-only, so this fails fast:
let err = DecisionTree::new(Algorithm::ID3, false).unwrap_err();
// -> Error::InvalidInput("Only CART algorithm is supported for regression tasks")

2.4.4. Hyperparameters and overfitting control

A tree grown with no limits keeps splitting until every leaf is pure (classification) or holds a single sample (regression). That tree memorizes the training set, noise included. This is the classic decision-tree failure mode.

The 5 growth parameters below are the full toolbox for trading training fit against generalization. Four of them are pre-pruning stopping rules that apply during growth. RustyML does no post-pruning. It has no ccp_alpha-style cost-complexity pruning, so all overfitting control happens before training starts.

The bounded setters return Result. Use .unwrap() or ? after them in a chain. with_max_depth and with_random_state return Self directly, with no Result.

One rule spans both min_samples_leaf and min_samples_split. Since you set them independently, no single builder can enforce it. min_samples_leaf must not exceed min_samples_split. RustyML checks this constraint at fit time and returns Error::InvalidParameter on a bad pairing. A bad pairing fails when you train, not when you build the tree.

Each parameter rejects a candidate split for a different reason:

• max_depth caps the number of edges on any root-to-leaf path. Some(0) forces the root to be a leaf. That leaf predicts the global majority class (or global mean), a useful sanity baseline.
• min_samples_split stops a node from splitting when it holds fewer samples than this value. It ends recursion early and yields a shallower tree.
• min_samples_leaf constrains the split search, not just the final pick. RustyML never considers a threshold that would leave a child below this minimum. Instead of collapsing the node into a leaf, the tree falls back to the best split whose children both satisfy the floor. This matches scikit-learn semantics. It is the parameter people misread most often: a rare category or a lone outlier does not throw away an otherwise good split.
• min_impurity_decrease rejects a split whose impurity decrease, scaled by N_t / N_total, falls below the threshold. N_t / N_total is the fraction of all training samples that reach the node. This node-weight scaling follows the scikit-learn convention. A large impurity drop deep in the tree, where few samples remain, counts for less than the same drop near the root.

The next example contrasts an overfit tree with a constrained one, on data with 2 deliberately mislabeled points. The unconstrained tree grows extra depth to isolate the noise and reaches 100% training accuracy. The constrained tree stays shallow and keeps every leaf populated, so it does not fit the noise:

use rustyml::machine_learning::{Algorithm, DecisionTree, Node, NodeType};
use ndarray::{array, Array1, Array2};

// Longest root-to-leaf path in edges (a bare leaf is depth 0)
fn depth(node: &Node) -> usize {
    match &node.node_type {
        NodeType::Leaf { .. } => 0,
        NodeType::Internal { .. } => {
            let mut d = 0;
            if let Some(l) = &node.left {
                d = d.max(depth(l));
            }
            if let Some(r) = &node.right {
                d = d.max(depth(r));
            }
            if let Some(children) = &node.children {
                for c in children.values() {
                    d = d.max(depth(c));
                }
            }
            1 + d
        }
    }
}

fn train_accuracy(tree: &DecisionTree, x: &Array2<f64>, y: &Array1<f64>) -> f64 {
    let preds = tree.predict(x).unwrap();
    let correct = preds
        .iter()
        .zip(y.iter())
        .filter(|(p, t)| (*p - *t).abs() < 0.5)
        .count();
    correct as f64 / y.len() as f64
}

fn main() {
    // Underlying rule: x <= 4 -> class 0, x >= 5 -> class 1.
    // 2 labels violate it: x = 1 and x = 8 are flipped noise.
    let x = array![[0.0], [1.0], [2.0], [3.0], [4.0], [5.0], [6.0], [7.0], [8.0], [9.0]];
    let y = array![0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 1.0];

    // Unconstrained: grows until pure, memorizing the noise
    let mut overfit = DecisionTree::new(Algorithm::CART, true).unwrap();
    overfit.fit(&x, &y).unwrap();

    // Constrained: 1 split deep, every leaf must retain >= 2 samples
    let mut constrained = DecisionTree::new(Algorithm::CART, true)
        .unwrap()
        .with_max_depth(1)
        .with_min_samples_leaf(2)
        .unwrap();
    constrained.fit(&x, &y).unwrap();

    println!(
        "unconstrained: depth {}, train acc {:.3}",
        depth(overfit.get_root().unwrap()),
        train_accuracy(&overfit, &x, &y)
    );
    println!(
        "constrained:   depth {}, train acc {:.3}",
        depth(constrained.get_root().unwrap()),
        train_accuracy(&constrained, &x, &y)
    );
}

The unconstrained tree reports a deeper structure and perfect training accuracy. The constrained tree has a single split, with 2 errors it deliberately keeps. On unseen data, the shallow tree is the one you want.

Real tuning sweeps these parameters against a validation split (see Train-Test Split). Score the result with the classification metrics.

2.4.5. Categorical features, missing values, and constant columns

RustyML has no separate categorical dtype. Every feature is an f64 column. You declare which columns hold discrete category codes with set_categorical_features(vec![...]). This is a &mut self setter. Call it between new and fit.

Under ID3 or C45, a declared column splits multi-way, with 1 child branch per distinct value. A multi-way split solves a pattern that a single numeric threshold cannot, such as “class 1 only when the code equals 1”. CART is binary by construction and ignores the declaration completely. On a CART tree, marking a column categorical does nothing. RustyML gives no warning.

use rustyml::machine_learning::{Algorithm, DecisionTree};
use ndarray::array;

fn main() {
    // 1 categorical feature (codes 0/1/2). Class depends on the code, not on any
    // single numeric cut: 0 -> class 0, 1 -> class 1, 2 -> class 0.
    let x = array![[0.0], [0.0], [1.0], [1.0], [2.0], [2.0]];
    let y = array![0.0, 0.0, 1.0, 1.0, 0.0, 0.0];

    let mut tree = DecisionTree::new(Algorithm::C45, true).unwrap();
    tree.set_categorical_features(vec![0]); // treat column 0 as categorical
    tree.fit(&x, &y).unwrap();

    // Each distinct code becomes its own branch, so the pattern is learned exactly
    println!("train preds: {:?}", tree.predict(&x).unwrap());

    // A category never seen in training routes to the node's fallback leaf: no error
    println!("unseen code 99 -> {:?}", tree.predict(&array![[99.0]]).unwrap());
}

RustyML canonicalizes category values by rounding to 6 decimals. 1.0000001 and 1.0000002 collapse into the same branch, while 1.0 and 2.0 stay distinct. Encode your codes as clean integers stored in f64, to avoid surprises.

At predict time, an unseen category cannot match any branch. It falls through to a stored fallback leaf, the majority prediction of the parent node’s training samples. You always get a valid prediction, never an error.

Even under ID3 or C45, min_samples_leaf still guards categorical splits. RustyML keeps a split as long as at least 2 branches each meet the leaf floor. One rare category with too few samples does not discard the whole multi-way split.

Missing values. RustyML has no NaN routing. fit and predict both check every input for finiteness. Any NaN or infinity in the feature matrix returns Error::NonFinite. Impute or drop missing entries before training (see Data Preprocessing).

Constant columns. A constant column is harmless. A numeric split is valid only between 2 distinct feature values. A column that never varies gives no candidate threshold, so RustyML skips it. A declared categorical column with fewer than 2 distinct values likewise gives no split. A constant feature costs a little search time but never corrupts the tree.

2.4.6. Inspecting the fitted tree

A fitted tree is easy to inspect. generate_tree_structure() returns a ready-to-print ASCII rendering of the tree: splits, thresholds, leaf classes, and probability vectors. It returns Error::NotFitted when you call it before training.

use rustyml::machine_learning::{Algorithm, DecisionTree};
use ndarray::array;

fn main() {
    let x = array![[0.0], [1.0], [2.0], [3.0]];
    let y = array![0.0, 0.0, 1.0, 1.0];

    let mut tree = DecisionTree::new(Algorithm::CART, true).unwrap();
    tree.fit(&x, &y).unwrap();

    print!("{}", tree.generate_tree_structure().unwrap());
    println!("features seen: {}", tree.get_n_features());
}

For programmatic inspection, get_root() -> Option<&Node> returns the raw tree. Node is public, with fields node_type, left, right, and children (an AHashMap<String, Box<Node>> for multi-way categorical nodes). NodeType is either Internal { feature_index, threshold, categories } or Leaf { value, class, probabilities }.

The depth helper in 2.4.4 walks this structure. You can walk it too, to extract feature-importance statistics or export the tree to another format.

The remaining getters are plain accessors. They are get_algorithm(), get_is_classifier(), get_n_features(), get_n_classes() (None for regression), get_parameters() (a Copy DecisionTreeParams), and get_categorical_features().

2.4.7. Determinism and seeding

Growth is greedy and deterministic, with one exception. When 2 or more candidate splits tie at the exact same selection score, the tree must break the tie.

With random_state = None and no crate-wide seed set, tie-breaking is deterministic. The last tied candidate wins. Repeated fits on the same data give a bit-for-bit identical tree. You do not need a seed for reproducibility, even on data with ties.

Set with_random_state(seed) to pick a uniformly random tied candidate from a seeded stream instead. The same seed reproduces the same tree. Different seeds may pick different, equally-scoring features.

use rustyml::machine_learning::{Algorithm, DecisionTree};
use ndarray::array;

fn main() {
    // Features 0 and 1 are identical columns, so their best splits tie exactly
    let x = array![[0.0, 0.0], [0.0, 0.0], [1.0, 1.0], [1.0, 1.0]];
    let y = array![0.0, 0.0, 1.0, 1.0];

    let fit_seeded = |seed: u64| {
        let mut t = DecisionTree::new(Algorithm::CART, true)
            .unwrap()
            .with_random_state(seed);
        t.fit(&x, &y).unwrap();
        t.generate_tree_structure().unwrap()
    };

    // Same seed -> identical tree, even though ties are broken at random
    assert_eq!(fit_seeded(7), fit_seeded(7));
    println!("seed 7 is reproducible");
}

random_state = Some(seed) uses that seed on its own and ignores any global seed. A tree left at random_state = None instead draws its tie-breaking randomness from the thread-local global stream, when one is active. Call set_global_seed(s) (paired with clear_global_seed()) to make a whole pipeline of unseeded models reproducible together.

Reproducibility and Random Seeds covers the mechanics and the reasoning behind routing all randomness through one seed.

2.4.8. Errors

Tree-specific failures live in the TreeError enum, re-exported as rustyml::machine_learning::TreeError. You reach it through the crate-wide Error::Tree variant. It is #[non_exhaustive] and has 2 members.

NotClassificationTree returns when you call predict_proba or predict_proba_one on a regression tree. CorruptStructure(&'static str) guards an invariant violation. A normally fitted and used model never triggers it. It protects tree traversal against a hand-built or otherwise broken node graph.

Every other error comes from the shared error surface in Error Handling:

• InvalidInput: bad labels, too few samples, or zero features.
• InvalidParameter: an out-of-range builder value, or the min_samples_leaf > min_samples_split cross-check.
• NonFinite: NaN or infinity in features.
• NotFitted: returned before training.
• EmptyInput: empty training or prediction data.
• DimensionMismatch { expected, found }: a prediction matrix with the wrong feature count.

use rustyml::machine_learning::{Algorithm, DecisionTree, TreeError};
use rustyml::error::Error;
use ndarray::array;

fn main() {
    let x = array![[0.0], [1.0], [2.0], [10.0], [11.0], [12.0]];
    let y = array![1.0, 1.0, 1.0, 10.0, 10.0, 10.0];

    let mut reg = DecisionTree::new(Algorithm::CART, false).unwrap();
    reg.fit(&x, &y).unwrap();

    // A regression tree has no class probabilities to report
    match reg.predict_proba(&x) {
        Err(Error::Tree(TreeError::NotClassificationTree)) => {
            println!("predict_proba is classification-only");
        }
        other => panic!("unexpected: {:?}", other),
    }
}

2.4.9. Persistence

A fitted DecisionTree serializes with save_to_path and load_from_path. These methods write and read the compact postcard binary format through serde. The whole structure round-trips, including the AHashMap children of multi-way categorical nodes. A loaded model reproduces predictions exactly.

You can name the file anything you want. The content is binary regardless of the file extension.

use rustyml::machine_learning::{Algorithm, DecisionTree};
use ndarray::array;

fn main() {
    let x = array![[0.0], [1.0], [2.0], [10.0], [11.0], [12.0]];
    let y = array![0.0, 0.0, 0.0, 1.0, 1.0, 1.0];

    let mut tree = DecisionTree::new(Algorithm::CART, true).unwrap();
    tree.fit(&x, &y).unwrap();

    tree.save_to_path("dt_model.bin").unwrap();
    let loaded = DecisionTree::load_from_path("dt_model.bin").unwrap();

    // Predictions survive the round trip bit-for-bit
    assert_eq!(tree.predict(&x).unwrap(), loaded.predict(&x).unwrap());
    println!("round trip OK");

    std::fs::remove_file("dt_model.bin").unwrap();
}

A failed read, a failed write, or a corrupt payload returns Error::Io. See Model Persistence in Depth for the format’s guarantees and version considerations.

2.4.10. Complexity and parallelism

Growing a node sorts each feature’s values once, then sweeps them with running impurity statistics. A node over n samples costs O(n_features * n log n). For a reasonably balanced tree, this compounds to roughly O(n_features * n log^2 n) overall. This is the standard cost of a CART-style learner. It is why a wide dataset (many features) dominates the training budget.

Prediction is a root-to-leaf walk, at O(depth) per sample. The trained tree does not store its own depth. The parallel gate instead assumes a walk of about 16 nodes as a stand-in value.

RustyML runs 2 independent rayon parallelizations, once the work clears a calibrated gate. During fit, the per-feature split search runs in parallel when n_samples * n_features clears the sort-scan gate. During predict or predict_proba, per-sample traversal runs in parallel when the sample count clears the tree-traversal gate.

A small problem stays single-threaded, to avoid parallel overhead. You do nothing to opt in. These gates pick a strategy only. They never change the result. Performance Tuning and Parallelism covers how to tune the thresholds.

2.5. Support Vector Machines

A support vector machine finds the decision boundary that sits as far as possible from the nearest points of each class. This is the maximum-margin hyperplane. RustyML ships 2 implementations that reach this goal from opposite directions.

SVC is the kernelized version. It lifts the data into a higher-dimensional space through a kernel function and finds a linear boundary there. Back in the original space, that boundary becomes curved. LinearSVC skips the kernel. It fits a straight hyperplane by minimizing hinge loss with stochastic gradient descent.

Both models are strictly binary classifiers, with no built-in one-vs-rest wrapper. Both use the same {0.0, 1.0} label domain, so you can swap one for the other without a rewrite of the target array. Past that shared goal, the 2 models differ in solver, regularization, and cost. The rest of this page helps you pick the right one for a problem.

2.5.1. When to reach for which

The choice comes down to 2 numbers: the sample count (n_samples) and the feature count (n_features). It also depends on whether a straight line can separate the classes.

SVC builds an n_samples x n_samples kernel matrix, also called the Gram matrix, before training starts. The Sequential Minimal Optimization (SMO) solver then works over that matrix. The matrix sets a hard limit: it costs O(n_samples^2) in memory and O(n_samples^2 * n_features) to build. At 10,000 samples, the Gram matrix alone takes about 800 MB of f64 values. At 260,000 samples, it would need hundreds of gigabytes.

SVC fits well when the boundary is truly nonlinear and n_samples stays modest, from hundreds to a few thousand rows. It does not scale to large datasets. That limit comes from the algorithm, not from this implementation.

LinearSVC never forms a kernel matrix. Each epoch runs a few matrix-vector passes over the data. That costs O(n_samples * n_features) in time and only O(n_features) in memory for the weight vector. It scales linearly in both dimensions, which makes it the default choice for wide, high-dimensional, or large-sample problems. Text classification with tens of thousands of sparse features is the classic case.

The catch is that LinearSVC can only fit a linear boundary. If the classes are not linearly separable, no amount of training fixes that. Use SVC with a kernel instead.

Watch for one trap. If you need only a linear boundary on a large dataset, do not use SVC with KernelType::Linear. That choice still pays the full O(n_samples^2) Gram-matrix cost, for a problem LinearSVC solves in linear memory. SVC with a linear kernel earns its cost only on small data, where you want the exact max-margin dual solution.

Both estimators are sensitive to feature scale. The RBF kernel measures squared Euclidean distance, and the SGD in LinearSVC takes fixed-size steps in feature space. Standardize your columns first (see Standardization and Normalization), unless they already share a range.

Both estimators also share one hard rule for y: every entry must be exactly 0.0 or 1.0. Any other value, such as a -1.0 from a textbook derivation, a 2.0, or a stray 0.5, causes an Error::InvalidInput at fit. Neither estimator remaps labels silently. If your labels are strings or arbitrary integers, run them through Label Encoding first.

2.5.2. SVC: kernels and the SMO solver

Construct an SVC with SVC::new(kernel, regularization_param, tol, max_iter). This call validates the arguments and returns a Result. Set the seed for reproducible training in a separate builder step, with with_random_state.

C is the regularization knob. It sets the upper bound on each dual coefficient, 0 <= alpha <= C. A large C lets the solver push the alphas high, fitting every training point with a narrow margin that tolerates few violations. A small C keeps the alphas small, widening the margin and accepting more slack. This is the opposite of the lambda intuition from linear models: a larger C means less regularization. Any non-positive or non-finite value fails at construction with Error::InvalidParameter.

SVC::default() gives an RBF kernel with gamma = 0.1, C = 1.0, tol = 0.001, and max_iter = 1000. This is a reasonable start when you have no prior information, though gamma almost always needs tuning.

Training runs Sequential Minimal Optimization (SMO). It repeatedly picks a pair of dual variables that violate the Karush-Kuhn-Tucker (KKT) conditions. It optimizes that pair analytically, holding the rest fixed. This repeats until the whole set satisfies the KKT conditions within tol, or until training hits the iteration cap.

The dual problem uses +1/-1 targets, because the y_i * y_j products in its objective only make sense with a sign. fit converts your {0.0, 1.0} column to +1/-1 once, on entry, and keeps that signed form internal. predict maps the sign back to {0.0, 1.0}. You never pass +1/-1 labels in, and, with one exception noted below, you never get them out either.

The Gram matrix and the initial error cache can build in parallel. The SMO inner loop stays sequential on purpose, so the optimization path stays reproducible for a given seed.

This example runs the whole construct-fit-predict loop with a linear kernel, on data a straight line separates cleanly:

use rustyml::machine_learning::{KernelType, SVC};
use ndarray::array;

fn main() {
    // Class 1 sits upper-right. Class 0 sits lower-left. A line separates them.
    let x = array![
        [2.0, 2.0], [3.0, 2.0], [2.0, 3.0], [3.0, 3.0],
        [-2.0, -2.0], [-3.0, -2.0], [-2.0, -3.0], [-3.0, -3.0],
    ];
    let y = array![1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0];

    let mut svc = SVC::new(KernelType::Linear, 10.0, 1e-3, 1000)
        .unwrap()
        .with_random_state(42);
    svc.fit(&x, &y).unwrap();

    // predict emits labels in {0.0, 1.0}, the same domain fit received
    let preds = svc.predict(&x).unwrap();
    println!("predictions: {:?}", preds);

    // decision_function returns the raw signed distance to the hyperplane
    let scores = svc.decision_function(&x).unwrap();
    println!("scores: {:?}", scores);
}

predict sets the threshold at zero. It maps >= 0.0 to 1.0 and everything else to 0.0. decision_function hands back the raw signed score. A positive score means the class-1.0 side, and its size shows how far the point sits from the boundary. Use decision_function when you want a confidence-like ordering instead of a hard label. There is also fit_predict, which fits and then predicts on the same matrix in 1 call.

The kernel zoo and the gamma coefficient

KernelType is the same enum that Kernel PCA uses, so learning it here pays off twice. It has 5 variants.

RBF is the default kernel and the one to try first for nonlinear data. It is a smooth, local similarity measure that only needs gamma tuned. Poly adds explicit interaction terms up to degree, but high degrees overflow quickly. A degree-400 polynomial on modestly large inputs pushes the decision value to infinity, which surfaces as Error::NonFinite at predict time.

Sigmoid mimics a two-layer network, but it is not positive-definite for all parameters, so it can behave erratically. Cosine measures angle rather than distance. It guards against zero vectors by returning 0.0 for them.

The gamma field is not a bare f64. It is a Gamma enum with 3 variants, matching scikit-learn’s 'scale', 'auto', and explicit choices:

• Gamma::Value(v): an explicit coefficient you supply.
• Gamma::Scale: resolved at fit time to 1 / (n_features * X.var()), where X.var() is the population variance of all entries of the training matrix.
• Gamma::Auto: resolved at fit time to 1 / n_features.

Scale and Auto are data-dependent, so they stay placeholders until fit sees the data. fit resolves them once and stores the concrete value. So a call to get_kernel() after fit returns a KernelType whose gamma is now a Gamma::Value. Scale fails with Error::InvalidInput if the data has zero variance, for example when every feature is constant, because the formula would divide by zero.

gamma controls how far a single training point’s influence reaches. A large gamma makes the RBF bumps tight and the boundary wiggly, a fast route to overfitting. A small gamma makes the bumps broad and the boundary smooth. It is the parameter you will tune most.

RBF on a problem no line can split

A kernel exists to separate data that a hyperplane cannot. Two concentric rings are the canonical example: an inner ring of one class, surrounded by an outer ring of the other. No straight line divides them, and a linear kernel is helpless here. The RBF kernel, which scores points by proximity, classifies every one correctly.

use rustyml::machine_learning::{Gamma, KernelType, SVC};
use ndarray::array;

fn main() {
    // Inner ring (radius 1) is class 1, outer ring (radius 5) is class 0.
    let x = array![
        [1.0, 0.0], [-1.0, 0.0], [0.0, 1.0], [0.0, -1.0],
        [5.0, 0.0], [-5.0, 0.0], [0.0, 5.0], [0.0, -5.0],
    ];
    let y = array![1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0];

    let mut svc = SVC::new(
        KernelType::RBF { gamma: Gamma::Value(0.5) },
        10.0,   // C
        1e-3,   // tol
        5000,   // max_iter, nonlinear problems need more SMO passes
    )
    .unwrap()
    .with_random_state(42);
    svc.fit(&x, &y).unwrap();

    let preds = svc.predict(&x).unwrap();
    let correct = preds.iter().zip(y.iter()).filter(|&(p, t)| p == t).count();
    println!("RBF accuracy on rings: {}/{}", correct, y.len());
}

Swap KernelType::RBF { .. } for KernelType::Linear on this dataset, and the classifier can no longer place every point on the right side. The rings are not linearly separable. That is exactly the failure mode a kernel exists to fix. Nonlinear problems also generally need a higher max_iter than linear ones, because the SMO solver has more support vectors to settle.

Inspecting the fitted model

After fit, SVC exposes the pieces of the trained model through getters. This is more than most estimators in this guide offer. The support vectors are the only training rows that matter to prediction, the ones whose dual coefficient came out non-zero. You can read them back directly:

use rustyml::machine_learning::{Gamma, KernelType, SVC};
use ndarray::array;

fn main() {
    let x = array![
        [2.0, 2.0], [3.0, 2.0], [2.0, 3.0], [3.0, 3.0],
        [-2.0, -2.0], [-3.0, -2.0], [-2.0, -3.0], [-3.0, -3.0],
    ];
    let y = array![1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0];

    let mut svc = SVC::new(
        KernelType::RBF { gamma: Gamma::Value(0.5) },
        5.0, 1e-3, 1000,
    )
    .unwrap()
    .with_random_state(42);
    svc.fit(&x, &y).unwrap();

    // Only rows with a non-zero alpha survive as support vectors.
    let support_vectors = svc.get_support_vectors().unwrap();
    let alphas = svc.get_alphas().unwrap();
    let labels = svc.get_support_vector_labels().unwrap();

    println!(
        "kept {} of {} rows as support vectors",
        support_vectors.nrows(),
        x.nrows()
    );
    println!("dual coefficients (alphas): {:?}", alphas);
    println!("support-vector labels (SMO's internal +/-1): {:?}", labels);
    println!("bias: {:?}", svc.get_bias());
    println!("SMO iterations actually run: {:?}", svc.get_actual_iterations());
    println!("resolved kernel: {:?}", svc.get_kernel());
}

get_support_vectors returns Option<&Array2<f64>>. get_alphas and get_support_vector_labels return Option<&Array1<f64>>. get_bias returns Option<f64>. All 4 are None before fit and Some after.

get_support_vector_labels is the one crack in the wall around the dual. It hands back the internal +1/-1 encoding, not the {0.0, 1.0} you handed to fit. So a -1.0 in that array denotes class 0.0. The label stays signed because predict folds it straight into the alpha_i * y_i coefficients. Threshold it yourself if you want the caller’s domain back.

get_actual_iterations reports how many SMO outer passes actually ran, always a number in [1, max_iter]. That count tells you whether the solver converged or hit the cap. A count equal to max_iter is a hint to raise the cap or loosen tol. get_kernel returns the resolved kernel. That is how you read back the concrete gamma that Gamma::Scale or Gamma::Auto produced.

Training can find no support vectors at all, for instance with single-class data where there is nothing to separate. In that case, fit returns Error::NotConverged instead of a degenerate all-zero model.

2.5.3. LinearSVC: hinge loss in the primal

LinearSVC solves the primal problem directly with minibatch stochastic gradient descent, minimizing hinge loss plus a regularization penalty. Construct it with LinearSVC::new(max_iter, learning_rate, penalty, fit_intercept, tol).

The labels are the same 0.0/1.0 that SVC takes. The story underneath is the same too: hinge loss needs signed targets, so fit remaps them to -1/+1 on entry. The difference is that nothing on LinearSVC’s public surface ever leaks that encoding back. The fitted state is a weight vector and a bias, and both live in feature space, not label space.

use rustyml::machine_learning::{LinearSVC, RegularizationType};
use ndarray::array;

fn main() {
    // The same 0.0 / 1.0 labels SVC takes. No remapping between the two.
    let x = array![
        [-5.0, 0.0], [-6.0, 0.0], [-7.0, 0.0], [-4.0, 0.0],
        [5.0, 0.0], [6.0, 0.0], [7.0, 0.0], [4.0, 0.0],
    ];
    let y = array![0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0];

    let mut model = LinearSVC::new(
        5000,                        // max_iter
        0.01,                        // learning_rate
        RegularizationType::L2(0.1), // penalty
        true,                        // fit_intercept
        1e-5,                        // tol
    )
    .unwrap()
    .with_random_state(42);
    model.fit(&x, &y).unwrap();

    let preds = model.predict(&x).unwrap();
    let scores = model.decision_function(&x).unwrap();
    println!("labels:  {:?}", preds);
    println!("scores:  {:?}", scores);
    println!("weights: {:?}", model.get_weights().unwrap());
    println!("bias:    {:?}", model.get_bias().unwrap());
    println!("stopped after {:?} epochs", model.get_actual_iterations());
}

fit shapes training with 2 details. It picks the minibatch size automatically, as clamp(n_samples / 10, 32, 512). You do not set that size. Each epoch shuffles the sample order before it slices the data into minibatches. This is why the seed matters (see below).

fit checks convergence on the root-mean-square change in the weights and bias between epochs. Training stops early once that change drops below tol. The default Loss::Hinge keeps a constant-size gradient near the margin, so a fixed learning rate often leaves the weights oscillating instead of settling. The example above shows this: it reaches the full 5000-epoch cap instead of stopping early. Loss::SquaredHinge has a gradient that shrinks to zero near the margin. Training with that loss usually stops well under max_iter, as the example below shows.

Left unchecked, a runaway learning_rate can push the weights to non-finite values. fit catches that and reports Error::NonFinite mid-training instead of returning garbage. Use a smaller learning_rate or stronger regularization to fix it.

Watch one threshold difference from SVC. LinearSVC::predict maps a decision value > 0.0 to class 1.0, and everything else, including exactly 0.0, to class 0.0. SVC maps >= 0.0 to 1.0 instead. Because both models share the same label domain, this is a real behavioral difference, not a bookkeeping detail. The 2 models break a zero score toward opposite classes.

decision_function here returns x * weights + bias. With fit_intercept = false, the bias stays exactly 0.0, and the score is a pure dot product.

Penalty and loss

RegularizationType is L1(lambda) or L2(lambda). L2, or ridge, applies a weights *= (1 - learning_rate * lambda) shrink at each step. This keeps all weights small but non-zero. L1, or lasso, applies a constant, sign-based subgradient pull that drives the weights of irrelevant features toward zero. Unlike LinearRegression and LogisticRegression, LinearSVC applies this pull as a plain subgradient step, not a proximal (soft-thresholding) step. So a weight rarely lands on exactly 0.0, though it still lands close to it.

If a feature carries no signal, a strong L1 penalty shrinks its weight closer to zero than it shrinks an informative feature’s weight. This helps when you suspect that many columns are noise. lambda = 0.0 is legal. It disables the penalty entirely.

2 builder methods extend the estimator past what new sets. with_loss switches between Loss::Hinge, the default, max(0, 1 - y*f(x)), and Loss::SquaredHinge, max(0, 1 - y*f(x))^2. Loss::SquaredHinge penalizes margin violations quadratically and is differentiable everywhere. That smoother objective lets some datasets converge more cleanly.

with_learning_rate_decay turns on an inverse-scaling schedule, where the effective rate at epoch t is learning_rate / (1 + decay * t). This lets SGD settle closer to the optimum, instead of hovering a fixed step away from it. It returns a Result, because the decay must be non-negative and finite.

use rustyml::machine_learning::{LinearSVC, Loss, RegularizationType};
use ndarray::array;

fn main() {
    let x = array![
        [-5.0, 0.0], [-6.0, 0.0], [-7.0, 0.0], [-4.0, 0.0],
        [5.0, 0.0], [6.0, 0.0], [7.0, 0.0], [4.0, 0.0],
    ];
    let y = array![0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0];

    let mut model = LinearSVC::new(10_000, 0.01, RegularizationType::L1(0.5), true, 1e-6)
        .unwrap()
        .with_loss(Loss::SquaredHinge)
        .with_learning_rate_decay(0.001)  // returns Result and checks the decay
        .unwrap()
        .with_random_state(0);
    model.fit(&x, &y).unwrap();

    println!("penalty: {:?}", model.get_penalty());
    println!("loss:    {:?}", model.get_loss());
    println!("preds:   {:?}", model.predict(&x).unwrap());
}

The getters mirror SVC’s: get_weights (Option<&Array1<f64>>), get_bias, get_penalty, get_loss, get_learning_rate, get_learning_rate_decay, get_tolerance, get_max_iterations, get_actual_iterations, and get_random_state. The weight vector length always matches n_features. So get_weights doubles as a feature-importance readout, once your columns share a common scale. LinearSVC is the linear-classifier cousin of Logistic Regression. The difference is the loss: hinge instead of log-loss. Hinge cares only about points near or across the margin, so it tends to give a boundary that ignores confidently-correct points entirely.

2.5.4. Reproducibility, persistence, and errors

Both estimators are non-deterministic by default. Both take a fixed seed through with_random_state(u64). For SVC, the seed drives the SMO working-set fallback, the randomized offset used when scanning for a second alpha to optimize. The same seed reproduces the same support vectors, bias, and predictions, bit for bit.

For LinearSVC, the seed drives the per-epoch minibatch shuffle. That fixes the exact sequence of gradient steps, and so the final weights too. Different seeds can land on different solutions, especially for LinearSVC on data where the shuffle order matters. Set the seed whenever you need runs to be comparable. See Reproducibility and Random Seeds for how this fits the crate-wide seeding story.

use rustyml::machine_learning::{Gamma, KernelType, SVC};
use ndarray::array;

fn main() {
    let x = array![
        [2.0, 2.0], [3.0, 2.0], [2.0, 3.0], [3.0, 3.0],
        [-2.0, -2.0], [-3.0, -2.0], [-2.0, -3.0], [-3.0, -3.0],
    ];
    let y = array![1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0];

    let train = || {
        let mut svc = SVC::new(KernelType::RBF { gamma: Gamma::Value(0.5) }, 5.0, 1e-3, 1000)
            .unwrap()
            .with_random_state(42);
        svc.fit(&x, &y).unwrap();
        svc.predict(&x).unwrap()
    };

    // Same seed, same data -> identical predictions.
    assert_eq!(train(), train());
    println!("same seed reproduces the model exactly");
}

Both models implement save_to_path and load_from_path. These methods serialize the entire fitted state to a compact postcard binary blob. That state is the support vectors and alphas for SVC, or the weights and bias for LinearSVC, plus every hyperparameter.

You can choose any file extension. The format stays binary regardless of the name. A round-trip reproduces predictions and decision scores exactly. This is the shallow end of Model Persistence in Depth.

use rustyml::machine_learning::{Gamma, KernelType, SVC};
use ndarray::array;
use std::fs::remove_file;

fn main() {
    let x = array![
        [2.0, 2.0], [3.0, 2.0], [2.0, 3.0], [3.0, 3.0],
        [-2.0, -2.0], [-3.0, -2.0], [-2.0, -3.0], [-3.0, -3.0],
    ];
    let y = array![1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0];

    let mut svc = SVC::new(KernelType::RBF { gamma: Gamma::Value(0.5) }, 5.0, 1e-3, 1000)
        .unwrap()
        .with_random_state(42);
    svc.fit(&x, &y).unwrap();

    svc.save_to_path("svc_model.bin").unwrap();      // postcard binary
    let loaded = SVC::load_from_path("svc_model.bin").unwrap();

    assert_eq!(svc.predict(&x).unwrap(), loaded.predict(&x).unwrap());
    println!("loaded model reproduces the original's predictions");

    remove_file("svc_model.bin").unwrap();
}

Every fallible entry point returns the crate’s Error (see Error Handling). The variants you will meet in practice:

The label domain still trips people, just not in the direction the SMO literature suggests. Both models want 0.0/1.0, so a y copied from a textbook derivation or from another library’s signed convention causes an InvalidInput at fit. Neither model converts it silently. Encode your labels into {0.0, 1.0} before training. After that, the only +1/-1 values you will meet are whatever get_support_vector_labels hands back.

2.6. Linear Discriminant Analysis

Linear Discriminant Analysis (LDA) does 2 jobs from a single fit. First, it works as a generative classifier. It models each class as a Gaussian with its own mean and a covariance shared across all classes. It then applies Bayes’ rule to pick the most probable label. Second, it works as a supervised dimensionality reducer. It projects the features onto the directions that separate the classes best. RustyML’s LDA gives both results from one fit call. predict, decision_function, and predict_proba serve the classifier. transform and fit_transform serve the projection. This page covers the API and the in-house linear algebra behind it. It also covers the Gaussian assumptions the model relies on, and what happens when the covariance matrix goes singular.

2.6.1. What the crate implements

RustyML implements LDA under rustyml::machine_learning::{LDA, Shrinkage, DiscriminantSolver}. The prelude also re-exports these types (see the prelude). The model takes an f64 feature matrix and i32 labels. LDA is a classifier, so labels are integer class ids, not floating-point values. The table below lists the API surface:

The crate does not implement QDA (quadratic discriminant analysis). It offers only the shared-covariance, linear-boundary variant. The getters get_classes, get_priors, get_means, get_overall_mean, and get_projection return None before fit runs. After fit runs, each getter returns its value wrapped in Some. Checking any of these getters tells you whether the model is trained.

The solver enum used to carry the plain name Solver. That name collided with the linear models’ own Solver enum. After a prelude glob import, Solver::GradientDescent resolved to the wrong enum and failed to compile. Both enums now carry names that match what they select. This crate uses DiscriminantSolver here and LeastSquaresSolver in the linear models.

2.6.2. A first classifier

Start with 3 well-separated clusters in 2D. LDA::default() resolves the component count automatically. You do not need to set a component count when you only want labels.

use ndarray::array;
use rustyml::machine_learning::LDA;

fn main() {
    // 3 tight, well-separated clusters, 3 samples each.
    let x = array![
        [1.0, 1.0], [1.5, 0.8], [0.8, 1.2],
        [5.0, 5.0], [5.2, 4.8], [4.8, 5.2],
        [9.0, 1.0], [9.2, 0.8], [8.8, 1.2],
    ];
    let y = array![0, 0, 0, 1, 1, 1, 2, 2, 2];

    let mut lda = LDA::default();
    lda.fit(&x, &y).unwrap();

    let preds = lda.predict(&x).unwrap();
    println!("predictions: {:?}", preds);

    // Per-class discriminant scores (n_samples, n_classes) and softmax posteriors.
    let scores = lda.decision_function(&x).unwrap();
    let proba = lda.predict_proba(&x).unwrap();
    println!("scores shape:   {:?}", scores.shape());
    println!("proba shape:    {:?}", proba.shape());
    println!("class order:    {:?}", lda.get_classes().unwrap());
}

predict, decision_function, and predict_proba share a single core computation. For a sample x, LDA computes a linear discriminant score for each class. The formula is score_j(x) = x * Sigma^-1 * mu_j - 0.5 * mu_j * Sigma^-1 * mu_j + ln(prior_j). Here Sigma is the shared covariance, mu_j is the class mean, and prior_j is the class frequency in the training set. decision_function returns these raw scores as an (n_samples, n_classes) matrix. predict takes the per-row argmax of the scores and maps it back to the original label. predict_proba applies a numerically stable row-wise softmax to the same scores, so each row sums to 1. The column order of decision_function and predict_proba follows get_classes(). get_classes() sorts the labels in ascending order. Do not assume this order matches the order labels first appeared in y.

The priors are pure class frequencies (n_class / n_samples). For this balanced set, each prior is 1/3. Class imbalance changes this. The ln(prior_j) term shifts the boundary toward the rarer class, the way a Bayes classifier should. To get equal priors, balance the training set. The LDA constructor has no priors= override.

2.6.3. Supervised projection and the n_classes - 1 ceiling

The projection side is what distinguishes LDA from PCA. LDA finds directions that maximize between-class scatter relative to within-class scatter. Only n_classes - 1 such directions carry non-zero separability, because the between-class scatter matrix has rank at most n_classes - 1. The crate enforces this limit. The usable component count is min(n_classes - 1, n_features), resolved at fit time. LDA::default() (or n_components = None) takes that maximum. LDA::new(k) requests exactly k components and fails at fit if k exceeds the cap.

A 2D scatter plot therefore needs at least 3 classes. 2 classes give only 1 discriminant axis. fit_transform trains the model and projects the data in a single call:

use ndarray::array;
use rustyml::machine_learning::{DiscriminantSolver, LDA, Shrinkage};

fn main() {
    let x = array![
        [1.0, 1.0], [1.5, 0.8], [0.8, 1.2],
        [5.0, 5.0], [5.2, 4.8], [4.8, 5.2],
        [9.0, 1.0], [9.2, 0.8], [8.8, 1.2],
    ];
    let y = array![0, 0, 0, 1, 1, 1, 2, 2, 2];

    // 3 classes -> at most 2 discriminant axes. Project straight to 2D.
    let mut lda = LDA::new(2)
        .unwrap()
        .with_solver(DiscriminantSolver::Eigen)
        .with_shrinkage(Shrinkage::Auto)
        .unwrap();

    let coords = lda.fit_transform(&x, &y).unwrap(); // (9, 2)
    println!("embedding shape: {:?}", coords.shape());
    for (row, &label) in coords.outer_iter().zip(y.iter()) {
        println!("class {label}: ({:.3}, {:.3})", row[0], row[1]);
    }

    // The projection matrix maps n_features -> n_components.
    let w = lda.get_projection().unwrap();
    println!("projection shape: {:?}", w.shape()); // [2, 2]
}

embedding shape: [9, 2]
... one line per sample: class <label>: (<axis 1>, <axis 2>) ...
projection shape: [2, 2]

The projection matrix W has shape (n_features, n_components). transform computes (x - xbar) * W. Here xbar is the mean of the whole training matrix. This value matches scikit-learn’s xbar_, and get_overall_mean() reads it back. RustyML matches scikit-learn’s (X - xbar_) @ scalings_ formula exactly. That match brings 2 details, and both changed recently. First, the projection is centered by the training mean. A transformed training set sits around the origin, not around the raw feature values. Second, the axes keep the scale the whitening produced. RustyML does not renormalize the axes to unit L2 norm. That scale is not cosmetic. It gives the projected data unit within-class covariance, the property the whole construction targets. A projection read from an older, unit-norm axis now shows a different offset and a different scale.

RustyML orders the columns by class separability, with the largest generalized eigenvalue first. The first component is therefore the single most discriminative axis. This ordering helps if you later want a 1D reduction from the same fit. A discriminant axis and its negation separate the classes equally well. The sign of any column is therefore arbitrary. Treat the orientation of a plotted axis as meaningless. Do not read directionality into it.

transform and fit_transform perform the same reduction step. fit_transform(&x, &y) runs fit(&x, &y) followed by transform(&x) in a single call.

2.6.4. Solvers and the in-house linear algebra

DiscriminantSolver selects how RustyML inverts the shared covariance for the classifier’s scoring coefficients Sigma^-1 * mu_c. This is an important and easy-to-miss fact. The solver choice only affects predict, decision_function, and predict_proba. The transform projection does not depend on the solver. RustyML always derives it from a symmetric eigendecomposition of the within-class covariance. Switching solvers changes the decision boundary’s numerics. It never changes the 2D embedding.

All 3 solvers reach identical labels on well-separated data. They differ only in how they handle an ill-conditioned Sigma. SVD and Eigen both build a pseudo-inverse. This pseudo-inverse discards any direction whose singular value or eigenvalue falls below a relative tolerance. A rank-deficient covariance therefore degrades gracefully instead of failing. LSQR avoids the inverse entirely. It solves each Sigma * coef_c = mu_c system iteratively. This choice saves memory when n_features is large.

The linear algebra is entirely in-house pure Rust. RustyML carries no LAPACK or nalgebra dependency (see prefer in-house numerics). The symmetric eigendecomposition uses Householder tridiagonalization followed by the implicit-shift QL iteration, the classic EISPACK tred2 and tql2 pair. The SVD uses a one-sided Jacobi method that computes small singular values to high relative accuracy. LSQR uses the Golub-Kahan bidiagonalization with running Givens rotations. Each method is deterministic to machine precision on the small-to-mid covariance matrices LDA produces. That determinism is why LDA needs no random seed.

2.6.5. Shrinkage and covariance regularization

With few samples per feature, the raw sample covariance is a noisy estimate. Its inverse amplifies that noise. Shrinkage pulls Sigma toward a scaled identity matrix. This trades a small amount of bias for a large reduction in variance. Shrinkage has 2 forms:

• Shrinkage::Auto: the Ledoit-Wolf closed-form optimal intensity. RustyML computes it from the data, with no tuning knob to set.
• Shrinkage::Manual(alpha): an explicit alpha in [0, 1]. A value of 0 applies no shrinkage. A value of 1 shrinks fully to the identity target. with_shrinkage validates alpha and returns Error::InvalidParameter if it falls outside [0, 1] or is not finite. This validation is why with_shrinkage returns a Result.

2 subtleties matter here. First, RustyML always adds a tiny fixed diagonal ridge to the covariance, about 1e-6 times the average variance. It adds this ridge regardless of the shrinkage choice, only to keep the factorizations stable. Shrinkage::Manual(0.0) and no shrinkage therefore produce identical results. Second, shrinkage here is independent of the solver, and this is a real divergence from scikit-learn. scikit-learn refuses shrinkage with its svd solver. RustyML applies shrinkage to the covariance before any solver runs. DiscriminantSolver::SVD with Shrinkage::Auto is therefore valid, and often the best combination. Choose Shrinkage::Auto as the default whenever you have collinear features or a thin sample-to-feature ratio.

2.6.6. Assumptions, and the symptoms when they break

LDA’s optimality rests on 2 assumptions. Each class follows a Gaussian distribution. All classes share one covariance matrix. The shared covariance is what makes the decision boundary linear, because the quadratic terms cancel. When these assumptions hold, LDA is data-efficient. It estimates a single covariance from all classes pooled together, instead of a separate covariance per class.

When the assumptions break, the symptoms are specific. Say the classes have genuinely different covariances, a condition called heteroscedastic data. One class forms a tight blob, and another forms a broad cloud. The true Bayes boundary is then quadratic. LDA’s forced-linear boundary systematically misclassifies samples in the region where the spreads differ. The wide class tends to dominate territory near the boundary. The crate has no QDA as an alternative. For strongly heteroscedastic data, use a non-linear classifier instead, such as a decision tree, an SVM with an RBF kernel, or a KNN classifier. A class that is multimodal or clearly non-Gaussian can also break the projection, even when the classifier still produces usable labels. Check a transform scatter plot before you trust the reduction.

2.6.7. Singular covariance and high dimensions

This section covers what happens when features are collinear, or when n_features exceeds n_samples, so the covariance becomes singular. RustyML does not raise a dedicated singular-matrix error. The crate places no guard on n_features against n_samples. The only sample-count checks are that n_samples must exceed n_classes, and each class needs at least 2 samples. A rank-deficient covariance gets absorbed in 2 ways. The always-on diagonal ridge nudges the covariance toward invertibility. The SVD and Eigen pseudo-inverse, and the whitening step inside the projection, zero out any direction whose eigenvalue falls below a relative tolerance. This projects onto the covariance’s non-null subspace instead of dividing by a value near 0.

use ndarray::array;
use rustyml::machine_learning::LDA;

fn main() {
    // The third column equals the sum of the first two, so the feature
    // matrix is rank-deficient. The shared covariance is therefore
    // singular.
    let x = array![
        [1.0, 2.0, 3.0],
        [1.5, 2.5, 4.0],
        [2.0, 3.0, 5.0],
        [6.0, 5.0, 11.0],
        [6.5, 4.5, 11.0],
        [7.0, 5.5, 12.5],
    ];
    let y = array![0, 0, 0, 1, 1, 1];

    // No singular-matrix error: the ridge and pseudo-inverse absorb the
    // rank deficiency.
    let mut lda = LDA::default();
    lda.fit(&x, &y).unwrap();

    let coords = lda.transform(&x).unwrap(); // (6, 1): 2 classes -> 1 component
    assert!(coords.iter().all(|v| v.is_finite()));
    println!("reduced to {:?}, all finite", coords.shape());
}

The projection has 1 remaining failure mode: Error::Computation, with context set to “Discriminant direction norm too small for stable projection”. RustyML raises this error only when a selected discriminant axis collapses to a near-zero vector, relative to the largest axis. The guard became relative once RustyML stopped renormalizing the axes to unit length. This failure happens when you request more components than the data can support in its degenerate subspace. Treat it as a hard numerical failure, not a validation slip. Reduce n_components, or regularize harder, to fix it. For n_features greater than n_samples, or for heavily collinear data, do not rely on the fixed 1e-6 ridge alone. Turn on Shrinkage::Auto instead. It conditions the covariance far better than the minimal stabilizer, and it makes the difference between a usable projection and one dominated by noise directions.

2.6.8. LDA versus PCA versus logistic regression

3 related methods are worth a direct comparison, because choosing among them is usually the real question.

PCA maximizes variance without ever looking at y. Its top components can therefore align with the direction that separates classes least, if that is where the spread lives. LDA maximizes separation directly, but it is capped at n_classes - 1 axes. Use LDA for a supervised low-dimensional view, or for a fast generative classifier, when the classes are roughly Gaussian. Use PCA for unsupervised compression, or when you need more than n_classes - 1 dimensions.

Against logistic regression, the split is generative versus discriminative. LDA models the class-conditional densities and applies Bayes’ rule. Logistic regression models P(y | x) directly and makes no assumption about the shape of each class. When the Gaussian and equal-covariance assumptions genuinely hold, LDA converges to a good boundary from fewer samples. When they do not hold, logistic regression is the more dependable linear classifier, because it never assumed a class shape. A common approach keeps both models and lets held-out accuracy decide, measured with the classification metrics.

2.6.9. Errors, persistence, and reproducibility

The error surface follows the crate-wide error handling conventions. LDA::new(0) returns Error::InvalidParameter, because components must be positive. An out-of-range Shrinkage::Manual also returns InvalidParameter, caught at build time. Structural problems surface at fit. An empty feature matrix, or a matrix with 0 feature columns, returns Error::EmptyInput. Fewer than 2 classes, n_samples not greater than n_classes, a class with only 1 sample, and an over-large n_components all return Error::InvalidInput. A non-finite entry in x returns Error::NonFinite. An x/y length mismatch returns Error::DimensionMismatch. At predict or transform time, an empty input matrix also returns Error::EmptyInput, a wrong feature count returns Error::DimensionMismatch, and calling an unfitted model returns Error::NotFitted("LDA").

use ndarray::array;
use rustyml::error::Error;
use rustyml::machine_learning::{LDA, Shrinkage};

fn main() {
    // Manual shrinkage must lie in [0, 1]. RustyML rejects 1.5 when you
    // build the model.
    match LDA::new(1).unwrap().with_shrinkage(Shrinkage::Manual(1.5)) {
        Err(Error::InvalidParameter { name, .. }) => println!("rejected parameter: {name}"),
        _ => unreachable!(),
    }

    // n_components is only bounded once the class count is known, so an over-large
    // request survives the constructor and fails at fit.
    let x = array![
        [1.0, 1.0], [1.5, 0.8], [0.8, 1.2],
        [5.0, 5.0], [5.2, 4.8], [4.8, 5.2],
    ];
    let y = array![0, 0, 0, 1, 1, 1]; // 2 classes -> max_components = 1
    let mut lda = LDA::new(2).unwrap();
    match lda.fit(&x, &y) {
        Err(Error::InvalidInput(msg)) => println!("fit rejected: {msg}"),
        _ => unreachable!(),
    }
}

A fitted LDA serializes with save_to_path and load_from_path. Both methods use the postcard binary format. The .bin extension is only a filename convention. The format stays binary regardless of the extension you choose. The round trip restores the classes, the priors, the means, the overall training mean, the projection, and the cached scoring coefficients. A loaded model predicts and transforms identically, with no re-fit needed. The overall mean is a field the format gained when transform started centering the projection. Blobs saved by an older version of RustyML will not load. Re-fit and re-save the model with the current version. See model persistence in depth for the format details.

use ndarray::array;
use rustyml::machine_learning::LDA;
use std::fs;

fn main() {
    let x = array![
        [1.0, 1.0], [1.5, 0.8], [0.8, 1.2],
        [5.0, 5.0], [5.2, 4.8], [4.8, 5.2],
        [9.0, 1.0], [9.2, 0.8], [8.8, 1.2],
    ];
    let y = array![0, 0, 0, 1, 1, 1, 2, 2, 2];

    let mut lda = LDA::new(2).unwrap();
    lda.fit(&x, &y).unwrap();
    let before = lda.predict(&x).unwrap();

    let path = "lda_model.bin";
    lda.save_to_path(path).unwrap();
    let loaded = LDA::load_from_path(path).unwrap();
    let after = loaded.predict(&x).unwrap();

    assert_eq!(before, after);
    fs::remove_file(path).unwrap();
    println!("round-trip predictions identical");
}

LDA is fully deterministic. It draws no random numbers, so the same data yields the same projection, the same coefficients, and the same labels on every run. It needs no seed to set, unlike the stochastic models in reproducibility and random seeds. The only run-to-run freedom is the arbitrary sign of each discriminant axis. That sign is a mathematical property of eigenvectors, not a sign of nondeterminism. Fitting parallelizes the per-class statistics and the final label scan, once the work clears the crate’s calibrated threshold. Parallelism never changes the result. It only changes how fast the result arrives. See performance tuning and parallelism for more on this topic.

2.7. KMeans Clustering

KMeans partitions samples into a fixed number of clusters. It alternates between 2 steps: it assigns each point to its nearest centroid, then it recomputes each centroid as the mean of its members. RustyML’s implementation is a parallel Lloyd’s algorithm with k-means++ seeding and best-of-n restarts. If you know scikit-learn’s sklearn.cluster.KMeans, most of that mental model applies here too. The one default that differs on purpose is n_init. See 2.7.1 for the reason.

2.7.1. How the estimator works

One call to fit runs the whole procedure n_init times. It keeps the run with the lowest inertia. Within a run, centroids start from k-means++ seeding. The first center is a uniformly random sample.

Each later center is drawn from the remaining points. The probability is proportional to the squared distance to the nearest center already chosen (the classic D^2 roulette wheel). This spreads the initial centers out, and it gives Lloyd’s iteration a better starting point than uniform random seeding. If every candidate’s squared distance is zero, the implementation picks that center uniformly at random instead. This case happens only when all points duplicate the already-chosen centers.

After seeding, each iteration does 2 things. The assignment step assigns every point to the nearest centroid. The update step replaces each centroid with the mean of its assigned points.

Iteration stops when the centroids stop moving. The convergence test compares the summed squared shift of all centroids against a variance-scaled tolerance. The threshold equals the mean per-feature population variance of the data, multiplied by tol. So tol is a relative tolerance, not an absolute distance. This matches the convention scikit-learn uses. A converged solution is a fixed point: every centroid equals the mean of its assigned points.

A fit that runs out of budget, instead of converging, gets one extra assignment pass before it returns. The Lloyd loop labels points against the current centroids, then installs the updated centroids. So stopping at max_iterations would leave labels and inertia describing the old centroids, while get_centroids reports the new ones. predict(x) would then disagree with get_labels(). The final pass re-assigns points against the centroids the model actually stores. scikit-learn re-runs its final E-step for the same reason.

The one deliberate difference from scikit-learn is the restart count. n_init defaults to 10 here. scikit-learn’s n_init='auto' is 1 for k-means++. This is not an oversight. scikit-learn’s default rests on its greedy k-means++, which draws 2 + ln(k) candidates per center and keeps the best. So one seeding already has low variance there.

RustyML uses plain k-means++: a single D^2 draw per center. Restarts exist to compensate for that higher-variance seeding. Pass with_n_init(1) for exact scikit-learn parity. Expect the fitted result to change when you do. Restarts derive their seeds deterministically from random_state, so a seeded fit stays reproducible.

2.7.2. Constructing a model

The constructor takes 3 positional parameters and validates them eagerly. An invalid configuration fails at construction, not at fit.

pub fn new(n_clusters: usize, max_iterations: usize, tolerance: f64) -> Result<Self, Error>

Any violation returns Error::InvalidParameter: n_clusters == 0, max_iterations == 0, or a tolerance that is zero, negative, NaN, or infinite. Note the asymmetry with the data-validation errors. An out-of-range hyperparameter returns InvalidParameter. A non-finite value in the data returns NonFinite instead. This distinction is a crate-wide convention.

KMeans::default() gives you n_clusters = 8, max_iterations = 300, tolerance = 1e-4, n_init = 10, and no seed. This configuration is always valid.

The 2 remaining settings are builder steps. Each one consumes the instance and returns it, so you can chain them after new. with_random_state(seed) cannot fail. with_n_init(n) returns Result, because 0 restarts triggers Error::InvalidParameter:

let mut km = KMeans::new(3, 300, 1e-4)
    .unwrap()
    .with_n_init(1)          // scikit-learn parity, returns Result
    .unwrap()
    .with_random_state(42);  // returns Self

Without a seed, k-means++ draws from entropy, and each fit produces a different partition. With a seed, the fit is reproducible. See 2.7.6 for details. This reproducibility covers every restart too. Each restart derives its own sub-seed deterministically from the seed you set.

2.7.3. Fitting, predicting, and reading results

3 methods drive the model. They differ in what they return and what they mutate.

fit(&mut self, data) trains the model in place and returns Result<&mut Self, Error>. It computes and stores the centroids, the training-set labels, the inertia, and the iteration count. All 4 values describe the winning restart.

predict(&self, data) assigns each row of a new matrix to its nearest fitted centroid. It returns an owned Array1<isize>. It borrows self immutably and does not change any stored state, so you can call it as many times as you want after one fit.

The labels use a signed type, not an unsigned one. This lets every clustering estimator in the crate share one label type. DBSCAN and Mean Shift both use -1 for noise, and the shared type feeds the metrics in 5.3. Clustering Metrics without a conversion. k-means itself never returns a negative label.

fit_predict(&mut self, data) calls fit, then returns a clone of the training labels. Use it when you want only the labels for the data you just trained on. Use fit plus predict instead when you want to score fresh points.

The fitted state is exposed through getters that return None before fit:

fit reports 3 data errors. It returns Error::EmptyInput when the matrix has zero rows. It returns Error::NonFinite when any value is NaN or infinite. It returns Error::InvalidInput when there are fewer samples than clusters, because you cannot place k centroids with fewer than k points.

predict adds 2 more errors. It returns Error::NotFitted when called before fit. It returns Error::DimensionMismatch when the feature count does not match the training data. It also runs the same empty-input and non-finite checks as fit.

2.7.4. Clustering three blobs end to end

3 tight, well-separated blobs make a standard sanity check. Any correct k=3 run must give every blob its own cluster. This example uses a small, deterministic dataset, with no random numbers in the data itself, and a fixed seed. It runs instantly, and the structure of the result is predictable.

use rustyml::machine_learning::KMeans;
use ndarray::{array, Array2};

fn main() {
    // 3 blobs of 5 points, centered at (0,0), (10,0), (5,10).
    let data: Array2<f64> = array![
        [-0.05,  0.03], [ 0.04, -0.02], [ 0.01,  0.05], [-0.03, -0.04], [ 0.02,  0.01],
        [ 9.95,  0.03], [10.04, -0.02], [10.01,  0.05], [ 9.97, -0.04], [10.02,  0.01],
        [ 4.95, 10.03], [ 5.04,  9.98], [ 5.01, 10.05], [ 4.97,  9.96], [ 5.02, 10.01],
    ];

    let mut km = KMeans::new(3, 300, 1e-4).unwrap().with_random_state(42);
    km.fit(&data).unwrap();

    let labels = km.get_labels().unwrap();
    let centroids = km.get_centroids().unwrap();
    println!("labels:    {:?}", labels);
    println!("centroids: {:?}", centroids);
    println!("inertia:   {:.6}", km.get_inertia().unwrap());
    println!("iters:     {}", km.get_actual_iterations().unwrap());

    // Score fresh points sitting on each blob's true center.
    let new_points = array![[0.0, 0.0], [10.0, 0.0], [5.0, 10.0]];
    let predicted = km.predict(&new_points).unwrap();
    println!("new-point labels: {:?}", predicted);
}

The exact cluster indices are arbitrary. k-means numbers clusters by discovery order, so the seed decides which blob becomes cluster 0. The structure of the result stays fixed:

labels:    the 5 points of each blob share one index. The 3 blobs get 3 distinct indices
           (some permutation of 0, 1, 2).
centroids: shape (3, 2). Rows sit within about 0.1 of (0,0), (10,0), (5,10), in some order.
inertia:   a small positive f64 (sum of squared point-to-centroid distances).
iters:     a handful, well below max_iterations.
new-point labels: each test point maps to the same cluster index as the blob it sits on.

Cluster indices depend on the permutation, so never compare labels across 2 separately fitted models by equality. Instead, use a permutation-invariant metric, such as the adjusted Rand index in 5.3. Clustering Metrics.

2.7.5. Choosing k

k-means cannot tell you k. You must supply it. 2 techniques narrow down a good value, and RustyML gives you the raw material for both.

The elbow method plots inertia against k. Inertia falls monotonically as k grows, because more centroids can only reduce the sum of squared distances. At k = n it reaches zero. Look for the elbow, the point where the drop flattens out. Read inertia directly from get_inertia.

The silhouette score is more decisive, because it has an interior optimum instead of a monotone trend. For each point, it measures how much closer that point sits to its own cluster than to the nearest other cluster. It averages these values to a score in [-1, 1], where higher is better. The metrics module provides it as silhouette_score. It takes the feature matrix, the labels, and a DistanceCalculationMetric. It is defined only for 2..=n-1 distinct clusters, so it cannot score k = 1.

use rustyml::machine_learning::KMeans;
use rustyml::metrics::silhouette_score;
use rustyml::math::DistanceCalculationMetric;
use ndarray::{array, Array2};

fn main() {
    let data: Array2<f64> = array![
        [-0.05,  0.03], [ 0.04, -0.02], [ 0.01,  0.05], [-0.03, -0.04], [ 0.02,  0.01],
        [ 9.95,  0.03], [10.04, -0.02], [10.01,  0.05], [ 9.97, -0.04], [10.02,  0.01],
        [ 4.95, 10.03], [ 5.04,  9.98], [ 5.01, 10.05], [ 4.97,  9.96], [ 5.02, 10.01],
    ];

    println!(" k   inertia   silhouette");
    for k in 1..=5usize {
        let mut km = KMeans::new(k, 300, 1e-4).unwrap().with_random_state(42);
        let labels = km.fit_predict(&data).unwrap();
        let inertia = km.get_inertia().unwrap();
        if k >= 2 {
            let s = silhouette_score(&data, &labels, DistanceCalculationMetric::Euclidean);
            println!("{k:2}  {inertia:8.4}   {s:7.4}");
        } else {
            println!("{k:2}  {inertia:8.4}      (n/a)");
        }
    }
}

On these 3 clean blobs, inertia drops sharply from k = 1 to k = 3, then flattens. The silhouette score peaks at k = 3. Both point at the true structure. Real data is messier, so the silhouette’s interior maximum tends to give a clearer signal. See 5.3. Clustering Metrics for the full list of metrics, including Davies-Bouldin and Calinski-Harabasz, which give independent second opinions.

2.7.6. Reproducible clusters: local seeds and the global seed

k-means++ is randomized, so an unseeded model produces a different partition on every fit. There are 2 ways to pin it down, and they combine in a predictable way.

The local seed is set with with_random_state(seed). It is self-contained: it seeds that estimator’s initialization RNG directly. It ignores any global state, and it never affects other components.

Each of the n_init restarts derives its own sub-seed from it, through a fixed mixing step, instead of by advancing one shared RNG. So a restart’s seeding does not depend on how much randomness the earlier restarts consumed. 2 models built with the same local seed, and fitted on the same data, produce identical centroids, labels, and inertia. The iteration count matches too, down to the last bit.

The global seed, set with rustyml::random::set_global_seed(seed), fixes every unseeded randomized component built and fitted on the same thread afterward. This includes k-means, the neural-network initializers, train_test_split, and more, all from one call. This mirrors Keras’ global-seed behavior. An unseeded k-means model draws an independent sub-seed from the global stream at fit time. To reproduce a run, set the global seed again before you re-fit.

2 caveats matter here. The global seed is thread-local, so set it on the thread that fits your models. Unseeded components also consume the stream in fit order, so their reproducibility depends on that order. A with_random_state seed avoids both problems. This is why it is the right tool for pinning a single estimator.

use rustyml::machine_learning::KMeans;
use rustyml::random::{set_global_seed, clear_global_seed};
use ndarray::{array, Array2};

fn main() {
    let data: Array2<f64> = array![
        [0.0, 0.0], [0.1, 0.0], [0.0, 0.1],
        [10.0, 0.0], [10.1, 0.0], [10.0, 0.1],
        [5.0, 10.0], [5.1, 10.0], [5.0, 10.1],
    ];

    // Local seed: identical results, independent of any global state.
    let mut a = KMeans::new(3, 300, 1e-4).unwrap().with_random_state(42);
    let mut b = KMeans::new(3, 300, 1e-4).unwrap().with_random_state(42);
    a.fit(&data).unwrap();
    b.fit(&data).unwrap();
    assert_eq!(a.get_labels().unwrap(), b.get_labels().unwrap());

    // Global seed: reset it before each fit to reproduce an unseeded run.
    set_global_seed(7);
    let mut c = KMeans::new(3, 300, 1e-4).unwrap();
    c.fit(&data).unwrap();
    let labels_c = c.get_labels().unwrap().clone();

    set_global_seed(7);
    let mut d = KMeans::new(3, 300, 1e-4).unwrap();
    d.fit(&data).unwrap();
    assert_eq!(&labels_c, d.get_labels().unwrap());

    clear_global_seed();
}

This determinism holds because the parallel arithmetic is designed for reproducibility, not just for speed. See 2.7.8 for how. For the full seeding model across the crate, see 7.1. Reproducibility and Random Seeds.

2.7.7. Failure modes and gotchas

Empty clusters. When Lloyd’s assignment leaves a centroid with no points, RustyML does not drop the cluster. It also does not leave a stale centroid in place. Each iteration reseeds every empty cluster with the single point farthest from its own assigned centroid, the point that contributes most to inertia. This keeps the model at exactly n_clusters centroids, and moves it out of the degenerate state on the next pass. So get_centroids always returns n_clusters rows. On pathological data, such as heavy duplication or k close to n, the labels can still resolve to fewer than k distinct values.

Sensitivity to feature scale. k-means minimizes Euclidean distance. So a feature measured in thousands dominates one measured in fractions, and the clustering effectively ignores the small-scale feature. There is no built-in standardization. Standardize your data first, with the tools in 4.2. Standardization and Normalization, whenever your features sit on different scales. This is the most common reason a k-means result looks wrong.

Sensitivity to outliers. Centroids are plain means, so a few extreme points can drag them off the dense region of their cluster and inflate inertia. If your data has outliers, either trim them, or use a density-based method that treats them as noise. 2.8. DBSCAN labels outliers explicitly. 2.9. Mean Shift finds modes without a preset k.

Non-globular clusters. k-means partitions space into Voronoi cells around the centroids. So it can recover only roughly convex, similarly sized blobs. Concentric rings or elongated manifolds defeat it, regardless of k. That geometry is what DBSCAN and the manifold methods in 2.12. t-SNE are for.

Too few samples. Requesting more clusters than you have points returns Error::InvalidInput. It does not silently clamp k. If k depends on your data, check n_clusters <= data.nrows() before you fit.

2.7.8. Parallelism and performance

The assignment step is the cost center. It compares every point against every centroid, each iteration, so this is where the parallelism lives. Each iteration computes all point-to-centroid projections as one parallel matrix product (see 6.2. Matrix Multiplication). It then finds each point’s nearest centroid with a short arg-min scan.

The centroid update accumulates each point’s row into its cluster’s running sum. It does this as a deterministic blocked reduction (see 6.3. Parallel Reductions). The k-means++ seeding parallelizes its distance passes the same way.

The word “deterministic” matters here. Each of these stages parallelizes only above a calibrated work threshold. The reduction always sums in a fixed block order, not in thread-arrival order. This is why the parallel path gives the same answer as the serial path. It is also why a seeded fit is reproducible on one machine, regardless of how many threads rayon uses.

Floating-point addition is not associative. Summing in whatever order the threads happen to finish would leak the thread count into the result. Across different machines or build targets, tiny last-bit differences in the arithmetic backend are still possible. Within one machine, a fixed seed gives a fixed answer.

For small inputs, everything runs single-threaded. This avoids rayon’s coordination overhead on data that does not need it. 3 thresholds control the crossover. The f64 scan gate covers the arg-min pass. The f64 sum gate covers the centroid-sum accumulation. The f64 cheap-map gate covers the division of each centroid by its cluster size.

The scan and sum gates live in rustyml::tuning::reduction. The cheap-map gate lives in rustyml::tuning::elementwise. Move them if you are clustering unusual shapes and want to shift a threshold. Most users never touch them. 7.3. Performance Tuning and Parallelism covers these knobs.

Building with the show_progress feature draws a live inertia and iteration bar during fit. This helps on large datasets, and it costs nothing when the feature is off.

2.7.9. Saving and loading a fitted model

KMeans derives serde’s Serialize and Deserialize. The generated save_to_path and load_from_path methods persist the whole model, including the centroids, labels, hyperparameters, and metadata, to a compact postcard binary blob. The file extension does not matter, because the format is always binary postcard. A loaded model predicts identically to the original, down to the last bit of the centroids.

The serialized layout gained the n_init field when restarts were added to the crate. So a blob written by an older version will not load. Re-fit the model and save it again instead.

use rustyml::machine_learning::KMeans;
use ndarray::{array, Array2};
use std::fs::remove_file;

fn main() {
    let data: Array2<f64> = array![
        [0.0, 0.0], [0.1, 0.0], [0.0, 0.1],
        [10.0, 0.0], [10.1, 0.0], [10.0, 0.1],
        [5.0, 10.0], [5.1, 10.0], [5.0, 10.1],
    ];

    let mut km = KMeans::new(3, 300, 1e-4).unwrap().with_random_state(42);
    km.fit(&data).unwrap();
    km.save_to_path("kmeans_model.bin").unwrap();

    let loaded = KMeans::load_from_path("kmeans_model.bin").unwrap();
    let original = km.predict(&data).unwrap();
    let restored = loaded.predict(&data).unwrap();
    assert_eq!(original, restored);

    remove_file("kmeans_model.bin").unwrap();
}

I/O and deserialization problems surface as Error::Io. A missing file on load is the common case. See 7.2. Model Persistence in Depth for the persistence format, versioning concerns, and how it works with the rest of the crate.

2.8. DBSCAN

DBSCAN (Density-Based Spatial Clustering of Applications with Noise) groups points that sit in dense regions. It labels the rest as noise. Unlike KMeans, DBSCAN does not need the number of clusters up front. It finds clusters of any shape, not only round blobs, and it marks outliers explicitly. In exchange, you set 2 density parameters, eps and min_samples, instead of k. This page covers the density model RustyML implements, a method to pick eps, what predict does, and the cost of the O(n^2) algorithm.

2.8.1. The density model: core, border, and noise points

DBSCAN assigns every training point one of 3 roles. The 2 parameters eps (the neighborhood radius) and min_samples (the density threshold) decide the role.

A point’s neighborhood is every point within eps of it. The boundary is inclusive, so a point at distance exactly eps counts as a neighbor. RustyML’s neighborhood query also counts the point itself, since its distance to itself is 0, and 0 is always <= eps. This fact sets the meaning of min_samples:

So min_samples counts the query point itself, the same way scikit-learn counts it. If you count a neighbor as only another nearby point, not the point itself, subtract 1 from min_samples. With min_samples = 2, any point with 1 other point within eps becomes a core point.

Clusters form by density connectivity. fit starts at an unvisited core point, claims it, then floods through the neighborhoods of core points and absorbs every point it reaches. A border point joins the cluster but does not extend the flood, because fit does not expand a border point’s neighborhood. Any point the flood never reaches stays -1.

2 facts follow from this design. First, fit has no random number generator, so the clustering is fully deterministic. fit processes points in ascending row order, and each neighbor list comes back sorted by index. The same input always produces the same labels and the same cluster ids. Second, when a border point sits within reach of 2 clusters, the cluster whose flood reaches the point first claims it. Because fit processes rows in order, that is the cluster with the lower id. This resolves the classic DBSCAN border ambiguity in a deterministic way instead of leaving it undefined.

2.8.2. Constructing and configuring the estimator

The constructor takes the 2 density parameters and validates them:

pub fn new(eps: f64, min_samples: usize) -> Result<Self, Error>
pub fn with_metric(self, metric: DistanceCalculationMetric) -> Result<Self, Error>

new returns Error::InvalidParameter if eps is non-positive or non-finite, or if min_samples is 0. The distance metric defaults to Euclidean. Change it with with_metric, which also returns Result because it validates the Minkowski order (see section 2.8.4). Default::default() gives eps = 0.5, min_samples = 5, and Euclidean. These numbers are placeholders. They are not good defaults for your data.

Getters expose the stored state after construction: get_epsilon, get_min_samples, and get_metric. Once the model is fitted, it also exposes get_labels() -> Option<&Array1<isize>> and get_core_sample_indices() -> Option<&Array1<usize>>. See Error Handling for how the error variants map to real failures.

2.8.3. Fitting and reading the labels

fit runs the clustering and stores the result. fit_predict does the same and also returns the label array. get_labels reads the stored labels after that. Labels have type Array1<isize>. Cluster ids run 0, 1, 2, ... in discovery order, and -1 marks noise. The signed isize type lets DBSCAN store noise in the same array as the cluster ids, with no separate mask.

use rustyml::machine_learning::DBSCAN;
use ndarray::Array2;

fn main() {
    // Two tight blobs plus one isolated point.
    let data = Array2::from_shape_vec(
        (9, 2),
        vec![
            0.0, 0.0, 0.1, 0.0, 0.0, 0.1, 0.1, 0.1, // blob A near the origin
            10.0, 10.0, 10.1, 10.0, 10.0, 10.1, 10.1, 10.1, // blob B near (10, 10)
            5.0, 5.0, // noise: far from both blobs
        ],
    )
    .unwrap();

    let mut dbscan = DBSCAN::new(0.5, 2).unwrap();
    let labels = dbscan.fit_predict(&data).unwrap();

    let n_clusters = labels.iter().filter(|&&l| l >= 0).map(|&l| l).max().map_or(0, |m| m + 1);
    let n_noise = labels.iter().filter(|&&l| l == -1).count();
    println!("labels     = {:?}", labels);
    println!("clusters   = {}", n_clusters);
    println!("noise pts  = {}", n_noise);

    // core_sample_indices holds only the rows that qualified as core points, sorted ascending.
    let cores = dbscan.get_core_sample_indices().unwrap();
    println!("core rows  = {:?}", cores);
}

Blob A becomes cluster 0, because fit discovers it first. Blob B becomes cluster 1. The lone point at (5, 5) stays -1. With min_samples = 2 and 4 points per blob, every blob point is a core point. So core_sample_indices is [0,1,2,3,4,5,6,7], and the noise row is absent.

labels     = [0, 0, 0, 0, 1, 1, 1, 1, -1], shape=[9], strides=[1], layout=CFcf (0xf), const ndim=1
clusters   = 2
noise pts  = 1
core rows  = [0, 1, 2, 3, 4, 5, 6, 7], shape=[8], strides=[1], layout=CFcf (0xf), const ndim=1

fit validates the input before it does any work. A zero-row matrix gives Error::EmptyInput. Any NaN or infinite value in the data gives Error::NonFinite. This shows the split of responsibility. A bad hyperparameter, such as a non-finite eps, gives InvalidParameter at construction. A bad value in the data gives NonFinite at fit time.

2.8.4. Distance metrics

with_metric accepts the 3 variants of DistanceCalculationMetric: Euclidean (the default, L2), Manhattan (L1), and Minkowski(p) (the general Lp norm). Minkowski(2.0) gives the same labels as Euclidean. Minkowski(1.0) gives the same labels as Manhattan. Use the named variants for L1 or L2. Reserve Minkowski for a fractional or higher order.

use rustyml::machine_learning::{DBSCAN, DistanceCalculationMetric};
use ndarray::array;

fn main() {
    let data = array![
        [0.0, 0.0], [0.1, 0.0], [0.0, 0.1],
        [5.0, 5.0], [5.1, 5.0], [5.0, 5.1],
    ];

    let mut dbscan = DBSCAN::new(0.5, 2)
        .unwrap()
        .with_metric(DistanceCalculationMetric::Manhattan)
        .unwrap();
    let labels = dbscan.fit_predict(&data).unwrap();
    println!("{:?}", labels); // two clusters, no noise
}

The constructor rejects a Minkowski order below 1 (including 0.5) or a non-finite order. Such an order breaks the triangle inequality, and that would break the neighborhood test. Changing the metric changes the units of eps, not only its meaning. The same points span a larger Manhattan distance than a Euclidean distance, so an eps tuned for one metric is wrong for another. Retune eps after every metric change. See Distance Metrics for the metric definitions and their tradeoffs.

2.8.5. Choosing eps: the k-distance heuristic

eps is the parameter people set wrong most often. This section gives a concrete method to pick it, instead of trial and error. For each point, measure the distance to its k-th nearest neighbor. Take k = min_samples, and count the point itself, so index 0 is the point at distance 0. Sort all the k-distances in ascending order and look at the curve. Points inside a dense cluster have a small k-distance. Noise points have a large one. The curve stays flat and low across the clustered points, then bends sharply upward at the “knee” as it starts to hit outliers. The k-distance at that knee makes a good eps value. It is large enough to connect real clusters, and small enough to leave outliers isolated.

This sorting step mirrors what a KNN query returns. You can compute the k-distances directly with the public metric dispatcher:

use rustyml::machine_learning::DistanceCalculationMetric;
use ndarray::array;

fn main() {
    // Dense cluster of 5 points plus 2 scattered outliers.
    let data = array![
        [0.0, 0.0], [0.2, 0.1], [0.1, 0.2], [0.3, 0.0], [0.0, 0.3],
        [4.0, 4.0], [8.0, 1.0],
    ];

    let metric = DistanceCalculationMetric::Euclidean;
    let min_samples = 3usize; // k for the k-distance graph
    let n = data.nrows();

    // k-distance of each point: the distance to its k-th nearest neighbor (self included).
    let mut k_dists: Vec<f64> = (0..n)
        .map(|i| {
            let mut d: Vec<f64> = (0..n)
                .map(|j| metric.distance(data.row(i), data.row(j)))
                .collect();
            d.sort_by(|a, b| a.partial_cmp(b).unwrap());
            d[min_samples - 1] // index 0 is self (distance 0)
        })
        .collect();

    k_dists.sort_by(|a, b| a.partial_cmp(b).unwrap());
    // Read the curve from left to right. The sharp rise near the end marks the knee.
    println!("sorted k-distances: {:?}", k_dists);
}

The flat prefix of the printed curve corresponds to the clustered points. Pick eps where the curve turns up. Reserve min_samples for the density floor. A common starting point is 2 * n_features. Raise it for noisy data, because more required neighbors means stricter noise rejection. Lower it toward n_features + 1 for clean, low-dimensional data. RustyML counts the point itself, so min_samples = 1 makes every point a core point. That produces 0 noise points, which is rarely what you want.

2.8.6. What predict does, and why it is not classic DBSCAN

Textbook DBSCAN has no predict method for new points. This gap is fundamental, not an oversight. Cluster membership in DBSCAN is transductive. A point’s label depends on the density of the whole neighborhood. Adding a new point to the data could turn it into a core point. It could also join 2 separate clusters into one, or shift where a border falls. There is no way to label a new point correctly without a new density analysis over the combined set.

RustyML still offers a predict method. It does something narrower and cheaper than a new density analysis. This section states exactly what it does:

use rustyml::machine_learning::DBSCAN;
use ndarray::array;

fn main() {
    let train = array![
        [0.0, 0.0], [0.1, 0.0], [0.0, 0.1], [0.1, 0.1],
        [10.0, 10.0], [10.1, 10.0], [10.0, 10.1], [10.1, 10.1],
    ];
    let mut dbscan = DBSCAN::new(0.5, 2).unwrap();
    dbscan.fit(&train).unwrap();

    // Assign each new point to the cluster of its nearest core point, if within eps.
    let queries = array![
        [0.05, 0.05],   // inside blob A  -> 0
        [10.05, 10.05], // inside blob B  -> 1
        [5.0, 5.0],     // far from all cores -> noise (-1)
    ];
    let preds = dbscan.predict(&queries).unwrap();
    println!("{:?}", preds); // preds is [0, 1, -1], one label per query.
}

predict finds each query’s nearest core point. It searches the core points saved during fit, not the full training set. It returns that core point’s cluster label if the query is within eps. Otherwise, it returns -1. predict picks the single nearest core point. It does not check every core point within eps. The eps gate is inclusive. predict never creates a new cluster, never promotes a query to a core point, and never runs the density flood again. So predict(x) does not give the same result as adding x to the training data and calling fit again. Treat predict as a fast, approximate way to assign held-out points to clusters fit already found. Call fit again on the enlarged set to get true DBSCAN semantics.

Calling predict before fit gives Error::NotFitted. A feature-count mismatch gives Error::DimensionMismatch. A non-finite query value gives Error::NonFinite. Empty input returns an empty array.

2.8.7. Two rings where KMeans fails

Shape is the main reason to choose DBSCAN over KMeans. KMeans partitions space with straight boundaries around k centroids, so it can only carve out convex, roughly round regions. DBSCAN follows density, so it can trace any shape. Two concentric rings show the difference clearly. The rings are not linearly separable around their shared center. KMeans cuts straight through both rings, while DBSCAN walks each ring as a connected chain.

use rustyml::machine_learning::{DBSCAN, KMeans};
use ndarray::Array2;

fn main() {
    // Build 2 concentric rings: inner radius 1, outer radius 4.
    let mut coords: Vec<f64> = Vec::new();
    let inner = 12usize;
    for k in 0..inner {
        let t = k as f64 / inner as f64 * std::f64::consts::TAU;
        coords.push(t.cos());
        coords.push(t.sin());
    }
    let outer = 28usize;
    for k in 0..outer {
        let t = k as f64 / outer as f64 * std::f64::consts::TAU;
        coords.push(4.0 * t.cos());
        coords.push(4.0 * t.sin());
    }
    let data = Array2::from_shape_vec((inner + outer, 2), coords).unwrap();

    // eps covers each ring's neighbor spacing but not the >= 3-unit gap between rings.
    let mut dbscan = DBSCAN::new(1.2, 2).unwrap();
    let db = dbscan.fit_predict(&data).unwrap();
    let db_clusters = db.iter().filter(|&&l| l >= 0).map(|&l| l).max().map_or(0, |m| m + 1);
    let db_noise = db.iter().filter(|&&l| l == -1).count();

    // KMeans with k = 2 (seeded for reproducibility).
    let mut km = KMeans::new(2, 100, 1e-4).unwrap().with_random_state(0);
    let km_labels = km.fit_predict(&data).unwrap();
    // km_inner0 and km_outer0 count how KMeans cluster 0 splits across the 2 rings.
    let km_inner0 = (0..inner).filter(|&i| km_labels[i] == 0).count();
    let km_outer0 = (inner..inner + outer).filter(|&i| km_labels[i] == 0).count();

    println!("DBSCAN: {} clusters, {} noise", db_clusters, db_noise);
    println!("KMeans cluster 0: {} inner-ring + {} outer-ring points", km_inner0, km_outer0);
}

DBSCAN recovers the 2 rings exactly. It finds 2 clusters and 0 noise points, with the inner ring as one label and the outer ring as the other. KMeans splits the plane with a line through the origin. So each of its 2 clusters mixes inner-ring and outer-ring points. KMeans cannot represent a ring at all.

DBSCAN: 2 clusters, 0 noise
KMeans cluster 0: 6 inner-ring + 14 outer-ring points

See Clustering Metrics to score these results with ground-truth labels or an intrinsic metric.

2.8.8. Cost, indexing, and parallelism

Naive DBSCAN runs in O(n^2) time. Every point runs a region query, and a brute-force region query scans all n points. RustyML lowers the constant with a kd-tree. During fit, RustyML builds one kd-tree over the data and answers each region query in about O(log n) average time. This only applies when the data has at most 8 features (DBSCAN_KD_TREE_MAX_DIMS). Above 8 dimensions, a kd-tree stops pruning well, because of the curse of dimensionality (almost every point looks “far” from every other point). So fit falls back to a brute-force scan. Clustering stays correct on that path, only slower. Reduce dimensionality first with PCA or t-SNE before you use high-dimensional data. This speeds up the index, and Euclidean neighborhoods lose meaning in high dimensions regardless of the index.

The brute-force region query runs in parallel across neighbors with rayon. This happens when the scan work (n_samples * n_features) clears a calibrated element gate (262_144 by default). The cluster-expansion loop stays sequential, because it is a flood-fill. So parallelism speeds up each region scan, not the overall control flow. predict runs in parallel over query points when n_queries * n_core_points * n_features clears the same gate. You can tune these gates through the crate::tuning facade, with no recompile needed. See Performance Tuning and Parallelism for how, and Parallel Reductions for the gate mechanism. As a guardrail, if a pathological dataset produces isize::MAX clusters, fit returns Error::Computation instead of an overflow in the label counter.

DBSCAN gives you clusters of any shape, automatic outlier detection, and no k to guess. In exchange, it costs quadratic time in the worst case. The kd-tree lowers this cost in low dimensions but does not remove it. DBSCAN is also sensitive to eps when clusters have different densities. A single global eps cannot fit both a dense cluster and a sparse cluster at once.

2.8.9. Persistence

A fitted DBSCAN serializes with save_to_path and load_from_path. Both methods use the compact postcard binary format. The saved data includes the hyperparameters and the fitted state that predict needs: the stored core points and their labels. A reloaded model predicts the same labels, with no need to see the original training set again.

use rustyml::machine_learning::DBSCAN;
use ndarray::array;

fn main() {
    let data = array![
        [0.0, 0.0], [0.1, 0.0], [0.0, 0.1], [0.1, 0.1],
        [10.0, 10.0], [10.1, 10.0], [10.0, 10.1], [10.1, 10.1],
    ];
    let mut dbscan = DBSCAN::new(0.5, 2).unwrap();
    dbscan.fit(&data).unwrap();

    let path = "dbscan_model.bin";
    dbscan.save_to_path(path).unwrap();

    let loaded = DBSCAN::load_from_path(path).unwrap();
    let preds = loaded.predict(&array![[0.05, 0.05], [10.05, 10.05], [5.0, 5.0]]).unwrap();
    println!("{:?}", preds); // preds is [0, 1, -1], the same labels fit produced.

    std::fs::remove_file(path).unwrap();
}

See Model Persistence in Depth for the format details, versioning caveats, and how persistence interacts with the rest of the crate.

2.9. Mean Shift

Mean Shift is a clustering algorithm. Use it when you do not know how many clusters the data holds, and you do not want to guess a number. Unlike KMeans, which needs k up front, Mean Shift finds the number of clusters from the density of the data. It needs only 1 input from you: the bandwidth. The algorithm moves every seed point uphill on the density surface until it stops at a mode.

The modes that survive become the cluster centers. Their count comes from the data, not from you. This freedom comes at a price: the bandwidth. It is the only setting that controls everything. A wrong bandwidth is the only way to get a bad result.

RustyML exposes the algorithm through MeanShift and a free function, estimate_bandwidth. Both re-export from rustyml::machine_learning. The kernel, the merge rule, the noise label, and the bandwidth estimator all match scikit-learn 1.9.0. On a reference 10-point dataset, the cluster centers, their numbering, and every label match scikit-learn’s output exactly.

2.9.1. Mode seeking with a flat kernel

Picture a kernel density estimate laid over your points. Each sample adds a small bump. Where samples accumulate, the bumps stack into peaks.

Mean Shift places a candidate center at a seed location. It replaces the center with the mean of the data around it. That mean point lies in the direction where density increases. The center climbs to the nearest peak, a mode, over repeated iterations. Repeat the process from many seeds to find every mode.

The kernel decides which points weight that mean, and this detail decides the algorithm’s behavior. RustyML uses the flat kernel, the same kernel scikit-learn uses. For a center c, every point within bandwidth of it counts once. Every point outside it counts zero. The next center is the plain mean of the points inside the ball: mean{ x_i : ||c - x_i|| <= bandwidth }.

The window is a hard ball with radius bandwidth. The number of points it holds at convergence is the mode’s intensity. The merge phase below ranks modes by this intensity.

An earlier version of this crate used a Gaussian kernel weighted over the whole dataset. RustyML removed the Gaussian kernel instead of keeping it as an option. It has no scikit-learn counterpart, so nothing validates it. It also does not produce the window point count the merge rule needs. MeanShift has one kernel, and it has no kernel parameter.

At a very small bandwidth, a shifted center’s ball can end up empty. A naive fix divides by zero and collapses the center to the origin. That would add a false cluster at (0, 0, ...), unrelated to any real point. RustyML avoids this. It leaves the center where it is and stops iterating. This matches the early exit scikit-learn takes for an empty neighborhood.

A point with no neighbors becomes its own mode. Because of this rule, a bandwidth that shrinks toward zero degrades gracefully. The result becomes one cluster per isolated point, not meaningless output.

After every seed converges, MeanShift removes duplicate modes with scikit-learn’s method: intensity-ordered greedy suppression. First, it ranks the converged modes by how many points their window held, from most to least. Then it walks that order. It keeps each mode and discards every other mode within one bandwidth of it.

A kept center is a true density mode. An earlier version averaged the suppressed modes into the kept center instead. That approach dragged the center off the density peak. It also made the result depend on the order in which the pass processed the seeds.

The number of surviving centers is the cluster count. This count comes entirely from the data and the bandwidth. You never state the cluster count directly. Each input sample gets the label of its nearest surviving center.

This 2-phase structure, first converge many seeds and then merge them, explains why a bandwidth that is slightly too large still gives clean results. Even when seeds converge to slightly different spots, the merge step still combines them.

2.9.2. Constructing a MeanShift

MeanShift::new takes the bandwidth and returns a Result. A bandwidth that is not positive and finite is a usage error. RustyML does not clamp it silently:

let ms = MeanShift::new(2.0)?            // the only required argument
    .with_max_iter(300)?                 // returns Result, validates > 0
    .with_tolerance(1e-4)?               // returns Result, validates positive and finite
    .with_bin_seeding(true)              // returns Self, infallible toggle
    .with_cluster_all(true);             // returns Self, infallible toggle

The split in return types is deliberate, and it is easy to miss. The 2 convergence setters validate their argument and return Result<Self, Error>. They need the ? operator. The 2 boolean toggles cannot fail and return Self. You can chain them directly.

MeanShift::default() equals new(1.0) with every other setting at its default. This is convenient for a first look. It is rarely the bandwidth you actually want. See 2.9.4 for more.

Invalid arguments come back as Error::InvalidParameter from new, with_max_iter, and with_tolerance. This is the same error type described in 1.6. Error Handling.

2.9.3. Fitting, predicting, and reading the results

fit takes a 2-D array, one sample per row. It runs the algorithm and returns &mut Self. predict maps new points to the learned centers and returns Array1<isize>. fit_predict does both, and returns the training labels directly.

The labels are signed because -1 is reserved for noise, the same convention DBSCAN and scikit-learn use. This convention lets the output of any clustering estimator feed any metric in 5.3. Clustering Metrics without a conversion. After fit, getters expose everything the model found.

use ndarray::Array2;
use rustyml::machine_learning::MeanShift;

fn main() {
    // 2 tight blobs: 5 points near (0, 0) and 5 near (20, 20).
    let data = Array2::from_shape_vec(
        (10, 2),
        vec![
            -0.1, 0.0, 0.1, 0.0, 0.0, -0.1, 0.0, 0.1, 0.0, 0.0, // blob A
            19.9, 20.0, 20.1, 20.0, 20.0, 19.9, 20.0, 20.1, 20.0, 20.0, // blob B
        ],
    )
    .unwrap();

    let mut ms = MeanShift::new(2.0).unwrap();
    let labels = ms.fit_predict(&data).unwrap();

    let centers = ms.get_cluster_centers().unwrap();
    println!("clusters found: {}", centers.nrows()); // emerges from the data: 2
    println!("labels: {:?}", labels);
    println!("samples per center: {:?}", ms.get_n_samples_per_center().unwrap());
    println!("iterations run: {}", ms.get_actual_iterations().unwrap());
}

The getters fall into 2 groups: results and echoed settings. get_cluster_centers returns Option<&Array2<f64>>, one row per cluster. get_labels returns Option<&Array1<isize>>. get_n_samples_per_center returns Option<&Array1<usize>>, the number of input samples assigned to each center. With cluster_all = true, these counts sum to the sample count. With cluster_all = false, the -1 outliers are excluded from the sum.

get_actual_iterations returns Option<usize>, the largest iteration count over all seeds. Use it to tell whether the run converged or hit max_iter. All 4 of these getters return None before fit and Some after.

The remaining getters, get_bandwidth, get_max_iterations, get_tolerance, get_bin_seeding, and get_cluster_all, simply read back the configuration.

predict has 4 failure modes. Calling it before fit returns Error::NotFitted. Passing an empty array returns Error::EmptyInput. Passing points whose feature count differs from the training data returns Error::DimensionMismatch. Passing data with NaN or infinite values returns Error::NonFinite. None of these errors are recoverable by retrying, so treat them as programming errors that surface at runtime.

2.9.4. Bandwidth: the hyperparameter that decides everything

Everything about a Mean Shift run flows from the bandwidth. It sets the radius of the flat kernel’s window. It also sets the merge radius, and the cutoff for what counts as an outlier when cluster_all is off. The bandwidth controls how far each seed can see, how strongly nearby modes collapse into one, and where noise begins.

A bandwidth too small causes over-segmentation. Seeds cannot reach across a cluster’s own spread, and modes multiply. In the extreme case, every well-separated point becomes its own cluster. A bandwidth too large causes under-segmentation instead. Distant blobs pull on each other until their modes drift together, and the merge step fuses them into one. Eventually the whole dataset becomes a single cluster.

No default bandwidth works for arbitrary data. The correct value is a length scale in the units of your features.

When you have no prior estimate, estimate_bandwidth gives a data-driven starting point. It takes the data and an optional quantile (default 0.3). It also takes an optional n_samples subsample size (default: all rows, clamped to the dataset size) and an optional random_state.

It computes k = max(1, floor(n * quantile)). Then it measures each point’s distance to its (k - 1)-th nearest neighbor, and returns the mean of those distances. This is a local-density statistic. It answers how far a typical point is from the edge of its own neighborhood, which is exactly what a bandwidth needs to be.

The (k - 1) term reproduces scikit-learn’s off-by-one behavior, because its neighbor query counts the query point itself. The result agrees with scikit-learn 1.9.0 to within 1e-14. A neighborhood of one point yields 0.0, the same result scikit-learn gives.

An earlier version returned a quantile of the whole pairwise-distance distribution instead. That is a global spread measure. It runs far larger than a bandwidth should on clustered data. It collapsed everything into a single cluster. If you tuned a bandwidth against the old estimator, re-estimate it now.

A quantile of 0.3 gives a typical short-to-medium neighborhood radius. This value tends to land near the intra-cluster scale. Larger quantiles bias toward larger bandwidths and fewer clusters.

use ndarray::Array2;
use rustyml::machine_learning::{MeanShift, estimate_bandwidth};

fn main() {
    // 3 well-separated blobs, 12 tight points each.
    let mut v: Vec<f64> = Vec::new();
    for (cx, cy) in [(0.0, 0.0), (10.0, 0.0), (5.0, 9.0)] {
        for k in 0..12u32 {
            v.push(cx + ((k * 7) % 5) as f64 * 0.05 - 0.1);
            v.push(cy + ((k * 3) % 5) as f64 * 0.05 - 0.1);
        }
    }
    let data = Array2::from_shape_vec((36, 2), v).unwrap();

    // A reasonable starting point straight from the data.
    let bw = estimate_bandwidth(&data, Some(0.3), None, Some(0)).unwrap();
    println!("estimated bandwidth: {:.3}", bw);

    // Sweep: small over-segments, large collapses everything into one.
    for bandwidth in [0.05_f64, 0.5, 3.0, 30.0] {
        let mut ms = MeanShift::new(bandwidth).unwrap();
        ms.fit(&data).unwrap();
        let k = ms.get_cluster_centers().unwrap().nrows();
        println!("bandwidth {bandwidth:>5} -> {k} clusters");
    }
}

The cluster count moves with the bandwidth in the direction you would expect. Because the exact counts depend on the data, treat the following as the shape of the output, not literal numbers:

estimated bandwidth: <small positive value>
bandwidth  0.05 -> many clusters      (blobs fragment; over-segmentation)
bandwidth   0.5 -> one cluster per blob
bandwidth     3 -> one cluster per blob
bandwidth    30 -> a single cluster   (all blobs merged; under-segmentation)

Follow a simple workflow. Call estimate_bandwidth once. Fit at that value. Look at the cluster count. Then increase the bandwidth if you got too many clusters, or decrease it if you got too few. Validate the choice with a metric that does not need ground-truth labels, such as the silhouette score in 5.3. Clustering Metrics.

estimate_bandwidth measures distances directly. Standardize your features first (see 4.2. Standardization and Normalization) to make the estimate meaningful when feature scales differ.

2.9.5. Bin seeding, cluster_all, and the outlier label

By default, every input point is a seed. This is the most thorough option. As 2.9.7 explains, it is also why fitting is deterministic. On dense datasets, using every point as a seed is wasteful, because thousands of seeds inside one blob all climb to the same mode.

with_bin_seeding(true) fixes this. It quantizes the feature space onto a grid whose cells have side length bandwidth, then keeps 1 representative seed per occupied cell. Fewer seeds mean fewer uphill walks and a faster fit.

The cost is a coarser, approximate seeding. A mode whose basin never contains a grid representative can be missed. Bin seeding is worth the trade when the seed loop dominates the run time and the data is dense enough to fill whole cells. On small or sparse data, it saves little, and it can only cost resolution.

cluster_all decides what happens to points that do not really belong to any mode. With the default true, MeanShift forces every point to its nearest center. Labels always fall in 0..n_clusters, and there is no concept of noise. Set it to false, and any point farther than one bandwidth from every center gets the label -1. This is scikit-learn’s noise value, the same value DBSCAN uses. The rule applies to the training labels from fit and to fresh points from predict.

The -1 label replaced an older sentinel value equal to n_clusters. Downstream code that counted distinct labels read that old sentinel as a real extra cluster. If your code compares a label against the cluster count, change it to label < 0.

use ndarray::Array2;
use rustyml::machine_learning::MeanShift;

fn main() {
    let data = Array2::from_shape_vec(
        (10, 2),
        vec![
            -0.1, 0.0, 0.1, 0.0, 0.0, -0.1, 0.0, 0.1, 0.0, 0.0,
            19.9, 20.0, 20.1, 20.0, 20.0, 19.9, 20.0, 20.1, 20.0, 20.0,
        ],
    )
    .unwrap();

    let mut ms = MeanShift::new(2.0).unwrap().with_cluster_all(false);
    ms.fit(&data).unwrap();

    // (10, 10) sits ~14 units from both blobs, far beyond the bandwidth of 2.0.
    let probe = Array2::from_shape_vec((1, 2), vec![10.0, 10.0]).unwrap();
    let pred = ms.predict(&probe).unwrap();

    if pred[0] < 0 {
        println!("outlier: label {}", pred[0]); // -1
    } else {
        println!("assigned to cluster {}", pred[0]);
    }
}

When you enable cluster_all = false, test for label < 0. Do not assume labels are dense. A naive labels.iter().max() no longer tells you the cluster count if every point happens to be noise. get_cluster_centers().unwrap().nrows() gives the reliable count.

2.9.6. Cost, convergence, and parallelism

Mean Shift has quadratic cost. Consider this before you run it on a large dataset. Each seed’s iteration touches all n points in d dimensions, to check which points fall inside the window and to average them. One seed costs O(iterations * n * d). With the default seeding, every point is a seed, so there are n seeds. A full fit costs roughly O(iterations * n^2 * d).

This is the same asymptotic class as DBSCAN’s pairwise scan. It is considerably heavier than KMeans’ O(iterations * n * k * d), whose k is usually far below n. Bin seeding cuts the seed count from n to the number of occupied cells. This reduces the constant, but it does not change the quadratic per-iteration term.

The convergence bound is max_iter (default 300). A seed stops early once its shift falls below tol. get_actual_iterations reports the largest count any seed needed. A value pinned at max_iter signals that some seed never settled.

The implementation runs in parallel only when doing so helps. In fit, the per-seed uphill walks are independent. They run across a Rayon pool once the total work (seeds times samples times features) clears RustyML’s calibrated scan-class gate. This gate defaults to 262,144 element-operations, and you can adjust it through crate::tuning. Below that gate, the loop stays serial, because the overhead of forking is not worth it.

Inside each seed, the implementation casts the weighted-mean computations as matrix-vector products. It deliberately keeps those serial when the seed axis alone already fills the pool. This avoids nested Rayon forks that would compete with each other. predict parallelizes its nearest-center scan under the same gate, keyed on samples times clusters times features.

For most datasets, you get parallelism automatically, without changing any setting. The tuning knobs, and the reasoning behind these thresholds, live in 7.3. Performance Tuning and Parallelism.

2.9.7. Reproducibility and persistence

Fitting a MeanShift is deterministic. It takes no random seed. With the default seeding, it starts from every point. With bin seeding, it starts from a fixed grid representative per cell. Neither approach draws from a random number generator, so two fits on identical data produce byte-identical centers and labels.

This is a real convenience over KMeans, whose centroid initialization is randomized and needs a seed to reproduce. See 7.1. Reproducibility and Random Seeds for the broader picture.

The only place randomness enters this module is estimate_bandwidth, and only when you ask it to subsample. If n_samples is smaller than the dataset, estimate_bandwidth shuffles indices to pick the subset. Pass a fixed random_state when you need the estimate itself to be reproducible. Requesting all rows, the default for n_samples, removes the randomness entirely, because there is nothing to sample.

A fitted model serializes to a compact postcard binary through save_to_path, and restores with load_from_path. The saved file carries the centers, labels, hyperparameters, and training metadata. A reloaded model predicts identically without a re-fit.

use ndarray::Array2;
use rustyml::machine_learning::MeanShift;

fn main() {
    let data = Array2::from_shape_vec(
        (6, 2),
        vec![0.0, 0.0, 0.1, 0.1, -0.1, 0.0, 10.0, 10.0, 10.1, 9.9, 9.9, 10.0],
    )
    .unwrap();

    let mut ms = MeanShift::new(2.0).unwrap();
    ms.fit(&data).unwrap();

    let path = "mean_shift_model.bin";
    ms.save_to_path(path).unwrap();
    let restored = MeanShift::load_from_path(path).unwrap();

    let before = ms.predict(&data).unwrap();
    let after = restored.predict(&data).unwrap();
    assert_eq!(before, after); // identical after the round trip

    std::fs::remove_file(path).unwrap();
    println!("round-trip predictions match");
}

One caveat applies to model files saved before the kernel and merge rule changed. The centers inside those files came from the old Gaussian-kernel run. A reloaded model clusters differently from a fresh fit on the same data. This happens silently, because nothing about the file announces which algorithm produced it. Re-fit the model and save it again.

Persist the model when the fit is expensive and the data is stable. Downstream services can then load and predict cheaply. 7.2. Model Persistence in Depth covers the format details and versioning caveats.

2.10. Principal Component Analysis

Principal Component Analysis (PCA) finds an orthonormal set of directions in feature space. These directions are the principal axes. The first axis captures the most variance in the data. The second axis is orthogonal to the first and captures the most of the remaining variance. Each later axis follows the same pattern.

Projecting the data onto the top k axes gives the k-dimensional linear subspace that keeps the most variance. This is the same subspace that gives the smallest squared reconstruction error. The subspace that keeps the most variance also discards the least information. This dual property is why PCA is a common first step to compress, denoise, or plot high-dimensional data.

RustyML’s PCA mirrors sklearn.decomposition.PCA. Fit it on a feature matrix. Use transform to get scores. Use inverse_transform to map scores back to feature space.

RustyML’s PCA differs from scikit-learn’s PCA in 3 ways. First, it centers the data but never scales it. The values you feed it matter more than in a pipeline that standardizes for you.

Second, the solver setting is an enum named SVDSolver, with 3 concrete strategies. The right choice depends on the number of features and the number of components you want.

Third, every factorization runs on hand-written pure Rust. There is no LAPACK and no BLAS. This makes the tradeoffs between solvers concrete rather than theoretical.

2.10.1. The surface you work with

PCA and SVDSolver live under rustyml::machine_learning::decomposition. Both types are also available through the prelude. The model is unsupervised. It takes an f64 feature matrix with samples as rows and features as columns. It uses no labels.

// Construction. n_components must be > 0, checked here.
PCA::new(n_components: usize) -> Result<PCA, Error>
PCA::default()                                         // n_components = 2, Full solver
pca.with_svd_solver(solver: SVDSolver) -> PCA          // builder, consumes and returns self

// Learning and projecting. fit takes &mut self.
pca.fit(&x)                -> Result<&mut PCA, Error>
pca.transform(&x)          -> Result<Array2<f64>, Error>   // (n_samples, n_components)
pca.fit_transform(&x)      -> Result<Array2<f64>, Error>
pca.inverse_transform(&scores) -> Result<Array2<f64>, Error> // (n_samples, n_features)

// Fitted state. mean, components, variances, and the sample/feature counts are None before
// fit, Some(&...) after. n_components and svd_solver are always available.
pca.get_mean()                     -> Option<&Array1<f64>>   // per-feature centering mean
pca.get_components()               -> Option<&Array2<f64>>   // (n_components, n_features), rows are axes
pca.get_explained_variance()       -> Option<&Array1<f64>>
pca.get_explained_variance_ratio() -> Option<&Array1<f64>>
pca.get_singular_values()          -> Option<&Array1<f64>>
pca.get_n_components() -> usize
pca.get_svd_solver()   -> SVDSolver
pca.get_n_samples()    -> Option<usize>
pca.get_n_features()   -> Option<usize>

The constructor checks only that n_components > 0. The harder bound, n_components <= min(n_samples, n_features), is checked at fit time, because it depends on the data. Use the getters to check whether the model is fitted. Each one returns None until a successful fit call sets it.

get_components returns the axes as rows of an (n_components, n_features) matrix. This is the same layout as scikit-learn’s components_. So transform is centered data times components.T, and inverse_transform is scores times components.

This example runs the full loop on a small 2-D data set and keeps 1 component:

use ndarray::array;
use rustyml::machine_learning::decomposition::PCA;

fn main() {
    let x = array![
        [2.5, 2.4],
        [0.5, 0.7],
        [2.2, 2.9],
        [1.9, 2.2],
        [3.1, 3.0],
        [2.3, 2.7],
    ];

    // fit_transform needs &mut self. It runs fit, then transform, on the same data.
    let mut pca = PCA::new(1).unwrap();
    let scores = pca.fit_transform(&x).unwrap();

    println!("scores shape: {:?}", scores.shape()); // [6, 1]
    println!("mean:         {:?}", pca.get_mean().unwrap());
    println!("component:    {:?}", pca.get_components().unwrap());
    println!("variance:     {:?}", pca.get_explained_variance().unwrap());
    println!("ratio:        {:?}", pca.get_explained_variance_ratio().unwrap());
    println!("singular:     {:?}", pca.get_singular_values().unwrap());
}

fit_transform(&x) is not a fused shortcut that gives a different result. It calls fit, then transform, on the same matrix. This is identical to running the two steps by hand. Use the two-step form to project a different matrix, for example new samples, through an already-fitted model. fit returns &mut Self, so you can chain a getter call directly onto it. The normal path, though, is to read state back with the get_* methods.

2.10.2. It centers, it does not scale

Before it decomposes the data, fit subtracts the per-feature mean and stores it in get_mean(). This is the only preprocessing step fit runs. It does not divide by the standard deviation. It does not whiten the data. It does not touch the units of the columns.

transform reuses the stored training mean to center new samples. inverse_transform adds the mean back. The mean is part of the fitted model, not a value computed fresh on each call.

Skipping this step is the most common cause of a misleading PCA result. Variance is not scale-invariant. Converting a feature from meters to millimeters multiplies its values by 1000, so its variance grows by 1000^2, a million times. PCA then assigns its first component to that one column.

The fix is to standardize the features before fitting. Standardizing puts every column on unit variance, so PCA compares them on equal footing. Use standardize with StandardizationAxis::Column. The next example shows the difference:

use ndarray::array;
use rustyml::machine_learning::decomposition::PCA;
use rustyml::utils::standardize::{standardize, StandardizationAxis};

fn main() {
    // Feature 0 uses tiny units. Feature 1 uses huge units. They carry the same
    // information, but very different variance.
    let x = array![
        [0.1, 1000.0],
        [0.2, 1005.0],
        [0.3, 1002.0],
        [0.4,  998.0],
        [0.5, 1010.0],
    ];

    let mut raw = PCA::new(2).unwrap();
    raw.fit(&x).unwrap();
    println!("raw ratios:          {:?}", raw.get_explained_variance_ratio().unwrap());

    let xs = standardize(&x, StandardizationAxis::Column).unwrap();
    let mut scaled = PCA::new(2).unwrap();
    scaled.fit(&xs).unwrap();
    println!("standardized ratios: {:?}", scaled.get_explained_variance_ratio().unwrap());
}

On the raw data, the first ratio is close to 1.0. Column 1’s numeric spread swamps the signal from column 0, so PC1 points almost exactly along column 1. After column standardization, the two features contribute about equally. The ratios then reflect the real correlation structure between them.

Whether to standardize is a modeling decision. If the features already share the same meaningful unit, for example pixel intensities, centering alone is enough. Make this decision on purpose, because rustyml does not make it for you. Statisticians describe this choice as PCA on the covariance matrix versus PCA on the correlation matrix. Standardizing first is the same as running PCA on the correlation matrix.

2.10.3. Choosing a solver, and the pure-Rust machinery behind it

SVDSolver selects how PCA computes the components. The name is not fully accurate. Only 1 of the 3 variants literally forms an SVD. The interface stays the same across all 3 variants. Whichever solver you choose, you get the same component rows, explained variances, and singular values. Each solver computes them through a different route, with a different cost and a different accuracy.

pub enum SVDSolver {
    Full,             // default: exact eigendecomposition of the covariance matrix
    Randomized(u64),  // randomized range finder, seeded by the u64
    PowerIteration,   // deflated power iteration for the top-k eigenpairs
}

The cost of each solver follows a clear pattern. Full and PowerIteration both build the dense d x d covariance. Both pay O(n * d^2) to form it and O(d^2) memory to hold it. The difference is what happens next. Full runs a complete O(d^3) eigendecomposition.

PowerIteration extracts k eigenpairs one at a time, using a bounded number of matrix-vector products for each, and deflates each one before finding the next. This wins on compute time when k is far smaller than d.

Randomized never builds a d x d matrix at all. Its working set is the n x k sketch and the k x d reduced projection. This makes it the memory-frugal choice when d is large. On small problems, all 3 solvers agree to several digits. The approximate solvers only show their speed advantage, or their numerical drift, once the matrices grow large.

use ndarray::array;
use rustyml::machine_learning::decomposition::{PCA, SVDSolver};

fn main() {
    let x = array![
        [2.5, 2.4],
        [0.5, 0.7],
        [2.2, 2.9],
        [1.9, 2.2],
        [3.1, 3.0],
        [2.3, 2.7],
        [2.0, 1.6],
        [1.0, 1.1],
    ];

    for solver in [SVDSolver::Full, SVDSolver::Randomized(42), SVDSolver::PowerIteration] {
        let mut pca = PCA::new(1).unwrap().with_svd_solver(solver);
        pca.fit(&x).unwrap();
        let sigma = pca.get_singular_values().unwrap();
        println!("{:?}: sigma_1 = {:.6}", solver, sigma[0]);
    }
}

PowerIteration and Randomized expose only a small part of their internal settings. PowerIteration has no public setting for iteration count or tolerance. Internally, it runs up to 1000 iterations per component, to a 1e-6 eigenvalue tolerance, with a fixed seed. None of these values is tunable through the public API.

PowerIteration can also fail to converge. This can happen when the data has less effective rank than the number of components you request. For example, duplicate columns or an exact linear dependency between features can cause this. When the deflation step runs out of variance to extract, fit returns Error::NotConverged. Full and Randomized never fail this way, because neither depends on a per-component convergence check.

Randomized exposes only its seed. The oversampling amount and the number of subspace iterations are fixed. So the full configuration surface for PCA is n_components plus with_svd_solver. This is small on purpose.

All 3 solvers run on the crate’s own machine_learning::linalg module. This module reimplements symmetric eigendecomposition, SVD (one-sided Jacobi), and QR (modified Gram-Schmidt) directly on ndarray arrays. There is no LAPACK dependency to install, link, or vendor. This matches the pure-Rust numerics approach the rest of the crate takes. The covariance and projection multiplies use the parallel gemmkit backend.

2.10.4. Reading the variance and choosing n_components

After a fit, 3 vectors describe how the variance spreads across the kept axes. get_singular_values returns the singular values, in descending order, of the centered data. get_explained_variance returns each singular value squared, divided by (n - 1). This is the actual variance along each component, in the units of the original data.

get_explained_variance_ratio divides each of those values by the total variance of the centered data. That total is the full trace, summed over all features, not just the ones you kept. So each entry in the ratio is the fraction of the overall variance that component explains.

This denominator choice matters. If you keep fewer components than features, the ratios sum to less than 1. The shortfall is exactly the variance you discarded.

This property makes the ratio the right tool for choosing n_components. The standard method uses a scree plot and a cumulative rule. Fit the full-rank decomposition once. Read the cumulative ratio. Keep the smallest number of axes that clears a variance target, for example 90%, 95%, or 99%.

use ndarray::array;
use rustyml::machine_learning::decomposition::PCA;

fn main() {
    // 8 samples, 3 features. Feature 2 is almost a linear echo of feature 0,
    // so the data is effectively low-rank.
    let x = array![
        [1.0, 0.2, 1.19],
        [2.0, 0.1, 2.09],
        [3.0, 0.4, 3.38],
        [4.0, 0.2, 4.19],
        [5.0, 0.5, 5.48],
        [6.0, 0.3, 6.29],
        [7.0, 0.1, 7.09],
        [8.0, 0.6, 8.58],
    ];

    // Fit the full decomposition (n_components = min(n_samples, n_features) = 3),
    // then decide how many axes to actually keep.
    let mut pca = PCA::new(3).unwrap();
    pca.fit(&x).unwrap();

    let ratio = pca.get_explained_variance_ratio().unwrap();
    let mut cumulative = 0.0;
    for (i, r) in ratio.iter().enumerate() {
        cumulative += r;
        println!("PC{}: individual = {:.4}, cumulative = {:.4}", i + 1, r, cumulative);
    }

    // Smallest k whose cumulative ratio clears 95%.
    let mut acc = 0.0;
    let mut k = ratio.len();
    for (i, r) in ratio.iter().enumerate() {
        acc += r;
        if acc >= 0.95 {
            k = i + 1;
            break;
        }
    }
    println!("keep {} component(s) to retain >= 95% variance", k);
}

One detail matters here. You fit once at full rank to read the spectrum. Then refit with the chosen k if you want the smaller projection. Refitting is cheap, and it gives you a model whose transform outputs exactly k columns.

If refitting is expensive for the data, use a shortcut instead. The first k component rows of the full fit are the same axes as a k-component fit would produce. The only difference is the sign convention, covered in the next section. So you can slice the full fit rather than refit it. Do not pick k from a single ratio in isolation. Read the cumulative curve instead, to see where the returns flatten out.

2.10.5. Reconstruction and inverse_transform

inverse_transform maps scores back to feature space, using the formula reconstructed = scores * components + mean. When you keep all min(n_samples, n_features) components, the round-trip is lossless, up to floating-point error. When you keep fewer components, the reconstruction is the orthogonal projection of the data onto the retained subspace. The residual is the variance in the axes you dropped. Measuring that residual is a direct way to put a number on the cost of compression.

use ndarray::array;
use rustyml::machine_learning::decomposition::PCA;

fn main() {
    let x = array![
        [1.0, 0.2, 1.19],
        [2.0, 0.1, 2.09],
        [3.0, 0.4, 3.38],
        [4.0, 0.2, 4.19],
        [5.0, 0.5, 5.48],
        [6.0, 0.3, 6.29],
    ];

    // Keep a single axis, then round-trip back into the original 3-D feature space.
    let mut pca = PCA::new(1).unwrap();
    pca.fit(&x).unwrap();

    let scores = pca.transform(&x).unwrap();                     // (6, 1)
    let reconstructed = pca.inverse_transform(&scores).unwrap(); // (6, 3)

    // Frobenius norm of the residual = the variance thrown away with PC2 and PC3.
    let err: f64 = (&reconstructed - &x).iter().map(|d| d * d).sum::<f64>().sqrt();
    println!("scores shape:         {:?}", scores.shape());
    println!("reconstruction shape: {:?}", reconstructed.shape());
    println!("reconstruction error: {:.6}", err);
}

Watch the shape contract, because it reverses transform. transform takes n_features columns and returns n_components columns. inverse_transform takes n_components columns and returns n_features columns. Pass it a matrix whose column count does not match n_components, and you get Error::DimensionMismatch, not a silent broadcast.

The sign fix, covered in the next section, flips a whole axis together with its scores. This makes reconstruction invariant to it. The product scores * components stays the same, whether or not an axis was negated. A sign flip never corrupts a round-trip.

2.10.6. Determinism: sign convention and seeds

Eigenvectors and singular vectors are only defined up to sign. -v spans the same axis as v. scikit-learn users know this as the reason PC signs sometimes flip between runs or between library versions.

RustyML fixes the sign after the decomposition, so the result stays deterministic. Each component row is negated, if needed, so its largest-magnitude loading becomes non-negative. As a result, all 3 solvers agree on the orientation of every axis on the same data. Repeated runs of any one solver also agree. The sign stays stable, and fit is reproducible with no extra step.

This has 2 consequences. First, the sign convention is specific to rustyml. It keys off the component vectors themselves, not off the U factor the way scikit-learn’s svd_flip does. So a component’s sign may differ from what scikit-learn prints for the same data. This difference is cosmetic. The axis, the variance it explains, and every reconstruction stay identical.

Second, the only real source of run-to-run variation is SVDSolver::Randomized. Its random sketch is seeded by the u64 you pass. The same seed gives bit-identical output. A different seed gives a slightly different approximate subspace. Full and PowerIteration carry no external randomness. PowerIteration seeds its own starting vector internally, with a fixed value.

For exact reproducibility across a pipeline, see Reproducibility and Random Seeds. The only lever here is the Randomized seed.

A fitted PCA serializes with save_to_path and load_from_path. These methods write and read the whole model (mean, components, variances, and singular values) as compact postcard binary. The file extension does not matter. The bytes stay postcard format regardless of the file name. A reloaded model transforms data identically to the original model:

pca.save_to_path("pca_model.bin")?;
let loaded = PCA::load_from_path("pca_model.bin")?;
let scores = loaded.transform(&x_new)?; // identical to the pre-save model

For the mechanics, including versioning concerns and when the binary format is safe for long-term storage, see Model Persistence in Depth.

2.10.7. Errors and edge cases

PCA validates its input and returns typed errors instead of panicking. The .unwrap() calls in these examples keep the code short. Do not use .unwrap() this way in production code. The table below lists the errors you can hit in practice, plus the one that is specific to the PowerIteration solver:

The NotConverged case is specific to PowerIteration. Full and Randomized never raise it, because neither depends on a strict per-component convergence check.

The 2-sample minimum is not arbitrary. Variance needs an n - 1 denominator, so a single row has nothing to decompose. The n_components ceiling of min(n_samples, n_features) is the rank bound. You cannot extract more orthogonal directions than the data spans.

Asking for more raises a parameter error, instead of a silently truncated result. rustyml checks this bound against the data, not against the model alone. So the same PCA::new(5) instance can succeed on a 100 x 20 matrix and fail on a 3 x 4 one.

PCA is linear by construction. It can only find directions that are linear combinations of the input features. Structure that lives on a curved manifold stays invisible to it. When a scree plot does not flatten, and reconstruction stays poor at every k, this is usually a signal to move to a nonlinear method.

Kernel PCA applies the same variance-maximizing idea in a kernel feature space. t-SNE is the tool for 2-D visualization of nonlinear neighborhood structure. If the goal is to separate labeled classes, rather than to capture raw variance, use the supervised counterpart, Linear Discriminant Analysis. It projects the data toward class separation instead of toward total spread.

2.11. Kernel PCA

KernelPCA runs PCA in an implicit feature space. It does not decompose the covariance of your data. Instead, it decomposes a centered kernel (Gram) matrix. This finds nonlinear structure that a linear projection cannot see.

Kernel PCA works with pairwise kernel values between samples. Its working object is an n x n matrix, not a d x d matrix. This fact alone drives the memory cost, the solver choice, and the main limitation: Kernel PCA has no inverse_transform.

This page is based on src/machine_learning/decomposition/kernel_pca.rs, the shared kernel types in src/machine_learning/types.rs, and the integration tests in tests/machine_learning/kernel_pca.rs.

The kernel machinery, the KernelType enum and the Gamma coefficient, is the same code that drives Support Vector Machines. Kernel PCA reuses this code unchanged. It does not define its own kernel types.

2.11.1. The kernel trick, and why plain PCA cannot see rings

PCA finds the directions of maximum variance in the input space. This works well when the structure is linear. It fails when the structure is not linear.

The classic failure case is 2 concentric rings. The inner ring has radius 0.5. The outer ring has radius 3.0. Radius alone separates the 2 classes perfectly. No straight line separates them, so no linear projection can either.

Feed these points to PCA, and the leading components capture only the angular spread of the outer ring. The radial information that distinguishes the classes never appears.

The kernel trick maps each point x through a nonlinear feature map phi(x). This maps x into a much higher-dimensional space. Ordinary PCA then runs in that space. In this lifted space, the rings can become linearly separable.

Kernel PCA never builds phi(x) directly. This is the trick. PCA in the lifted space needs only inner products of the form phi(xi) * phi(xj). A kernel function K(xi, xj) computes this inner product directly, without building phi.

For the RBF kernel, K(x, y) = exp(-gamma * ||x - y||^2). Its implicit feature space has infinite dimensions. Each kernel value is still a single scalar, and you can compute it directly.

The RBF value depends only on the distance between 2 points. This encodes the radial structure that plain PCA discards. This is why the RBF kernel splits the 2 rings. 2.11.7 has a runnable example.

2.11.2. Double centering: the non-obvious core

PCA in the lifted space needs centered features: phi_centered(xi) = phi(xi) - (1/n) * sum_k phi(xk). You cannot subtract this mean directly, because you never have phi in hand.

The object Kernel PCA actually decomposes is the matrix of inner products of the centered features. You can write these inner products entirely in terms of the raw kernel matrix K. Expanding phi_centered(xi) * phi_centered(xj) gives the double-centering identity:

Kc[i, j] = K[i, j] - row_means[i] - row_means[j] + overall_mean

row_means[i] is the mean of row i of K. overall_mean is the mean of the whole matrix. In matrix form, this is Kc = H * K * H. H is the centering matrix H = I - (1/n) * J. I is the identity matrix, and J is the n x n all-ones matrix.

This is called double centering because H applies on both sides of K. It subtracts the row mean and the column mean, then adds the grand mean back, so the formula does not remove it twice. A custom Kernel PCA implementation often misses that + overall_mean term. Missing it biases every projection.

fit implements this. It computes the per-row means and the overall mean of the training kernel matrix (kernel_means). It then rewrites each entry in place as K[i,j] - row_mean[i] - row_mean[j] + overall_mean (center_kernel_matrix).

H * K * H has a direct consequence. Every row of Kc sums to zero, so every column of the resulting projection has a mean of zero. The test test_centering_training_output_has_near_zero_column_means checks that each projected component averages within 1e-9 of zero.

New points need a different, asymmetric formula. Many naive implementations give up here and refuse out-of-sample transforms.

When you project a new sample, its cross-kernel row against the training set needs the training statistics, not its own. center_cross_kernel_matrix subtracts the training row means and the mean of the new row itself, then adds the training overall mean. RustyML implements this, so transform on unseen data works correctly. See 2.11.6.

2.11.3. Constructing the estimator

The constructor takes the kernel and the number of components. It validates both and returns a Result:

pub fn new(kernel: KernelType, n_components: usize) -> Result<Self, Error>
pub fn with_eigen_solver(self, eigen_solver: EigenSolver) -> Self

n_components == 0 gets rejected right away, with Error::InvalidParameter naming the field. The relationship n_components <= n_samples cannot be checked at construction time, because the sample count is not known yet. fit enforces it instead, again with Error::InvalidParameter. A failed fit leaves the model untouched. The test test_fit_n_components_greater_than_n_samples_returns_invalid_parameter confirms the fitted-state getters stay None after a failed fit. You never end up with a half-mutated estimator.

The eigen solver defaults to EigenSolver::Dense. Set it with the chaining builder with_eigen_solver. The Default implementation gives you an RBF kernel with gamma = 0.1, n_components = 2, and the dense solver:

use rustyml::machine_learning::decomposition::kernel_pca::{EigenSolver, KernelPCA};
use rustyml::machine_learning::{Gamma, KernelType};
use ndarray::array;

fn main() {
    // 6 points in 2-D. RBF kernel, keep 2 components.
    let x = array![
        [1.0, 0.0],
        [0.0, 1.0],
        [-1.0, 0.0],
        [0.0, -1.0],
        [2.0, 0.5],
        [-0.5, 2.0],
    ];

    let mut kpca = KernelPCA::new(KernelType::RBF { gamma: Gamma::Value(0.5) }, 2)
        .unwrap()
        .with_eigen_solver(EigenSolver::Dense);

    let projected = kpca.fit_transform(&x).unwrap();
    assert_eq!(projected.nrows(), 6);
    assert_eq!(projected.ncols(), 2);

    // Fitted state is exposed through getters.
    println!("kept {} components", kpca.get_n_components());
    println!("training samples: {:?}", kpca.get_n_samples()); // Some(6)
    let eigenvalues = kpca.get_eigenvalues().unwrap();
    println!("leading eigenvalue: {}", eigenvalues[0]);
}

The getters mirror the internal state. get_kernel, get_n_components, and get_eigen_solver return by value. get_n_samples and get_n_features return Option<usize> (None before fitting). get_eigenvalues and get_eigenvectors return Option<&Array1<f64>> and Option<&Array2<f64>>.

The stored eigenvectors are the columns from the centered kernel matrix’s eigendecomposition. These are the per-sample coefficients, traditionally written alpha, with shape n_samples x n_components. They are not the input-space directions that PCA gives you. Kernel PCA has no meaningful loading vector to inspect. This is the flip side of working in an implicit space.

2.11.4. Kernels and choosing gamma

KernelType has 5 variants, shared with SVC:

Linear reduces Kernel PCA back to ordinary PCA, up to the centering convention. Use it only as a baseline. RBF is the default. Use RBF when you suspect nonlinear, distance-based structure, like the rings.

Poly captures polynomial interactions. Cosine normalizes away magnitude and keeps only direction. This helps with high-dimensional sparse data.

Sigmoid is not a true (Mercer) kernel. Its centered Gram matrix can be indefinite. This interacts with the eigenvalue handling in 2.11.6.

Constructor validation is strict, and specific to each kernel. Poly requires degree > 0, a positive finite gamma, and a finite coef0. RBF requires a positive finite gamma. Sigmoid only requires its parameters to be finite. gamma = 0 is accepted for Sigmoid (test test_new_sigmoid_gamma_zero_accepted), because a zero coefficient is a legitimate, if degenerate, sigmoid. Any violation returns Error::InvalidParameter, naming the field.

The gamma coefficient has type Gamma. It is either an explicit value or a data-dependent rule resolved at fit time:

fit resolves Scale and Auto once, using the training data’s variance and feature count. It stores the resolved value, so the training matrix and every later transform call use the same coefficient. Gamma::Scale fails with Error::InvalidInput if the data has zero variance (all-constant features), because the formula divides by it.

For the RBF kernel, gamma is an inverse squared bandwidth: gamma = 1 / (2 * sigma^2). A large gamma (small bandwidth) makes the kernel see only near neighbors. The Gram matrix approaches the identity matrix, every point looks maximally distinct, and the projection overfits noise. A small gamma (large bandwidth) makes every pair look similar. The Gram matrix approaches a constant matrix, and the leading component captures nothing.

The useful range sits where the typical value of gamma * ||x - y||^2, for nearby points, is close to 1. A practical starting point is Gamma::Scale. From there, decrease gamma if the projection looks like noise. Increase gamma if distinct clusters merge together. Kernel PCA is unsupervised, so it has no built-in cross-validation for this choice. The concentric-rings separability metric in the test suite (class_separability) is the kind of downstream signal you can tune against.

2.11.5. Eigen solvers: exact versus iterative

EigenSolver selects how RustyML extracts the top n_components eigenpairs of the centered kernel matrix. All 3 solvers are pure-Rust, in-house implementations. The crate does not depend on a second linear-algebra library for this step (nalgebra remains only a dev-dependency, for test cross-checks).

The solver choice affects speed and the numerical path, not the result. Dense, Lanczos, and PowerIteration agree on the leading eigenvalues. They produce projections that are identical up to a per-column sign flip. The tests confirm this directly: test_eigensolver_dense_vs_lanczos_agree and test_eigensolver_dense_vs_power_iteration_agree compare column norms (sign-agnostic) and the top eigenvalue, to tolerances of 1e-5 and 1e-4. The sign ambiguity comes from the eigenvectors themselves, not from a bug. Fix the sign yourself downstream if you need a canonical one.

The iterative solvers do not save memory. All 3 solvers operate on the same n x n centered kernel matrix, which must exist in full before any decomposition starts. Lanczos and PowerIteration skip the O(n^3) cost of a full decomposition, when n_components is far smaller than n_samples. Neither one avoids the O(n^2) matrix itself. 2.11.8 covers that limit.

// A few components from a large kernel matrix: skip the full O(n^3) decomposition.
let kpca = KernelPCA::new(KernelType::RBF { gamma: Gamma::Scale }, 3)
    .unwrap()
    .with_eigen_solver(EigenSolver::Lanczos);

Kernel PCA carries no randomness in any solver. There is no seed to set. 2 runs on the same data and machine produce bit-identical output. The test test_determinism_dense_solver asserts exact equality (assert_allclose(..., 0.0)). This differs from t-SNE, whose stochastic initialization needs seeding.

2.11.6. Fitting, transforming, and out-of-sample projection

The 3 entry points are inherent methods. KernelPCA also implements the crate’s Fit, Transform, and FitTransform traits. These traits forward to the inherent methods, so generic code can treat KernelPCA like PCA:

pub fn fit<S>(&mut self, x: &ArrayBase<S, Ix2>) -> Result<&mut Self, Error>
pub fn transform<S>(&self, x: &ArrayBase<S, Ix2>) -> Result<Array2<f64>, Error>
pub fn fit_transform<S>(&mut self, x: &ArrayBase<S, Ix2>) -> Result<Array2<f64>, Error>

fit needs at least 2 samples. 1 row returns Error::InvalidInput. 0 rows return Error::EmptyInput. Non-finite input returns Error::NonFinite.

fit resolves gamma, builds and centers the training kernel matrix, extracts the eigenpairs, and stores everything a later transform needs. This includes a copy of the full training matrix, which every transform call reuses.

transform projects any matrix with the same feature count as the training data, including data the model has never seen. It builds the cross-kernel matrix between the new points and the stored training samples. It centers this matrix with the training statistics (see 2.11.2), then projects it onto the stored eigenvectors.

The projected coordinate on component k is (Kc * v_k) / sqrt(lambda_k). This 1/sqrt(lambda) scaling turns raw eigenvectors into properly normalized principal components. Calling transform before fit returns Error::NotFitted. A feature-count mismatch returns Error::DimensionMismatch.

use rustyml::machine_learning::decomposition::kernel_pca::KernelPCA;
use rustyml::machine_learning::{Gamma, KernelType};
use ndarray::array;

fn main() {
    let x_train = array![
        [1.0, 0.0],
        [0.0, 1.0],
        [-1.0, 0.0],
        [0.0, -1.0],
        [2.0, 0.5],
        [-0.5, 2.0],
        [1.5, -1.5],
        [-2.0, 1.0],
    ];

    // Default solver is Dense. No builder call needed.
    let mut kpca = KernelPCA::new(KernelType::RBF { gamma: Gamma::Value(0.5) }, 2).unwrap();
    kpca.fit(&x_train).unwrap();

    // Points the model has never seen, same feature count.
    let x_new = array![[0.3, 0.3], [1.8, -1.2]];
    let projected_new = kpca.transform(&x_new).unwrap();
    assert_eq!(projected_new.nrows(), 2);
    assert_eq!(projected_new.ncols(), 2);
    println!("{projected_new:?}");
}

fit_transform is a shortcut. It fits the model, then transforms the same data. The test test_fit_transform_equals_fit_then_transform confirms it matches the 2-call path to 1e-10.

fit_transform is a convenience, not an optimization. Internally, it just calls fit, then transform, on the same matrix. transform rebuilds the kernel matrix from scratch either way, so the cost is the same as calling both separately. Use the 2-step form when you need to project a different matrix through an already-fitted model.

For a proper Mercer kernel on distinct points, the centered Gram matrix is positive semidefinite. Every kept eigenvalue is strictly positive (test test_eigenvalues_are_positive_after_fit also confirms they come out sorted in descending order).

Kernel PCA does not hard-fail on non-positive eigenvalues, though. A centered Gram matrix is only positive semidefinite up to round-off error. Non-Mercer kernels, like Sigmoid, can produce genuinely negative trailing eigenvalues. fit only rejects non-finite eigenvalues (NaN or Inf maps to Error::Computation), instead of rejecting the whole fit. Any component whose eigenvalue is not meaningfully positive, below a relative 1e-12 * lambda_max threshold, gets a projection scale of 0.0. This zeroes that column instead of producing Inf or NaN.

This keeps the requested n_components dimensionality, while quietly discarding degenerate directions. The test test_fit_indefinite_kernel_negative_eigenvalue_is_tolerated drives this path with a Sigmoid kernel. It checks that the offending column comes out all zeros, and the rest stays finite.

2.11.7. Worked example: separating concentric rings

This is the case plain PCA cannot handle. There are 2 rings, radially separable but linearly entangled. Run the RBF kernel, and the 2 classes land in distinguishable regions of component space:

use rustyml::machine_learning::decomposition::kernel_pca::KernelPCA;
use rustyml::machine_learning::{Gamma, KernelType};
use ndarray::Array2;
use std::f64::consts::PI;

fn main() {
    // Inner ring r = 0.5, outer ring r = 3.0. No line separates them.
    let n = 12;
    let mut data: Vec<f64> = Vec::new();
    for i in 0..n {
        let a = 2.0 * PI * i as f64 / n as f64;
        data.push(0.5 * a.cos());
        data.push(0.5 * a.sin());
    }
    for i in 0..n {
        let a = 2.0 * PI * i as f64 / n as f64;
        data.push(3.0 * a.cos());
        data.push(3.0 * a.sin());
    }
    let x = Array2::from_shape_vec((2 * n, 2), data).unwrap();

    let mut kpca = KernelPCA::new(KernelType::RBF { gamma: Gamma::Value(0.5) }, 2).unwrap();
    let proj = kpca.fit_transform(&x).unwrap();

    // The RBF kernel encodes radial distance, so the rings separate along a component.
    let inner_mean: f64 = (0..n).map(|i| proj[[i, 0]]).sum::<f64>() / n as f64;
    let outer_mean: f64 = (n..2 * n).map(|i| proj[[i, 0]]).sum::<f64>() / n as f64;
    println!("inner-ring mean of component 0: {inner_mean:.4}");
    println!("outer-ring mean of component 0: {outer_mean:.4}");
    println!("gap between ring means: {:.4}", (inner_mean - outer_mean).abs());
}

Swap KernelType::RBF { .. } for KernelType::Linear, and the gap between the ring means collapses. The linear projection is dominated by the outer ring’s angular variation. It never encodes the radius. The test test_rbf_separates_radial_clusters_better_than_linear makes this quantitative, with a Fisher-style separability score. It asserts that the RBF projection beats the linear one by a comfortable margin.

2.11.8. What Kernel PCA cannot do

Kernel PCA has no inverse_transform. Plain PCA has one. You can map a low-dimensional code back to the input space, because the projection is a linear map with a clean transpose.

Kernel PCA cannot do this. This is not an oversight. It is the pre-image problem. A projected point lives in the implicit feature space. To invert it, you need an input x whose feature map phi(x) lands at that location.

For most kernels, RBF above all, the feature map is nonlinear, infinite-dimensional, and not surjective. An arbitrary point in feature space usually has no exact pre-image. It has only approximate ones, found through a separate nonlinear optimization.

RustyML does not ship that approximation. Kernel PCA is strictly a forward, one-way projection. Use it for visualization, for denoising by projection, or as a nonlinear feature stage that feeds a downstream classifier. Do not use it for reconstruction.

The Gram matrix is O(n^2). That is the real ceiling. fit builds an n x n matrix of f64 values. Memory grows as 8 * n^2 bytes, regardless of feature count. This is roughly 800 MB at n = 10,000, and 3.2 GB at n = 20,000.

Time is worse. Building the matrix is O(n^2 * d), through a single parallel GEMM. The dense eigendecomposition is O(n^3). Switching to Lanczos or PowerIteration trims the decomposition cost, when you need only a handful of components. Nothing removes the O(n^2) matrix itself.

In practice, Kernel PCA is comfortable into the low thousands of samples. It starts to hurt in the tens of thousands. Past that point, subsample a representative set to fit on, then transform the rest (each new batch pays O(m * n * d) for this). Or use a method that never forms the full kernel matrix.

Every transform call also carries the training set with it. The projection is defined relative to the stored training samples, so transform rebuilds an m x n cross-kernel matrix for m new points. PCA’s transform cost is independent of the training size. Kernel PCA’s transform cost scales with n forever. Budget for it.

The parallel machinery starts automatically above internal size gates, keyed on the element count of the kernel matrix. Roughly, the centering scans parallelize once n^2 clears a few hundred thousand elements. The elementwise centering parallelizes once it clears a few million elements. The kernel GEMM has its own FLOPs gate. You do not configure any of this per call. See 7.3. Performance Tuning and Parallelism for the tunable gates.

2.11.9. Persistence

KernelPCA derives Serialize and Deserialize, and exposes the standard pair:

pub fn save_to_path(&self, path: &str) -> Result<(), Error>
pub fn load_from_path(path: &str) -> Result<Self, Error>

Serialization uses the compact postcard binary format. The .bin, .dat, or any other extension in the path is just a filename. The bytes are always binary.

A round-tripped model reproduces transform output exactly. The test test_save_load_round_trip asserts equality to 1e-12.

Check what gets serialized. A fitted Kernel PCA stores the entire training matrix, along with the eigenvectors and the centering statistics, because transform needs all of it. The saved file grows with your training set. This is another consequence of the same O(n^2)/stored-samples design. Remember this before you persist a model fit on a large corpus.

use rustyml::machine_learning::decomposition::kernel_pca::KernelPCA;
use rustyml::machine_learning::{Gamma, KernelType};
use ndarray::array;
use std::fs;

fn main() {
    let x = array![
        [1.0, 0.0],
        [0.0, 1.0],
        [-1.0, 0.0],
        [0.0, -1.0],
        [2.0, 0.5],
        [-0.5, 2.0],
        [1.5, -1.5],
        [-2.0, 1.0],
    ];

    let mut kpca = KernelPCA::new(KernelType::RBF { gamma: Gamma::Value(0.5) }, 2).unwrap();
    kpca.fit(&x).unwrap();
    let before = kpca.transform(&x).unwrap();

    let path = "kpca_model.bin";
    kpca.save_to_path(path).unwrap();
    let loaded = KernelPCA::load_from_path(path).unwrap();
    let after = loaded.transform(&x).unwrap();

    assert_eq!(before.shape(), after.shape());
    fs::remove_file(path).unwrap();
}

A missing file surfaces as Error::Io (test test_load_from_nonexistent_path_returns_io_error). For the general error taxonomy, see 1.6. Error Handling. For persistence patterns across the crate, see 7.2. Model Persistence in Depth.

2.12. t-SNE

TSNE is RustyML’s implementation of t-distributed Stochastic Neighbor Embedding (t-SNE). It is a common method to turn high-dimensional data into a 2-D or 3-D plot. TSNE lives in machine_learning::manifold. Unlike the linear reducers in 2.10. Principal Component Analysis and 2.11. Kernel PCA, it learns no reusable projection. It embeds only the points you give it.

If you know scikit-learn, this is sklearn.manifold.TSNE. The API mirrors it on purpose. It exposes a perplexity, a learning_rate, an n_iter, a choice between an exact and a Barnes-Hut gradient, and PCA or random initialization.

2.12.1. What t-SNE is for, and what it is not

t-SNE is a visualization tool. Its only job is to place similar high-dimensional points near each other on a low-dimensional canvas. This lets a human eye see the cluster structure in the data. Most misuse of t-SNE comes from treating it as a general-purpose dimensionality reducer. It is not one.

The design makes this restriction concrete. In the estimator traits, PCA and Kernel PCA implement both Transform (project new, unseen data through a fitted model) and FitTransform. TSNE implements only FitTransform. It has no transform method, no stored fitted state, and no way to add a point to an existing embedding. The embedding coordinates are the optimization variables themselves, not the output of a learned function.

Use PCA to project new samples later. Use t-SNE to plot the data you already have. The 2 tools complement each other. A common pipeline runs PCA first and feeds the result to t-SNE (see 2.12.9).

The inherent method takes &self and returns a fresh embedding, so it never mutates the model. You can call it directly, or through the FitTransform trait. The trait method takes &mut self. It just forwards to the inherent method.

2.12.2. How t-SNE works

t-SNE measures similarity twice. It measures once in the original space and once in the embedding. Then it moves the embedding until the two agree.

In the high-dimensional space, t-SNE builds a set of pairwise affinities. For each point i, it centers a Gaussian on that point. It converts squared distances to the neighbors of i into conditional probabilities p_{j|i}. Near points get a high probability. Far points get a probability close to zero.

The width of the Gaussian is not fixed. RustyML solves it per point, with a binary search over sigma. The search matches the entropy of each point’s neighbor distribution to a target you set. That target is the perplexity. Perplexity reads as an effective neighbor count, roughly how many neighbors each point should feel.

RustyML runs the search for up to 50 bisection steps, to a tolerance of 1e-5. The self-distance always maps to exactly zero, so a point is never its own neighbor. The conditional probabilities are then symmetrized into joint probabilities p_ij. These joint probabilities sum to 1 over all pairs.

In the embedding, t-SNE uses a different, heavier-tailed kernel. This kernel is the Student-t distribution with 1 degree of freedom, where q_ij is proportional to (1 + ||y_i - y_j||^2)^-1. This is the trick that gives the “t” its name, and it fixes the crowding problem. A Gaussian in 2-D cannot make room for all the points that were moderately distant neighbors in, say, 50-D.

There is simply not enough area, and everything collapses into a blob. The t-distribution’s fat tail lets moderately similar points sit far apart in the map, without paying much probability cost. Clusters separate cleanly instead of crushing together.

The 2 distributions are matched by minimizing the Kullback-Leibler divergence KL(P || Q), with gradient descent on the embedding coordinates. KL divergence is asymmetric on purpose. It heavily penalizes placing a high-p (truly near) pair far apart in the map. It barely penalizes placing a low-p (truly far) pair close together. That asymmetry is why t-SNE preserves local neighborhoods faithfully, and treats global distances as expendable. It also drives all the plot-reading caveats in 2.12.8.

2.12.3. Constructing a model

TSNE::new takes the 4 core hyperparameters. It validates them up front. On a bad value, it returns Error::InvalidParameter instead of failing later at fit time.

use ndarray::array;
use rustyml::machine_learning::manifold::t_sne::{TSNE, TSNEMethod};

fn main() {
    // Two loose groups of points in 3-D feature space.
    let x = array![
        [0.0, 0.0, 0.0],
        [0.2, 0.1, -0.1],
        [-0.1, 0.2, 0.1],
        [0.1, -0.2, 0.0],
        [5.0, 5.0, 5.0],
        [5.2, 4.9, 5.1],
        [4.8, 5.1, 4.9],
        [5.1, 5.0, 4.8],
    ];

    // new(n_components, perplexity, learning_rate, n_iter) -> Result<TSNE, Error>.
    let tsne = TSNE::new(2, 3.0, 200.0, 300)
        .unwrap()
        .with_method(TSNEMethod::Exact)
        .unwrap();

    // fit_transform takes &self and returns an (n_samples, n_components) array.
    let embedding = tsne.fit_transform(&x).unwrap();
    assert_eq!(embedding.shape(), &[8, 2]);
    println!("embedding shape: {:?}", embedding.shape());
}

The constructor’s validation rules are narrow but strict:

TSNE::default() gives new(2, 30.0, 200.0, 1000), with PCA initialization and Barnes-Hut. This is a reasonable starting point for a real dataset, not a toy-sized one. The builder methods set everything else. Each builder returns Self for chaining, except with_method. with_method returns Result, because Barnes-Hut must validate its angle and its dimensionality:

Every stored field has a matching getter: get_n_components, get_perplexity, get_learning_rate, get_n_iter, get_random_state, get_init, get_method, and get_min_grad_norm. Use these to check how a chain of builder calls resolved.

2.12.4. Exact versus Barnes-Hut

TSNEMethod sets the cost tradeoff. RustyML’s default is not the exact method. This differs from what many users assume.

TSNEMethod::Exact builds the full dense n x n joint-probability matrix. It computes the gradient over every pair on each iteration. This costs O(n^2) per step, in both time and memory. TSNEMethod::Exact supports any n_components.

TSNEMethod::BarnesHut { angle } keeps the affinities sparse. Each point talks only to its k = ceil(3 * perplexity) + 1 nearest neighbors. It summarizes the repulsive forces with a space-partitioning tree. This gives roughly O(n log n) per iteration.

The angle (theta) is in [0, 1) and trades accuracy for speed. A larger angle opens tree cells sooner, so it runs faster but coarser. The value 0.5 is the standard balance. The tree lives in the embedding space, so Barnes-Hut supports only n_components of 3 or less.

TSNE::new picks Barnes-Hut with angle = 0.5 whenever n_components is 3 or less. This covers every visualization case. Otherwise it falls back to Exact. with_method enforces the same constraints. It rejects an angle outside [0, 1) with InvalidParameter. It also rejects Barnes-Hut paired with more than 3 components, with the same error.

use ndarray::array;
use rustyml::machine_learning::{TSNE, TSNEMethod};

fn main() {
    let x = array![
        [0.0, 0.0], [1.0, 0.0], [0.0, 1.0], [1.0, 1.0],
        [8.0, 8.0], [9.0, 8.0], [8.0, 9.0], [9.0, 9.0],
    ];

    // Default for 2 components: Barnes-Hut with angle 0.5.
    let bh = TSNE::new(2, 3.0, 200.0, 300).unwrap();
    assert_eq!(bh.get_method(), TSNEMethod::BarnesHut { angle: 0.5 });

    // Coarser, faster tree.
    let coarse = TSNE::new(2, 3.0, 200.0, 300)
        .unwrap()
        .with_method(TSNEMethod::BarnesHut { angle: 0.8 })
        .unwrap();

    // Exact O(n^2) gradient, the only option once n_components > 3.
    let exact = TSNE::new(2, 3.0, 200.0, 300)
        .unwrap()
        .with_method(TSNEMethod::Exact)
        .unwrap();

    for model in [bh, coarse, exact] {
        let emb = model.fit_transform(&x).unwrap();
        assert_eq!(emb.ncols(), 2);
    }
}

The Barnes-Hut gradient is scaled to match the exact path. The factor of 4 is folded in. The same learning_rate fits both methods, so you can switch between them without retuning. Both methods are bit-reproducible. The tree build is deterministic. The normalizer is summed in a fixed order, so the result does not depend on thread scheduling.

2.12.5. Perplexity and the sample-count rules

Perplexity is the hyperparameter that most changes the shape of your plot. It sets the effective neighbor count that each point calibrates its Gaussian to. A low perplexity emphasizes local structure and fractures the data into many small islands. A high perplexity blends larger neighborhoods and can smear distinct clusters together.

Values from 5 to 50 cover almost every use. The right value rises as your dataset grows.

There are 2 limits on perplexity, and they differ. fit_transform, not TSNE::new, enforces the first one. perplexity must be strictly less than the number of samples, or fit_transform returns InvalidParameter.

The second limit is about Barnes-Hut sparsity, not correctness. The Barnes-Hut path keeps k = ceil(3 * perplexity) + 1 neighbors per point, capped at n - 1. Some t-SNE implementations reject a perplexity above roughly n / 3 as an error. RustyML does not. Past that point, the neighbor cap has already saturated at n - 1, so every point already lists every other point as a neighbor. The embedding stays correct, but Barnes-Hut then loses its usual speed advantage over Exact for that data.

The enforced perplexity < n check is the one that matters, and it is the same for both methods. This is also why t-SNE on a handful of points gives little insight. With 20 samples, the enforced rule already caps perplexity below 20. A run near that ceiling has little real neighbor structure to find, and the clusters in the plot are mostly noise.

use ndarray::array;
use rustyml::error::Error;
use rustyml::machine_learning::{TSNE, TSNEMethod};

fn main() {
    let x = array![[0.0, 0.0], [1.0, 1.0], [2.0, 0.5]]; // 3 samples

    // perplexity must be < n_samples.
    // Here 3.0 is not < 3.
    let model = TSNE::new(2, 3.0, 200.0, 100)
        .unwrap()
        .with_method(TSNEMethod::Exact)
        .unwrap();

    match model.fit_transform(&x) {
        Err(Error::InvalidParameter { .. }) => {
            println!("perplexity too large for this sample count");
        }
        other => panic!("expected InvalidParameter, got {other:?}"),
    }
}

2.12.6. Initialization, seeding, and reproducibility

Init chooses the starting point for optimization. This choice affects reproducibility.

Init::PCA is the default. It starts the embedding from the top principal components of the input, rescaled to a small spread. When PCA succeeds, this method is deterministic and ignores random_state. The same data always gives the same starting layout, and so the same final embedding. It also gives the optimizer a well-spread starting layout. This is why Init::PCA is the default.

PCA init can fail in 2 cases. The input has fewer features than n_components, or its leading component is degenerate (zero spread). When either happens, Init::PCA falls back to the same random init that Init::Random uses. That fallback does consult random_state (or the global seed), so determinism returns with it.

Init::Random starts from tiny random noise. It seeds this noise through the crate’s central RNG. Apart from the PCA fallback above, this is the only path that consults random_state. Set a seed with with_random_state, and the run becomes bit-reproducible. Leave it unset, and the start comes from entropy, so every run differs.

use ndarray::array;
use rustyml::machine_learning::{Init, TSNE, TSNEMethod};
use rustyml::set_global_seed;

fn main() {
    let x = array![
        [0.0, 0.0], [1.0, 0.5], [0.5, 1.0],
        [6.0, 6.0], [7.0, 6.5], [6.5, 7.0],
    ];

    // Explicit per-model seed: reproducible regardless of any global seed.
    let seeded = TSNE::new(2, 2.0, 200.0, 250)
        .unwrap()
        .with_init(Init::Random)
        .with_random_state(42)
        .with_method(TSNEMethod::Exact)
        .unwrap();
    let e1 = seeded.fit_transform(&x).unwrap();
    let e2 = seeded.fit_transform(&x).unwrap();
    assert_eq!(e1, e2); // bit-identical

    // An unseeded random-init model becomes reproducible only because
    // this code sets the thread-local global seed first.
    set_global_seed(7);
    let unseeded = TSNE::new(2, 2.0, 200.0, 250)
        .unwrap()
        .with_init(Init::Random)
        .with_method(TSNEMethod::Exact)
        .unwrap();
    let _emb = unseeded.fit_transform(&x).unwrap();
}

The random_state field feeds make_rng. Every randomized component in the crate resolves its seed through this same path. An explicit Some(seed) is independent and never touches the global stream. A None seed defers to whatever set_global_seed set on the current thread.

This has 2 practical effects. With the default PCA init, the seed does not matter. Do not expect with_random_state to change anything, unless you also switch to Init::Random. The parallel reductions in the optimizer are also order-stable. Because of this, a fixed configuration reproduces bit for bit on a given machine, no matter the thread count. Chapter 7.1. Reproducibility and Random Seeds covers the full mechanics, including the thread-local nature of the global seed.

2.12.7. Inside the optimizer

The gradient descent loop is not plain SGD. Its phases explain most of what n_iter and learning_rate do.

The first 250 iterations run under early exaggeration (or all iterations, if n_iter is smaller). During this phase, t-SNE multiplies every joint probability p_ij by 12. This inflates the attractive forces. Tight clusters pull together and carve out empty space between groups before the layout settles. When the phase ends, the exaggeration factor drops back to 1, and the map fine-tunes.

If n_iter is too low, the run may stop inside the exaggeration phase. This leaves an over-contracted embedding that has not yet fine-tuned. This is one reason the defaults use 1000 iterations. Real runs rarely want fewer than a few hundred iterations.

Momentum follows the same schedule. It is 0.5 during exaggeration and 0.8 afterward. This gives the refinement phase more inertia through the flat parts of the KL landscape. Each coordinate also has an adaptive gain (Jacobs’ delta-bar-delta). This gain grows when the gradient keeps its sign, and decays when the step oscillates. The gain is floored at 0.01.

So learning_rate sets only a base step size. The effective per-parameter rate adapts as the run proceeds. After every step, the embedding is re-centered on the origin, to stop it from drifting. This is why the output columns always come back with a mean near zero.

min_grad_norm controls early stopping. After the exaggeration phase ends, the loop checks the largest absolute gradient entry on each iteration. It stops as soon as that value drops below the threshold (the default is 1e-7). This saves iterations once the map has converged. The loop skips this check during exaggeration, because the inflated gradients there would never trip it. Set min_grad_norm to 0.0 to disable early stopping and always run the full n_iter.

// The full builder surface, for reference.
let tsne = TSNE::new(2, 30.0, 200.0, 1000)?   // n_components, perplexity, lr, n_iter
    .with_init(Init::PCA)                       // or Init::Random
    .with_random_state(0)                       // only affects Init::Random
    .with_method(TSNEMethod::Exact)?            // or BarnesHut { angle }
    .with_min_grad_norm(0.0);                   // disable early stopping

Enable the show_progress feature (1.2. Installation and Feature Flags) to make the loop compute and show the live KL divergence on each iteration. Without the feature, the loop skips that pass entirely. It costs nothing in a production build.

2.12.8. Reading a t-SNE plot

t-SNE optimizes local neighborhoods, and it discards global geometry. Because of this, the plot misrepresents several things. Read it with that in mind.

The distance between 2 clusters means nothing. 2 blobs that land far apart on the canvas are not more different than 2 blobs that land close together. The gap between clusters is an artifact of the KL objective and the optimizer’s random walk. It is not a measurement.

The same holds for cluster size. How spread out a cluster’s points look is not a measure of its true variance. Dense regions get inflated, and sparse regions get compressed, to keep local neighborhoods intact. Never treat a between-cluster distance or a cluster diameter from a t-SNE map as a quantitative result.

The embedding depends on initialization, and the optimization is non-convex. So a single run is only 1 sample from a distribution of possible layouts. Run several. Vary the seed under Init::Random, or sweep perplexity across values such as 5, 30, and 50. Trust only the structure that survives across runs. A cluster that appears at one perplexity and dissolves at another is a likely artifact, not a finding.

Do not feed t-SNE coordinates into a downstream model as features. They are a picture, not a reusable representation. There is no transform, so they do not apply to new data. They also change across runs, and carry no meaning as distances. For reusable, projectable features, use PCA instead.

A common and valid workflow runs the reverse order. Cluster in the original space, for example with KMeans. Then color a t-SNE plot by the cluster label, to check the clustering by eye.

2.12.9. Cost, scaling, and preprocessing wide data

Both methods pay an O(n^2) setup cost, once, up front. This setup computes pairwise distances and calibrates the per-point sigmas. Barnes-Hut still runs an O(n^2) neighbor search before its cheaper O(n log n) iterations start. So the per-iteration cost differs, O(n^2) for Exact against O(n log n) for Barnes-Hut. Neither method escapes the quadratic setup term. The exact path also needs a full n x n matrix in memory.

In round numbers, exact t-SNE stays comfortable into the low thousands of points. Beyond that, Barnes-Hut lets you push toward tens of thousands of points. Let the default method choice handle this.

The other lever is the width of your data. The distance computation costs O(n^2 * d), for d input features. High-dimensional Euclidean distances are also noisy. The standard remedy reduces wide data to about 50 dimensions with PCA first, then runs t-SNE on that. This helps both speed and quality.

This step differs from Init::PCA. Init::PCA only uses the top 2 or 3 components, to seed the starting layout. Here, PCA compresses the input itself, before t-SNE ever runs.

use ndarray::array;
use rustyml::machine_learning::{PCA, TSNE, TSNEMethod};

fn main() {
    // 10 samples, 8 raw features (imagine hundreds in a real problem).
    let x = array![
        [0.10, 0.21, 0.05, 0.30, 0.11, 0.02, 0.22, 0.14],
        [0.12, 0.19, 0.08, 0.28, 0.09, 0.05, 0.20, 0.10],
        [0.09, 0.23, 0.03, 0.31, 0.13, 0.01, 0.24, 0.16],
        [0.11, 0.20, 0.06, 0.29, 0.10, 0.03, 0.21, 0.12],
        [0.80, 0.70, 0.90, 0.10, 0.60, 0.85, 0.15, 0.75],
        [0.82, 0.68, 0.88, 0.12, 0.62, 0.83, 0.17, 0.73],
        [0.78, 0.72, 0.91, 0.09, 0.58, 0.87, 0.14, 0.77],
        [0.81, 0.69, 0.89, 0.11, 0.61, 0.84, 0.16, 0.74],
        [0.40, 0.45, 0.42, 0.55, 0.38, 0.47, 0.52, 0.44],
        [0.42, 0.43, 0.44, 0.53, 0.36, 0.49, 0.50, 0.46],
    ];

    // Step 1: compress with PCA (here to 4 dims, about 50 for wide data).
    let mut pca = PCA::new(4).unwrap();
    let reduced = pca.fit_transform(&x).unwrap();

    // Step 2: run t-SNE on the compact representation.
    let tsne = TSNE::new(2, 3.0, 200.0, 300)
        .unwrap()
        .with_method(TSNEMethod::Exact)
        .unwrap();
    let embedding = tsne.fit_transform(&reduced).unwrap();
    assert_eq!(embedding.shape(), &[10, 2]);
}

The precompute pass and the per-iteration passes switch to RustyML’s parallel primitives once the pairwise work is large enough. The GEMM calls parallelize on their own. The reductions stay in a fixed order, so the result does not depend on thread count. To tune throughput, see 7.3. Performance Tuning and Parallelism, which covers the gating thresholds.

2.12.10. Errors from fit_transform

Construction validates the 4 hyperparameters. The rest of the checks run at fit_transform time. All of them return the crate-wide Error type:

Every failure is a typed Error, not a panic. You can match on the cause and react. Retry with a smaller perplexity, clean non-finite rows, or stop, as the example above shows.

2.13. Isolation Forest

IsolationForest is the unsupervised anomaly detector in RustyML.

Most anomaly detectors model where normal data lives, then flag points that fall outside that region. IsolationForest inverts this approach. It uses the fact that anomalies are few and different. A tree of random axis-aligned splits isolates an anomaly into its own leaf after only a few cuts. A point buried in a dense cluster needs many more cuts before the tree isolates it. RustyML averages this cut count over a forest of trees, then normalizes the result into an anomaly score.

This estimator matches sklearn.ensemble.IsolationForest, the same subsampling-based ensemble from Liu, Ting, and Zhou. The API now lines up with scikit-learn method for method. score_samples, decision_function, and predict mean what they mean in Python, including the sign of the scores. If you port code written against an older RustyML, read Section 2.13.4 first. Both the score sign and the method names changed there.

2.13.1. The isolation principle and the anomaly score

Each tree grows on a random subsample of the data. The build step repeats one simple move. It picks a feature at random. It picks a split threshold at random, between that feature’s minimum and maximum value in the current node. It sends points below the threshold to the left child and the rest to the right child.

The tree recurses until a node holds 1 point. It also stops when every value of the chosen feature is equal in that node, because there is nothing left to split. It stops too when it hits the depth cap. A point far from the bulk of the data needs only a handful of random cuts to isolate. A point in the middle of a dense cluster needs many more cuts.

The tree structure that records this stays minimal. An isolation tree does not need class counts or impurity. It only needs where it split and how many points reached each leaf:

pub enum IsolationTree {
    Leaf { size: usize },
    Internal {
        feature: usize,
        threshold: f64,
        left: Box<IsolationTree>,
        right: Box<IsolationTree>,
    },
}

The raw signal for a sample is its path length, h(x). This is the number of edges from the root to the leaf where the sample lands. A tree has a depth cap, so a leaf can still hold several points the tree did not fully separate. The path length adds a correction for the subtree that would have continued below that leaf. A leaf at depth d holding size points contributes d + c(size) to the path length.

c(n) estimates the average extra depth needed to isolate n points. c(n) is also the normalization constant. It equals the expected path length of a failed search in a binary search tree of n points:

c(n) = 2 * H(n-1) - 2 * (n-1) / n, where H(m) is the m-th harmonic number.

RustyML computes H(m) exactly, as a running sum, when n <= 50. Above that, it switches to the asymptotic form ln(m) + gamma + 1 / (2 * m). Here gamma is the Euler-Mascheroni constant, and m = n - 1. This approximation keeps the error under 1e-3. RustyML pins 2 edge cases: c(n) = 0 when n <= 1, and c(2) = 1. The final score for a sample averages h(x) over every tree, then applies Liu et al.’s formula, negated:

s(x) = -2^(-E[h(x)] / c(n)).

The score lands in the range [-1, 0). A lower score means more anomalous. This is the reverse of the earlier RustyML convention.

A short average path is easy to isolate, so it is anomalous. It drives the score toward -1. A long average path is hard to isolate, so it is normal. It drives the score toward 0.

The negation puts the score on the usual convention where negative marks the rejected class. It also makes score_samples numerically identical to scikit-learn’s version.

The neutral point is s = -0.5. At this point, a sample’s expected path length equals c(n), so its isolation cost matches the tree average. Liu et al. call this the “no distinct anomaly” regime. Contamination::Auto uses -0.5 as its cutoff. Treat -0.5 as no signal, not as confirmed normal. A point earns “normal” only when it scores clearly above -0.5.

The c(n) in the denominator uses n = sample_size. This is the realized per-tree subsample size, covered in the next 2 sections. It is not max_samples. This distinction matters when your dataset is smaller than max_samples, because normalizing by the wrong n would shift every score.

RustyML’s test suite checks the closed form directly. It fits identical points with max_samples >= n_rows. Every score comes out exactly -0.5, because E[h(x)] = c(sample_size) cancels the exponent.

2.13.2. Constructing the forest

IsolationForest has 2 entry points. IsolationForest::default() gives the standard configuration. IsolationForest::new(n_estimators, max_samples) sets the 2 structural parameters. It validates them and returns Result<Self, Error>. 3 builder methods refine the result further. Each one consumes and returns self, so you can chain them:

impl IsolationForest {
    pub fn new(n_estimators: usize, max_samples: usize) -> Result<Self, Error>;
    pub fn with_max_depth(self, max_depth: usize) -> Result<Self, Error>;
    pub fn with_random_state(self, seed: u64) -> Self;
    pub fn with_contamination(self, contamination: Contamination) -> Result<Self, Error>;
}

use rustyml::machine_learning::IsolationForest;

fn main() {
    // Standard config: 100 trees, subsample of 256 rows each,
    // depth auto = ceil(log2(256)) = 8, no seed.
    let _a = IsolationForest::default();

    // Explicit config, with a seed and a hand-set depth cap that overrides the auto value.
    let forest = IsolationForest::new(100, 256)
        .unwrap()
        .with_max_depth(10)
        .unwrap()
        .with_random_state(42);

    assert_eq!(forest.get_n_estimators(), 100);
    assert_eq!(forest.get_max_samples(), 256);
    assert_eq!(forest.get_max_depth(), 10);
    assert_eq!(forest.get_random_state(), Some(42));
}

new rejects n_estimators == 0 or max_samples == 0. It returns Error::InvalidParameter, naming the field that failed. with_max_depth(0) rejects the same way.

Note one asymmetry. An explicit depth of 0 is invalid. The auto-computed depth can still be 0. For example, max_samples == 1 gives ceil(log2(1)) = 0. This is fine, because a subsample of 1 point has nothing to split anyway.

Every field has a getter. get_n_estimators, get_max_samples, get_max_depth, get_random_state (returns Option<u64>), and get_contamination work right after construction. 4 more getters make sense only after fitting: get_n_features, get_sample_size, get_offset (returns Option<f64>, the resolved score cutoff), and get_trees (returns Option<&Vec<IsolationTree>>, None until fitted).

Understand the auto depth before you override it. ceil(log2(max_samples)) estimates the height of a balanced tree over the subsample. Anomalies are roughly isolated by that height already. Splitting deeper mostly separates points inside dense normal clusters. The c(size) leaf correction already accounts for that effect.

A bigger max_depth buys almost no accuracy, and it costs more build time. Leave max_depth on auto, unless you have a specific reason to change it.

2.13.3. Subsampling, sample_size, and why 256

Isolation Forest makes an unusual design choice. Each tree trains on a small random subsample, not on the whole dataset. This is a feature, not a shortcut. RustyML draws sample_size = min(max_samples, n_rows) rows for each tree, without replacement, through a partial Fisher-Yates shuffle.

The default max_samples = 256 comes directly from the original paper. Small subsamples defeat 2 failure modes of density-based detectors. Swamping happens when normal points near a cluster of anomalies start to look anomalous. Masking happens when a dense clump of anomalies hides its own members from each other. Both problems get worse as each tree sees more data, because large samples let anomalies form their own mini-clusters. Those mini-clusters are no longer easy to isolate.

A 256-row subsample is large enough to profile the shape of the normal region. It is also small enough that anomalies stay sparse and stay easy to isolate. A bigger subsample mostly buys higher build cost and worse detection quality. That is why max_samples is a subsample budget, not a “use more data” knob.

This design has 2 consequences. First, when your dataset has fewer rows than max_samples, sample_size clamps to n_rows. Every tree then sees the whole dataset. The forest still works, because each tree still differs through its random splits. It just no longer subsamples. fit checks for this case and never panics.

Second, normalization uses c(sample_size), not c(max_samples). RustyML reads sample_size back from the fitted model instead of assuming max_samples. get_sample_size() reports the realized value after fit.

2.13.4. Fitting and reading scores

fit takes a 2-D f64 array, where rows are samples and columns are features. It returns Result<&mut Self, Error>. fit records n_features and sample_size, builds every tree, then resolves the contamination rule into a stored score cutoff, the offset. Scoring then exposes 4 surfaces. Each one shifts or thresholds the surface before it, in a chain:

pub fn fit<S>(&mut self, x: &ArrayBase<S, Ix2>) -> Result<&mut Self, Error>
where S: Data<Elem = f64> + Send + Sync;

pub fn score_samples<S>(&self, x: &ArrayBase<S, Ix2>) -> Result<Array1<f64>, Error>
where S: Data<Elem = f64>;                  // 1 score per row, in [-1, 0). Lower means more anomalous

pub fn score_sample(&self, sample: &[f64]) -> Result<f64, Error>;   // 1 row, as a slice

pub fn decision_function<S>(&self, x: &ArrayBase<S, Ix2>) -> Result<Array1<f64>, Error>
where S: Data<Elem = f64>;                  // score_samples(x) minus offset. Negative marks an outlier

pub fn predict<S>(&self, x: &ArrayBase<S, Ix2>) -> Result<Array1<i32>, Error>
where S: Data<Elem = f64>;                  // sign of decision_function: {-1 outlier, +1 inlier}

Each name means exactly what it means in scikit-learn. score_samples gives the raw anomaly score. decision_function shifts that score so zero marks the decision boundary. predict gives the sign of the decision value. A strictly negative value gives -1. Everything else gives +1, so a sample that lands exactly on the cutoff counts as an inlier.

Pick the method the task needs. Use score_samples to rank samples, set your own cutoff, or feed a downstream calibrator. Use decision_function for a signed margin. Use predict for a hard {-1, +1} decision.

Earlier RustyML versions differ here in 2 ways. Both changes alter results silently, instead of failing to compile.

First, the scores are now negated. The old scale was [0, 1], where a higher score meant more anomalous. The new scale is [-1, 0), where a lower score means more anomalous. Flip every comparison that ranks or thresholds a score.

Second, the per-sample slice method is now called score_sample (singular). It replaces the old anomaly_score method. The old batch form, where predict returned scores, is gone. The old predict_labels(&x, contamination) method is gone too. score_samples replaces the first. predict, together with the fitted contamination rule, replaces the second.

score_sample scores one point at a time, for a live stream or an ad-hoc query. It takes a &[f64] slice instead of a matrix. fit_predict runs fit, then predict, on the same matrix. It returns labels, the standard “label the training set” convenience for unsupervised models.

use ndarray::array;
use rustyml::machine_learning::IsolationForest;

fn main() {
    let train = array![[0.0, 0.0], [0.1, 0.0], [0.0, 0.1], [0.1, 0.1], [30.0, 30.0]];
    let mut forest = IsolationForest::new(80, 32).unwrap().with_random_state(1);
    forest.fit(&train).unwrap();

    // Score points one at a time through the slice API.
    let normal = forest.score_sample(&[0.05, 0.05]).unwrap();
    let weird = forest.score_sample(&[30.0, 30.0]).unwrap();
    assert!(weird < normal); // lower score means more anomalous

    // The batch form, and the same values shifted by the fitted cutoff.
    let scores = forest.score_samples(&train).unwrap();
    let decision = forest.decision_function(&train).unwrap();
    let offset = forest.get_offset().unwrap(); // -0.5 under Contamination::Auto
    assert!((decision[0] - (scores[0] - offset)).abs() < 1e-12);
}

Every entry point validates its input and returns a typed Error. fit returns Error::EmptyInput on zero rows. fit returns Error::NonFinite on any NaN or infinite value. score_samples also checks Error::DimensionMismatch, when the feature count differs from training, and Error::NotFitted, before fit runs.

decision_function and predict propagate those same errors unchanged. score_sample checks only Error::NotFitted and Error::DimensionMismatch for its single slice. It does not check the slice for NaN or infinite values.

The batch calls accept any ArrayBase backing. This includes non-contiguous views, such as a transposed array. You do not need to copy .t() into a fresh buffer before scoring.

2.13.5. A worked example: injecting outliers into a blob

This is the standard sanity check for an anomaly detector. Build a tight cluster of inliers. Drop in a few points that clearly do not belong. Confirm that the forest ranks those points at the top.

use ndarray::Array2;
use rustyml::machine_learning::{Contamination, IsolationForest};

fn main() {
    // 20 inliers in a tight blob near (5, 5)...
    let mut rows: Vec<f64> = Vec::new();
    for i in 0..20 {
        let t = i as f64;
        rows.push(5.0 + 0.15 * t.sin());
        rows.push(5.0 + 0.15 * t.cos());
    }
    // ...plus 3 injected outliers far from the cluster.
    for &(x, y) in &[(30.0, 30.0), (-20.0, 40.0), (50.0, -10.0)] {
        rows.push(x);
        rows.push(y);
    }
    let n = rows.len() / 2; // 23 rows
    let data = Array2::from_shape_vec((n, 2), rows).unwrap();

    // A 15% contamination budget, resolved into a fixed cutoff at fit time.
    let mut forest = IsolationForest::new(100, 256)
        .unwrap()
        .with_random_state(42)
        .with_contamination(Contamination::Fraction(0.15))
        .unwrap();
    let labels = forest.fit_predict(&data).unwrap();
    let scores = forest.score_samples(&data).unwrap();

    // The 3 injected rows (indices 20..23) should score well below the blob.
    for i in (n - 3)..n {
        println!("outlier row {i}: score {:.3}", scores[i]);
    }

    let flagged: Vec<usize> = labels
        .iter()
        .enumerate()
        .filter(|&(_, &l)| l == -1)
        .map(|(i, _)| i)
        .collect();
    println!("cutoff (offset): {:.3}", forest.get_offset().unwrap());
    println!("flagged as outliers (-1): {flagged:?}");
}

The 3 outlier points land near the bottom of the score range. The blob rows sit close to zero:

outlier row 20: score -0.735
outlier row 21: score -0.779
outlier row 22: score -0.788
cutoff (offset): -0.419
flagged as outliers (-1): [2, 20, 21, 22]

Note what the cutoff is, and what it is not. Contamination::Fraction(0.15) sets the offset to the 15th percentile of the training scores. It separates off roughly the lowest 15% of samples. Here that is 4 rows out of 23, 1 more than the 3 rows actually injected.

The exact count depends on where the scores fall. It does not depend on a fixed ceil(0.15 * n), so an inlier can get swept in. This is what contamination means, and it motivates the next section.

2.13.6. Turning scores into labels: contamination

contamination is a rule you set on the builder. It is not an argument to a prediction call. fit resolves it into a single stored number, the offset:

pub enum Contamination {
    Auto,          // the paper's cutoff: offset = -0.5
    Fraction(f64), // offset = the 100*c-th percentile of the TRAINING scores. c is in (0.0, 0.5]
}

Contamination::Auto is the default. It pins the cutoff at -0.5, the score where a sample isolates exactly as fast as the forest average. Contamination::Fraction(c) instead reads the cutoff from the training scores, at the 100*c-th percentile. It uses NumPy’s linear interpolation for that percentile. This choice makes get_offset() equal scikit-learn’s offset_ numerically. It does more than flag a comparable number of rows.

A Fraction value must be finite and inside (0.0, 0.5]. Otherwise, with_contamination returns Error::InvalidParameter. The upper bound of 0.5 encodes the assumption that anomalies are the minority.

Resolving the cutoff at fit time is the whole point of this design. The rule becomes model state. A sample gets the same label whether you score it alone, in a slice, or in the full batch.

The earlier design worked differently. It took a quantile of whatever batch it received, so it was transductive. A single-row call always came back -1. Splitting a test set in half changed its labels.

Changing the rule after fitting requires a new call to fit, because RustyML computes the offset from the training scores.

Contamination is still a budget, not a discovery. You tell the model what proportion of the training data you expect to be anomalous. fit places the cutoff accordingly, whether or not that many anomalous points actually exist. If you overestimate, the cutoff sweeps in normal points as false positives, as the worked example showed. If you underestimate, the cutoff labels real anomalies as inliers. The model cannot know the true rate, so there is no way around this tradeoff.

When you have ground-truth labels for a validation set, sweep the fraction. Pick the value that best trades precision against recall for your cost structure. The tools in 5.2. Classification Metrics apply here, if you treat -1 as the positive class.

When you do not have ground truth, prefer the raw score_samples values. Threshold them against a level you can justify. For example, use a percentile from a clean reference window or an absolute score cutoff. Do not just commit to a fixed proportion. If your idea of “outlier” is really “sparse region,” rather than “few and different,” compare against DBSCAN, which labels low-density points as noise.

2.13.7. Seeding and reproducible forests

An unseeded forest draws its randomness from entropy, so 2 fits differ. Pass with_random_state(seed) to make the whole forest reproducible. The same seed and the same data give bit-identical scores.

use ndarray::array;
use rustyml::machine_learning::IsolationForest;

fn main() {
    let data = array![[0.0, 0.0], [0.5, 0.5], [1.0, 1.0], [2.0, 2.0], [50.0, 50.0]];

    let mut a = IsolationForest::new(30, 20).unwrap().with_random_state(13);
    a.fit(&data).unwrap();
    let sa = a.score_samples(&data).unwrap();

    let mut b = IsolationForest::new(30, 20).unwrap().with_random_state(13);
    b.fit(&data).unwrap();
    let sb = b.score_samples(&data).unwrap();

    assert_eq!(sa, sb); // bit-identical, regardless of thread scheduling
}

This determinism is stronger than it looks, and the reason matters for the parallel build described in the next section. RustyML does not share a single RNG across the trees. Tree i gets its own generator, seeded with seed.wrapping_add(i). Each tree’s seed depends only on its index.

So the forest comes out identical whether the trees build serially or across rayon threads, in any order the scheduler picks. Parallelism can never change the result. This is a deliberate contrast with the neural-network components in this crate, where reproducibility can depend on order.

You can fix randomness globally instead of passing a seed to every constructor. rustyml::set_global_seed seeds an unseeded forest from the process-global, thread-local, random stream:

use ndarray::array;
use rustyml::machine_learning::IsolationForest;
use rustyml::set_global_seed;

fn main() {
    set_global_seed(2026);
    let data = array![[0.0, 0.0], [0.5, 0.5], [1.0, 1.0], [50.0, 50.0]];

    // No with_random_state: per-tree seeds derive from the global stream.
    let mut forest = IsolationForest::new(30, 16).unwrap();
    let _labels = forest.fit_predict(&data).unwrap();
}

An explicit with_random_state always wins over the global seed. It also never consumes the global stream, so mixing the two stays predictable. The global seed is thread-local. Set it on the same thread that builds the model. See 7.1. Reproducibility and Random Seeds for the full seed-resolution rules.

2.13.8. Parallel training and prediction

Training and scoring both parallelize through rayon. A gate keeps tiny workloads serial, to avoid fork/join overhead. fit builds trees across a into_par_iter once n_estimators reaches its threshold of 10 trees. The default 100-tree forest always trains in parallel. Each tree is an independent subsample-and-build step, with its own index-seeded RNG. The work parallelizes cleanly, with no shared mutable state and no effect on the result.

score_samples parallelizes over rows once the estimated traversal work clears the calibrated tree-traversal gate. That work estimate is samples times trees times average path length. Below the gate, small batches score serially. decision_function and predict inherit this behavior, because they are thin wrappers over score_samples. score_sample always runs serially, because it handles only a single sample.

Isolation Forest scales well on wide forests and large scoring batches. Training cost is roughly n_estimators * sample_size * log(sample_size), and it is embarrassingly parallel across trees. Per-sample scoring is O(n_estimators * tree_depth), and it is independent across rows. See 7.3. Performance Tuning and Parallelism to tune the gates or study the cost model.

2.13.9. Saving and loading a forest

A fitted forest serializes to a compact postcard binary, through save_to_path and load_from_path. Every RustyML estimator exposes this same pair of methods. The whole state travels: every tree, the hyperparameters, n_features, sample_size, and the fitted offset. A reloaded model scores and labels identically to the original, byte for byte.

use ndarray::array;
use rustyml::machine_learning::IsolationForest;

fn main() {
    let data = array![[0.0, 0.0], [0.1, 0.1], [0.2, 0.0], [10.0, 10.0]];
    let mut forest = IsolationForest::new(50, 32).unwrap().with_random_state(7);
    forest.fit(&data).unwrap();
    let before = forest.score_samples(&data).unwrap();

    let path = "isolation_forest_model.bin";
    forest.save_to_path(path).unwrap();

    let loaded = IsolationForest::load_from_path(path).unwrap();
    let after = loaded.score_samples(&data).unwrap();

    assert_eq!(before, after); // scoring survives the round trip exactly
    std::fs::remove_file(path).unwrap();
}

load_from_path returns Error::Io when the file is missing, or when the bytes are not a valid serialized forest. RustyML persists sample_size, so a loaded model keeps the correct c(n) normalization, even though it never sees the training data again. Its scores stay on the same scale as before you saved it.

One migration note applies. The stored offset changed sign along with the scores. A forest saved by an older version resolves its cutoff on the wrong side of zero. Re-fit and re-save any persisted forest. For versioning and format details across the crate, see 7.2. Model Persistence in Depth.

3. Neural Networks

RustyML ships a small, Keras-shaped deep-learning framework, written in pure Rust. You stack layers into a Sequential model, compile it with an optimizer and a loss, and call fit/predict. Every tensor that flows through the framework is a Tensor, which is just ndarray::ArrayD<f32>. It is single-precision, has a dynamic rank, and runs with no GPU and no autograd tape. Each layer implements forward and backward by hand, and hands its parameters to the optimizer through a flat view. This makes the whole framework deterministic and easy to debug. If you have used Keras, you will recognize the shape of the API. The differences are strict f32 precision, explicit input dimensions, and Result-returning constructors, and this chapter explains them.

Read Chapter 1 before this chapter. Read Working with ndarray and Installation and Feature Flags too, since the neural_network feature gates this whole module. Read Error Handling as well, since layer and loss constructors return Result. A model has this end-to-end shape:

use rustyml::neural_network::{
    sequential::Sequential,
    layers::{Activation, Dense},
    optimizers::Adam,
    losses::MeanSquaredError,
};
use ndarray::Array;

fn main() {
    let x = Array::ones((8, 4)).into_dyn(); // 8 samples, 4 features
    let y = Array::ones((8, 1)).into_dyn(); // 8 samples, 1 target

    let mut model = Sequential::new();
    model
        .add(Dense::new(4, 16, Activation::ReLU).unwrap())
        .add(Dense::new(16, 1, Activation::Linear).unwrap())
        .compile(
            Adam::new(0.001, 0.9, 0.999, 1e-8, 0.0).unwrap(),
            MeanSquaredError::new(),
        );

    model.fit(&x, &y, 5).unwrap();
    let preds = model.predict(&x).unwrap();
    println!("prediction shape: {:?}", preds.shape());
}

The Sequential Model is the container that turns a pile of layers into a trainable network. It owns the training loop, the optimizer, and the loss, and exposes add, compile, fit, train_batch, evaluate, predict, summary, and weight save/load. It also holds the batch-shuffle seed (set_seed) and the learning-rate pair (learning_rate / set_learning_rate). Read this section first, even if you need only one specific layer.

Dense Layers and Activations covers the fully connected layer, the main layer of tabular models and the output stage of most networks. It also covers the Activation enum (ReLU, Sigmoid, Tanh, Softmax, Linear), which you fold into a layer or use as a standalone layer. Read this section second. Everything after it assumes you know how a Dense layer declares its input_dim and units.

Loss Functions is the objective half of compile. It covers mean squared error and mean absolute error for regression, and binary, categorical, and sparse-categorical cross-entropy for classification. Read this section closely. Its averaging conventions differ on purpose: some average per element, others average per prediction site, and switching between them quietly rescales your effective learning rate. CategoricalCrossEntropy and SparseCategoricalCrossEntropy take a from_logits flag that changes whether you need a Softmax on the output. BinaryCrossEntropy has no such flag, and always expects a probability in (0, 1).

Optimizers is the update half of compile. It covers SGD with momentum, Adam, AdamW, RMSprop, and AdaGrad. This section also covers clip-by-global-norm (global_clipnorm), coupled versus decoupled weight decay, and mid-training learning-rate scheduling. Sections 3.1 through 3.4 together give you a complete, trainable feed-forward network.

The remaining sections add specialized layers. All of them plug into the same Sequential. Convolutional Layers provides 1D/2D/3D convolution, plus depthwise and separable variants for spatial data. Pooling Layers provides parameter-free max and average downsampling, plus their global variants. Recurrent Layers covers SimpleRNN, LSTM, and GRU for sequences. Regularization and Normalization Layers covers dropout (including spatial dropout), Gaussian noise, and batch, layer, group, and instance normalization. These layers depend on mode: they behave differently in fit than in predict, and the model switches the mode for you.

Saving and Loading Weights closes the chapter. save_to_path and load_from_path persist weights only, in postcard binary format. They do not persist the architecture. Rebuild the identical layer stack in code, then load the weights into it. Read this section once you have a model worth keeping. See Model Persistence in Depth for the format details and version caveats.

3.1. The Sequential Model

Sequential is the container that turns a stack of layers into a trainable model. The Keras workflow transfers almost without change. You call new() for an empty model, add() layers in order, compile() it with an optimizer and a loss, then call fit() and predict().

RustyML differs from Keras in 3 ways. Training is full-batch by default. There is no callback machinery, so you write your own loops for early stopping and learning-rate schedules. A width mismatch between adjacent layers surfaces as a panic instead of a Result value.

Sequential lives in rustyml::neural_network::sequential. The layers, optimizers, and losses that plug into it each have their own page (3.2, 3.4, 3.3).

3.1.1. The lifecycle at a glance

The 5 calls below are the entire public surface you need to train a network. Each builder method returns &mut Self, so add and compile chain together. fit returns Result<History, Error>, with one loss value per epoch. predict returns Result<Tensor, Error>.

let mut model = Sequential::new();
model
    .add(/* a layer */)
    .add(/* another layer */)
    .compile(/* optimizer */, /* loss */);
model.summary();
let history = model.fit(&x, &y, epochs)?; // epochs: u32 (history.loss() returns one f32 per epoch)
let y_hat = model.predict(&x_new)?;       // Tensor = ArrayD<f32>

The prelude exports everything on this page: Sequential, History, Dense, Activation, every optimizer and loss, and the Tensor alias. One glob import covers every example below:

use rustyml::prelude::*; // Sequential, History, Dense, Activation, Adam, SGD, losses, Tensor

Every tensor in the framework has type Tensor, an alias for ndarray::ArrayD<f32>. It has a dynamic rank and always holds f32 values. Call .into_dyn() to convert a statically-ranked array, such as Array2 or Array3, into IxDyn before it enters a layer. RustyML has no f64 path. The whole stack uses single precision for cache and SIMD performance.

3.1.2. Building the model: new and add

Sequential::new() creates an empty model with no optimizer, no loss, and no shuffle seed. add takes any L: 'static + Layer by value, boxes it as Box<dyn Layer>, and appends it to the model:

pub fn add<L: 'static + Layer>(&mut self, layer: L) -> &mut Self;

add consumes the layer, so you construct and move it in one expression: model.add(Dense::new(2, 8, Activation::ReLU).unwrap()). Layer constructors can fail on their own. For example, a Dense layer with a zero dimension returns Error::InvalidParameter. That is why .unwrap() appears on the layer constructor, not on add.

add does no cross-layer validation. It never checks that a layer’s input width matches the previous layer’s output width. A Box<dyn Layer> exposes no such contract at insertion time. A mismatch surfaces only when data flows through the model. See 3.1.8. Keras’s functional API works the opposite way: it fails when you connect incompatible layers while you build the graph.

The Layer trait, briefly

You do not need to implement Layer to use Sequential. Reading its contract explains what fit and predict call. The core methods are:

pub trait Layer: std::any::Any + Send + Sync {
    fn forward(&mut self, input: &Tensor) -> Result<Tensor, Error>;   // training pass, caches state for backward
    fn predict(&self, input: &Tensor) -> Result<Tensor, Error>;       // eval pass, takes &self, writes no caches
    fn backward(&mut self, grad_output: &Tensor) -> Result<Tensor, Error>;
    fn param_count(&self) -> TrainingParameters;
    fn output_shape(&self) -> String;
    fn layer_type(&self) -> &str;
    // parameters(), layer_type(), output_shape(), set_training_if_mode_dependent() have defaults
}

Layer has 2 forward paths instead of 1. forward(&mut self, ...) runs during training, taking &mut self to stash the intermediate tensors each layer needs for its backward pass. predict(&self, ...) runs during inference, taking &self, writing no caches, and putting mode-dependent layers, such as dropout and batch normalization, into their inference behavior. The Layer trait requires Send + Sync. Because predict borrows &self, a layer can serve concurrent inference calls with no lock. This split means predict never disturbs training state, and it costs less than a training forward pass.

Backward propagation is pure math. It does not sanitize NaN or Inf values on purpose. A non-finite gradient propagates and surfaces at the next forward pass or as a NaN loss, instead of being masked silently.

3.1.3. compile: wiring the optimizer and loss

pub fn compile<O, LFunc>(&mut self, optimizer: O, loss: LFunc) -> &mut Self
where O: 'static + Optimizer, LFunc: 'static + Loss;

compile stores the optimizer and loss as trait objects. It returns &mut Self so it chains off the last add. That is all it does. It runs no shape inference and allocates no weights. Each layer allocates its own weights in its constructor, using Xavier/Glorot initialization.

Training is the only thing that needs compile. fit returns Error::NeuralNetwork(NnError::NotCompiled(_)) if the optimizer or the loss is missing. predict never reads the optimizer or the loss, so it works on an uncompiled model. evaluate sits between the two: it needs a loss to score with, but no optimizer to step with. This split matters for serving. After you load weights into a fresh architecture, you can call predict right away without a compiled optimizer.

Pick the optimizer and loss from 3.4 and 3.3.

Each loss family normalizes its value in a different way. MeanSquaredError, MeanAbsoluteError, and BinaryCrossEntropy average over every element. CategoricalCrossEntropy sums over the trailing class axis, then averages over the prediction sites. A prediction site is a sample for a [batch, classes] target. For a channels-last convolutional softmax head that predicts 1 class per pixel, a prediction site is a pixel. The divisor is then batch * height * width.

Switching loss families rescales the gradient magnitude, which changes the effective learning rate. Re-tune the step size after you change the loss.

3.1.4. summary: reading the architecture

summary() prints a Keras-style table to stdout. Use it to check the wiring and the parameter count before you spend epochs training the model. Here is the table for a Dense(2 -> 8, ReLU) layer followed by a Dense(8 -> 2, Softmax) layer:

Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 8)              │            24 │
│ dense_1 (Dense)                 │ (None, 2)              │            18 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
 Total params: 42 (168 B)
 Trainable params: 42 (168 B)
 Non-trainable params: 0 (0 B)

RustyML generates layer names per type, in Keras style. The first Dense layer is named dense, the next is dense_1, and so on. None in the output shape is the batch dimension. It stays unknown until data arrives.

Parameter counts come from each layer’s param_count() method. A Dense(in -> out) layer reports in * out + out parameters, the weight matrix plus the bias vector. So 2 * 8 + 8 = 24, and 8 * 2 + 2 = 18. The byte figures assume 4 bytes per f32.

Layers split into 3 groups. Trainable parameters count toward “Trainable”, frozen parameters count toward “Non-trainable”, and parameter-free layers, such as activations and pooling, contribute 0. summary borrows &self and never changes the model, so call it at any time, before or after training.

3.1.5. fit and the training loop

pub fn fit(&mut self, x: &Tensor, y: &Tensor, epochs: u32) -> Result<History, Error>;

fit is full-batch. Each epoch runs exactly one gradient step over the entire x and y you provide. There is only one batch, so nothing is shuffled, and the model never reads its shuffle seed. epochs counts these full-dataset steps. Keras’s fit, by contrast, splits data into mini-batches and shuffles every epoch by default. Use fit_with_batches instead if one gradient step per epoch converges too slowly for your dataset. Also use it if a single forward pass does not fit your memory budget (see below).

Each epoch, fit runs one train_batch call. The subsection below uses this same public, single-step method to build custom loops. train_batch performs these steps, in order:

1. Forward through every layer in training mode.
2. Compute the scalar loss.
3. Compute the loss gradient with respect to the output.
4. Advance the optimizer’s global step once. This lets Adam advance its bias-correction timestep once per step, not once per layer.
5. Backpropagate through the layers in reverse, so each layer stashes its parameter gradients.
6. Apply clip-by-global-norm, if the optimizer requests it (see Optimizer::global_clipnorm).
7. Update every layer’s parameters.

step() runs before the per-layer update() calls. This order keeps step-dependent optimizers correct across multiple layers.

fit returns a History. Its entire API is loss(): one f32 per epoch, in epoch order. epochs = 0 gives an empty slice. Each entry is the mean per-sample loss measured during the epoch, before that epoch’s own update. Every batch contributes the loss from the forward pass that ran before that batch’s own weight update. So each entry describes the weights the model held while the epoch ran, never the weights the epoch ends with.

Treating the last entry as the trained model’s final loss is wrong in both directions. While training converges, the entry reads above the truth, because the epoch’s own updates already improved on the weights it measured. Once the step size overshoots, the entry reads below the truth, because those same updates made things worse. This is Keras’s convention, not an accident of the loop. evaluate (see below) reports the loss of the model you hold right now.

The show_progress feature only adds a display. It is not the only way to read the loss. When it is on, fit renders a live progress bar. The bar shows the current epoch’s loss with 6 decimal places. fit_with_batches renders a similar bar that tracks the running average loss as its batches complete:

[dependencies]
rustyml = { version = "0.14", features = ["neural_network", "show_progress"] }

[00:00:00] ████████████████████████████████████████ 400/400 | Loss: <current loss>

Without the feature, training runs silently. The returned History holds the same numbers either way, so your code never depends on whether the feature is on.

Mini-batch training with fit_with_batches

pub fn fit_with_batches(&mut self, x: &Tensor, y: &Tensor, epochs: u32, batch_size: usize)
    -> Result<History, Error>;

fit_with_batches is the mini-batch loop. It reshuffles the sample order at the start of every epoch, then trains on fixed-size chunks. The shuffle makes this the one place where the model’s seed matters. Set it with Sequential::new_with_seed(seed) or set_seed(seed) for a reproducible shuffle order (see 7.1). This seed governs only the shuffle. It does not affect weight initialization, which each layer seeds through its own with_random_state call.

A batch_size of 0, or one larger than the dataset, returns Error::InvalidParameter. A batch_size equal to n_samples degenerates to a single full-batch step per epoch, matching fit.

Its History entries mean what fit’s entries mean, except for one refinement that shows only when batch_size does not divide the dataset evenly. Each batch contributes to the epoch loss in proportion to its sample count, not as 1 vote per batch. So a short trailing batch pulls the epoch figure less than a full batch does. This makes each entry exactly the dataset-wide mean per-sample loss, which matches what Keras reports. Keras’s loss metric accumulates every batch with sample_weight = batch_size, and a plain mean over batches would not match that. A test pins both the weighting and the before-the-update timing against numbers taken from Keras 3.15: tests/neural_network/sequential.rs::test_batch_losses_and_epoch_mean_match_keras.

use ndarray::Array;
use rustyml::prelude::*;

fn main() {
    let x = Array::ones((8, 3)).into_dyn();
    let y = Array::zeros((8, 1)).into_dyn();

    // Seed the per-epoch shuffle so the run is reproducible.
    let mut model = Sequential::new_with_seed(0);
    model
        .add(Dense::new(3, 6, Activation::ReLU).unwrap())
        .add(Dense::new(6, 1, Activation::Sigmoid).unwrap())
        .compile(
            SGD::new(0.05, 0.9, false, 0.0).unwrap(),
            BinaryCrossEntropy::new(),
        );

    // 4 mini-batches of 2 samples per epoch, reshuffled every epoch.
    let history = model.fit_with_batches(&x, &y, 5, 2).unwrap();
    assert_eq!(history.loss().len(), 5);

    // External LR schedule: read the step size, halve it, write it back. The optimizer keeps
    // its momentum buffers across the change.
    let lr = model.learning_rate().unwrap();
    model.set_learning_rate(lr * 0.5);
    let resumed = model.fit_with_batches(&x, &y, 5, 2).unwrap();

    // Training resumes where it left off rather than restarting.
    assert!(resumed.loss()[0] < *history.loss().last().unwrap());
    assert_eq!(model.predict(&x).unwrap().shape(), &[8, 1]);
}

set_learning_rate, used above, is the hook for external schedules such as step decay or warmup. It retunes the step size in place. The optimizer keeps all of its accumulated state, such as momentum buffers and Adam moments, across the change. It does nothing if the model has not been compiled.

learning_rate() is its read half. It returns None on an uncompiled model. This read-scale-write pattern avoids keeping a second copy of the rate beside the model, which could drift out of sync with the optimizer. Unlike the optimizer constructors, set_learning_rate validates nothing. learning_rate() returns exactly the value you set, even a zero or a negative rate.

Writing the loop yourself: train_batch and evaluate

pub fn train_batch(&mut self, x: &Tensor, y: &Tensor) -> Result<f32, Error>;
pub fn evaluate(&self, x: &Tensor, y: &Tensor) -> Result<f32, Error>;

train_batch is the single step that both fit variants build on. It is public. Any other epoch structure is a loop you write, instead of a fork of the library. Examples include curriculum ordering, a per-step schedule, or a probe between steps.

The whole of x is the batch. Nothing is split, nothing is shuffled, and mode-dependent layers run in training mode. The returned f32 is the loss from the forward pass, measured before this call’s own update. This is the number fit records for each epoch, and the value that fit_with_batches averages, weighted by sample count, into each History entry. Keras calls this method train_on_batch.

train_batch validates its own inputs instead of trusting the caller to have done it. So calling it on an uncompiled model returns a NotCompiled error, not a panic.

evaluate is the other half. It runs one inference-mode forward pass over the whole of x and scores it with the compiled loss. It updates nothing: no gradients, no parameters, and no batch-norm running statistics. It borrows &self, so scoring a model between training steps cannot disturb it. It also draws from no random number generator, so calling it inside a fit_with_batches loop cannot perturb the shuffle stream.

Layers behave exactly as they do in predict. Dropout and noise layers act as the identity, and batch normalization reads its running statistics. So on a model with such layers, evaluate and the number fit recorded for the same data disagree. Training-mode dropout inflates the number fit records. evaluate gives the more accurate estimate of the two.

Together, train_batch and evaluate turn early stopping, learning-rate schedules, and checkpoint selection into ordinary code you write. The loop below takes one full-batch step at a time. It scores the model it holds after each step. It halves the step size after 10 steps without an improvement, and stops once the schedule runs out:

use ndarray::Array;
use rustyml::prelude::*;

fn main() {
    // 8 points of y = 2x + 1.
    let x = Array::from_shape_vec((8, 1), (0..8).map(|i| i as f32 / 8.0).collect::<Vec<_>>())
        .unwrap()
        .into_dyn();
    let y = x.mapv(|v| 2.0 * v + 1.0);

    let mut model = Sequential::new();
    model
        .add(Dense::new(1, 8, Activation::Tanh).unwrap().with_random_state(0))
        .add(Dense::new(8, 1, Activation::Linear).unwrap().with_random_state(0))
        .compile(SGD::new(0.1, 0.9, false, 0.0).unwrap(), MeanSquaredError::new());

    let start = model.evaluate(&x, &y).unwrap();
    let (mut best, mut stale, mut previous) = (start, 0, start);

    for _ in 0..500 {
        // The step reports the loss it started from. This stack has no dropout, so that
        // number is the previous `evaluate` measured again, one step stale.
        let during = model.train_batch(&x, &y).unwrap();
        assert!((during - previous).abs() < 1e-5);

        // This one scores the weights the step produced.
        let after = model.evaluate(&x, &y).unwrap();
        previous = after;

        if after < best - 1e-6 {
            best = after;
            stale = 0;
            continue;
        }

        // 10 steps without progress: halve the step size, read from the optimizer instead of
        // a copy kept here. Once it is that small, there is nothing left to try.
        stale += 1;
        if stale == 10 {
            let lr = model.learning_rate().unwrap();
            if lr < 1e-4 {
                break;
            }
            model.set_learning_rate(lr * 0.5);
            stale = 0;
        }
    }

    assert!(best < start / 100.0);
}

3.1.6. predict: forward-only inference

pub fn predict(&self, x: &Tensor) -> Result<Tensor, Error>;

predict runs the inference forward path (Layer::predict) through every layer and returns the output Tensor. It borrows &self, allocates no backward caches, and puts mode-dependent layers into inference behavior. Dropout is disabled, and batch normalization uses its running statistics, which is the correct behavior for serving. predict is deterministic: 2 calls on the same input return identical tensors. Unlike training and evaluation, predict does not need compile.

The result shape is whatever the last layer emits, (batch, output_dim) for a Dense tail. Input must match each layer’s expectations. A Dense layer requires a 2D (batch, features) tensor and returns Error::InvalidInput for anything else.

3.1.7. A complete example: learning XOR

XOR is the smallest problem that a linear model cannot solve. It is the classic proof that a hidden layer does real work. This example uses 2 Dense layers: a Tanh hidden layer and a Softmax head over 2 classes. Training with Adam and categorical cross-entropy separates the classes cleanly. The targets are one-hot: class 0 is [1, 0], and class 1 is [0, 1]. Seeding both layers’ weight initialization with with_random_state(0) makes the run reproducible.

use ndarray::Array;
use rustyml::prelude::*;

fn main() {
    // XOR inputs, shape (4, 2).
    let x = Array::from_shape_vec((4, 2), vec![0.0_f32, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 1.0])
        .unwrap()
        .into_dyn();

    // One-hot targets: XOR is class 1 for (0,1) and (1,0), class 0 otherwise.
    let y = Array::from_shape_vec((4, 2), vec![1.0_f32, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0])
        .unwrap()
        .into_dyn();

    let mut model = Sequential::new();
    model
        .add(Dense::new(2, 8, Activation::Tanh).unwrap().with_random_state(0))
        .add(Dense::new(8, 2, Activation::Softmax).unwrap().with_random_state(0))
        .compile(
            Adam::new(0.1, 0.9, 0.999, 1e-8, 0.0).unwrap(),
            CategoricalCrossEntropy::new(false),
        );

    model.summary();
    let history = model.fit(&x, &y, 400).unwrap();
    assert!(*history.loss().last().unwrap() < history.loss()[0] / 100.0);

    let preds = model.predict(&x).unwrap();
    for i in 0..4 {
        // argmax over the 2 class probabilities
        let class = if preds[[i, 0]] >= preds[[i, 1]] { 0 } else { 1 };
        println!("row {i} -> class {class}  probs [{:.3}, {:.3}]", preds[[i, 0]], preds[[i, 1]]);
    }
}

After 400 full-batch epochs, the network assigns the 2 true XOR rows to class 1 and the 2 false rows to class 0. Each probability sits at or near 1.000. The History records that descent. Its last epoch sits more than 2 orders of magnitude below its first, which is what the assertion checks. The CategoricalCrossEntropy::new(false) argument tells the loss that the head already produces probabilities, from the Softmax layer, so the loss does not apply its own log-softmax. Pass true only when your last layer emits raw logits.

3.1.8. Errors, shape checking, and the mismatch panic

fit, fit_with_batches, and train_batch validate the model and the inputs before touching any layer. The checks run in this order:

1. An optimizer is present.
2. A loss is present.
3. The model has at least 1 layer.
4. The inputs have a batch axis.
5. The inputs are not empty.
6. x and y agree on batch size.

evaluate runs the same checks, except for the optimizer check, since only a parameter update needs an optimizer. The table below maps each failure to its error variant. All of them are rustyml::error::Error values:

The rank-0 row needs an explanation. A 0-dimensional tensor is not an empty one. It holds exactly 1 element, so is_empty does not reject it, and the batch-axis index that follows has nothing to read. This used to panic. Now it returns InvalidInput.

These are recoverable Result values. The next example shows the compile requirement. It shows how training and evaluation each demand something different from compile, and how predict needs neither:

use ndarray::Array;
use rustyml::error::Error;
use rustyml::neural_network::NnError;
use rustyml::prelude::*;

fn main() {
    let x = Array::ones((4, 2)).into_dyn();
    let y = Array::ones((4, 1)).into_dyn();

    let mut model = Sequential::new();
    model.add(Dense::new(2, 1, Activation::Linear).unwrap());

    // fit needs an optimizer and a loss. Without compile, it fails fast and names the first
    // thing it found missing.
    match model.fit(&x, &y, 1) {
        Err(Error::NeuralNetwork(NnError::NotCompiled(missing))) => assert_eq!(missing, "optimizer"),
        other => panic!("expected NotCompiled, got {other:?}"),
    }

    // evaluate updates nothing, so it asks only for the loss it scores with.
    match model.evaluate(&x, &y) {
        Err(Error::NeuralNetwork(NnError::NotCompiled(missing))) => {
            assert_eq!(missing, "loss function")
        }
        other => panic!("expected NotCompiled, got {other:?}"),
    }

    // predict, by contrast, never needs compile: it is forward-only.
    assert_eq!(model.predict(&x).unwrap().shape(), &[4, 1]);
}

One case is a genuine gap in the Result-based error handling. A dimension mismatch between adjacent layers is not a Result. It is a panic. add never checks that consecutive layers agree. So a Dense(2 -> 4) layer feeding a Dense(8 -> 2) layer builds without complaint. The inconsistency surfaces only when data reaches the second layer’s matrix multiply, on the first call to fit or predict:

let mut model = Sequential::new();
model
    .add(Dense::new(2, 4, Activation::ReLU).unwrap())    // emits 4 columns
    .add(Dense::new(8, 2, Activation::Softmax).unwrap()); // expects 8, a mismatch

let x = Array::ones((3, 2)).into_dyn();
let _ = model.predict(&x); // panics inside the GEMM, does not return Err

thread 'main' panicked at gemmkit-ndarray-0.1.2/src/fused.rs:85:5:
assertion `left == right` failed: gemmkit-ndarray: A.cols (4) != B.rows (8)
  left: 4
 right: 8

This panic comes from the matrix-product backend, not from RustyML. Dense::forward hands the product straight to gemmkit-ndarray, so the assertion that fires belongs to the backend. The path in the message is a crates.io registry path, not a path in this repository. This is expected behavior, not a sign of an internal bug.

Treat inter-layer widths as a build-time invariant that you must enforce. Each layer’s input_dim must equal the previous layer’s units. A mismatch is a programming error, not bad input, so it aborts instead of returning an ordinary Error. Get the widths right in the constructors, which is what summary checks for you.

Then this panic never fires. Get them wrong, and the first data that flows through the model reveals it. The panic message names the 2 extents that disagree: here, the 4 columns the first layer emits against the 8 rows the second layer expects.

Each building block has its own page from here. See Dense layers and activations, loss functions, optimizers, the convolutional and recurrent layers, and saving and loading weights.

3.2. Dense Layers and Activations

Dense is the main layer of a feedforward network. It applies a linear map, input * W + b, then an elementwise nonlinearity. RustyML fuses that nonlinearity into the layer itself. It does not treat the nonlinearity as a separate stage.

This page covers the constructor and its fused-activation design. It also covers the exact initialization scheme and the five activations, with their gradients and failure modes. It covers the GEMM-backed compute path and how to read or inject weights. It assumes you have already read the Sequential model page. The layers here are what you stack inside a Sequential model.

3.2.1. The fused-activation design

The constructor takes the activation as its third argument, not as a separate layer:

pub fn new(
    input_dim: usize,
    units: usize,
    activation: impl Into<Activation>,
) -> Result<Dense, Error>

Internally the layer stores an Activation value. This is a plain Copy enum with 5 variants. It is not a generic type parameter Dense<A>, and it is not a Box<dyn Activation>. That choice is deliberate. A generic parameter would monomorphize Dense 5 ways.

It would also force weight deserialization to probe every Dense<A> pairing to find the concrete type on disk. A trait object would add an indirection to every elementwise call. The runtime enum keeps Dense a single concrete type. The persistence layer can then downcast every saved layer to exactly one struct (see Saving and Loading Weights). The activation math stays a pure, stateless function, and the layer calls it inside its own forward and backward passes.

Keras makes the same fusion choice, with Dense(units, activation="relu"). RustyML has no implicit “no activation”. You must always pass one. To get a pure linear layer, pass Activation::Linear, the identity. This is the equivalent of Keras’ activation=None. Every example below that ends in a regression head does exactly this.

Fusing is the right default. RustyML also provides the activations as standalone layers (ReLU, Sigmoid, Tanh, Softmax, Linear). Use these when you need something between the linear map and the nonlinearity. A normalization layer is the most common case (see Regularization and Normalization Layers). The two forms are equivalent:

// These two stacks compute the same thing.
model.add(Dense::new(64, 32, Activation::ReLU).unwrap());          // fused: one cached output

model.add(Dense::new(64, 32, Activation::Linear).unwrap())         // split: identity Dense ...
     .add(ReLU::new());                                            // ... then a standalone ReLU

Prefer the fused form. It caches a single activated tensor instead of two. The backward pass then differentiates the activation in terms of that cached output (see 3.2.5). Use the split form only when a layer must sit between W*x + b and the nonlinearity.

3.2.2. Constructing and sizing a layer

Dense::new(input_dim, units, activation) returns Result<Dense, Error>. Both dimensions must be non-zero. Passing 0 for either yields Error::InvalidParameter (see Error Handling). input_dim is the number of features per row. units is the number of neurons, and therefore the output width.

The parameter count is input_dim * units + units: one weight per (input, output) pair, plus one bias per output. param_count() reports it as TrainingParameters::Trainable(n). output_shape() renders (None, units), where None is the dynamic batch dimension, mirroring Keras’ summary. A Dense(4, 3, ...) therefore holds a 4 x 3 weight matrix and a 1 x 3 bias. That is 12 + 3 = 15 trainable scalars.

Input must be a 2-D tensor of shape (batch, input_dim). A 1-D or 3-D input is rejected with Error::InvalidInput, rather than being silently reshaped. If you are feeding the output of a convolutional or recurrent stack, flatten it to 2 dimensions first. The following program constructs a layer, inspects it, and injects known weights:

use ndarray::Array2;
use rustyml::neural_network::layers::layer_weight::LayerWeight;
use rustyml::neural_network::layers::{Activation, Dense, TrainingParameters};
use rustyml::neural_network::traits::Layer;

fn main() {
    // A 4 -> 3 dense layer with a fused ReLU activation.
    let mut dense = Dense::new(4, 3, Activation::ReLU).unwrap();

    // 4*3 weights + 3 bias = 15 trainable scalars.
    assert_eq!(dense.param_count(), TrainingParameters::Trainable(15));
    println!("output shape: {}", dense.output_shape()); // (None, 3)

    // Read the freshly initialized parameters without cloning them.
    match dense.get_weights() {
        LayerWeight::Dense(w) => {
            println!("weight {:?}, bias {:?}", w.weight.shape(), w.bias.shape());
        }
        _ => unreachable!(),
    }

    // Inject known weights. The shapes are validated against the layer's config.
    let weights =
        Array2::from_shape_vec((4, 3), (0..12).map(|v| v as f32).collect::<Vec<f32>>()).unwrap();
    let bias = Array2::zeros((1, 3));
    dense.set_weights(weights, bias).unwrap();
}

set_weights checks both shapes and returns Error::NeuralNetwork(NnError::WeightShape) on a mismatch. A (3, 3) weight for a 4 -> 3 layer, or a (1, 4) bias, is rejected rather than truncated.

3.2.3. Weight initialization

Weights use Xavier/Glorot uniform initialization. Each element is drawn from Uniform(-limit, +limit), where limit = sqrt(6 / (input_dim + units)). Biases start at exactly zero. This is the fan-in-plus-fan-out Glorot scheme. It matches Keras’ Dense default exactly (glorot_uniform kernel, zeros bias).

RustyML applies Glorot regardless of the activation. It does not switch to He/Kaiming initialization for ReLU layers, even though He is the textbook match for rectifiers. For shallow nets this rarely matters. For a deep ReLU stack, early convergence may be slightly slower than a He-initialized equivalent. If that happens, initialize with set_weights from your own scaled draw.

Initialization draws through the crate’s shared RNG. By default the seed comes from the process-global seed if one is set, otherwise from entropy. Two runs then give different weights unless you fix the randomness. There are 2 ways to make it reproducible. Set a thread-local global seed with rustyml::random::set_global_seed(...) before you construct the model. This also fixes dropout masks and the fit-time batch shuffle.

You can instead seed one layer explicitly, with Dense::new(...)?.with_random_state(seed). This re-runs Glorot with that seed and leaves the global stream untouched. The full seeding model is in Reproducibility and Random Seeds. It also explains why an explicit per-layer seed does not change the seeds handed to unseeded layers.

3.2.4. The five activations

Activation has exactly 5 variants. The crate has no LeakyReLU, ELU, GELU, or Swish. Each variant’s backward pass is expressed in terms of the activated output a = f(x), not the pre-activation x. The layer caches that output, not the input.

ReLU is the default hidden-layer choice. It is cheap, and it does not saturate on the positive side. Its failure mode is the dead neuron. The derivative is 0 for x <= 0, so a neuron whose pre-activation is negative for every example in the batch gets zero gradient.

It never updates, and it stays off for good. A high learning rate makes this worse. It pushes neurons into the dead region early. Glorot init has no leaky variant. Your options are a smaller learning rate and well-scaled inputs, or, if you inject your own weights, a better initial scale.

Sigmoid squashes values to (0, 1). Its derivative a * (1 - a) peaks at 0.25 when a = 0.5, and it decays toward zero as the output saturates. Stacking sigmoids in a deep hidden path throttles gradients: the classic vanishing-gradient problem. Use Sigmoid for a single binary output, paired with binary cross-entropy, or as a gate. Do not use it as a deep hidden nonlinearity. At extreme inputs, f32 saturates it to exactly 0.0 or 1.0.

Tanh maps to (-1, 1). Unlike sigmoid, it is zero-centered, which tends to make hidden-layer optimization better behaved. Its gradient 1 - a^2 still reaches 1 near the origin. It saturates the same way sigmoid does at the tails. It is the natural bounded, zero-centered hidden activation, and the recurrent layers use it internally.

Softmax turns a row of logits into a probability distribution over the last axis. The forward pass subtracts each row’s maximum before it takes the exponent. This makes the largest term exp(0) = 1, and the sum always >= 1. That shift makes softmax overflow-proof and shift-invariant: adding a constant to every logit leaves the output unchanged.

Softmax needs at least a 2-D input. A 1-D tensor returns Error::InvalidInput. Its backward pass is the true Jacobian-vector product across the row, not an elementwise multiply, and each gradient row sums to zero.

Softmax belongs on the output layer of a classifier, paired with the correct loss configuration. Mid-network, softmax is mechanically valid: the Jacobian backward is correct, so a Dense(..., Softmax) buried in the stack still trains. It is almost never what you want, though. It collapses the representation onto a simplex, discards all magnitude information, and saturates worse than ReLU or tanh.

Linear is the identity. Its gradient is 1, and its output is unbounded. Use it for regression heads, and whenever the loss function expects raw logits.

The loss function must match the activation on the output head. There are 2 correct pairings for multi-class classification. Mixing them silently corrupts training:

• Softmax head + CategoricalCrossEntropy::new(false). The final Dense emits probabilities. The loss consumes probabilities. Correct.
• Linear head + CategoricalCrossEntropy::new(true). The final Dense emits raw logits. The loss applies a numerically stable log-softmax internally, and returns the fused (softmax(z) - y) gradient in one step. Also correct, and more numerically stable.

The 2 pairings produce the same gradient mathematically. The fused from_logits = true path avoids the intermediate -y/p division and the separate softmax step. It therefore degrades more gracefully when a predicted probability is tiny. Do not mix the two pairings.

A softmax head with from_logits = true applies softmax twice. A linear head with from_logits = false feeds raw logits to a loss that expects probabilities. Loss details live in Loss Functions.

// More stable alternative to a Softmax head: emit logits, and fuse the softmax into the loss.
model
    .add(Dense::new(8, 3, Activation::Linear).unwrap())    // raw logits, NOT probabilities
    .compile(optimizer, CategoricalCrossEntropy::new(true)); // from_logits = true

The activations are pure math, with no NaN/Inf sanitization. A NaN propagates through untouched. tanh saturates to 1 or -1 at large inputs. ReLU of a large negative input is 0. A NaN anywhere in a softmax row contaminates the whole row through the normalizer. Non-finite values surface downstream, as a NaN loss, not at the activation step.

3.2.5. Forward, backward, and the GEMM path

The forward pass is activation(input * W + b). It runs as a single call into the gemmkit backend. The product, the per-column bias, and, for ReLU, the activation, all run in one pass. The bias and activation apply in the kernel’s epilogue while the output tile is still in registers.

The fused result matches the unfused product plus scalar activation bit for bit, with one exception. The fused ReLU epilogue maps a NaN pre-activation to 0.0. The standalone ReLU activation used elsewhere in the crate instead propagates NaN. A NaN pre-activation already means a diverged model. The cached-output ReLU derivative then treats that 0.0 like a dead unit. It backpropagates a zero gradient there, instead of propagating the NaN further back.

Whether the product threads is gemmkit’s decision, not the layer’s. It compares batch * input_dim * units against a work gate, 589,824 by default. That count is the raw product, not a FLOP count with a factor of 2. Below the gate, the product stays on one thread. Above it, the worker count ramps with the work, rather than moving straight to the full machine.

Small layers in a tight loop stay serial by design, because the per-call dispatch would dominate. These products also nest safely inside an outer parallel region. The backend is described in full in Matrix Multiplication, and the tunable gates in Performance Tuning and Parallelism.

The elementwise activation that follows has its own, separate parallelism gate. ReLU is a memory-bound “cheap map” whose crossover sits at 4,000,000 elements. At any practical layer size, it runs serial. Sigmoid, Tanh, and Softmax are exp-dominated, and go parallel above 131,072 elements. Moving these gates only trades serial for parallel. The results are identical, and every product is run-to-run deterministic on the same machine.

forward caches the input and the activated output for the backward pass. predict is the eval-mode twin. It computes the same values, but writes no caches. The backward pass first differentiates the activation using the cached output.

It then computes 3 quantities. The weight gradient is input^T * grad, a GEMM. The bias gradient is the column-sum over the batch. The input gradient is grad * W^T, another GEMM.

Calling backward before forward returns Error::NeuralNetwork(NnError::ForwardPassNotRun). An upstream gradient whose shape does not match the cached output returns Error::ShapeMismatch. Both are errors, never panics.

3.2.6. Reading and setting weights

There is no standalone weights() / bias() getter. The accessor is get_weights() from the Layer trait. It returns a LayerWeight enum. For a dense layer, that is LayerWeight::Dense(DenseLayerWeight { weight, bias }), where weight and bias are Cow<Array2<f32>> borrowed from the live layer, with no clone.

weight has shape (input_dim, units), and bias has shape (1, units), as the construction example in 3.2.2 shows. To write parameters, use set_weights(weights, bias), which validates both shapes. This same LayerWeight enum is the on-disk weight format. Anything you can read here is what round-trips through save and load, in Saving and Loading Weights.

3.2.7. Two worked models

A regression net ends in a Linear head, and trains against mean squared error. The hidden layer fuses ReLU. The output layer fuses Linear, because a regression target is unbounded:

use ndarray::Array;
use rustyml::neural_network::layers::{Activation, Dense};
use rustyml::neural_network::losses::MeanSquaredError;
use rustyml::neural_network::optimizers::SGD;
use rustyml::neural_network::sequential::Sequential;

fn main() {
    // 4 samples, 3 features, 1 continuous target each.
    let x = Array::from_shape_vec(
        (4, 3),
        vec![0.0, 0.1, 0.2, 1.0, 0.9, 0.8, 0.2, 0.1, 0.0, 0.9, 1.0, 0.8],
    )
    .unwrap()
    .into_dyn();
    let y = Array::from_shape_vec((4, 1), vec![0.3, 2.7, 0.3, 2.7])
        .unwrap()
        .into_dyn();

    let mut model = Sequential::new();
    model
        .add(Dense::new(3, 8, Activation::ReLU).unwrap()) // hidden, fused ReLU
        .add(Dense::new(8, 1, Activation::Linear).unwrap()) // regression head: identity
        .compile(SGD::new(0.05, 0.9, false, 0.0).unwrap(), MeanSquaredError::new());

    model.fit(&x, &y, 20).unwrap();

    let preds = model.predict(&x).unwrap();
    println!("prediction shape: {:?}", preds.shape()); // [4, 1]
}

A classifier ends in a Softmax head over one-hot targets, paired with CategoricalCrossEntropy::new(false), because the head emits probabilities:

use ndarray::Array;
use rustyml::neural_network::layers::{Activation, Dense};
use rustyml::neural_network::losses::CategoricalCrossEntropy;
use rustyml::neural_network::optimizers::Adam;
use rustyml::neural_network::sequential::Sequential;

fn main() {
    // 4 samples, 3 features, 2 classes (one-hot targets).
    let x = Array::from_shape_vec(
        (4, 3),
        vec![0.0, 0.1, 0.2, 1.0, 0.9, 0.8, 0.1, 0.0, 0.2, 0.8, 1.0, 0.9],
    )
    .unwrap()
    .into_dyn();
    let y = Array::from_shape_vec((4, 2), vec![1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0])
        .unwrap()
        .into_dyn();

    let mut model = Sequential::new();
    model
        .add(Dense::new(3, 8, Activation::ReLU).unwrap())
        .add(Dense::new(8, 2, Activation::Softmax).unwrap()) // probability head
        .compile(
            Adam::new(0.01, 0.9, 0.999, 1e-8, 0.0).unwrap(),
            CategoricalCrossEntropy::new(false),
        );

    model.fit(&x, &y, 20).unwrap();

    let probs = model.predict(&x).unwrap();
    // Each row is a distribution over the 2 classes, summing to 1.
    println!("class-probability shape: {:?}", probs.shape()); // [4, 2]
}

Swap the head to Activation::Linear, and the loss to CategoricalCrossEntropy::new(true). This classifier then trains on the more stable fused-logits path, and predict produces raw scores instead of probabilities. Which optimizer to compile with, and how learning rate and momentum interact with the dead-neuron and saturation behavior above, is the subject of Optimizers.

3.3. Loss Functions

A loss function is the scalar objective the training loop drives toward zero. It is also the source of the first gradient that flows backward through the network. RustyML ships 5 losses in rustyml::neural_network::losses. MeanSquaredError and MeanAbsoluteError handle regression. BinaryCrossEntropy handles binary and multi-label classification. CategoricalCrossEntropy and SparseCategoricalCrossEntropy handle mutually exclusive multi-class problems. Each loss is a small, unit-like struct that implements the Loss trait. This lets every loss compose the same way inside a Sequential model. You can also call a loss directly, for evaluation or for a hand-rolled training loop.

All 5 losses carry no per-example state. They are cheap to construct and to copy (Debug + Clone + Copy + PartialEq). They differ in the shapes they accept, how they normalize, and how they stay numerically stable at the edges of their domains. This page states those facts straight from the implementation, so you know what a loss does before you compile it into a model.

3.3.1. The Loss trait and averaging conventions

Every loss implements two methods:

pub trait Loss {
    fn compute_loss(&self, y_true: &Tensor, y_pred: &Tensor) -> Result<f32, Error>;
    fn compute_grad(&self, y_true: &Tensor, y_pred: &Tensor) -> Result<Tensor, Error>;
}

Tensor is an alias for ndarray::ArrayD<f32>, a dynamically dimensioned f32 array. Note the argument order: y_true comes first, then y_pred. Both methods return a Result. Shape and label validation happen up front, instead of the code panicking mid-computation. See Error Handling for the error types. compute_grad returns the gradient of compute_loss with respect to the predictions. The two are always exactly consistent, so the gradient is not an approximation.

The losses do not all normalize the same way, on purpose. The regression and binary losses average over every element of the tensor (y.len()), and treat each output as an independent target. CategoricalCrossEntropy instead sums over the trailing class axis and averages over the prediction sites, the product of every axis before it. For the usual [batch, classes] target, that divisor is the batch size. For the [batch, height, width, classes] output of a channels-last Conv2D softmax head, the divisor is batch * height * width, 1 prediction per pixel. This matches Keras’ default sum_over_batch_size reduction. SparseCategoricalCrossEntropy accepts rank-2 predictions only, so its divisor is always the batch size. Switching a model from, for example, MeanSquaredError to CategoricalCrossEntropy rescales the gradient magnitude by roughly the number of classes. This changes the effective learning rate. If a learning rate that worked well before suddenly diverges or stalls after you change the loss, check the rescaling first. Adjust the rate on the optimizer to compensate.

The input shapes also differ. Getting them wrong is the most common cause of a rejected fit call. The table below lists every contract, checked against each loss’s validation code.

3.3.2. Mean Squared Error

MSE is the default loss for regression. The forward value is mean((y_pred - y_true)^2) over all elements. The gradient is 2 * (y_pred - y_true) / n, where n is the total element count. The factor of 2 comes from the derivative of the squared term. RustyML folds that factor into the gradient, not into the loss, so compute_grad is the exact gradient of compute_loss.

Because the error term is squared, MSE weights large residuals quadratically. A prediction that is off by 10 contributes 100 times more to the loss than one off by 1. MSE is the right choice when large errors are truly worse and your targets cluster around the true value in a roughly Gaussian way. MSE is the wrong choice when your data has heavy-tailed noise or outliers that you do not want the model to fit closely.

use rustyml::neural_network::Tensor;
use rustyml::neural_network::losses::MeanSquaredError;
use rustyml::neural_network::traits::Loss;
use ndarray::Array;

fn main() {
    let mse = MeanSquaredError::new();

    let y_true: Tensor = Array::from_shape_vec(vec![2, 2], vec![1.0_f32, 2.0, 3.0, 4.0])
        .unwrap()
        .into_dyn();
    let y_pred: Tensor = Array::from_shape_vec(vec![2, 2], vec![1.0_f32, 3.0, 5.0, 4.0])
        .unwrap()
        .into_dyn();

    let loss = mse.compute_loss(&y_true, &y_pred).unwrap();
    let grad = mse.compute_grad(&y_true, &y_pred).unwrap();

    println!("MSE loss: {loss:.4}");        // mean of squared diffs
    println!("grad shape: {:?}", grad.shape());
}

If the two shapes differ, both methods return Err(Error::ShapeMismatch { .. }). This stops an ndarray broadcast panic from escaping. For the metrics you report after training, such as R2 and explained variance, see Regression Metrics. The loss here is the training objective, not the evaluation report.

3.3.3. Mean Absolute Error

MAE is the loss that tolerates outliers better than MSE. The forward value is mean(|y_pred - y_true|), so residuals contribute linearly, not quadratically. An outlier that is off by 10 adds 10 times as much to the loss as one off by 1, not 100 times. Use MAE when your targets contain outliers that you do not want to dominate training.

The gradient needs more care. The derivative of |x| is sign(x). The MAE gradient is +1/n where the prediction is above the target and -1/n where it is below. At an exact tie, where y_pred == y_true, RustyML returns 0.0 instead of an arbitrary +1/n or -1/n. This is a deliberate subgradient choice. A perfect prediction gives a true zero gradient. The kink at zero is real. The gradient size does not shrink as the prediction approaches the target, unlike MSE, whose gradient scales with the residual. Plain SGD on MAE can therefore oscillate around the optimum. Pairing MAE with an adaptive optimizer that scales steps by gradient history reduces this oscillation in practice.

The sign logic has a final branch. It returns f32::NAN for any residual that is neither greater than 0, less than 0, nor equal to 0. This branch only fires for a NaN residual. Finite inputs never reach it. If a NaN value has already entered your predictions, MAE does not convert it to a clean number. This matches the library’s policy: surface non-finite values instead of hiding them.

use rustyml::neural_network::Tensor;
use rustyml::neural_network::losses::{MeanAbsoluteError, MeanSquaredError};
use rustyml::neural_network::traits::Loss;
use ndarray::Array;

fn main() {
    // 4 clean points, plus 1 large outlier in the last position.
    let y_true: Tensor = Array::from_shape_vec(vec![5], vec![1.0_f32, 2.0, 3.0, 4.0, 5.0])
        .unwrap()
        .into_dyn();
    let y_pred: Tensor = Array::from_shape_vec(vec![5], vec![1.0_f32, 2.0, 3.0, 4.0, 15.0])
        .unwrap()
        .into_dyn();

    let mse = MeanSquaredError::new();
    let mae = MeanAbsoluteError::new();

    // The 10-unit error adds about 100/5 to MSE, but only about 10/5 to MAE.
    // The outlier dominates MSE. It does not dominate MAE.
    println!("MSE: {:.4}", mse.compute_loss(&y_true, &y_pred).unwrap());
    println!("MAE: {:.4}", mae.compute_loss(&y_true, &y_pred).unwrap());
}

3.3.4. Binary Cross-Entropy

BinaryCrossEntropy fits problems where each output is an independent yes/no decision. This covers 1 binary label per sample, or several independent binary labels per sample (multi-label classification). The forward value is mean(-[y * ln(p) + (1 - y) * ln(1 - p)]) over every element. Here, y is a 0.0/1.0 label and p is a predicted probability. Because it averages over every element, a multi-label output of shape [batch, labels] averages over batch * labels. This is the conventional mean binary cross-entropy.

Get 2 things right. First, the label format: y_true must hold 0.0 or 1.0 floats, not class indices or logits. Second, y_pred must be a probability in (0, 1). This loss has no logits mode. You must squash your network’s output into (0, 1) yourself, in practice with a Sigmoid final layer (see Dense Layers and Activations). This is a real difference from Keras. Keras’ BinaryCrossentropy accepts a from_logits flag. RustyML’s does not, so the Sigmoid layer is mandatory, not optional.

A prediction of exactly 0.0 or 1.0 could make ln(0) and division by zero produce NaN or Inf. To prevent this, the loss clips every probability into [1e-7, 1 - 1e-7] before it takes a log, in both the forward and gradient paths. Consider a prediction of exactly 1.0 for a positive label, a fully confident and correct case. It yields a tiny but finite loss (about 1e-7) and a finite gradient, not a zero loss and a NaN gradient.

use rustyml::neural_network::Tensor;
use rustyml::neural_network::losses::BinaryCrossEntropy;
use rustyml::neural_network::traits::Loss;
use ndarray::Array;

fn main() {
    let bce = BinaryCrossEntropy::new();

    // Labels are 0.0 or 1.0. Predictions are probabilities in (0, 1).
    let y_true: Tensor = Array::from_shape_vec(vec![4], vec![0.0_f32, 1.0, 1.0, 0.0])
        .unwrap()
        .into_dyn();
    let y_pred: Tensor = Array::from_shape_vec(vec![4], vec![0.1_f32, 0.9, 0.8, 0.2])
        .unwrap()
        .into_dyn();

    println!("BCE loss: {:.4}", bce.compute_loss(&y_true, &y_pred).unwrap());

    // Even at the boundary, clipping keeps the loss finite.
    let hard_pred: Tensor = Array::from_shape_vec(vec![4], vec![0.0_f32, 1.0, 1.0, 0.0])
        .unwrap()
        .into_dyn();
    let loss = bce.compute_loss(&y_true, &hard_pred).unwrap();
    assert!(loss.is_finite());
    println!("BCE at boundary predictions: {loss:.6}");
}

3.3.5. Categorical Cross-Entropy

CategoricalCrossEntropy handles mutually exclusive multi-class classification with one-hot targets. The last axis is always the class axis. Every axis before it indexes an independent prediction site. y_true and y_pred of shape [batch, classes] give 1 prediction per sample. The [batch, height, width, classes] output of a channels-last Conv2D softmax head gives 1 prediction per pixel, the per-pixel segmentation case. The loss sums the cross-entropy over the class axis. It then divides by the total number of prediction sites, the product of the leading axes. That divisor is batch at rank 2, and batch * height * width at rank 4.

The loss also requires at least a 2-D, non-empty input. A 1-D input is rejected with Error::InvalidInput. A 1-D tensor has no leading axis, which would leave a divisor of 1 and a softmax over the only axis present. An empty input is rejected with Error::EmptyInput.

The constructor takes 1 boolean argument, from_logits. It controls 2 distinct modes:

CategoricalCrossEntropy::new(false) // y_pred is already a probability distribution
CategoricalCrossEntropy::new(true)  // y_pred is raw logits, the loss applies softmax internally

With from_logits = false, the default, y_pred should be a probability distribution along its last axis, the output of a Softmax layer. Following Keras, the loss first renormalizes each row with y_pred / sum(y_pred, axis=-1). Only then does it clip into [1e-7, 1 - 1e-7], giving -sum(y_true * ln(q_clipped)) / sites.

A softmax row already sums to 1, so this division does not change the loss value. The division is still part of what compute_grad differentiates. The gradient is (sum(y_true) - y_true / q_clipped) / (row_sum * sites). The whole bracket is divided by the row sum. Written in terms of the unnormalized p, this is (sum(y_true) / row_sum - y_true / p) / sites. That is the familiar -y/p term, plus a term that stays constant across each row. A softmax backward pass cancels any row-constant term, so a softmax head trains the same either way. The extra term only matters when this loss reads a head that is not a softmax.

The renormalization has 1 practical benefit. A head whose rows do not quite sum to 1 gets scored as the distribution it implies, instead of being penalized for its scale.

With from_logits = true, the loss treats y_pred as raw, unnormalized logits. It applies softmax itself, using a numerically stable log-softmax. Prefer this mode for training, for 2 reasons.

The first reason is stability. Computing softmax(z) and then ln(...) naively overflows exp for large logits. It also loses precision when it takes the log of an already-clipped near-zero probability. The internal path instead subtracts each prediction site’s maximum before it exponentiates, so exp never overflows. It then computes log_softmax = z - logsumexp(z) directly, so it never logs a clipped probability. This normalization runs within 1 prediction site, never across sites. The pixels of a rank-4 conv head therefore do not compete with each other for probability mass.

The second reason is efficiency. The gradient with respect to the logits collapses to the fused form (softmax(z) - y_true) / sites. This form is cheaper to compute. It also avoids the -y/p division, which explodes when p is near 0.

This has 1 practical consequence. The gradient in logits mode is with respect to the logits. Your network’s final layer must therefore output logits, not probabilities. Use a Linear final activation, with no Softmax layer. If you keep a Softmax layer and also pass from_logits = true, the model applies softmax twice, which corrupts training.

use rustyml::neural_network::Tensor;
use rustyml::neural_network::losses::CategoricalCrossEntropy;
use rustyml::neural_network::traits::Loss;
use ndarray::Array;

fn main() {
    // Probability mode: y_pred must already be a softmax distribution.
    let cce_probs = CategoricalCrossEntropy::new(false);
    let y_true: Tensor = Array::from_shape_vec(
        vec![3, 3],
        vec![1.0_f32, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0],
    )
    .unwrap()
    .into_dyn();
    let probs: Tensor = Array::from_shape_vec(
        vec![3, 3],
        vec![0.8_f32, 0.1, 0.1, 0.2, 0.7, 0.1, 0.1, 0.2, 0.7],
    )
    .unwrap()
    .into_dyn();
    println!("CCE (probs): {:.4}", cce_probs.compute_loss(&y_true, &probs).unwrap());

    // Logits mode: y_pred is raw logits, and the loss applies softmax internally.
    let cce_logits = CategoricalCrossEntropy::new(true);
    let logits: Tensor = Array::from_shape_vec(vec![1, 3], vec![1.0_f32, 2.0, 0.5])
        .unwrap()
        .into_dyn();
    let label: Tensor = Array::from_shape_vec(vec![1, 3], vec![0.0_f32, 1.0, 0.0])
        .unwrap()
        .into_dyn();
    let loss = cce_logits.compute_loss(&label, &logits).unwrap();
    let grad = cce_logits.compute_grad(&label, &logits).unwrap();
    // grad is the fused (softmax(logits) - one_hot) / prediction sites.
    println!("CCE (logits): {loss:.4}, grad shape {:?}", grad.shape());
}

3.3.6. Sparse Categorical Cross-Entropy

SparseCategoricalCrossEntropy computes the same loss as CategoricalCrossEntropy, but it takes integer class labels instead of one-hot vectors. y_true is a [batch, 1] tensor of class indices, stored as f32 and rounded to the nearest integer. y_pred is a [batch, num_classes] tensor of probabilities or logits. The tests confirm the equivalence. For the same predictions, the sparse loss on integer labels and the dense CCE on the matching one-hot encoding agree to floating-point round-off. Their gradients agree too.

The shape rules are strict. Without them, a malformed label would cause an opaque index-out-of-bounds panic. y_pred must be exactly 2-D. A 3-D prediction is rejected with Error::InvalidInput. y_true must be exactly [batch, 1]. A 1-D label vector or a [batch, 2] shape is rejected. This is a real difference from Keras, whose sparse categorical crossentropy takes a 1-D [batch] label. Each label must be finite, non-negative, and strictly less than num_classes. Negative or out-of-range labels come back as Error::InvalidInput. A batch-size mismatch between labels and predictions comes back as Error::DimensionMismatch. Validation happens once, up front. It extracts a Vec<usize> of class indices before any arithmetic runs.

The memory savings grow with the number of classes. A dense one-hot target is [batch, num_classes] of f32, or batch * num_classes * 4 bytes, almost all of it zeros. The sparse label is [batch, 1], or batch * 4 bytes. This is a num_classes-fold reduction on the target tensor, and you never build the one-hot matrix at all. For a vocabulary of tens of thousands of classes, this is the difference between a trivial label tensor and one larger than the predictions themselves. Use the sparse loss when your labels are naturally integers and num_classes is large. Use the dense CCE when you already have one-hot, or soft, targets.

The from_logits flag works the same way as it does for dense CCE. When false, the loss expects per-row probabilities. It renormalizes each row before the clip, exactly as CategoricalCrossEntropy does, so it scores an unnormalized head as the distribution it implies. The row-normalizer is also part of what compute_grad differentiates. Because of this, every class picks up a shared 1 / (batch * row_sum) term in the gradient, not only the labeled class. A softmax head cancels this shared term in its backward pass, the same way it does for dense CCE. When true, the loss expects logits and applies the same stable log-softmax. The fused gradient is softmax with 1 subtracted at the true class, divided by the batch.

The probability path also sums the per-sample losses serially, instead of with a parallel reduction. This keeps the reported loss from drifting with thread scheduling. See Reproducibility and Random Seeds.

use rustyml::neural_network::Tensor;
use rustyml::neural_network::losses::SparseCategoricalCrossEntropy;
use rustyml::neural_network::traits::Loss;
use ndarray::Array;

fn main() {
    let scce = SparseCategoricalCrossEntropy::new(false);

    // Labels are class indices in a [batch, 1] column, not one-hot rows.
    let y_true: Tensor = Array::from_shape_vec(vec![3, 1], vec![0.0_f32, 1.0, 2.0])
        .unwrap()
        .into_dyn();
    let y_pred: Tensor = Array::from_shape_vec(
        vec![3, 3],
        vec![0.8_f32, 0.1, 0.1, 0.2, 0.7, 0.1, 0.1, 0.2, 0.7],
    )
    .unwrap()
    .into_dyn();

    let loss = scce.compute_loss(&y_true, &y_pred).unwrap();
    let grad = scce.compute_grad(&y_true, &y_pred).unwrap();
    println!("SCCE loss: {loss:.4}");        // equals dense CCE on the one-hot equivalent
    println!("grad shape: {:?}", grad.shape());

    // RustyML rejects out-of-range and negative labels, instead of wrapping them silently.
    let bad: Tensor = Array::from_shape_vec(vec![3, 1], vec![0.0_f32, 1.0, 9.0])
        .unwrap()
        .into_dyn();
    assert!(scce.compute_loss(&bad, &y_pred).is_err());
}

3.3.7. Pairing activations with losses

The final-layer activation and the loss form a matched pair. Mismatch them, and you either feed the loss the wrong domain (probabilities where it wants logits, or the reverse), or you apply softmax twice. The table below lists the correct combinations, and why each one works.

The distinction between the two classification rows matters. Sigmoid plus BinaryCrossEntropy fits independent binary outputs, where a sample can belong to several labels at once. Softmax plus CategoricalCrossEntropy fits mutually exclusive classes, where the probabilities across classes sum to 1. Do not use softmax where the labels are independent. Do not use sigmoid where the labels compete.

The model below builds the preferred multi-class setup: a Linear output feeding CategoricalCrossEntropy in logits mode. It has no Softmax layer, because the loss applies the softmax itself.

use rustyml::neural_network::sequential::Sequential;
use rustyml::neural_network::layers::{Activation, Dense};
use rustyml::neural_network::optimizers::Adam;
use rustyml::neural_network::losses::CategoricalCrossEntropy;
use ndarray::Array;

fn main() {
    // 4 samples, 3 features -> 3 classes.
    let x = Array::from_shape_vec(
        vec![4, 3],
        vec![
            0.1_f32, 0.2, 0.7, 0.9, 0.1, 0.0, 0.2, 0.8, 0.1, 0.7, 0.2, 0.1,
        ],
    )
    .unwrap()
    .into_dyn();
    // One-hot targets: classes 2, 0, 1, 0.
    let y = Array::from_shape_vec(
        vec![4, 3],
        vec![
            0.0_f32, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0,
        ],
    )
    .unwrap()
    .into_dyn();

    let mut model = Sequential::new();
    model
        .add(Dense::new(3, 8, Activation::ReLU).unwrap())
        .add(Dense::new(8, 3, Activation::Linear).unwrap()); // logits, no Softmax

    model.compile(
        Adam::new(0.01, 0.9, 0.999, 1e-8, 0.0).unwrap(),
        CategoricalCrossEntropy::new(true), // from_logits = true
    );

    model.fit(&x, &y, 5).unwrap();
    let preds = model.predict(&x).unwrap();
    println!("prediction shape: {:?}", preds.shape());
}

3.3.8. Numerical stability and edge behavior

3 mechanisms keep these losses finite at the edges of their domains. Each guards a different failure.

Probability clipping guards the cross-entropy losses whenever they take probabilities as input. This covers BinaryCrossEntropy, and CategoricalCrossEntropy / SparseCategoricalCrossEntropy in from_logits = false mode. Every probability is clamped into [1e-7, 1 - 1e-7] before any ln. A prediction of exactly 0.0 or 1.0 therefore produces a large but finite loss and a finite gradient, instead of NaN or Inf.

The clip guards against log(0). It does not gate the gradient. Unlike Keras’ autodiff, compute_grad evaluates at the clipped probability. It does not zero out clipped positions. Loss and gradient agree everywhere inside the interval.

For the categorical losses, the row renormalization happens before the clip. A row that sums to zero still yields a non-finite result, exactly as it does in Keras. The tests check this directly: fully confident predictions and worst-case boundary predictions both stay finite.

The stable log-softmax guards the logits paths. Instead of forming softmax(z) and logging it, the loss subtracts each prediction site’s maximum before it exponentiates. It then computes log_softmax = z - logsumexp(z) directly. This makes from_logits = true more than a convenience. It is a more reliable way to train a classifier. It never overflows exp. It never logs a clipped probability. It returns the well-conditioned fused gradient softmax(z) - y. Prefer this mode when you have a choice.

Nothing sanitizes NaN. MAE’s sign branch returns f32::NAN for a NaN residual, and a NaN prediction propagates through the other losses too. A non-finite value stays non-finite, by design, so divergence surfaces loudly instead of hiding. To tame large but finite gradients before they diverge, clip on the optimizer with clip-by-global-norm, instead of clamping inside the loss.

After training, to score a model, use the dedicated evaluators in Classification Metrics and Regression Metrics. A loss is tuned to be a smooth training signal, not a human-readable report.

3.4. Optimizers

An optimizer in RustyML turns gradients into parameter updates. A layer’s backward pass stashes a gradient for every trainable tensor. The optimizer then walks those tensors and moves each one downhill. You pick an optimizer when you call compile on a Sequential model. You also pick a loss function at the same time. After that, fit drives the optimizer for you.

RustyML has 5 optimizers: SGD, Adam, AdamW, RMSprop, and AdaGrad. All 5 live in rustyml::neural_network::optimizers. The prelude re-exports them. Every constructor returns Result<Self, Error> because it checks its hyperparameters first. Each example below ends its constructor call with .unwrap(), or with real error handling.

The algorithms match the ones in Keras and PyTorch. 2 things differ. The constructors take positional arguments. No argument has a default value. Gradient clipping and weight decay are also part of the optimizer itself, not the training loop.

3.4.1. The optimizer interface and how updates flow

Every optimizer implements the Optimizer trait. The trait has 5 methods, and you use all of them. The training loop calls 3 of the methods for you. A learning-rate schedule reads and writes the other 2, learning_rate and set_learning_rate (see 3.4.8):

pub trait Optimizer {
    fn step(&mut self);                                         // once per batch
    fn update(&mut self, layer: &mut dyn Layer, grad_scale: f32); // once per layer
    fn global_clipnorm(&self) -> Option<f32>;                        // clip threshold, or None
    fn learning_rate(&self) -> f32;                            // current step size
    fn set_learning_rate(&mut self, learning_rate: f32);       // scheduling hook
}

The training loop calls step exactly once per batch, before it touches any layer. It then calls update once per layer. A stateful optimizer advances its notion of time inside step. Adam and AdamW increment a bias-correction timestep there. SGD, RMSprop, and AdaGrad only use step to rewind an internal cursor.

This is why Adam advances its timestep once per batch, not once per layer. A hand-rolled training loop must call step once per batch, not once per layer or once per parameter. Calling it more often breaks Adam’s bias-correction math without any error message.

update receives a grad_scale factor from the training loop. The loop computes grad_scale from global_clipnorm (see below). When clipping is off, grad_scale is 1.0. Inside update, the optimizer asks the layer for its parameters through layer.parameters(). This call returns a flat ParamGrad for each trainable tensor. A ParamGrad holds a mutable value slice, a matching grad slice, and a decays: bool flag.

The decays flag keeps weight decay honest. Weight matrices and convolution or recurrent kernels carry decays = true. Biases and normalization gamma and beta carry decays = false. RustyML never decays a parameter with decays = false, no matter what weight_decay you set. Layers set this flag, and optimizers respect it. You do not manage it yourself.

Layers expose parameters as flat &mut [f32] slices. Because of this, a single per-element kernel handles any tensor shape. Each kernel also switches to a Rayon parallel path once a tensor crosses an element-count threshold (see 7.3).

An optimizer keys its per-parameter state by the position at which a layer yields each tensor. The parameter order must therefore stay stable across steps, and it does. A given optimizer instance belongs to one model. Do not share an optimizer instance across models.

3.4.2. SGD

SGD::new(learning_rate, momentum, nesterov, weight_decay) // all f32 except nesterov: bool

SGD is plain stochastic gradient descent. It supports optional momentum, Nesterov acceleration, and decoupled weight decay. With momentum = 0.0, the update is the textbook form:

param -= lr * grad

Set momentum > 0.0 to accumulate a per-parameter velocity buffer. You can also add the Nesterov look-ahead step:

v    = momentum * v + grad
step = grad + momentum * v   (nesterov = true)   or   v   (nesterov = false)
param -= lr * step

0.9 is the conventional momentum value. nesterov only matters when momentum is non-zero. The weight_decay here is decoupled, the SGDW formulation. Before the gradient step, SGD shrinks weight tensors by param *= (1 - lr * weight_decay), independent of the gradient. AdamW uses this same decoupling, applied here to SGD.

use rustyml::prelude::*;
use rustyml::neural_network::sequential::Sequential;
use ndarray::Array;

fn main() {
    // Tiny inline regression: y = 2*x
    let x = Array::from_shape_vec((4, 1), vec![0.5_f32, 1.0, 1.5, 2.0])
        .unwrap()
        .into_dyn();
    let y = Array::from_shape_vec((4, 1), vec![1.0_f32, 2.0, 3.0, 4.0])
        .unwrap()
        .into_dyn();

    let mut model = Sequential::new();
    model
        .add(Dense::new(1, 1, Activation::Linear).unwrap())
        // learning_rate, momentum, nesterov, weight_decay
        .compile(SGD::new(0.05, 0.9, true, 0.0).unwrap(), MeanSquaredError::new());

    let before = model.predict(&x).unwrap();
    model.fit(&x, &y, 10).unwrap();
    let after = model.predict(&x).unwrap();
    println!("shapes: {:?} -> {:?}", before.shape(), after.shape());
}

Use SGD with momentum when you want fine control and predictable behavior. Its dynamics are well understood. Its single moving average is cheap to keep. SGD with momentum still sets the generalization standard that other methods get measured against.

The cost is sensitivity to the learning rate. A rate too large makes training diverge. A rate too small makes training slow. Adam removes much of this sensitivity.

3.4.3. Adam

Adam::new(learning_rate, beta1, beta2, epsilon, weight_decay) // all f32

Adam keeps 2 per-parameter moving averages. The first moment, m, tracks the mean gradient. The second moment, v, tracks the mean squared gradient. Adam uses both to give every parameter its own adaptive step size. The full update, run per element, is:

t += 1                                  (advanced once per batch, in step())
m = beta1*m + (1 - beta1)*grad
v = beta2*v + (1 - beta2)*grad^2
m_hat = m / (1 - beta1^t)               <- bias correction
v_hat = v / (1 - beta2^t)               <- bias correction
param -= lr * m_hat / (sqrt(v_hat) + epsilon)

Bias correction matters because m and v start at zero. On the first few steps, this pulls both moments toward zero. The pull is strong for v, because beta2 = 0.999 keeps v close to zero at t = 1.

Dividing by (1 - beta^t) rescales the moments. At t = 1, the denominator 1 - beta1 cancels the (1 - beta1) factor already in m. This recovers the raw gradient. As t grows, beta^t approaches 0, and the correction fades to a no-op.

This is why the timestep advances inside step, once per batch, and not inside update. With full-batch fit, t equals the epoch count. With fit_with_batches, t advances once per mini-batch. The timestep saturates instead of overflowing. Extremely long training runs therefore stay well defined.

Adam adds epsilon outside the square root. RMSprop and AdaGrad (3.4.5 and 3.4.6) add their epsilon inside it. This looks like an inconsistency, but it is deliberate. Keras splits epsilon placement the same way. RustyML matches Keras optimizer by optimizer, instead of forcing one form on all 3.

As a result, epsilon is not on the same scale in every optimizer. Read the scale note under RMSprop before you carry an epsilon value from one optimizer to another.

Adam’s weight_decay implements classic coupled L2 regularization. Adam folds weight_decay * param into the gradient before the moment update. The penalty therefore flows through m and v, and the adaptive 1 / (sqrt(v_hat) + epsilon) denominator rescales it. This coupling is usually not what you want. See AdamW, next, for the alternative.

With weight_decay = 0.0, the coupling has no effect, and Adam and AdamW become byte-for-byte identical. RustyML keeps the coupling in Adam on purpose. This is a deliberate divergence from Keras 3.

Keras 3’s base optimizer applies decoupled decay to every optimizer. This makes Adam(weight_decay=x) and AdamW(weight_decay=x) numerically the same in Keras. RustyML instead follows PyTorch. In PyTorch, torch.optim.Adam(weight_decay=) is coupled, and AdamW’s is not. This keeps the 2 names genuinely distinct in RustyML.

use rustyml::prelude::*;
use rustyml::neural_network::sequential::Sequential;
use ndarray::Array;

fn main() {
    // Tiny inline regression: y = x0 + 2*x1
    let x = Array::from_shape_vec((4, 2), vec![0.0_f32, 1.0, 1.0, 0.0, 1.0, 1.0, 2.0, 1.0])
        .unwrap()
        .into_dyn();
    let y = Array::from_shape_vec((4, 1), vec![2.0_f32, 1.0, 3.0, 4.0])
        .unwrap()
        .into_dyn();

    let mut model = Sequential::new();
    model
        .add(Dense::new(2, 8, Activation::ReLU).unwrap())
        .add(Dense::new(8, 1, Activation::Linear).unwrap())
        // learning_rate, beta1, beta2, epsilon, weight_decay
        .compile(Adam::new(0.01, 0.9, 0.999, 1e-8, 0.0).unwrap(), MeanSquaredError::new());

    model.fit(&x, &y, 50).unwrap();

    let preds = model.predict(&x).unwrap();
    println!("prediction shape: {:?}", preds.shape());
}

Adam::new(0.001, 0.9, 0.999, 1e-8, 0.0) is the default choice. Use this configuration when you are not sure which optimizer to pick. It tolerates a wide range of learning rates. It converges quickly on messy loss surfaces. It needs almost no tuning to start training.

3.4.4. AdamW

AdamW::new(learning_rate, beta1, beta2, epsilon, weight_decay) // identical signature to Adam

AdamW runs the same moment math and bias correction as Adam. The only difference is where weight decay enters the update. This difference matters as soon as you turn decay on. AdamW is decoupled, the Loshchilov and Hutter formulation.

AdamW shrinks the weight directly, param *= (1 - lr * weight_decay), before an ordinary Adam step. The decay never touches m or v. The adaptive denominator never divides it.

This distinction is not cosmetic. In Adam’s coupled scheme, the weight_decay * param term rides through the second moment v. A parameter that has seen large gradients gets a large v and a large denominator. It therefore gets less effective decay than a quiet parameter. Regularization strength ends up tangled with each weight’s gradient history. This is the opposite of the uniform shrink that weight decay is supposed to give.

AdamW severs this link. Every weight decays by the same (1 - lr * weight_decay) factor, regardless of its gradients. For this reason, AdamW generalizes better. It is the standard choice when you care about regularizing a model.

The practical rule is this: use AdamW for any non-zero weight decay with an adaptive optimizer, not Adam’s weight_decay. Use plain Adam, with weight_decay = 0.0, when you apply no regularization at all. At weight_decay = 0.0, the 2 optimizers run the same algorithm. There is no reason to prefer Adam over AdamW, except habit.