3.1. The Sequential Model

Sequential is the container that turns a stack of layers into a trainable model. The Keras workflow transfers almost without change. You call new() for an empty model, add() layers in order, compile() it with an optimizer and a loss, then call fit() and predict().

RustyML differs from Keras in 3 ways. Training is full-batch by default. There is no callback machinery, so you write your own loops for early stopping and learning-rate schedules. A width mismatch between adjacent layers surfaces as a panic instead of a Result value.

Sequential lives in rustyml::neural_network::sequential. The layers, optimizers, and losses that plug into it each have their own page (3.2, 3.4, 3.3).

3.1.1. The lifecycle at a glance

The 5 calls below are the entire public surface you need to train a network. Each builder method returns &mut Self, so add and compile chain together. fit returns Result<History, Error>, with one loss value per epoch. predict returns Result<Tensor, Error>.

let mut model = Sequential::new();
model
    .add(/* a layer */)
    .add(/* another layer */)
    .compile(/* optimizer */, /* loss */);
model.summary();
let history = model.fit(&x, &y, epochs)?; // epochs: u32 (history.loss() returns one f32 per epoch)
let y_hat = model.predict(&x_new)?;       // Tensor = ArrayD<f32>

The prelude exports everything on this page: Sequential, History, Dense, Activation, every optimizer and loss, and the Tensor alias. One glob import covers every example below:

use rustyml::prelude::*; // Sequential, History, Dense, Activation, Adam, SGD, losses, Tensor

Every tensor in the framework has type Tensor, an alias for ndarray::ArrayD<f32>. It has a dynamic rank and always holds f32 values. Call .into_dyn() to convert a statically-ranked array, such as Array2 or Array3, into IxDyn before it enters a layer. RustyML has no f64 path. The whole stack uses single precision for cache and SIMD performance.

3.1.2. Building the model: new and add

Sequential::new() creates an empty model with no optimizer, no loss, and no shuffle seed. add takes any L: 'static + Layer by value, boxes it as Box<dyn Layer>, and appends it to the model:

pub fn add<L: 'static + Layer>(&mut self, layer: L) -> &mut Self;

add consumes the layer, so you construct and move it in one expression: model.add(Dense::new(2, 8, Activation::ReLU).unwrap()). Layer constructors can fail on their own. For example, a Dense layer with a zero dimension returns Error::InvalidParameter. That is why .unwrap() appears on the layer constructor, not on add.

add does no cross-layer validation. It never checks that a layer’s input width matches the previous layer’s output width. A Box<dyn Layer> exposes no such contract at insertion time. A mismatch surfaces only when data flows through the model. See 3.1.8. Keras’s functional API works the opposite way: it fails when you connect incompatible layers while you build the graph.

The Layer trait, briefly

You do not need to implement Layer to use Sequential. Reading its contract explains what fit and predict call. The core methods are:

pub trait Layer: std::any::Any + Send + Sync {
    fn forward(&mut self, input: &Tensor) -> Result<Tensor, Error>;   // training pass, caches state for backward
    fn predict(&self, input: &Tensor) -> Result<Tensor, Error>;       // eval pass, takes &self, writes no caches
    fn backward(&mut self, grad_output: &Tensor) -> Result<Tensor, Error>;
    fn param_count(&self) -> TrainingParameters;
    fn output_shape(&self) -> String;
    fn layer_type(&self) -> &str;
    // parameters(), layer_type(), output_shape(), set_training_if_mode_dependent() have defaults
}

Layer has 2 forward paths instead of 1. forward(&mut self, ...) runs during training, taking &mut self to stash the intermediate tensors each layer needs for its backward pass. predict(&self, ...) runs during inference, taking &self, writing no caches, and putting mode-dependent layers, such as dropout and batch normalization, into their inference behavior. The Layer trait requires Send + Sync. Because predict borrows &self, a layer can serve concurrent inference calls with no lock. This split means predict never disturbs training state, and it costs less than a training forward pass.

Backward propagation is pure math. It does not sanitize NaN or Inf values on purpose. A non-finite gradient propagates and surfaces at the next forward pass or as a NaN loss, instead of being masked silently.

3.1.3. compile: wiring the optimizer and loss

pub fn compile<O, LFunc>(&mut self, optimizer: O, loss: LFunc) -> &mut Self
where O: 'static + Optimizer, LFunc: 'static + Loss;

compile stores the optimizer and loss as trait objects. It returns &mut Self so it chains off the last add. That is all it does. It runs no shape inference and allocates no weights. Each layer allocates its own weights in its constructor, using Xavier/Glorot initialization.

Training is the only thing that needs compile. fit returns Error::NeuralNetwork(NnError::NotCompiled(_)) if the optimizer or the loss is missing. predict never reads the optimizer or the loss, so it works on an uncompiled model. evaluate sits between the two: it needs a loss to score with, but no optimizer to step with. This split matters for serving. After you load weights into a fresh architecture, you can call predict right away without a compiled optimizer.

Pick the optimizer and loss from 3.4 and 3.3.

Each loss family normalizes its value in a different way. MeanSquaredError, MeanAbsoluteError, and BinaryCrossEntropy average over every element. CategoricalCrossEntropy sums over the trailing class axis, then averages over the prediction sites. A prediction site is a sample for a [batch, classes] target. For a channels-last convolutional softmax head that predicts 1 class per pixel, a prediction site is a pixel. The divisor is then batch * height * width.

Switching loss families rescales the gradient magnitude, which changes the effective learning rate. Re-tune the step size after you change the loss.

3.1.4. summary: reading the architecture

summary() prints a Keras-style table to stdout. Use it to check the wiring and the parameter count before you spend epochs training the model. Here is the table for a Dense(2 -> 8, ReLU) layer followed by a Dense(8 -> 2, Softmax) layer:

Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense)                   │ (None, 8)              │            24 │
│ dense_1 (Dense)                 │ (None, 2)              │            18 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
 Total params: 42 (168 B)
 Trainable params: 42 (168 B)
 Non-trainable params: 0 (0 B)

RustyML generates layer names per type, in Keras style. The first Dense layer is named dense, the next is dense_1, and so on. None in the output shape is the batch dimension. It stays unknown until data arrives.

Parameter counts come from each layer’s param_count() method. A Dense(in -> out) layer reports in * out + out parameters, the weight matrix plus the bias vector. So 2 * 8 + 8 = 24, and 8 * 2 + 2 = 18. The byte figures assume 4 bytes per f32.

Layers split into 3 groups. Trainable parameters count toward “Trainable”, frozen parameters count toward “Non-trainable”, and parameter-free layers, such as activations and pooling, contribute 0. summary borrows &self and never changes the model, so call it at any time, before or after training.

3.1.5. fit and the training loop

pub fn fit(&mut self, x: &Tensor, y: &Tensor, epochs: u32) -> Result<History, Error>;

fit is full-batch. Each epoch runs exactly one gradient step over the entire x and y you provide. There is only one batch, so nothing is shuffled, and the model never reads its shuffle seed. epochs counts these full-dataset steps. Keras’s fit, by contrast, splits data into mini-batches and shuffles every epoch by default. Use fit_with_batches instead if one gradient step per epoch converges too slowly for your dataset. Also use it if a single forward pass does not fit your memory budget (see below).

Each epoch, fit runs one train_batch call. The subsection below uses this same public, single-step method to build custom loops. train_batch performs these steps, in order:

1. Forward through every layer in training mode.
2. Compute the scalar loss.
3. Compute the loss gradient with respect to the output.
4. Advance the optimizer’s global step once. This lets Adam advance its bias-correction timestep once per step, not once per layer.
5. Backpropagate through the layers in reverse, so each layer stashes its parameter gradients.
6. Apply clip-by-global-norm, if the optimizer requests it (see Optimizer::global_clipnorm).
7. Update every layer’s parameters.

step() runs before the per-layer update() calls. This order keeps step-dependent optimizers correct across multiple layers.

fit returns a History. Its entire API is loss(): one f32 per epoch, in epoch order. epochs = 0 gives an empty slice. Each entry is the mean per-sample loss measured during the epoch, before that epoch’s own update. Every batch contributes the loss from the forward pass that ran before that batch’s own weight update. So each entry describes the weights the model held while the epoch ran, never the weights the epoch ends with.

Treating the last entry as the trained model’s final loss is wrong in both directions. While training converges, the entry reads above the truth, because the epoch’s own updates already improved on the weights it measured. Once the step size overshoots, the entry reads below the truth, because those same updates made things worse. This is Keras’s convention, not an accident of the loop. evaluate (see below) reports the loss of the model you hold right now.

The show_progress feature only adds a display. It is not the only way to read the loss. When it is on, fit renders a live progress bar. The bar shows the current epoch’s loss with 6 decimal places. fit_with_batches renders a similar bar that tracks the running average loss as its batches complete:

[dependencies]
rustyml = { version = "0.14", features = ["neural_network", "show_progress"] }

[00:00:00] ████████████████████████████████████████ 400/400 | Loss: <current loss>

Without the feature, training runs silently. The returned History holds the same numbers either way, so your code never depends on whether the feature is on.

Mini-batch training with fit_with_batches

pub fn fit_with_batches(&mut self, x: &Tensor, y: &Tensor, epochs: u32, batch_size: usize)
    -> Result<History, Error>;

fit_with_batches is the mini-batch loop. It reshuffles the sample order at the start of every epoch, then trains on fixed-size chunks. The shuffle makes this the one place where the model’s seed matters. Set it with Sequential::new_with_seed(seed) or set_seed(seed) for a reproducible shuffle order (see 7.1). This seed governs only the shuffle. It does not affect weight initialization, which each layer seeds through its own with_random_state call.

A batch_size of 0, or one larger than the dataset, returns Error::InvalidParameter. A batch_size equal to n_samples degenerates to a single full-batch step per epoch, matching fit.

Its History entries mean what fit’s entries mean, except for one refinement that shows only when batch_size does not divide the dataset evenly. Each batch contributes to the epoch loss in proportion to its sample count, not as 1 vote per batch. So a short trailing batch pulls the epoch figure less than a full batch does. This makes each entry exactly the dataset-wide mean per-sample loss, which matches what Keras reports. Keras’s loss metric accumulates every batch with sample_weight = batch_size, and a plain mean over batches would not match that. A test pins both the weighting and the before-the-update timing against numbers taken from Keras 3.15: tests/neural_network/sequential.rs::test_batch_losses_and_epoch_mean_match_keras.

use ndarray::Array;
use rustyml::prelude::*;

fn main() {
    let x = Array::ones((8, 3)).into_dyn();
    let y = Array::zeros((8, 1)).into_dyn();

    // Seed the per-epoch shuffle so the run is reproducible.
    let mut model = Sequential::new_with_seed(0);
    model
        .add(Dense::new(3, 6, Activation::ReLU).unwrap())
        .add(Dense::new(6, 1, Activation::Sigmoid).unwrap())
        .compile(
            SGD::new(0.05, 0.9, false, 0.0).unwrap(),
            BinaryCrossEntropy::new(),
        );

    // 4 mini-batches of 2 samples per epoch, reshuffled every epoch.
    let history = model.fit_with_batches(&x, &y, 5, 2).unwrap();
    assert_eq!(history.loss().len(), 5);

    // External LR schedule: read the step size, halve it, write it back. The optimizer keeps
    // its momentum buffers across the change.
    let lr = model.learning_rate().unwrap();
    model.set_learning_rate(lr * 0.5);
    let resumed = model.fit_with_batches(&x, &y, 5, 2).unwrap();

    // Training resumes where it left off rather than restarting.
    assert!(resumed.loss()[0] < *history.loss().last().unwrap());
    assert_eq!(model.predict(&x).unwrap().shape(), &[8, 1]);
}

set_learning_rate, used above, is the hook for external schedules such as step decay or warmup. It retunes the step size in place. The optimizer keeps all of its accumulated state, such as momentum buffers and Adam moments, across the change. It does nothing if the model has not been compiled.

learning_rate() is its read half. It returns None on an uncompiled model. This read-scale-write pattern avoids keeping a second copy of the rate beside the model, which could drift out of sync with the optimizer. Unlike the optimizer constructors, set_learning_rate validates nothing. learning_rate() returns exactly the value you set, even a zero or a negative rate.

Writing the loop yourself: train_batch and evaluate

pub fn train_batch(&mut self, x: &Tensor, y: &Tensor) -> Result<f32, Error>;
pub fn evaluate(&self, x: &Tensor, y: &Tensor) -> Result<f32, Error>;

train_batch is the single step that both fit variants build on. It is public. Any other epoch structure is a loop you write, instead of a fork of the library. Examples include curriculum ordering, a per-step schedule, or a probe between steps.

The whole of x is the batch. Nothing is split, nothing is shuffled, and mode-dependent layers run in training mode. The returned f32 is the loss from the forward pass, measured before this call’s own update. This is the number fit records for each epoch, and the value that fit_with_batches averages, weighted by sample count, into each History entry. Keras calls this method train_on_batch.

train_batch validates its own inputs instead of trusting the caller to have done it. So calling it on an uncompiled model returns a NotCompiled error, not a panic.

evaluate is the other half. It runs one inference-mode forward pass over the whole of x and scores it with the compiled loss. It updates nothing: no gradients, no parameters, and no batch-norm running statistics. It borrows &self, so scoring a model between training steps cannot disturb it. It also draws from no random number generator, so calling it inside a fit_with_batches loop cannot perturb the shuffle stream.

Layers behave exactly as they do in predict. Dropout and noise layers act as the identity, and batch normalization reads its running statistics. So on a model with such layers, evaluate and the number fit recorded for the same data disagree. Training-mode dropout inflates the number fit records. evaluate gives the more accurate estimate of the two.

Together, train_batch and evaluate turn early stopping, learning-rate schedules, and checkpoint selection into ordinary code you write. The loop below takes one full-batch step at a time. It scores the model it holds after each step. It halves the step size after 10 steps without an improvement, and stops once the schedule runs out:

use ndarray::Array;
use rustyml::prelude::*;

fn main() {
    // 8 points of y = 2x + 1.
    let x = Array::from_shape_vec((8, 1), (0..8).map(|i| i as f32 / 8.0).collect::<Vec<_>>())
        .unwrap()
        .into_dyn();
    let y = x.mapv(|v| 2.0 * v + 1.0);

    let mut model = Sequential::new();
    model
        .add(Dense::new(1, 8, Activation::Tanh).unwrap().with_random_state(0))
        .add(Dense::new(8, 1, Activation::Linear).unwrap().with_random_state(0))
        .compile(SGD::new(0.1, 0.9, false, 0.0).unwrap(), MeanSquaredError::new());

    let start = model.evaluate(&x, &y).unwrap();
    let (mut best, mut stale, mut previous) = (start, 0, start);

    for _ in 0..500 {
        // The step reports the loss it started from. This stack has no dropout, so that
        // number is the previous `evaluate` measured again, one step stale.
        let during = model.train_batch(&x, &y).unwrap();
        assert!((during - previous).abs() < 1e-5);

        // This one scores the weights the step produced.
        let after = model.evaluate(&x, &y).unwrap();
        previous = after;

        if after < best - 1e-6 {
            best = after;
            stale = 0;
            continue;
        }

        // 10 steps without progress: halve the step size, read from the optimizer instead of
        // a copy kept here. Once it is that small, there is nothing left to try.
        stale += 1;
        if stale == 10 {
            let lr = model.learning_rate().unwrap();
            if lr < 1e-4 {
                break;
            }
            model.set_learning_rate(lr * 0.5);
            stale = 0;
        }
    }

    assert!(best < start / 100.0);
}

3.1.6. predict: forward-only inference

pub fn predict(&self, x: &Tensor) -> Result<Tensor, Error>;

predict runs the inference forward path (Layer::predict) through every layer and returns the output Tensor. It borrows &self, allocates no backward caches, and puts mode-dependent layers into inference behavior. Dropout is disabled, and batch normalization uses its running statistics, which is the correct behavior for serving. predict is deterministic: 2 calls on the same input return identical tensors. Unlike training and evaluation, predict does not need compile.

The result shape is whatever the last layer emits, (batch, output_dim) for a Dense tail. Input must match each layer’s expectations. A Dense layer requires a 2D (batch, features) tensor and returns Error::InvalidInput for anything else.

3.1.7. A complete example: learning XOR

XOR is the smallest problem that a linear model cannot solve. It is the classic proof that a hidden layer does real work. This example uses 2 Dense layers: a Tanh hidden layer and a Softmax head over 2 classes. Training with Adam and categorical cross-entropy separates the classes cleanly. The targets are one-hot: class 0 is [1, 0], and class 1 is [0, 1]. Seeding both layers’ weight initialization with with_random_state(0) makes the run reproducible.

use ndarray::Array;
use rustyml::prelude::*;

fn main() {
    // XOR inputs, shape (4, 2).
    let x = Array::from_shape_vec((4, 2), vec![0.0_f32, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 1.0])
        .unwrap()
        .into_dyn();

    // One-hot targets: XOR is class 1 for (0,1) and (1,0), class 0 otherwise.
    let y = Array::from_shape_vec((4, 2), vec![1.0_f32, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0])
        .unwrap()
        .into_dyn();

    let mut model = Sequential::new();
    model
        .add(Dense::new(2, 8, Activation::Tanh).unwrap().with_random_state(0))
        .add(Dense::new(8, 2, Activation::Softmax).unwrap().with_random_state(0))
        .compile(
            Adam::new(0.1, 0.9, 0.999, 1e-8, 0.0).unwrap(),
            CategoricalCrossEntropy::new(false),
        );

    model.summary();
    let history = model.fit(&x, &y, 400).unwrap();
    assert!(*history.loss().last().unwrap() < history.loss()[0] / 100.0);

    let preds = model.predict(&x).unwrap();
    for i in 0..4 {
        // argmax over the 2 class probabilities
        let class = if preds[[i, 0]] >= preds[[i, 1]] { 0 } else { 1 };
        println!("row {i} -> class {class}  probs [{:.3}, {:.3}]", preds[[i, 0]], preds[[i, 1]]);
    }
}

After 400 full-batch epochs, the network assigns the 2 true XOR rows to class 1 and the 2 false rows to class 0. Each probability sits at or near 1.000. The History records that descent. Its last epoch sits more than 2 orders of magnitude below its first, which is what the assertion checks. The CategoricalCrossEntropy::new(false) argument tells the loss that the head already produces probabilities, from the Softmax layer, so the loss does not apply its own log-softmax. Pass true only when your last layer emits raw logits.

3.1.8. Errors, shape checking, and the mismatch panic

fit, fit_with_batches, and train_batch validate the model and the inputs before touching any layer. The checks run in this order:

1. An optimizer is present.
2. A loss is present.
3. The model has at least 1 layer.
4. The inputs have a batch axis.
5. The inputs are not empty.
6. x and y agree on batch size.

evaluate runs the same checks, except for the optimizer check, since only a parameter update needs an optimizer. The table below maps each failure to its error variant. All of them are rustyml::error::Error values:

The rank-0 row needs an explanation. A 0-dimensional tensor is not an empty one. It holds exactly 1 element, so is_empty does not reject it, and the batch-axis index that follows has nothing to read. This used to panic. Now it returns InvalidInput.

These are recoverable Result values. The next example shows the compile requirement. It shows how training and evaluation each demand something different from compile, and how predict needs neither:

use ndarray::Array;
use rustyml::error::Error;
use rustyml::neural_network::NnError;
use rustyml::prelude::*;

fn main() {
    let x = Array::ones((4, 2)).into_dyn();
    let y = Array::ones((4, 1)).into_dyn();

    let mut model = Sequential::new();
    model.add(Dense::new(2, 1, Activation::Linear).unwrap());

    // fit needs an optimizer and a loss. Without compile, it fails fast and names the first
    // thing it found missing.
    match model.fit(&x, &y, 1) {
        Err(Error::NeuralNetwork(NnError::NotCompiled(missing))) => assert_eq!(missing, "optimizer"),
        other => panic!("expected NotCompiled, got {other:?}"),
    }

    // evaluate updates nothing, so it asks only for the loss it scores with.
    match model.evaluate(&x, &y) {
        Err(Error::NeuralNetwork(NnError::NotCompiled(missing))) => {
            assert_eq!(missing, "loss function")
        }
        other => panic!("expected NotCompiled, got {other:?}"),
    }

    // predict, by contrast, never needs compile: it is forward-only.
    assert_eq!(model.predict(&x).unwrap().shape(), &[4, 1]);
}

One case is a genuine gap in the Result-based error handling. A dimension mismatch between adjacent layers is not a Result. It is a panic. add never checks that consecutive layers agree. So a Dense(2 -> 4) layer feeding a Dense(8 -> 2) layer builds without complaint. The inconsistency surfaces only when data reaches the second layer’s matrix multiply, on the first call to fit or predict:

let mut model = Sequential::new();
model
    .add(Dense::new(2, 4, Activation::ReLU).unwrap())    // emits 4 columns
    .add(Dense::new(8, 2, Activation::Softmax).unwrap()); // expects 8, a mismatch

let x = Array::ones((3, 2)).into_dyn();
let _ = model.predict(&x); // panics inside the GEMM, does not return Err

thread 'main' panicked at gemmkit-ndarray-0.1.2/src/fused.rs:85:5:
assertion `left == right` failed: gemmkit-ndarray: A.cols (4) != B.rows (8)
  left: 4
 right: 8

This panic comes from the matrix-product backend, not from RustyML. Dense::forward hands the product straight to gemmkit-ndarray, so the assertion that fires belongs to the backend. The path in the message is a crates.io registry path, not a path in this repository. This is expected behavior, not a sign of an internal bug.

Treat inter-layer widths as a build-time invariant that you must enforce. Each layer’s input_dim must equal the previous layer’s units. A mismatch is a programming error, not bad input, so it aborts instead of returning an ordinary Error. Get the widths right in the constructors, which is what summary checks for you.

Then this panic never fires. Get them wrong, and the first data that flows through the model reveals it. The panic message names the 2 extents that disagree: here, the 4 columns the first layer emits against the 8 rows the second layer expects.

Each building block has its own page from here. See Dense layers and activations, loss functions, optimizers, the convolutional and recurrent layers, and saving and loading weights.