3. Neural Networks

RustyML ships a small, Keras-shaped deep-learning framework, written in pure Rust. You stack layers into a Sequential model, compile it with an optimizer and a loss, and call fit/predict. Every tensor that flows through the framework is a Tensor, which is just ndarray::ArrayD<f32>. It is single-precision, has a dynamic rank, and runs with no GPU and no autograd tape. Each layer implements forward and backward by hand, and hands its parameters to the optimizer through a flat view. This makes the whole framework deterministic and easy to debug. If you have used Keras, you will recognize the shape of the API. The differences are strict f32 precision, explicit input dimensions, and Result-returning constructors, and this chapter explains them.

Read Chapter 1 before this chapter. Read Working with ndarray and Installation and Feature Flags too, since the neural_network feature gates this whole module. Read Error Handling as well, since layer and loss constructors return Result. A model has this end-to-end shape:

use rustyml::neural_network::{
    sequential::Sequential,
    layers::{Activation, Dense},
    optimizers::Adam,
    losses::MeanSquaredError,
};
use ndarray::Array;

fn main() {
    let x = Array::ones((8, 4)).into_dyn(); // 8 samples, 4 features
    let y = Array::ones((8, 1)).into_dyn(); // 8 samples, 1 target

    let mut model = Sequential::new();
    model
        .add(Dense::new(4, 16, Activation::ReLU).unwrap())
        .add(Dense::new(16, 1, Activation::Linear).unwrap())
        .compile(
            Adam::new(0.001, 0.9, 0.999, 1e-8, 0.0).unwrap(),
            MeanSquaredError::new(),
        );

    model.fit(&x, &y, 5).unwrap();
    let preds = model.predict(&x).unwrap();
    println!("prediction shape: {:?}", preds.shape());
}

The Sequential Model is the container that turns a pile of layers into a trainable network. It owns the training loop, the optimizer, and the loss, and exposes add, compile, fit, train_batch, evaluate, predict, summary, and weight save/load. It also holds the batch-shuffle seed (set_seed) and the learning-rate pair (learning_rate / set_learning_rate). Read this section first, even if you need only one specific layer.

Dense Layers and Activations covers the fully connected layer, the main layer of tabular models and the output stage of most networks. It also covers the Activation enum (ReLU, Sigmoid, Tanh, Softmax, Linear), which you fold into a layer or use as a standalone layer. Read this section second. Everything after it assumes you know how a Dense layer declares its input_dim and units.

Loss Functions is the objective half of compile. It covers mean squared error and mean absolute error for regression, and binary, categorical, and sparse-categorical cross-entropy for classification. Read this section closely. Its averaging conventions differ on purpose: some average per element, others average per prediction site, and switching between them quietly rescales your effective learning rate. CategoricalCrossEntropy and SparseCategoricalCrossEntropy take a from_logits flag that changes whether you need a Softmax on the output. BinaryCrossEntropy has no such flag, and always expects a probability in (0, 1).

Optimizers is the update half of compile. It covers SGD with momentum, Adam, AdamW, RMSprop, and AdaGrad. This section also covers clip-by-global-norm (global_clipnorm), coupled versus decoupled weight decay, and mid-training learning-rate scheduling. Sections 3.1 through 3.4 together give you a complete, trainable feed-forward network.

The remaining sections add specialized layers. All of them plug into the same Sequential. Convolutional Layers provides 1D/2D/3D convolution, plus depthwise and separable variants for spatial data. Pooling Layers provides parameter-free max and average downsampling, plus their global variants. Recurrent Layers covers SimpleRNN, LSTM, and GRU for sequences. Regularization and Normalization Layers covers dropout (including spatial dropout), Gaussian noise, and batch, layer, group, and instance normalization. These layers depend on mode: they behave differently in fit than in predict, and the model switches the mode for you.

Saving and Loading Weights closes the chapter. save_to_path and load_from_path persist weights only, in postcard binary format. They do not persist the architecture. Rebuild the identical layer stack in code, then load the weights into it. Read this section once you have a model worth keeping. See Model Persistence in Depth for the format details and version caveats.