3.5. Convolutional Layers

RustyML ships 10 convolutional layers. Conv1D, Conv2D, and Conv3D are the plain layers. Conv1DTranspose, Conv2DTranspose, and Conv3DTranspose are their 3 transposed counterparts. DepthwiseConv1D, DepthwiseConv2D, SeparableConv1D, and SeparableConv2D are the depthwise and separable pairs. All 10 live in rustyml::neural_network::layers, and the layers glob re-exports them.

They share 1 design. A layer struct holds the weights, the bias, and the activation. It holds no cache and no gradient, because a forward pass takes &self and parks both in the Ctx of the pass. The layer delegates the actual numerics for the plain convolutions to 1 dimension-generic engine.

Tensors are channels-last here, and a kernel carries its taps first. The constructor takes the configuration of the layer and nothing else. It never takes an input shape, because a kernel extent comes from the input and the input is not there yet. UnaryLayer::build reads the shape, sizes every array, and draws the weights. The model build calls it once per layer.

This page covers the layouts, the constructor, padding, and dilation rules, and the im2col plus GEMM engine behind the standard convolutions. It also covers the depthwise and separable factorization with its cost math, and the transposed convolutions that run a convolution backwards to grow a tensor. It then covers the border layers that resize the spatial axes by hand. The last 3 sections cover the layers that reorder axes, repeat a vector, or reverse an axis. They also cover the layer that does nothing at all, and the error types.

3.5.1. Tensor Layouts and the Layer Family

Every convolution here is channels-last. The spatial axes follow the batch axis, and the channel axis comes last. A Conv2D reads [batch, height, width, channels] and writes [batch, out_height, out_width, filters]. This is the NHWC convention.

A tensor already in that order needs no permutation. Build inputs by hand with ndarray, and put the channels last. The weight tensor follows the same rule: the kernel taps come first, then the input channels, then the filters.

That is not only an interface choice. With the channel axis innermost, a kernel tap at a given output position is Cin contiguous floats. This lets the engine build its im2col matrix from run copies, instead of a scalar gather. It also lets the flat weight matrix line up with the im2col matrix without any permutation.

Every shape in the table follows that 1 convention, so a kernel exported from another channels-last framework drops in without a permutation. Note that the 3 transposed kernels carry their filter axis before their input-channel axis, which is the reverse of the plain ones. Section 3.5.5 explains why.

The 4 depthwise and separable layers need more explanation. A depthwise layer has no filters argument at all. It applies depth_multiplier kernels to each input channel and emits C * dm channels. Input channel c’s multiplier m lands at output channel c * dm + m.

A separable layer holds 2 weight tensors because it fuses 2 convolutions into 1 layer. The depthwise stage emits its channels in that same c * dm + m order. This means the pointwise weight’s rows already match the depthwise output, so the layer needs no repacking between the stages. Section 3.5.4 covers this in more detail.

The forward math is cross-correlation, which is the usual convention. The engine does not flip the kernel. The engine adds the bias last, once per filter, after the multiply-accumulate step. Weights start from Xavier/Glorot uniform bounds. Biases start at zero.

Every pass takes a Ctx. That context carries the training flag, and it holds every cache and every gradient of the pass, so no layer holds one. Ctx::training() picks the mode that parks a cache, and Ctx::inference() picks the mode that parks nothing. The following example sets the weights by hand and runs 1 inference pass. It shows the layout and the Valid-padding output size.

use ndarray::{Array, Array1, Array4};
use rustyml::neural_network::layers::*;
use rustyml::neural_network::traits::UnaryLayer;
use rustyml::neural_network::{Ctx, Shape};

fn main() {
    // Channels-last: [batch, height, width, channels]. The constructor allocates nothing.
    let mut layer = Conv2D::new(1, (2, 2), (1, 1), Activation::Linear).unwrap();

    // `build` reads the channel count off the last axis and sizes the kernel.
    layer.build(&Shape::with_free_batch(&[1, 4, 4, 1])).unwrap();

    // Weights are [kernel_h, kernel_w, channels, filters]. Bias is [filters].
    let weights = Array4::from_elem((2, 2, 1, 1), 1.0f32);
    let bias = Array1::zeros(1);
    layer.set_weights(weights, bias).unwrap();

    let pixels: Vec<f32> = (1..=16).map(|v| v as f32).collect();
    let x = Array::from_shape_vec((1, 4, 4, 1), pixels).unwrap().into_dyn();

    // The forward pass takes `&self` and the context of the pass. An inference context
    // parks nothing at all.
    let mut ctx = Ctx::inference();
    let out = layer.forward(&x, &mut ctx).unwrap();
    // Valid padding: (4 - 2)/1 + 1 = 3 along each spatial axis -> [1, 3, 3, 1].
    assert_eq!(out.shape(), &[1, 3, 3, 1]);
    // An all-ones 2x2 kernel sums each window: the first is 1 + 2 + 5 + 6 = 14.
    assert_eq!(out[[0, 0, 0, 0]], 14.0);
    println!("output shape: {:?}", out.shape());
}

3.5.2. Constructors, Padding, Dilation, and Output Shapes

The constructors take the hyperparameters as positional arguments. The kernel and stride argument shape tracks the rank. Every 1D layer uses a scalar kernel_size and stride. The 2D and 3D layers take tuples instead.

A separable layer inserts a depth_multiplier argument before the activation. A depthwise layer has no filters argument. It derives its output width from the input, and it takes its depth multiplier through a builder method.

Conv1D::new(filters, kernel_size: usize, stride: usize, activation) -> Result<Conv1D, Error>
Conv2D::new(filters, kernel_size: (usize, usize), strides: (usize, usize), activation) -> Result<Conv2D, Error>
Conv3D::new(filters, kernel_size: (usize, usize, usize), strides: (usize, usize, usize), activation) -> Result<Conv3D, Error>
DepthwiseConv1D::new(kernel_size: usize, stride: usize, activation) -> Result<DepthwiseConv1D, Error>
DepthwiseConv1D::with_depth_multiplier(self, depth_multiplier: usize) -> Result<DepthwiseConv1D, Error>   // builder, defaults to 1
DepthwiseConv2D::new(kernel_size: (usize, usize), strides: (usize, usize), activation) -> Result<DepthwiseConv2D, Error>
DepthwiseConv2D::with_depth_multiplier(self, depth_multiplier: usize) -> Result<DepthwiseConv2D, Error>   // builder, defaults to 1
SeparableConv1D::new(filters, kernel_size: usize, stride: usize, depth_multiplier: usize, activation) -> Result<SeparableConv1D, Error>
SeparableConv2D::new(filters, kernel_size: (usize, usize), strides: (usize, usize), depth_multiplier: usize, activation) -> Result<SeparableConv2D, Error>

activation takes impl Into<Activation>. You can pass an Activation variant (Activation::ReLU, Activation::Sigmoid, Activation::Tanh, Activation::Softmax { axis: -1 }, Activation::Linear), or a standalone activation layer such as ReLU::new(). An embedded softmax accepts the default axis -1 alone, so the build refuses any other axis. See 3.2. Dense Layers and Activations for more about the activation set.

No constructor takes an input shape. The model build supplies it, and the layer reads the channel count and the spatial extents from it. The batch axis stays free, so 1 built layer serves every batch size.

4 builder methods refine a constructed layer. Each one consumes and returns self, so you can chain them. with_padding(PaddingType) switches the padding mode. with_dilation_rate(...) spaces the kernel taps, and it returns a Result.

with_random_state(u64) records the seed of the Xavier draw, and the build spends it. On an unbuilt layer it draws nothing, because there is no array yet. Call it before you assign custom weights or start training (see 7.1. Reproducibility and Random Seeds).

with_use_bias(bool) decides whether the layer holds a bias at all. All 10 convolution layers carry it, and every one of them defaults to true.

with_use_bias(false) removes the bias from the layer, and not only from the sum. The layer then reports a parameter count of the kernel alone. It exposes 1 array in place of 2, and a checkpoint of it holds 1 path fewer. Where a normalization layer follows the convolution, use it, because the shift of that layer makes the bias redundant.

set_weights(...) installs weights and a bias that you control. It needs a built layer, and it returns Error::NeuralNetwork(NnError::NotBuilt) on a layer that holds no array yet. It then checks every array against the shape the build settled.

Note that a depthwise layer’s set_weights takes a kernel and an Array1 bias. A separable layer’s takes 3 arrays instead: depthwise weights, pointwise weights, then bias. The bias argument is optional on every one of them. Pass None for a layer built with with_use_bias(false), and pass the array for every other layer. The other order returns Error::InvalidParameter.

Padding uses the PaddingType enum, which has 2 variants: Valid (the default) and Same. Conv1D accepts a third mode, which a later paragraph covers. The 2 shared variants set the output size as follows:

• Valid applies no padding. It computes output values only where the kernel fully overlaps the input. The formula is out = (in - keff) / stride + 1 (integer floor division), on each spatial axis. keff is the effective kernel of the dilation rule below, and it equals the kernel size at the default dilation of 1.
• Same zero-pads the borders so out = ceil(in / stride). At stride == 1 this keeps the spatial size exactly. At a larger stride, the output is the input length divided by the stride, rounded up. The layer splits the total padding evenly and puts the extra cell on the trailing edge (pad_before = pad_total / 2). The pooling engine splits an odd total the same way.

Valid is the 1 mode that needs the kernel to fit the input axis. Same supplies the missing cells, so an effective kernel longer than the input axis stays legal there. The rule therefore runs at the build and not in a constructor. with_padding and with_dilation_rate both run after construction, and both decide whether the rule applies. The build is the first step that sees the padding mode and the input shape together.

A Conv2D with a 3x3 kernel over a 2x2 input constructs under every mode. Its build then returns Error::InvalidInput under Valid, and it succeeds under Same. The refusal names the layer, the axis, and the effective kernel extent that the axis must reach. Section 3.5.10 shows both halves.

Conv1D takes a third mode, ConvPadding::Causal, which no other convolution accepts. It puts all keff - 1 pad cells on the leading edge and none on the trailing edge. An output position therefore never reads a later input position. Its output length is the Same one, ceil(in / stride). Use it for a sequence model that must not look ahead.

Conv1D::with_padding takes impl Into<ConvPadding>, so PaddingType::Valid and PaddingType::Same still pass through unchanged. The enum is in the prelude, and it also lives at rustyml::neural_network::layers::ConvPadding.

with_dilation_rate spaces the kernel taps. A dilation of d on an axis puts d - 1 empty cells between neighboring taps, so k taps span an effective kernel of keff = (k - 1) * d + 1 input cells. The window still advances by the stride, and the offset of tap t at output position o is o * stride + t * d, never (o * stride + t) * d. A dilated layer therefore widens its receptive field and adds no parameter, which is what a stack of dilated convolutions over a long sequence needs. The default is 1 on every axis, and it gives a solid kernel.

with_dilation_rate returns a Result. A dilation of 0 gives Error::InvalidParameter on every layer. On Conv1D, Conv2D, Conv3D, and the 3 transposed layers, a dilation above 1 together with a stride above 1 gives the same error. There, the 2 factors have no agreed meaning together. DepthwiseConv1D, DepthwiseConv2D, SeparableConv1D, and SeparableConv2D accept that pair instead, and read the stride and the dilation as 2 independent settings.

The 1D layers take a scalar, and the 2D and 3D layers take a tuple, exactly as their kernel argument does. All 10 convolution layers carry this method.

The following numbers show the rule in practice. A [N, 8, 8, 1] input through a 3x3 kernel gives 3 results. Valid padding at stride 1 gives (8-3)/1+1 = 6, so the output is [N, 6, 6, F]. Same padding at stride 1 gives 8, so the output is [N, 8, 8, F]. Same padding at stride 2 gives ceil(8/2) = 4. A Conv1D over length 6, with kernel 3 and stride 2, under Valid padding, gives (6-3)/2+1 = 2.

The next program runs both settings on a Conv1D over the values 1 to 8. An all-ones kernel sums each window, so every output value names the input positions the window reached.

use ndarray::{Array, Array1, Array3};
use rustyml::neural_network::layers::*;
use rustyml::neural_network::traits::UnaryLayer;
use rustyml::neural_network::{Ctx, Shape};

fn main() {
    // 1 sample, 8 steps, 1 channel, holding the values 1 to 8.
    let steps: Vec<f32> = (1..=8).map(|v| v as f32).collect();
    let x = Array::from_shape_vec((1, 8, 1), steps).unwrap().into_dyn();

    let make = |dilation: usize, padding: ConvPadding| {
        let mut layer = Conv1D::new(1, 3, 1, Activation::Linear)
            .unwrap()
            .with_dilation_rate(dilation)
            .unwrap()
            .with_padding(padding);
        // Both builder calls come before the build, so the geometry is settled when the
        // layer sizes its kernel.
        layer.build(&Shape::with_free_batch(&[1, 8, 1])).unwrap();
        layer
            .set_weights(Array3::from_elem((3, 1, 1), 1.0f32), Array1::zeros(1))
            .unwrap();
        layer
    };

    let mut ctx = Ctx::inference();

    // Dilation 2 spans (3 - 1) * 2 + 1 = 5 cells, so Valid keeps 8 - 5 + 1 = 4 positions.
    let out = make(2, ConvPadding::Valid).forward(&x, &mut ctx).unwrap();
    assert_eq!(out.shape(), &[1, 4, 1]);
    // Position 0 reads steps 1, 3 and 5: 1 + 3 + 5 = 9.
    assert_eq!(out[[0, 0, 0]], 9.0);

    // Causal padding keeps the length and reads no later step.
    let causal = make(1, ConvPadding::Causal).forward(&x, &mut ctx).unwrap();
    assert_eq!(causal.shape(), &[1, 8, 1]);
    // Position 0 sees 2 pad cells and step 1, so it holds 1.
    assert_eq!(causal[[0, 0, 0]], 1.0);
    // Position 7 sees steps 6, 7 and 8: 6 + 7 + 8 = 21.
    assert_eq!(causal[[0, 7, 0]], 21.0);

    println!("valid {:?}, causal {:?}", out.shape(), causal.shape());
}

Conv3D applies the same rule on depth, height, and width, each on its own. The 3 transposed layers take the same arguments and the same PaddingType. Both of their output rules grow the axis instead of shrinking it, and section 3.5.5 gives them. model.summary() prints these output shapes and the per-layer parameter counts. Use it to check a stack before you train it. See 3.1. Sequential Model for the full description of summary.

3.5.3. The im2col + GEMM Engine

Conv1D, Conv2D, and Conv3D do not each carry their own loop nest. A plain convolution is the same operation at every rank. Only the number of spatial axes changes. All 3 layers delegate their forward and backward numerics to 1 implementation in convolution_engine.rs.

This implementation is generic over the spatial rank R = ndim - 2. The layer wrapper keeps the public API, the weight storage, and the activation. The engine does the arithmetic, and the context of the pass holds what the backward pass needs.

The engine uses im2col plus GEMM, the same strategy the major frameworks use. For the forward pass, it gathers each output window into a row. This forms an [out_plane, k_plane*Cin] matrix, whose columns align with the flat weight matrix [k_plane*Cin, F]. 1 matrix multiply then produces [out_plane, F], and the engine adds the bias per filter.

Under the channels-last layout, this gather is a run of Cin-wide copy_from_slice calls, not a scalar-at-a-time walk. The product lands directly on a contiguous slab of the output, with no scatter. Trading a 6-deep loop nest for a matrix multiply lets the layer use the crate’s tuned in-house GEMM, instead of a naive 3-level loop. See 6.2. Matrix Multiplication for more on that GEMM.

The backward pass runs 2 GEMMs per batch item. One computes the weight gradient. The other computes the input-gradient columns, which the engine then scatters back (col2im) into the input-gradient tensor.

The engine gates parallelism on estimated FLOPs, not on element counts. A 7x7x512 convolution and a 3x3x3 convolution can share the same output-element count, but their costs differ by a wide margin. The forward gate compares 2 * batch * F * out_plane * Cin*k against CONV_PARALLEL_MIN_FLOPS (default 4,000,000, tunable at runtime through rustyml::tuning).

Below this threshold, the forward pass runs serial. Above it, the forward pass parallelizes over (batch item, output-position block) tasks. This lets 1 large image fill every core, even at batch == 1. Each task builds its own im2col block and runs its own GEMM into a disjoint output region.

The backward pass parallelizes over batch items. It reduces the weight and bias partials in batch order, which keeps results bit-reproducible across runs on the same machine. It also routes each item’s GEMMs through a switch. The switch keeps them parallel while the batch is too short to fill the thread pool. It flips them to serial once the batch alone saturates the pool. This way, a batch task never forks rayon again inside its own GEMM.

The parallel path and the serial path return the same numbers, so you rarely need to touch the gate. If you profile a workload that sits just under the threshold, see 7.3. Performance Tuning and Parallelism for how to move it.

3.5.4. Depthwise and Separable Convolutions

A standard convolution mixes across channels and across space in 1 step. Every output channel is a weighted sum over all input channels and all kernel taps. That coupling is where the parameters live. The weight tensor holds F * Cin * kh * kw values.

Depthwise separable convolution factors this operation into 2 cheaper stages. The first stage is a depthwise convolution that filters each input channel on its own (spatial mixing only, no cross-channel mixing). The second stage is a pointwise 1x1 convolution that recombines the channels (cross-channel mixing only, no spatial extent). This is the idea behind MobileNet and Xception. RustyML exposes both stages.

DepthwiseConv2D is the first stage alone. It carries depth_multiplier kernels of size kh x kw for each input channel, and emits C * depth_multiplier output channels. This is why it has no filters argument. It does no channel recombination, so it cannot mix channels on its own. In practice, you almost always pair it with a 1x1 convolution downstream.

depth_multiplier defaults to 1. Set it with with_depth_multiplier, which returns a Result because it rejects 0. A separable layer takes its own depth_multiplier as a positional argument. It expands the intermediate channel count to Cin * depth_multiplier, before the pointwise stage collapses it back to filters.

The parameter counts are the point of this factorization. Consider Cin input channels, F output filters, and a kh x kw kernel:

• Standard Conv2D: F * Cin * kh * kw + F.
• DepthwiseConv2D: C * dm * kh * kw + C * dm (at the default dm = 1, this is C * kh * kw + C).
• SeparableConv2D: dm * Cin * kh * kw (depthwise) + F * Cin * dm (pointwise) + F (bias).

Ignoring the bias term, the separable-to-standard ratio is 1/F + 1/(kh*kw). The savings grow with both the filter count and the kernel area. Each parameter costs 1 multiply-accumulate per output position, so this same ratio is also the compute (FLOP) ratio.

As an example, take a 64-filter 3x3 convolution over 3 channels. The standard layer has 64*3*3*3 + 64 = 1792 parameters. The separable equivalent has 27 + 192 + 64 = 283 parameters, about 6.3 times fewer. A depthwise-only layer over those 3 channels has just 30 parameters.

The following program builds all 3 layers and checks the counts. param_count() returns a ParamCounts, and total() adds its trainable and its non-trainable field. A convolution layer holds no non-trainable element, so its total equals its trainable count.

use rustyml::neural_network::Shape;
use rustyml::neural_network::layers::*;
use rustyml::neural_network::traits::{LayerBase, UnaryLayer};

fn main() {
    let input = Shape::with_free_batch(&[1, 32, 32, 3]); // [batch, H, W, channels]

    // Standard 64-filter 3x3 convolution over 3 input channels.
    let mut standard = Conv2D::new(64, (3, 3), (1, 1), Activation::ReLU).unwrap();
    standard.build(&input).unwrap();

    // Separable equivalent: depthwise (depth_multiplier = 1) then a pointwise 1x1 to 64 filters.
    let mut separable = SeparableConv2D::new(64, (3, 3), (1, 1), 1, Activation::ReLU).unwrap();
    separable.build(&input).unwrap();

    // Depthwise-only emits C * depth_multiplier = 3 channels and does no cross-channel mixing.
    let mut depthwise = DepthwiseConv2D::new((3, 3), (1, 1), Activation::ReLU).unwrap();
    depthwise.build(&input).unwrap();

    println!("standard  Conv2D params: {}", standard.param_count().total());
    println!("separable Conv2D params: {}", separable.param_count().total());
    println!("depthwise Conv2D params: {}", depthwise.param_count().total());

    assert_eq!(standard.param_count().total(), 1792); // 64*3*3*3 + 64
    assert_eq!(separable.param_count().total(), 283); // 27 + 192 + 64
    assert_eq!(depthwise.param_count().total(), 30); // 3*3*3 + 3
}

The same math holds on a sequence, where wide channel counts are just as common. A 1D kernel drops the height axis, so a depthwise stage is [k, C, dm] and a pointwise stage is [1, Cin*dm, F]. Everything else carries over unchanged: the c * dm + m channel order, the padding rules, and the builder methods. The following program runs the same comparison over a 128-step sequence carrying 32 channels, where the separable form is about 4.5 times smaller.

use ndarray::Array3;
use rustyml::neural_network::layers::*;
use rustyml::neural_network::traits::{LayerBase, UnaryLayer};
use rustyml::neural_network::{Ctx, Shape};

fn main() {
    // [batch, length, channels]: 128 time steps carrying 32 channels.
    let input = Shape::with_free_batch(&[1, 128, 32]);

    // Standard 64-filter width-5 convolution over 32 input channels.
    let mut standard = Conv1D::new(64, 5, 1, Activation::ReLU).unwrap();
    standard.build(&input).unwrap();

    // Separable equivalent: depthwise width-5, then a pointwise 1-tap to 64 filters.
    let mut separable = SeparableConv1D::new(64, 5, 1, 1, Activation::ReLU).unwrap();
    separable.build(&input).unwrap();

    // Depthwise-only emits C * depth_multiplier = 32 channels and mixes nothing across them.
    let mut depthwise = DepthwiseConv1D::new(5, 1, Activation::ReLU).unwrap();
    depthwise.build(&input).unwrap();

    println!("standard  Conv1D params: {}", standard.param_count().total());
    println!("separable Conv1D params: {}", separable.param_count().total());
    println!("depthwise Conv1D params: {}", depthwise.param_count().total());

    assert_eq!(standard.param_count().total(), 10304); // 64*32*5 + 64
    assert_eq!(separable.param_count().total(), 2272); // 160 + 2048 + 64
    assert_eq!(depthwise.param_count().total(), 192); // 32*5 + 32

    // Valid padding: (128 - 5)/1 + 1 = 124 positions, each carrying 64 filters.
    let x = Array3::<f32>::zeros((1, 128, 32)).into_dyn();
    let mut ctx = Ctx::inference();
    assert_eq!(separable.forward(&x, &mut ctx).unwrap().shape(), &[1, 124, 64]);
}

When channel counts run high and you want to cut both parameters and compute, use these layers. Wide feature maps are one example. A model meant to run on modest hardware is another. In exchange, you accept a modeling tradeoff: the factored form is strictly less expressive than a full convolution, for the same F and kernel.

Their engine differs from the standard layers’ engine. A depthwise pass uses a direct loop nest, because the channels are independent and im2col buys little here. It parallelizes over (batch item, output row) tasks. Output rows are disjoint, so this split needs no merge. It gates on NAIVE_CONV_PARALLEL_MIN_FLOPS (default 1,000,000).

A separable layer runs its depthwise stage through this same naive path. It then routes its pointwise stage back through the shared im2col plus GEMM engine. A 1-tap convolution is exactly a per-position cross-channel matrix multiply.

All 4 layers share 1 depthwise loop nest, across both ranks. A [batch, length, channels] tensor holds its values in the same order as [batch, 1, length, channels], and a [k, C, dm] kernel in the same order as [1, k, C, dm]. The 1D layers therefore set the height terms of the shared geometry to 1 and pass the flat slices of their own rank-3 arrays. This repacks nothing, and the 1D and the 2D form cannot drift apart.

3.5.5. Transposed Convolutions

Conv1DTranspose, Conv2DTranspose, and Conv3DTranspose run a convolution backwards over the spatial axes. A convolution shrinks a tensor, so its transpose grows one. This is the layer a decoder or a generator uses to climb back to the resolution the matching Conv2D consumed. Unlike UpSampling2D, it learns how it grows, because it carries a kernel, and a bias when use_bias is true.

The name is literal. A convolution is a linear map, so it has a transpose, and that transpose is what this layer computes. The forward pass here is exactly the gradient a plain convolution computes with respect to its input. The backward pass here is exactly a plain forward pass. RustyML implements it that way. The transposed engine reuses every geometry helper of the standard engine and only reverses the direction of its 2 matrix products.

So a transposed convolution costs about what the convolution it transposes costs. Read “deconvolution” as a synonym to avoid. The layer recovers the shape, never the values.

Each input position writes its own value times the whole kernel into the output, starting at position * stride. Wherever the stride is below the kernel size, neighboring windows overlap, and the overlapping writes accumulate. Wherever the stride is above the kernel size, some output positions receive nothing at all. A layer with a bias adds it once per output position, at the end, so those positions hold exactly the bias. A layer built with with_use_bias(false) adds nothing, and those positions stay at 0.

The output size follows 1 rule per padding mode, applied to each axis on its own:

keff is the effective kernel of section 3.5.2, so a dilation of 1 reads these rules with the plain kernel size. The constructors mirror the plain ones argument for argument, and the same 4 builder methods with_padding, with_dilation_rate, with_random_state, and with_use_bias refine them:

Conv1DTranspose::new(filters, kernel_size: usize, stride: usize, activation) -> Result<Conv1DTranspose, Error>
Conv2DTranspose::new(filters, kernel_size: (usize, usize), strides: (usize, usize), activation) -> Result<Conv2DTranspose, Error>
Conv3DTranspose::new(filters, kernel_size: (usize, usize, usize), strides: (usize, usize, usize), activation) -> Result<Conv3DTranspose, Error>

1 difference in the weight layout is easy to miss. The kernel is [k..., filters, channels], so the filter axis comes before the input-channel axis. That is the reverse of the plain convolution kernel. The reason is the direction of the pass: a transposed convolution reads channels and writes filters.

This reversal is why an initializer never reads the shape of a kernel to work out its 2 fans. A rule that reads the last 2 axes gives these 3 layers the pair the wrong way round. The layer reports its own fan_in and fan_out by name instead, so the layout cannot mislead the draw. Section 3.2.3 covers the Initializer and Fans values in full.

set_weights therefore refuses a kernel laid out for the matching Conv2D when the filter count and the channel count differ. When the 2 counts are equal, the 2 layouts have the same shape, and no shape check can separate them. param_count is unaffected, since the product is the same either way.

A second difference is that these layers put no lower bound on the input spatial size, under any padding mode. A plain convolution under Valid padding refuses an input smaller than its effective kernel, because such a convolution has no complete window. A transposed convolution has the opposite geometry, so a 1x1 input under a 3x3 kernel is legal. This is the normal first step of a decoder that starts from a Dense head.

The following example grows a 2x2 feature map into 4x4 and checks 2 of the 16 output values by hand.

use ndarray::{Array, Array1, Array4};
use rustyml::neural_network::layers::*;
use rustyml::neural_network::traits::UnaryLayer;
use rustyml::neural_network::{Ctx, Shape};

fn main() {
    // Channels-last: [batch, height, width, channels].
    let mut layer = Conv2DTranspose::new(1, (3, 3), (2, 2), Activation::Linear)
        .unwrap()
        .with_padding(PaddingType::Same);
    layer.build(&Shape::with_free_batch(&[1, 2, 2, 1])).unwrap();

    // Weights are [kernel_h, kernel_w, filters, channels]: the filter axis comes first.
    let taps: Vec<f32> = (1..=9).map(|v| v as f32).collect();
    let weights = Array4::from_shape_vec((3, 3, 1, 1), taps).unwrap();
    layer.set_weights(weights, Array1::zeros(1)).unwrap();

    let x = Array::from_shape_vec((1, 2, 2, 1), vec![1.0f32, 2.0, 3.0, 4.0])
        .unwrap()
        .into_dyn();
    let mut ctx = Ctx::inference();
    let out = layer.forward(&x, &mut ctx).unwrap();

    // Same padding: 2 * 2 = 4 along each spatial axis -> [1, 4, 4, 1].
    assert_eq!(out.shape(), &[1, 4, 4, 1]);
    // Input (0, 0) scales the whole kernel from output (0, 0), so out[0, 0] is 1 * w[0, 0].
    assert_eq!(out[[0, 0, 0, 0]], 1.0);
    // Output (2, 2) is the one position all 4 inputs reach: 1*9 + 2*7 + 3*3 + 4*1 = 36.
    assert_eq!(out[[0, 2, 2, 0]], 36.0);
    println!("output shape: {:?}", out.shape());
}

The overlap has a visible cost. When the stride does not divide the kernel size, some output positions collect more kernel taps than their neighbors do. A trained model turns that imbalance into a regular grid of bright and dark cells. This is the checkerboard artifact.

A kernel size that the stride divides evenly removes it. A 4x4 kernel at stride 2 is a safer default than a 3x3 kernel at stride 2. The alternative is UpSampling2D followed by a plain Conv2D, which cannot produce the artifact at all.

The round trip back to the original size is exact only when the convolution kept every input position. Under Valid that means the kernel is at least the stride, and the stride divides input - kernel. The convolution otherwise drops the trailing positions its last window could not reach, and the transposed convolution has nothing to rebuild them from. An 8-wide axis at kernel 3 and stride 2 convolves to 3 positions, and those 3 positions transpose back to 7, not 8.

Under Same the condition is that the stride divides the input, and a stride that does not divide it overshoots to the next multiple instead. RustyML has no output-padding argument to absorb that difference. Pick a stride that divides, or resize afterward with a border layer from section 3.5.7.

The 3 layers share the standard engine’s parallelism gate, tuning::conv::set_parallel_min_flops, because both engines run the same 2 matrix-product shapes. The transposed forward pass parallelizes over batch items only. Its scatter accumulates into overlapping output positions, so splitting 1 image across threads would need a merge that batch items never need. Below a full batch the per-item products run in parallel instead, so a batch of 1 still uses every core.

3.5.6. Building a Small CNN

Convolutions emit rank-3, rank-4, or rank-5 tensors, but a Dense classifier head needs a rank-2 [batch, features] matrix. Flatten bridges the 2. Flatten::new() takes no argument and returns the layer directly, with no Result, because it has nothing to misconfigure. It reshapes [batch, ...] into [batch, product-of-the-rest], and it reads that product from the shape the build hands it.

Reshape is the general form of the same idea. Reshape::new(target_shape: Vec<isize>) builds a parameter-free layer that rewrites every axis after the batch axis. The target_shape never names the batch axis. Axis 0 passes through untouched, so 1 instance serves every batch size. The 2 layers now differ only in what they compute, and neither one takes an input shape.

At most 1 entry of target_shape may be -1. That axis takes whatever extent makes the element count match. Reshape::new(vec![-1, 2]) on a [batch, 4] input gives [batch, 2, 2], since 4 / 2 = 2. Reshape::new(vec![-1]) collapses every axis after the batch axis into 1 axis, so it is exactly Flatten. An empty target_shape is legal, and gives the rank-1 output shape [batch].

The 2 layers serve different directions. Flatten goes from a convolution stack into a Dense head. Reshape also goes the other way, and turns a Dense output back into a volume. A decoder needs that direction.

From there, the transposed convolutions of section 3.5.5 grow the spatial axes back with a learned kernel. The parameter-free UpSampling1D, UpSampling2D, and UpSampling3D layers do the same without one. See 3.6. Pooling Layers for the upsampling shapes and their Interpolation modes.

The constructor rejects more than 1 -1, a 0, and any value below -1, and each case returns Error::InvalidParameter. An element count that cannot match is a forward-time error, and returns Error::ShapeMismatch. For example, vec![2, 3] needs 6 elements per sample, and a [5, 4] input holds only 4.

Reshape::new rejects a 0 in target_shape on purpose. No non-empty input can ever match a 0 axis. The constructor raises this error early, not at the first forward pass. The set of valid programs stays the same, and only the moment of the error moves earlier.

Like Flatten, Reshape moves no data. It reads and writes in C order, so the last axis varies fastest. The channels-last layout puts the channel axis innermost, so a reshape that splits or merges the trailing axes regroups channels before spatial positions. Flatten uses that same order, so the 2 layers agree on where each element lands. Read a target shape with that order in mind.

The following program folds a rank-2 batch into a volume through 1 inferred axis. forward_mut is the entry point of a caller that drives 1 layer by hand. It builds the layer from the tensor it receives, and then it completes the pass:

use ndarray::Array2;
use rustyml::neural_network::Ctx;
use rustyml::neural_network::layers::*;
use rustyml::neural_network::traits::UnaryLayer;

fn main() {
    // 5 samples of 4 features each.
    let x = Array2::<f32>::from_shape_fn((5, 4), |(n, k)| (n * 4 + k) as f32).into_dyn();

    // The target shape names the axes after the batch axis. -1 takes 4 / 2 = 2.
    let mut layer = Reshape::new(vec![-1, 2]).unwrap();

    // `forward_mut` builds the layer from the tensor, then runs the pass. A training
    // context keeps what the backward pass needs.
    let mut ctx = Ctx::training();
    let folded = layer.forward_mut(&x, &mut ctx).unwrap();

    assert_eq!(folded.shape(), &[5, 2, 2]);
    // C order: the last axis varies fastest, so sample 0 folds into [[0, 1], [2, 3]].
    assert_eq!(folded[[0, 1, 0]], 2.0);

    // A gradient of the output shape restores the input shape.
    let grad = layer.backward(&folded, &mut ctx).unwrap();
    assert_eq!(grad.shape(), &[5, 4]);

    println!("folded shape: {:?}", folded.shape());
}

The next example builds a complete stack. A Conv2D runs over synthetic 1-channel 8x8 images. Flatten then feeds the result into a Dense regression head. The Sequential model trains the stack for a few epochs.

use ndarray::{Array2, Array4};
use rustyml::neural_network::Shape;
use rustyml::neural_network::layers::*;
use rustyml::neural_network::losses::*;
use rustyml::neural_network::optimizers::*;
use rustyml::neural_network::sequential::SequentialBuilder;

fn main() {
    // Synthetic "images": 6 samples, 8x8, 1 channel.
    let mut x = Array4::<f32>::zeros((6, 8, 8, 1));
    for n in 0..6 {
        for i in 0..8 {
            for j in 0..8 {
                x[[n, i, j, 0]] = ((n + i + j) as f32 * 0.1).sin();
            }
        }
    }
    let x = x.into_dyn();

    // 3 regression targets per sample.
    let y = Array2::<f32>::from_shape_fn((6, 3), |(n, k)| (n as f32) * 0.01 + (k as f32) * 0.1)
        .into_dyn();

    let mut model = SequentialBuilder::new()
        // [N, 8, 8, 1] -> Conv2D(4 filters, 3x3, Valid) -> [N, 6, 6, 4]
        .add(
            Conv2D::new(4, (3, 3), (1, 1), Activation::ReLU)
                .unwrap()
                .with_random_state(42),
        )
        // [N, 6, 6, 4] -> Flatten -> [N, 144]
        .add(Flatten::new())
        // [N, 144] -> Dense -> [N, 3]
        .add(Dense::new(3, Activation::Linear).unwrap().with_random_state(7))
        .build(&Shape::with_free_batch(x.shape()))
        .unwrap();
    model.compile(SGD::new(0.01, 0.0, false, 0.0).unwrap(), MeanSquaredError::new());

    model.summary();
    model.fit(&x, &y, 3).unwrap();

    let pred = model.predict(&x).unwrap();
    assert_eq!(pred.shape(), &[6, 3]);
    println!("prediction shape: {:?}", pred.shape());
}

The Dense head names no input width at all. The build threads the flattened feature count 6 * 6 * 4 = 144 into it. The 1 number a reader used to compute by hand is now impossible to get wrong. summary() prints the shape at each stage and the parameter budget:

Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ conv2d (Conv2D)                 │ (None, 6, 6, 4)        │            40 │
│ flatten (Flatten)               │ (None, 144)            │             0 │
│ dense (Dense)                   │ (None, 3)              │           435 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
 Total params: 475 (1900 B)
 Trainable params: 475 (1900 B)
 Non-trainable params: 0 (0 B)

To shrink the spatial extent before the dense head, insert a pooling layer between the convolution and the flatten step. Save the trained stack with the tools in 3.9. Saving and Loading Weights.

3.5.7. Explicit Borders: ZeroPadding and Cropping

PaddingType::Same is a policy. It computes the total padding for you. It splits an odd total by a fixed rule, pad_before = pad_total / 2, so the extra cell lands on the trailing edge. When you want a different split, or padding that no convolution follows, reach for a border layer instead.

The family has 6 members in 2 halves. ZeroPadding1D, ZeroPadding2D, and ZeroPadding3D add zero positions at the ends of the spatial axes. Cropping1D, Cropping2D, and Cropping3D remove positions there. Each half is the backward pass of the other half. All 6 leave the batch axis and the channel axis untouched, and none of them holds a parameter.

Every constructor takes 1 argument and returns the layer directly, with no Result. A border amount is a usize, so no value is invalid at construction. The model build refuses a cropping layer that removes too much instead, as soon as the input extent reaches it.

The argument accepts 3 forms, and the rank decides which ones apply:

Read the 2D tuple form with care. ZeroPadding2D::new((1, 2)) gives 1 row at the top and the bottom, plus 2 columns at the left and the right. It does not give 1 row at the top and 2 at the bottom. Only the 1D pair form names the 2 ends of 1 axis.

A cropping layer must leave at least 1 position on each spatial axis. Cropping1D::new((2, 3)) on a length-5 input removes all 5 steps, so the build returns Error::InvalidInput and names the axis. Cropping1D::new((2, 2)) on the same input leaves 1 step, and it succeeds.

Neither half moves data across the channel axis. A pad allocates a zero tensor and copies the input into the middle of it. A crop copies the interior out. Because each half is the other half’s backward pass, a ZeroPadding2D followed by a Cropping2D with the same amounts is the identity on values.

A border layer that no build has touched refuses forward, so the program below drives it with forward_mut. The program shows the round trip, then puts a pad in front of a Valid convolution so the convolution keeps the spatial size:

use ndarray::{Array2, Array4};
use rustyml::neural_network::layers::*;
use rustyml::neural_network::losses::*;
use rustyml::neural_network::optimizers::*;
use rustyml::neural_network::sequential::SequentialBuilder;
use rustyml::neural_network::traits::UnaryLayer;
use rustyml::neural_network::{Ctx, Shape};

fn main() {
    // 4 samples, 6x6 pixels, 1 channel. Every value is above 0.
    let x = Array4::<f32>::from_shape_fn((4, 6, 6, 1), |(n, i, j, _)| {
        ((n + i + j) as f32 + 1.0) * 0.05
    })
    .into_dyn();

    // 1 zero row at the top only, and 1 zero column at each side: 6x6 -> 7x8.
    let mut ctx = Ctx::inference();
    let mut pad = ZeroPadding2D::new(((1, 0), (1, 1)));
    let padded = pad.forward_mut(&x, &mut ctx).unwrap();
    assert_eq!(padded.shape(), &[4, 7, 8, 1]);
    assert_eq!(padded[[0, 0, 0, 0]], 0.0);

    // A crop with the same amounts cancels the pad exactly.
    let mut crop = Cropping2D::new(((1, 0), (1, 1)));
    assert_eq!(crop.forward_mut(&padded, &mut ctx).unwrap(), x);

    // In a model: 6x6 grows to 8x8, so the 3x3 Valid convolution gives 6x6 back.
    let y = Array2::<f32>::from_shape_fn((4, 2), |(n, k)| (n as f32) * 0.1 + (k as f32) * 0.01)
        .into_dyn();

    let mut model = SequentialBuilder::new()
        .add(ZeroPadding2D::new(1))
        .add(Conv2D::new(2, (3, 3), (1, 1), Activation::ReLU).unwrap())
        .add(Flatten::new())
        .add(Dense::new(2, Activation::Linear).unwrap())
        .build(&Shape::with_free_batch(x.shape()))
        .unwrap();
    model.compile(SGD::new(0.01, 0.0, false, 0.0).unwrap(), MeanSquaredError::new());

    model.fit(&x, &y, 3).unwrap();

    let pred = model.predict(&x).unwrap();
    assert_eq!(pred.shape(), &[4, 2]);
    println!("prediction shape: {:?}", pred.shape());
}

summary() prints a real output shape for a border layer, and it needs no forward pass to do so. The build gave the layer the shape that reaches it, and compute_output_shape adds the 2 amounts of each spatial axis to it. Every layer of this chapter answers that way now, so a printed shape is a shape the model produces.

3.5.8. Permute, RepeatVector, and Reverse

3 more parameter-free layers round out the shape family. Permute reorders the axes after the batch axis. RepeatVector turns 1 vector per sample into a sequence of identical steps. Reverse flips the order of the positions inside 1 axis and changes no shape.

Permute::new(dims) names the new order. dims counts from 1 and never includes the batch axis. Permute::new(vec![2, 1]) on a [batch, steps, features] input gives [batch, features, steps]. The entries must be a permutation of 1..=dims.len(), so each axis appears exactly once. The input rank must be dims.len() + 1, and a rank the layer does not serve is a forward-time Error::InvalidInput.

Permute::new(vec![]) returns Error::InvalidParameter. Such a layer reorders nothing, and every other layer here needs a batch axis and at least 1 more axis.

Permute differs from Reshape in a way worth stating plainly. A reshape reads the same buffer in the same order under a new shape, so every value keeps its place in memory order. A permute reads the buffer in a new order, so the values land at new positions. Both layers allocate a new tensor, but only the permute pays for a scattered copy.

A permute that leaves the last axis last keeps whole rows contiguous and stays close to copy speed. A permute that moves the last axis cuts the contiguous run to 1 element and costs several times more. When the model allows a choice, put the permute where the tensor is small.

RepeatVector::new(n) takes a [batch, features] input and gives [batch, n, features]. Every one of the n steps holds the same vector. n must be greater than 0, and a 0 is an Error::InvalidParameter at construction. The input rank must be 2.

RepeatVector exists for the decoder side of an encoder-decoder model. A recurrent layer returns only its last hidden state by default, a rank-2 tensor, and a recurrent layer needs a rank-3 input. RepeatVector bridges the 2, so an LSTM can feed another LSTM over a sequence of a different length. This layer emits the same vector at every step, which is the standard way to seed a decoder with a fixed context. A plain stack needs no bridge, because with_return_sequences(true) makes the lower layer emit a rank-3 sequence of its own. See 3.7. Recurrent Layers for both paths.

The following program runs both layers by hand:

use ndarray::{Array2, Array3};
use rustyml::neural_network::Ctx;
use rustyml::neural_network::layers::*;
use rustyml::neural_network::traits::UnaryLayer;

fn main() {
    // 2 samples, 2 steps, 3 features.
    let x = Array3::<f32>::from_shape_fn((2, 2, 3), |(n, t, f)| (n * 6 + t * 3 + f) as f32)
        .into_dyn();

    // dims counts from 1 and skips the batch axis, so (2, 1) swaps steps and features.
    let mut permute = Permute::new(vec![2, 1]).unwrap();
    let mut ctx = Ctx::training();
    let swapped = permute.forward_mut(&x, &mut ctx).unwrap();
    assert_eq!(swapped.shape(), &[2, 3, 2]);
    // Sample 0 was [[0, 1, 2], [3, 4, 5]], and it transposes to [[0, 3], [1, 4], [2, 5]].
    assert_eq!(swapped[[0, 1, 0]], 1.0);
    assert_eq!(swapped[[0, 0, 1]], 3.0);

    // The gradient runs back through the inverse order, from the same context.
    let grad = permute.backward(&swapped, &mut ctx).unwrap();
    assert_eq!(grad.shape(), &[2, 2, 3]);

    // RepeatVector turns 1 vector per sample into a sequence of identical steps.
    let state = Array2::<f32>::from_shape_fn((2, 3), |(n, f)| (n * 3 + f) as f32).into_dyn();
    let mut repeat = RepeatVector::new(4).unwrap();
    let sequence = repeat.forward_mut(&state, &mut ctx).unwrap();
    assert_eq!(sequence.shape(), &[2, 4, 3]);
    assert_eq!(sequence[[0, 0, 2]], sequence[[0, 3, 2]]);

    println!("swapped {:?}, sequence {:?}", swapped.shape(), sequence.shape());
}

Both layers report a real output shape from summary() with no forward pass, for the same reason the border layers do. Permute applies its order to the shape the build hands it, and RepeatVector inserts its step axis into that shape.

Reverse::new(axis) reverses the order along 1 axis. The axis counts against the FULL rank, and a negative axis counts back from the end, which follows Concatenate. Position k of that axis leaves at position n - 1 - k, and every other axis stays as it was. The layer holds no array, changes no shape, and its backward pass is the same reversal applied to the upstream gradient.

The layer refuses 2 inputs. Axis 0 is the batch axis, and reversing it would put every sample against the target of another sample. The layer refuses an input of rank below 3 as well. A rank-2 tensor is 1 feature vector per sample, and it holds no axis whose order carries meaning.

That second refusal is the one that matters in practice. It is what a recurrent branch gives when return_sequences stays unset. Without the refusal, the layer would reorder the features of every sample and report nothing.

The layer reports its type with the axis in it, as Reverse(1) rather than Reverse. A checkpoint compares the type of the layer at each position and the shapes it built for. This layer changes no shape and holds no array. The axis is therefore the only thing that separates 2 of them. The reported type is the only field a strict load would see it in.

The main use is the time axis of a bidirectional model. See 3.7. Recurrent Layers, section 3.7.9.

3.5.9. Identity

Identity is the layer that does nothing. It returns its input unchanged at every rank and every shape, and its backward pass returns the gradient it receives. Identity::new() takes no argument and returns the layer directly, with no Result, because it has nothing to misconfigure. It also implements Default.

A layer that does nothing sounds useless until a program builds a model instead of a person writing it out by hand. A function that picks among several layers needs something to return when the choice is “no operation”. A stack whose depth is a runtime value needs a filler. An experiment that compares an architecture with and without a layer needs both models to keep the same layer count. This way, summary() and a saved weight file still line up. All 3 read better with a layer that does nothing than with an Option at every position.

The layer copies. It cannot borrow, because a layer returns an owned tensor. The copy is 1 linear pass and runs at memory speed, but it is not free. Remove the layer rather than keep it in a model you have finished.

A rank-0 tensor has no batch axis and gives Error::InvalidInput. A tensor with a zero extent gives Error::EmptyInput. Reshape, Permute, RepeatVector, and the border layers draw the same 2 boundaries.

use ndarray::Array2;
use rustyml::neural_network::layers::*;
use rustyml::neural_network::losses::*;
use rustyml::neural_network::optimizers::*;
use rustyml::neural_network::sequential::SequentialBuilder;
use rustyml::neural_network::traits::UnaryLayer;
use rustyml::neural_network::{Ctx, Shape};

fn main() {
    // 2 samples, 3 features.
    let x = Array2::<f32>::from_shape_fn((2, 3), |(n, f)| (n * 3 + f) as f32).into_dyn();

    let mut identity = Identity::new();
    let mut ctx = Ctx::training();
    let out = identity.forward_mut(&x, &mut ctx).unwrap();
    assert_eq!(out, x);

    // The gradient goes back exactly as it arrived.
    let grad = identity.backward(&out, &mut ctx).unwrap();
    assert_eq!(grad, out);

    // A stack whose depth is a runtime value. The filler keeps the layer count fixed.
    let depth = 2;
    let mut builder = SequentialBuilder::new();
    for step in 0..3 {
        if step < depth {
            builder = builder.add(Dense::new(3, Linear::new()).unwrap());
        } else {
            builder = builder.add(Identity::new());
        }
    }
    let mut model = builder.build(&Shape::with_free_batch(x.shape())).unwrap();
    model.compile(
        SGD::new(0.01, 0.0, false, 0.0).unwrap(),
        MeanSquaredError::new(),
    );
    model.summary();

    println!("output shape {:?}", model.predict(&x).unwrap().shape());
}

summary() prints a real output shape for this layer with no forward pass. Identity changes no extent, so its output shape is the shape the build hands it.

3.5.10. Errors and How to Avoid Them

The convolution layers and the shape layers validate their configuration closely. They return typed errors instead of a panic on bad input. The table below lists the common cases and their error variants.

2 of these error paths need more explanation. The first is the kernel that does not fit its input axis. The build checks it, and only under Valid padding. A constructor cannot check it, because with_padding and with_dilation_rate both run after the constructor, and both decide whether the rule applies at all. Under Valid a kernel that does not fit leaves no complete window, so the axis would have 0 positions.

A 0 extent is not an answer. It passes through every later layer of the stack, and it turns up far from the layer that made it. The build returns InvalidInput instead, and the message names the layer, the axis, and the effective kernel extent that the axis must reach. The build creates nothing past that layer. Under Same, and under Causal on a Conv1D, the padding supplies the missing cells. The same layer builds and runs, and it returns the size those rules give.

The second is the runtime extent that disagrees with the build. Every built layer checks it, and it checks every axis except the batch axis. A Conv2D built for (None, 5, 5, 2) refuses a 3-channel tensor and a 2-row tensor alike, with InvalidInput. The message names the build shape and the axis at fault. The batch axis stays free, so a partial final mini-batch always passes. The build shape is the contract, and a layer no longer trusts a caller to honor it.

The following program exercises these recoverable paths:

use ndarray::Array;
use rustyml::error::Error;
use rustyml::neural_network::layers::*;
use rustyml::neural_network::traits::UnaryLayer;
use rustyml::neural_network::{Ctx, Shape};

fn main() {
    // 1. A kernel larger than the input constructs. Only Valid padding refuses it, and the
    //    build is where that happens, before the layer holds a single weight.
    let mut small_input = Conv2D::new(1, (3, 3), (1, 1), Activation::Linear).unwrap();
    let message = match small_input.build(&Shape::with_free_batch(&[1, 2, 2, 1])) {
        Ok(()) => panic!("a 3x3 Valid kernel does not fit a 2x2 input"),
        Err(error) => error.to_string(),
    };
    assert!(message.contains("Conv2D"), "{message}");
    assert!(message.contains("effective kernel extent, which is 3"), "{message}");

    // The same layer under Same padding builds and runs, because the padding supplies the
    // missing cells.
    let tiny = Array::ones((1, 2, 2, 1)).into_dyn();
    let mut ctx = Ctx::inference();
    let mut padded = Conv2D::new(1, (3, 3), (1, 1), Activation::Linear)
        .unwrap()
        .with_padding(PaddingType::Same);
    padded.build(&Shape::with_free_batch(&[1, 2, 2, 1])).unwrap();
    assert_eq!(padded.forward(&tiny, &mut ctx).unwrap().shape(), &[1, 2, 2, 1]);

    // 2. A zero depth multiplier is rejected by the builder rather than panicking.
    let bad_dm = DepthwiseConv2D::new((2, 2), (1, 1), Activation::Linear)
        .unwrap()
        .with_depth_multiplier(0);
    assert!(matches!(bad_dm, Err(Error::InvalidParameter { .. })));

    // 3. A dilation above 1 needs a stride of 1, so the builder refuses the pair.
    let bad_dilation = Conv2D::new(1, (3, 3), (2, 2), Activation::Linear)
        .unwrap()
        .with_dilation_rate((2, 2));
    assert!(matches!(bad_dilation, Err(Error::InvalidParameter { .. })));

    // 4. An extent that disagrees with the build shape, on a spatial axis.
    let mut conv = Conv2D::new(1, (3, 3), (1, 1), Activation::Linear).unwrap();
    conv.build(&Shape::with_free_batch(&[1, 5, 5, 1])).unwrap();
    let small = Array::ones((1, 2, 5, 1)).into_dyn(); // height 2, built for 5
    assert!(matches!(
        conv.forward(&small, &mut ctx),
        Err(Error::InvalidInput(_))
    ));

    // 5. The same check covers the channel axis, on every layer of the family.
    let mut dw = DepthwiseConv2D::new((2, 2), (1, 1), Activation::Linear).unwrap();
    dw.build(&Shape::with_free_batch(&[1, 4, 4, 2])).unwrap();
    let wrong_channels = Array::ones((1, 4, 4, 3)).into_dyn(); // 3 channels, built for 2
    assert!(matches!(
        dw.forward(&wrong_channels, &mut ctx),
        Err(Error::InvalidInput(_))
    ));

    // 6. The batch axis is never checked, so any batch size passes.
    let batch_of_9 = Array::ones((9, 5, 5, 1)).into_dyn();
    assert_eq!(
        conv.forward(&batch_of_9, &mut ctx).unwrap().shape(),
        &[9, 3, 3, 1]
    );

    println!("all error paths behaved as documented");
}

The ForwardPassNotRun error most often arrives during training. The backward pass takes back what the matching forward pass parked in the context. Call backward with a fresh context, and the layer returns this variant instead of reading stale state. A context that an inference pass left empty gives the same result. A second backward against the same context returns it too, because the first one took the cache away.

Inside Sequential, the framework handles this order for you. You meet this error only when you drive layers by hand. See 1.6. Error Handling for the full error taxonomy and the smart constructors behind these variants.