Prepare data

Enable the utils feature for this chapter. Add ndarray = "=0.17.2" to your dependencies. The examples also work with the default RustyML features.

Choose array types

Use Array2<f64> for classical model features. Each row contains one sample. Each column contains one feature. A matrix with shape (100, 4) contains 100 samples and four features.

Use Array1<f64> for regression targets. Class label types depend on the receiving API. Keep target order aligned with feature rows.

array! creates small arrays directly. Array2::from_shape_vec builds matrices from flat values in row order. Its shape must match the number of values. Handle its shape error before passing data to RustyML.

Use .view() to borrow an array without copying its elements. Many preprocessing functions accept owned arrays and views through generic array references.

Split functions take ownership of their arrays. Clone them only when you need the originals later.

Neural network layers use their own tensor shapes. Check each layer before reshaping data.

Split before fitting preprocessing

train_test_split(x, y, test_size, random_state) returns (x_train, x_test, y_train, y_test) inside a Result. Pass Some(0.33) to reserve about one third for testing. None selects the default test fraction, 0.3.

Pass Some(42) for a fixed random seed. None uses the global seed stream when set, or system entropy. See Random seeds. The split rounds the test count and keeps at least one sample on each side.

Use train_test_split_stratified for classification when both subsets need each class. It requires labels that implement Clone + Eq + Hash. Integer and string labels support these bounds. Floating-point labels do not implement Eq.

Each class must contain at least two samples. The function splits each class separately. Rounding can make its total test fraction differ from the requested fraction.

Fit StandardScaler on training features only. Transform training and held-out features with the same fitted scaler. This sequence prevents test information from changing the training transformation. Keep the scaler with the model for later predictions.

The scaler stores each feature’s population mean and variance. It subtracts the mean and divides by the standard deviation by default. Constant or near-constant features use a divisor of 1.0.

fit replaces stored statistics. fit_transform fits and transforms the same input. partial_fit updates statistics from additional training batches.

The stateless standardize function computes fresh statistics from every input. Use the stateful scaler when you need consistent transformations across datasets.

Run data_prepare.rs:

use ndarray::array;
use rustyml::utils::{StandardScaler, train_test_split, train_test_split_stratified};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let x = array![
        [1.0, 10.0],
        [2.0, 20.0],
        [3.0, 30.0],
        [4.0, 40.0],
        [5.0, 50.0],
        [6.0, 60.0]
    ];
    let y = array![0, 0, 0, 1, 1, 1];
    let (x_train, x_test, y_train, y_test) =
        train_test_split(x.clone(), y.clone(), Some(0.33), Some(42))?;
    println!("train: {:?}, labels: {}", x_train.dim(), y_train.len());
    println!("test: {:?}, labels: {}", x_test.dim(), y_test.len());
    let mut scaler = StandardScaler::new();
    scaler.fit(&x_train)?;
    let z_train = scaler.transform(&x_train)?;
    let z_test = scaler.transform(&x_test)?;
    println!("scaled: {:?}, {:?}", z_train.dim(), z_test.dim());
    let (_, _, strat_train, strat_test) = train_test_split_stratified(x, y, Some(0.33), Some(42))?;
    println!(
        "class 0 counts: train {}, test {}",
        strat_train.iter().filter(|&&v| v == 0).count(),
        strat_test.iter().filter(|&&v| v == 0).count()
    );
    Ok(())
}

train: (4, 2), labels: 4
test: (2, 2), labels: 2
scaled: (4, 2), (2, 2)
class 0 counts: train 2, test 1

Encode labels consistently

to_categorical accepts nonnegative i32 labels. Each label selects its corresponding column. Pass a fixed class count to preserve the same width across subsets. Without a class count, the function uses the largest label plus one.

to_categorical_with_mapping accepts a slice of hashable labels. It assigns indices in first-seen order. Save the returned mapping. Use those indices when encoding later data. Calling it independently on test labels can change column meanings. Decide how your application handles labels absent from the training mapping.

to_sparse_categorical selects each row’s largest value. It returns column indices as i32 labels. Ties select the lowest column index. Use your saved mapping to recover original string labels. This function also accepts finite score matrices.

Run data_labels.rs:

use ndarray::array;
use rustyml::utils::{to_categorical, to_categorical_with_mapping, to_sparse_categorical};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let labels = array![0_i32, 2, 1];
    let encoded = to_categorical(&labels, Some(3))?;
    println!("one-hot shape: {:?}", encoded.dim());
    println!("first row: {:?}", encoded.row(0).to_vec());
    println!("decoded: {:?}", to_sparse_categorical(&encoded)?.to_vec());
    let (encoded, mapping) = to_categorical_with_mapping(&["cat", "dog", "cat"], None)?;
    println!("cat: {}, dog: {}", mapping["cat"], mapping["dog"]);
    println!("mapped shape: {:?}", encoded.dim());
    Ok(())
}

one-hot shape: (3, 3)
first row: [1.0, 0.0, 0.0]
decoded: [0, 2, 1]
cat: 0, dog: 1
mapped shape: (3, 2)

Handle preprocessing errors

The examples return errors with ?. Production code can match rustyml::error::Error variants. Include a fallback arm because this enum is non-exhaustive.

Check dataset values before training. Split utilities do not perform full feature validation.