4. Data Preprocessing

Raw data almost never arrives in the shape a model wants. Feature columns can span very different scales. Class labels can come as strings or as non-consecutive integers. You also need an honest held-out set before you fit anything. This chapter covers the preprocessing helpers in rustyml::utils: splitting, standardizing, normalizing, and label encoding. They all live behind the utils feature flag, and operate directly on the ndarray arrays your models already consume.

If you come from scikit-learn, note this module’s shape. Most of it is plain functions, not stateful fit/transform estimators. Each call computes its statistics from the array you pass it, and returns a fresh array. The exceptions are the scalers: StandardScaler, MinMaxScaler, MaxAbsScaler, RobustScaler, and Normalizer. These do carry the fit/transform contract across a train/test boundary. Each stores what it learned from the training matrix, so later batches go through the identical map. Either way, the ordering discipline is yours. Split your data first, then fit the scaling. This keeps anything derived from test rows out of training. Every function returns Result<_, Error> and validates its input first. Each function rejects an empty dataset, a shape mismatch, or a non-finite value, where that check applies. Read Error Handling if you have not.

Read the sections in pipeline order. Split first (4.1). Scale the training features next (4.2). Then encode labels, if your model or loss needs one-hot targets (4.3). The example below runs that exact sequence:

use ndarray::{Array1, array};
use rustyml::utils::train_test_split::train_test_split;
use rustyml::utils::standardize::{StandardizationAxis, standardize};
use rustyml::utils::label_encoding::to_categorical;

fn main() {
    // Six samples, two features on very different scales, two classes.
    let x = array![
        [1.0, 20.0],
        [2.0, 21.0],
        [3.0, 19.0],
        [4.0, 22.0],
        [5.0, 18.0],
        [6.0, 23.0],
    ];
    let y: Array1<i32> = array![0, 1, 0, 1, 0, 1];

    // 1. Split first, so scaling never sees the test rows.
    let (x_train, _x_test, y_train, _y_test) =
        train_test_split(x, y, Some(0.34), Some(42)).unwrap();

    // 2. Standardize each feature column to zero mean, unit variance.
    let x_train = standardize(&x_train, StandardizationAxis::Column).unwrap();

    // 3. One-hot encode the integer labels for a categorical loss.
    let y_train = to_categorical(&y_train, None).unwrap();

    println!("x_train shape: {:?}", x_train.shape());
    println!("y_train shape: {:?}", y_train.shape());
}

4.1. Train-Test Split

Train-Test Split carves your feature matrix and labels into disjoint training and test subsets. This lets evaluation numbers reflect unseen data. train_test_split does a plain random partition. train_test_split_stratified splits each class independently, so both sides keep the same label proportions. Choose the stratified variant whenever a class is rare, since a plain split can drop that class from one side entirely. test_size defaults to 0.3. The label array is generic over its element type. Passing a random_state seed makes the shuffle reproducible (see Reproducibility and Random Seeds).

4.2. Standardization and Normalization

Standardization and Normalization are 2 different rescalings that people often confuse. standardize computes z-scores. It subtracts the mean, then divides by the standard deviation, so each feature ends up with zero mean and unit variance. normalize rescales each row or column to unit norm, under an L1, L2, Max, or custom Lp order.

Each operation also has a stateful form. A stateful form remembers its training statistics, so a test split or a single live sample gets scaled by the numbers the model trained under. The stateful forms are StandardScaler and Normalizer. MinMaxScaler (bounded range), MaxAbsScaler (magnitude only, keeping structural zeros), and RobustScaler (median and IQR, so outliers do not set the scale) join them.

Column standardization is the everyday choice before any distance-based or gradient-based model. KNN, SVM, PCA, and neural networks all misbehave when features live on different scales. Per-row unit-norm normalization instead suits direction-only vectors, such as TF-IDF rows compared by cosine similarity. Both operations handle a degenerate lane instead of dividing by a vanishing scale. standardize forces the divisor to 1.0, so a constant feature centers to zeros. normalize leaves a near-zero lane exactly as it is.

4.3. Label Encoding

Label Encoding converts between the integer labels a dataset ships with and the one-hot matrices a categorical loss expects. to_categorical turns a column of non-negative i32 labels into a one-hot matrix. to_categorical_with_mapping does the same for arbitrary hashable labels, strings included, and also returns the label-to-index map, so you can decode later. to_sparse_categorical runs the inverse. It reduces a one-hot or softmax-probability matrix to the per-row argmax, as i32 labels. Use to_sparse_categorical to turn a network’s softmax output back into predicted classes. If you know Keras, this trio matches its to_categorical. One-hot targets are exactly what Categorical Cross-Entropy consumes.