2. Classical Machine Learning

Classical machine learning covers everything in RustyML that is not a neural network: linear models, trees, kernel methods, clustering, and dimensionality reduction. These algorithms train fast, need little data, and produce models you can inspect. Reach for them first. Move to Chapter 3 only when a problem needs a deep network. Every estimator in this chapter lives under rustyml::machine_learning and shares one small contract. Construct it with new, which validates its arguments and returns Result. LinearRegression::new is the one exception: it cannot fail, so it returns Self directly. Train the model with fit, and run inference with predict, unless the model reduces dimensionality. The dimensionality-reduction transformers use transform and fit_transform instead of predict. Learn this rhythm once, and it carries across all 14 models in this chapter.

Read Chapter 1 before this chapter. Read Working with ndarray first, since every model consumes an Array2<f64> feature matrix. Read Error Handling too, since constructors and fit/predict all return the crate’s Result. Chapter 4 covers encoding and scaling your features. Chapter 5 covers the accuracy, silhouette, and R^2 scores you use to judge these models.

Every model in this chapter follows the shape below, with only the details changing:

use rustyml::machine_learning::LinearRegression;
use ndarray::array;

fn main() {
    // construct -> fit -> predict, the rhythm every estimator repeats
    let mut model = LinearRegression::new(true);
    let x = array![[1.0, 2.0], [2.0, 3.0], [3.0, 4.0]];
    let y = array![6.0, 9.0, 12.0];
    model.fit(&x, &y).unwrap();

    let preds = model.predict(&array![[4.0, 5.0]]).unwrap();
    println!("prediction: {:?}", preds);
}

Each section stands alone, so jump straight to the model you need. Reading in order moves from the simplest estimators to the most involved.

Supervised learning predicts a target from labeled examples. Linear Regression fits a continuous target, with optional L1/L2 regularization and a choice of a gradient-descent or a closed-form solver. It is the best place to learn the fit/predict loop. Logistic Regression reuses that gradient machinery for binary classification. K-Nearest Neighbors skips training and classifies by proximity, with a selectable distance metric and weighting scheme. Decision Trees split the feature space into readable if/else rules, using the ID3, C4.5, or CART algorithm, with pruning. Support Vector Machines covers 2 models: a kernelized SVC (SMO solver) for curved boundaries, and a fast LinearSVC for wide, high-dimensional data. Linear Discriminant Analysis classifies and reduces dimensions at the same time, by modeling each class as a Gaussian with a shared covariance.

Clustering groups unlabeled data. KMeans partitions points into a fixed number of clusters, using k-means++ initialization. It is fast, and the usual default choice. DBSCAN finds clusters of any shape by density, and labels outliers as noise. It needs no cluster count in advance. Mean Shift also finds the number of clusters on its own, by climbing a density surface. Score all 3 methods with the clustering metrics.

Dimensionality reduction compresses features while it keeps their structure. Principal Component Analysis is the linear default for decorrelation and compression. Kernel PCA extends it to nonlinear structure, through RBF, polynomial, and other kernels. t-SNE embeds high-dimensional data into 2 or 3 dimensions, for visualization only. It learns no reusable projection. It exposes fit_transform alone, with no out-of-sample transform.

Anomaly detection stands as its own family. Isolation Forest scores how easily each point isolates under random splits, and flags outliers without needing any labels.