2.5. Support Vector Machines

A support vector machine finds the decision boundary that sits as far as possible from the nearest points of each class. This is the maximum-margin hyperplane. RustyML ships 2 implementations that reach this goal from opposite directions.

SVC is the kernelized version. It lifts the data into a higher-dimensional space through a kernel function and finds a linear boundary there. Back in the original space, that boundary becomes curved. LinearSVC skips the kernel. It fits a straight hyperplane by minimizing hinge loss with stochastic gradient descent.

Both models are strictly binary classifiers, with no built-in one-vs-rest wrapper. Both use the same {0.0, 1.0} label domain, so you can swap one for the other without a rewrite of the target array. Past that shared goal, the 2 models differ in solver, regularization, and cost. The rest of this page helps you pick the right one for a problem.

2.5.1. When to reach for which

The choice comes down to 2 numbers: the sample count (n_samples) and the feature count (n_features). It also depends on whether a straight line can separate the classes.

SVC builds an n_samples x n_samples kernel matrix, also called the Gram matrix, before training starts. The Sequential Minimal Optimization (SMO) solver then works over that matrix. The matrix sets a hard limit: it costs O(n_samples^2) in memory and O(n_samples^2 * n_features) to build. At 10,000 samples, the Gram matrix alone takes about 800 MB of f64 values. At 260,000 samples, it would need hundreds of gigabytes.

SVC fits well when the boundary is truly nonlinear and n_samples stays modest, from hundreds to a few thousand rows. It does not scale to large datasets. That limit comes from the algorithm, not from this implementation.

LinearSVC never forms a kernel matrix. Each epoch runs a few matrix-vector passes over the data. That costs O(n_samples * n_features) in time and only O(n_features) in memory for the weight vector. It scales linearly in both dimensions, which makes it the default choice for wide, high-dimensional, or large-sample problems. Text classification with tens of thousands of sparse features is the classic case.

The catch is that LinearSVC can only fit a linear boundary. If the classes are not linearly separable, no amount of training fixes that. Use SVC with a kernel instead.

Watch for one trap. If you need only a linear boundary on a large dataset, do not use SVC with KernelType::Linear. That choice still pays the full O(n_samples^2) Gram-matrix cost, for a problem LinearSVC solves in linear memory. SVC with a linear kernel earns its cost only on small data, where you want the exact max-margin dual solution.

Both estimators are sensitive to feature scale. The RBF kernel measures squared Euclidean distance, and the SGD in LinearSVC takes fixed-size steps in feature space. Standardize your columns first (see Standardization and Normalization), unless they already share a range.

Both estimators also share one hard rule for y: every entry must be exactly 0.0 or 1.0. Any other value, such as a -1.0 from a textbook derivation, a 2.0, or a stray 0.5, causes an Error::InvalidInput at fit. Neither estimator remaps labels silently. If your labels are strings or arbitrary integers, run them through Label Encoding first.

2.5.2. SVC: kernels and the SMO solver

Construct an SVC with SVC::new(kernel, regularization_param, tol, max_iter). This call validates the arguments and returns a Result. Set the seed for reproducible training in a separate builder step, with with_random_state.

C is the regularization knob. It sets the upper bound on each dual coefficient, 0 <= alpha <= C. A large C lets the solver push the alphas high, fitting every training point with a narrow margin that tolerates few violations. A small C keeps the alphas small, widening the margin and accepting more slack. This is the opposite of the lambda intuition from linear models: a larger C means less regularization. Any non-positive or non-finite value fails at construction with Error::InvalidParameter.

SVC::default() gives an RBF kernel with gamma = 0.1, C = 1.0, tol = 0.001, and max_iter = 1000. This is a reasonable start when you have no prior information, though gamma almost always needs tuning.

Training runs Sequential Minimal Optimization (SMO). It repeatedly picks a pair of dual variables that violate the Karush-Kuhn-Tucker (KKT) conditions. It optimizes that pair analytically, holding the rest fixed. This repeats until the whole set satisfies the KKT conditions within tol, or until training hits the iteration cap.

The dual problem uses +1/-1 targets, because the y_i * y_j products in its objective only make sense with a sign. fit converts your {0.0, 1.0} column to +1/-1 once, on entry, and keeps that signed form internal. predict maps the sign back to {0.0, 1.0}. You never pass +1/-1 labels in, and, with one exception noted below, you never get them out either.

The Gram matrix and the initial error cache can build in parallel. The SMO inner loop stays sequential on purpose, so the optimization path stays reproducible for a given seed.

This example runs the whole construct-fit-predict loop with a linear kernel, on data a straight line separates cleanly:

use rustyml::machine_learning::{KernelType, SVC};
use ndarray::array;

fn main() {
    // Class 1 sits upper-right. Class 0 sits lower-left. A line separates them.
    let x = array![
        [2.0, 2.0], [3.0, 2.0], [2.0, 3.0], [3.0, 3.0],
        [-2.0, -2.0], [-3.0, -2.0], [-2.0, -3.0], [-3.0, -3.0],
    ];
    let y = array![1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0];

    let mut svc = SVC::new(KernelType::Linear, 10.0, 1e-3, 1000)
        .unwrap()
        .with_random_state(42);
    svc.fit(&x, &y).unwrap();

    // predict emits labels in {0.0, 1.0}, the same domain fit received
    let preds = svc.predict(&x).unwrap();
    println!("predictions: {:?}", preds);

    // decision_function returns the raw signed distance to the hyperplane
    let scores = svc.decision_function(&x).unwrap();
    println!("scores: {:?}", scores);
}

predict sets the threshold at zero. It maps >= 0.0 to 1.0 and everything else to 0.0. decision_function hands back the raw signed score. A positive score means the class-1.0 side, and its size shows how far the point sits from the boundary. Use decision_function when you want a confidence-like ordering instead of a hard label. There is also fit_predict, which fits and then predicts on the same matrix in 1 call.

The kernel zoo and the gamma coefficient

KernelType is the same enum that Kernel PCA uses, so learning it here pays off twice. It has 5 variants.

RBF is the default kernel and the one to try first for nonlinear data. It is a smooth, local similarity measure that only needs gamma tuned. Poly adds explicit interaction terms up to degree, but high degrees overflow quickly. A degree-400 polynomial on modestly large inputs pushes the decision value to infinity, which surfaces as Error::NonFinite at predict time.

Sigmoid mimics a two-layer network, but it is not positive-definite for all parameters, so it can behave erratically. Cosine measures angle rather than distance. It guards against zero vectors by returning 0.0 for them.

The gamma field is not a bare f64. It is a Gamma enum with 3 variants, matching scikit-learn’s 'scale', 'auto', and explicit choices:

• Gamma::Value(v): an explicit coefficient you supply.
• Gamma::Scale: resolved at fit time to 1 / (n_features * X.var()), where X.var() is the population variance of all entries of the training matrix.
• Gamma::Auto: resolved at fit time to 1 / n_features.

Scale and Auto are data-dependent, so they stay placeholders until fit sees the data. fit resolves them once and stores the concrete value. So a call to get_kernel() after fit returns a KernelType whose gamma is now a Gamma::Value. Scale fails with Error::InvalidInput if the data has zero variance, for example when every feature is constant, because the formula would divide by zero.

gamma controls how far a single training point’s influence reaches. A large gamma makes the RBF bumps tight and the boundary wiggly, a fast route to overfitting. A small gamma makes the bumps broad and the boundary smooth. It is the parameter you will tune most.

RBF on a problem no line can split

A kernel exists to separate data that a hyperplane cannot. Two concentric rings are the canonical example: an inner ring of one class, surrounded by an outer ring of the other. No straight line divides them, and a linear kernel is helpless here. The RBF kernel, which scores points by proximity, classifies every one correctly.

use rustyml::machine_learning::{Gamma, KernelType, SVC};
use ndarray::array;

fn main() {
    // Inner ring (radius 1) is class 1, outer ring (radius 5) is class 0.
    let x = array![
        [1.0, 0.0], [-1.0, 0.0], [0.0, 1.0], [0.0, -1.0],
        [5.0, 0.0], [-5.0, 0.0], [0.0, 5.0], [0.0, -5.0],
    ];
    let y = array![1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0];

    let mut svc = SVC::new(
        KernelType::RBF { gamma: Gamma::Value(0.5) },
        10.0,   // C
        1e-3,   // tol
        5000,   // max_iter, nonlinear problems need more SMO passes
    )
    .unwrap()
    .with_random_state(42);
    svc.fit(&x, &y).unwrap();

    let preds = svc.predict(&x).unwrap();
    let correct = preds.iter().zip(y.iter()).filter(|&(p, t)| p == t).count();
    println!("RBF accuracy on rings: {}/{}", correct, y.len());
}

Swap KernelType::RBF { .. } for KernelType::Linear on this dataset, and the classifier can no longer place every point on the right side. The rings are not linearly separable. That is exactly the failure mode a kernel exists to fix. Nonlinear problems also generally need a higher max_iter than linear ones, because the SMO solver has more support vectors to settle.

Inspecting the fitted model

After fit, SVC exposes the pieces of the trained model through getters. This is more than most estimators in this guide offer. The support vectors are the only training rows that matter to prediction, the ones whose dual coefficient came out non-zero. You can read them back directly:

use rustyml::machine_learning::{Gamma, KernelType, SVC};
use ndarray::array;

fn main() {
    let x = array![
        [2.0, 2.0], [3.0, 2.0], [2.0, 3.0], [3.0, 3.0],
        [-2.0, -2.0], [-3.0, -2.0], [-2.0, -3.0], [-3.0, -3.0],
    ];
    let y = array![1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0];

    let mut svc = SVC::new(
        KernelType::RBF { gamma: Gamma::Value(0.5) },
        5.0, 1e-3, 1000,
    )
    .unwrap()
    .with_random_state(42);
    svc.fit(&x, &y).unwrap();

    // Only rows with a non-zero alpha survive as support vectors.
    let support_vectors = svc.get_support_vectors().unwrap();
    let alphas = svc.get_alphas().unwrap();
    let labels = svc.get_support_vector_labels().unwrap();

    println!(
        "kept {} of {} rows as support vectors",
        support_vectors.nrows(),
        x.nrows()
    );
    println!("dual coefficients (alphas): {:?}", alphas);
    println!("support-vector labels (SMO's internal +/-1): {:?}", labels);
    println!("bias: {:?}", svc.get_bias());
    println!("SMO iterations actually run: {:?}", svc.get_actual_iterations());
    println!("resolved kernel: {:?}", svc.get_kernel());
}

get_support_vectors returns Option<&Array2<f64>>. get_alphas and get_support_vector_labels return Option<&Array1<f64>>. get_bias returns Option<f64>. All 4 are None before fit and Some after.

get_support_vector_labels is the one crack in the wall around the dual. It hands back the internal +1/-1 encoding, not the {0.0, 1.0} you handed to fit. So a -1.0 in that array denotes class 0.0. The label stays signed because predict folds it straight into the alpha_i * y_i coefficients. Threshold it yourself if you want the caller’s domain back.

get_actual_iterations reports how many SMO outer passes actually ran, always a number in [1, max_iter]. That count tells you whether the solver converged or hit the cap. A count equal to max_iter is a hint to raise the cap or loosen tol. get_kernel returns the resolved kernel. That is how you read back the concrete gamma that Gamma::Scale or Gamma::Auto produced.

Training can find no support vectors at all, for instance with single-class data where there is nothing to separate. In that case, fit returns Error::NotConverged instead of a degenerate all-zero model.

2.5.3. LinearSVC: hinge loss in the primal

LinearSVC solves the primal problem directly with minibatch stochastic gradient descent, minimizing hinge loss plus a regularization penalty. Construct it with LinearSVC::new(max_iter, learning_rate, penalty, fit_intercept, tol).

The labels are the same 0.0/1.0 that SVC takes. The story underneath is the same too: hinge loss needs signed targets, so fit remaps them to -1/+1 on entry. The difference is that nothing on LinearSVC’s public surface ever leaks that encoding back. The fitted state is a weight vector and a bias, and both live in feature space, not label space.

use rustyml::machine_learning::{LinearSVC, RegularizationType};
use ndarray::array;

fn main() {
    // The same 0.0 / 1.0 labels SVC takes. No remapping between the two.
    let x = array![
        [-5.0, 0.0], [-6.0, 0.0], [-7.0, 0.0], [-4.0, 0.0],
        [5.0, 0.0], [6.0, 0.0], [7.0, 0.0], [4.0, 0.0],
    ];
    let y = array![0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0];

    let mut model = LinearSVC::new(
        5000,                        // max_iter
        0.01,                        // learning_rate
        RegularizationType::L2(0.1), // penalty
        true,                        // fit_intercept
        1e-5,                        // tol
    )
    .unwrap()
    .with_random_state(42);
    model.fit(&x, &y).unwrap();

    let preds = model.predict(&x).unwrap();
    let scores = model.decision_function(&x).unwrap();
    println!("labels:  {:?}", preds);
    println!("scores:  {:?}", scores);
    println!("weights: {:?}", model.get_weights().unwrap());
    println!("bias:    {:?}", model.get_bias().unwrap());
    println!("stopped after {:?} epochs", model.get_actual_iterations());
}

fit shapes training with 2 details. It picks the minibatch size automatically, as clamp(n_samples / 10, 32, 512). You do not set that size. Each epoch shuffles the sample order before it slices the data into minibatches. This is why the seed matters (see below).

fit checks convergence on the root-mean-square change in the weights and bias between epochs. Training stops early once that change drops below tol. The default Loss::Hinge keeps a constant-size gradient near the margin, so a fixed learning rate often leaves the weights oscillating instead of settling. The example above shows this: it reaches the full 5000-epoch cap instead of stopping early. Loss::SquaredHinge has a gradient that shrinks to zero near the margin. Training with that loss usually stops well under max_iter, as the example below shows.

Left unchecked, a runaway learning_rate can push the weights to non-finite values. fit catches that and reports Error::NonFinite mid-training instead of returning garbage. Use a smaller learning_rate or stronger regularization to fix it.

Watch one threshold difference from SVC. LinearSVC::predict maps a decision value > 0.0 to class 1.0, and everything else, including exactly 0.0, to class 0.0. SVC maps >= 0.0 to 1.0 instead. Because both models share the same label domain, this is a real behavioral difference, not a bookkeeping detail. The 2 models break a zero score toward opposite classes.

decision_function here returns x * weights + bias. With fit_intercept = false, the bias stays exactly 0.0, and the score is a pure dot product.

Penalty and loss

RegularizationType is L1(lambda) or L2(lambda). L2, or ridge, applies a weights *= (1 - learning_rate * lambda) shrink at each step. This keeps all weights small but non-zero. L1, or lasso, applies a constant, sign-based subgradient pull that drives the weights of irrelevant features toward zero. Unlike LinearRegression and LogisticRegression, LinearSVC applies this pull as a plain subgradient step, not a proximal (soft-thresholding) step. So a weight rarely lands on exactly 0.0, though it still lands close to it.

If a feature carries no signal, a strong L1 penalty shrinks its weight closer to zero than it shrinks an informative feature’s weight. This helps when you suspect that many columns are noise. lambda = 0.0 is legal. It disables the penalty entirely.

2 builder methods extend the estimator past what new sets. with_loss switches between Loss::Hinge, the default, max(0, 1 - y*f(x)), and Loss::SquaredHinge, max(0, 1 - y*f(x))^2. Loss::SquaredHinge penalizes margin violations quadratically and is differentiable everywhere. That smoother objective lets some datasets converge more cleanly.

with_learning_rate_decay turns on an inverse-scaling schedule, where the effective rate at epoch t is learning_rate / (1 + decay * t). This lets SGD settle closer to the optimum, instead of hovering a fixed step away from it. It returns a Result, because the decay must be non-negative and finite.

use rustyml::machine_learning::{LinearSVC, Loss, RegularizationType};
use ndarray::array;

fn main() {
    let x = array![
        [-5.0, 0.0], [-6.0, 0.0], [-7.0, 0.0], [-4.0, 0.0],
        [5.0, 0.0], [6.0, 0.0], [7.0, 0.0], [4.0, 0.0],
    ];
    let y = array![0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0];

    let mut model = LinearSVC::new(10_000, 0.01, RegularizationType::L1(0.5), true, 1e-6)
        .unwrap()
        .with_loss(Loss::SquaredHinge)
        .with_learning_rate_decay(0.001)  // returns Result and checks the decay
        .unwrap()
        .with_random_state(0);
    model.fit(&x, &y).unwrap();

    println!("penalty: {:?}", model.get_penalty());
    println!("loss:    {:?}", model.get_loss());
    println!("preds:   {:?}", model.predict(&x).unwrap());
}

The getters mirror SVC’s: get_weights (Option<&Array1<f64>>), get_bias, get_penalty, get_loss, get_learning_rate, get_learning_rate_decay, get_tolerance, get_max_iterations, get_actual_iterations, and get_random_state. The weight vector length always matches n_features. So get_weights doubles as a feature-importance readout, once your columns share a common scale. LinearSVC is the linear-classifier cousin of Logistic Regression. The difference is the loss: hinge instead of log-loss. Hinge cares only about points near or across the margin, so it tends to give a boundary that ignores confidently-correct points entirely.

2.5.4. Reproducibility, persistence, and errors

Both estimators are non-deterministic by default. Both take a fixed seed through with_random_state(u64). For SVC, the seed drives the SMO working-set fallback, the randomized offset used when scanning for a second alpha to optimize. The same seed reproduces the same support vectors, bias, and predictions, bit for bit.

For LinearSVC, the seed drives the per-epoch minibatch shuffle. That fixes the exact sequence of gradient steps, and so the final weights too. Different seeds can land on different solutions, especially for LinearSVC on data where the shuffle order matters. Set the seed whenever you need runs to be comparable. See Reproducibility and Random Seeds for how this fits the crate-wide seeding story.

use rustyml::machine_learning::{Gamma, KernelType, SVC};
use ndarray::array;

fn main() {
    let x = array![
        [2.0, 2.0], [3.0, 2.0], [2.0, 3.0], [3.0, 3.0],
        [-2.0, -2.0], [-3.0, -2.0], [-2.0, -3.0], [-3.0, -3.0],
    ];
    let y = array![1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0];

    let train = || {
        let mut svc = SVC::new(KernelType::RBF { gamma: Gamma::Value(0.5) }, 5.0, 1e-3, 1000)
            .unwrap()
            .with_random_state(42);
        svc.fit(&x, &y).unwrap();
        svc.predict(&x).unwrap()
    };

    // Same seed, same data -> identical predictions.
    assert_eq!(train(), train());
    println!("same seed reproduces the model exactly");
}

Both models implement save_to_path and load_from_path. These methods serialize the entire fitted state to a compact postcard binary blob. That state is the support vectors and alphas for SVC, or the weights and bias for LinearSVC, plus every hyperparameter.

You can choose any file extension. The format stays binary regardless of the name. A round-trip reproduces predictions and decision scores exactly. This is the shallow end of Model Persistence in Depth.

use rustyml::machine_learning::{Gamma, KernelType, SVC};
use ndarray::array;
use std::fs::remove_file;

fn main() {
    let x = array![
        [2.0, 2.0], [3.0, 2.0], [2.0, 3.0], [3.0, 3.0],
        [-2.0, -2.0], [-3.0, -2.0], [-2.0, -3.0], [-3.0, -3.0],
    ];
    let y = array![1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0];

    let mut svc = SVC::new(KernelType::RBF { gamma: Gamma::Value(0.5) }, 5.0, 1e-3, 1000)
        .unwrap()
        .with_random_state(42);
    svc.fit(&x, &y).unwrap();

    svc.save_to_path("svc_model.bin").unwrap();      // postcard binary
    let loaded = SVC::load_from_path("svc_model.bin").unwrap();

    assert_eq!(svc.predict(&x).unwrap(), loaded.predict(&x).unwrap());
    println!("loaded model reproduces the original's predictions");

    remove_file("svc_model.bin").unwrap();
}

Every fallible entry point returns the crate’s Error (see Error Handling). The variants you will meet in practice:

The label domain still trips people, just not in the direction the SMO literature suggests. Both models want 0.0/1.0, so a y copied from a textbook derivation or from another library’s signed convention causes an InvalidInput at fit. Neither model converts it silently. Encode your labels into {0.0, 1.0} before training. After that, the only +1/-1 values you will meet are whatever get_support_vector_labels hands back.