5.2. Classification Metrics

Classification metrics live in rustyml::metrics, next to the regression metrics and the clustering metrics. Every function and type on this page is re-exported flat from that module, so use rustyml::metrics::{accuracy, ConfusionMatrix, roc_auc}; is all the import you need. The interface splits along a single axis that confuses people coming from scikit-learn. Some entry points take hard class labels, some take a decision threshold, and some take raw probabilities or scores. Getting the label representation wrong is the most common mistake with this module.

5.2.1. The module’s conventions, and how it signals errors

The estimators in Chapter 2 return Result<_, Error>. The functions in this module do not. They panic on a precondition violation instead. This is deliberate. The classification functions here are pure array -> scalar code that pulls in only ndarray and ahash. On a shape mismatch, the module panics the same way ndarray does, instead of returning the crate’s error type.

Every function checks 2 preconditions: equal length and non-empty input. A violation panics with dimension mismatch: expected N, found M or input is empty: .... This wording matches the crate’s Error variants on purpose. A metric is not the place to recover from bad input. If y_true and y_pred have different lengths, the bug is upstream. The panic surfaces it at the call site, instead of returning a misleading 0.0.

Arguments are always (y_true, y_pred), ground truth first. Order does not matter for the symmetric metrics, such as accuracy. For ConfusionMatrix::new and roc_auc, argument order decides which array counts as truth. Keep y_true first as a habit.

The module uses 3 label representations, listed in the table below. The compiler enforces them, but the panic messages do not explain the design.

5.2.2. Accuracy, and why it lies under imbalance

The free function accuracy(&y_true, &y_pred) returns the fraction of exactly matching labels. It compares each pair within f64::EPSILON. This makes it built for discrete class labels stored as f64, such as 0.0, 1.0, and 2.0. It works the same way for binary and multi-class problems. The comparison is symmetric, so swapping the arguments changes nothing.

Do not feed accuracy probabilities. It does no thresholding, so 0.87 and 1.0 count as a mismatch. Threshold probabilities yourself first. ConfusionMatrix does not do this for you either. It takes only hard 0.0/1.0 labels and panics on anything else.

Accuracy has a real weakness: it gives a poor summary of imbalanced data. The module offers 2 more honest metrics for that case. Consider a screening problem with 100 samples and only 5 positives. A model that predicts the negative class for every sample scores 95% accuracy. It also catches 0 of the cases that matter.

use rustyml::metrics::{accuracy, ConfusionMatrix};
use ndarray::Array1;

fn main() {
    // Imbalanced problem: 100 samples, only 5 positives.
    let mut truth = vec![0.0f64; 100];
    for t in truth.iter_mut().take(5) {
        *t = 1.0;
    }
    let y_true = Array1::from(truth);
    // A model that always predicts the majority (negative) class.
    let y_pred = Array1::from(vec![0.0f64; 100]);

    println!("accuracy:          {:.3}", accuracy(&y_true, &y_pred)); // ~0.95
    let cm = ConfusionMatrix::new(&y_true, &y_pred);
    println!("recall:            {:.3}", cm.recall());            // 0.0 (catches nothing)
    println!("balanced accuracy: {:.3}", cm.balanced_accuracy()); // 0.5 (chance level)
    println!("MCC:               {:.3}", cm.mcc());               // 0.0 (no correlation)
}

balanced_accuracy averages recall and specificity, so a majority-class predictor is pinned at 0.5 no matter how skewed the classes are. The Matthews correlation coefficient (mcc) goes further. It folds all 4 cells of the confusion matrix into a single correlation in [-1, 1]. It reads 0 for this degenerate model, because there is no association between prediction and truth at all. These 2 numbers show whether 95% accuracy is actually good.

5.2.3. The binary confusion matrix

ConfusionMatrix is a small Copy struct holding 4 counts: true positives, false positives, true negatives, and false negatives. Every scalar it exposes is derived from those 4 numbers. ConfusionMatrix::new(&y_true, &y_pred) takes 2 f64 arrays of hard labels. Every entry must be exactly 0.0 or 1.0. Nothing is binarized. A probability, an unbounded decision-function score, or a -1/+1 label makes the constructor panic instead of converting the value silently.

scikit-learn’s confusion_matrix also requires hard labels, for the same reason. Threshold the scores before the call. This also forces a clear choice of where the cutoff sits.

An earlier version of ConfusionMatrix::new binarized both arguments at a hardcoded 0.5. This silently corrupted a probabilistic ground truth. It also cut an unbounded score at a meaningless point, and it counted NaN as negative. Some code may depend on that old behavior. Add an explicit mapv(|p| if p >= 0.5 { 1.0 } else { 0.0 }) before the call to restore it.

For a different label pair, such as the -1/+1 an outside margin classifier emits, use ConfusionMatrix::new_with_labels(&y_true, &y_pred, negative_label, positive_label). This is the binary form of scikit-learn’s labels=[neg, pos]. RustyML’s own SVC and LinearSVC predict 0.0/1.0, so plain new already fits them. The 2 arguments accept independent storage types. Mixing an owned array with a view is fine, so ConfusionMatrix::new(&y_test, &model.predict(&x)?.view()) compiles.

use ndarray::array;
use rustyml::metrics::ConfusionMatrix;

fn main() {
    // Hard 0/1 labels: 5 real positives, 3 real negatives.
    let y_true = array![1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0];
    let y_pred = array![1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0];
    let cm = ConfusionMatrix::new(&y_true, &y_pred);

    let (tp, fp, tn, fn_) = cm.get_counts();
    println!("TP={tp} FP={fp} TN={tn} FN={fn_}"); // TP=3 FP=1 TN=2 FN=2
    println!("accuracy    {:.3}", cm.accuracy());
    println!("precision   {:.3}", cm.precision());   // 3/4
    println!("recall      {:.3}", cm.recall());      // 3/5
    println!("specificity {:.3}", cm.specificity()); // 2/3
    println!("f1          {:.3}", cm.f1_score());
    print!("{}", cm.summary());
}

get_counts() returns the raw (tp, fp, tn, fn) tuple. The derived accessors each return an f64: accuracy, error_rate (exactly 1 - accuracy), precision, recall, specificity, f1_score, mcc, and balanced_accuracy. Each metric has its own convention for a zero denominator, chosen on purpose:

• precision and recall return 0.0 when the denominator is empty (no positive predictions, or no actual positives).
• specificity returns 1.0 when there are no actual negatives. This 0/0 case counts as nothing to get wrong.
• mcc returns 0.0 when any marginal sum is zero, because the coefficient is undefined there.

These conventions match the per-class conventions in the multi-class matrix, so the two stay consistent.

summary() renders the matrix and all 8 derived metrics as a formatted table, with each metric to 4 decimal places. It is meant for logs and notebooks. Do not parse the string. Treat it as output for a human reader.

Confusion Matrix:
+-----------------+--------------------+--------------------+
|                 | Predicted Positive | Predicted Negative |
+-----------------+--------------------+--------------------+
| Actual Positive | TP: 3              | FN: 2              |
| Actual Negative | FP: 1              | TN: 2              |
+-----------------+--------------------+--------------------+

Performance Metrics:
- Accuracy:          0.6250
- Balanced Accuracy: 0.6333
...

5.2.4. Precision, recall, and the tradeoff

Precision and recall answer different questions, and choosing which one to optimize is a domain decision, not a statistical one. Precision (TP / (TP + FP)) measures how many of the flagged samples are real. Recall (TP / (TP + FN)) measures how many of the real cases the model catches. The 2 metrics pull against each other, because both depend on the decision threshold. A lower threshold flags more samples: recall rises, because it misses fewer real cases, but precision falls, because more flags are false alarms. A higher threshold reverses the tradeoff.

In medical screening, a false negative can be fatal. You tune for high recall there and accept the false alarms that a follow-up test filters out. In fraud review, every flag costs an analyst’s time and annoys a legitimate customer. Precision matters more there, so you tolerate missing some fraud rather than drowning the team in false positives. No threshold is correct in the abstract. Only the threshold that matches the cost of the 2 error types is right.

f1_score combines precision and recall into their harmonic mean, 2PR / (P + R). The choice of the harmonic mean over the arithmetic mean is the point. The arithmetic mean of precision 1.0 and recall 0.0 is a flattering 0.5. The harmonic mean is 0.0 there, because it is dominated by its smaller input. F1 rewards a classifier only when both precision and recall are high, which fits a classifier that must avoid both false alarms and missed detections. When precision and recall are both 0, the crate returns 0.0 instead of dividing by 0.

F1 weights the 2 errors equally. When the errors are not equally costly, report precision and recall separately. Pick the threshold deliberately, instead of chasing the highest F1 score.

Precision, recall, and F1 are all methods on the confusion matrix: cm.precision(), cm.recall(), cm.f1_score(). There are no free precision_score(y_true, y_pred) functions, the way scikit-learn has them. The matrix walks the data once, then answers each question from 4 integers. A per-metric free function would recount the whole array on every call instead. Porting a scikit-learn script means collapsing a run of *_score calls into a single ConfusionMatrix, then reading the numbers off it.

Two more single-number summaries sit alongside them, also as methods on the matrix. cm.balanced_accuracy() is the mean of the 2 classes’ recalls. It is the honest counterweight to accuracy’s behavior under imbalance. A 9-to-1 classifier that reads 0.9 accuracy reads 0.5 here, the score a coin flip deserves. cm.mcc() is the Matthews correlation coefficient, a correlation between the true and the predicted labeling that runs from -1, through 0 at chance, to +1. It climbs only when all 4 cells of the matrix look good, which makes it the hardest of these numbers to inflate.

use ndarray::array;
use rustyml::metrics::ConfusionMatrix;

fn main() {
    let y_true = array![0.0, 1.0, 1.0, 0.0, 1.0, 1.0, 0.0, 0.0];
    let y_pred = array![0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0];

    // One pass builds the matrix. Every metric after that just reads the counts.
    let cm = ConfusionMatrix::new(&y_true, &y_pred);
    println!("precision          {:.3}", cm.precision());
    println!("recall             {:.3}", cm.recall());
    println!("f1                 {:.3}", cm.f1_score());
    println!("balanced accuracy  {:.3}", cm.balanced_accuracy());
    println!("mcc                {:.3}", cm.mcc());

    // TP=3, FP=1, TN=3, FN=1: precision = 3/4 and recall = 3/4, so F1 is 3/4 too.
    let (tp, fp, tn, fn_) = cm.get_counts();
    assert_eq!((tp, fp, tn, fn_), (3, 1, 3, 1));
    assert!((cm.f1_score() - 0.75).abs() < 1e-12);
}

5.2.5. Threshold-free evaluation: ROC AUC and PR curves

Everything above uses a single fixed threshold. Sometimes you want to evaluate the ranking a model produces, independent of where you eventually cut it. ROC AUC and the precision-recall curve give you that view. These functions take bool labels (true for the positive class) and f64 scores. The scores can be probabilities or any monotone decision value.

roc_auc(&labels, &scores) returns the area under the ROC curve, computed with the Mann-Whitney U statistic. This gives it a clean interpretation: it is the probability that a randomly chosen positive sample scores above a randomly chosen negative one. A score of 1.0 is a perfect ranking, and a score of 0.5 is a coin flip. A score below 0.5 means the model ranks samples backwards. Tied scores receive their average rank. A model whose scores are all equal therefore lands at exactly 0.5, instead of a result that depends on array order.

AUC does not need a threshold, and it does not depend on class balance. Its weakness shows on heavily imbalanced data. A large negative class can make the curve look good there, while precision at any usable threshold stays poor. This is why you also check average precision.

average_precision(&labels, &scores) is the area under the precision-recall curve, computed as the precision-weighted sum of recall increments. On imbalanced problems, it gives a more honest headline number, because its baseline is the positive rate, not a fixed 0.5. It does not get an easy boost from an easy negative class.

use ndarray::array;
use rustyml::metrics::{average_precision, precision_recall_curve, roc_auc, roc_curve};

fn main() {
    // Scores for 6 samples. `true` marks the positive class.
    let labels = array![true, false, true, false, true, false];
    let scores = array![0.95, 0.4, 0.7, 0.3, 0.6, 0.2];

    println!("ROC AUC:      {:.3}", roc_auc(&labels, &scores));
    println!("avg precision {:.3}", average_precision(&labels, &scores));

    // Full sweep: (fpr, tpr, thresholds), all equal length, starting at the (0,0) origin.
    let (fpr, tpr, thresholds) = roc_curve(&labels, &scores);
    println!("ROC points: {}", fpr.len());
    println!("first tpr={:.2} fpr={:.2}", tpr[0], fpr[0]);
    assert_eq!(thresholds[0], f64::INFINITY); // the origin classifies nothing as positive

    // precision/recall carry 1 extra closing point beyond the thresholds,
    // and run in ascending-threshold / descending-recall order.
    let (precision, recall, pr_thresholds) = precision_recall_curve(&labels, &scores);
    assert_eq!(precision.len(), pr_thresholds.len() + 1);
    assert_eq!(precision[precision.len() - 1], 1.0); // closing point: recall 0, precision 1
    assert_eq!(recall[recall.len() - 1], 0.0);
}

roc_curve returns (fpr, tpr, thresholds) as 3 equal-length Array1<f64> arrays. There is 1 point per distinct score, in decreasing order, prefixed with the (0, 0) origin. The origin’s threshold is f64::INFINITY, the only value that classifies nothing as positive. Unlike a finite max_score + 1.0, infinity stays distinguishable from the top real threshold even when scores are large (1e17 + 1.0 == 1e17). This matches scikit-learn.

One difference remains, and it is deliberate. RustyML always returns the full sweep, so it keeps every collinear interior point. scikit-learn’s default, drop_intermediate=True, discards those points instead. RustyML’s point count can therefore be larger, while the curve itself, and roc_auc, stay identical. Integrating this curve with the trapezoid rule reproduces roc_auc exactly.

precision_recall_curve returns (precision, recall, thresholds) in the opposite order. Thresholds ascend, so recall descends along the arrays. The final (precision = 1, recall = 0) closing point sits at the low-recall end, where it belongs. precision and recall are therefore 1 element longer than thresholds. Watch for that off-by-1 difference when you zip the arrays. scikit-learn makes the same ordering distinction between the 2 curve functions, and the output matches element for element.

An earlier version appended the closing point at the high-recall end. This left recall monotone in neither direction. Code that depends on a particular orientation needs a check against the current behavior.

All 4 ranking functions reject NaN scores with a panic (scores must not contain NaN). This is not pedantry. f64::total_cmp sorts NaN as the most extreme value, so a stray NaN would silently count as the most confident prediction and corrupt the ranking. roc_auc and roc_curve also require at least 1 positive and 1 negative label. average_precision and precision_recall_curve require at least 1 positive label. A single-class input gives a degenerate curve, so it panics instead of returning a meaningless number.

These functions are binary-only. The crate has no one-vs-rest or macro-averaged multi-class AUC. For per-class ROC on a multi-class problem, binarize each class, then call roc_auc once per class.

5.2.6. The multi-class confusion matrix and averaging

For more than 2 classes, use MulticlassConfusionMatrix. It takes usize labels for both truth and predictions. Its class axis is the sorted union of every label seen in either input. A class that appears only in the predictions (a hallucinated class) still gets a row and a column. matrix() exposes the full K x K count grid as an ArrayView2<usize>, with rows indexed by true class and columns by predicted class. labels() gives the label at each index, and n_classes() gives the dimension.

use ndarray::array;
use rustyml::metrics::{Average, MulticlassConfusionMatrix};

fn main() {
    let y_true = array![0usize, 1, 2, 2, 1, 0, 2];
    let y_pred = array![0usize, 2, 2, 2, 1, 0, 1];
    let cm = MulticlassConfusionMatrix::new(&y_true, &y_pred);

    println!("classes:  {:?}", cm.labels());            // [0, 1, 2]
    println!("support:  {:?}", cm.support());           // true-sample count per class
    println!("accuracy: {:.3}", cm.accuracy());
    println!("recall:   {:?}", cm.per_class_recall());

    // Aggregation strategy is an explicit argument, not a hidden default.
    println!("macro F1:    {:.3}", cm.f1(Average::Macro));
    println!("micro F1:    {:.3}", cm.f1(Average::Micro));
    println!("weighted F1: {:.3}", cm.f1(Average::Weighted));

    // For the per-class numbers, read per_class_*. They return a Vec<f64> in label order.
    println!("per-class F1: {:?}", cm.per_class_f1());

    print!("{}", cm.summary());              // count grid + per-class report
}

The per-class views, per_class_precision, per_class_recall, and per_class_f1, return a Vec<f64> in label order. They use the same zero-denominator convention as the binary matrix: 0.0 for a class that is never predicted or never true. support() returns the number of ground-truth samples per class. The weighted average uses that count.

The aggregated precision, recall, and f1 methods each take an Average argument. This is where the macro, micro, and weighted distinction matters:

• Average::Macro is the unweighted mean of the per-class scores. Every class counts equally, regardless of its size, so a rare but important class is not drowned out by a common one. Report this on imbalanced multi-class problems.
• Average::Weighted weights each per-class score by that class’s support. It measures performance on a typical sample, and it tracks accuracy more closely than macro averaging does.
• Average::Micro pools the counts across all classes before it computes the metric. This type supports only single-label classification: exactly 1 predicted class per sample. There, micro precision, micro recall, and micro F1 collapse to the same value: accuracy. The implementation returns accuracy directly for Micro. Micro F1 and accuracy are the same number by construction in a single-label setting. A different reported value is a mistake.

Those 3 variants are the whole set. There is no counterpart to scikit-learn’s average="binary". For a single class’s one-vs-rest number, index into per_class_precision, per_class_recall, or per_class_f1 by label position. cm.labels() gives that exact order.

summary() prints the count grid, followed by a scikit-learn-style per-class report with a macro avg and weighted avg footer. It sizes the table to the labels and counts present. This is also the type’s only reporting entry point. There is no free-function version.

5.2.7. Probability-based and agreement metrics

3 more functions complete the module. They take probabilities or paired labelings, not a confusion matrix.

log_loss(&y_true, &y_prob) computes multi-class cross-entropy: y_true holds each sample’s true class index as a usize. y_prob is an Array2<f64> with 1 row per sample and 1 column per class. Only the probability assigned to the true class contributes to the score. Each row is renormalized to sum to 1 before scoring, so a row that is not already a normalized distribution is still handled consistently. The selected probability is then clamped away from 0 and 1, so the logarithm stays finite. A confidently wrong prediction therefore gets a large but finite penalty instead of +inf.

Lower is better.

top_k_accuracy(&y_true, &y_prob, k) counts a sample as correct if its true class is among the k highest-probability classes. A class ties into the top-k set if fewer than k classes are strictly more probable. A boundary tie therefore counts in the sample’s favor. Report this metric when a correct answer within the top 5 classes is an acceptable bar. It panics if k == 0, if a label is out of range for the probability columns, or if y_prob contains NaN. A NaN true-class probability would defeat the p > true_prob comparison and miscount the sample as a hit.

cohen_kappa(&y_true, &y_pred) measures agreement between 2 labelings, corrected for chance. The formula is (p_o - p_e) / (1 - p_e). Here, p_o is the observed agreement (accuracy), and p_e is the agreement expected from the marginal label frequencies alone. It runs from -1, through 0 at chance, to 1 at perfect agreement. It shows whether the accuracy is actually better than a model that guesses in proportion to the class frequencies. That is a sharper question than raw accuracy answers on skewed data.

use ndarray::array;
use rustyml::metrics::{cohen_kappa, log_loss, top_k_accuracy};

fn main() {
    let y_true = array![0usize, 1, 2];
    // Row i = predicted class distribution for sample i.
    let y_prob = array![
        [0.8, 0.1, 0.1],
        [0.1, 0.7, 0.2],
        [0.2, 0.2, 0.6],
    ];

    println!("log loss:  {:.3}", log_loss(&y_true, &y_prob));    // lower is better
    println!("top-2 acc: {:.3}", top_k_accuracy(&y_true, &y_prob, 2));

    // cohen_kappa compares 2 hard labelings, not probabilities.
    let y_pred = array![0usize, 1, 1];
    println!("kappa:     {:.3}", cohen_kappa(&y_true, &y_pred));
}

5.2.8. End-to-end: evaluating a logistic regression classifier

This section puts the metrics together with the logistic regression model from Chapter 2. LogisticRegression::predict already returns hard {0.0, 1.0} labels as an Array1<f64>, exactly what ConfusionMatrix and accuracy want. The only conversion left is the bool label array that roc_auc needs, alongside the predict_proba scores.

use ndarray::{array, Array1};
use rustyml::machine_learning::LogisticRegression;
use rustyml::metrics::{accuracy, roc_auc, ConfusionMatrix};

fn main() {
    // 2 well-separated clusters in 2-D feature space.
    let x_train = array![
        [1.0, 1.0], [1.5, 2.0], [2.0, 1.5],
        [6.0, 5.0], [5.5, 6.5], [6.5, 5.5]
    ];
    let y_train = array![0.0, 0.0, 0.0, 1.0, 1.0, 1.0];

    let mut model = LogisticRegression::new(true, 0.5, 500, 1e-6).unwrap();
    model.fit(&x_train, &y_train).unwrap();

    // Held-out test set with known labels.
    let x_test = array![[1.2, 1.4], [2.2, 1.8], [5.8, 6.0], [6.2, 5.2]];
    let y_test = array![0.0, 0.0, 1.0, 1.0];

    // predict -> hard {0.0, 1.0} labels, ready for the label-based metrics.
    let y_pred: Array1<f64> = model.predict(&x_test).unwrap();

    println!("accuracy: {:.3}", accuracy(&y_test, &y_pred));

    let cm = ConfusionMatrix::new(&y_test, &y_pred);
    print!("{}", cm.summary());

    // Ranking quality from the raw probabilities: bool labels + f64 scores.
    let scores = model.predict_proba(&x_test).unwrap();
    let labels = y_test.mapv(|v| v >= 0.5);
    println!("ROC AUC: {:.3}", roc_auc(&labels, &scores));
}

This data is cleanly separable, so the model classifies the test set perfectly, and every metric reads 1.0. This is a useful check that the pipeline works, but the interesting decisions happen elsewhere. To study the precision/recall tradeoff on a real classifier, feed the predict_proba output into roc_curve and precision_recall_curve from section 5.2.5. Inspect the whole sweep, instead of committing to the model’s built-in 0.5 threshold.

This is the difference between reporting a single accuracy number and reporting the chosen operating point together with the error it accepts. The second report is the one that survives review. When labels arrive as strings or categories rather than 0.0/1.0, convert them first with label encoding. This gives them the f64 or usize form these metrics expect.