Skip to main content

Advanced Metrics for Classification and Regression

Understanding advanced metrics for classification and regression is essential for evaluating and interpreting models, especially when basic metrics like accuracy or mean squared error are insufficient. This article will explore several advanced metrics, their mathematical foundations, and practical applications, focusing on how these metrics are used to assess the performance of models in a general context.

1. Introduction to Advanced Metrics​

1.1 Why Advanced Metrics?​

Basic metrics like accuracy and mean squared error provide a general sense of model performance, but they can be misleading or insufficient in certain situations. Advanced metrics offer a more nuanced view, helping to identify specific strengths and weaknesses of a model or approach.

1.2 When to Use Advanced Metrics​

  • Imbalanced Data: In cases where the data is imbalanced, metrics like accuracy can be misleading, and advanced metrics like Precision-Recall AUC or Matthews Correlation Coefficient (MCC) become more relevant.
  • Specific Performance Needs: Depending on the application, different metrics may prioritize different aspects of performance, such as sensitivity to false positives or false negatives.

2. Advanced Metrics for Classification​

2.1 Matthews Correlation Coefficient (MCC)​

Matthews Correlation Coefficient (MCC) is a metric used for evaluating the quality of binary classifications. It takes into account true and false positives and negatives and is generally regarded as a balanced measure, even for imbalanced data.

MCC=TP×TN−FP×FN(TP+FP)(TP+FN)(TN+FP)(TN+FN)\text{MCC} = \frac{TP \times TN - FP \times FN}{\sqrt{(TP + FP)(TP + FN)(TN + FP)(TN + FN)}}

Where:

  • TP: True Positives
  • TN: True Negatives
  • FP: False Positives
  • FN: False Negatives

MCC returns a value between -1 and +1:

  • +1: Perfect prediction
  • 0: No better than random prediction
  • -1: Total disagreement between prediction and reality

2.2 Precision-Recall AUC​

Precision-Recall AUC is used to evaluate models on datasets with imbalanced classes. Unlike ROC-AUC, which plots True Positive Rate (TPR) vs. False Positive Rate (FPR), Precision-Recall curves plot Precision vs. Recall.

  • Precision: Measures the accuracy of positive predictions.
  • Recall: Measures the ability to find all positive instances.

The term “PR AUC” is ambiguous. scikit-learn's average_precision_score computes the non-interpolated weighted mean of precisions over recall increments, while trapezoidal auc(recall, precision) linearly interpolates between operating points and can be optimistic. State which summary you report and compare it with the positive-class prevalence baseline.

2.3 F-beta Score​

The F-beta Score is a generalization of the F1-score that allows you to weigh recall more than precision, or vice versa, depending on the value of β\beta:

Fβ=(1+β2)×Precision×Recall(β2×Precision)+Recall\text{F}_\beta = (1 + \beta^2) \times \frac{\text{Precision} \times \text{Recall}}{(\beta^2 \times \text{Precision}) + \text{Recall}}

Where:

  • β=1\beta = 1: F1-Score, balanced between Precision and Recall.
  • β>1\beta > 1: More weight to Recall.
  • β<1\beta < 1: More weight to Precision.

This metric is useful when you need to focus on reducing false negatives (higher recall) or false positives (higher precision).

2.4 Balanced Accuracy​

Balanced Accuracy is another metric that adjusts for imbalanced datasets. It is the average of sensitivity (True Positive Rate) and specificity (True Negative Rate):

Balanced Accuracy=12(TPTP+FN+TNTN+FP)\text{Balanced Accuracy} = \frac{1}{2} \left(\frac{TP}{TP + FN} + \frac{TN}{TN + FP}\right)

This metric is particularly useful when the class distribution is skewed, providing a better indicator of performance than raw accuracy.

3. Advanced Metrics for Regression​

3.1 Mean Absolute Percentage Error (MAPE)​

Mean Absolute Percentage Error (MAPE) measures the accuracy of a forecasted value compared to the actual value, expressed as a percentage:

MAPE=1n∑i=1n∣yi−y^iyi∣×100\text{MAPE} = \frac{1}{n} \sum_{i=1}^{n} \left| \frac{y_i - \hat{y}_i}{y_i} \right| \times 100

Where:

  • yiy_i is the actual value.
  • y^i\hat{y}_i is the predicted value.

MAPE can be useful when relative error is meaningful, but it is undefined at a zero target and unstable near zero. scikit-learn returns a very large finite value rather than infinity when y_true is zero. It also weights equal absolute errors differently according to target magnitude, so confirm that this matches the decision cost.

3.2 Mean Squared Logarithmic Error (MSLE)​

Mean Squared Logarithmic Error (MSLE) is similar to Mean Squared Error (MSE) but applies logarithmic transformation, making it useful when you care more about the relative differences rather than the absolute differences:

MSLE=1n∑i=1n(log⁡(1+yi)−log⁡(1+y^i))2\text{MSLE} = \frac{1}{n} \sum_{i=1}^{n} \left( \log(1 + y_i) - \log(1 + \hat{y}_i) \right)^2

MSLE requires non-negative targets and predictions. Because it compares log1p values, it emphasizes relative differences more than ordinary MSE; it is not suitable when negative values are meaningful.

3.3 Symmetric Mean Absolute Percentage Error (SMAPE)​

Symmetric Mean Absolute Percentage Error (SMAPE) is a variation of MAPE that symmetrically treats the absolute differences between actual and predicted values:

SMAPE=1n∑i=1n∣y^i−yi∣(∣y^i∣+∣yi∣)/2×100\text{SMAPE} = \frac{1}{n} \sum_{i=1}^{n} \frac{\left| \hat{y}_i - y_i \right|}{(\left|\hat{y}_i \right| + \left| y_i \right|) / 2} \times 100

Under this displayed definition, each term is bounded by 200% when the denominator is nonzero. The value is undefined when both actual and predicted values are zero unless the implementation defines an explicit convention. Other publications use a different factor or denominator, so document the exact formula.

3.4 Relative Squared Error (RSE)​

Relative Squared Error (RSE) compares the sum of squared errors of the prediction to that of a simple baseline model (typically, predicting the mean):

RSE=∑i=1n(yi−y^i)2∑i=1n(yi−yˉ)2\text{RSE} = \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2}

Where yˉ\bar{y} is the mean of the actual values.

RSE provides a measure of how much worse or better the model is compared to the baseline model.

4. Practical Examples Using Scikit-learn​

4.1 Calculating Advanced Classification Metrics​

Let’s calculate some advanced classification metrics using Scikit-learn:

from sklearn.metrics import (
    average_precision_score,
    balanced_accuracy_score,
    fbeta_score,
    matthews_corrcoef,
)

y_test = [0, 0, 0, 0, 1, 1]
y_prob = [0.05, 0.20, 0.40, 0.65, 0.70, 0.90]
y_pred = [int(score >= 0.5) for score in y_prob]

mcc = matthews_corrcoef(y_test, y_pred)
f2_score = fbeta_score(y_test, y_pred, beta=2)
balanced_accuracy = balanced_accuracy_score(y_test, y_pred)

print(f"MCC: {mcc:.2f}")
print(f"F2-Score: {f2_score:.2f}")
print(f"Balanced Accuracy: {balanced_accuracy:.2f}")

average_precision = average_precision_score(y_test, y_prob)
print(f"Average precision: {average_precision:.2f}")

4.2 Calculating Advanced Regression Metrics​

Let’s calculate some advanced regression metrics using Scikit-learn:

from sklearn.metrics import mean_absolute_percentage_error, mean_squared_log_error

y_test = [1.0, 2.0, 4.0, 8.0]
y_pred = [1.2, 1.8, 5.0, 7.5]

mape = mean_absolute_percentage_error(y_test, y_pred)
msle = mean_squared_log_error(y_test, y_pred)

print(f"MAPE: {100 * mape:.2f}%")
print(f"MSLE: {msle:.2f}")

5. Best Practices and Considerations​

5.1 Choosing the Right Metric​

Selecting the appropriate metric is crucial and should be based on the specific goals of your analysis:

  • Use MCC or balanced accuracy when performance across both binary classes should influence one summary, but also show the underlying confusion matrix.
  • Use MAPE or a precisely defined SMAPE only when their behavior near zero and their implicit weighting match the application.
  • Use a precision-recall curve or average precision to study ranking with a rare positive class. Choose the deployed threshold separately from explicit false-positive and false-negative costs.
  • Evaluate calibration with proper scoring rules such as log loss or Brier score when probability values themselves drive decisions.

5.2 Interpreting Advanced Metrics​

Advanced metrics often require more careful interpretation than simpler metrics like accuracy or MSE. For example, MCC is particularly useful in binary classification with imbalanced data, but its interpretation is less straightforward than accuracy.

6. Conclusion​

6.1 Recap of Key Concepts​

Advanced metrics provide different views of model performance. Understanding their domains, baselines, averaging rules, and failure modes is crucial for tuning models and making informed decisions.

6.2 Next Steps​

In future articles, we will explore model evaluation in imbalanced datasets and how to interpret and visualize these advanced metrics effectively.

Advanced metrics in classification and regression provide a more nuanced and comprehensive evaluation of model performance. By mastering these metrics, you can better understand the strengths and weaknesses of your models, leading to more informed decisions and better outcomes in your data analysis projects.