Skip to main content

Linear Regression Common Mistakes and Best Practices

1. Confusing prediction assumptions with inference assumptions​

OLS minimizes squared residuals without requiring normally distributed errors. Normality, independence, and constant conditional variance matter for familiar finite-sample tests and standard errors, not for computing coefficients. Prediction still depends on the conditional mean being represented well enough in the deployment population.

Inspect residuals against fitted values and important features. For repeated, grouped, or temporal data, use a split and an uncertainty method that respect dependence. Never infer causation from a fitted coefficient without an identification argument and an appropriate study design.

2. Learning from the test set​

Imputation, scaling, polynomial expansion, feature selection, and target-informed filtering must be learned inside the training data—and inside each training fold during tuning. Keep the final test set isolated until choices are complete.

from sklearn.datasets import load_diabetes
from sklearn.linear_model import Ridge
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)
pipeline = make_pipeline(StandardScaler(), Ridge())
search = GridSearchCV(
    pipeline,
    {"ridge__alpha": [0.1, 1.0, 10.0, 100.0]},
    scoring="neg_root_mean_squared_error",
    cv=5,
)
search.fit(X_train, y_train)
test_r2 = search.score(X_test, y_test)
print(search.best_params_)
print(f"Test R²: {test_r2:.3f}")

3. Treating R2R^2 as universal proof of fit​

Test R2R^2 is relative to a mean-prediction baseline and may be negative. It does not express error in target units, establish causality, or reveal which observations fail. Report MAE or RMSE, residual summaries, uncertainty where needed, and performance across relevant subgroups or time periods.

4. Reading coefficients as feature importance​

Raw coefficient magnitudes reflect feature units. Correlated predictors can make coefficients unstable, signs can change when variables are added, and omitted confounders can distort associations. Examine units, uncertainty, correlation, and stability across resamples. Ridge stabilizes prediction under collinearity but changes the estimand; Lasso can select one of several correlated features arbitrarily.

5. Ignoring leverage, outliers, and extrapolation​

Large residuals and high-leverage points are different. Investigate data quality and influence rather than deleting observations automatically. OLS emphasizes large residuals through squaring. Robust regression or a transformed target may be justified, but the choice must reflect the data-generating process.

A linear model extends indefinitely beyond the observed feature range. Mark such predictions as extrapolations and constrain deployment if the linear relationship is not defensible there.

Checklist​

  • Define the prediction population, split strategy, baseline, and metric before fitting.
  • Put every learned transform inside the validation pipeline.
  • Diagnose residual structure and influential observations.
  • Separate predictive accuracy, coefficient interpretation, and causal claims.
  • Record feature units and reject unsupported extrapolation.