Skip to main content

Spatial Cross-Validation and Leakage

Random cross-validation assumes that held-out observations represent the future prediction task. Nearby train and test locations often share environmental context, acquisition artifacts, repeated entities, or labels, so random folds can measure interpolation beside known samples when deployment requires prediction into a new region.

Prerequisites​

Complete spatial regression. You should know the difference between tuning data, a final test, model residual dependence, and geographic extrapolation.

Learning goals​

  • Define the deployment unit, distance, time, and extrapolation target before splitting.
  • Keep all observations from one spatial group in a single fold.
  • Compare random and spatially separated validation without declaring either universally correct.
  • Fit preprocessing, feature engineering, imputation, and model selection inside each training fold.

Split the observation process, not just rows​

The synthetic target is a smooth, known field on a 12×1212\times12 grid. A three-neighbor regressor uses coordinates only. Four shuffled folds place nearly every test point beside training points. Four quadrant folds hold out one contiguous 6×66\times6 region at a time, increasing train–test separation and requiring interpolation inward from its borders.

GroupKFold guarantees that each declared group appears in one test fold and not in that fold's training set. It does not create a geographic buffer, balance environmental covariates, respect time automatically, or prove independence. Groups must be defined from the real deployment question before model comparison. A separate final region remains necessary if the same spatial folds tune hyperparameters.

Worked example​

import numpy as np
from scipy.spatial.distance import cdist
from sklearn.model_selection import GroupKFold, KFold, cross_val_score
from sklearn.neighbors import KNeighborsRegressor

coordinates = np.array([(row, column) for row in range(12) for column in range(12)], dtype=float)
target = np.sin(coordinates[:, 0] / 2) + np.cos(coordinates[:, 1] / 3) + 0.15 * coordinates[:, 0]
model = KNeighborsRegressor(n_neighbors=3, weights="distance")
random_cv = KFold(n_splits=4, shuffle=True, random_state=57)
groups = (coordinates[:, 0] >= 6).astype(int) * 2 + (coordinates[:, 1] >= 6).astype(int)
spatial_cv = GroupKFold(n_splits=4)

random_rmse = -cross_val_score(model, coordinates, target, cv=random_cv,
scoring="neg_root_mean_squared_error")
spatial_rmse = -cross_val_score(model, coordinates, target, cv=spatial_cv, groups=groups,
scoring="neg_root_mean_squared_error")
random_separation = np.mean([
cdist(coordinates[test], coordinates[train]).min(axis=1).mean()
for train, test in random_cv.split(coordinates)
])
spatial_separation = np.mean([
cdist(coordinates[test], coordinates[train]).min(axis=1).mean()
for train, test in spatial_cv.split(coordinates, target, groups)
])
print(f"samples={len(target)} field=known_synthetic model=3nn_coordinates")
print(f"random_cv_rmse={random_rmse.mean():.3f} mean_nearest_train_m={random_separation:.3f}")
print(f"quadrant_cv_rmse={spatial_rmse.mean():.3f} mean_nearest_train_m={spatial_separation:.3f}")

Expected output:

samples=144 field=known_synthetic model=3nn_coordinates
random_cv_rmse=0.157 mean_nearest_train_m=1.003
quadrant_cv_rmse=0.652 mean_nearest_train_m=2.528

The comparison exposes geographic leakage for this intended separated-region task. Quadrant error is not inherently more “true”: for deployment beside dense existing sensors, the random-like interpolation distance may be relevant.

Leakage checklist​

Split before fitting scalers, imputers, encoders, interpolation surfaces, neighborhood summaries, target encodings, embeddings, or feature selection. Recompute graph and raster features using training-available inputs only. Keep repeated sites, households, devices, tiles, and temporal revisits together when their shared information would not exist at prediction time. Audit buffer distances in the CRS units and report whether test covariates leave the training range.

Limitations and common mistakes​

  • Blocks that are too small preserve dependence; blocks that are too large can test extrapolation beyond the intended task.
  • Spatial separation can confound location with time, sensor, institution, or protected-group composition.
  • Stratifying labels after defining blocks can split the very unit that must remain isolated.
  • Comparing many block sizes on the final test set leaks model-selection information.
  • Cross-validation estimates predictive performance under a split design, not a causal effect.

Practice​

Write two deployment statements: prediction beside existing sensors and prediction in a new district. For each, choose groups, buffers, time ordering, tuning folds, final test geography, and error slices.

Next steps​

Next, study how the reporting units themselves can change estimates in MAUP and the ecological fallacy.

Primary sources​

Checked 2026-10-18: D. R. Roberts et al., Cross-validation strategies for structured data; scikit-learn 1.9, GroupKFold; scikit-learn, cross-validation guidance and grouped splitters.