Aligning Validation with Deployment in Spatial Prediction: Target-Weighted Cross-Validation

Abstract. Reliable estimation of predictive performance is essential for spatial environmental modeling, where machine-learning models are commonly used to generate spatially continuous maps from unevenly distributed observations. Standard cross-validation (CV) procedures implicitly assume that validation data are representative of prediction conditions across the target domain. In practice, this assumption is often violated due to preferential or clustered sampling designs, leading to biased estimates of predictive performance and associated uncertainty.

We introduce a deployment-oriented validation framework for spatial prediction based on weighted CV, which aims to align validation tasks with the distribution of prediction tasks across a specified target domain. The framework encompasses importance-weighted cross-validation (IWCV) as a general solution, and a calibration-based formulation, Target-Weighted Cross-Validation (TWCV), that leverages spatially meaningful task descriptors, including environmental covariates and prediction distance.

Through controlled simulation experiments, we show that conventional non-spatial and spatial CV strategies can exhibit substantial bias under realistic sampling designs, whereas the proposed weighted CV approaches with spatial task descriptors nearly eliminate or substantially reduce this bias when validation tasks adequately cover the deployment-task space. A case study on mapping nitrogen dioxide (NO₂) concentrations across Germany demonstrates that standard CV approaches can overestimate prediction error due to sampling biases, while weighted CV yields estimates that are more consistent with deployment conditions.

The proposed framework separates validation task generation from risk estimation and can provide approximately unbiased estimators of deployment risk under appropriate coverage and weighting conditions. It provides a practical approach for improving the reliability of performance assessment in environmental mapping applications where sample distributions differ from prediction domains.