kNNDM: k-fold Nearest Neighbour Distance Matching Cross-Validation for map accuracy estimation

Linnenbrink, Jan; Milà, Carles; Ludwig, Marvin; Meyer, Hanna

doi:https://doi.org/10.5194/egusphere-2023-1308

Preprints

https://doi.org/10.5194/egusphere-2023-1308

Preprints

05 Jul 2023

| 05 Jul 2023

kNNDM: k-fold Nearest Neighbour Distance Matching Cross-Validation for map accuracy estimation

Jan Linnenbrink, Carles Milà, Marvin Ludwig, and Hanna Meyer

Abstract. Random and spatial Cross-Validation (CV) methods are commonly used to evaluate machine learning-based spatial prediction models, and the obtained performance values are often interpreted as map accuracy estimates. However, the appropriateness of such approaches is currently the subject of controversy. For the common case where no probability sample for validation purposes is available, in Milà et al. (2022) we proposed the Nearest Neighbour Distance Matching (NNDM) Leave-One-Out (LOO) CV method. This method produces a distribution of geographical Nearest Neighbour Distances (NND) between test and train locations during CV that matches the distribution of NND between prediction and training locations. Hence, it creates predictive conditions during CV that are comparable to what is required when predicting a defined area. Although NNDM LOO CV produced largely reliable map accuracy estimates in our analysis, as a LOO-based method, it cannot be applied to large datasets found in many studies.

Here, we propose a novel k-fold CV strategy for map accuracy estimation inspired by the concepts of NNDM LOO CV: the k-fold NNDM (kNNDM) CV. The kNNDM algorithm tries to find a k-fold configuration such that the Empirical Cumulative Distribution Function (ECDF) of NND between test and train locations during CV is matched to the ECDF of NND between prediction and training locations.

We tested kNNDM CV in a simulation study with different sampling distributions and compared it to other CV methods including NNDM LOO CV. We found that kNNDM CV performed similarly to NNDM LOO CV and produced reasonably reliable map accuracy estimates across sampling patterns with strong reductions in computation time for large sample sizes. Furthermore, we found a positive linear association between the quality of the match of the two ECDFs in kNNDM and the reliability of the map accuracy estimates.

kNNDM provided the advantages of our original NNDM LOO CV strategy while bypassing its sample size limitations.

Received: 14 Jun 2023 – Discussion started: 05 Jul 2023

Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this preprint. The responsibility to include appropriate place names lies with the authors.

Download & links

Preprint (PDF, 1574 KB)

Notice on discussion status
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
Preprint (1574 KB)

Download & links

The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.

Journal article(s) based on this preprint

07 Aug 2024

kNNDM CV: k-fold nearest-neighbour distance matching cross-validation for map accuracy estimation

Jan Linnenbrink, Carles Milà, Marvin Ludwig, and Hanna Meyer

Geosci. Model Dev., 17, 5897–5912, https://doi.org/10.5194/gmd-17-5897-2024,https://doi.org/10.5194/gmd-17-5897-2024, 2024

Short summary

Jan Linnenbrink, Carles Milà, Marvin Ludwig, and Hanna Meyer

Interactive discussion

Status: closed

CC1: 'Comment on egusphere-2023-1308', Nils Tjaden, 07 Jul 2023

Just a quick hint that https://doi.org/10.1016/j.jag.2023.103364 was just published - may or may not be relevant for your discussion.

Citation: https://doi.org/10.5194/egusphere-2023-1308-CC1
RC1:
'Comment on egusphere-2023-1308', Italo Goncalves, 23 Aug 2023
The manuscript presents a much-needed methodology for cross-validation of spatial data. In my opinion, the strongest point is the use of the W statistic to identify the best CV split. However, there are a few points which I feel should be addressed in the discussion.
The proposed methodology using clustering algorithms seems valid, but how can we know if it provides the best possible result? An algorithm that optimizes the W statistic directly as a function of the CV fold indices would be more desirable, instead of relying on the clustering algorithm´s internal metric as a proxy. As a suggestion for future work, I recommend using a genetic algorithm to assign CV indices to the data points directly.

The W statistic explained 60% of the variability in map accuracy, but would this be consistent across different datasets? At least one more case study would be needed to verify this.

Minor comment:
Line 90: cross “the”.
Citation: https://doi.org/10.5194/egusphere-2023-1308-RC1
RC2: 'Comment on egusphere-2023-1308', Anonymous Referee #2, 23 Aug 2023

The study proposes a novel cross-validation method for spatial data that aims to deliver more representative measurements of spatial map accuracy than commonly-used methods. This is a relevant concern for GMD readers with the rise in use of machine learning methods for geoscientific modelling. Issues with model evaluation in the spatial setting have been identified in a number of recent studies. The paper is well-written and contributes a practical solution for a common issue.
In my opinion, the most exciting/innovative idea in this work is the concept of defining the evaluation method based on the desired data for which he model is intended to return predictions. This would require researchers to more carefully define the purpose of their models before and during the model creation process, which should be common practice. In reality, this is often not done, or done in a ‘standard’ way which doesn’t accurately reflect the intended use of the model.
The method presented in this paper is a very practical solution to this, where the desired target dataset is an input of the evaluation algorithm and therefore researchers are required to clearly consider and define it. I think this is a significant contribution to model development methodology and should be more clearly emphasised in the manuscript. The possibilities, benefits and disadvantages of this concept could also be discussed - for example, when models are used in production, the prediction area is a moving target; would that require continual re-evaluation?
The paper suggests that kNNDM is, essentially, a computationally-cheaper alternative to the previously-published method by the authors, NNDM LOO. In the article, the only limitation of leave-one-out CV methods described is that of computational time. However, to my knowledge, even if computation is not considered, LOO CV methods may not be the optimum method due to higher variation in the resulting models (due to the bias-variance tradeoff). Could this explain why kNNDM 10-fold seems to perform better in the case of strong clusters (Figure 5)? For me, this would be more convincing than the computation speedup comparison, which is relatively trivial given that LOO CV is the most extreme version of k-fold CV.
Following on from this, it seems likely that the value of k would impact the results. Use of 10 folds is very common; is there theoretical justification for this? It would be useful to see some comparisons of the results with multiple values of k.
In Figure 1, it is shown that the W statistic will also be larger if training points are regularly distributed, as well as when clustered. Does this mean that the null hypothesis might be rejected for regularly distributed datapoints? Does this explain why NNDM LOO performed better for regularly distributed data (Figure 5)?
Finally, I would recommend testing the method on at least one additional dataset, as the results presumably depend on the spatial autocorrelation present in the dataset used.
Minor comment: I assume the hyperparameters of the models are not tuned as it is not mentioned, but this could be stated explicitly.

Citation: https://doi.org/10.5194/egusphere-2023-1308-RC2
AC1: 'Comment on egusphere-2023-1308', Jan Linnenbrink, 19 Oct 2023

The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2023/egusphere-2023-1308/egusphere-2023-1308-AC1-supplement.pdf

Citation: https://doi.org/10.5194/egusphere-2023-1308-AC1

Interactive discussion

Status: closed

CC1: 'Comment on egusphere-2023-1308', Nils Tjaden, 07 Jul 2023

Just a quick hint that https://doi.org/10.1016/j.jag.2023.103364 was just published - may or may not be relevant for your discussion.

Citation: https://doi.org/10.5194/egusphere-2023-1308-CC1
RC1:
'Comment on egusphere-2023-1308', Italo Goncalves, 23 Aug 2023
The manuscript presents a much-needed methodology for cross-validation of spatial data. In my opinion, the strongest point is the use of the W statistic to identify the best CV split. However, there are a few points which I feel should be addressed in the discussion.
The proposed methodology using clustering algorithms seems valid, but how can we know if it provides the best possible result? An algorithm that optimizes the W statistic directly as a function of the CV fold indices would be more desirable, instead of relying on the clustering algorithm´s internal metric as a proxy. As a suggestion for future work, I recommend using a genetic algorithm to assign CV indices to the data points directly.

The W statistic explained 60% of the variability in map accuracy, but would this be consistent across different datasets? At least one more case study would be needed to verify this.

Minor comment:
Line 90: cross “the”.
Citation: https://doi.org/10.5194/egusphere-2023-1308-RC1
RC2: 'Comment on egusphere-2023-1308', Anonymous Referee #2, 23 Aug 2023

The study proposes a novel cross-validation method for spatial data that aims to deliver more representative measurements of spatial map accuracy than commonly-used methods. This is a relevant concern for GMD readers with the rise in use of machine learning methods for geoscientific modelling. Issues with model evaluation in the spatial setting have been identified in a number of recent studies. The paper is well-written and contributes a practical solution for a common issue.
In my opinion, the most exciting/innovative idea in this work is the concept of defining the evaluation method based on the desired data for which he model is intended to return predictions. This would require researchers to more carefully define the purpose of their models before and during the model creation process, which should be common practice. In reality, this is often not done, or done in a ‘standard’ way which doesn’t accurately reflect the intended use of the model.
The method presented in this paper is a very practical solution to this, where the desired target dataset is an input of the evaluation algorithm and therefore researchers are required to clearly consider and define it. I think this is a significant contribution to model development methodology and should be more clearly emphasised in the manuscript. The possibilities, benefits and disadvantages of this concept could also be discussed - for example, when models are used in production, the prediction area is a moving target; would that require continual re-evaluation?
The paper suggests that kNNDM is, essentially, a computationally-cheaper alternative to the previously-published method by the authors, NNDM LOO. In the article, the only limitation of leave-one-out CV methods described is that of computational time. However, to my knowledge, even if computation is not considered, LOO CV methods may not be the optimum method due to higher variation in the resulting models (due to the bias-variance tradeoff). Could this explain why kNNDM 10-fold seems to perform better in the case of strong clusters (Figure 5)? For me, this would be more convincing than the computation speedup comparison, which is relatively trivial given that LOO CV is the most extreme version of k-fold CV.
Following on from this, it seems likely that the value of k would impact the results. Use of 10 folds is very common; is there theoretical justification for this? It would be useful to see some comparisons of the results with multiple values of k.
In Figure 1, it is shown that the W statistic will also be larger if training points are regularly distributed, as well as when clustered. Does this mean that the null hypothesis might be rejected for regularly distributed datapoints? Does this explain why NNDM LOO performed better for regularly distributed data (Figure 5)?
Finally, I would recommend testing the method on at least one additional dataset, as the results presumably depend on the spatial autocorrelation present in the dataset used.
Minor comment: I assume the hyperparameters of the models are not tuned as it is not mentioned, but this could be stated explicitly.

Citation: https://doi.org/10.5194/egusphere-2023-1308-RC2
AC1: 'Comment on egusphere-2023-1308', Jan Linnenbrink, 19 Oct 2023

The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2023/egusphere-2023-1308/egusphere-2023-1308-AC1-supplement.pdf

Citation: https://doi.org/10.5194/egusphere-2023-1308-AC1

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision | EF: Editorial file upload

AR by Jan Linnenbrink on behalf of the Authors (07 Nov 2023) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (03 Dec 2023) by Rohitash Chandra

RR by Italo Goncalves (05 Dec 2023)

RR by Ute Mueller (07 Jan 2024)

RR by Anonymous Referee #4 (07 Jan 2024)

Suggestions for revision or reasons for rejection

This paper is an improvement on the efficiency of the authors' previous work NNDM LOO CV, which is significant from the results (Fig. 8). However, some necessary modifications need to be made before the paper can be accepted. In fact, I don't think there are too many technical problems in this paper, but there is a lot of room for improvement in the writing of the paper. In order to let readers better understand their work and expand the influence of the paper, I suggest that the author consider my opinions.

1) The title of the paper is suggested to be changed to kNNDM CV: k-fold... Or simply remove kNNDM, because k-fold NNDM is more accurate, but this abbreviation is too long.

2) Since the biggest contribution of the paper is to improve the efficiency of NNDM LOO CV method, the abstract needs to explain the specific efficiency improvement shown by the experiment (For example: how many times improved).

3) I noticed that Mila et al., 2022 had been cited 20 times in the paper, which generally only needs to be cited once, not every time it is mentioned.

4) In Part 2 I noticed that the premise of understanding this paper seems to be reading (Mila et al., 2022), on the basis that I think each paper should be independent. Therefore, I suggest authors supplement the introduction of NNDM LOO method and related concepts in this paper?

5) How does the paper implement k-fold with w statistic? Authors seem to want the reader to understand their thoughts from the algorithm, which is often difficult. I suggest they devote some content to addressing their idea directly. Figure 2 also looks too complicated. The opinions of the given Community comments, I also went to see Wang's paper (https://doi.org/10.1016/j.jag.2023.103364), the paper compared easier to understand, which can be refered.

6) In my opinion, Figure 1 and Figure 3 are also unnecessary. These conclusions can be explained through demonstration or reference. If necessary, they should be included in the experimental part instead of here. The authors should note that since the data is presented in a later section, the reader will have more confusion about how these results were obtained.

7) FIgure 7. A line chart might be better.

8) Additional simulation should be placed in the paper, not in a supplement. Figures 1 and 3 can instead be added to the supplement file.

Hide

ED: Reconsider after major revisions (25 Jan 2024) by Rohitash Chandra

AR by Jan Linnenbrink on behalf of the Authors (04 Mar 2024) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (08 Apr 2024) by Rohitash Chandra

RR by Ute Mueller (15 Apr 2024)

Suggestions for revision or reasons for rejection

Note that line numbers refer to the track changes version of the document.

I agree with the authors that figures 1 and 3 should be kept in the manuscript. There are still some minor issues that should be addressed, in part grammatical corrections but also some improvements

Line 6, Line 80, 86, 101 replace “train” by “training”. There might be other instances, so please correct as necessary

Line 58: “In Mila et al (2022)” is not necessary, nor is it necessary to start a new paragraph

line 81: “closely” instead of “close”

line 97: write 1_{r}(d_j) instead of 1{d_j<=r} and define the function 1_{r}(d_j) as
1_{r}(d_j)=\lbrace 0 if d_j>r // 1 if d_j \leq r

line 100 1_{r}(d_{ij}) instead of 1{d_{ij}<=r}

line 102 “less than or equal to “ instead of “equal or lower than”

line 106 write 1_{r}(d^*_j) instead of 1{d^*_j<=r}

lines 122 to 125: set this out formally defining H_0 and H_alt, you can then remove the frmal statements from figure 2 and simplify it a little at least

Lines 127-129: This is not clear” If, however, the null hypothesis is rejected (p-value < 0.05),
we proceed to cluster the training points based on their coordinates into a range of qi ∈ Q clusters, where Q is defined as an integer sequence of length 100 ranging between k and N “ Do you mean that “ we proceed to cluster the training points into qi ∈ Q clusters, where Q is defined as an integer sequence of length 100 with values between k and N”? Incidentally you have not stated the value of k anywhere. What is its minimum? Also it seems that Q is defined before the process starts, so may first define that in the text also in that order

I agree with reviewer 3 that Figure 2 is too complicated. It can be simplified if stanrd flow chart symbols are used and for example “For each q i^ Compute Gj* (r, L) compute W statistic
between Gj*(r, L) and Gij(r)” is replaced by “For each q i^ compute Gj* (r, L) and
W statistic” also , you forgot that q is also an output

Line 180: “extremely clustered instead of “extreme clustered”

Hide

RR by Wen Luo (17 Apr 2024)

ED: Publish as is (17 Jun 2024) by Rohitash Chandra

AR by Jan Linnenbrink on behalf of the Authors (18 Jun 2024) Manuscript

Journal article(s) based on this preprint

07 Aug 2024

kNNDM CV: k-fold nearest-neighbour distance matching cross-validation for map accuracy estimation

Jan Linnenbrink, Carles Milà, Marvin Ludwig, and Hanna Meyer

Geosci. Model Dev., 17, 5897–5912, https://doi.org/10.5194/gmd-17-5897-2024,https://doi.org/10.5194/gmd-17-5897-2024, 2024

Short summary

Jan Linnenbrink, Carles Milà, Marvin Ludwig, and Hanna Meyer

Model code and software

kNNDM: k-fold Nearest Neighbour Distance Matching Cross-Validation for map accuracy estimation Jan Linnenbrink, Carles Milà, Marvin Ludwig, and Hanna Meyer https://doi.org/10.6084/m9.figshare.23514135.v1

Jan Linnenbrink, Carles Milà, Marvin Ludwig, and Hanna Meyer

Viewed

Total article views: 1,360 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	BibTeX	EndNote
1,015	299	46	1,360	46	36

HTML: 1,015
PDF: 299
XML: 46
Total: 1,360
BibTeX: 46
EndNote: 36

Views and downloads (calculated since 05 Jul 2023)

Month	HTML	PDF	XML	Total
Jul 2023	247	73	10	330
Aug 2023	96	30	4	130
Sep 2023	49	24	1	74
Oct 2023	61	21	6	88
Nov 2023	23	12	1	36
Dec 2023	44	12	4	60
Jan 2024	95	14	2	111
Feb 2024	43	28	2	73
Mar 2024	64	16	0	80
Apr 2024	71	18	5	94
May 2024	61	21	3	85
Jun 2024	96	14	5	115
Jul 2024	54	13	3	70
Aug 2024	11	3	0	14
Sep 2024	0

Cumulative views and downloads (calculated since 05 Jul 2023)

Month	HTML	PDF	XML	Total
Jul 2023	247	73	10	330
Aug 2023	96	30	4	130
Sep 2023	49	24	1	74
Oct 2023	61	21	6	88
Nov 2023	23	12	1	36
Dec 2023	44	12	4	60
Jan 2024	95	14	2	111
Feb 2024	43	28	2	73
Mar 2024	64	16	0	80
Apr 2024	71	18	5	94
May 2024	61	21	3	85
Jun 2024	96	14	5	115
Jul 2024	54	13	3	70
Aug 2024	11	3	0	14
Sep 2024	0

Viewed (geographical distribution)

Total article views: 1,358 (including HTML, PDF, and XML) Thereof 1,358 with geography defined and 0 with unknown origin.

Country	#	Views	%

Cited

Latest update: 03 Sep 2024

Download

The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.

Preprint (1574 KB)
Metadata XML

Short summary

Estimation of map accuracy based on Cross-Validation (CV) in spatial modeling is pervasive but controversial. Here, we build upon our previous work and propose a novel, prediction-oriented k-fold CV strategy for map accuracy estimation in which the distribution of geographical distances between prediction and training points is taken into account when constructing the CV folds. Our method produces more reliable estimates than other CV methods and can be used for large datasets.


Total:	0
HTML:	0
PDF:	0
XML:	0

kNNDM: k-fold Nearest Neighbour Distance Matching Cross-Validation for map accuracy estimation

Journal article(s) based on this preprint

Interactive discussion

Interactive discussion

Peer review completion

Suggestions for revision or reasons for rejection

Suggestions for revision or reasons for rejection

Suggestions for revision or reasons for rejection

Journal article(s) based on this preprint

Model code and software

Viewed

Viewed (geographical distribution)

Cited

3 citations as recorded by crossref.