Towards improving the spatial testability of aftershock forecast models

Khawaja, Muhammad Asim; Maleki Asayesh, Behnam; Hainzl, Sebastian; Schorlemmer, Danijel

doi:https://doi.org/10.5194/egusphere-2023-309

Preprints

https://doi.org/10.5194/egusphere-2023-309

Preprints

09 Mar 2023

| 09 Mar 2023

Towards improving the spatial testability of aftershock forecast models

Muhammad Asim Khawaja, Behnam Maleki Asayesh, Sebastian Hainzl, and Danijel Schorlemmer

Abstract. Aftershock forecast models are usually provided on a uniform spatial grid, and the receiver operating characteristic (ROC) curve is often employed for evaluation, drawing a binary comparison of earthquake occurrences or non-occurrence for each grid cell. However, synthetic tests show flaws in using ROC for aftershock forecast ranking. We suggest a twofold improvement in the testing strategy. First, we propose to replace ROC with the Matthews correlation coefficient (MCC) and the F1 curve. We also suggest using a multi-resolution test grid adapted to the earthquake density. We conduct a synthetic experiment where we analyze aftershock distributions stemming from a Coulomb Failure (∆CFS) model, including stress activation and shadow regions. Using these aftershock distributions, we test the true ∆CFS model as well as a simple distance-based forecast (R), only predicting activation. The standard test cannot clearly distinguish between both forecasts, particularly in the case of some outliers. However, using both MCC-F1 instead of ROC curves and a simple radial multi-resolution grid improves the test capabilities significantly. Our findings suggest that to conduct meaningful tests, we should have at least 8 % and 5 % cells with observed earthquakes to differentiate between a near-perfect forecast model and an informationless forecast using ROC and MCC-F1, respectively. While we cannot change the observed data, we can adjust the spatial grid using a data-driven approach to reduce the disparity between the number of earthquakes and the total number of cells. Using the recently introduced Quadtree approach to generate multi-resolution grids, we test real aftershock forecast models for Chi-Chi and Landers aftershocks following the suggested guideline. Despite the improved tests, we find that the simple R model still outperforms the ∆CFS model in both cases, indicating that the latter should not be applied without further model adjustments.

Received: 24 Feb 2023 – Discussion started: 09 Mar 2023

Download & links

Preprint (PDF, 10361 KB)

Notice on discussion status
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
Preprint (10361 KB)

Supplement (1181 KB)

Download & links

The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.

Journal article(s) based on this preprint

31 Jul 2023

Towards improving the spatial testability of aftershock forecast models

Asim M. Khawaja, Behnam Maleki Asayesh, Sebastian Hainzl, and Danijel Schorlemmer

Nat. Hazards Earth Syst. Sci., 23, 2683–2696, https://doi.org/10.5194/nhess-23-2683-2023,https://doi.org/10.5194/nhess-23-2683-2023, 2023

Short summary

Muhammad Asim Khawaja et al.

Interactive discussion

Status: closed

CC1:
'Comment on egusphere-2023-309', Linxuan Li, 17 Apr 2023

This systematic research is important for this broad field and is the fundamental work for aftershock prediction. I very much enjoyed reading the paper. I have some questions that I would like to discuss with the author:
1) I wonder how you generate multiple aftershock sequences. If you use “delta CFS clock advance model”, the distribution of aftershocks is determined (lambda is given), do you change the parameters of the mainshock? Also, in Section 3.4, I also want to know the procedure of “simulating catalogs with different earthquake numbers but a fixed number of cells containing earthquakes”.
2) Figure 1: Perhaps I miss some points, but I think you should get different ROCs for R-model when you add SEs. In other words, you should have both red and blue dotted lines. Plus, I don’t understand why you use an inverse cumulative distribution function for R-model.
3) Line 231: What are (delta_CFS)^(1%) and R^(99%)? What are the criteria for selecting 1% and 99%?
4) Line 265-266: Why do you choose the aftershocks based on such criteria? There are several methods that may help distinguish aftershocks, like ETAS. At least, I think the author can change the parameters (radius and time) to verify the stability of their conclusions.

Citation: https://doi.org/10.5194/egusphere-2023-309-CC1
- AC1: 'Reply on CC1', Muhammad Asim Khawaja, 23 May 2023
  
  Thank you for your appreciation and for reading our work.
  Ans1:
  Generating aftershocks (or earthquakes) based on the probability map of a model does not involve any changes in the mainshock parameters. However, it is about simulating events based on the probability distribution.
  In simulating catalogs with different earthquake numbers but a fixed number of cells containing earthquakes, there can be two ways to achieve this programmatically. We can keep simulating catalogs until we get the catalog with the desired number of earthquakes and active cells. This can be computationally expensive. Secondly, we can simulate one earthquake at a time and keep simulating until we get the desired number of active cells. In this work, we use the latter approach.
  Please find all the codes to reproduce results here:
  https://github.com/khawajasim/aftershock_forecast_testing.
  The functions to simulate catalogs can be found in the module “utils”, along with the required documentation.
  
  Ans2:
  We decided to have just one ROC for R-model because it has a positive forecast everywhere and the ROCs remain similar. Since our goal is to observe the behavior of ROC when an earthquake occurs in a negative region of ΔCFS so introducing ROCs for the R-model would only clutter the figures with more lines without adding meaningful information.
  We use an inverse cumulative distribution for one model (which happens to be R-model in this case), and a cumulative distribution for the other model, because we think that in this way we can easily distinguish visually whether two curves are clearly separated or not. We can also use cumulative distribution for the R-model and inverse cumulative for ΔCFS.
  Ans3:
  It refers to the difference between the performance of the ΔCFS and the R model. ΔCFS(1%) and R-model(99%) mean that we allow 1% flexibility or overlap according to our criteria. Alternatively, this could be set to 2.5% (i.e. ΔCFS(2.5%) and R-model(97.5%)), or 5% or 10%, etc. However, we have used 1% because we want to allow less overlap between the performance of the ΔCFS and the R model.
  
  Ans4:
  We selected earthquakes that occurred within the one year after the mainshock with a horizontal distance of less than 100 km from the mainshock fault, because the seismicity in this spatial and temporal vicinity to the mainshock is usually dominated by aftershocks, and previous studies have used similar selection rules (Sharma et al. 2020, Asayesh et al. 2022, cited in the manuscript). However, some of the selected earthquakes may be background events unrelated to the mainshocks. We refrain from labeling the test data with a sophisticated approach such as the ETAS model, because a simpler approach is that the forecast models already account for possible stationary background activity. Here we have ignored background activity, which may have a small effect on our result, but won’t change the main result because the background activity is only a small fraction of the total activity. However, we have now clarified in the revised manuscript that we have ignored possible background activity in our test of the two real aftershock sequences.
  Note that these are just two case studies for aftershock forecast evaluation (for Chi-Chi and Landers), that we have discussed here. There are several options for other case studies or data selections. However, the final conclusion of this study does not depend on this evaluation.
  
  Sharma, S., Hainzl, S., Zöeller, G., and Holschneider, M.: Is Coulomb stress the best choice for aftershock forecasting?, Journal of Geophysical Research: Solid Earth, 125, e2020JB019 553, 2020.
  
  Asayesh, B. M., Zafarani, H., Hainzl, S., and Sharma, S.: Effects of large aftershocks on spatial aftershock forecasts during the 2017–2019 western Iran sequence, Geophysical Journal International, 232, 147–161, 2022
  
  Citation: https://doi.org/10.5194/egusphere-2023-309-AC1
CC2:
'Comment on egusphere-2023-309', Behnam Malekiasayesh, 20 Apr 2023

Due to the importance of the forecasting of aftershocks following big earthquake and testability of these forecasting methods, this research is important. I have enjoyed the paper and have some questions/suggestions that I would like to discuss with the author:
1. The MCC-F1 curve provides the opportunity to obtain the best threshold points. I am wondering if the minimum distance of all the points of the MCC-F1 curve from point of perfect performance (1, 1) is equal to the distance of the best threshold point and the point of perfect performance?
2. It seems that there is a little difference between calculated MCC-F1 metrics in this study and Cao et al. (2020) study. I am interested about the advantages of your suggested MCC-F1 metric.

Citation: https://doi.org/10.5194/egusphere-2023-309-CC2
- AC2: 'Reply on CC2', Muhammad Asim Khawaja, 23 May 2023
  
  Hello Behnam, Thank you for providing some comments from readers’ perspective.
  Ans1:
  The best possible values of MCC and F1 are 1. So the best performance of the forecast in terms of MCC and F1 is the one closest to 1. So the performance of (MCC, F1) nearest to (1, 1) provides the best performance.
  Ans2:
  For MCC-F1, the procedure of drawing the curve between MCC and F1 is the same as proposed by Cao et al. (2020). We differ slightly from the idea of Cao et al. (2020) for computing a single performance metric using the MCC-F1 curve. Cao et al. (2020) propose to compute the average distance of the MCC-F1 curve from the point of ideal performance, i.e. (1, 1). However, we slightly differ from this idea. MCC already incorporates all the entries of a confusion matrix, i.e. True Positive (TP), False Positive (FP), True Negative (TN), False Negative (TN). It never increases or decreases monotonically with varying decision thresholds, unlike in the ROC curve. Thus with average distance, we can never achieve the perfect performance of one for an ideal forecast. On the other hand, the highest performance can reveal the maximum potential of the forecast, so we choose the testing metric based on the minimum distance from the point of ideal performance (1,1).
  
  Citation: https://doi.org/10.5194/egusphere-2023-309-AC2
RC1:
'Comment on egusphere-2023-309', Jose Bayona, 26 Apr 2023

General comments:
I have read the EGUsphere-2023-209 manuscript titled "Towards improving the spatial testability of aftershock forecast models" by Asim M. Khawaja, Behnam Maleki Asayesh, Sebastian Hainzl, and Danijel Schorlemmer. In this manuscript, the authors propose to use Matthew correlation coefficients and F1 curves (MCC-F1), as well as radial multi-resolution spatial grids to improve the testability of aftershock forecasting models. They build on the work of Parsons (2020), who first discussed the utility of the Receiver Operating Characteris (ROC) curve for evaluating Coulumb Failure (∆CFS) aftershock models. Based on aftershock simulations provided by a relatively simple ∆CFS model, the authors find that replacing ROC curves with MCC-F1 curves, and using radial spatial grids, instead of rectangular grids, allows to differentiate a near perfect ∆CFS model (from which the synthetic data are generated) from a non-informative random reference model. In addition, they find that such discrimination is possible if the aftershocks occur in a few percentage of the total analysed spatial cells. Finally, they test their method against real aftershock data, finding that the reference model outperforms the ∆CFS model in forecasting the spatial distribution of the aftershocks of the 1999 Mw 7.7 Chi-Chi and 1992 Mw 7.3 Landers earthquakes.
I think the manuscript is well written, informative, and certainly could be of use to the earthquake forecasting community in improving our current understanding of the clustering nature of earthquakes. However, I also consider that it could benefit from a few improvements, which I describe below.
Specific comments:
Line 29: “physics-based”
Lines 34-35: I agree that ideally only models that have prospectively evaluated should be considered for decision-making. However, in practice, peer-reviewed models can also considered for this purpose.
Line 41: Provide a reference for the ROC.
Line 86: Provide magnitudes, years of occurrence and references for the Chi Chi and Landers earthquakes.
Line 128: The first statement is quite vague in my opinion. Either elaborate more or delete.
Line 166: Provide a reference for the Ellsworth B magnitude-area relationship.
Line 177: What is x in this context?
Line 182: The authors can also mention dynamic triggering (e.g., Hardebeck and Harris, 2022)
Lines 195-196: The authors mention that, based on their results, the ROC is unable to perform meaningful testing, but is not it also that the models they use are notoriously limited in their ability to forecast aftershock locations? I would be curious to see a similar analysis using more complex physics-based models or statistical models like ETAS.
Line 326: The authors could add "and computationally less expensive".
Technical corrections:
Figs. 1a and 1b. The legends are not very informative. Are the Coulumb stress changes in MPa? Have they been normalised?
Figs. 2b and 3b. There is a typo in the y-label of these figures.
Table 1. The authors probably mean “Multi”
References:
Hardebeck, J.L. and Harris, R.A., 2022. Earthquakes in the shadows: Why aftershocks occur at surprising locations. The Seismic Record, 2(3), pp.207-216.

Citation: https://doi.org/10.5194/egusphere-2023-309-RC1
- AC3: 'Reply on RC1', Muhammad Asim Khawaja, 23 May 2023
  
  Thank you for taking the time to review the manuscript and appreciating the effort.
  Line 29 (Response): Corrected.
  Lines 34-35 (Response): We have changed the wording and tone of the sentence.
  … “It is desirable to use those forecast models for societal decision making that have proven their applicability through testing”…
  Line 41(Response): Done.
  Line 86 (Response): Noted.
  Line 128 (Response): Modified the sentence to clarify the meaning.
  ... "The reliability of the testing models for any type of dataset is primarily associated" ...
  Line 166 (Response): We have now provided the reference in the text.
  Line 177 (Response): ‘x’ refers to ∆CFS in this case. We have modified and replaced x with “Model”.
  Line 182 (Response): Thank you for the suggestion. We have added the reference.
  Lines 195-196 (Response): These lines do not discuss the forecast testing based on a real earthquake catalog. Instead, it refers to the synthetic experiment. In this experiment, we use the simple ∆CFS model as a seismicity generator, and the same is being used for the forecast evaluation. Thus in this context, the ∆CFS is the perfect forecast model because it is also the catalog generator, so it should be able to perform better than any competing uninformative model.
  We think that we failed to convey this in the first version of the manuscript, so we have modified the text slightly to make it clearer. Please see line 201 of the revised annotated manuscript.
  Line 326 (Response): In this manuscript, we have not discussed the issue of computational resources in the context of aftershock forecast testing, so it would be inappropriate to add this to the conclusion.
  Figs. 1a and 1b. (Response): The forecasts shown in the Figures 1a and 1b show the forecast rates (ƛ), and their calculation is provided in the manuscript. We have further elaborated the text and added this to the legend and caption of the figure.
  Figs. 2b and 3b. (Response): Thank you for pointing this out. We have corrected them.
  Table 1. (Response): We have corrected it.
  
  Citation: https://doi.org/10.5194/egusphere-2023-309-AC3
RC2:
'Comment on egusphere-2023-309', Anonymous Referee #2, 04 May 2023

In this manuscript, the authors have discussed the testing of earthquakes/aftershock forecasting models. With a wide range of techniques and models being developed for earthquake forecasting, it becomes important to analyze the testing metrics in detail for a meaningful evaluation of the testing metrics is an important issue. In this paper, the authors discussed binary testing metrics, i.e. ROC curve. I share some concerns:

1- The manuscript discussing the MCC-F1 is still not published but is available as a preprint. Why do you still consider it important to propose earthquake forecast testing?

2- Can you justify whether changing the grid of the earthquake forecast model is changing the forecast itself or just affecting the test's outcomes? How will you prove that?

3- Did you also consider any other binary classification evaluation metric other than MCC-F1 and ROC curve? e.g. precision recall curve?

Citation: https://doi.org/10.5194/egusphere-2023-309-RC2
- AC4: 'Reply on RC2', Muhammad Asim Khawaja, 23 May 2023
  
  Ans1:
  Although the manuscript proposing the MCC-F1 curve has not yet been published in a journal, but we find it useful. However, MCC and F1 are already published and discussed performance measures in the literature. We find the MCC-F1 curve useful because like ROC, it provides performance across different decision thresholds.
  Ans2:
  To address this question, we would like to refer to a recent study (Khawaja et al. 2023) which showed that aggregating a forecast on a multi-resolution grid does not change the consistency of a forecast model, as long as a model is actually consistent with the actual observation.
  Khawaja, Asim M., et al. "Statistical power of spatial earthquake forecast tests." Geophysical Journal International 233.3 (2023): 2053-2066.
  Ans3:
  Yes, we considered the performance measures that work using varying thresholds, for example, the Precision-Recall curve. The Precision involves TP and FP, while Recall involves TP and FN. Similarly, both together take into consideration TP, FP, and FN, leaving out the effect of TN. Therefore, we preferred a performance metric that takes into account all four entries of a confusion matrix.
  However, the novelty of this study is to provide a guideline on the minimum amount of data required to evaluate earthquake forecast models in terms of binary classification.
  
  Citation: https://doi.org/10.5194/egusphere-2023-309-AC4

Interactive discussion

Status: closed

CC1:
'Comment on egusphere-2023-309', Linxuan Li, 17 Apr 2023

This systematic research is important for this broad field and is the fundamental work for aftershock prediction. I very much enjoyed reading the paper. I have some questions that I would like to discuss with the author:
1) I wonder how you generate multiple aftershock sequences. If you use “delta CFS clock advance model”, the distribution of aftershocks is determined (lambda is given), do you change the parameters of the mainshock? Also, in Section 3.4, I also want to know the procedure of “simulating catalogs with different earthquake numbers but a fixed number of cells containing earthquakes”.
2) Figure 1: Perhaps I miss some points, but I think you should get different ROCs for R-model when you add SEs. In other words, you should have both red and blue dotted lines. Plus, I don’t understand why you use an inverse cumulative distribution function for R-model.
3) Line 231: What are (delta_CFS)^(1%) and R^(99%)? What are the criteria for selecting 1% and 99%?
4) Line 265-266: Why do you choose the aftershocks based on such criteria? There are several methods that may help distinguish aftershocks, like ETAS. At least, I think the author can change the parameters (radius and time) to verify the stability of their conclusions.

Citation: https://doi.org/10.5194/egusphere-2023-309-CC1
- AC1: 'Reply on CC1', Muhammad Asim Khawaja, 23 May 2023
  
  Thank you for your appreciation and for reading our work.
  Ans1:
  Generating aftershocks (or earthquakes) based on the probability map of a model does not involve any changes in the mainshock parameters. However, it is about simulating events based on the probability distribution.
  In simulating catalogs with different earthquake numbers but a fixed number of cells containing earthquakes, there can be two ways to achieve this programmatically. We can keep simulating catalogs until we get the catalog with the desired number of earthquakes and active cells. This can be computationally expensive. Secondly, we can simulate one earthquake at a time and keep simulating until we get the desired number of active cells. In this work, we use the latter approach.
  Please find all the codes to reproduce results here:
  https://github.com/khawajasim/aftershock_forecast_testing.
  The functions to simulate catalogs can be found in the module “utils”, along with the required documentation.
  
  Ans2:
  We decided to have just one ROC for R-model because it has a positive forecast everywhere and the ROCs remain similar. Since our goal is to observe the behavior of ROC when an earthquake occurs in a negative region of ΔCFS so introducing ROCs for the R-model would only clutter the figures with more lines without adding meaningful information.
  We use an inverse cumulative distribution for one model (which happens to be R-model in this case), and a cumulative distribution for the other model, because we think that in this way we can easily distinguish visually whether two curves are clearly separated or not. We can also use cumulative distribution for the R-model and inverse cumulative for ΔCFS.
  Ans3:
  It refers to the difference between the performance of the ΔCFS and the R model. ΔCFS(1%) and R-model(99%) mean that we allow 1% flexibility or overlap according to our criteria. Alternatively, this could be set to 2.5% (i.e. ΔCFS(2.5%) and R-model(97.5%)), or 5% or 10%, etc. However, we have used 1% because we want to allow less overlap between the performance of the ΔCFS and the R model.
  
  Ans4:
  We selected earthquakes that occurred within the one year after the mainshock with a horizontal distance of less than 100 km from the mainshock fault, because the seismicity in this spatial and temporal vicinity to the mainshock is usually dominated by aftershocks, and previous studies have used similar selection rules (Sharma et al. 2020, Asayesh et al. 2022, cited in the manuscript). However, some of the selected earthquakes may be background events unrelated to the mainshocks. We refrain from labeling the test data with a sophisticated approach such as the ETAS model, because a simpler approach is that the forecast models already account for possible stationary background activity. Here we have ignored background activity, which may have a small effect on our result, but won’t change the main result because the background activity is only a small fraction of the total activity. However, we have now clarified in the revised manuscript that we have ignored possible background activity in our test of the two real aftershock sequences.
  Note that these are just two case studies for aftershock forecast evaluation (for Chi-Chi and Landers), that we have discussed here. There are several options for other case studies or data selections. However, the final conclusion of this study does not depend on this evaluation.
  
  Sharma, S., Hainzl, S., Zöeller, G., and Holschneider, M.: Is Coulomb stress the best choice for aftershock forecasting?, Journal of Geophysical Research: Solid Earth, 125, e2020JB019 553, 2020.
  
  Asayesh, B. M., Zafarani, H., Hainzl, S., and Sharma, S.: Effects of large aftershocks on spatial aftershock forecasts during the 2017–2019 western Iran sequence, Geophysical Journal International, 232, 147–161, 2022
  
  Citation: https://doi.org/10.5194/egusphere-2023-309-AC1
CC2:
'Comment on egusphere-2023-309', Behnam Malekiasayesh, 20 Apr 2023

Due to the importance of the forecasting of aftershocks following big earthquake and testability of these forecasting methods, this research is important. I have enjoyed the paper and have some questions/suggestions that I would like to discuss with the author:
1. The MCC-F1 curve provides the opportunity to obtain the best threshold points. I am wondering if the minimum distance of all the points of the MCC-F1 curve from point of perfect performance (1, 1) is equal to the distance of the best threshold point and the point of perfect performance?
2. It seems that there is a little difference between calculated MCC-F1 metrics in this study and Cao et al. (2020) study. I am interested about the advantages of your suggested MCC-F1 metric.

Citation: https://doi.org/10.5194/egusphere-2023-309-CC2
- AC2: 'Reply on CC2', Muhammad Asim Khawaja, 23 May 2023
  
  Hello Behnam, Thank you for providing some comments from readers’ perspective.
  Ans1:
  The best possible values of MCC and F1 are 1. So the best performance of the forecast in terms of MCC and F1 is the one closest to 1. So the performance of (MCC, F1) nearest to (1, 1) provides the best performance.
  Ans2:
  For MCC-F1, the procedure of drawing the curve between MCC and F1 is the same as proposed by Cao et al. (2020). We differ slightly from the idea of Cao et al. (2020) for computing a single performance metric using the MCC-F1 curve. Cao et al. (2020) propose to compute the average distance of the MCC-F1 curve from the point of ideal performance, i.e. (1, 1). However, we slightly differ from this idea. MCC already incorporates all the entries of a confusion matrix, i.e. True Positive (TP), False Positive (FP), True Negative (TN), False Negative (TN). It never increases or decreases monotonically with varying decision thresholds, unlike in the ROC curve. Thus with average distance, we can never achieve the perfect performance of one for an ideal forecast. On the other hand, the highest performance can reveal the maximum potential of the forecast, so we choose the testing metric based on the minimum distance from the point of ideal performance (1,1).
  
  Citation: https://doi.org/10.5194/egusphere-2023-309-AC2
RC1:
'Comment on egusphere-2023-309', Jose Bayona, 26 Apr 2023

General comments:
I have read the EGUsphere-2023-209 manuscript titled "Towards improving the spatial testability of aftershock forecast models" by Asim M. Khawaja, Behnam Maleki Asayesh, Sebastian Hainzl, and Danijel Schorlemmer. In this manuscript, the authors propose to use Matthew correlation coefficients and F1 curves (MCC-F1), as well as radial multi-resolution spatial grids to improve the testability of aftershock forecasting models. They build on the work of Parsons (2020), who first discussed the utility of the Receiver Operating Characteris (ROC) curve for evaluating Coulumb Failure (∆CFS) aftershock models. Based on aftershock simulations provided by a relatively simple ∆CFS model, the authors find that replacing ROC curves with MCC-F1 curves, and using radial spatial grids, instead of rectangular grids, allows to differentiate a near perfect ∆CFS model (from which the synthetic data are generated) from a non-informative random reference model. In addition, they find that such discrimination is possible if the aftershocks occur in a few percentage of the total analysed spatial cells. Finally, they test their method against real aftershock data, finding that the reference model outperforms the ∆CFS model in forecasting the spatial distribution of the aftershocks of the 1999 Mw 7.7 Chi-Chi and 1992 Mw 7.3 Landers earthquakes.
I think the manuscript is well written, informative, and certainly could be of use to the earthquake forecasting community in improving our current understanding of the clustering nature of earthquakes. However, I also consider that it could benefit from a few improvements, which I describe below.
Specific comments:
Line 29: “physics-based”
Lines 34-35: I agree that ideally only models that have prospectively evaluated should be considered for decision-making. However, in practice, peer-reviewed models can also considered for this purpose.
Line 41: Provide a reference for the ROC.
Line 86: Provide magnitudes, years of occurrence and references for the Chi Chi and Landers earthquakes.
Line 128: The first statement is quite vague in my opinion. Either elaborate more or delete.
Line 166: Provide a reference for the Ellsworth B magnitude-area relationship.
Line 177: What is x in this context?
Line 182: The authors can also mention dynamic triggering (e.g., Hardebeck and Harris, 2022)
Lines 195-196: The authors mention that, based on their results, the ROC is unable to perform meaningful testing, but is not it also that the models they use are notoriously limited in their ability to forecast aftershock locations? I would be curious to see a similar analysis using more complex physics-based models or statistical models like ETAS.
Line 326: The authors could add "and computationally less expensive".
Technical corrections:
Figs. 1a and 1b. The legends are not very informative. Are the Coulumb stress changes in MPa? Have they been normalised?
Figs. 2b and 3b. There is a typo in the y-label of these figures.
Table 1. The authors probably mean “Multi”
References:
Hardebeck, J.L. and Harris, R.A., 2022. Earthquakes in the shadows: Why aftershocks occur at surprising locations. The Seismic Record, 2(3), pp.207-216.

Citation: https://doi.org/10.5194/egusphere-2023-309-RC1
- AC3: 'Reply on RC1', Muhammad Asim Khawaja, 23 May 2023
  
  Thank you for taking the time to review the manuscript and appreciating the effort.
  Line 29 (Response): Corrected.
  Lines 34-35 (Response): We have changed the wording and tone of the sentence.
  … “It is desirable to use those forecast models for societal decision making that have proven their applicability through testing”…
  Line 41(Response): Done.
  Line 86 (Response): Noted.
  Line 128 (Response): Modified the sentence to clarify the meaning.
  ... "The reliability of the testing models for any type of dataset is primarily associated" ...
  Line 166 (Response): We have now provided the reference in the text.
  Line 177 (Response): ‘x’ refers to ∆CFS in this case. We have modified and replaced x with “Model”.
  Line 182 (Response): Thank you for the suggestion. We have added the reference.
  Lines 195-196 (Response): These lines do not discuss the forecast testing based on a real earthquake catalog. Instead, it refers to the synthetic experiment. In this experiment, we use the simple ∆CFS model as a seismicity generator, and the same is being used for the forecast evaluation. Thus in this context, the ∆CFS is the perfect forecast model because it is also the catalog generator, so it should be able to perform better than any competing uninformative model.
  We think that we failed to convey this in the first version of the manuscript, so we have modified the text slightly to make it clearer. Please see line 201 of the revised annotated manuscript.
  Line 326 (Response): In this manuscript, we have not discussed the issue of computational resources in the context of aftershock forecast testing, so it would be inappropriate to add this to the conclusion.
  Figs. 1a and 1b. (Response): The forecasts shown in the Figures 1a and 1b show the forecast rates (ƛ), and their calculation is provided in the manuscript. We have further elaborated the text and added this to the legend and caption of the figure.
  Figs. 2b and 3b. (Response): Thank you for pointing this out. We have corrected them.
  Table 1. (Response): We have corrected it.
  
  Citation: https://doi.org/10.5194/egusphere-2023-309-AC3
RC2:
'Comment on egusphere-2023-309', Anonymous Referee #2, 04 May 2023

In this manuscript, the authors have discussed the testing of earthquakes/aftershock forecasting models. With a wide range of techniques and models being developed for earthquake forecasting, it becomes important to analyze the testing metrics in detail for a meaningful evaluation of the testing metrics is an important issue. In this paper, the authors discussed binary testing metrics, i.e. ROC curve. I share some concerns:

1- The manuscript discussing the MCC-F1 is still not published but is available as a preprint. Why do you still consider it important to propose earthquake forecast testing?

2- Can you justify whether changing the grid of the earthquake forecast model is changing the forecast itself or just affecting the test's outcomes? How will you prove that?

3- Did you also consider any other binary classification evaluation metric other than MCC-F1 and ROC curve? e.g. precision recall curve?

Citation: https://doi.org/10.5194/egusphere-2023-309-RC2
- AC4: 'Reply on RC2', Muhammad Asim Khawaja, 23 May 2023
  
  Ans1:
  Although the manuscript proposing the MCC-F1 curve has not yet been published in a journal, but we find it useful. However, MCC and F1 are already published and discussed performance measures in the literature. We find the MCC-F1 curve useful because like ROC, it provides performance across different decision thresholds.
  Ans2:
  To address this question, we would like to refer to a recent study (Khawaja et al. 2023) which showed that aggregating a forecast on a multi-resolution grid does not change the consistency of a forecast model, as long as a model is actually consistent with the actual observation.
  Khawaja, Asim M., et al. "Statistical power of spatial earthquake forecast tests." Geophysical Journal International 233.3 (2023): 2053-2066.
  Ans3:
  Yes, we considered the performance measures that work using varying thresholds, for example, the Precision-Recall curve. The Precision involves TP and FP, while Recall involves TP and FN. Similarly, both together take into consideration TP, FP, and FN, leaving out the effect of TN. Therefore, we preferred a performance metric that takes into account all four entries of a confusion matrix.
  However, the novelty of this study is to provide a guideline on the minimum amount of data required to evaluate earthquake forecast models in terms of binary classification.
  
  Citation: https://doi.org/10.5194/egusphere-2023-309-AC4

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision | EF: Editorial file upload

ED: Publish as is (02 Jul 2023) by Filippos Vallianatos

AR by Muhammad Asim Khawaja on behalf of the Authors (06 Jul 2023) Manuscript

Journal article(s) based on this preprint

31 Jul 2023

Towards improving the spatial testability of aftershock forecast models

Asim M. Khawaja, Behnam Maleki Asayesh, Sebastian Hainzl, and Danijel Schorlemmer

Nat. Hazards Earth Syst. Sci., 23, 2683–2696, https://doi.org/10.5194/nhess-23-2683-2023,https://doi.org/10.5194/nhess-23-2683-2023, 2023

Short summary

Muhammad Asim Khawaja et al.

Supplement

https://doi.org/10.5194/egusphere-2023-309-supplement

Muhammad Asim Khawaja et al.

Viewed

Total article views: 331 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	Supplement	BibTeX	EndNote
232	82	17	331	33	5	4

HTML: 232
PDF: 82
XML: 17
Total: 331
Supplement: 33
BibTeX: 5
EndNote: 4

Views and downloads (calculated since 09 Mar 2023)

Month	HTML	PDF	XML	Total
Mar 2023	84	31	5	120
Apr 2023	62	18	6	86
May 2023	45	13	4	62
Jun 2023	22	3	0	25
Jul 2023	19	17	2	38

Cumulative views and downloads (calculated since 09 Mar 2023)

Month	HTML	PDF	XML	Total
Mar 2023	84	31	5	120
Apr 2023	62	18	6	86
May 2023	45	13	4	62
Jun 2023	22	3	0	25
Jul 2023	19	17	2	38

Viewed (geographical distribution)

Total article views: 370 (including HTML, PDF, and XML) Thereof 370 with geography defined and 0 with unknown origin.

Country	#	Views	%

Latest update: 31 Jul 2023

Download

The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.

Preprint (10361 KB)
Metadata XML

Short summary

Testing a forecast model is important for further improving it. One way of evaluating the spatial distribution of the forecast is by conducting a binary comparison of forecast and observation. We find that an already used testing metric for evaluating the spatial distribution of forecasts is incapable of differentiating between a perfect and an uninformative forecast model. Thus, we suggest using a newly proposed testing metric and representation of the forecast to conduct meaningful testing.


Total:	0
HTML:	0
PDF:	0
XML:	0