09 Mar 2023
 | 09 Mar 2023
Status: this preprint is open for discussion.

Towards improving the spatial testability of aftershock forecast models

Muhammad Asim Khawaja, Behnam Maleki Asayesh, Sebastian Hainzl, and Danijel Schorlemmer

Abstract. Aftershock forecast models are usually provided on a uniform spatial grid, and the receiver operating characteristic (ROC) curve is often employed for evaluation, drawing a binary comparison of earthquake occurrences or non-occurrence for each grid cell. However, synthetic tests show flaws in using ROC for aftershock forecast ranking. We suggest a twofold improvement in the testing strategy. First, we propose to replace ROC with the Matthews correlation coefficient (MCC) and the F1 curve. We also suggest using a multi-resolution test grid adapted to the earthquake density. We conduct a synthetic experiment where we analyze aftershock distributions stemming from a Coulomb Failure (∆CFS) model, including stress activation and shadow regions. Using these aftershock distributions, we test the true ∆CFS model as well as a simple distance-based forecast (R), only predicting activation. The standard test cannot clearly distinguish between both forecasts, particularly in the case of some outliers. However, using both MCC-F1 instead of ROC curves and a simple radial multi-resolution grid improves the test capabilities significantly. Our findings suggest that to conduct meaningful tests, we should have at least 8 % and 5 % cells with observed earthquakes to differentiate between a near-perfect forecast model and an informationless forecast using ROC and MCC-F1, respectively. While we cannot change the observed data, we can adjust the spatial grid using a data-driven approach to reduce the disparity between the number of earthquakes and the total number of cells. Using the recently introduced Quadtree approach to generate multi-resolution grids, we test real aftershock forecast models for Chi-Chi and Landers aftershocks following the suggested guideline. Despite the improved tests, we find that the simple R model still outperforms the ∆CFS model in both cases, indicating that the latter should not be applied without further model adjustments.

Muhammad Asim Khawaja et al.

Status: open (until 20 Apr 2023)

Comment types: AC – author | RC – referee | CC – community | EC – editor | CEC – chief editor | : Report abuse

Muhammad Asim Khawaja et al.

Muhammad Asim Khawaja et al.


Total article views: 113 (including HTML, PDF, and XML)
HTML PDF XML Total Supplement BibTeX EndNote
80 28 5 113 15 1 1
  • HTML: 80
  • PDF: 28
  • XML: 5
  • Total: 113
  • Supplement: 15
  • BibTeX: 1
  • EndNote: 1
Views and downloads (calculated since 09 Mar 2023)
Cumulative views and downloads (calculated since 09 Mar 2023)

Viewed (geographical distribution)

Total article views: 110 (including HTML, PDF, and XML) Thereof 110 with geography defined and 0 with unknown origin.
Country # Views %
  • 1
Latest update: 29 Mar 2023
Short summary
Testing a forecast model is important for further improving it. One way of evaluating the spatial distribution of the forecast is by conducting a binary comparison of forecast and observation. We find that an already used testing metric for evaluating the spatial distribution of forecasts is incapable of differentiating between a perfect and an uninformative forecast model. Thus, we suggest using a newly proposed testing metric and representation of the forecast to conduct meaningful testing.