Analysis of geomagnetic observatory data and detection of geomagnetic jerks with the MOSFiT software package

da Silva, Marcos Vinicius; Pinheiro, Katia; Ohlert, Achim; Matzka, Jürgen

doi:https://doi.org/10.5194/egusphere-2023-79

Preprints

https://doi.org/10.5194/egusphere-2023-79

Preprints

13 Jun 2023

| 13 Jun 2023

Analysis of geomagnetic observatory data and detection of geomagnetic jerks with the MOSFiT software package

Marcos Vinicius da Silva, Katia Pinheiro, Achim Ohlert, and Jürgen Matzka

Abstract. MOSFiT (Magnetic Observatories and Stations Filtering Tool) is a python package to visualize and filter data from magnetic observatories and magnetometer stations. The purpose of MOSFiT is to automatically isolate and analyze the secular variation (SV) information measured by geomagnetic observatories data. External field contributions may be reduced by selecting data according to local time and geomagnetic indices and by subtracting the magnetospheric field predictions of the CHAOS-7 model. MOSFiT calculates the SV by annual differences of monthly means and geomagnetic jerk occurrence time and amplitude are automatically calculated by fitting two straight-line segments in a user-defined time interval of the SV time series. Here, we present the new python package, validate it against independent results from previous publications, and show its application. In particular, we quantify the RMS misfit between SV derived from processing schemes and the SV predicted by CHAOS-7. Analysing the INTERMAGNET quasi-definitive data with MOSFIT allows for a timely investigation of SV such as the detection of recent geomagnetic jerks. It can also be used for data selection for e.g., external field studies as well as for quality control of geomagnetic observatory data.

Received: 20 Jan 2023 – Discussion started: 13 Jun 2023

Download & links

Preprint (PDF, 4807 KB)

Notice on discussion status
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
Preprint (4807 KB)

Supplement (4066 KB)

Download & links

The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.

Journal article(s) based on this preprint

18 Dec 2023

Analysis of geomagnetic observatory data and detection of geomagnetic jerks with the MOSFiT software package

Marcos Vinicius da Silva, Katia J. Pinheiro, Achim Ohlert, and Jürgen Matzka

Geosci. Instrum. Method. Data Syst., 12, 271–283, https://doi.org/10.5194/gi-12-271-2023,https://doi.org/10.5194/gi-12-271-2023, 2023

Short summary

Marcos Vinicius da Silva et al.

Interactive discussion

Status: closed

CC1: 'Comment on egusphere-2023-79', J. Miquel Torta, 14 Jun 2023

To validate the methods provided in this manuscript, in Sect. 4.2 we compare the automatic detection of the 2007, 2011 and 2014 jerks using observatory data from NGK, EBR, TAM and ASC with that given by Torta et al. (2015). The comparison is presented in Table 4. The results show a very good agreement for the time of occurrence of the jerks as well as for the amplitudes at all four observatories except for the 2011 time of occurrence jerk reported by Torta et al. for ASC, which was 2012.0. However Fig. 6 of this manuscript, and indeed Fig 1 of Torta et al. (2015), indicate that both the observatory data and the CHAOS model prediction show a jerkk around 2010. After reviewing our results and notes we can confirm that in fact the occurrence found with our method was 2010.2 (which is very close to the result given in this manuscript), but due to a transcription error we wrote 2012.0. We would be grateful if the authors could indicate this (e.g. as a personal communication) in any revisions to their original manuscript.
J. M. Torta

Citation: https://doi.org/10.5194/egusphere-2023-79-CC1
RC1:
'Comment on egusphere-2023-79', Seiki Asari, 16 Aug 2023

Basically I find the paper worth publishing, in which a new python tool MOSFiT for magnetic observatory data analysis is introduced. As the source program has been made open, it can be used by observers and researchers widely for different purposes, not just for secular variation analysis but for data processing as well. To improve the manuscript before publishing, I would give a suggestion as below, for which some further computation (but not tough) is needed to supplement their quantitative results. I would like the authors to consider it and make a necessary revision.
On top of its useful function for jerk detection, MOSFiT is featured with various methods for external field mitigation. To characterize the different methods more in detail, I suggest that quantitative comparison of SV misfits derived with all those methods be made in Section 4.1, where only two cases (CHAOS-7 corrected and uncorrected SVs) are compared in the manuscript. Analyses for KP, QD and NT method, or even for a combination of their use with the CHAOS-7 method, can also be made to complement Table 3 with their outcomes. Furthermore, Figure 9 and the text for its explanation in Section 5 may be moved to Section 4.1, as a qualitative illustration to support the quantitative analyses, which can be referenced from Section 5 now dedicated solely for jerk discussion. (Note there is redundancy in L229-231 for describing Fig.9. Some have already been stated previously in L225- 229)
It is often observed that K-indices are rather small in recovery phase of storms, when the ring currents remain, still lowering X-component level. Therefore I imagine, a combination of the KP-method and CHOAS-7 method (or otherwise a selection with Dst-index) would be even more effective for excluding external fields from data.
Typos:
L15 variesranges
L17 Abrupt changes in SV ‘trend’ (as described properly in L145)
L118 can be can be
L193 latest the end

S. Asari

Citation: https://doi.org/10.5194/egusphere-2023-79-RC1
- AC1: 'Reply on RC1', Marcos Vinicius Siqueira da Silva, 20 Sep 2023
  
  Dear Seiki Asari,
  thank you for your comments.
  Please, find attached our answer.
  Marcos Vinicius.
  
  Citation: https://doi.org/10.5194/egusphere-2023-79-AC1
RC2:
'Comment on egusphere-2023-79', Jan Reda, 17 Aug 2023

GENERAL COMMENTS
The authors present their experiences related to the analysis of jerks, which are unpredictable events that suddenly speed up secular variations (SV) of the Earth's magnetic field.
Identifying jerks is not an easy task. The authors of this article undertook the difficult task of creating generally available software that performs such a task. The software, called MOSFIT (Magnetic Observatories and Stations Filtering Tool), works with geomagnetic data of Definitive or Quasi-Definitive status in IAGA-2002 format, which are easily available, for example, on the WDC (World Data Center) Edinburgh server. This applies to both INTERMAGNET observatory data and other observatories.
The MOSFIT package can also be used for quality control of data from geomagnetic observatories. The software has filtering, selection, and data visualization capabilities. MOSFIT was written in Python language, which is becoming increasingly popular. The authors provide a link to the software, as well as a guide to facilitate the use of the software package.
In the first chapters, the authors introduce the reader well to the subject of jerks and the problems that arise when detecting them. This is about isolating SV from final recordings of the entire geomagnetic field that also contain the influence of the external field. This is really a big challenge, because not all jerks are very expressive.
In order to isolate jerks, the package offers Hampel filtering as the first step. In the next stage, the contribution of the external field can be reduced by selecting data according to local midnight, Kp geomagnetic indices, quiet days, disturbed days, and by subtracting the magnetospheric field based on the CHAOS-7 model.
It is worth emphasizing that the package offers the possibility of visualization at every stage of work.
The authors presented the detection of jerks from 2007, 2011, and 2014 and compared the results with an analysis performed and published by another researcher. Generally, it can be stated that the consistency of detection was very good, both in terms of the time of occurrence of the jerk and its amplitude.
In summary, I am happy to recommend the manuscript for publication after making minor corrections listed below, in sections SPECIFIC COMMENTS and TECHNICAL, LANGUAGE AND OTHER REMARKS.

SPECIFIC COMMENTS
In the bibliography (References), it would be good to add DOI (Digital Object Identifier) identifiers if they are available. DOI identifiers are important because they add credibility to the source and make it easier to access publications. Currently, providing DOI’s is basically a standard in bibliographies.

TECHNICAL, LANGUAGE AND OTHER REMARKS
Lines 1,7,44,155,236     The word python should be rather capitalized in this context.
Line 15                      There is missing space between “varies” and “ranges”
Line 39                      Rather should be “close-to-final” (missing ”-“)
Line 58                      “computacional” -> “computational”
Line 126                    There is missing space between “time” and “series”
Lines 174,183          “table” -> “Table”
Line 232                    “Fig 8” -> “Fig. 8” (missing dot)
Line 236                    Should be rather “one-minute”
Line 282                    Change the capitalisation: Journal of the American Statistical
                                    Association
Line 310                    Change the captalisation: Generalized Hampel Filters

Citation: https://doi.org/10.5194/egusphere-2023-79-RC2
- AC2: 'Reply on RC2', Marcos Vinicius Siqueira da Silva, 20 Sep 2023
  
  Dear Jan Reda,
  thank you for your comments.
  Please, find attached our answer.
  Marcos Vinicius.
  
  Citation: https://doi.org/10.5194/egusphere-2023-79-AC2
RC3:
'Comment on egusphere-2023-79', Rudi Cop, 18 Aug 2023

The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2023/egusphere-2023-79/egusphere-2023-79-RC3-supplement.pdf

Citation: https://doi.org/10.5194/egusphere-2023-79-RC3
- AC3: 'Reply on RC3', Marcos Vinicius Siqueira da Silva, 20 Sep 2023
  
  Dear Rudi Cop,
  thank you for your comments.
  Please, find attached our answer.
  Marcos Vinicius.
  
  Citation: https://doi.org/10.5194/egusphere-2023-79-AC3

Interactive discussion

Status: closed

CC1: 'Comment on egusphere-2023-79', J. Miquel Torta, 14 Jun 2023

To validate the methods provided in this manuscript, in Sect. 4.2 we compare the automatic detection of the 2007, 2011 and 2014 jerks using observatory data from NGK, EBR, TAM and ASC with that given by Torta et al. (2015). The comparison is presented in Table 4. The results show a very good agreement for the time of occurrence of the jerks as well as for the amplitudes at all four observatories except for the 2011 time of occurrence jerk reported by Torta et al. for ASC, which was 2012.0. However Fig. 6 of this manuscript, and indeed Fig 1 of Torta et al. (2015), indicate that both the observatory data and the CHAOS model prediction show a jerkk around 2010. After reviewing our results and notes we can confirm that in fact the occurrence found with our method was 2010.2 (which is very close to the result given in this manuscript), but due to a transcription error we wrote 2012.0. We would be grateful if the authors could indicate this (e.g. as a personal communication) in any revisions to their original manuscript.
J. M. Torta

Citation: https://doi.org/10.5194/egusphere-2023-79-CC1
RC1:
'Comment on egusphere-2023-79', Seiki Asari, 16 Aug 2023

Basically I find the paper worth publishing, in which a new python tool MOSFiT for magnetic observatory data analysis is introduced. As the source program has been made open, it can be used by observers and researchers widely for different purposes, not just for secular variation analysis but for data processing as well. To improve the manuscript before publishing, I would give a suggestion as below, for which some further computation (but not tough) is needed to supplement their quantitative results. I would like the authors to consider it and make a necessary revision.
On top of its useful function for jerk detection, MOSFiT is featured with various methods for external field mitigation. To characterize the different methods more in detail, I suggest that quantitative comparison of SV misfits derived with all those methods be made in Section 4.1, where only two cases (CHAOS-7 corrected and uncorrected SVs) are compared in the manuscript. Analyses for KP, QD and NT method, or even for a combination of their use with the CHAOS-7 method, can also be made to complement Table 3 with their outcomes. Furthermore, Figure 9 and the text for its explanation in Section 5 may be moved to Section 4.1, as a qualitative illustration to support the quantitative analyses, which can be referenced from Section 5 now dedicated solely for jerk discussion. (Note there is redundancy in L229-231 for describing Fig.9. Some have already been stated previously in L225- 229)
It is often observed that K-indices are rather small in recovery phase of storms, when the ring currents remain, still lowering X-component level. Therefore I imagine, a combination of the KP-method and CHOAS-7 method (or otherwise a selection with Dst-index) would be even more effective for excluding external fields from data.
Typos:
L15 variesranges
L17 Abrupt changes in SV ‘trend’ (as described properly in L145)
L118 can be can be
L193 latest the end

S. Asari

Citation: https://doi.org/10.5194/egusphere-2023-79-RC1
- AC1: 'Reply on RC1', Marcos Vinicius Siqueira da Silva, 20 Sep 2023
  
  Dear Seiki Asari,
  thank you for your comments.
  Please, find attached our answer.
  Marcos Vinicius.
  
  Citation: https://doi.org/10.5194/egusphere-2023-79-AC1
RC2:
'Comment on egusphere-2023-79', Jan Reda, 17 Aug 2023

GENERAL COMMENTS
The authors present their experiences related to the analysis of jerks, which are unpredictable events that suddenly speed up secular variations (SV) of the Earth's magnetic field.
Identifying jerks is not an easy task. The authors of this article undertook the difficult task of creating generally available software that performs such a task. The software, called MOSFIT (Magnetic Observatories and Stations Filtering Tool), works with geomagnetic data of Definitive or Quasi-Definitive status in IAGA-2002 format, which are easily available, for example, on the WDC (World Data Center) Edinburgh server. This applies to both INTERMAGNET observatory data and other observatories.
The MOSFIT package can also be used for quality control of data from geomagnetic observatories. The software has filtering, selection, and data visualization capabilities. MOSFIT was written in Python language, which is becoming increasingly popular. The authors provide a link to the software, as well as a guide to facilitate the use of the software package.
In the first chapters, the authors introduce the reader well to the subject of jerks and the problems that arise when detecting them. This is about isolating SV from final recordings of the entire geomagnetic field that also contain the influence of the external field. This is really a big challenge, because not all jerks are very expressive.
In order to isolate jerks, the package offers Hampel filtering as the first step. In the next stage, the contribution of the external field can be reduced by selecting data according to local midnight, Kp geomagnetic indices, quiet days, disturbed days, and by subtracting the magnetospheric field based on the CHAOS-7 model.
It is worth emphasizing that the package offers the possibility of visualization at every stage of work.
The authors presented the detection of jerks from 2007, 2011, and 2014 and compared the results with an analysis performed and published by another researcher. Generally, it can be stated that the consistency of detection was very good, both in terms of the time of occurrence of the jerk and its amplitude.
In summary, I am happy to recommend the manuscript for publication after making minor corrections listed below, in sections SPECIFIC COMMENTS and TECHNICAL, LANGUAGE AND OTHER REMARKS.

SPECIFIC COMMENTS
In the bibliography (References), it would be good to add DOI (Digital Object Identifier) identifiers if they are available. DOI identifiers are important because they add credibility to the source and make it easier to access publications. Currently, providing DOI’s is basically a standard in bibliographies.

TECHNICAL, LANGUAGE AND OTHER REMARKS
Lines 1,7,44,155,236     The word python should be rather capitalized in this context.
Line 15                      There is missing space between “varies” and “ranges”
Line 39                      Rather should be “close-to-final” (missing ”-“)
Line 58                      “computacional” -> “computational”
Line 126                    There is missing space between “time” and “series”
Lines 174,183          “table” -> “Table”
Line 232                    “Fig 8” -> “Fig. 8” (missing dot)
Line 236                    Should be rather “one-minute”
Line 282                    Change the capitalisation: Journal of the American Statistical
                                    Association
Line 310                    Change the captalisation: Generalized Hampel Filters

Citation: https://doi.org/10.5194/egusphere-2023-79-RC2
- AC2: 'Reply on RC2', Marcos Vinicius Siqueira da Silva, 20 Sep 2023
  
  Dear Jan Reda,
  thank you for your comments.
  Please, find attached our answer.
  Marcos Vinicius.
  
  Citation: https://doi.org/10.5194/egusphere-2023-79-AC2
RC3:
'Comment on egusphere-2023-79', Rudi Cop, 18 Aug 2023

The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2023/egusphere-2023-79/egusphere-2023-79-RC3-supplement.pdf

Citation: https://doi.org/10.5194/egusphere-2023-79-RC3
- AC3: 'Reply on RC3', Marcos Vinicius Siqueira da Silva, 20 Sep 2023
  
  Dear Rudi Cop,
  thank you for your comments.
  Please, find attached our answer.
  Marcos Vinicius.
  
  Citation: https://doi.org/10.5194/egusphere-2023-79-AC3

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision | EF: Editorial file upload

AR by Marcos Vinicius Siqueira da Silva on behalf of the Authors (12 Oct 2023) Author's response Author's tracked changes Manuscript

ED: Publish as is (13 Oct 2023) by Valery Korepanov

AR by Marcos Vinicius Siqueira da Silva on behalf of the Authors (22 Oct 2023) Author's response Manuscript

Journal article(s) based on this preprint

18 Dec 2023

Analysis of geomagnetic observatory data and detection of geomagnetic jerks with the MOSFiT software package

Marcos Vinicius da Silva, Katia J. Pinheiro, Achim Ohlert, and Jürgen Matzka

Geosci. Instrum. Method. Data Syst., 12, 271–283, https://doi.org/10.5194/gi-12-271-2023,https://doi.org/10.5194/gi-12-271-2023, 2023

Short summary

Marcos Vinicius da Silva et al.

Supplement

https://doi.org/10.5194/egusphere-2023-79-supplement

Model code and software

Magnetic Observatories and Stations Filtering Tool (MOSFiT) Marcos Vinicius da Silva https://github.com/marcosv9/MOSFiT-package

Marcos Vinicius da Silva et al.

Viewed

Total article views: 425 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	Supplement	BibTeX	EndNote
297	96	32	425	40	14	17

HTML: 297
PDF: 96
XML: 32
Total: 425
Supplement: 40
BibTeX: 14
EndNote: 17

Views and downloads (calculated since 13 Jun 2023)

Month	HTML	PDF	XML	Total
Jun 2023	96	24	9	129
Jul 2023	68	19	4	91
Aug 2023	62	13	8	83
Sep 2023	37	17	6	60
Oct 2023	15	15	4	34
Nov 2023	6	1	0	7
Dec 2023	13	7	1	21

Cumulative views and downloads (calculated since 13 Jun 2023)

Month	HTML	PDF	XML	Total
Jun 2023	96	24	9	129
Jul 2023	68	19	4	91
Aug 2023	62	13	8	83
Sep 2023	37	17	6	60
Oct 2023	15	15	4	34
Nov 2023	6	1	0	7
Dec 2023	13	7	1	21

Viewed (geographical distribution)

Total article views: 414 (including HTML, PDF, and XML) Thereof 414 with geography defined and 0 with unknown origin.

Country	#	Views	%

Latest update: 18 Dec 2023

Download

The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.

Preprint (4807 KB)
Metadata XML

Short summary

Geomagnetic observatories are dedicated to the long-term monitoring of the Earth's magnetic field. Their time-series contain information mainly about the Earth's core and the near-Earth space environment. Our Python code access a global database with the most recent observatory data and allows to separate the information on the Earth's core. At the same time, it allows for an efficient check of the data quality. We present the code, validate it and explain its usage.


Total:	0
HTML:	0
PDF:	0
XML:	0