the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 20 Dec 2022
Near-real time detection of unexpected atmospheric events using Principal Component Analysis on the IASI radiances
Adrien Vu Van
Anne Boynard
Pascal Prunet
Dominique Jolivet
Olivier Lezeaux
Patrice Henry
Claude Camy-Peyret
Lieven Clarisse
Bruno Franco
Pierre-François Coheur
Cathy Clerbaux
Abstract. The three IASI instruments on-board the Metop family of satellites have been sounding the atmospheric composition since 2006. More than 30 atmospheric gases can be measured from the IASI radiance spectra, allowing the improvement of weather forecasting, and the monitoring of atmospheric chemistry and climate variables.
The early detection of extreme events such as fires, pollution episodes, volcanic eruptions, or industrial releases is key to take safety measures to protect inhabitants and the environment in the impacted areas. With its near real time observations and good horizontal coverage, IASI can contribute to the series of monitoring systems for the systematic and continuous detection of exceptional atmospheric events, in order to support operational decisions.
In this paper, we describe a new approach for the near real time detection and characterization of unexpected events, which relies on the principal component analysis (PCA) of IASI radiance spectra. By analysing both the IASI raw and compressed spectra, we applied a PCA-granule based method on various past well documented extreme events such as volcanic eruptions, fires, anthropogenic pollutions and industrial accidents. We demonstrate that the method is well suited to detect spectral signatures for reactive and weak absorbing gases, even for sporadic events. Long-term records are also generated for fire and volcanic events, by analysing the available IASI/Metop-B data record.
-
Notice on discussion status
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
-
Preprint
(1848 KB)
-
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(1848 KB) - BibTeX
- EndNote
- Final revised paper
Journal article(s) based on this preprint
Adrien Vu Van et al.
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2022-1372', Anonymous Referee #2, 10 Jan 2023
General comments
The paper describes a method for detection of unusual IASI spectra based on the noise normalised (PC) reconstruction residual. Each channel of the noise normalised reconstruction residual is a linear function of the radiances. 11 channels corresponding to absorption lines of selected gases are chosen for the detection (in the appendix a larger set of channels (ranges) is presented but the relation and role of each of these channel sets is not clear). While, by construction, the method is well suited to identify unusual spectra, the allocation to specific molecules can cause false detections as illustrated in the paper with the (false) HNO3 detection in a volcanic plume.
Specific comments
Section 3.1 needs some corrections. N is defined as the instrument noise covariance matrix, which is consistent with its use in equation 1, but not in the following equations, where it should be the matrix square root of the instrument noise. Line 102: this is not a projection (look up the definition). Line 107: “conservatively”?? Line 115: “optimal number” optimal in what sense? Line 97: is it really necessary to give a formula for the covariance matrix, especially since this formula is not the best way to actually compute it.
The small size (120000) of the training database is problematic because the computation of the 8461*8461 covariance matrix will be affected by instrument noise as well as unusual spectra (which can be hard to avoid).
There is no evidence presented for the usefulness of the thinning towards the poles “in order to not over-represent high latitudes”.
Line 131: then? And how can a random selection help to “represent all the conditions” – should be better to keep all.
Section 3.3: 20? Actually 1 PC is enough to “depict (sic!) most of the atmospheric variability”. While 150 PC is a good choice this is not related to any percentage of the total variance but rather the signal to noise ratio of the remaining PCs.
Section 4:
The details, especially regarding the detection thresholds, are hard to understand and should be rewritten. Also I miss justifications for the choices made.
Why not apply thresholds for each individual spectrum, instead of the granule min and max? Faster (line 152)? No, I don’t believe so.
The use of 3 different thresholds seems unnecessarily complicated. And the second threshold might be counterproductive in case most of the granule is affected by a similar anomaly.
The term “signal intensity” is used without being introduced. It is simply the (absolute value of) the (noise normalised) residual and the new term does only confuse.
It is not clear why a second training set was used for the computation of thresholds. Why different threshold for day and night?
Line 442-443 and 506-507: This is what was already discussed on page 10. I feel it is wrong to talk about “artefact” and “reconstruction error” in this context. The detection method of the paper is nothing but the identification of reconstruction error. That an unusual perturbation in a limited number of channels can affect the reconstruction residual in other channels is natural and unavoidable (as you can convince yourself by looking at a two dimensional space with 1 retained PC). Actually maybe the biggest contribution of the paper is that is shows that this kind of “cross talk” seems to be relatively rare in practice.
Technical corrections
Line 19: horizontal => spatial
Line 120: insert “as”
Line 145: “maximum of information”?
Line 154: noise normalised “residuals”
Line 182: I don’t understand this sentence
Citation: https://doi.org/10.5194/egusphere-2022-1372-RC1 -
AC1: 'Reply on RC1', Adrien Vu Van, 17 Feb 2023
The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2022/egusphere-2022-1372/egusphere-2022-1372-AC1-supplement.pdf
-
AC1: 'Reply on RC1', Adrien Vu Van, 17 Feb 2023
-
RC2: 'Comment on egusphere-2022-1372', Anonymous Referee #1, 12 Jan 2023
Review of “Near-real time detection of unexpected atmospheric events using Principal Component Analysis on the IASI radiances” by Adrien Vu Van et al.
The manuscript describes a PCA based method on real time detection and characterization of atmospheric events. For this, they are applying measured data from three IASI satellites. The manuscript is nicely written and offers interesting application of the PCA on detection of extreme events. However, some clarifications are needed, as described below.
Major comment
The methodology description needs to be improved. Even though majority of the methodology is described in previous study, it would be important to provide here necessary details on the method for replicating the analyses with similar data. e.g. PCA could be described more clearly and it is not clear how to you get GMI and GMA from the PC’s. This makes it more difficult to follow the results from the case studies.
Specific comments
Abstract: Point out the focus and true novelty of this manuscript in the abstract. Now it sounds more like the introduction
line 117, Antonelli 2004 is not in the list of references. In addition, with Atkinson 2008 and 2010 they are not the original or the best references of methods for defining the optimal number of components
Line 173: Define IASI-PCA-GE more clearly
Section 3.3.: As the explained variance is not really increasing after ~25 components, using 150 PC sounds a bit of overfitting. How did you define the number? How many PC would e.g. Scree test or Kaiser criterion suggest?
Lines 501-506: With this high number of observations in the training set, it is not probable that few outliers would affect drastically to the sensitivity of the method. As already the Atkinson papers pointed out, there has been suspicions that PCA might not be the best method for this type of analysis. Have you considered other possible factorization methods like EFA, NMF or PMF discussed e.g. in Isokääntä et al. 2020 (https://doi.org/10.5194/amt-13-2995-2020)?
In addition, have you considered accounting for the geophysical parameter possibly acting as confounding factors in your analysis?Conclusions: point out that the method can be used as online tool for detecting extreme events, as mentioned in the text earlier.
Citation: https://doi.org/10.5194/egusphere-2022-1372-RC2 -
AC2: 'Reply on RC2', Adrien Vu Van, 17 Feb 2023
The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2022/egusphere-2022-1372/egusphere-2022-1372-AC2-supplement.pdf
-
AC2: 'Reply on RC2', Adrien Vu Van, 17 Feb 2023
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2022-1372', Anonymous Referee #2, 10 Jan 2023
General comments
The paper describes a method for detection of unusual IASI spectra based on the noise normalised (PC) reconstruction residual. Each channel of the noise normalised reconstruction residual is a linear function of the radiances. 11 channels corresponding to absorption lines of selected gases are chosen for the detection (in the appendix a larger set of channels (ranges) is presented but the relation and role of each of these channel sets is not clear). While, by construction, the method is well suited to identify unusual spectra, the allocation to specific molecules can cause false detections as illustrated in the paper with the (false) HNO3 detection in a volcanic plume.
Specific comments
Section 3.1 needs some corrections. N is defined as the instrument noise covariance matrix, which is consistent with its use in equation 1, but not in the following equations, where it should be the matrix square root of the instrument noise. Line 102: this is not a projection (look up the definition). Line 107: “conservatively”?? Line 115: “optimal number” optimal in what sense? Line 97: is it really necessary to give a formula for the covariance matrix, especially since this formula is not the best way to actually compute it.
The small size (120000) of the training database is problematic because the computation of the 8461*8461 covariance matrix will be affected by instrument noise as well as unusual spectra (which can be hard to avoid).
There is no evidence presented for the usefulness of the thinning towards the poles “in order to not over-represent high latitudes”.
Line 131: then? And how can a random selection help to “represent all the conditions” – should be better to keep all.
Section 3.3: 20? Actually 1 PC is enough to “depict (sic!) most of the atmospheric variability”. While 150 PC is a good choice this is not related to any percentage of the total variance but rather the signal to noise ratio of the remaining PCs.
Section 4:
The details, especially regarding the detection thresholds, are hard to understand and should be rewritten. Also I miss justifications for the choices made.
Why not apply thresholds for each individual spectrum, instead of the granule min and max? Faster (line 152)? No, I don’t believe so.
The use of 3 different thresholds seems unnecessarily complicated. And the second threshold might be counterproductive in case most of the granule is affected by a similar anomaly.
The term “signal intensity” is used without being introduced. It is simply the (absolute value of) the (noise normalised) residual and the new term does only confuse.
It is not clear why a second training set was used for the computation of thresholds. Why different threshold for day and night?
Line 442-443 and 506-507: This is what was already discussed on page 10. I feel it is wrong to talk about “artefact” and “reconstruction error” in this context. The detection method of the paper is nothing but the identification of reconstruction error. That an unusual perturbation in a limited number of channels can affect the reconstruction residual in other channels is natural and unavoidable (as you can convince yourself by looking at a two dimensional space with 1 retained PC). Actually maybe the biggest contribution of the paper is that is shows that this kind of “cross talk” seems to be relatively rare in practice.
Technical corrections
Line 19: horizontal => spatial
Line 120: insert “as”
Line 145: “maximum of information”?
Line 154: noise normalised “residuals”
Line 182: I don’t understand this sentence
Citation: https://doi.org/10.5194/egusphere-2022-1372-RC1 -
AC1: 'Reply on RC1', Adrien Vu Van, 17 Feb 2023
The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2022/egusphere-2022-1372/egusphere-2022-1372-AC1-supplement.pdf
-
AC1: 'Reply on RC1', Adrien Vu Van, 17 Feb 2023
-
RC2: 'Comment on egusphere-2022-1372', Anonymous Referee #1, 12 Jan 2023
Review of “Near-real time detection of unexpected atmospheric events using Principal Component Analysis on the IASI radiances” by Adrien Vu Van et al.
The manuscript describes a PCA based method on real time detection and characterization of atmospheric events. For this, they are applying measured data from three IASI satellites. The manuscript is nicely written and offers interesting application of the PCA on detection of extreme events. However, some clarifications are needed, as described below.
Major comment
The methodology description needs to be improved. Even though majority of the methodology is described in previous study, it would be important to provide here necessary details on the method for replicating the analyses with similar data. e.g. PCA could be described more clearly and it is not clear how to you get GMI and GMA from the PC’s. This makes it more difficult to follow the results from the case studies.
Specific comments
Abstract: Point out the focus and true novelty of this manuscript in the abstract. Now it sounds more like the introduction
line 117, Antonelli 2004 is not in the list of references. In addition, with Atkinson 2008 and 2010 they are not the original or the best references of methods for defining the optimal number of components
Line 173: Define IASI-PCA-GE more clearly
Section 3.3.: As the explained variance is not really increasing after ~25 components, using 150 PC sounds a bit of overfitting. How did you define the number? How many PC would e.g. Scree test or Kaiser criterion suggest?
Lines 501-506: With this high number of observations in the training set, it is not probable that few outliers would affect drastically to the sensitivity of the method. As already the Atkinson papers pointed out, there has been suspicions that PCA might not be the best method for this type of analysis. Have you considered other possible factorization methods like EFA, NMF or PMF discussed e.g. in Isokääntä et al. 2020 (https://doi.org/10.5194/amt-13-2995-2020)?
In addition, have you considered accounting for the geophysical parameter possibly acting as confounding factors in your analysis?Conclusions: point out that the method can be used as online tool for detecting extreme events, as mentioned in the text earlier.
Citation: https://doi.org/10.5194/egusphere-2022-1372-RC2 -
AC2: 'Reply on RC2', Adrien Vu Van, 17 Feb 2023
The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2022/egusphere-2022-1372/egusphere-2022-1372-AC2-supplement.pdf
-
AC2: 'Reply on RC2', Adrien Vu Van, 17 Feb 2023
Peer review completion
Journal article(s) based on this preprint
Adrien Vu Van et al.
Adrien Vu Van et al.
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 243 | 141 | 14 | 398 | 5 | 6 |
- HTML: 243
- PDF: 141
- XML: 14
- Total: 398
- BibTeX: 5
- EndNote: 6
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(1848 KB) - Metadata XML