the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 14 Feb 2024
Air mass factor calculation using deep neural network technique for tropospheric NO2 retrieval from space
Abstract. We performed a feasibility study on using deep neural network (DNN) techniques to calculate the air mass factor (AMF) for the satellite remote sensing of tropospheric nitrogen dioxide NO2. Satellite remote sensing in the UV and visible wavelength ranges is widely used to study the emission and distribution of tropospheric NO2, which is one of the most crucial gases in both climate change and air pollution. One of the largest sources of uncertainty in NO2 satellite retrievals is the AMF, especially when enhanced trace gas concentrations are present in the lower troposphere. Computing the AMF is a very time-consuming task that is usually performed using radiative transfer models. In general, the practical use of such models for satellite remote sensing is limited by the available computational power. We constructed two DNN models to calculate the tropospheric AMF (Trop-AMF-Net) and the altitude-dependent box-AMF (Box-AMF-Net). Trop-AMF-Net consists of five multilayer perceptrons and a linear transform. Box-AMF-Net is an encoder-decoder framework that combines Trop-AMF-Net with a long short-term memory network. We prepared two test datasets, one of which reflects the actual observed NO2 measurement pattern and the other of which assumes a uniform distribution. For the former, we assumed that the NO2 observation was performed by the Global Observing SATellite for Greenhouse Gases and Water Cycle (GOSAT-GW), which is planned for launch in 2024. For this test dataset, Trop-AMF-Net could reproduce the tropospheric AMFs with root-mean-squared percentage errors (RMSPE) of 0.121 % and 0.156 % when the model was trained using the same and uniform distributions, respectively. The RMSPEs of the box-AMFs in the troposphere predicted by Box-AMF-Net are 0.284 % for the test and training datasets when the observed pattern was used. The computation time for 10,000 samples using a combination of a central processing unit and graphics processing unit is 3.7 seconds. The RMSPE and computation time are approximately 30 times smaller and two times shorter compared to those of the commonly used look-up table and interpolation technique. Our feasibility study also highlights the importance of training the model with a dataset that is consistent with the test use.
This preprint has been withdrawn.
-
Withdrawal notice
This preprint has been withdrawn.
-
Preprint
(4793 KB)
Interactive discussion
Status: closed
-
CC1: 'Comment on egusphere-2024-194', Ben Veihelmann, 15 Feb 2024
The manuscript deals with the air mass computation (AMF) performed in satellite based NO2 retrievals. Typically, the largest uncertainties in this AMF computation are related to uncertainties in the aerosol and cloud conditions. The experiment described in the manuscript is limited to clear sky conditions, which excludes the most challening observation conditions in this regard. This limitation should be discussed upfront. It may be useful to add a brief discussion on the potential of DNN based approaches to AMF computation in conditions with aerosol and clouds.
Citation: https://doi.org/10.5194/egusphere-2024-194-CC1 -
AC1: 'Reply on CC1', Yajun Xu, 25 Mar 2024
Thank you for your valuable comment.
Drawing from our experience, we consider that the proposed DNN will also perform robustly in scenarios involving aerosols and clouds.
Currently, we are preparing a new dataset for testing the performance of DNN for AMF computation in conditions with aerosol and clouds.
The forthcoming results will be added to the manuscript.Citation: https://doi.org/10.5194/egusphere-2024-194-AC1
-
AC1: 'Reply on CC1', Yajun Xu, 25 Mar 2024
-
RC1: 'Comment on egusphere-2024-194', Anonymous Referee #1, 15 Mar 2024
The manuscript showcases computational advancements through the proposed neural network approach for AMF calculation. While this is positive in this context, the conceptual framework still seems a bit far to operational tools. Particularly, the adoption of the clear-sky assumption sidesteps significant challenges inherent in AMF computation under variable aerosol and cloud conditions, which are part of real-world applicability.
Moreover, the indications of potential overfitting to training data within the models underscore the necessity for a more robust generalization strategy. This aspect is critical for ensuring the models' efficacy beyond controlled or idealized scenarios, which the current study somewhat overlooks.
Despite these shortcomings, it is commendable that the authors present this study as a preliminary framework, highlighting the hurdles that need addressing for operational use. This transparency is necessary for setting the stage for future research and development in this area. By refining the discussion points highlighted in the attached review and making necessary technical adjustments as outlined, the paper holds the potential to contribute to the ongoing dialogue in the field.
- AC2: 'Reply on RC1', Yajun Xu, 25 Mar 2024
-
RC2: 'Comment on egusphere-2024-194', Anonymous Referee #2, 21 Mar 2024
The problem targeted by the paper is speeding up the computation of the air mass index (AMF). The physical model is the radiative transfer model (RTM) that takes into account the interactions between emission, absorption and scattering along the light path, which is computational costly to evaluate. The baseline method to speed up AMF calculation is via using a look up table (LUT), built using discrete samples of the input variable space, and numerical interpolation.
The paper investigates training deep neural networks (DNNs) to replace RTM/LUT in the calculation of AMF and box-AMF. The former can be solved as a 1D regression problem with 5 input variables (SZA, VZA, RAA, surface albedo, terrain height), while the latter can be solved as a sequence prediction problem with the same input variables. Under this framework, the paper proposes two DNN models, Top-AMF-Net and Box-AMF-Net. The former mainly contains multilayer perceptrons (MLPs) with leaky ReLU activation functions, while the latter mainly employs Long Short Term Memory (LSTM) for sequence prediction, in addition to attention modules and the incorporation of Trop-AMF-Net as an encoder.
Training and testing datasets were contructed using RTM. Two types of distributions (normal and uniform) were employed in sampling the input variables to generate the data. The experimental results show favourable results both Trop-AMF-Net and Box-AMF-Net, whereby both DNNs generally outperform LUT+interpolation in terms of accuracy in predicting AMF/Box-AMF, provided that the testing and training data were drawn from the same distributions.
In my opinion, the design of the methods (particularly the DNNs) was sound and the paper was overall well executed. The following suggestions are provided to hopefully add to a decent paper:
1. The results in Fig 7, specifically the No. 1a, 1b and 2b tests, seem to suggest overfitting to the dominant ground truth AMF; that is, the trained model is biased towards predicting the dominant AMF values that exist in the training datasets (which also happened to be the dominant AMF values in the testing dataset which was drawn from the same distribution). More analysis of the result is suggested, e.g., plotting the RMSE of the 2-dimensional visualisations of the data using t-SNE.
2. It would be helpful to analyse if the primary baseline approach (LUT) is given the same number of "parameters" as the DNN alternative, so as to convince that no method was handicapped by restricting its representation/learning capacity. The number of LUT parameters could be the number of points or entries in the LUT.
3. In the testing set which was normally distributed, the LUT was still defined over uniformly sampled grid points; this could be seen as a disadvantage compared to the DNN setting which can rely on similar distributions in the training and testing datasets. An LUT variant where a non-uniform grid is used, e.g., higher resolution in regions where the unform distribution are denser, could be interesting for comparisons.
Citation: https://doi.org/10.5194/egusphere-2024-194-RC2
Interactive discussion
Status: closed
-
CC1: 'Comment on egusphere-2024-194', Ben Veihelmann, 15 Feb 2024
The manuscript deals with the air mass computation (AMF) performed in satellite based NO2 retrievals. Typically, the largest uncertainties in this AMF computation are related to uncertainties in the aerosol and cloud conditions. The experiment described in the manuscript is limited to clear sky conditions, which excludes the most challening observation conditions in this regard. This limitation should be discussed upfront. It may be useful to add a brief discussion on the potential of DNN based approaches to AMF computation in conditions with aerosol and clouds.
Citation: https://doi.org/10.5194/egusphere-2024-194-CC1 -
AC1: 'Reply on CC1', Yajun Xu, 25 Mar 2024
Thank you for your valuable comment.
Drawing from our experience, we consider that the proposed DNN will also perform robustly in scenarios involving aerosols and clouds.
Currently, we are preparing a new dataset for testing the performance of DNN for AMF computation in conditions with aerosol and clouds.
The forthcoming results will be added to the manuscript.Citation: https://doi.org/10.5194/egusphere-2024-194-AC1
-
AC1: 'Reply on CC1', Yajun Xu, 25 Mar 2024
-
RC1: 'Comment on egusphere-2024-194', Anonymous Referee #1, 15 Mar 2024
The manuscript showcases computational advancements through the proposed neural network approach for AMF calculation. While this is positive in this context, the conceptual framework still seems a bit far to operational tools. Particularly, the adoption of the clear-sky assumption sidesteps significant challenges inherent in AMF computation under variable aerosol and cloud conditions, which are part of real-world applicability.
Moreover, the indications of potential overfitting to training data within the models underscore the necessity for a more robust generalization strategy. This aspect is critical for ensuring the models' efficacy beyond controlled or idealized scenarios, which the current study somewhat overlooks.
Despite these shortcomings, it is commendable that the authors present this study as a preliminary framework, highlighting the hurdles that need addressing for operational use. This transparency is necessary for setting the stage for future research and development in this area. By refining the discussion points highlighted in the attached review and making necessary technical adjustments as outlined, the paper holds the potential to contribute to the ongoing dialogue in the field.
- AC2: 'Reply on RC1', Yajun Xu, 25 Mar 2024
-
RC2: 'Comment on egusphere-2024-194', Anonymous Referee #2, 21 Mar 2024
The problem targeted by the paper is speeding up the computation of the air mass index (AMF). The physical model is the radiative transfer model (RTM) that takes into account the interactions between emission, absorption and scattering along the light path, which is computational costly to evaluate. The baseline method to speed up AMF calculation is via using a look up table (LUT), built using discrete samples of the input variable space, and numerical interpolation.
The paper investigates training deep neural networks (DNNs) to replace RTM/LUT in the calculation of AMF and box-AMF. The former can be solved as a 1D regression problem with 5 input variables (SZA, VZA, RAA, surface albedo, terrain height), while the latter can be solved as a sequence prediction problem with the same input variables. Under this framework, the paper proposes two DNN models, Top-AMF-Net and Box-AMF-Net. The former mainly contains multilayer perceptrons (MLPs) with leaky ReLU activation functions, while the latter mainly employs Long Short Term Memory (LSTM) for sequence prediction, in addition to attention modules and the incorporation of Trop-AMF-Net as an encoder.
Training and testing datasets were contructed using RTM. Two types of distributions (normal and uniform) were employed in sampling the input variables to generate the data. The experimental results show favourable results both Trop-AMF-Net and Box-AMF-Net, whereby both DNNs generally outperform LUT+interpolation in terms of accuracy in predicting AMF/Box-AMF, provided that the testing and training data were drawn from the same distributions.
In my opinion, the design of the methods (particularly the DNNs) was sound and the paper was overall well executed. The following suggestions are provided to hopefully add to a decent paper:
1. The results in Fig 7, specifically the No. 1a, 1b and 2b tests, seem to suggest overfitting to the dominant ground truth AMF; that is, the trained model is biased towards predicting the dominant AMF values that exist in the training datasets (which also happened to be the dominant AMF values in the testing dataset which was drawn from the same distribution). More analysis of the result is suggested, e.g., plotting the RMSE of the 2-dimensional visualisations of the data using t-SNE.
2. It would be helpful to analyse if the primary baseline approach (LUT) is given the same number of "parameters" as the DNN alternative, so as to convince that no method was handicapped by restricting its representation/learning capacity. The number of LUT parameters could be the number of points or entries in the LUT.
3. In the testing set which was normally distributed, the LUT was still defined over uniformly sampled grid points; this could be seen as a disadvantage compared to the DNN setting which can rely on similar distributions in the training and testing datasets. An LUT variant where a non-uniform grid is used, e.g., higher resolution in regions where the unform distribution are denser, could be interesting for comparisons.
Citation: https://doi.org/10.5194/egusphere-2024-194-RC2
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 387 | 107 | 31 | 525 | 18 | 19 |
- HTML: 387
- PDF: 107
- XML: 31
- Total: 525
- BibTeX: 18
- EndNote: 19
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1