Using machine learning algorithm to retrieve cloud fraction based on FY-4A AGRI observations

Xia, Jinyi; Guan, Li

doi:https://doi.org/10.5194/egusphere-2024-977

Preprints

https://doi.org/10.5194/egusphere-2024-977

Preprints

10 Jun 2024

| 10 Jun 2024

Using machine learning algorithm to retrieve cloud fraction based on FY-4A AGRI observations

Jinyi Xia and Li Guan

Abstract. Cloud fraction as a vital component of meteorological satellite products plays an essential role in environmental monitoring, disaster detection, climate analysis and other research areas. A long short-term memory (LSTM) machine learning algorithm is used in this paper to retrieve the cloud fraction of AGRI (Advanced Geosynchronous Radiation Imager) onboard FY-4A satellite based on its full-disc level-1 radiance observation. Correction has been made subsequently to the retrieved cloud fraction in areas where solar glint occurs using a correction curve fitted with sun-glint angle as weight. The algorithm includes two steps: the cloud detection is conducted firstly for each AGRI field of view to identify whether it is clear sky, partial cloud or overcast cloud coverage within the observation field. Then the cloud fraction is retrieved for the scene identified as partly cloudy. The 2B-CLDCLASS-LIDAR cloud fraction product from Cloudsat& CALIPSO active remote sensing satellite is employed as the truth to assess the accuracy of the retrieval algorithm. Comparison with the operational AGRI level 2 cloud fraction product is also conducted at the same time. During daytime, the probability of detection (POD) for clear sky, partly cloudy, and overcast scenes in the official operational cloud detection product were 0.5359, 0.7041, and 0.7826, respectively. The POD for cloud detection using the LSTM algorithm were 0.8294, 0.7223, and 0.8435. While the operational product often misclassified clear sky scenes as cloudy, the LSTM algorithm improved the discrimination of clear sky scenes, albeit with a higher false alarm rate compared to the operational product. For partly cloudy scenes, the mean error (ME) and root-mean-square error (RMSE) of the operational product were 0.2374 and 0.3269. The LSTM algorithm exhibited lower ME (0.1134) and RMSE (0.1897) than the operational product. The large reflectance in the sun-glint region resulted in significant cloud fraction retrieval errors using the LSTM algorithm. However, after applying the correction, the accuracy of cloud cover retrieval in this region greatly improved. During nighttime, the LSTM model demonstrated improved POD for clear sky and partly cloudy scenes compared to the operational product, while maintaining a similar POD value for overcast scenes and a lower false alarm rate. For partly cloudy scenes at night, the operational product exhibited a positive mean error, indicating an overestimation of cloud cover, whereas the LSTM model showed a negative mean error, indicating an underestimation of cloud cover. The LSTM model also exhibited a lower RMSE compared to the operational product.

Received: 01 Apr 2024 – Discussion started: 10 Jun 2024

Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this preprint. The responsibility to include appropriate place names lies with the authors.

Download & links

Preprint (PDF, 1171 KB)

Notice on discussion status
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
Preprint (1171 KB)

Download & links

The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.

Journal article(s) based on this preprint

25 Nov 2024

Retrieval of cloud fraction using machine learning algorithms based on FY-4A AGRI observations

Jinyi Xia and Li Guan

Atmos. Meas. Tech., 17, 6697–6706, https://doi.org/10.5194/amt-17-6697-2024,https://doi.org/10.5194/amt-17-6697-2024, 2024

Short summary

Jinyi Xia and Li Guan

Interactive discussion

Status: closed

RC1:
'Comment on egusphere-2024-977', Anonymous Referee #1, 30 Jun 2024
General comments:
In this manuscript, the authors present a machine learning based cloud cover and fraction retrieval framework from AGRI observations onboard FY-4 satellites using LSTM network. As a reviewer who did the initial assessment before it came to a full review, I find the manuscript largely improved by the authors with the majority of my concerns addressed. However, there still exist some minor issues in the contents that could possibly confuse the readers. Therefore, I would suggest a minor revision before it gets published in this journal.
Specific comments:
Line 34~35, ‘the accuracy,… greatly improved’ should be ‘the accuracy gets greatly improved’.

Line 82, ‘is the based on’ should be ‘is based on’.

Line 83, ‘Such as’ should be ‘For example,’ or ‘For instance’.

Line 123, ‘It has total 14 channels’ should be ‘It has 14 channels in total’.

Line 129, when referring to spatial resolution of satellite imagers, it should be mentioned as ‘4km at nadir’ as the pixel size varies by geolocation.

Line 154, ‘for download’ should be ‘to download’.

Line 163, ‘distance difference’ should be ’spatial distance’.

Line 241, ‘hidden layer size’ can be confusing for readers whether that is the number of hidden layers, or the number of neurons on each hidden layer. Personally I would believe the number 3 refers to the number of hidden layers. But the saying on Line 238 which reads as ‘having a hidden layer that is too large’ also seems to indicate the latter meaning. Please explain in a clear way.

Line 251~252, are both models A and B using the same settings of batch size, optimizer and loss function? As far as I know, classification models (model A) and regression models (model B) usually use different loss functions.

Line 278, ‘convercast’ should be ‘overcast’.

Line 309, ‘inverted’ should be ‘retrieved’.

Line 310, ‘in this text’ should be ‘in this paper’.

Line 311, there should be an ‘and’ before ‘operational products’.

Line 315, again ‘inverted’, recommend the authors revise all ‘invert’ when referring to retrievals.

Line 349~352, the description of components (lines, dots, etc.) on Figure 2(b) should also appear in the caption of Figure 2 so that readers don’t need to look for the sentences when looking at the figure itself.

Table 4, what truth value are the statistics calculated against? According to Line 190~191 in this manuscript, 2B-CLDCLASS-LIDAR data is only available before July 2019.
Citation: https://doi.org/10.5194/egusphere-2024-977-RC1
- AC1: 'Reply on RC1', Jinyi Xia, 30 Jun 2024
  
  Thank you for pointing out the issues, here are my responses to all the comments:
  1-7: I will make corrections in the next round of manuscript submission.
  8: 'hidden layer size' refers to the number of hidden layers. I will clarify this in the next manuscript submission.
  9: The batch size for the classification model is 500 (line 251), and for the regression model is 60 (line 266). Both models use the Adam optimizer (lines 256 and 267), the classification model uses cross-entropy as the loss function (line 252), and the regression model uses mean squared error (line 268).
  10-15: I will make corrections in the next round of manuscript submission.
  16: Tables 4 and 5 present errors in the sun glint region during June to July 2019, with the ground truth being 2B-CLDCLASS-LIDAR data. Line 190 contains an error. Data before August 2019 is available.
  
  Citation: https://doi.org/10.5194/egusphere-2024-977-AC1
RC2:
'Comment on egusphere-2024-977', Anonymous Referee #3, 30 Jun 2024
Review comments for “Using machine learning algorithm 1 to retrieve cloud fraction based on FY-4A AGRI observations” by Xia, J. and Guan, L.

General comments:
This manuscript presents a novel application of a Long Short-Term Memory (LSTM) machine learning algorithm to retrieve cloud fraction using data from the FY-4A AGRI satellite. The study addresses an important aspect of meteorological research, providing a valuable contribution to the field by improving cloud detection and fraction retrieval accuracy. The manuscript is well-structured and comprehensive, detailing the methodology, data sources, and validation processes effectively. However, there are areas where the manuscript could be further strengthened. The introduction could benefit from a deeper explanation of why LSTM models are particularly well-suited for this application compared to other machine learning techniques. Additionally, while the results are promising, a more thorough discussion of potential biases and limitations, especially regarding the higher false alarm rates observed with the LSTM model, would provide a balanced view. Including confidence intervals or significance tests for the reported metrics would also add to the rigor of the statistical analysis.
The manuscript has the potential to be a valuable resource for researchers and practitioners in meteorology and environmental monitoring, but significant improvements should be made before.

Specific Comments
The manuscript primarily discusses the method, but the abstract and the content focuses only on the results. There is insufficient explanation about the method (machine learning and correction), its characteristics, and why it leads to improvements.

The manuscript lacks a discussion about the impact of different machine learning structures on the retrieval of cloud fraction. Please provide more results regarding this.

The data resolution of CloudSat and CALIPSO is not consistent with AGRI. It is crucial to discuss this dataset uncertainty and its impact on the retrieval.

The captions of the figures and tables are too simplified and do not provide essential information.

Line 63: What is the spatial resolution of the instrument?

Lines 65-81: Summarize these old and classic references, emphasizing only the important ones.

Lines 98-113: The logic is not clear. Why did the authors conduct this work, and what is the advantage of the LSTM compared to other frameworks?

Table 1: A reference should be provided.

Lines 149-155: Since cloud fraction is important for building the training dataset, the algorithm for the joint product should be introduced. Additionally, how the uncertainty in the dataset affects the retrieval should also be discussed.

Lines 253-258: When discussing LSTM, the use of time series information is implied. Is this the case in your study?

Minor corrections:
Lines 11, 43,62, 275 and etc., add a comma before the “and” or “or” in a list

Line 15, “Correction” should be “Corrections”

Line 49, “determined” is too strong, it should be “influenced”

Line 66, better to replace “encompassing” with “including”

Line 81, it should be made clear that why “season” and “climate” will influence the thresholds?

Line 203, “and demonstrates” not correct, or just remove the period

Line 206, remove the extra comma

Line 276, remove the quotation mark, and add “the” before the POD and FAR

Line 283, “retrieving cloud fraction”-> “retrieving cloud fractions”

Line 301, “Operational” -> “operational”

Lines 308-309, Line 317, and etc. , why did you use “cloud amount” and “inverted” instead of “cloud fractions” and “retrieved”?

Lines 467-471, use the abbr. “FAR” instead of “false alarm rate”

Lines 473-478, it is not appropriate to compare the “algorithm” to the “operation product”, so replace the “than” in Line 475 and 478 with “compared to”
Citation: https://doi.org/10.5194/egusphere-2024-977-RC2
- AC2:
  'Reply on RC2', Jinyi Xia, 30 Jun 2024
  Thank you for your comments on my manuscript, here is my response.
  I will provide explanations on the methods (machine learning and corrections) in the next manuscript update.
  
  I conducted experiments on the impact of different machine learning algorithms on cloud fraction retrieval, and I will include this content in the manuscript.
  
  The data resolution of CloudSat and CALIPSO differs from AGRI. FY-4A AGRI and 2B-CLDCLASS-LIDAR data are spatially matched with a distance and time difference of under 1.5 km and 15 minutes respectively. For effective collocation of 2B-CLDCLASS-LIDAR cloud fraction data within AGRI pixels, a minimum of two 2B-CLDCLASS-LIDAR pixels are needed within each AGRI field of view. The average cloud fraction of these pixels is used for that AGRI pixel.
  
  I will modify the titles of the figures and tables in the next manuscript upload.
  
  The spatial resolution of FY4A AGRI is 4km; I will add this information.
  
  Thank you for your suggestions, I will make corrections according to your feedback.
  
  I only pointed out the shortcomings of the threshold method, with LSTM performing better than the threshold method. I did not describe the advantages of LSTM compared to other frameworks. I will amend this based on the feedback.
  
  I will provide the reference for Table 1.
  
  I will add an introduction to the joint product algorithm.
  
  This paper only uses the LSTM algorithm for classification and regression, excluding time series information.
  
  For minor corrections, I will carefully make revisions and thoroughly recheck the entire text.
  
  Citation: https://doi.org/10.5194/egusphere-2024-977-AC2
RC3:
'Comment on egusphere-2024-977', Anonymous Referee #2, 01 Jul 2024
General comments:
A simple LSTM performs detection and cloud fraction regression, using FY-4A AGRI data as input, with labels based on 2B-CLDCLASS-LIDAR. The task is important for meteorological and climate applications, and the authors do an excellent job of providing motivation for their work. However, the proposed machine learning method is unusual for a few reasons, and the unusual aspects are not justified by much explanation or comparison with other, more standard machine learning practices.
If the authors complete some major changes, this work has the potential to be valuable.
Line-by-line comments:
191-196: Why do you only use data from May and June 2019? FY-4A was launched earlier, so shouldn’t there be more data available? It would be good to take a random sample from a larger window of time with more seasonal variability. If there is some other reason only to use this time window, it needs to be stated here.

239-241: “Typically, the optimal size of the hidden layer is determined by trying different sizes and evaluating their performance on a validation set.” Correct. Did you do this? If so, the results should be in the paper.

241: Your hidden layer sizes are extremely small. Also, you need to state how many hidden layers you have. My subsequent analysis will assume you only have 1 hidden layer. Even so, this needs to be explicitly stated.

244-246: This is simply a restatement of the definition of batch size, and not an explanation. I’d remove both sentences. Instead, you can simply say something like “A larger batch size typically reduces the training time per epoch for complex reasons” and then cite some paper about batch size, e.g. [1].

254: Are the inputs to the network normalized? If not, they should be.

267: “a batch size of 60 is chosen due to the limited sample number in dataset B.” If you have limited data, that doesn’t mean you need to decrease the batch size. Given the extremely small size of your LSTM, I don’t think you will have a problem with overfitting, so you can safely use as large of a batch size as you have the VRAM for.

268: This is fine for the cloud fraction task, but you shouldn’t / can’t use MSE for the cloud detection task, as it is a classification and not a regression.

Figure 2:
(a) circular lines should be thicker, text should be larger

(b) missing a legend for the colors

(c) put the degree sign ° on the tick labels in the color bar

(d) missing a legend for the colors, and you should use different marker types (e.g. triangles and circles) to help colorblind readers

Major changes requested:
You use a temporal / recurrent network architecture (LSTM) but have no description of how the temporal aspect of your data is used by the network. You need to describe this, in detail. What are the timesteps? Is there a warmup time for your LSTM (a minimum time before it attains a stable state and a reasonable accuracy)? On the other hand, if there is no time series information in use here, you should not be using an LSTM!

There needs to be some comparison with other models. A (single-layer?) network with only 3 hidden units is a very unusual choice. This is a tiny neural network. You could write down the full definition of this network on a piece of paper.
Why even use a neural network at that point, instead of something like a random forest? The latter would be more interpretable, easier to train / justify, and potentially more powerful than your very small model.

If you’re going to keep such a small network, I’d like to see a comparison with linear regression.

How does it compare with a larger LSTM?

Is the temporal component helpful (i.e. how does the LSTM compare with a standard MLP)?

Why not make use of spatial context? You could consider using a convolutional neural network or a ConvLSTM.

I think you should use a larger dataset, or at least one that isn’t restricted to one month. There is seasonal variability to clouds, and you want your algorithm to be effective year-round.

You are comparing your network with an operational cloud product. This is not a very fair comparison, as the operational cloud product is designed with different goals and principles (and physics) in mind than 2B-CLDCLASS-LIDAR was, whereas your method is directly trained on the 2B-CLDCLASS-LIDAR product.
You should spend more time breaking down these differences

Having a reasonable standard baseline (like a regular neural network or a random forest) would further help contextualize your results.

References
[1] Smith, Samuel L., et al. "Don't decay the learning rate, increase the batch size." arXiv preprint arXiv:1711.00489 (2017).
Citation: https://doi.org/10.5194/egusphere-2024-977-RC3
- AC3:
  'Reply on RC3', Jinyi Xia, 16 Jul 2024
  Thank you for your comments, here is my reply:
  Due to the lack of time series information in this paper, using LSTM is not reasonable, so I have switched to using the Random Forest algorithm.
  
  A single observation from FY4A AGRI, the northern and southern hemispheres contain data from different seasons, climates, and surface types. Therefore, the training dataset required for training the model does not need to cover a long period of time.
  
  Additional algorithms for the 2B-CLDCLASS-LIDAR product were provided, along with an analysis of its differences from operational products. The 2B-CLDCLASS-LIDAR product, derived from the active remote sensing instrument CPR-CALIOP, is currently the most accurate cloud fraction product. When evaluating the accuracy of cloud retrieval algorithms against operational products, using 2B-CLDCLASS-LIDAR as the reference value is the only choice.
  
  Citation: https://doi.org/10.5194/egusphere-2024-977-AC3

Interactive discussion

Status: closed

RC1:
'Comment on egusphere-2024-977', Anonymous Referee #1, 30 Jun 2024
General comments:
In this manuscript, the authors present a machine learning based cloud cover and fraction retrieval framework from AGRI observations onboard FY-4 satellites using LSTM network. As a reviewer who did the initial assessment before it came to a full review, I find the manuscript largely improved by the authors with the majority of my concerns addressed. However, there still exist some minor issues in the contents that could possibly confuse the readers. Therefore, I would suggest a minor revision before it gets published in this journal.
Specific comments:
Line 34~35, ‘the accuracy,… greatly improved’ should be ‘the accuracy gets greatly improved’.

Line 82, ‘is the based on’ should be ‘is based on’.

Line 83, ‘Such as’ should be ‘For example,’ or ‘For instance’.

Line 123, ‘It has total 14 channels’ should be ‘It has 14 channels in total’.

Line 129, when referring to spatial resolution of satellite imagers, it should be mentioned as ‘4km at nadir’ as the pixel size varies by geolocation.

Line 154, ‘for download’ should be ‘to download’.

Line 163, ‘distance difference’ should be ’spatial distance’.

Line 241, ‘hidden layer size’ can be confusing for readers whether that is the number of hidden layers, or the number of neurons on each hidden layer. Personally I would believe the number 3 refers to the number of hidden layers. But the saying on Line 238 which reads as ‘having a hidden layer that is too large’ also seems to indicate the latter meaning. Please explain in a clear way.

Line 251~252, are both models A and B using the same settings of batch size, optimizer and loss function? As far as I know, classification models (model A) and regression models (model B) usually use different loss functions.

Line 278, ‘convercast’ should be ‘overcast’.

Line 309, ‘inverted’ should be ‘retrieved’.

Line 310, ‘in this text’ should be ‘in this paper’.

Line 311, there should be an ‘and’ before ‘operational products’.

Line 315, again ‘inverted’, recommend the authors revise all ‘invert’ when referring to retrievals.

Line 349~352, the description of components (lines, dots, etc.) on Figure 2(b) should also appear in the caption of Figure 2 so that readers don’t need to look for the sentences when looking at the figure itself.

Table 4, what truth value are the statistics calculated against? According to Line 190~191 in this manuscript, 2B-CLDCLASS-LIDAR data is only available before July 2019.
Citation: https://doi.org/10.5194/egusphere-2024-977-RC1
- AC1: 'Reply on RC1', Jinyi Xia, 30 Jun 2024
  
  Thank you for pointing out the issues, here are my responses to all the comments:
  1-7: I will make corrections in the next round of manuscript submission.
  8: 'hidden layer size' refers to the number of hidden layers. I will clarify this in the next manuscript submission.
  9: The batch size for the classification model is 500 (line 251), and for the regression model is 60 (line 266). Both models use the Adam optimizer (lines 256 and 267), the classification model uses cross-entropy as the loss function (line 252), and the regression model uses mean squared error (line 268).
  10-15: I will make corrections in the next round of manuscript submission.
  16: Tables 4 and 5 present errors in the sun glint region during June to July 2019, with the ground truth being 2B-CLDCLASS-LIDAR data. Line 190 contains an error. Data before August 2019 is available.
  
  Citation: https://doi.org/10.5194/egusphere-2024-977-AC1
RC2:
'Comment on egusphere-2024-977', Anonymous Referee #3, 30 Jun 2024
Review comments for “Using machine learning algorithm 1 to retrieve cloud fraction based on FY-4A AGRI observations” by Xia, J. and Guan, L.

General comments:
This manuscript presents a novel application of a Long Short-Term Memory (LSTM) machine learning algorithm to retrieve cloud fraction using data from the FY-4A AGRI satellite. The study addresses an important aspect of meteorological research, providing a valuable contribution to the field by improving cloud detection and fraction retrieval accuracy. The manuscript is well-structured and comprehensive, detailing the methodology, data sources, and validation processes effectively. However, there are areas where the manuscript could be further strengthened. The introduction could benefit from a deeper explanation of why LSTM models are particularly well-suited for this application compared to other machine learning techniques. Additionally, while the results are promising, a more thorough discussion of potential biases and limitations, especially regarding the higher false alarm rates observed with the LSTM model, would provide a balanced view. Including confidence intervals or significance tests for the reported metrics would also add to the rigor of the statistical analysis.
The manuscript has the potential to be a valuable resource for researchers and practitioners in meteorology and environmental monitoring, but significant improvements should be made before.

Specific Comments
The manuscript primarily discusses the method, but the abstract and the content focuses only on the results. There is insufficient explanation about the method (machine learning and correction), its characteristics, and why it leads to improvements.

The manuscript lacks a discussion about the impact of different machine learning structures on the retrieval of cloud fraction. Please provide more results regarding this.

The data resolution of CloudSat and CALIPSO is not consistent with AGRI. It is crucial to discuss this dataset uncertainty and its impact on the retrieval.

The captions of the figures and tables are too simplified and do not provide essential information.

Line 63: What is the spatial resolution of the instrument?

Lines 65-81: Summarize these old and classic references, emphasizing only the important ones.

Lines 98-113: The logic is not clear. Why did the authors conduct this work, and what is the advantage of the LSTM compared to other frameworks?

Table 1: A reference should be provided.

Lines 149-155: Since cloud fraction is important for building the training dataset, the algorithm for the joint product should be introduced. Additionally, how the uncertainty in the dataset affects the retrieval should also be discussed.

Lines 253-258: When discussing LSTM, the use of time series information is implied. Is this the case in your study?

Minor corrections:
Lines 11, 43,62, 275 and etc., add a comma before the “and” or “or” in a list

Line 15, “Correction” should be “Corrections”

Line 49, “determined” is too strong, it should be “influenced”

Line 66, better to replace “encompassing” with “including”

Line 81, it should be made clear that why “season” and “climate” will influence the thresholds?

Line 203, “and demonstrates” not correct, or just remove the period

Line 206, remove the extra comma

Line 276, remove the quotation mark, and add “the” before the POD and FAR

Line 283, “retrieving cloud fraction”-> “retrieving cloud fractions”

Line 301, “Operational” -> “operational”

Lines 308-309, Line 317, and etc. , why did you use “cloud amount” and “inverted” instead of “cloud fractions” and “retrieved”?

Lines 467-471, use the abbr. “FAR” instead of “false alarm rate”

Lines 473-478, it is not appropriate to compare the “algorithm” to the “operation product”, so replace the “than” in Line 475 and 478 with “compared to”
Citation: https://doi.org/10.5194/egusphere-2024-977-RC2
- AC2:
  'Reply on RC2', Jinyi Xia, 30 Jun 2024
  Thank you for your comments on my manuscript, here is my response.
  I will provide explanations on the methods (machine learning and corrections) in the next manuscript update.
  
  I conducted experiments on the impact of different machine learning algorithms on cloud fraction retrieval, and I will include this content in the manuscript.
  
  The data resolution of CloudSat and CALIPSO differs from AGRI. FY-4A AGRI and 2B-CLDCLASS-LIDAR data are spatially matched with a distance and time difference of under 1.5 km and 15 minutes respectively. For effective collocation of 2B-CLDCLASS-LIDAR cloud fraction data within AGRI pixels, a minimum of two 2B-CLDCLASS-LIDAR pixels are needed within each AGRI field of view. The average cloud fraction of these pixels is used for that AGRI pixel.
  
  I will modify the titles of the figures and tables in the next manuscript upload.
  
  The spatial resolution of FY4A AGRI is 4km; I will add this information.
  
  Thank you for your suggestions, I will make corrections according to your feedback.
  
  I only pointed out the shortcomings of the threshold method, with LSTM performing better than the threshold method. I did not describe the advantages of LSTM compared to other frameworks. I will amend this based on the feedback.
  
  I will provide the reference for Table 1.
  
  I will add an introduction to the joint product algorithm.
  
  This paper only uses the LSTM algorithm for classification and regression, excluding time series information.
  
  For minor corrections, I will carefully make revisions and thoroughly recheck the entire text.
  
  Citation: https://doi.org/10.5194/egusphere-2024-977-AC2
RC3:
'Comment on egusphere-2024-977', Anonymous Referee #2, 01 Jul 2024
General comments:
A simple LSTM performs detection and cloud fraction regression, using FY-4A AGRI data as input, with labels based on 2B-CLDCLASS-LIDAR. The task is important for meteorological and climate applications, and the authors do an excellent job of providing motivation for their work. However, the proposed machine learning method is unusual for a few reasons, and the unusual aspects are not justified by much explanation or comparison with other, more standard machine learning practices.
If the authors complete some major changes, this work has the potential to be valuable.
Line-by-line comments:
191-196: Why do you only use data from May and June 2019? FY-4A was launched earlier, so shouldn’t there be more data available? It would be good to take a random sample from a larger window of time with more seasonal variability. If there is some other reason only to use this time window, it needs to be stated here.

239-241: “Typically, the optimal size of the hidden layer is determined by trying different sizes and evaluating their performance on a validation set.” Correct. Did you do this? If so, the results should be in the paper.

241: Your hidden layer sizes are extremely small. Also, you need to state how many hidden layers you have. My subsequent analysis will assume you only have 1 hidden layer. Even so, this needs to be explicitly stated.

244-246: This is simply a restatement of the definition of batch size, and not an explanation. I’d remove both sentences. Instead, you can simply say something like “A larger batch size typically reduces the training time per epoch for complex reasons” and then cite some paper about batch size, e.g. [1].

254: Are the inputs to the network normalized? If not, they should be.

267: “a batch size of 60 is chosen due to the limited sample number in dataset B.” If you have limited data, that doesn’t mean you need to decrease the batch size. Given the extremely small size of your LSTM, I don’t think you will have a problem with overfitting, so you can safely use as large of a batch size as you have the VRAM for.

268: This is fine for the cloud fraction task, but you shouldn’t / can’t use MSE for the cloud detection task, as it is a classification and not a regression.

Figure 2:
(a) circular lines should be thicker, text should be larger

(b) missing a legend for the colors

(c) put the degree sign ° on the tick labels in the color bar

(d) missing a legend for the colors, and you should use different marker types (e.g. triangles and circles) to help colorblind readers

Major changes requested:
You use a temporal / recurrent network architecture (LSTM) but have no description of how the temporal aspect of your data is used by the network. You need to describe this, in detail. What are the timesteps? Is there a warmup time for your LSTM (a minimum time before it attains a stable state and a reasonable accuracy)? On the other hand, if there is no time series information in use here, you should not be using an LSTM!

There needs to be some comparison with other models. A (single-layer?) network with only 3 hidden units is a very unusual choice. This is a tiny neural network. You could write down the full definition of this network on a piece of paper.
Why even use a neural network at that point, instead of something like a random forest? The latter would be more interpretable, easier to train / justify, and potentially more powerful than your very small model.

If you’re going to keep such a small network, I’d like to see a comparison with linear regression.

How does it compare with a larger LSTM?

Is the temporal component helpful (i.e. how does the LSTM compare with a standard MLP)?

Why not make use of spatial context? You could consider using a convolutional neural network or a ConvLSTM.

I think you should use a larger dataset, or at least one that isn’t restricted to one month. There is seasonal variability to clouds, and you want your algorithm to be effective year-round.

You are comparing your network with an operational cloud product. This is not a very fair comparison, as the operational cloud product is designed with different goals and principles (and physics) in mind than 2B-CLDCLASS-LIDAR was, whereas your method is directly trained on the 2B-CLDCLASS-LIDAR product.
You should spend more time breaking down these differences

Having a reasonable standard baseline (like a regular neural network or a random forest) would further help contextualize your results.

References
[1] Smith, Samuel L., et al. "Don't decay the learning rate, increase the batch size." arXiv preprint arXiv:1711.00489 (2017).
Citation: https://doi.org/10.5194/egusphere-2024-977-RC3
- AC3:
  'Reply on RC3', Jinyi Xia, 16 Jul 2024
  Thank you for your comments, here is my reply:
  Due to the lack of time series information in this paper, using LSTM is not reasonable, so I have switched to using the Random Forest algorithm.
  
  A single observation from FY4A AGRI, the northern and southern hemispheres contain data from different seasons, climates, and surface types. Therefore, the training dataset required for training the model does not need to cover a long period of time.
  
  Additional algorithms for the 2B-CLDCLASS-LIDAR product were provided, along with an analysis of its differences from operational products. The 2B-CLDCLASS-LIDAR product, derived from the active remote sensing instrument CPR-CALIOP, is currently the most accurate cloud fraction product. When evaluating the accuracy of cloud retrieval algorithms against operational products, using 2B-CLDCLASS-LIDAR as the reference value is the only choice.
  
  Citation: https://doi.org/10.5194/egusphere-2024-977-AC3

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision | EF: Editorial file upload

AR by Jinyi Xia on behalf of the Authors (17 Jul 2024) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (23 Jul 2024) by Jian Xu

RR by Anonymous Referee #2 (25 Jul 2024)

Suggestions for revision or reasons for rejection

Overview:

The authors use a random forest algorithm to estimate cloud fraction from a geosynchronous, multispectral imager. The task is well motivated, and the data is well explained. However, the methodology used is not sufficiently justified. It is unclear exactly what algorithm the authors are using and whether their choices of hyperparameters are reasonable. Additionally, comparing the results from version 4 with results from version 3 of the paper suggests the current method is not as effective as it should be. The authors need to compare their results with other methods to show their competitiveness. Comparing with an operational product only shows that the operational product can be improved, but is not enough to show that the proposed method is a substantial contribution to scientific progress.

Comments:

In your response to reviewer 3, you claim: "If the training set is sufficiently accurate, regardless of the machine learning algorithm used, the results will be similar." This is quite wrong. It is also inconsistent with your own results. Version 3 of the paper was, as far as I understand, using an LSTM with 1 layer, 3 hidden units, and no temporal features. This should be equivalent to a 1-layer, 3-hidden-unit multi-layer perceptron (MLP) with a tanh activation function, so I’ll call it an MLP. Version 4 of the paper uses a random forest (RF) instead. The numbers reported for the MLP and the RF are very different, with the RF achieving better performance in some areas and worse performance in others. You reported in V3 that POD for cloud detection with the MLP for clear sky, cloudy, and overcast was 0.8294, 0.7223, and 0.8435, respectively. In V4 you report that these numbers for the RF are 0.6984, 0.8972, and 0.8613, respectively, marking a significant drop in clear sky skill, a significant increase in cloudy skill, and a small increase in overcast skill. With only the results you’ve reported so far, it seems like the choice of algorithm is extremely important.

Given these results, I think that readers will need to be convinced that you have selected a reasonable algorithm with reasonable hyperparameters. Given the wide difference in results between the two algorithms so far, I don’t think you are reaching the ceiling of possible model skill given your data.
• I think you need to experiment with more hyperparameters for the random forest (and report those experiments here) and/or compare with some other baseline algorithms with reasonable hyperparameter choices.
• Given that your RF does not consistently outperform a 3-hidden-unit MLP (an incredibly small network), I’d be surprised if it significantly improves upon a linear regression model or a model that simply outputs the mean of the training data. These would be useful tests for you, and they would help to understand the behavior of your evaluation metrics.
• I expect you can achieve better performance with larger trees / forests, or with an MLP with more than 1 layer and more than 3 hidden units per layer. I think you should try both. Trying more hyperparameters for your RF implementation should be very simple.
• If you already have code that can train an LSTM on your data, it should not be too complicated to try something like a 3-layer MLP with, for example, 16 hidden units per layer. This is still a very simple network, but much more expressive than the 3-unit MLP you tried before. I also think a well-designed convolutional neural network would significantly outperform your current results.

In my previous review, I suggested you use data with broader seasonal coverage. You responded: “A single observation from FY4A AGRI, the northern and southern hemispheres contain data from different seasons, climates, and surface types. Therefore, the training dataset required for training the model does not need to cover a long period of time.” I disagree. While your data covers both hemispheres, there is a relationship between latitude, seasonality, and clouds. I see no reason not to use data from more times of year. If there is too much raw data, you can randomly sample the same amount of data you currently have, but from more seasons / years.

Finally, you need to discuss the numbers achieved by other methods which attempt the same or similar tasks. For cloud detection, your highest recall (POD) is about 0.9, and your highest precision (1 – FAR) is about 0.82, but you also get values as low as 0.67. This seems low to me, especially when these numbers aren’t far from multilayer cloud detection numbers using similar inputs and 2B-CLDCLASS-LIDAR labels (Ding et al 2022. Multilayer detection is a (much) harder task. Still, it’s hard to compare, as you only report precision/recall (POD / FAR) while other papers seem to report accuracy. You should include accuracy alongside POD / FAR in your results. As for the ME / RMSE values, these seem high to me. ME is a little misleading when compared to RMSE, as your positive / negative errors cancel out, but the values are still quite high. Just because your method outperforms the operational product does not mean it is competitive, it just means the operational product is bad. The results (especially in light of the inconsistency between the MLP results and the RF results) suggest to me that you have more work to do when it comes to selecting and training the model. If you can show comparisons with state-of-the-art methods that indicate otherwise, I'll change my mind.

Line-by-line comments:
130: I can’t find Qu et. al. in your citations. I stumbled upon this, and I have not checked all other citations.

139: I tried to look up the Hu et. al. paper, but the DOI links to a different author / title than the one you list.

307: You cite Quesada-Ruiz et. al. 2022, which is a very specific type of random forest algorithm called AFGRRF. Do you use their method, or a more general random forest algorithm? If it is the former case, you need to explicitly state that you use this method as well as briefly summarize the method (here you are only summarizing the general random forest algorithm). If instead you use a more general random forest, you should probably cite a seminal random forest paper, a survey paper, and/or a paper that applies (regular) random forests to similar data.
I’ve noticed that Quesada-Ruiz et. al. have a publicly available implementation of AFGRRF in R. There are plenty of other publicly available implementations of random forests, so I’m curious why you use this method. It is important that you justify the choice of method in the text.

316-332: Citations are needed throughout this section to back up your claims. This is especially true in the absence of experiments showing that your hyperparameter choices are competitive.

323-325: If you’re using sqrt(M) for your Mtry parameter, shouldn’t you be using 4 and 3 instead of 3 and 2? Sqrt(14) is closer to 4 than to 3, and sqrt(8) is closer to 3 than to 2.

324: Which channels are not available at night? You should include this in Table 1.

540: You already abbreviated RMSE, no need to list it again here.

References
L. Ding et al., "Imbalanced Multi-layer Cloud Classification with Advanced Baseline Imager (ABI) and CloudSat/CALIPSO Data," 2022 IEEE International Conference on Big Data (Big Data), Osaka, Japan, 2022, pp. 5902-5909, doi: 10.1109/BigData55660.2022.10020783.

Hide

RR by Anonymous Referee #3 (04 Aug 2024)

ED: Reconsider after major revisions (09 Aug 2024) by Jian Xu

AR by Jinyi Xia on behalf of the Authors (30 Aug 2024) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (11 Sep 2024) by Jian Xu

RR by Anonymous Referee #2 (27 Sep 2024)

Suggestions for revision or reasons for rejection

The paper is much improved. The authors included more details on the ML algorithms which has alleviated most of the concerns I had about the validity of the results. The new results between the RF and MLP are much more consistent than the old results between the RF and LSTM. Still, the paper would benefit from more polish throughout, and some parts of the text are still confusing or inconsistent.

Overall comments:
I am starting to feel confident that I mostly understand your method. Still, it has taken me several re-reads of the paper. Most of your readers will not take the time to understand it as well as your reviewers do. Some simple changes will make the paper much easier to follow:
• You should summarize what the inputs / outputs of your cloud detection and cloud fraction models are. If you can make this into a figure / flowchart, that’s even better. You should include the actual shapes of those inputs / outputs.
• You should consolidate some of your results tables to highlight the comparison between your method and the operational approach.

Line-by-line comments:

210: Instead of “A dataset .. and B dataset” it is clearer to say “dataset A… and dataset B”

218: “To ensure the balance and representativeness of the samples, the proportions of different cloud fraction samples in dataset A are set at 5:1:1:1:1:1:5”
• This makes sense. It would be interesting to see a comparison with the results of your previous approach (the 1:1:1:1:1:1:1 dataset), perhaps in the supplementary material.
• I’m still a little confused about dataset B. In line 190 you say “at least two 2B-CLDCLASS-LIDAR pixels are required within each AGRI field of view. The cloud fraction average of these pixels is used as the cloud fraction for that AGRI pixel.” Are you sampling so that the dataset approximates the 5:1:1:1:1:1:5 ratio? Also, if the labels are averaged between multiple 2B-CLDCLASS-LIDAR pixels, having a discretization like this doesn’t make much sense.

265: You say MLP uses stochastic gradient descent, but you later report that your solver is Adam. These are different optimizers. You can just say “the model’s weights are trained in a supervised manner using backpropagation.”

265: What loss function do you use to train the cloud detection and cloud fraction MLPs? I would assume cross-entropy for the cloud detection model, and MSE for the cloud fraction model, but you need to mention it.

278: “Hidden layer size… Hidden layer neuron count”
Hidden layer size and hidden layer neuron count are the same thing. This explains why in the previous version of your paper I believed your MLP had only 3 neurons. You should instead use “number of hidden layers” to describe what you are currently calling “hidden layer size” and you can use “hidden layer size” to describe what you are currently calling “hidden layer neuron count.” The term “neuron” has become outdated as MLPs / neural networks have drifted away from their original neuroscientific inspiration, and if you need to refer to a single node in a hidden layer you can call it a "hidden unit."

277-288:
Getting results with different numbers of layers and different hidden layer sizes is great. Still, it would be nice to see the actual results. A few notes on this section:
• This should be a table. It is difficult to read at present.
• Since you have all of these results, you should put them in the supplementary material, so interested readers can benefit from your experiments for their own work.
• You don’t need to try so many batch sizes. I would just pick one and keep it consistent across all experiments. Usually it’s a reasonable choice to pick the biggest batch size you can fit into your GPU’s VRAM.
• You don’t need to report results with different activation functions. It is extremely well established in the literature at this point that ReLU and ReLU variants are almost always the best choice.
• You should use the terms themselves and explain what they mean rather than listing variable names (e.g. LearnRateDropPeriod).

311-474: You should find a way to combine the operational product tables with the tables of your method. Directly comparing them will highlight the whole point of your paper.

414: There is a lot of white space in this and other figures. Trimming some white space would make the panels bigger and more legible.
Panel (b):
• “y = 0.8092 + 0.2441”: missing an “x”
• I’m confused by these results. Line 205 reads: “the cloud fraction is the average of cloud fractions at different layers.” Why are the true cloud fraction values in this plot discretized to [0, 0.16, 0.33, …]? Isn’t your model predicting an average? This is related to my above comment about line 218.

459: Panel (a) should have a color bar, like the other panels.

488: Panel (a) should have a color bar. Panel (c) has a color bar that is difficult to read. It should be consistent with panel (b). Also you could trim some white space in this figure, and make it large enough to fill the page width (like Figure 3).

497: “CloudSat” not “Cloudsat”, and “CALIPSO” not “Calypso”

References: I still cannot find the Hu, J. paper anywhere on the internet, with either the DOI or the title. Please make sure all citations have a verifiable link or DOI.

Hide

ED: Publish subject to minor revisions (review by editor) (05 Oct 2024) by Jian Xu

AR by Jinyi Xia on behalf of the Authors (10 Oct 2024) Author's response Author's tracked changes Manuscript

ED: Publish as is (11 Oct 2024) by Jian Xu

AR by Jinyi Xia on behalf of the Authors (11 Oct 2024)

Journal article(s) based on this preprint

25 Nov 2024

Retrieval of cloud fraction using machine learning algorithms based on FY-4A AGRI observations

Jinyi Xia and Li Guan

Atmos. Meas. Tech., 17, 6697–6706, https://doi.org/10.5194/amt-17-6697-2024,https://doi.org/10.5194/amt-17-6697-2024, 2024

Short summary

Jinyi Xia and Li Guan

Viewed

Total article views: 518 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	BibTeX	EndNote
367	119	32	518	18	14

HTML: 367
PDF: 119
XML: 32
Total: 518
BibTeX: 18
EndNote: 14

Views and downloads (calculated since 10 Jun 2024)

Month	HTML	PDF	XML	Total
Jun 2024	107	44	10	161
Jul 2024	144	55	19	218
Aug 2024	35	6	1	42
Sep 2024	17	6	0	23
Oct 2024	35	4	2	41
Nov 2024	29	4	0	33

Cumulative views and downloads (calculated since 10 Jun 2024)

Month	HTML	PDF	XML	Total
Jun 2024	107	44	10	161
Jul 2024	144	55	19	218
Aug 2024	35	6	1	42
Sep 2024	17	6	0	23
Oct 2024	35	4	2	41
Nov 2024	29	4	0	33

Viewed (geographical distribution)

Total article views: 510 (including HTML, PDF, and XML) Thereof 510 with geography defined and 0 with unknown origin.

Country	#	Views	%

Latest update: 25 Nov 2024

Download

The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.

Preprint (1171 KB)
Metadata XML

Short summary

This study presents a method for estimating cloud cover from FY4A AGRI observations using LSTM neural networks. The results demonstrate excellent performance in distinguishing clear sky scenes and reducing errors in cloud cover estimation. It shows significant improvements compared to existing methods.


Total:	0
HTML:	0
PDF:	0
XML:	0