the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Retrieval of surface solar irradiance from satellite using machine learning: pitfalls and perspectives
Hadrien Verbois
Yves-Marie Saint-Drenan
Vadim Becquet
Benoit Gschwind
Philippe Blanc
Abstract. Knowledge of the solar surface irradiance (SSI) spatial and temporal characteristics is critical in many domains, the first of which is likely solar energy. While meteorological ground stations can provide accurate measurements of SSI locally, they are sparsely distributed worldwide. SSI estimations derived from satellite imagery are thus crucial to gain a finer understanding of the solar resource. To infer SSI from satellite images is, however, not straightforward and it has been the focus of many researchers in the past thirty to forty years. For long, the emphasis has been on empirical models (simple parameterization linking the reflectance to the clear-sky index) and on physical models. Recently, new satellite SSI retrieval methods are emerging, which directly infer the SSI from the satellite images using machine learning. Although only a few such works have been published, their practical efficiency has already been questioned.
The objective of this paper is to better understand the potential and the pitfalls of this new coming family of methods. To do so, simple multi-layer-perceptron (MLP) models are constructed with different training datasets of satellite-based radiance measurements from Meteosat Second Generation (MSG) with collocated SSI ground measurements from Meteo-France. The performance of the models is evaluated on a test dataset independent from the training set both in space and time.
We found that the data-driven model’s performance is very dependent on the training set. On the one hand, even a simple MLP can significantly outperform a state-of-the-art physical retrieval method, provided the training set is sufficiently large and similar enough to the test set. On the other hand, in certain configurations, the data-driven model can dramatically underperform even in stations located close to the training set.
-
Notice on discussion status
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
-
Preprint
(2128 KB)
-
Supplement
(1 KB)
-
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(2128 KB) - Metadata XML
-
Supplement
(1 KB) - BibTeX
- EndNote
- Final revised paper
Journal article(s) based on this preprint
Hadrien Verbois et al.
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2023-243', Anonymous Referee #1, 11 May 2023
GENERAL COMMETS:
- I think this manuscript addresses a very important point, the pitfals and drawbacks of AI for the retrieval of SSI. For example, the analysis of the results at the mediterean stations presented in this manuscript illustrate the challenges associated with machine learning. The net is only able to learn from local relations between reflection, surface albedo, atmosphere and SSI. It knows nothing about physics. Hence, it has to be expected that the performance decreases significantly if it is applied in regions with quite different conditions concerning aerosol load, cloud types, H20 and surface albedo. Within this scope it has to be taken into account that in many regions almost no in-situ data are available for the training or retraining. Hence, no learning of regional relations is possible then, but physical retrieval methods do not have these problems and it would be interesting to see the results between the current network with CAMS in Africa. Hence, the question why AI is needed for the retrieval of SSI should be addressed in more detail in the manuscript.. Further, it is difficult to know what the net has really learned (black box approach), If we do not know what the net learns, we can’t learn either (and our intelligence might expire on the long run). This should be discussed in more detail as well, based on the results presented in the manuscript. These points are partly addressed in the conclusion (e.g. L370ff) but should be discussed in more detail.
- Poor performance of AI could also result from wrong training or training architecture. However, the comparison with the established CAMS show that the training has been done well (5.1.1). This is very good, because it shows that the discussed pitfalls are not due to failures in the training or the chosen training method.
- Figure 5: It seems to me that the main benefit of the machine learning is that it corrects differences in SSI induced by the different viewing geometries of ground based and satellite observations. We are aware of this effect, as significant differences are apparent when SSI retrieved from Meteosat East is compared with those from Meteosat prime for the same regions. So far these effects are not considered in many physical methods e.g. in CAMS, but it might be possible to implement appropriate “slant column” geometry corrections, which would increase the comparability of ground based and satellite based SSI. Please discuss this issue.
- Please consider that other physical retrieval methods might perform better or worse than CAMS, hence that the network might have a lower/higher performance when compared with other models. Please mention this briefly.
- 5.2.2 Impact of aerosols: This is not a really a fair analysis, AOD (and H20) have not been given at predictors for the learning, hence the network could not learn anything about the relation of AOD variations and SSI, SAT reflection. It can just learn locally some kind of mean clear sky state. Contrarily AOD is used in CAMS as “predictor”. Please mention that the performance of ML might be better if AOD data were used as predictor in addition. Of course, it is not easy to find an accurate AOD raster data set, but this problem concerns AI as well as phyical methods. Further, here, AOD from Aeronet is used, which is not available for CAMS elsewhere. Hence the capability of CAMS (or any other sat retrieval) concerning AOD variations is probably much lower as in the example. This should be also mentioned.
- Please add more information about the in-situ data. Do they all have the same maintenance, calibrations cycles and so on. Hence can the same accuracy be expected for all pyranometers ?
- Throughout the manuscript. Please avoid the separation between physical methods and clear sky index methods. They are physical methods as well !
DETAILED COMMENTS:
- Abstract: “the first of which is likely solar energy”. This depends on the viewpoint. Please delete “the first of” and rephrase accordingly, it is also quite important for climate, tourism,...
- Abstract: "For long, the emphasis has been on empirical models (simple parameterization linking the reflectance to the clear-sky index) and on physical models” The use of the clear sky index follows also physical laws, hence please rephrase. Please see also the general comments.
- L25 “…index methods without explicit physical cloud models”, L29 "empirical" please see 2.) and general comments . The use of the clear sky index follows also physical laws and the cloud index is a measure for the cloud transmission, thus, not without physical cloud model, please rephrase
- Line 55 “pixels of ca. 4 by 5 km”. it might be closer to 3.2x5.5 please check.
- L104: “ML model must be fully online” Please explain why ?
- L 195 “Three tricks are applied:” Please use a more appropriate term instead of tricks.
- L 370 please consider to add surface albedo here
- L 385: Another option is to improve the physical methods, without AI e.g. as demonstrated by HelioMon. The accuracy of HelioMont is already close to that of BSRN stations, why fuss with AI ? Please consider to add this option to the manuscript. In the Alps it is questionable if any network would be able to learn the complex relations for all regions, because taking the spatial heterogeneity into account there are not enough ground stations.
Citation: https://doi.org/10.5194/egusphere-2023-243-RC1 -
AC1: 'Reply on RC1', Hadrien Verbois, 30 Jul 2023
Dear reviewer,
Thank you very much for your thorough review. We greatly appreciate the time and effort you committed to reviewing this manuscript. We have endeavored to answer all your points and provided a detailed answer to each of them.
Please find our replies in the supplement pdf, where they are interlaced with your comments.
-
RC2: 'Comment on egusphere-2023-243', Anonymous Referee #2, 27 May 2023
This is a very interesting paper that deals with limitations and perspectives for the calculation of surface solar irradiance (SSI) using machine learning techniques.
The paper deals with an aspect including a number of more or less .. easy to explain, sources of errors and uncertainties. The work is high level and ends up in a publication with unique in my opinion results worth being published in AMT.
Some comments towards manuscript improvement
Abstract
At the moment the abstract is a bit like a general discussion and some metrics there, especially summarized comparisons of ML and CAMS, could be useful for a reader that will be intrigued to read more about it.Introduction
What I am missing is some basic state of the art of current datasets (including CAMS) and their performance evaluation.
Data
AERONET does not provide AOD every minute and also in cloudy days, so some clarification could be included as a short paragraph in 2.2 e.g. how AERONET data used , which wavelength for aerosol optical depth used etc.
Section 3
It is impressive the choice of using 3 hours (12 instants) as a basic hierarchy of the method. Could you explain how this choice has been decided? isn’t it 3 hours relatively .. long ?
The 3 set ups could be of course more complex but I personally find the choice really appropriate here.
I am a bit puzzled by the fact that the kc=1 limitation of CAMS does not have a more visual impact on the statistics. Or is it a major factor of the ML better performance ?
Figure 5: based on the definition given in lines 80 – 85 and the aerosol issues discussed after fig. 5 there should be clear sky index higher than 1 not visible in the figure.
Aerosols: It is clear that the ML inputs does not include any aerosol information so figure 7 is more or less expected. A very rough predictor including an aerosol climatology (more in summer less in winter) would for sure improve this negative correlation shown in fig. 7 . Especially because this has an impact on “high solar irradiance” cases.
Fig. 7 needs a bit more explanation as it is not clear if the points are based on instants, hourly or daily values.
I find difficult to understand how the ML can outperform CAMS for clear skies in the related bins of fig. 5 and still have these aerosol related aspects shown in fig. 7.
Maybe the authors could discuss:
In general it is understandable that the paper does not introduce a method to be used in different areas but it is a kind of sensitivity study on the ML performance. For this case a really unique dataset is used with a huge number of stations. However, it would be nice to comment on perspectives of an actual application of such system. Indirectly this study can assess some kind of realistic cases of limited or not, ground-based data available that can be used for applying such methods in different areas.
The whole France and so many stations is a huge area, but still could be very different than another area with different cloud/aerosol conditions which the same results with the same number of stations and analysis can vary. E.g. aerosol (not captured) effects in N. Africa will have a crucial effect on the statistics as well as areas with different and more clouds.
Finally I can say that problems such as the spatial (difference of point/station to grid/satellite) and the temporal (15 min satellite frequency vs 1 minute measurements integrated to hourly), seem to somehow dealt in a nice way with the ML training.
Minor
“Note that, since night-time is flagged as failing QC, 30% is a high requirement”, I don’t understand this maybe you could clarify.
Congratulations for a very interesting work.
Citation: https://doi.org/10.5194/egusphere-2023-243-RC2 -
AC2: 'Reply on RC2', Hadrien Verbois, 30 Jul 2023
Dear reviewer,
Thank you very much for your thorough review. We greatly appreciate the time and effort you committed to reviewing this manuscript. We have endeavored to answer all your points and provided a detailed answer to each of them.
Please find our replies in the supplement pdf, where they are interlaced with your comments.
-
AC2: 'Reply on RC2', Hadrien Verbois, 30 Jul 2023
-
RC3: 'Comment on egusphere-2023-243', Anonymous Referee #3, 27 May 2023
comments:
The machine learning based techniques are very popular in recent years, and these methods do provide very good performance in various fields. And people start to question is it possilble for ML to replace the traditional physical method.
I think the authors tried to try to give some explanation from some extend. In this paper, the authors used physical method and ML based method to infer SSI from satellite images. And the authors gave an overall exploration of the MLP and analysed pitfalls and drawbacks of the method. It is very interesting that the result from MLP is better than CAMS from some aspects.
This paper is of high standard, I have some questions about some points, and hope the authors can consider.
1. There are many machine learning methods, and MLP is just a very basic ML method, why you chose this method? From the introduction part, I can’t see some information about how MLP is used in the past researches, how it works in this field? Of course, this paper explained the performance of AI and physical method, but MLP can’t represent AI techinque, could you explain why don’t you use some other state-of-art AI techniques?
I think some more information should be given in introduction part as well.
2. I think another index should also be considered, that is efficiency. What is the running time of MLP and CAMS respectively? This is also a very important index to see the performance of the two methods.
3. According to 5.3.3, as to ML methods, input training dataset plays very important part in the result, it should include all the necessary information it needs. So that is why a correlation analysis between input variable and target variable is necessary. Just as the authors have analysis, in some locations, ML method shows poor performance because it doesn’t have direct access to that information, is it because satellite can’t cover that area which lead to this problem?
Citation: https://doi.org/10.5194/egusphere-2023-243-RC3 -
AC3: 'Reply on RC3', Hadrien Verbois, 30 Jul 2023
Dear reviewer,
Thank you very much for your thorough review. We greatly appreciate the time and effort you committed to reviewing this manuscript. We have endeavored to answer all your points and provided a detailed answer to each of them.
Please find our replies in the supplement pdf, where they are interlaced with your comments.
-
AC3: 'Reply on RC3', Hadrien Verbois, 30 Jul 2023
-
RC4: 'Comment on egusphere-2023-243', Anonymous Referee #4, 06 Jun 2023
This paper presents a study of the determination of surface solar irradiance from satellite using a multi-layer perceptron (MLP) compared to the state-of-the-art Copernicus Atmosphere Monitoring Service (CAMS) retrieval model. The paper is well written and well structured. The scientific question is relevant to the community and to the journal's field of application. However, the title does not accurately describe the content of the article, which deals with one particular model, namely an MLP with a hidden layer. Machine learning is a broad field that cannot be reduced to MLPs. In several places in the text, the conclusions are intended to be extended to machine learning, but they should be limited to the MLP used here.
The sensitivity study is interesting, but raises some questions about the selection of predictors for the MLP model. For example, it is noted that not including AOD at 500nm leads to underestimation. So why not include AOD 500 as a predictor? In general, the comparison between MLP and CAMS seems biased, since CAMS seems to have access to more variables (especially thanks to the clear-sky model). For a neutral comparison, wouldn't it be better to list all the variables used by CAMS (including those hidden in the clear-sky model) and use them as predictors of the MLP model? How can MLP be expected to take into account the effect of AOD if it has no access to AOD values? The same applies to albedo.
The ML model is assigned 9 pixels, whereas for CAMS, only 1 pixel is used. Why not considering an average of the 9 pixels for CAMS? This would give an idea of the contribution of MLP compared with a simple spatial average.
Here are some detailed comments:
- l. 52: Why did the authors only use 3 bands out of the 12 available? And why not use the HRV channel which is always available for France?
- l. 60: The link at the bottom of the page that describes the instrument is broken.
- l. 100: « The inverse transformation is applied to the network predictions before starting to analyze its performance ». As it is not explicit can the authors confirm that they apply the inverse transformation with the mean irradiance computed from the training set? And why do they normalized by the mean instead of removing the mean and normalizing with the standard deviation?
- About the MLP structure: How the configuration (number of neurons, activations, initialisation) was chosen?
- l. 134: « Regularization is implemented through an early stopping procedure, which stop training if the validation error does not decrease for more than 20 epochs. ». The description of the regularization is incomplete. What is the minimum variation used to stop the training?
- l. 136: « Because the last layer uses a linear activation function,… ». Why choosing a linear activation while a RELU should resolve the positiveness problem?
- l. 179: Are 8 years of data really needed for the training? What about the sensitivity about the size of the training set?
- Table 3: The MBE values seem to be incorrect. They are not coherent with Fig. 4b
- l. 235: CAM instead of CAMS
- l. 246: « Furthermore, CAMS seems to handle situations for which the clear-sky is close to one better than ML model». Is that seen from the yellow "spot" near kc=1 that is more diffuse for ML? This statement is not clear.
- Figure 3: « ML model (a and c) and for CAMS (b and d) » should be « CAMS model (a and c) and for ML (b and d) »
- l. 256: Contradiction with statement line 233 while MBE was of the order of 50W.M-2
- l. 263: « Figure 4 ». It should be Fig. 5.
- l. 315: « As discussed in Section 4.2.2, we repeat the experiment 20 times for each
choice of N, with different randomly chosen training stations at each iteration ». Is the subsampling of stations ensuring that they are well distributed in space? How do the authors achieve this?
Citation: https://doi.org/10.5194/egusphere-2023-243-RC4 -
AC4: 'Reply on RC4', Hadrien Verbois, 30 Jul 2023
Dear reviewer,
Thank you very much for your thorough review. We greatly appreciate the time and effort you committed to reviewing this manuscript. We have endeavored to answer all your points and provided a detailed answer to each of them.
Please find our replies in the supplement pdf, where they are interlaced with your comments.
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2023-243', Anonymous Referee #1, 11 May 2023
GENERAL COMMETS:
- I think this manuscript addresses a very important point, the pitfals and drawbacks of AI for the retrieval of SSI. For example, the analysis of the results at the mediterean stations presented in this manuscript illustrate the challenges associated with machine learning. The net is only able to learn from local relations between reflection, surface albedo, atmosphere and SSI. It knows nothing about physics. Hence, it has to be expected that the performance decreases significantly if it is applied in regions with quite different conditions concerning aerosol load, cloud types, H20 and surface albedo. Within this scope it has to be taken into account that in many regions almost no in-situ data are available for the training or retraining. Hence, no learning of regional relations is possible then, but physical retrieval methods do not have these problems and it would be interesting to see the results between the current network with CAMS in Africa. Hence, the question why AI is needed for the retrieval of SSI should be addressed in more detail in the manuscript.. Further, it is difficult to know what the net has really learned (black box approach), If we do not know what the net learns, we can’t learn either (and our intelligence might expire on the long run). This should be discussed in more detail as well, based on the results presented in the manuscript. These points are partly addressed in the conclusion (e.g. L370ff) but should be discussed in more detail.
- Poor performance of AI could also result from wrong training or training architecture. However, the comparison with the established CAMS show that the training has been done well (5.1.1). This is very good, because it shows that the discussed pitfalls are not due to failures in the training or the chosen training method.
- Figure 5: It seems to me that the main benefit of the machine learning is that it corrects differences in SSI induced by the different viewing geometries of ground based and satellite observations. We are aware of this effect, as significant differences are apparent when SSI retrieved from Meteosat East is compared with those from Meteosat prime for the same regions. So far these effects are not considered in many physical methods e.g. in CAMS, but it might be possible to implement appropriate “slant column” geometry corrections, which would increase the comparability of ground based and satellite based SSI. Please discuss this issue.
- Please consider that other physical retrieval methods might perform better or worse than CAMS, hence that the network might have a lower/higher performance when compared with other models. Please mention this briefly.
- 5.2.2 Impact of aerosols: This is not a really a fair analysis, AOD (and H20) have not been given at predictors for the learning, hence the network could not learn anything about the relation of AOD variations and SSI, SAT reflection. It can just learn locally some kind of mean clear sky state. Contrarily AOD is used in CAMS as “predictor”. Please mention that the performance of ML might be better if AOD data were used as predictor in addition. Of course, it is not easy to find an accurate AOD raster data set, but this problem concerns AI as well as phyical methods. Further, here, AOD from Aeronet is used, which is not available for CAMS elsewhere. Hence the capability of CAMS (or any other sat retrieval) concerning AOD variations is probably much lower as in the example. This should be also mentioned.
- Please add more information about the in-situ data. Do they all have the same maintenance, calibrations cycles and so on. Hence can the same accuracy be expected for all pyranometers ?
- Throughout the manuscript. Please avoid the separation between physical methods and clear sky index methods. They are physical methods as well !
DETAILED COMMENTS:
- Abstract: “the first of which is likely solar energy”. This depends on the viewpoint. Please delete “the first of” and rephrase accordingly, it is also quite important for climate, tourism,...
- Abstract: "For long, the emphasis has been on empirical models (simple parameterization linking the reflectance to the clear-sky index) and on physical models” The use of the clear sky index follows also physical laws, hence please rephrase. Please see also the general comments.
- L25 “…index methods without explicit physical cloud models”, L29 "empirical" please see 2.) and general comments . The use of the clear sky index follows also physical laws and the cloud index is a measure for the cloud transmission, thus, not without physical cloud model, please rephrase
- Line 55 “pixels of ca. 4 by 5 km”. it might be closer to 3.2x5.5 please check.
- L104: “ML model must be fully online” Please explain why ?
- L 195 “Three tricks are applied:” Please use a more appropriate term instead of tricks.
- L 370 please consider to add surface albedo here
- L 385: Another option is to improve the physical methods, without AI e.g. as demonstrated by HelioMon. The accuracy of HelioMont is already close to that of BSRN stations, why fuss with AI ? Please consider to add this option to the manuscript. In the Alps it is questionable if any network would be able to learn the complex relations for all regions, because taking the spatial heterogeneity into account there are not enough ground stations.
Citation: https://doi.org/10.5194/egusphere-2023-243-RC1 -
AC1: 'Reply on RC1', Hadrien Verbois, 30 Jul 2023
Dear reviewer,
Thank you very much for your thorough review. We greatly appreciate the time and effort you committed to reviewing this manuscript. We have endeavored to answer all your points and provided a detailed answer to each of them.
Please find our replies in the supplement pdf, where they are interlaced with your comments.
-
RC2: 'Comment on egusphere-2023-243', Anonymous Referee #2, 27 May 2023
This is a very interesting paper that deals with limitations and perspectives for the calculation of surface solar irradiance (SSI) using machine learning techniques.
The paper deals with an aspect including a number of more or less .. easy to explain, sources of errors and uncertainties. The work is high level and ends up in a publication with unique in my opinion results worth being published in AMT.
Some comments towards manuscript improvement
Abstract
At the moment the abstract is a bit like a general discussion and some metrics there, especially summarized comparisons of ML and CAMS, could be useful for a reader that will be intrigued to read more about it.Introduction
What I am missing is some basic state of the art of current datasets (including CAMS) and their performance evaluation.
Data
AERONET does not provide AOD every minute and also in cloudy days, so some clarification could be included as a short paragraph in 2.2 e.g. how AERONET data used , which wavelength for aerosol optical depth used etc.
Section 3
It is impressive the choice of using 3 hours (12 instants) as a basic hierarchy of the method. Could you explain how this choice has been decided? isn’t it 3 hours relatively .. long ?
The 3 set ups could be of course more complex but I personally find the choice really appropriate here.
I am a bit puzzled by the fact that the kc=1 limitation of CAMS does not have a more visual impact on the statistics. Or is it a major factor of the ML better performance ?
Figure 5: based on the definition given in lines 80 – 85 and the aerosol issues discussed after fig. 5 there should be clear sky index higher than 1 not visible in the figure.
Aerosols: It is clear that the ML inputs does not include any aerosol information so figure 7 is more or less expected. A very rough predictor including an aerosol climatology (more in summer less in winter) would for sure improve this negative correlation shown in fig. 7 . Especially because this has an impact on “high solar irradiance” cases.
Fig. 7 needs a bit more explanation as it is not clear if the points are based on instants, hourly or daily values.
I find difficult to understand how the ML can outperform CAMS for clear skies in the related bins of fig. 5 and still have these aerosol related aspects shown in fig. 7.
Maybe the authors could discuss:
In general it is understandable that the paper does not introduce a method to be used in different areas but it is a kind of sensitivity study on the ML performance. For this case a really unique dataset is used with a huge number of stations. However, it would be nice to comment on perspectives of an actual application of such system. Indirectly this study can assess some kind of realistic cases of limited or not, ground-based data available that can be used for applying such methods in different areas.
The whole France and so many stations is a huge area, but still could be very different than another area with different cloud/aerosol conditions which the same results with the same number of stations and analysis can vary. E.g. aerosol (not captured) effects in N. Africa will have a crucial effect on the statistics as well as areas with different and more clouds.
Finally I can say that problems such as the spatial (difference of point/station to grid/satellite) and the temporal (15 min satellite frequency vs 1 minute measurements integrated to hourly), seem to somehow dealt in a nice way with the ML training.
Minor
“Note that, since night-time is flagged as failing QC, 30% is a high requirement”, I don’t understand this maybe you could clarify.
Congratulations for a very interesting work.
Citation: https://doi.org/10.5194/egusphere-2023-243-RC2 -
AC2: 'Reply on RC2', Hadrien Verbois, 30 Jul 2023
Dear reviewer,
Thank you very much for your thorough review. We greatly appreciate the time and effort you committed to reviewing this manuscript. We have endeavored to answer all your points and provided a detailed answer to each of them.
Please find our replies in the supplement pdf, where they are interlaced with your comments.
-
AC2: 'Reply on RC2', Hadrien Verbois, 30 Jul 2023
-
RC3: 'Comment on egusphere-2023-243', Anonymous Referee #3, 27 May 2023
comments:
The machine learning based techniques are very popular in recent years, and these methods do provide very good performance in various fields. And people start to question is it possilble for ML to replace the traditional physical method.
I think the authors tried to try to give some explanation from some extend. In this paper, the authors used physical method and ML based method to infer SSI from satellite images. And the authors gave an overall exploration of the MLP and analysed pitfalls and drawbacks of the method. It is very interesting that the result from MLP is better than CAMS from some aspects.
This paper is of high standard, I have some questions about some points, and hope the authors can consider.
1. There are many machine learning methods, and MLP is just a very basic ML method, why you chose this method? From the introduction part, I can’t see some information about how MLP is used in the past researches, how it works in this field? Of course, this paper explained the performance of AI and physical method, but MLP can’t represent AI techinque, could you explain why don’t you use some other state-of-art AI techniques?
I think some more information should be given in introduction part as well.
2. I think another index should also be considered, that is efficiency. What is the running time of MLP and CAMS respectively? This is also a very important index to see the performance of the two methods.
3. According to 5.3.3, as to ML methods, input training dataset plays very important part in the result, it should include all the necessary information it needs. So that is why a correlation analysis between input variable and target variable is necessary. Just as the authors have analysis, in some locations, ML method shows poor performance because it doesn’t have direct access to that information, is it because satellite can’t cover that area which lead to this problem?
Citation: https://doi.org/10.5194/egusphere-2023-243-RC3 -
AC3: 'Reply on RC3', Hadrien Verbois, 30 Jul 2023
Dear reviewer,
Thank you very much for your thorough review. We greatly appreciate the time and effort you committed to reviewing this manuscript. We have endeavored to answer all your points and provided a detailed answer to each of them.
Please find our replies in the supplement pdf, where they are interlaced with your comments.
-
AC3: 'Reply on RC3', Hadrien Verbois, 30 Jul 2023
-
RC4: 'Comment on egusphere-2023-243', Anonymous Referee #4, 06 Jun 2023
This paper presents a study of the determination of surface solar irradiance from satellite using a multi-layer perceptron (MLP) compared to the state-of-the-art Copernicus Atmosphere Monitoring Service (CAMS) retrieval model. The paper is well written and well structured. The scientific question is relevant to the community and to the journal's field of application. However, the title does not accurately describe the content of the article, which deals with one particular model, namely an MLP with a hidden layer. Machine learning is a broad field that cannot be reduced to MLPs. In several places in the text, the conclusions are intended to be extended to machine learning, but they should be limited to the MLP used here.
The sensitivity study is interesting, but raises some questions about the selection of predictors for the MLP model. For example, it is noted that not including AOD at 500nm leads to underestimation. So why not include AOD 500 as a predictor? In general, the comparison between MLP and CAMS seems biased, since CAMS seems to have access to more variables (especially thanks to the clear-sky model). For a neutral comparison, wouldn't it be better to list all the variables used by CAMS (including those hidden in the clear-sky model) and use them as predictors of the MLP model? How can MLP be expected to take into account the effect of AOD if it has no access to AOD values? The same applies to albedo.
The ML model is assigned 9 pixels, whereas for CAMS, only 1 pixel is used. Why not considering an average of the 9 pixels for CAMS? This would give an idea of the contribution of MLP compared with a simple spatial average.
Here are some detailed comments:
- l. 52: Why did the authors only use 3 bands out of the 12 available? And why not use the HRV channel which is always available for France?
- l. 60: The link at the bottom of the page that describes the instrument is broken.
- l. 100: « The inverse transformation is applied to the network predictions before starting to analyze its performance ». As it is not explicit can the authors confirm that they apply the inverse transformation with the mean irradiance computed from the training set? And why do they normalized by the mean instead of removing the mean and normalizing with the standard deviation?
- About the MLP structure: How the configuration (number of neurons, activations, initialisation) was chosen?
- l. 134: « Regularization is implemented through an early stopping procedure, which stop training if the validation error does not decrease for more than 20 epochs. ». The description of the regularization is incomplete. What is the minimum variation used to stop the training?
- l. 136: « Because the last layer uses a linear activation function,… ». Why choosing a linear activation while a RELU should resolve the positiveness problem?
- l. 179: Are 8 years of data really needed for the training? What about the sensitivity about the size of the training set?
- Table 3: The MBE values seem to be incorrect. They are not coherent with Fig. 4b
- l. 235: CAM instead of CAMS
- l. 246: « Furthermore, CAMS seems to handle situations for which the clear-sky is close to one better than ML model». Is that seen from the yellow "spot" near kc=1 that is more diffuse for ML? This statement is not clear.
- Figure 3: « ML model (a and c) and for CAMS (b and d) » should be « CAMS model (a and c) and for ML (b and d) »
- l. 256: Contradiction with statement line 233 while MBE was of the order of 50W.M-2
- l. 263: « Figure 4 ». It should be Fig. 5.
- l. 315: « As discussed in Section 4.2.2, we repeat the experiment 20 times for each
choice of N, with different randomly chosen training stations at each iteration ». Is the subsampling of stations ensuring that they are well distributed in space? How do the authors achieve this?
Citation: https://doi.org/10.5194/egusphere-2023-243-RC4 -
AC4: 'Reply on RC4', Hadrien Verbois, 30 Jul 2023
Dear reviewer,
Thank you very much for your thorough review. We greatly appreciate the time and effort you committed to reviewing this manuscript. We have endeavored to answer all your points and provided a detailed answer to each of them.
Please find our replies in the supplement pdf, where they are interlaced with your comments.
Peer review completion
Post-review adjustments
Journal article(s) based on this preprint
Hadrien Verbois et al.
Hadrien Verbois et al.
Viewed
| HTML | XML | Total | Supplement | BibTeX | EndNote | |
|---|---|---|---|---|---|---|
| 406 | 167 | 29 | 602 | 40 | 10 | 10 |
- HTML: 406
- PDF: 167
- XML: 29
- Total: 602
- Supplement: 40
- BibTeX: 10
- EndNote: 10
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(2128 KB) - Metadata XML