the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 08 Aug 2023
Evaluation of surface shortwave downward radiation forecasts by the numerical weather prediction model AROME
Abstract. With the worlwide development of the solar energy sector, the need for reliable surface shortwave downward radiation (SWD) forecasts has significantly increased in recent years. SWD forecasts of a few hours to a few days based on numerical weather prediction (NWP) models are essential to facilitate the incorporation of solar energy into the electric grid and ensure network stability. However, errors in NWP models can be substantial. In order to characterize in detail the performances of AROME, the operational NWP model of the French weather service Météo-France, a full year of hourly AROME forecasts is compared to corresponding in situ SWD measurements from 168 high-quality pyranometers covering France. In addition, to classify cloud scenes at high temporal frequency and over the whole territory, cloud products derived from the Satellite Application Facility for Nowcasting and Very Short Range Forecasting (SAF NWC) from geostationary satellites are also used. The 2020 bias is 18 W m-2 and the root-mean-square-error is 98 W m-2. The situations that contribute the most to the bias correspond to cloudy skies in the model and in the observations, situations that are very frequent (66 %) and characterized by an annual bias of 24 W m-2. Part of this positive bias probably comes from an underestimation of cloud fraction in AROME, although this is not fully addressed in this study due to lack of consistent observations at kilometer resolution. The other situations have less impact on SWD errors. Missed cloudy situations and erroneously predicted clouds, which correspond on average to clouds with a low impact on the SWD, also have low occurrence (4 % and 11 %). Likewise, well-predicted clear sky conditions are characterized by a low bias (3 W m-2). When limited to overcast situations in the model, the bias in cloudy skies is small (198 W m-2) but results from large compensating errors. Indeed, further investigations show that high clouds are systematically associated with a SWD positive bias while low clouds are associated with a negative bias. This detailed analysis shows that the errors result from a combination of incorrect cloud optical properties and cloud fraction errors, highlighting the need for a more detailed evaluation of cloud properties. This study also provides valuable insights into the potential improvement of AROME physical parametrization.
-
Notice on discussion status
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
-
Preprint
(3185 KB)
-
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(3185 KB) - Metadata XML
- BibTeX
- EndNote
- Final revised paper
Journal article(s) based on this preprint
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2023-1181', Anonymous Referee #1, 22 Aug 2023
In their article 'Evaluation of surface shortwave downward radiation forecasts by the numerical weather prediction model AROME', the authors attribute shortwave radiation biases to cloud regimes using satellite an pyranometer data.
The article is a valuable contribution, providing a detailed quantification of model deficiencies and, by this, guidance for model developers.
Minor Comments
- Line 142: The results can be very sensitive to the choice of such thresholds. Did you investigate this? If so, please add a sentence on the reason of your choice in the article.
- Line 147-150: While neighborhood strategies are well-suited for verification purposes, this double penalty problem is exactly what PV power forecast have to struggle with. One could argue that this strategy is not suitable for solving the problems targeted in this study. An alternative strategy would be the usage of ensemble forecasts. As I understood from the following, you tried more than one value for neighborhood size and the results were not sensitive to this choice? Please clarify.
- Line 173 (Figure 2): In the following, several terms are introduced that describe a combination of several cloud regimes. For example, "low clouds" which are cloud classes CR0, CR1, CR4. Another example is geometrically thin clouds which are CR1, CR2, CR3. A table here that summarizes these groupings of cloud regimes would be helpful. It would also help to avoid inconsistencies like for geometrically thick clouds which are CR4, CR5 and CR7 in line 286 and 438, but only CR5 and CR7 in line 464.
- Line 173 (Figure 2): Furthermore, a more descriptive naming of the cloud regimes might be beneficial for the readability of the manuscript.
- Line 174: Another threshold is introduced here. Again, as far as I understand, the sensitivity to this parameter has already been investigated. A short comment here would be helpful.
- Line 177: It is not clear which special cloud regime is meant. CR7?
- Line 195: This description is a bit misleading and on first reading it sounds like (SWD_mod/SWD_obs)/SWD_modclear. What you meant is probably SWD_mod/SWD_modclear for the model and SWD_obs/SWD_modclear for observations. One (or two) short formulas are always clearer than a description in words.
- Line 201: This clear-sky underestimation of SWD is not surprising. There is a decreasing aerosol trend in Central Europe in the last decades. Older climatologies (like Tegen) are thus overestimating present-day aerosol loadings in Central Europe.
- Line 247: As mentioned before, there is also a positive bias in Tegen AOD for Central Europe.
- Line 416: "...and the SWD higher (15%)" Do you mean SD?
- Line 428-431: I did not fully understand this paragraph. It starts with the description of the second pair of values, the conclusion seems to be rather suitable for the third pair of values which is not described anywhere else.
- Line 446: For this section where you verify the satellite-based cloud classification of your approach, SYNOP data on high, mid and low cloud fraction might be helpful as an independent dataset looking from a different perspective. Have you considered this?
- Line 481: Did you separate between low and high clouds here? For low clouds, the consideration of snow might even further pronounce the already existing bias.
Language
Please double-check the following formulations which do not sound correct to me (as a non-native speaker):
- Lines 10-12: A verb seems to be missing in this sentence
- Line 21: There is more than one physical parameterization in AROME, so I would suggest either "of the AROME physical parameterizations" or "of the AROME physical parameterization package"
- Line 38-39: I am not sure if I understand what is meant by "optimal reserves optimizing storage"
- Line 47: Although I understand what is meant by "infra-hour", I am not sure if this is an English term
- Line 49: terme -> term
- Line 108: The "It" is misleading. I assume the French operational configuration of AROME is meant and not the model AROME.
- Line 165: A separate sentence would be clearer to read: "The neighborhood size is comparable to the satellite pixel size."
- Line 176: "Another"
- Line 204: "is slightly"
- Line 204: Is delayed meant in a temporal context? If not, I would rather suggest a different word like "displaced" or "shifted"
- Line 265: "not detected"
- Line 332: "Hit cases"
- Line 399: Missing space
- Line 409: "section"
- Line 542: Do you mean "Gb"?
Citation: https://doi.org/10.5194/egusphere-2023-1181-RC1 - AC1: 'Reply on RC1', Marie-Adèle Magnaldo, 11 Nov 2023
-
RC2: 'Comment on egusphere-2023-1181', Anonymous Referee #2, 06 Sep 2023
In the manuscript “Evaluation of surface shortwave downward radiation forecasts by the numerical weather prediction model AROME”, the authors use shortwave observations from the French pyranometer network together with a geostationary satellite cloud classification to characterise and attribute the shortwave radiation bias in the AROME forecast model.
I find the topic appropriate for the journal, and of interest to the wider research community. Despite the limitations of the satellite data set, the authors manage to pinpoint systematic model errors in the representation of low and high clouds, giving valuable guidance for future model development efforts.
My comments largely address wording/spelling/typo corrections and some minor clarifications of the discussion. In some places, I think the discussion could be a bit more concise.
Specific comments:
Abstract: While SWD is defined on the first line, it might be helpful for clarity to mention somewhere in the abstract body how the sign of the bias is defined (e.g. where a positive bias is first mentioned, one might add “i.e. too much shortwave radiation reaching the ground” or similar).
L90: Single sites are not necessarily unrepresentative, particularly if long time series are considered that cover a variety of cloud regimes. I find the wording a bit strong here.
L137: I find the reference to Table 1 here a bit confusing. The text talks about 15 cloud types, yet the table lists only 6. The merging of the cloud types is discussed later in the text. Maybe instead of referencing Table1 in line 137, the text should refer the reader to section 2.2.3 where the table is explained (and first reference the table in that section).
L145: What is the sensitivity to size of neighbourhood, or choosing the nearest-neighbour instead of using a neighbourhood approach?
L188: I don’t really see a more pronounced seasonal cycle. Does that statement apply to the low cloud category only, or to all cloud? If it applies to only low clouds, it is not easy to make out the seasonal cycle of CR0+CR1+CR4 in Fig. 3 since the categories are not grouped together. Please clarify.
L208: I’m not that familiar with various configurations of AROME. How does HARMONIE-AROME differ from the AROME version presented in the manuscript? Maybe a half-sentence would be useful here, e.g. stating that HARMONIE-AROME uses the same microphysics and radiation schemes (if that is the case), to indicate that the results from the cited study apply.
Fig 12 does not add much to the discussion. I think it (and the few sentences discussing it) could probably be left out.
Section 4.1.2: I found this section most difficult to follow, and also somewhat repetitive. It seems to mainly confirm conclusions that were already drawn previously from analyzing SWD and SD in cloud classes/types. That does not really surprise me, since I would expect the informational content found in SWD, SD and clear sky index (extensively discussed in sections 3.3) to be the same as found in the transmittance and its standard deviation.
E.g. L185: “some optically thin clouds are not detected by NWC SAF product, which is a known caveat of passive sensors” – L418: “clouds may be present but not detected”
Similarly, we have already seen that the contribution to the SWD bias from false alarm and missed cases is relatively small (corresponding to the conclusions on Line 430 and 434).
I would like this section to be more concise (what new information does the transmittance perspective contribute, that hasn’t been seen the section 3.3 previously?), or maybe some of the additional information could be wrapped in with the discussions throughout section 3.3, eliminating 4.1.2.
L498: I am not sure how relevant lacking supercooled liquid is likely to be over France. The greatest impact on radiation is found in cold regions where models erroneously produce ice-only clouds. I’d expect most LC in France to contain liquid water under most conditions, except maybe for some winter conditions. Nevertheless AROME is used elsewhere, and the Scandinavian countries could certainly benefit from an improved representation of supercooled liquid.
Technical comments:
L25: should this link be in the text, or in a reference? Not sure what the journal’s style guide suggests, but a long link in the text disrupts the flow a bit.
L38/39: wording:” optimal reserves optimizing storage.”
L49: typo: short-terme, should be term, also: satellite should not be capitalised
L90: typo: word “not” is used twice
L218: Wording suggestion: Use “In contrast, “instead of “on the contrary” (applies to several places throughout the text)
L249: wording: “local and punctual” – suggestion: just use “localised”. “punctual” means “being on time”. Or if you are referring to time, then maybe “temporary” or “short-lived” might be better.
L294: The year is missing in reference Antoine et al. – looks like it is not published yet. In this case, maybe there should be a further comment behind the name, e.g. “in preprint”, “under review” or similar (not sure what the journal prefers).
Fig 11: Labels of the cloud type (CRx) in each panel would make it easier to follow the discussion in the text
L393: remove parentheses around reference to Lucas-Picher et al.
L428: put Ackermann reference in parentheses
L520 and following: sentences are repeated
L542: Should the units be Gb, not Go?
Citation: https://doi.org/10.5194/egusphere-2023-1181-RC2 - AC2: 'Reply on RC2', Marie-Adèle Magnaldo, 11 Nov 2023
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2023-1181', Anonymous Referee #1, 22 Aug 2023
In their article 'Evaluation of surface shortwave downward radiation forecasts by the numerical weather prediction model AROME', the authors attribute shortwave radiation biases to cloud regimes using satellite an pyranometer data.
The article is a valuable contribution, providing a detailed quantification of model deficiencies and, by this, guidance for model developers.
Minor Comments
- Line 142: The results can be very sensitive to the choice of such thresholds. Did you investigate this? If so, please add a sentence on the reason of your choice in the article.
- Line 147-150: While neighborhood strategies are well-suited for verification purposes, this double penalty problem is exactly what PV power forecast have to struggle with. One could argue that this strategy is not suitable for solving the problems targeted in this study. An alternative strategy would be the usage of ensemble forecasts. As I understood from the following, you tried more than one value for neighborhood size and the results were not sensitive to this choice? Please clarify.
- Line 173 (Figure 2): In the following, several terms are introduced that describe a combination of several cloud regimes. For example, "low clouds" which are cloud classes CR0, CR1, CR4. Another example is geometrically thin clouds which are CR1, CR2, CR3. A table here that summarizes these groupings of cloud regimes would be helpful. It would also help to avoid inconsistencies like for geometrically thick clouds which are CR4, CR5 and CR7 in line 286 and 438, but only CR5 and CR7 in line 464.
- Line 173 (Figure 2): Furthermore, a more descriptive naming of the cloud regimes might be beneficial for the readability of the manuscript.
- Line 174: Another threshold is introduced here. Again, as far as I understand, the sensitivity to this parameter has already been investigated. A short comment here would be helpful.
- Line 177: It is not clear which special cloud regime is meant. CR7?
- Line 195: This description is a bit misleading and on first reading it sounds like (SWD_mod/SWD_obs)/SWD_modclear. What you meant is probably SWD_mod/SWD_modclear for the model and SWD_obs/SWD_modclear for observations. One (or two) short formulas are always clearer than a description in words.
- Line 201: This clear-sky underestimation of SWD is not surprising. There is a decreasing aerosol trend in Central Europe in the last decades. Older climatologies (like Tegen) are thus overestimating present-day aerosol loadings in Central Europe.
- Line 247: As mentioned before, there is also a positive bias in Tegen AOD for Central Europe.
- Line 416: "...and the SWD higher (15%)" Do you mean SD?
- Line 428-431: I did not fully understand this paragraph. It starts with the description of the second pair of values, the conclusion seems to be rather suitable for the third pair of values which is not described anywhere else.
- Line 446: For this section where you verify the satellite-based cloud classification of your approach, SYNOP data on high, mid and low cloud fraction might be helpful as an independent dataset looking from a different perspective. Have you considered this?
- Line 481: Did you separate between low and high clouds here? For low clouds, the consideration of snow might even further pronounce the already existing bias.
Language
Please double-check the following formulations which do not sound correct to me (as a non-native speaker):
- Lines 10-12: A verb seems to be missing in this sentence
- Line 21: There is more than one physical parameterization in AROME, so I would suggest either "of the AROME physical parameterizations" or "of the AROME physical parameterization package"
- Line 38-39: I am not sure if I understand what is meant by "optimal reserves optimizing storage"
- Line 47: Although I understand what is meant by "infra-hour", I am not sure if this is an English term
- Line 49: terme -> term
- Line 108: The "It" is misleading. I assume the French operational configuration of AROME is meant and not the model AROME.
- Line 165: A separate sentence would be clearer to read: "The neighborhood size is comparable to the satellite pixel size."
- Line 176: "Another"
- Line 204: "is slightly"
- Line 204: Is delayed meant in a temporal context? If not, I would rather suggest a different word like "displaced" or "shifted"
- Line 265: "not detected"
- Line 332: "Hit cases"
- Line 399: Missing space
- Line 409: "section"
- Line 542: Do you mean "Gb"?
Citation: https://doi.org/10.5194/egusphere-2023-1181-RC1 - AC1: 'Reply on RC1', Marie-Adèle Magnaldo, 11 Nov 2023
-
RC2: 'Comment on egusphere-2023-1181', Anonymous Referee #2, 06 Sep 2023
In the manuscript “Evaluation of surface shortwave downward radiation forecasts by the numerical weather prediction model AROME”, the authors use shortwave observations from the French pyranometer network together with a geostationary satellite cloud classification to characterise and attribute the shortwave radiation bias in the AROME forecast model.
I find the topic appropriate for the journal, and of interest to the wider research community. Despite the limitations of the satellite data set, the authors manage to pinpoint systematic model errors in the representation of low and high clouds, giving valuable guidance for future model development efforts.
My comments largely address wording/spelling/typo corrections and some minor clarifications of the discussion. In some places, I think the discussion could be a bit more concise.
Specific comments:
Abstract: While SWD is defined on the first line, it might be helpful for clarity to mention somewhere in the abstract body how the sign of the bias is defined (e.g. where a positive bias is first mentioned, one might add “i.e. too much shortwave radiation reaching the ground” or similar).
L90: Single sites are not necessarily unrepresentative, particularly if long time series are considered that cover a variety of cloud regimes. I find the wording a bit strong here.
L137: I find the reference to Table 1 here a bit confusing. The text talks about 15 cloud types, yet the table lists only 6. The merging of the cloud types is discussed later in the text. Maybe instead of referencing Table1 in line 137, the text should refer the reader to section 2.2.3 where the table is explained (and first reference the table in that section).
L145: What is the sensitivity to size of neighbourhood, or choosing the nearest-neighbour instead of using a neighbourhood approach?
L188: I don’t really see a more pronounced seasonal cycle. Does that statement apply to the low cloud category only, or to all cloud? If it applies to only low clouds, it is not easy to make out the seasonal cycle of CR0+CR1+CR4 in Fig. 3 since the categories are not grouped together. Please clarify.
L208: I’m not that familiar with various configurations of AROME. How does HARMONIE-AROME differ from the AROME version presented in the manuscript? Maybe a half-sentence would be useful here, e.g. stating that HARMONIE-AROME uses the same microphysics and radiation schemes (if that is the case), to indicate that the results from the cited study apply.
Fig 12 does not add much to the discussion. I think it (and the few sentences discussing it) could probably be left out.
Section 4.1.2: I found this section most difficult to follow, and also somewhat repetitive. It seems to mainly confirm conclusions that were already drawn previously from analyzing SWD and SD in cloud classes/types. That does not really surprise me, since I would expect the informational content found in SWD, SD and clear sky index (extensively discussed in sections 3.3) to be the same as found in the transmittance and its standard deviation.
E.g. L185: “some optically thin clouds are not detected by NWC SAF product, which is a known caveat of passive sensors” – L418: “clouds may be present but not detected”
Similarly, we have already seen that the contribution to the SWD bias from false alarm and missed cases is relatively small (corresponding to the conclusions on Line 430 and 434).
I would like this section to be more concise (what new information does the transmittance perspective contribute, that hasn’t been seen the section 3.3 previously?), or maybe some of the additional information could be wrapped in with the discussions throughout section 3.3, eliminating 4.1.2.
L498: I am not sure how relevant lacking supercooled liquid is likely to be over France. The greatest impact on radiation is found in cold regions where models erroneously produce ice-only clouds. I’d expect most LC in France to contain liquid water under most conditions, except maybe for some winter conditions. Nevertheless AROME is used elsewhere, and the Scandinavian countries could certainly benefit from an improved representation of supercooled liquid.
Technical comments:
L25: should this link be in the text, or in a reference? Not sure what the journal’s style guide suggests, but a long link in the text disrupts the flow a bit.
L38/39: wording:” optimal reserves optimizing storage.”
L49: typo: short-terme, should be term, also: satellite should not be capitalised
L90: typo: word “not” is used twice
L218: Wording suggestion: Use “In contrast, “instead of “on the contrary” (applies to several places throughout the text)
L249: wording: “local and punctual” – suggestion: just use “localised”. “punctual” means “being on time”. Or if you are referring to time, then maybe “temporary” or “short-lived” might be better.
L294: The year is missing in reference Antoine et al. – looks like it is not published yet. In this case, maybe there should be a further comment behind the name, e.g. “in preprint”, “under review” or similar (not sure what the journal prefers).
Fig 11: Labels of the cloud type (CRx) in each panel would make it easier to follow the discussion in the text
L393: remove parentheses around reference to Lucas-Picher et al.
L428: put Ackermann reference in parentheses
L520 and following: sentences are repeated
L542: Should the units be Gb, not Go?
Citation: https://doi.org/10.5194/egusphere-2023-1181-RC2 - AC2: 'Reply on RC2', Marie-Adèle Magnaldo, 11 Nov 2023
Peer review completion
Journal article(s) based on this preprint
Data sets
Cloud satellite products developed by the NWC Eumetsat SAF NWC Eumetsat SAF https://www.icare.univ-lille.fr/
satellite product from Copernicus Atmosphere Monitoring Service CAMS https://atmosphere.copernicus.eu/
Observations of shortwave downward radiation from the operational observation network of Météo-France Météo-France https://donneespubliques.meteofrance.fr/?fond=produit&id_produit=298&id_rubrique=32
AROME forecasts Météo-France https://doi.org/10.5281/zenodo.7928622
Model code and software
AROME Météo-France http://www.umr-cnrm.fr/accord/
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 259 | 94 | 19 | 372 | 12 | 14 |
- HTML: 259
- PDF: 94
- XML: 19
- Total: 372
- BibTeX: 12
- EndNote: 14
Viewed (geographical distribution)
| Country | # | Views | % |
|---|---|---|---|
| France | 1 | 95 | 26 |
| United States of America | 2 | 87 | 23 |
| Germany | 3 | 44 | 12 |
| China | 4 | 16 | 4 |
| United Kingdom | 5 | 14 | 3 |
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
- 95
Marie-Adèle Magnaldo
Sébastien Riette
Christine Lac
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(3185 KB) - Metadata XML