Analysis of model error in forecast errors of Extended Atmospheric Lorenz' 05 Systems and the ECMWF system

Bednář, Hynek; Kantz, Holger

doi:https://doi.org/10.5194/egusphere-2023-1464

Preprints

https://doi.org/10.5194/egusphere-2023-1464

Preprints

04 Aug 2023

| 04 Aug 2023

Analysis of model error in forecast errors of Extended Atmospheric Lorenz' 05 Systems and the ECMWF system

Hynek Bednář and Holger Kantz

Abstract. The forecast error growth as a function of lead time of atmospheric phenomena is caused by initial and model errors. When studying the initial error growth, it turns out that small scale phenomena, which contribute little to the forecast product, significantly affect the ability to predict this. The question under investigation is whether omitting these atmospheric phenomena will improve the predictability of the resulting value. The topic is studied in the extended Lorenz (2005) system. This system shows that omitting small spatiotemporal scales will reduce predictability more than modeling it. Generally, a system with model error (omitting phenomena) will not improve predictability. A theory explaining and describing this behavior is developed, with the difference between systems (model error) produced at each time step seen as the error of the initial conditions. The resulting model error is then defined as the sum of the increments of the time evolution of the initial conditions so defined. The theory is compared to the fit parameters that define the model error in certain approximations of the average forecast error growth. Parameters are interpreted in this context, and the hypotheses are used to estimate the errors described in the theory. It is proposed how to distinguish increments to prediction error growth from small spatiotemporal-scales phenomena and model error. Results are presented for the error growth of the ECMWF system, where a 40 % reduction in model error between 1987 and 2011 is calculated based on the developed theory, while over the same time, the instability of the system with respect to initial condition errors has grown.

Received: 30 Jun 2023 – Discussion started: 04 Aug 2023

Download & links

Hynek Bednář and Holger Kantz

Status: final response (author comments only)

RC1:
'Comment on egusphere-2023-1464', Anonymous Referee #1, 04 Sep 2023

The authors consider the question of whether omitting small spatiotemporal scales atmospheric phenomena will improve the predictability of the resulting value, using the Lorenz (2005) systems denote by L05-1, L05-2, L05-3. This system shows that omitting small spatiotemporal scales will reduce predictability more than modeling it. The authors claimed that initial errors of magnitudes that are comparable to real weather forecasts do not play a dominant role in the L05 systems, while model errors do. Moreover, they present an explanation of the observed error growth in terms of an time-averaged model error called the drift.
The authors showed that for the multi-scale systems L05-2 and L05-3, the initial error growth can be well described by the power law or the extended power law, while a simple exponential growth or the quadratic hypothesis with model error are less appropriate. We showed that in the L05 and ECMWF data the error growth in the early stage grows faster than the approximation of the whole curve, probably due to the presence of only smaller spatiotemporal scales in this part.
The paper seem to be doing a good job recalling the basic notions of initial error growth and model error growth. The L05-3 system was introduced by the authors in an earlier paper to extend the 2-scale, L05-2 system to three scales. The parameters of these systems are set so that all scales behave chaotically. Though it is not totally clear how robust the results are if the parameters are perturbed. The explanation of the initial decline and subsequent growth of the rate of model error growth by the notion of ``drift" is a nice attempt, though it is not totally clear if this is special for the L-05 systems.
Overall, I think the results are worth publishing and the efforts is a nice demonstration of illustrating the effects of different scale on error growth.
I have only a few comments for improvement of the paper:

- In the abstract, where is the claim "Generally, a

system with model error (omitting phenomena) will not improve predictability." supported in the maintext? How general is it? This seems to be a very strong statement. If not, I suggest weaken this statement.
- Although it maybe natural, it would be good to give a sentence of explanation about why choosing L05-2 and L05-3 as "reality"
- Is "this" in line 9 ``initial error growth"? Perhaps good to be more specific.
- How about adding references to the relevant figures after "as we will see in numerical simulations" in line 270?
- Would you explain why geometric mean is used rather than the usual arithmetic mean in model error growth and drift terms in (11), (14), (15), (18) ?
- The drift d(tau) at the beginning of line 271 is not defined yet, It does not seems to be the drift VECTOR in line 269. Please clarify.
- Please be consistent in terminology. For example, is ``the drift D(tau)" on Page 279 the same as d(tau) in line 271? Is it the same as the "the averaged drift D" in line 283?
- Perhaps include a table summarizing the heavy notation involved.
- The reference list will look better it it was itemized.
- I am not totally convinced (or understand) that the notion of the "drift" introduced really explain the model error growth as claimed. It seems that, taking time-average without an absolute value is similar to looking at the original system, when the system is ergodic. Perhaps the authors can explain more on what ``explain" means other than showing another summary statistics of the system.

Citation: https://doi.org/10.5194/egusphere-2023-1464-RC1
- AC1: 'Reply on RC1', Hynek Bednar, 19 Sep 2023
  
  We are grateful to the referee for devoting their time to our manuscript. The valuable comments and suggestions will help us to improve the paper.
  
  We will here respond to comments made:
  
  The parameters of these systems are set so that all scales behave chaotically. Though it is not totally clear how robust the results are if the parameters are perturbed.
  
  As long as parameters are such that all scales are chaotic, we do not expect any qualitative changes with respect to the studied scenario. We tried to ensure the robustness of the results by considering two cases of "reality" (L05-2 and L05-3 systems). Furthermore, we tested as "reality" the L05-1 system with 360 variables and as "model" the L05-1 system with 180 and 90 variables. The results are consistent with the presented results. We are aware of the need to test the results on "real" systems.
  The explanation of the initial decline and subsequent growth of the rate of model error growth by the notion of ``drift" is a nice attempt, though it is not totally clear if this is special for the L-05 systems.
  
  "Drift" was used by Orrell (2002) to explain the initial decline and subsequent growth of the rate of model error growth for the ECMWF system (500 hPa, Northern hemisphere for 10 d in October 1999 and total energy globally over a 15 d period in December 2000) . Therefore, the results do not appear to apply only to the L05 system. We have further confirmed the behavior resulting from "drift" on the ECMWF system data in Section 5.
  In the abstract, where is the claim "Generally, a system with model error (omitting phenomena) will not improve predictability." supported in the maintext? How general is it? This seems to be a very strong statement. If not, I suggest weaken this statement.
  
  We have replaced "Generally" with "In other words" (Lines 11 – 12).
  Although it maybe natural, it would be good to give a sentence of explanation about why choosing L05-2 and L05-3 as "reality"
  
  A full explanation of why L05-2 and L05-3 systems are selected as the reality is given in Section 2.2. In addition, we have added a sentence to the introduction on lines 128 – 129 ("The omitted scale is the small scale for the L05-2 system and the small and medium scale for the L05-3 system. "). Information on why we do not use the L05-2 system as model and L05-3 system as reality is given on lines 614-615.
  Is "this" in line 9 ``initial error growth? Perhaps good to be more specific.
  
  The word "product" was added to line 9.
  How about adding references to the relevant figures after "as we will see in numerical simulations" in line 270?
  
  Reference was added.
  Would you explain why geometric mean is used rather than the usual arithmetic mean in model error growth and drift terms in (11), (14), (15), (18) ?
  
  We added an explanation on line 200: "The geometric mean is chosen because of its suitability for comparison with growth governed by the largest Lyapunov exponent. For further information, see Bednar et al. (2014) or Ding and Li (2011). "
  The drift d(tau) at the beginning of line 271 is not defined yet, It does not seems to be the drift VECTOR in line 269. Please clarify.
  
  d(tau) was changed to the absolute value of drift d(tau).
  Please be consistent in terminology. For example, is ``the drift D(tau)" on Page 279 the same as d(tau) in line 271? Is it the same as the "the averaged drift D" in line 283?
  
  We improved consistency in terminology. We related D(t) to eq. (18) and d(tau) to eq. (17).
  Perhaps include a table summarizing the heavy notation involved.
  
  We itemized the numbers of the equations. We hope this will help readability.
  The reference list will look better it it was itemized.
  
  We itemized the reference list.
  I am not totally convinced (or understand) that the notion of the "drift" introduced really explain the model error growth as claimed. It seems that, taking time-average without an absolute value is similar to looking at the original system, when the system is ergodic. Perhaps the authors can explain more on what ``explain" means other than showing another summary statistics of the system.
  
  The difference is that it is the summation of vectors created from the difference in time evolution of different systems (but with the same initial conditions) after one time step. The model errors at successive time steps as vectors are not strongly correlated, and that therefore accumulating their absolute values is very different from accumulating them as vectors, where the absolute values sum will grow much faster than the vector valued sum, and that this slower error growth now gives a better explanation of the deviation of the trajectories.
  References:
  
  Bednář, H., Raidl, A., and Mikšovský, J.: Initial Error Growth and Predictability of Chaotic Low-dimensional Atmospheric Model, IJAC, 11, 256–264, https://doi.org/10.1007/s11633-014-0788-3 2014.
  
  Ding, R., Li, J.: Comparisons of two ensemble mean methods in measuring the average error growth and the predictability, Acta Meteorol Sin, 25, 395–404, https://doi.org/10.1007/s13351-011-0401-4, 2011.
  
  Orrell, D.: Role of the metric in forecast error growth: how chaotic is the weather?, Tellus, 54, 350-362, https://doi.org/10.1034/j.1600-0870.2002.01389.x, 2002.
  
  Citation: https://doi.org/10.5194/egusphere-2023-1464-AC1
RC2:
'Comment on egusphere-2023-1464', Quan Wang, 29 Oct 2023
The growth of forecast errors with respect to the lead time of atmospheric phenomena is primarily influenced by initial errors and model errors. An important question is whether omitting atmospheric phenomena, which contribute little to the final value, will improve the predictability of the resulting value. This paper uses the L05 systems and the ECMWF systems to investigate the question. Based on some numerical results, this article finds that the average prediction error grows faster in a model where small-scale phenomena are omitted, but the model error is therefore created, compared to a model where all phenomena are present, but the average forecast error growth is scale-dependent. Using these results in this paper tries to explain why omitting atmospheric phenomena, which contribute little to the final value, will not improve the predictability of the resulting value. However, this paper does not provide a complete theory to show this. Although this article says that a theory explaining and describing this behavior is developed, I did not find any strict mathematical theory in this article. In addition, there are some statement which is not clear. For instance,
Line 20: …“the instability of the system with respect to initial condition errors has grown”…, the instability is not clear?

Line 95:” …. the constant b in Eq. (5) which, irrespective of initial condition errors, will lead to a deviation of the model solution from reality…”. It seems that there is no a constant b in Eq. (5).

Line 120: “….Including small spatiotemporal scales, i.e., improving the model's spatial and temporal resolution, therefore enhances the instability with respect to initial condition error”…. the exact meaning of the instability is not clear.

Line 140: “….We measure the error magnitude e(t) after fixed time intervals ..”. there is not any expression for e(t).

5.Line 160: “…. For this scheme to be meaningful, we have to ensure that the reference trajectory is on the system's attractor and that the repetition of this scheme samples the whole attractor with correct weights (the invariant measure)….”. the existence of attractors in this system is not clear.
6.Line 195: There is no definition of .
Line 230: There is no definition of .

Line 245: There is no definition of .

For these reasons I recommend rejection.
Citation: https://doi.org/10.5194/egusphere-2023-1464-RC2
- AC2: 'Reply on RC2', Hynek Bednar, 31 Oct 2023
  
  We are grateful to the referee for devoting his time to our manuscript. We will here respond to comments made to support the validity of the article for publishing:
  
  This paper tries to explain why omitting atmospheric phenomena, which contribute little to the final value, will not improve the predictability of the resulting value. However, this paper does not provide a complete theory to show this. Although this article says that a theory explaining and describing this behavior is developed, I did not find any strict mathematical theory in this article.
  
  The developed theory is not strictly mathematical but is based on a strictly mathematical theory describing the model error growth (Drift - Section 2.4), presented by Orrell (Orrell et al., 2001; Orrell, 2002) and on a strictly mathematical theory of classical low dimensional chaos, where one observes an exponential error growth of a tiny initial error whose exponent is given by the largest Lyapunov exponent of the system. Our extension that sees Drift produced at each time step as the error of the initial conditions is based on an experiment with Lorenz L05 systems (Appendix A) and explains and describes the model error growth in this experimental setting (Section 4). The derived results are then verified in the ECMWF systems (Section 5). Because it is not a theory in a strictly mathematical sense. We replace the term “theory” with the term “hypothesis”. We believe that our hypothesis Eq. (21) is as worthy of publication as other already commonly accepted experiment-based hypotheses such as Eqs. (2)-(6).
  
  Line 20: …“the instability of the system with respect to initial condition errors has grown”…, the instability is not clear?
  
  By instability we mean the error growth rate of the initial conditions of ECMWF systems, which is expressed by the Lambda parameter from Eq. (5). The values can be seen in Figure 15a - blue curve. More details can be found in Section 5. For better understanding, we have added "(error growth rate)" to the text.
  
  Line 95:” …. the constant b in Eq. (5) which, irrespective of initial condition errors, will lead to a deviation of the model solution from reality…”. It seems that there is no a constant b in Eq. (5).
  
  b has been replaced in the text by betha. We thank the referee for spotting this misprint.
  
  Line 120: “….Including small spatiotemporal scales, i.e., improving the model's spatial and temporal resolution, therefore enhances the instability with respect to initial condition error”…. the exact meaning of the instability is not clear.
  
  By instability, we mean the error growth rate of the initial conditions. Brisch and Kantz (2019) and Zhang et al. (2019) associated initial error growth with scale-dependent error growth, where tiny errors grow much faster than larger ones. Lorenz (1969) gave a sketch of such error growth: a typical quantity to be predicted is a superposition of the dynamics on different scales. After a fast growth of the small-scale errors with saturation at these very same small scales, the large-scale errors continue to grow at a slower rate until even these saturate. We have added "(error growth rate)" to the text.
  
  Line 140: “….We measure the error magnitude e(t) after fixed time intervals ..”. there is not any expression for e(t).
  
  On line 140, e(t) is defined as the error magnitude after fixed time intervals. The expressions for the settings are shown on lines 192, 226 and 245.
  
  Line 160: “…. For this scheme to be meaningful, we have to ensure that the reference trajectory is on the system's attractor and that the repetition of this scheme samples the whole attractor with correct weights (the invariant measure)….”. the existence of attractors in this system is not clear.
  Lorenz L05 systems are widely accepted chaotic systems with a positive largest Lyapunov exponent (which is computed and presented). For L96 system (Lorenz, 1996) the existence of attractor has been shown, and because our system can be expected to be in the same model class, we expect the existence of a chaotic attractor.
  
  Line 195: There is no definition of . Line 230: There is no definition of . Line 245: There is no definition of .
  
  It is probably meant that Eqs. (11), (14) and (15) are not definitions from a strictly mathematical sence, so we replace the expression "is defined" by "is calculated".
  
  References:
  
  Brisch, J., and Kantz, H.: Power law error growth in multi-hierarchical chaotic system-a dynamical mechanism for finite prediction horizon, New J. Phys., 21, 1–7, https://doi.org/10.1088/1367-2630/ab3b4c, 2019.
  
  Lorenz, E. N.: The predictability of a flow which possesses many scales of motion, Tellus, 21, 289–307, https://doi.org/10.1111/j.2153-3490.1969.tb00444.x, 1969.
  
  Lorenz, E. N.: Predictability: a problem partly solved, in: Predictability of Weather and Climate, edited by: Palmer, T., and Hagedorn, R., Cambridge University Press, Cambridge, UK, 1–18, https://doi.org/10.1017/CBO9780511617652.004, 1996.
  
  Orrell, D., Smith, L., Barkmeijer, J., and Palmer, T. N.: Model error in weather forecasting, Nonlin. Processes Geophys., 8, 357–371, https://doi.org/10.5194/npg-8-357-2001, 2001.
  
  Orrell, D.: Role of the metric in forecast error growth: how chaotic is the weather?, Tellus, 54, 350-362, https://doi.org/10.1034/j.1600-0870.2002.01389.x, 2002.
  
  Zhang, F., Sun, Q., Magnusson, L., Buizza, R., Lin, S. H.,Chen J. H., and Emanuel K.: What is the Predictability Limit of Multilatitude Weather, J. Atmos. Sci., 76, 1077–1091, https://doi.org/10.1175/JAS-D-18-0269.1, 2019.
  
  Lorenz, E. N.: The predictability of a flow which possesses many scales of motion, Tellus, 21, 289–307, https://doi.org/10.1111/j.2153-3490.1969.tb00444.x, 1969.
  
  Citation: https://doi.org/10.5194/egusphere-2023-1464-AC2

Hynek Bednář and Holger Kantz

Data sets

Analysis of model error in forecast errors of Extended Atmospheric Lorenz' 05 Systems and the ECMWF system Hynek Bednář http://www.doi.org/10.17605/OSF.IO/2EWXB

Model code and software

Analysis of model error in forecast errors of Extended Atmospheric Lorenz' 05 Systems and the ECMWF system Hynek Bednář http://www.doi.org/10.17605/OSF.IO/2EWXB

Hynek Bednář and Holger Kantz

Viewed

Total article views: 350 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	BibTeX	EndNote
241	88	21	350	15	16

HTML: 241
PDF: 88
XML: 21
Total: 350
BibTeX: 15
EndNote: 16

Views and downloads (calculated since 04 Aug 2023)

Month	HTML	PDF	XML	Total
Aug 2023	86	24	5	115
Sep 2023	39	13	4	56
Oct 2023	38	12	4	54
Nov 2023	12	8	2	22
Dec 2023	10	1	1	12
Jan 2024	10	3	1	14
Feb 2024	16	10	1	27
Mar 2024	13	10	0	23
Apr 2024	17	7	3	27

Cumulative views and downloads (calculated since 04 Aug 2023)

Month	HTML	PDF	XML	Total
Aug 2023	86	24	5	115
Sep 2023	39	13	4	56
Oct 2023	38	12	4	54
Nov 2023	12	8	2	22
Dec 2023	10	1	1	12
Jan 2024	10	3	1	14
Feb 2024	16	10	1	27
Mar 2024	13	10	0	23
Apr 2024	17	7	3	27

Viewed (geographical distribution)

Total article views: 349 (including HTML, PDF, and XML) Thereof 349 with geography defined and 0 with unknown origin.

Country	#	Views	%

Latest update: 28 Apr 2024

Short summary

The forecast error growth of atmospheric phenomena is caused by initial and model errors. When studying the initial error growth, it turns out that small scale phenomena, which contribute little to the forecast product, significantly affect the ability to predict this. With a negative result, we investigate (in the extended Lorenz (2005) system) whether omitting these phenomena will improve predictability. A theory explaining and describing this behavior is developed.


Total:	0
HTML:	0
PDF:	0
XML:	0