the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 12 Jun 2024
Development of high-order global dynamical core using discontinuous Galerkin method for atmospheric LES and proposal of test cases: SCALE-DG v0.8.0
Abstract. Focusing on future global atmospheric simulations with grid spacing of O(10–100 m), we developed a global nonhydrostatic atmospheric dynamical core with high-order accuracy by applying discontinuous Galerkin method (DGM) both horizontally and vertically. Further, considering a global large-eddy simulation (LES), a Smagorinsky–Lilly turbulence model was introduced to the proposed global dynamical core in the DGM framework. By conducting several tests with various polynomial orders (p), the impact of high-order DGM on atmospheric flows was investigated. To show high-order numerical convergence, a few modifications were made in the experimental setup of existing test cases. In addition, we proposed an idealized test case to validate global LES models, which is a global extension of idealized planetary boundary layer (PBL) turbulence experiment performed in our previous studies. The error norms from the deterministic test cases, such as linear advection and gravity wave test cases, show an optimal order of spatial accuracy with about p + 1-order when the temporal and round-off errors are sufficiently small. In the climatic test cases, such as the Held-Suarez test, the kinetic energy spectra indicate the advantage of effective resolutions when large polynomial orders are used. In the LES experiment, the global model provided a reasonable vertical structure of PBL and energy spectra since the results under shallow atmosphere approximation well reproduce those obtained in the plane computational domain.
- Preprint
(19295 KB) - Metadata XML
-
Supplement
(6507 KB) - BibTeX
- EndNote
Status: final response (author comments only)
-
RC1: 'Comment on egusphere-2024-1477', Valeria Barra, 16 Jul 2024
General comments:
The paper presents a numerical dynamical core (dycore) for global non-hydrostatic atmospheric simulations. The numerical discretization uses the high-order discontinuous Galerkin method (DGM) both horizontally and vertically and targets global atmospheric simulations in the setting of large-eddy simulation (LES), with grid spacing of O(10–100 m). The paper presents several numerical experiments to verify the numerical framework adopted. The problems are scientifically important and the work seems to have been carried out with care. The scientific significance of the work and novelty compared to other DG dycores are the aspects that concern me the most. The grammatical correctness of the English language is also another aspect that would require further revisions.
I am in general in favor of acceptance to GMD, provided the authors can convincingly respond to the comments.
Specific comments:
- My main concern is with the aim and motivation of the work. Authors state that “Recently developed supercomputers have enabled us to conduct high-resolution global atmospheric simulations using a sub-kilometer horizontal grid spacing”, without commenting or expanding on whether this should really be undertaken as a scientific endeavor. Just because we can, it does not mean that we should. The authors do not seem to outweigh the pros and cons of conducting such numerical simulations, especially in light of the carbon footprint and computational costs associated with said sub-kilometer scale simulations.
- In the literature review, the authors seem to miss to mention the The Nonhydrostatic Unified Model of the Atmosphere (NUMA), which also successfully used DGM
- The authors mention multiple times that they conduct classical numerical experiments to validate their numerical model. However, they seem to confuse Validation with Verification. In Numerical Analysis, the concepts of Verification and Validation (V&V) can be oversimplified in a succinct manner by saying that “verification is solving the equations right” (verifying the numerics) and “validation is solving the right equations” (verifying the physics - often done by comparing model results with actual data given by observations).
- The authors seem to have chosen favorable examples/results and have not sufficiently provided explanations on reasons behind some degraded results, such as when a less than theoretical convergence rate was achieved. Also, the use of filters to overcome numerical instabilities was not always comprehensively justified and their effect on the quality of the results was not extensively elaborated on.
- Section 3, line 331: Can the authors elaborate a little more on why they consider “difficult” to directly evaluate the numerical convergence in those cases?
- In Sec 3.1 for the Linear Advection experiment, I wonder if the authors also verified their solver using a slotted cylinder example. This is often used in the literature because the sharp features of the geometry would particularly challenge the solver. I would appreciate if the authors would conduct such numerical experiments and would compare their convergence rates with the results reported in the literature, e.g., Guba et al. “Optimization-based limiters for the spectral element method” (2014) https://doi.org/10.1016/j.jcp.2014.02.029 looking at the results without limiters. In the same section, regarding the numerical results in Figure 1, the authors have not sufficiently explained why the case with 𝛼 = 0, i.e., no singularity in the coordinates on the cubed-sphere corners, in almost all cases presents larger numerical errors.
- Sec 3.3, line 443: authors mention the modal filter as one potential reason for the degraded sub-optimal convergence. Shouldn’t it help instead? Can they elaborate on this further?
- Section 3.5, Caption of Figure 11: Can the authors explain why they presented numerical results for the highest resolution case with a temporal average over only 300 days as opposed to 1000 days for the other cases? Was it too computationally expensive to perform the highest resolution simulation over 1000 days, or the model presented difficulties over 300 days, such as it suffered from numerical instabilities/crashes?
Technical corrections (the referee will use italic font for addition to the quoted text where appropriate):
- Line 5: “the impact of high-order DGM on atmospheric flows was investigated”. I would rephrase this with another sentence along the lines of: “the impact of high-order DGM on the quality or accuracy of the numerical simulations of atmospheric flows was investigated”
- Line 16: “In the near future”
- Line 18: “Then, large-eddy simulation (LES) is a promising strategy, since in LES”
- Line 33-34: Rephrase “In the context of DGM, KT2023 investigated the problem with the order of accuracy necessary for LES”
- Line 35: Add plural for generic or non specific countable nouns in English, i.e., “modal filters are used”, or add an article if you want to use singular nouns
- Line 36-37: Authors mention “2000-2010” but then they survey literature belonging to the following decade
- Line 64: Please introduce the FDM acronym before using it
- Line 73: “the impact of high-order DGM on the atmospheric flows”. I would rephrase this, similar to the Abstract sentence.
- Line 71-72: “We focused” and then “We attempt”. Please check grammar consistency of temporal tenses throughout the text
- Line 86: “required” -> “requiring”?
- Line 155: “angular velocity of the planet”
- Line 156: “In the numerical experiments”.
- Line 172: “is essentially the same as”
- Line 175: “In the absence of a vertical”
- Line 183: “D is the divergence of the three-dimensional velocity”
- Line 194: “For further details on the turbulence model”
- Line 227: “For the numerical flux of the inviscid terms”
- Line 229: reword “considered”
- Line 230: “transformations, and is formulated as”
- Line 270: “restrict the time step” (remove “to”)
- Line 280: “in the case of the diagonally implicit RK scheme”
- Line 288: “To obtain the solutions of the nonlinear equation system”
- Line 291: “In the case of the collocation approach”
- Line 301: “When using the HEVI approach”
- Line 302: “entries of the matrices”
- Line 304: Rephrase with: “For high-order methods, numerical instability is likely to occur in advection-dominated flows, because the discrete advection operator is oscillatory.”
- Line 310: Remove “represents” or “is”
- Line 317: “the order of the filter”
- Line 318: “at the final stage of the RK scheme”
- Line 320. Rename Section 3 “Verification of the dynamical core”
- Line 322: “we mainly focused on the impact of the polynomial order on the effective”. There are several missing articles throughout the text. I stopped correcting all of them after some point. The authors should more carefully proof-read for English correctness.
- Line 339: I know 𝛼 is used in the literature to denote the angle between the axis of the solid body rotation and the North pole. However, the authors should be careful because they also previously used (𝛼, 𝛽, 𝜁) for the local coordinates on the cubed-sphere.
- Line 379: “errors”
- Line 381: “a modal filter” and “was investigated”
- Line 392: “except for the horizontal wind”
- Line 427: “details on the sponge layer”
- Line 446-447 avoid repetition of “includes” and “included” in the same sentence by using a synonym
- Line 455: “stretched”
- Line 473: “evaluation of the horizontal resolution”
- Line 501: “by using similar spatial resolution”
- Section A3: reword the section title “Investigation on the degradation of the optimal numerical convergence”
- Figure A2: remove bold text in caption
- Figure A3: remove bold text in caption
- Line 741: I am sorry, but even in the acknowledgement sentence in which the authors thank the company they used for the English editing, there is a grammatical error “We would like to thank Editage for the English language editing”
- AC1: 'Reply on RC1', Yuta Kawai, 15 Oct 2024
-
RC2: 'Comment on egusphere-2024-1477', Anonymous Referee #2, 18 Aug 2024
This paper presents a global nonhydrostatic dry atmospheric dynamical core discretised using high order discontinous Galerkin methods. The motivation for using high order methods comes from the authors' previous work which showed that when using an LES turbulence model, the order of accuracy of the spatial discretisation needs to be sufficiently high. This also motivates the use of DG methods as they avoid the large computational stencil required for high order grid point methods. The new model is tested using a range of well known 3D problems, with some changes made to avoid the development of small scales in order to make convergence analysis possible.
The work presented here is of interest and has been carefully done, however it is not clear exactly what is new about this model and how it relates to previous DG / high-order discretisations. In some places additional clarifications and/or references are required and the language needs to be more precise - e.g. words like "attempt" or "about" should be avoided. In many places the results are qualified with "about" and I wonder if it is possible to be more precise, or more confident in what is described. I have highlighted these places below.
I recommend numbering all the equations - this makes it much easier when others discuss your paper!
In the results section, the information on the number of elements, polynomial orders and resulting equatorial resolution is hard to read
- could this information be summarised in a table for each test?Many of the comments below require only minor changes to clarify the text. However, I have selected "major revisions" because I have requested a lot of clarifications and in particular, I think it is very important that the novelty of the method is clarified with reference to other DG dycore publications.
Introduction:
===========line 18: "inertia subrange" should be "inertial subrange"
lines 26-28: This sentence is confusing and the `e-folding time' is not defined or referred to again. As the use of high order methods is motivated by this study, I think it is worth adding another sentence or two here to adequately explain this previous result.
line 30: "which is recognised as a local spectral method" - what do you mean by "local spectral method" or can you cite something here?
line 35: Again this description of the previous result needs either simplifying or clarifying - as it is written it provokes questions: What was special about the case with upwinded numerical flux and sufficiently high order modal filter? What is "sufficiently high order" in this case? What happens more generally?
line 44: Clarify what you mean by "effective resolution significantly apart from the grid spacing". What is your definition of "effective resolution".
line 46: "eight grid spacing" might be clearer as "eight grid lengths". Do you have a reference for this claim?
line 49: "than that in plane domains" the "that" is not needed
line 50: "archive" - I think you mean "achieve"
line 52: "the pole problem" - add a short description of this for those not familiar with the issue.
line 53: "we can suffer..." the "we" here is confusing as I don't think you mean the work you are describing in the paper. I suggest rephrasing as "However, significantly high resolution global simulations can suffer from..."
line 58: "The Climate Machine..." What order is used in this model? How is what you have done different? This section describes a some other DG dycores but it is not clear how the work presented here differs from each one, especially as line 70-71 says "This study includes several progresses from previous studies..." Firstly, "progresses" is not quite the right word here. You could say "This study build on progress from previous studies..." or "This model includes several algorithmic features from previous studies..." but more importantly it needs to be clearer what "progresses" or "features" and what previous studies.
line 76: "provide a chance to modify the experimental setup" - Make it clear that the standard tests were modified for a particular reason related to your aims and the features of your method.
line 79: "Even when the aim of this study..." I am not sure what this sentence means.
line 86: should be "cost required for numerical stabilization" (the "for" is missing)
line 88: "semi-discretization" should be "semi-discretized"
line 89: "which is an extension of..." - more detail would clarify the novelty, e.g. "which extends... by..."
Model description:
===============line 101: "the Jacobian are denoted" - "are" should be "is". On line 104 you give the expression for the Jacobian of the vertical coordinate transformation - you could do the same here for the horizontal coordinate transformation, for consistency and clarity.
line 109: This clarification of the different notation for the coordinate variables is distracting - can you move it to where you actually use this different notation for the first time (I think in the next section (2.2), line 180 onwards)?
equation 1: should S_{SGS} also depend on grad(q) (as in line 137)?
line 151: "where $\delta_S$ is an index..." I think it is a switch rather than an index.
line 208: "effective horizontal grid spacing" - how is this related to the "effective resolution" you talk about elsewhere? Or is it just the spacing between the nodes? In which case, is "effective" the right word?
line 253: should the j be a second subscript of s? it looks like it isn't.
line 270: "severely restrict to the timestep" should be "severely restrict the timestep"
line 319: Should \alpha_m be \alpha_i where i is the index of the highest mode? Or is m the index of the highest mode?
Validation of dynamical core
=======================line 332: as "for this test case" after "Thus" to clarify. Also, do you mean "by focussing on the energy spectra"? This would be clearer if you define what you mean by "effective resolution" and how it relates to the spectra.
line 348: "we set to D=..." should be "we set D=..."
line 355: "Figure 1 shows the numerical errors..." from this I was expecting a spatial plot of the error - the figure shows the dependence of the errors at 12 days on horizontal resolution (as described in the caption) - this description should be in the text too. I'm not sure that the qualifying "about" is necessary in the next sentence.
line 360: "there is less difference between the angles" - less difference in what?
line 367: "This study considered..." should be "This study considers..." or even "This study presents..."
line 375: "effective grid spacing" - is this the same as N_{e, h} as described in the previous test setup? Make sure you are consistent.
line 378: "we set the Courant number against the..." What does this mean? "against" doesn't make sense here - do you mean the acoustic Courant number? Can you define C_{rh, cs} with a formula?
line 402: "is well known..." Do you have any citations here?
Figure 4: The x axis labels should be the same units in each figure.
line 422: "at about z < 15km" - why "about"?
line 423: "a fully explicit" - do you mean the HEVE scheme described earlier? If so, then refer to it.
line 424: "against the" - same comment as earlier - do you mean acoustic Courant number?
line 422: "at about z > 15km" - why "about"?
line 426: "the 1/4 sector" - what is this?
line 429: "to ensure the numerical stability" should be "to ensure numerical stability; "which are summarized" should be "which is summarized"
line 437: "We consider..." I'm not sure what this sentence means.
line 446: "in mid-latitude" should be "in the mid-latitudes"
line 447: Remove "are included"
line 450-1: should be "the adiabatic inviscid primitive equations"
line 455: "stretch" should be "stretched" and could you please specify what stretching you used?
line 456: "about 350m" - why about?
line 457: "against the" - same comment as before - is this the acoustic Courant number?
line 473: "for" should be "of the"
line 475: "cancellation of vertical errors" - I don't understand why this happens - is it really that clean? Can you explain more?
line 478: "sufficiently small compared to... for example" - you cite a specific example here so could you also give the magnitude in that specific example so the reader can compare themselves?
Figure 7: Make it clear in the caption that the top plot is the "reference" solution, or order the plots so that the resolution is increasing either moving upwards or downwards - at the moment it is confusing comparing the different figures.
line 496: "against the" - same comment as before, do you mean the acoustic Courant number?
line 500: say that these are averaged fields.
line 501: "using nearly spatial resolution" - do you mean "nearly the same spatial resolution"? Please clarify.
line 509 and 510: "resolutions" should be "resolution" in both cases
line 510: "about 50km" - why about?
line 513: "by" should be "using"
Figure 9: "averaging" should be "averaged"
line 523: "about 10-20 grids" - what does this mean? grid cells rather than grids?
line 524: "Thus, ..." Could you clarify this sentence? If you replaced "in relatively small polynomial order" with "when using lower polynomial order", would that say what you mean?
line 540: "with 200" should be "of 200"
Figure 12: What is the polynomial order and resolution for these results?
line 542: What is the form of the sponge layer? Refer to appendix A2 again.
line 547: Do you mean the acoustic Courant number? Why "about" 0.438?
line 558: "eight grids" do you mean "eight grid cells"?
line 560: "required polynomial order is p>3" - required for what?; "which is true for results obtained in this study" - make it clear you have shown this by describing the difference for p=3.
Figure 13: "averaging" should be "averaged"; "variable" should be "variance"
Conclusions:
==========line 563: "our previous studies" - cite specifically which ones.
line 567: "considering advantages" - what does this mean? do you mean that the DGM method has advantages? I don't think "considering" is the right word.
lines 593, 594 and 600: "grids" should, I think, be "grid cells"
Code and data availability:
====================="crate" should be create
Could the data go on GitHub LFS?
Appendix A
=========line 627: Can you cite the previous studies that did this?
line 638: Say that alpha is given below.
line 670: why does this equation go on to the next line and what is the dot at the start of line 671 for?
Appendix B
=========line 730: "inertia subrange" should be "inertial subrange"
References:
=========Check formatting - some titles are in all caps.
Citation: https://doi.org/10.5194/egusphere-2024-1477-RC2 - AC2: 'Reply on RC2', Yuta Kawai, 15 Oct 2024
-
RC3: 'Comment on egusphere-2024-1477', Anonymous Referee #3, 02 Sep 2024
Review of https://doi.org/10.5194/egusphere-2024-1477
Yuta Kawai1 and Hirofumi Tomita
Development of high-order global dynamical core using
discontinuous Galerkin method for atmospheric LES and proposal
of test cases: SCALE-DG v0.8.0
General comments:
1. The authors make the claim that for global LES modeling (100km
grid spacing), high-order DG methods will be important in this
context. I dont object to this argument (and dont request
any changes), but I will mention that I dont find the arguments
persuasive. If the arguments are correct, I think DG methods would
be more common in regional models, which often run in the LES regime.
2. One issue not address in this paper is the timestep. DG methods
with the values of p proposed here will be quite expensive. A good
comparison showing how expensive high order DG can be compared to
finite volumes is given in Brdar et al,
https://doi.org/10.1007/s00162-012-0264-z which compares the DG based
DUNE model with the finite volume (operational weather forecast model)
COSMO.
See also my comment below in the conclusions about numerical efficiency.
3. While reading the text, it was clear that the authors use different
settings (timestepping, filtering, and Smagorinsky diffusion) for
the different test cases. This can be good practice during
the development process in order to test specific characteristics
of the dycore. But it is also useful to present results with
the dycore configured as it would be used in practice.
As the authors mention in their conclusions, they have not yet
run the model with realistic topography (which is well known
to create a lot of problems with high order element methods), and
thus the "operational" configuration of SCALE-DG, especially with
regards to how much filtering/diffusion will ultimately be
needed, may not be known.One suggestion would be to also include all test results with
the same configuration used for the planetary boundary layer turbulence
test. If the authors consider that beyond the scope of this paper,
I would request to add a table summarizing all the settings used
for each test.
Specific comments4. line 36: (or line 55) "... some researchers have successfully
developed global nonhydrostatic atmospheric dynamical cores based
... element-based methods"The authors mention some research codes, but neglect recent and
larger efforts using high-order element based methods from major
modeling centers. These include E3SM: ( Caldwell et al., JAMES 2021
e2021MS002544, Donahue et al, JAMES 2024 e2024MS004314 ), the Korian
KIM model (Hong et al, 2018,
https://link.springer.com/article/10.1007/s13143-018-0028-9), and
NRL's NEPTUNE NEPTUNE Model, Kelly et al, 2024,
https://arxiv.org/abs/2405.06076
5. line 75:"Few such studies for global nonhydrostatic dynamical cores are
available although the numerical convergence characteristics of DGM
was investigated for regional dynamical core..."For the citations of regional DG dynamical cores, the authors should also cite:
Brdar et al, https://doi.org/10.1007/s00162-012-0264-zThis part of the introduction focuses only on three dimensional
models, and gives the impression there is limited work on DG
for global atmospheric modeling. There are quite a few papers looking
at DG on the cubed-sphere grid for the shallow water equations, such
as Nair MWR 2005, Ullrich GMD 2014, and the very recent entropy stable
formulations: Ricardo et al, 2024
https://www.sciencedirect.com/science/article/pii/S0021999124000123I also think that the NUMA model from Giraldo et al. (cited in this
text for their regional configuration) has a global version that runs
both DG and CG, but I dont have a reference for that.
A key model that needs to be mentioned is NEPTUNE, which is a global
high order element based method that uses CG and DG, making it quite
similar to SCALE-DG. NEPTUNE is one of the few models I know that is
using 3D higher order elements (as proposed here).
(See NEPTUNE references in Kelly et al, 2024, https://arxiv.org/abs/2405.06076)
6. line 84:
"However, they did not consider the vector Laplacian operator for the
vector quantities (for 85 example, momentum). This might be because
the rigorous form of vector Laplacian is so complex that it may not be
worth the computational cost required numerical stabilization."The authors should note that vector viscosity for both DG and CG was developed in:
Ullrich 2014, https://gmd.copernicus.org/articles/7/3017/2014/gmd-7-3017-2014.pdf
as well (for CG) in
Guba et al., https://gmd.copernicus.org/articles/7/2803/2014/gmd-7-2803-2014.pdf7. line 470 " In addition, the effective resolution is apparently
higher than that of the low-order global dynamical core."See my general comment above. The authors are running different
dissipation/fiter settings for each test case. As the authors mention,
SCALE-DG needs stronger filtering when running more realistic test
cases presented later. It is also very likely that even more
dissipation will be needed when realistic topography is added. For
the baroclinic instability test case, (as well as Held-Suarez) models
are recommend to run with their operational diffusion settings, which
is the case for the FV results. Thus I would qualify this statement,
and note that this might be due to the SCALE-DG's high order
discretization, but it could also be due to using filtering levels that
would not be practical in realistic problems.9. line 567: "and high computational efficiency in recent parallel supercomputers, over
grid-point methods."I doubt this statement is true - given the small timestep required by high order DG
(see comment above). One might be able to make the case that the methods achieve
higher FLOP counts, but most people would interpret computationally efficiency
in terms of time-to-solution.
10. Terminology: The authors use the phrase ""eight grids" and "10~20
grids" several times. It's clear what they mean, but this is an
unusual phrasing and I think technically incorrect because they are
referring to the grid spacing or grid cell width, not the grid itself.
I'd suggest changing to $8 \Delta x$.Citation: https://doi.org/10.5194/egusphere-2024-1477-RC3 - AC3: 'Reply on RC3', Yuta Kawai, 15 Oct 2024
Model code and software
Source codes and setting files for numerical experiments in this study Yuta Kawai https://doi.org/10.5281/zenodo.10901697
Viewed
| HTML | XML | Total | Supplement | BibTeX | EndNote | |
|---|---|---|---|---|---|---|
| 345 | 143 | 32 | 520 | 34 | 14 | 11 |
- HTML: 345
- PDF: 143
- XML: 32
- Total: 520
- Supplement: 34
- BibTeX: 14
- EndNote: 11
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1