the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 15 Apr 2025
Linear Meta-Model optimization for regional climate models (LiMMo version 1.0)
Abstract. A new tool for objective parameter tuning of regional climate models is presented. The climate model output was emulated using a linear regression approach for each grid point on a monthly mean basis (Linear Meta-Model – LiMMo). This linear approximation showed high accuracy over a 6-year period. The error norm between the Meta-Model and the observational data sets was minimized using the gradient-based, limited-memory Broyden-Fletcher-Goldfarb-Shanno method with box constraints. The LiMMo framework was applied to the state-of-the-art regional climate model ICON-CLM, tuned to the E-OBS and HOAPS observational data sets. Different optimization objectives were explored by assigning varying weights to model variables in the error norm definition. The combination of a linear emulator with fast gradient-based optimization allows the proposed method to scale linearly with the number of model variables and parameters, facilitating the tuning of dozens of parameters simultaneously.
- Preprint
(3541 KB) - Metadata XML
- BibTeX
- EndNote
Status: closed
-
RC1: 'Comment on egusphere-2025-710', Anonymous Referee #1, 12 May 2025
This manuscript discusses a tuning method (LiMMo) for regional climate models. LiMMo is based on linear regression. In the manuscript, LiMMo is tested on a sample tuning run.
Major comment:
My main question relates to regularization of LiMMo’s regression. LiMMo appears to use linear regression without any sort of regularization, e.g., ridge regression or LASSO regression. A common problem with un-regularized linear regression is that it sometimes yields large “optimal” parameter values that delicately cancel each other’s effects. Then those parameter values lead to a poor result when used in a non-linear model like ICON-CLM. However, LiMMo doesn’t seem to suffer from this problem in the example tuning run presented (see Fig. 6 and the discussion in the manuscript). Please discuss how this problem is avoided in your run. In addition, please do a tuning run in which the range of each parameter, p_max - p_min, is doubled, and then recalculate the R2 values and re-create the plots in Fig. (6). In general, with the range doubled, does regression yield large parameter values that behave poorly in ICON-CLM?
Minor comments:
Equation (7): What are the values of Delta_p_m used in your tuning runs? How is Delta_p_m related to p_m_min and p_m_max?
Lines 320-321: “Test samples were generated by simultaneously varying these parameters within the Latin Hypercube around the minimum and maximum values”. Why does this sentence say “around” rather than “between”? Are the samples allowed to include values less than p_min or greater than p_max? Can you give more details about how this Latin Hypercube sample is constructed?
Equation (9): The logical switches, p_l, must take integer values of 0 or 1, but linear regression would seem to yield optimal values of p_l that are real numbers. How does LiMMo convert between the real values yielded by regression and the integer values of, e.g., Fig. (9)?
Equations (14)-(15): Please clarify the notation “min/max”. I was initially confused by whether Eqn. (14) was to be interpreted as really two equations, one for REG_min and one for REG_max, or instead whether REG_min/max was a single variable. It wasn’t clear until I reached Eqn. (15) that the former interpretation is the intended one. To clarify, the authors could, for example, simply write the equation before Eqn. (14) as an equation for REG_min and state that a similar equation holds for REG_max.
What is plotted in Fig. 6 is not clear to me. I am guessing that Fig. 6 evaluates whether the regression model yields the same result as the ICON-CLM model run for the same configuration and set of parameter values. Is this true? What does each grey dot represent? Is it a single grid point for a single month? Readers might be interested to see plots of other variables, in addition to rsds and pr_amount.
What is plotted in Fig. 7 is also unclear. It is apparently meant to assess the “linear approximation error”, but then it plots RMSE relative to obs. However, it’s possible for both the regression and ICON-CLM to have the same RMSE but different spatial patterns. A different spatial pattern would indicate that the linear approximation error is large, but this large error wouldn’t be reflected in RMSE. Also, I don’t understand how the ICON-CLM result can sometimes have lower RMSE (better accuracy) than the regression. The regression is approximating the optimum, but often ICON-CLM appears to do even better than the optimum. In addition to Fig. 7, it might be helpful to simply plot spatial maps of, e.g., pr_amount from the regression model next to pr_amount from ICON-CLM.
Citation: https://doi.org/10.5194/egusphere-2025-710-RC1 - AC1: 'Reply on RC1', Sergei Petrov, 09 Jul 2025
-
RC2: 'Comment on egusphere-2025-710', Anonymous Referee #2, 28 May 2025
First of all, I thank the authors for this articles. I think its core concept and developments are valuable, interesting and worth publishing. I had a great time reading it.
1. My biggest comment on the content is the following. For the study to be complete, I believe there lacks a comparison with another method (e.g., quadratic regression) which could be expected to result in a better set of parameters at the expense of more time and computing resources. I do not request adding this comparison to the study. If you do, I think it would be extremely valuable (I know that your goal is to be cost-efficient, but the method is anyways, yet here we are talking about presenting it for the first time, which could deserve a one-time investment in order to more clearly situate its pros and cons with respects to other methods), but if you don’t, please at least develop on the potential problems brought by the linear approximation. I think that the current manuscript version is very superficial on that point, highlighting the cost advantage but overlooking the cons.
2. Another comment for the content is about L327 ("occasionally yielding negative precipitation values"). Doesn't this deserve more attention than a remark? We wonder whether this has consequences for the optimization, if you would advise something to go around this (e.g., add a conditional check in the regression to forbid out of range variable results and make those fall back to the range limit, in this case zero. I don't know, really, it's a proposition).
3. The remaining of my comments are about the form. The most serious is about the introduction, which, in my opinion, is problematic.
First, it relies on the abstract. For instance, the objective (designing a flexible cost-effective tuning tool) is not clearly stated (it is only the first sentence of the abstract), although all the decisions to reach it are developed from paragraph 2. In addition, the LiMMo acronym is employed without introduction (except from the abstract), relying on the implicit understanding from the reader about the paper’s goal.
Second, the bibliography presented is very weak, emphasizing 2 examples of tuning techniques in the first paragraph, then 3 previous articles using quadratic regression, and the “well established” reference to justify the sample number formula in the context of Monte Carlo. There is no reference for Meta-Models in general, linear regression approaches and how it is perceived by previous Meta-Model studies (no mention in previous research would even be surprising enough to precisely mention that it’s lacking), gradient-based optimizations, or the choice of objective functions (“RMSE is not enough” by Liemohn et al., 2021, for instance, is clearly relevant). You may then answer that some things are original ideas, but in this case, well there is a third point.
Third, many of the introduction is about explaining the choices made for LiMMo and stating about the advantages of the method. In my opinion, this is not introductory but rather about the methodology or even conclusion (for paragraphs L59 and L66 about the applicability of LiMMo).
In the end, the introduction just feels like a detailed abstract, which, I believe, is not what it should be. The introduction should present the field of your study (i.e., model parameter optimization), explain what has already been proposed in the literature (more extensively than in the current version, explain their procedure as you do for you: how do they choose the metrics, the weights, the optimization process, etc.), the pros, the cons. Then you explain the paper aims to fill the gap of flexible, cost-efficient parameter optimization by proposing a new method. You introduce BRIEFLY your choices, which you develop only in the methodology.
4. I’d actually suggest two different sections, one “materials” presenting ICON-CLM, its variables, tuning parameters and observational datasets, and one separate “the LiMMo framework” presenting the error norm, linear approximation, gradient-based optimization and the formulas of sensitivity. The separate section on LiMMo would facilitate the readability and applicability of your method by the readers, I believe. And then the “results” section is, in fact, entirely an application of your parameter optimization framework, from the sensitivity to the selection of parameters, to ICON-CLM.
5. Now more specific comments:
L26-28: This relies on the implicit assumption that we understand what you are going to do, i.e., use Meta-Model. Then, in this context, you choose the regression-based approach. Please make it explicit, e.g., writing in the previous sentence: "This approach is referred as objective tuning, or objective calibration, and this is the focus of our study." In addition, the detail about "for each grid point and time step" seems fairly early in the text, and feels more like a methodology part explanation.
Paragraph starting L29: The use of hyphens (which should be en or em dashes depending on the convention) for explaining the minimum number of required dynamic simulations is confusing in my opinion, because subjects are numbers and I'd very naturally read them "minus" at the first pass. Consider using colons, "i.e." or "that is", instead.
L35,36: Your explanation corresponds to two formulas, i.e., two interpretations of the linear regression. If the reference simulation is fixed, then the formula is indeed N + 1, but if there is a new reference simulation each time you change of parameters, then the formula is simply 2 N. I think mentioning the 2 N option is confusing, especially since this is not the chosen approach.
L51: Is it "perfect", really?
L86: Consider using a table rather than a list
L98: Please highlight the nature of the E-OBS dataset (assimilating model, satellite-based product, satellite-station merge, ...)
L105: Please provide the version and associated reference for the COARE algorithm.
L219: Please mention somewhere (for instance in the introduction) existing objective ways to define weights in Multi-Criteria Decision-Making (e.g., entropy weights)
L255: Please split the paragraph after single norm evaluation O(...) and an additional sentence clarifying that the optimization then seeks for the vector p that minimizes Eq. 11. Address the optimization method in another paragraph.
Relating to the axes and titles in Fig. 3 and 4, please specify clearly that "Score = Eq. 4 = Objective function value", and that "Score gradient = Gradient norm value = Eq. 11" and that iterations are made with the p vector (or something else, if I misunderstood). Otherwise, please make the terms more consistent.
Eq. 14: I'd highly prefer to see the sensitivity benchmark in the methodology section. In the current version, those case-independent equations feels odd after having introduced a result section.
Fig. 5: Please use a uniform decreasing intensity colormap such as cmocean's amp here. The diverging one of the current version makes no sense because the center (white) is not indicated and has no meaning anyways. Moreover, please address the contrast between text and background (use white text color under a certain threshold of background intensity). Also, consider using the average rather than the sum, so as to include the column in the coloring.
L301: "The heat flux..." I find that there are too many exceptions in the figure (rsds is the lowest for rlh, or pr is quite high as well although not mentioned) to state this that way.
L320: You mean one single sample vector p, right? (it's a single simulation in Fig. 6, correct?)
L321: "around tthe minimum and maximum values" What does this mean? The parameters were not taken out of their range, were they?
L321: "Due to limited..." It is unclear whether this subset is the same as in this paragraph's second sentence or a additional filtering within this subset.
Fig. 7: I do not understand what are the markers for. Is that for several sets of parameters? Please clarify this because it is not easily understandable after Fig. 6. Also, consider bigger markers or other shapes. It is currently difficult to distinguish between round-like markers (penta and hectagonal).
Section 3.4: Please remind the readers that introducing logical switches does not affect the optimization results for continuous parameters, and that the process simply consists of computing the error using the regression Eq. 9 for all new eight possibilities.
L414: "objectively" does not make sense if it's adapted to the user's priorities.
Citation: https://doi.org/10.5194/egusphere-2025-710-RC2 - AC2: 'Reply on RC2', Sergei Petrov, 09 Jul 2025
Status: closed
-
RC1: 'Comment on egusphere-2025-710', Anonymous Referee #1, 12 May 2025
This manuscript discusses a tuning method (LiMMo) for regional climate models. LiMMo is based on linear regression. In the manuscript, LiMMo is tested on a sample tuning run.
Major comment:
My main question relates to regularization of LiMMo’s regression. LiMMo appears to use linear regression without any sort of regularization, e.g., ridge regression or LASSO regression. A common problem with un-regularized linear regression is that it sometimes yields large “optimal” parameter values that delicately cancel each other’s effects. Then those parameter values lead to a poor result when used in a non-linear model like ICON-CLM. However, LiMMo doesn’t seem to suffer from this problem in the example tuning run presented (see Fig. 6 and the discussion in the manuscript). Please discuss how this problem is avoided in your run. In addition, please do a tuning run in which the range of each parameter, p_max - p_min, is doubled, and then recalculate the R2 values and re-create the plots in Fig. (6). In general, with the range doubled, does regression yield large parameter values that behave poorly in ICON-CLM?
Minor comments:
Equation (7): What are the values of Delta_p_m used in your tuning runs? How is Delta_p_m related to p_m_min and p_m_max?
Lines 320-321: “Test samples were generated by simultaneously varying these parameters within the Latin Hypercube around the minimum and maximum values”. Why does this sentence say “around” rather than “between”? Are the samples allowed to include values less than p_min or greater than p_max? Can you give more details about how this Latin Hypercube sample is constructed?
Equation (9): The logical switches, p_l, must take integer values of 0 or 1, but linear regression would seem to yield optimal values of p_l that are real numbers. How does LiMMo convert between the real values yielded by regression and the integer values of, e.g., Fig. (9)?
Equations (14)-(15): Please clarify the notation “min/max”. I was initially confused by whether Eqn. (14) was to be interpreted as really two equations, one for REG_min and one for REG_max, or instead whether REG_min/max was a single variable. It wasn’t clear until I reached Eqn. (15) that the former interpretation is the intended one. To clarify, the authors could, for example, simply write the equation before Eqn. (14) as an equation for REG_min and state that a similar equation holds for REG_max.
What is plotted in Fig. 6 is not clear to me. I am guessing that Fig. 6 evaluates whether the regression model yields the same result as the ICON-CLM model run for the same configuration and set of parameter values. Is this true? What does each grey dot represent? Is it a single grid point for a single month? Readers might be interested to see plots of other variables, in addition to rsds and pr_amount.
What is plotted in Fig. 7 is also unclear. It is apparently meant to assess the “linear approximation error”, but then it plots RMSE relative to obs. However, it’s possible for both the regression and ICON-CLM to have the same RMSE but different spatial patterns. A different spatial pattern would indicate that the linear approximation error is large, but this large error wouldn’t be reflected in RMSE. Also, I don’t understand how the ICON-CLM result can sometimes have lower RMSE (better accuracy) than the regression. The regression is approximating the optimum, but often ICON-CLM appears to do even better than the optimum. In addition to Fig. 7, it might be helpful to simply plot spatial maps of, e.g., pr_amount from the regression model next to pr_amount from ICON-CLM.
Citation: https://doi.org/10.5194/egusphere-2025-710-RC1 - AC1: 'Reply on RC1', Sergei Petrov, 09 Jul 2025
-
RC2: 'Comment on egusphere-2025-710', Anonymous Referee #2, 28 May 2025
First of all, I thank the authors for this articles. I think its core concept and developments are valuable, interesting and worth publishing. I had a great time reading it.
1. My biggest comment on the content is the following. For the study to be complete, I believe there lacks a comparison with another method (e.g., quadratic regression) which could be expected to result in a better set of parameters at the expense of more time and computing resources. I do not request adding this comparison to the study. If you do, I think it would be extremely valuable (I know that your goal is to be cost-efficient, but the method is anyways, yet here we are talking about presenting it for the first time, which could deserve a one-time investment in order to more clearly situate its pros and cons with respects to other methods), but if you don’t, please at least develop on the potential problems brought by the linear approximation. I think that the current manuscript version is very superficial on that point, highlighting the cost advantage but overlooking the cons.
2. Another comment for the content is about L327 ("occasionally yielding negative precipitation values"). Doesn't this deserve more attention than a remark? We wonder whether this has consequences for the optimization, if you would advise something to go around this (e.g., add a conditional check in the regression to forbid out of range variable results and make those fall back to the range limit, in this case zero. I don't know, really, it's a proposition).
3. The remaining of my comments are about the form. The most serious is about the introduction, which, in my opinion, is problematic.
First, it relies on the abstract. For instance, the objective (designing a flexible cost-effective tuning tool) is not clearly stated (it is only the first sentence of the abstract), although all the decisions to reach it are developed from paragraph 2. In addition, the LiMMo acronym is employed without introduction (except from the abstract), relying on the implicit understanding from the reader about the paper’s goal.
Second, the bibliography presented is very weak, emphasizing 2 examples of tuning techniques in the first paragraph, then 3 previous articles using quadratic regression, and the “well established” reference to justify the sample number formula in the context of Monte Carlo. There is no reference for Meta-Models in general, linear regression approaches and how it is perceived by previous Meta-Model studies (no mention in previous research would even be surprising enough to precisely mention that it’s lacking), gradient-based optimizations, or the choice of objective functions (“RMSE is not enough” by Liemohn et al., 2021, for instance, is clearly relevant). You may then answer that some things are original ideas, but in this case, well there is a third point.
Third, many of the introduction is about explaining the choices made for LiMMo and stating about the advantages of the method. In my opinion, this is not introductory but rather about the methodology or even conclusion (for paragraphs L59 and L66 about the applicability of LiMMo).
In the end, the introduction just feels like a detailed abstract, which, I believe, is not what it should be. The introduction should present the field of your study (i.e., model parameter optimization), explain what has already been proposed in the literature (more extensively than in the current version, explain their procedure as you do for you: how do they choose the metrics, the weights, the optimization process, etc.), the pros, the cons. Then you explain the paper aims to fill the gap of flexible, cost-efficient parameter optimization by proposing a new method. You introduce BRIEFLY your choices, which you develop only in the methodology.
4. I’d actually suggest two different sections, one “materials” presenting ICON-CLM, its variables, tuning parameters and observational datasets, and one separate “the LiMMo framework” presenting the error norm, linear approximation, gradient-based optimization and the formulas of sensitivity. The separate section on LiMMo would facilitate the readability and applicability of your method by the readers, I believe. And then the “results” section is, in fact, entirely an application of your parameter optimization framework, from the sensitivity to the selection of parameters, to ICON-CLM.
5. Now more specific comments:
L26-28: This relies on the implicit assumption that we understand what you are going to do, i.e., use Meta-Model. Then, in this context, you choose the regression-based approach. Please make it explicit, e.g., writing in the previous sentence: "This approach is referred as objective tuning, or objective calibration, and this is the focus of our study." In addition, the detail about "for each grid point and time step" seems fairly early in the text, and feels more like a methodology part explanation.
Paragraph starting L29: The use of hyphens (which should be en or em dashes depending on the convention) for explaining the minimum number of required dynamic simulations is confusing in my opinion, because subjects are numbers and I'd very naturally read them "minus" at the first pass. Consider using colons, "i.e." or "that is", instead.
L35,36: Your explanation corresponds to two formulas, i.e., two interpretations of the linear regression. If the reference simulation is fixed, then the formula is indeed N + 1, but if there is a new reference simulation each time you change of parameters, then the formula is simply 2 N. I think mentioning the 2 N option is confusing, especially since this is not the chosen approach.
L51: Is it "perfect", really?
L86: Consider using a table rather than a list
L98: Please highlight the nature of the E-OBS dataset (assimilating model, satellite-based product, satellite-station merge, ...)
L105: Please provide the version and associated reference for the COARE algorithm.
L219: Please mention somewhere (for instance in the introduction) existing objective ways to define weights in Multi-Criteria Decision-Making (e.g., entropy weights)
L255: Please split the paragraph after single norm evaluation O(...) and an additional sentence clarifying that the optimization then seeks for the vector p that minimizes Eq. 11. Address the optimization method in another paragraph.
Relating to the axes and titles in Fig. 3 and 4, please specify clearly that "Score = Eq. 4 = Objective function value", and that "Score gradient = Gradient norm value = Eq. 11" and that iterations are made with the p vector (or something else, if I misunderstood). Otherwise, please make the terms more consistent.
Eq. 14: I'd highly prefer to see the sensitivity benchmark in the methodology section. In the current version, those case-independent equations feels odd after having introduced a result section.
Fig. 5: Please use a uniform decreasing intensity colormap such as cmocean's amp here. The diverging one of the current version makes no sense because the center (white) is not indicated and has no meaning anyways. Moreover, please address the contrast between text and background (use white text color under a certain threshold of background intensity). Also, consider using the average rather than the sum, so as to include the column in the coloring.
L301: "The heat flux..." I find that there are too many exceptions in the figure (rsds is the lowest for rlh, or pr is quite high as well although not mentioned) to state this that way.
L320: You mean one single sample vector p, right? (it's a single simulation in Fig. 6, correct?)
L321: "around tthe minimum and maximum values" What does this mean? The parameters were not taken out of their range, were they?
L321: "Due to limited..." It is unclear whether this subset is the same as in this paragraph's second sentence or a additional filtering within this subset.
Fig. 7: I do not understand what are the markers for. Is that for several sets of parameters? Please clarify this because it is not easily understandable after Fig. 6. Also, consider bigger markers or other shapes. It is currently difficult to distinguish between round-like markers (penta and hectagonal).
Section 3.4: Please remind the readers that introducing logical switches does not affect the optimization results for continuous parameters, and that the process simply consists of computing the error using the regression Eq. 9 for all new eight possibilities.
L414: "objectively" does not make sense if it's adapted to the user's priorities.
Citation: https://doi.org/10.5194/egusphere-2025-710-RC2 - AC2: 'Reply on RC2', Sergei Petrov, 09 Jul 2025
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 1,016 | 62 | 20 | 1,098 | 15 | 29 |
- HTML: 1,016
- PDF: 62
- XML: 20
- Total: 1,098
- BibTeX: 15
- EndNote: 29
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1