the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 11 Oct 2024
Parameterisation toolbox for physical-biogeochemical model compatible with FABM. Case study: the coupled 1D GOTM-ECOSMO E2E for the Sylt-Romo Bight, North Sea
Abstract. Mathematical models serve as invaluable tool for comprehending marine ecosystems. The performance of these models is often highly dependent on their parameters. Traditionally, refining these models involved a time-intensive trial-and-error approach to identify model parameter values that are able to reproduce observations well. However, as ecosystem models grow in complexity, this approach becomes impractical. With advancements in computing power, optimization techniques have emerged as a viable alternative. Yet, these techniques often exhibit model-specific tailoring, limiting their broader application. In this study, we introduce a parameterisation toolbox founded on a Particle Swarm Optimizer (PSO) implemented in the Framework for Aquatic Biogeochemical Models (FABM), which allows its reuse between numerous existing models in FABM, and thus makes the optimizer more accessible to the community. The PSO toolbox's effectiveness is demonstrated through its implementation on a 1D physical-biogeochemical model (GOTM-ECOSMO E2E), which successfully parameterised the Sylt-Romo Bight ecosystem. The toolbox was able to identify most of the tuned parameters and to suggest potential ranges for poorly constrained parameters. In addition, the toolbox uncovers a number of parameter sets with notable differences in some parameter values, but resulting in not much difference in biomass and fluxes. Furthermore, by experimenting with optimisation models of varying complexity, the toolbox was able to define an optimal model for the Sylt-Romo Bight.
- Preprint
(43542 KB) - Metadata XML
- BibTeX
- EndNote
Status: closed
-
RC1: 'Comment on egusphere-2024-2710', Anonymous Referee #1, 14 Nov 2024
General comments:
The particle swam optimizer toolbox described in this paper represents a novel approach to the problem of parameter optimization in marine ecosystem models. The toolbox allows a range of model parameters to be constrained simultaneously. This represents a considerable advantage over tuning individual parameters or variables independently which may lead to improvements in the performance of one variable at the cost of another. The authors demonstrate the successful implementation of the particle swarm optimizer approach within a regional ecosystem. A toolbox is presented that is compatible with FABM and this will significantly aid the portability of the toolbox to other ecosystem models and will support it’s uptake by the wider ecosystem modelling community.
The advantages and disadvantages of toolbox are clearly stated in the paper. However, there are several points I feel require further discussion as listed below.
Specific comments:
Abstract
Line 2: I would argue that several recent studies go beyond a ‘trial-and-error’ approach.
Line 10: Sentence beginning ‘The toolbox was able to…’ the tool box itself didn’t identify the parameters but defined optimal values
Introduction
Line 21: are they incomplete or not devised?
Paragraph starting line 30: Yumruktepe et al., 2023 have described a successful parameter optimization framework utilizing ARGO floats and the same (GOTM-FABM-ECOSMO) model set-up as used in this study. The PSO would potentially fit nicely within the ARGO toolbox allowing identification of a parameter set that is appropriate over larger areas without excessive computational requirements.
Paragraph starting line 46: These hypotheses are referred to again on line 169 but are not mentioned later in the paper, including in the discussion or the section on future work. Has or will the model be used to test these hypotheses?
Parameterization toolbox
Line 60: should bound be boundary?
Line 64: I find the analogy with the two boats confuses the issue - according to the current description the two boats will quickly converge close to the deepest point between them (but not at the deepest point in the lake). The description beginning on line 77 is easy to follow even for a non-mathematician, so maybe the boat analogy could be removed altogether – if kept it should be improved.
Model configuration and set up
No mention is given to the performance of GOTM in simulating the complex physical regime of the region. This is relevant as poorly constrained physics will impact the final parameter set.
Line 189: Was a one-year spin-up enough to achieve a stable ecosystem state?
The choice of parameter space within which parameters are allowed to vary is not discussed. Some model parameters are more tightly constrained than others (based on existing knowledge) so the size of the parameter space within which an individual parameter is allowed to vary may be parameter dependent. Are the authors sure the defined parameter spaces do not exceed meaningful ranges in each individual case?
In cases where the final optimized parameters are significantly different to the reference value some discussion may be merited. For example, looking at Figure 4, the optimal maximum growth rate for large phytoplankton is almost double the reference value. The value of 2.5 is high compared to values typically used and may lie close to the upper boundary of the defined parameter space? Can this high value be justified through consideration of the specific ecosystem processes at work in the study region? Or is there some compensation within the model (such as high growth rates balance by high grazing rates?)
In general consideration should be given to parameters which exhibit very large changes from the reference values. How does the new (optimal) parameter compare to that used in other models? And does it remain within a scientifically meaningful range? What does this say about the system being studied?
Results and discussion
Line 264/5: Why does the model underrepresent phytoplankton biomass at other times of the year?
Paragraph starting on line 268: Presumably river input does not account for NO3 removal. Also why does the lack of river input not impact phosphate in the model?
The 8-year period simulated is still relatively short and does not indicate robustness over longer decadal/multidecadal periods. The robustness seen may also come at a cost – looking at Figure 5 is there some evidence that the model is tuned towards a mean state and underrepresents interannual variability?
Section 3.2
297: Have you defined one parameter set, within which several of the parameters can be allowed to vary within a defined range? If you have define several parameter sets which are actually different then the system dynamics cannot be identical. I feel further discussion is needed around this subject. Choosing a parameter set that achieves the right result for the wrong reasons means the parameter set is likely less portable to other locations or time frames and will also likely have implications for the representation of higher trophic levels within the model system.
Conclusions
Line 327: Neither the hydrodynamic complexity or the success of GOTM in representing this has been discussed in the paper.
Line 333: It may be better to point to the doi given at the end of the paper here? At least an institutional rather than personal site is required to ensure secure long-term access.
Line 350:
I see the identification of different parameter sets as highlighting the limitation of the model and the method, care should be taken not to achieve the right results for the wrong reasons if the model is to be used to study the system dynamics. Although the possibility of defining stable parameter ranges is, however, a plus.
Figures A2 and A3: I did not understand why the black dots sometimes appear to be outside the range of the grey dots.
Other minor comments
The manuscript should be checked for typos/grammatical errors e.g.
Line 1: insert ‘an’ after ‘serve as’
Line 2: remove ‘in recent years’
Line 5: insert a comma after ‘system’ or rewrite the sentence to be clearer
Line 44: remove ‘to’ after ‘also’
Citation: https://doi.org/10.5194/egusphere-2024-2710-RC1 -
AC1: 'Reply on RC1', Hoa Nguyen, 21 Nov 2024
Dear Reviewer,
Thank you very much for taking the time to review my paper and for providing constructive feedback!
I will carefully review and address all of your comments in detail in my formal response later. For now, I would like to provide a quick clarification on a few points:
- Regarding the hypotheses mentioned in the paragraph starting on line 46, these will indeed be investigated later with the model.
- For the parameter space dependency, you are correct that it depends on the parameters. To address this, in our study, we scaled the parameter spaces to the range [0, 1] to avoid this dependency. This detail may not have been sufficiently clear, and I will ensure it is better described in the revised manuscript.
- About Figures A2 and A3, the gray dots were present under the black dots but were not easily visible. I will adjust the size of the gray dots in the revised manuscript to make them more apparent.
Thank you again for your valuable feedback!
Best regards, Hoa
Citation: https://doi.org/10.5194/egusphere-2024-2710-AC1
-
AC1: 'Reply on RC1', Hoa Nguyen, 21 Nov 2024
-
RC2: 'Comment on egusphere-2024-2710', Anonymous Referee #2, 16 Jan 2025
Review of “Parameterisation toolbox for physical-biogeochemical model compatible with FABM. Case study: the coupled 1D GOTM-ECOSMO E2E for the Sylt-Romo Bight, North Sea”
by Hoa Nguyen et al.General comments
The manuscript by Nguyen et al. deals with a key aspect of ecosystem and biogeochemical modelling. Although parameter optimization is apparently a technical aspect, it is deeply rooted in the ecological/biogeochemical understanding. The toolbox presented here appears to be a useful development owing to its implementation within FABM, a generalized framework for the configuration of a wide range of models. Below I provide my general and specific comments.
1) Although the manuscript is generally clearly written, I found many grammar errors (e.g. number concordance) in the Introduction. Please revise grammar thoroughly throughout the text. Also, pay attention to the formatting of the references.
2) Regarding the PSO algorithm (section 3.1 Demonstration of the PSO toolbox)
What is the likelihood that the optimization gets trapped in a local minimum? Is there a means to ensure the PSO algorithm is exploratory enough to avoid local minima? Did the authors made tests in this regard? For example, running the PSO against a model-generated dataset where (a) exact parameter values are known, and (b) the optimized model should be able to converge exactly to the solution.
I missed here a more extended discussion of how the PSO performs in comparison to other parameter optimization approaches (e.g. in terms of number of iterations, total number of executions needed, convergence speed, convergence accuracy…).
3) Section 3.2 (Multiple parameter sets can reproduce observations equally well despite differences in values) is very welcome, as it puts in the spotlight a common (but often overlooked) problem in parameter optimization. However, it falls short at deciphering whether the similar model outputs with different parameter sets result from mutually-compensating processes or from low sensitivity. I am not proposing authors to discover which is the case, but just to let readers know that different scenarios are possible and discuss them briefly. Again, improved use of the literature will enrich this section.
4) Section 3.3 (Optimising model complexity with PSO: top-down control by macrobenthos in the marine ecosystem around Sylt Road). In my modest view this is clearly the most interesting/exciting section. The ability to evaluate models of different complexity in a unified framework is powerful and can illuminate key aspects of ecosystem functioning. Authors may want to further highlight this aspect.Specific comments
L22: “remain incomplete and have not been devised” please use more appropriate wording. Nobody knows what complete bgc model equations look like.
L27: “alternative” to what? or is parameter optimization just a means to enable simplistic models to reproduce observed state variables and fluxes?
L32: “subjective”? I guess the authors meant “objective” or at least quantitative, in the sense that a misfit function is minimized. The three papers cited above (Prieß et al., 2013; Falls et al., 2022; Kern et al., 2024) cannot be deemed to be proposing “subjective” approaches.
L95: perhaps replace neighborhood by population to remain consistent in terms of wording (I understood them as equivalent).
L133: “A quick guide to the PSO Toolbox:” can possibly be removed, or merged into the title of the subsection (currently “PSO with FABM”) to make it more explicit.
L135: suggest replacing “validated” with the less judgmental “evaluated” (usually preferred in modeling frameworks)
L137: “and the intervals and frequency for randomly resetting parameters rather than inferring them from parameters and skill scores from the previous iteration”: this is hard to understand, please explain more clearly or give readers more background information.
L146: Please explain here what the difference is between ECOSMO and ECOSMO E2E.
L153: it seems “calibration” is used as a synonym for “optimization”. Perhaps stick to one term for clarity.
L159: please specify that the Wadden Sea is at the coastal margin of the North Sea.
L186: see comment on L146.
L190: same as L135. Check throughout.
L191-193: please refer to Table A1 and distinguish model parameters (lie “aa”) using italics, or different typography… to help readers.
L204: is guess “the number of model evaluations per iteration” is equivalent to “population size”. Please specify to help readers .
L208: not necessarily the “duration”, but the computing time. If all model evaluations are done in parallel for a given iteration step, there should not be noticeable increases in “duration”.
L225: how does this compare to other optimization approaches? e.g., those cited in the Introduction? for example, Falls et al. (2022) saw convergence during the first 10-20 iterations with a biased random key genetic algorithm. Please discuss this aspect more in depth.
L242: in my own experience, convergence speed may be related to sensitivity, with faster convergence for more sensitive parameters. This information might be a useful addition here.
L248: “ZIt can be seen that about two thirds of the tuned parameters converge on certain values”. And the others? Do they converge? Please rephrase in a less vague way. The meaning of “certain” here is unclear.
Figure 4: why are there several black dots for a given parameter? Do they correspond to different experiments? It feels like some info is missing here.
L267-L275: it seems that the optimized model better captures the mean seasonal cycle, but still fails to capture the interannual variability…? some explanation, even if speculative, to let readers imagine potential workarounds?
L375: fix repetition in this sentence
Figure A2 (and elsewhere): how is convergence defined?
Figures A2-A3: I see that, for some parameters, the best parameter value a at given iteration (black) is far apart from the majority of parameter values evaluated (gray). After several iterations, the black dots and the gray cloud converge. Why does it sometimes take so long? Does this point to the need for adjusting the PSO algorithm to improve performance?Citation: https://doi.org/10.5194/egusphere-2024-2710-RC2
Status: closed
-
RC1: 'Comment on egusphere-2024-2710', Anonymous Referee #1, 14 Nov 2024
General comments:
The particle swam optimizer toolbox described in this paper represents a novel approach to the problem of parameter optimization in marine ecosystem models. The toolbox allows a range of model parameters to be constrained simultaneously. This represents a considerable advantage over tuning individual parameters or variables independently which may lead to improvements in the performance of one variable at the cost of another. The authors demonstrate the successful implementation of the particle swarm optimizer approach within a regional ecosystem. A toolbox is presented that is compatible with FABM and this will significantly aid the portability of the toolbox to other ecosystem models and will support it’s uptake by the wider ecosystem modelling community.
The advantages and disadvantages of toolbox are clearly stated in the paper. However, there are several points I feel require further discussion as listed below.
Specific comments:
Abstract
Line 2: I would argue that several recent studies go beyond a ‘trial-and-error’ approach.
Line 10: Sentence beginning ‘The toolbox was able to…’ the tool box itself didn’t identify the parameters but defined optimal values
Introduction
Line 21: are they incomplete or not devised?
Paragraph starting line 30: Yumruktepe et al., 2023 have described a successful parameter optimization framework utilizing ARGO floats and the same (GOTM-FABM-ECOSMO) model set-up as used in this study. The PSO would potentially fit nicely within the ARGO toolbox allowing identification of a parameter set that is appropriate over larger areas without excessive computational requirements.
Paragraph starting line 46: These hypotheses are referred to again on line 169 but are not mentioned later in the paper, including in the discussion or the section on future work. Has or will the model be used to test these hypotheses?
Parameterization toolbox
Line 60: should bound be boundary?
Line 64: I find the analogy with the two boats confuses the issue - according to the current description the two boats will quickly converge close to the deepest point between them (but not at the deepest point in the lake). The description beginning on line 77 is easy to follow even for a non-mathematician, so maybe the boat analogy could be removed altogether – if kept it should be improved.
Model configuration and set up
No mention is given to the performance of GOTM in simulating the complex physical regime of the region. This is relevant as poorly constrained physics will impact the final parameter set.
Line 189: Was a one-year spin-up enough to achieve a stable ecosystem state?
The choice of parameter space within which parameters are allowed to vary is not discussed. Some model parameters are more tightly constrained than others (based on existing knowledge) so the size of the parameter space within which an individual parameter is allowed to vary may be parameter dependent. Are the authors sure the defined parameter spaces do not exceed meaningful ranges in each individual case?
In cases where the final optimized parameters are significantly different to the reference value some discussion may be merited. For example, looking at Figure 4, the optimal maximum growth rate for large phytoplankton is almost double the reference value. The value of 2.5 is high compared to values typically used and may lie close to the upper boundary of the defined parameter space? Can this high value be justified through consideration of the specific ecosystem processes at work in the study region? Or is there some compensation within the model (such as high growth rates balance by high grazing rates?)
In general consideration should be given to parameters which exhibit very large changes from the reference values. How does the new (optimal) parameter compare to that used in other models? And does it remain within a scientifically meaningful range? What does this say about the system being studied?
Results and discussion
Line 264/5: Why does the model underrepresent phytoplankton biomass at other times of the year?
Paragraph starting on line 268: Presumably river input does not account for NO3 removal. Also why does the lack of river input not impact phosphate in the model?
The 8-year period simulated is still relatively short and does not indicate robustness over longer decadal/multidecadal periods. The robustness seen may also come at a cost – looking at Figure 5 is there some evidence that the model is tuned towards a mean state and underrepresents interannual variability?
Section 3.2
297: Have you defined one parameter set, within which several of the parameters can be allowed to vary within a defined range? If you have define several parameter sets which are actually different then the system dynamics cannot be identical. I feel further discussion is needed around this subject. Choosing a parameter set that achieves the right result for the wrong reasons means the parameter set is likely less portable to other locations or time frames and will also likely have implications for the representation of higher trophic levels within the model system.
Conclusions
Line 327: Neither the hydrodynamic complexity or the success of GOTM in representing this has been discussed in the paper.
Line 333: It may be better to point to the doi given at the end of the paper here? At least an institutional rather than personal site is required to ensure secure long-term access.
Line 350:
I see the identification of different parameter sets as highlighting the limitation of the model and the method, care should be taken not to achieve the right results for the wrong reasons if the model is to be used to study the system dynamics. Although the possibility of defining stable parameter ranges is, however, a plus.
Figures A2 and A3: I did not understand why the black dots sometimes appear to be outside the range of the grey dots.
Other minor comments
The manuscript should be checked for typos/grammatical errors e.g.
Line 1: insert ‘an’ after ‘serve as’
Line 2: remove ‘in recent years’
Line 5: insert a comma after ‘system’ or rewrite the sentence to be clearer
Line 44: remove ‘to’ after ‘also’
Citation: https://doi.org/10.5194/egusphere-2024-2710-RC1 -
AC1: 'Reply on RC1', Hoa Nguyen, 21 Nov 2024
Dear Reviewer,
Thank you very much for taking the time to review my paper and for providing constructive feedback!
I will carefully review and address all of your comments in detail in my formal response later. For now, I would like to provide a quick clarification on a few points:
- Regarding the hypotheses mentioned in the paragraph starting on line 46, these will indeed be investigated later with the model.
- For the parameter space dependency, you are correct that it depends on the parameters. To address this, in our study, we scaled the parameter spaces to the range [0, 1] to avoid this dependency. This detail may not have been sufficiently clear, and I will ensure it is better described in the revised manuscript.
- About Figures A2 and A3, the gray dots were present under the black dots but were not easily visible. I will adjust the size of the gray dots in the revised manuscript to make them more apparent.
Thank you again for your valuable feedback!
Best regards, Hoa
Citation: https://doi.org/10.5194/egusphere-2024-2710-AC1
-
AC1: 'Reply on RC1', Hoa Nguyen, 21 Nov 2024
-
RC2: 'Comment on egusphere-2024-2710', Anonymous Referee #2, 16 Jan 2025
Review of “Parameterisation toolbox for physical-biogeochemical model compatible with FABM. Case study: the coupled 1D GOTM-ECOSMO E2E for the Sylt-Romo Bight, North Sea”
by Hoa Nguyen et al.General comments
The manuscript by Nguyen et al. deals with a key aspect of ecosystem and biogeochemical modelling. Although parameter optimization is apparently a technical aspect, it is deeply rooted in the ecological/biogeochemical understanding. The toolbox presented here appears to be a useful development owing to its implementation within FABM, a generalized framework for the configuration of a wide range of models. Below I provide my general and specific comments.
1) Although the manuscript is generally clearly written, I found many grammar errors (e.g. number concordance) in the Introduction. Please revise grammar thoroughly throughout the text. Also, pay attention to the formatting of the references.
2) Regarding the PSO algorithm (section 3.1 Demonstration of the PSO toolbox)
What is the likelihood that the optimization gets trapped in a local minimum? Is there a means to ensure the PSO algorithm is exploratory enough to avoid local minima? Did the authors made tests in this regard? For example, running the PSO against a model-generated dataset where (a) exact parameter values are known, and (b) the optimized model should be able to converge exactly to the solution.
I missed here a more extended discussion of how the PSO performs in comparison to other parameter optimization approaches (e.g. in terms of number of iterations, total number of executions needed, convergence speed, convergence accuracy…).
3) Section 3.2 (Multiple parameter sets can reproduce observations equally well despite differences in values) is very welcome, as it puts in the spotlight a common (but often overlooked) problem in parameter optimization. However, it falls short at deciphering whether the similar model outputs with different parameter sets result from mutually-compensating processes or from low sensitivity. I am not proposing authors to discover which is the case, but just to let readers know that different scenarios are possible and discuss them briefly. Again, improved use of the literature will enrich this section.
4) Section 3.3 (Optimising model complexity with PSO: top-down control by macrobenthos in the marine ecosystem around Sylt Road). In my modest view this is clearly the most interesting/exciting section. The ability to evaluate models of different complexity in a unified framework is powerful and can illuminate key aspects of ecosystem functioning. Authors may want to further highlight this aspect.Specific comments
L22: “remain incomplete and have not been devised” please use more appropriate wording. Nobody knows what complete bgc model equations look like.
L27: “alternative” to what? or is parameter optimization just a means to enable simplistic models to reproduce observed state variables and fluxes?
L32: “subjective”? I guess the authors meant “objective” or at least quantitative, in the sense that a misfit function is minimized. The three papers cited above (Prieß et al., 2013; Falls et al., 2022; Kern et al., 2024) cannot be deemed to be proposing “subjective” approaches.
L95: perhaps replace neighborhood by population to remain consistent in terms of wording (I understood them as equivalent).
L133: “A quick guide to the PSO Toolbox:” can possibly be removed, or merged into the title of the subsection (currently “PSO with FABM”) to make it more explicit.
L135: suggest replacing “validated” with the less judgmental “evaluated” (usually preferred in modeling frameworks)
L137: “and the intervals and frequency for randomly resetting parameters rather than inferring them from parameters and skill scores from the previous iteration”: this is hard to understand, please explain more clearly or give readers more background information.
L146: Please explain here what the difference is between ECOSMO and ECOSMO E2E.
L153: it seems “calibration” is used as a synonym for “optimization”. Perhaps stick to one term for clarity.
L159: please specify that the Wadden Sea is at the coastal margin of the North Sea.
L186: see comment on L146.
L190: same as L135. Check throughout.
L191-193: please refer to Table A1 and distinguish model parameters (lie “aa”) using italics, or different typography… to help readers.
L204: is guess “the number of model evaluations per iteration” is equivalent to “population size”. Please specify to help readers .
L208: not necessarily the “duration”, but the computing time. If all model evaluations are done in parallel for a given iteration step, there should not be noticeable increases in “duration”.
L225: how does this compare to other optimization approaches? e.g., those cited in the Introduction? for example, Falls et al. (2022) saw convergence during the first 10-20 iterations with a biased random key genetic algorithm. Please discuss this aspect more in depth.
L242: in my own experience, convergence speed may be related to sensitivity, with faster convergence for more sensitive parameters. This information might be a useful addition here.
L248: “ZIt can be seen that about two thirds of the tuned parameters converge on certain values”. And the others? Do they converge? Please rephrase in a less vague way. The meaning of “certain” here is unclear.
Figure 4: why are there several black dots for a given parameter? Do they correspond to different experiments? It feels like some info is missing here.
L267-L275: it seems that the optimized model better captures the mean seasonal cycle, but still fails to capture the interannual variability…? some explanation, even if speculative, to let readers imagine potential workarounds?
L375: fix repetition in this sentence
Figure A2 (and elsewhere): how is convergence defined?
Figures A2-A3: I see that, for some parameters, the best parameter value a at given iteration (black) is far apart from the majority of parameter values evaluated (gray). After several iterations, the black dots and the gray cloud converge. Why does it sometimes take so long? Does this point to the need for adjusting the PSO algorithm to improve performance?Citation: https://doi.org/10.5194/egusphere-2024-2710-RC2
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 187 | 61 | 199 | 447 | 14 | 15 |
- HTML: 187
- PDF: 61
- XML: 199
- Total: 447
- BibTeX: 14
- EndNote: 15
Viewed (geographical distribution)
| Country | # | Views | % |
|---|---|---|---|
| United States of America | 1 | 119 | 26 |
| Germany | 2 | 33 | 7 |
| Norway | 3 | 29 | 6 |
| France | 4 | 22 | 4 |
| Romania | 5 | 21 | 4 |
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
- 119