the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 19 Dec 2022
Data-driven methods to estimate the committor function in conceptual ocean models
Abstract. In recent years, several climate subsystems have been identified that may undergo a relatively rapid transition compared to the changes in their forcing. Such transitions are rare events in general and simulating long-enough trajectories in order to gather sufficient data to determine transition statistics would be too expensive. Conversely, rare-events algorithms like TAMS (Trajectory-Adaptive Multilevel Sampling) encourage the transition while keeping track of the model statistics. However, this algorithm relies on a score function whose choice is crucial to ensure its efficiency. The optimal score function, called committor function, is in practice very difficult to compute. In this paper, we compare different data-based methods (Analogue Markov Chains, Neural Networks, Reservoir Computing, Dynamical Galerkin Approximation) to estimate the committor from trajectory data. We apply these methods on two models of the Atlantic Ocean circulation featuring very different dynamical behavior. We compare these methods in terms of two measures, evaluating how close the estimate is from the true committor, and in terms of the computational time. We find that all methods are able to extract information from the data in order to provide a good estimate of the committor. Analogue Markov Chains provide a very reliable estimate of the true committor in simple models but prove not so robust when applied to systems with a more complex phase space. Neural network methods clearly stand out by their relatively low testing time, and their training time scales more favorably with the complexity of the model than the other methods. In particular, feedforward neural networks consistently achieve the best performance when trained with enough data, making this method promising for committor estimation in sophisticated climate models.
-
Notice on discussion status
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
-
Preprint
(4091 KB)
-
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(4091 KB) - Metadata XML
- BibTeX
- EndNote
- Final revised paper
Journal article(s) based on this preprint
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2022-1362', Anonymous Referee #1, 03 Feb 2023
In the paper "Data-driven methods to estimate the committor function in conceptual ocean models", the authors compare a broad spectrum of data-driven methods for estimating the committor, i.e. the probability to transition conditioned on a starting point. The methods are compared for two climate-relevant models, namely an ocean box model for the AMOC and a model for wind-driven ocean circulation, both of which feature bi-stability, with correspondingly rare noise-induced transitions between the system's fixed points. The main result of the paper consists of a detailed introduction of all four considered methods (feed-forward neural networks, reservoir computing, analogue Markov chain, dynamical Galerkin approximation) and a comparison of their skill in predicting the committor as a function of the amount of provided training data (in the form of seen transitions).
The paper treats a very important problem, in that data-driven methods are clearly the way forward towards more complex models to predict (noise-induced) tipping point behavior in climate models, and push the boundary of methodology for implementing rare event algorithms in such complex models. The manuscript is very readable, cires the relevant literature, and explains a good amount of detail. Concretely, the detailed comparison of the large number of data-driven approaches makes the presented result applicable to a wide range of problems within climate science and beyond.
As a consequence, I am in favor of publication after minor revisions and clarifications as indicated below.
Minor points:
- p10: Equation (21) is unclear to me: u and v are time-series. The inner product <,> is, by context, to be interpreted over the state-space and not time? If so, the index i on the rhs is the temporal index. Why is there a sum over the time index? Is this a time-integral? If so, the rhs is a vector and the lhs a scalar? Fortunately, it seems that equation (21) is not used anywhere later on, but arguably the notation in the section could be clarified.
- p22: The explanation around lines 555 to 565 about 'aborted transitions' is not very satisfactory. In particular, the amount of 'aborted trajectories' is quantified by the committor itself by definition. There cannot be a large fraction of trajectories that reach q=0.5 and then abort. Instead, the fraction of trajectories reaching q=0.5 but not transitioning is exactly 0.5, and the same is true for any other value of the committor (for example, 10% of all trajectories reaching q=0.9 eventually abort). It is therefore unclear how one model can show more aborted transitions than another, or what an aborted transition even is. Maybe I am misunderstanding the intention of this paragraph.- line 180: change "hte" to "the"
- line 362: "using as less data as" to "using as little data as"
- line 543: "this phenomena" to "this phenomenon"
Citation: https://doi.org/10.5194/egusphere-2022-1362-RC1 - AC1: 'Reply on RC1', Valérian Jacques-Dumas, 20 Mar 2023
-
RC2: 'Comment on egusphere-2022-1362', Anonymous Referee #2, 05 Feb 2023
The primary objective of the paper is to provide a thorough empirical study of the popular data-driven methods, i.e. methods that require either simulated or observed transition trajectories, for solving the committor functions. The authors focus on four approaches: analogue methods, neural networks, reservoir computing and dynamical Galerkin approximation and utilizes the AMOC and the double-gyre model to demonstrate their performances. The paper provides a concise review of the four approaches mentioned, followed by detailed numerical evidence and discussions about their accuracies and scalings. On the whole, I found the exposition and goal of the paper to be clear. The numerical results, though I am uncertain of their generality, seem to be rather reasonable benchmarks and good starting points for applications to rare event algorithms. Therefore I'm inclined to accept the publication after minor revisions.
Minor comments:
- Figure 3: as described in the caption, there should be “confidence intervals” for FFNN training time in subplot (c). Or is it just too narrow to see?
- It would be helpful to provide more details about the scaling issue in the AMG method. Overall AMG method seems to be rather competitive especially when the number of available data is limited (which is typically the case in real application). Does the computational bottleneck come from the increasingly expensive KD-tree search or the eigenvalue algorithms for the large analogue matrix? For the latter case, one can exploit the sparsity structure in the analogue matrix and use iterative eigen-solvers. Matrices of size 1e4 by 1e4 seem still efficient to work with.
- The authors focus on the scaling issue from one perspective– the data dimension. However when applying the algorithm to other models, the variable dimension typically plays a critical role and efficient algorithms with moderate complexity growth with the model dimension is indeed the research target in the committor function community. I don’t think it is necessary to provide numerical evidence in this angle but I would suggest the author include a brief discussion and clarify this different definition about “high-dimensional” problems.
- The discussion about computational time is rather long for all models. I would suggest adding theoretical scalings of, say O(N), O(N^2), etc. to figure 3 and 5 to make a clear visualization and comparison.
Typos:
line 19: remove “the”
line 45: “algorithms” to “algorithm”
line 141: “trees” to “tree”
line 170: “computed” to “compute”
line 180: “hte” to “the”
line 224: what does (7) mean?
line 362: “less” to “little”
line 521: “method” to “methods”
line 601: “appproximation” to “approximation”
Citation: https://doi.org/10.5194/egusphere-2022-1362-RC2 - AC2: 'Reply on RC2', Valérian Jacques-Dumas, 20 Mar 2023
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2022-1362', Anonymous Referee #1, 03 Feb 2023
In the paper "Data-driven methods to estimate the committor function in conceptual ocean models", the authors compare a broad spectrum of data-driven methods for estimating the committor, i.e. the probability to transition conditioned on a starting point. The methods are compared for two climate-relevant models, namely an ocean box model for the AMOC and a model for wind-driven ocean circulation, both of which feature bi-stability, with correspondingly rare noise-induced transitions between the system's fixed points. The main result of the paper consists of a detailed introduction of all four considered methods (feed-forward neural networks, reservoir computing, analogue Markov chain, dynamical Galerkin approximation) and a comparison of their skill in predicting the committor as a function of the amount of provided training data (in the form of seen transitions).
The paper treats a very important problem, in that data-driven methods are clearly the way forward towards more complex models to predict (noise-induced) tipping point behavior in climate models, and push the boundary of methodology for implementing rare event algorithms in such complex models. The manuscript is very readable, cires the relevant literature, and explains a good amount of detail. Concretely, the detailed comparison of the large number of data-driven approaches makes the presented result applicable to a wide range of problems within climate science and beyond.
As a consequence, I am in favor of publication after minor revisions and clarifications as indicated below.
Minor points:
- p10: Equation (21) is unclear to me: u and v are time-series. The inner product <,> is, by context, to be interpreted over the state-space and not time? If so, the index i on the rhs is the temporal index. Why is there a sum over the time index? Is this a time-integral? If so, the rhs is a vector and the lhs a scalar? Fortunately, it seems that equation (21) is not used anywhere later on, but arguably the notation in the section could be clarified.
- p22: The explanation around lines 555 to 565 about 'aborted transitions' is not very satisfactory. In particular, the amount of 'aborted trajectories' is quantified by the committor itself by definition. There cannot be a large fraction of trajectories that reach q=0.5 and then abort. Instead, the fraction of trajectories reaching q=0.5 but not transitioning is exactly 0.5, and the same is true for any other value of the committor (for example, 10% of all trajectories reaching q=0.9 eventually abort). It is therefore unclear how one model can show more aborted transitions than another, or what an aborted transition even is. Maybe I am misunderstanding the intention of this paragraph.- line 180: change "hte" to "the"
- line 362: "using as less data as" to "using as little data as"
- line 543: "this phenomena" to "this phenomenon"
Citation: https://doi.org/10.5194/egusphere-2022-1362-RC1 - AC1: 'Reply on RC1', Valérian Jacques-Dumas, 20 Mar 2023
-
RC2: 'Comment on egusphere-2022-1362', Anonymous Referee #2, 05 Feb 2023
The primary objective of the paper is to provide a thorough empirical study of the popular data-driven methods, i.e. methods that require either simulated or observed transition trajectories, for solving the committor functions. The authors focus on four approaches: analogue methods, neural networks, reservoir computing and dynamical Galerkin approximation and utilizes the AMOC and the double-gyre model to demonstrate their performances. The paper provides a concise review of the four approaches mentioned, followed by detailed numerical evidence and discussions about their accuracies and scalings. On the whole, I found the exposition and goal of the paper to be clear. The numerical results, though I am uncertain of their generality, seem to be rather reasonable benchmarks and good starting points for applications to rare event algorithms. Therefore I'm inclined to accept the publication after minor revisions.
Minor comments:
- Figure 3: as described in the caption, there should be “confidence intervals” for FFNN training time in subplot (c). Or is it just too narrow to see?
- It would be helpful to provide more details about the scaling issue in the AMG method. Overall AMG method seems to be rather competitive especially when the number of available data is limited (which is typically the case in real application). Does the computational bottleneck come from the increasingly expensive KD-tree search or the eigenvalue algorithms for the large analogue matrix? For the latter case, one can exploit the sparsity structure in the analogue matrix and use iterative eigen-solvers. Matrices of size 1e4 by 1e4 seem still efficient to work with.
- The authors focus on the scaling issue from one perspective– the data dimension. However when applying the algorithm to other models, the variable dimension typically plays a critical role and efficient algorithms with moderate complexity growth with the model dimension is indeed the research target in the committor function community. I don’t think it is necessary to provide numerical evidence in this angle but I would suggest the author include a brief discussion and clarify this different definition about “high-dimensional” problems.
- The discussion about computational time is rather long for all models. I would suggest adding theoretical scalings of, say O(N), O(N^2), etc. to figure 3 and 5 to make a clear visualization and comparison.
Typos:
line 19: remove “the”
line 45: “algorithms” to “algorithm”
line 141: “trees” to “tree”
line 170: “computed” to “compute”
line 180: “hte” to “the”
line 224: what does (7) mean?
line 362: “less” to “little”
line 521: “method” to “methods”
line 601: “appproximation” to “approximation”
Citation: https://doi.org/10.5194/egusphere-2022-1362-RC2 - AC2: 'Reply on RC2', Valérian Jacques-Dumas, 20 Mar 2023
Peer review completion
Journal article(s) based on this preprint
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 252 | 140 | 16 | 408 | 9 | 3 |
- HTML: 252
- PDF: 140
- XML: 16
- Total: 408
- BibTeX: 9
- EndNote: 3
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
Cited
Valérian Jacques-Dumas
René M. van Westen
Freddy Bouchet
Henk A. Dijkstra
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(4091 KB) - Metadata XML