the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Datadriven methods to estimate the committor function in conceptual ocean models
Valérian JacquesDumas
René M. van Westen
Freddy Bouchet
Henk A. Dijkstra
Abstract. In recent years, several climate subsystems have been identified that may undergo a relatively rapid transition compared to the changes in their forcing. Such transitions are rare events in general and simulating longenough trajectories in order to gather sufficient data to determine transition statistics would be too expensive. Conversely, rareevents algorithms like TAMS (TrajectoryAdaptive Multilevel Sampling) encourage the transition while keeping track of the model statistics. However, this algorithm relies on a score function whose choice is crucial to ensure its efficiency. The optimal score function, called committor function, is in practice very difficult to compute. In this paper, we compare different databased methods (Analogue Markov Chains, Neural Networks, Reservoir Computing, Dynamical Galerkin Approximation) to estimate the committor from trajectory data. We apply these methods on two models of the Atlantic Ocean circulation featuring very different dynamical behavior. We compare these methods in terms of two measures, evaluating how close the estimate is from the true committor, and in terms of the computational time. We find that all methods are able to extract information from the data in order to provide a good estimate of the committor. Analogue Markov Chains provide a very reliable estimate of the true committor in simple models but prove not so robust when applied to systems with a more complex phase space. Neural network methods clearly stand out by their relatively low testing time, and their training time scales more favorably with the complexity of the model than the other methods. In particular, feedforward neural networks consistently achieve the best performance when trained with enough data, making this method promising for committor estimation in sophisticated climate models.
Valérian JacquesDumas et al.
Status: closed

RC1: 'Comment on egusphere20221362', Anonymous Referee #1, 03 Feb 2023
In the paper "Datadriven methods to estimate the committor function in conceptual ocean models", the authors compare a broad spectrum of datadriven methods for estimating the committor, i.e. the probability to transition conditioned on a starting point. The methods are compared for two climaterelevant models, namely an ocean box model for the AMOC and a model for winddriven ocean circulation, both of which feature bistability, with correspondingly rare noiseinduced transitions between the system's fixed points. The main result of the paper consists of a detailed introduction of all four considered methods (feedforward neural networks, reservoir computing, analogue Markov chain, dynamical Galerkin approximation) and a comparison of their skill in predicting the committor as a function of the amount of provided training data (in the form of seen transitions).
The paper treats a very important problem, in that datadriven methods are clearly the way forward towards more complex models to predict (noiseinduced) tipping point behavior in climate models, and push the boundary of methodology for implementing rare event algorithms in such complex models. The manuscript is very readable, cires the relevant literature, and explains a good amount of detail. Concretely, the detailed comparison of the large number of datadriven approaches makes the presented result applicable to a wide range of problems within climate science and beyond.
As a consequence, I am in favor of publication after minor revisions and clarifications as indicated below.
Minor points:
 p10: Equation (21) is unclear to me: u and v are timeseries. The inner product <,> is, by context, to be interpreted over the statespace and not time? If so, the index i on the rhs is the temporal index. Why is there a sum over the time index? Is this a timeintegral? If so, the rhs is a vector and the lhs a scalar? Fortunately, it seems that equation (21) is not used anywhere later on, but arguably the notation in the section could be clarified.
 p22: The explanation around lines 555 to 565 about 'aborted transitions' is not very satisfactory. In particular, the amount of 'aborted trajectories' is quantified by the committor itself by definition. There cannot be a large fraction of trajectories that reach q=0.5 and then abort. Instead, the fraction of trajectories reaching q=0.5 but not transitioning is exactly 0.5, and the same is true for any other value of the committor (for example, 10% of all trajectories reaching q=0.9 eventually abort). It is therefore unclear how one model can show more aborted transitions than another, or what an aborted transition even is. Maybe I am misunderstanding the intention of this paragraph. line 180: change "hte" to "the"
 line 362: "using as less data as" to "using as little data as"
 line 543: "this phenomena" to "this phenomenon"
Citation: https://doi.org/10.5194/egusphere20221362RC1  AC1: 'Reply on RC1', Valérian JacquesDumas, 20 Mar 2023

RC2: 'Comment on egusphere20221362', Anonymous Referee #2, 05 Feb 2023
The primary objective of the paper is to provide a thorough empirical study of the popular datadriven methods, i.e. methods that require either simulated or observed transition trajectories, for solving the committor functions. The authors focus on four approaches: analogue methods, neural networks, reservoir computing and dynamical Galerkin approximation and utilizes the AMOC and the doublegyre model to demonstrate their performances. The paper provides a concise review of the four approaches mentioned, followed by detailed numerical evidence and discussions about their accuracies and scalings. On the whole, I found the exposition and goal of the paper to be clear. The numerical results, though I am uncertain of their generality, seem to be rather reasonable benchmarks and good starting points for applications to rare event algorithms. Therefore I'm inclined to accept the publication after minor revisions.
Minor comments:
 Figure 3: as described in the caption, there should be “confidence intervals” for FFNN training time in subplot (c). Or is it just too narrow to see?
 It would be helpful to provide more details about the scaling issue in the AMG method. Overall AMG method seems to be rather competitive especially when the number of available data is limited (which is typically the case in real application). Does the computational bottleneck come from the increasingly expensive KDtree search or the eigenvalue algorithms for the large analogue matrix? For the latter case, one can exploit the sparsity structure in the analogue matrix and use iterative eigensolvers. Matrices of size 1e4 by 1e4 seem still efficient to work with.
 The authors focus on the scaling issue from one perspective– the data dimension. However when applying the algorithm to other models, the variable dimension typically plays a critical role and efficient algorithms with moderate complexity growth with the model dimension is indeed the research target in the committor function community. I don’t think it is necessary to provide numerical evidence in this angle but I would suggest the author include a brief discussion and clarify this different definition about “highdimensional” problems.
 The discussion about computational time is rather long for all models. I would suggest adding theoretical scalings of, say O(N), O(N^2), etc. to figure 3 and 5 to make a clear visualization and comparison.
Typos:
line 19: remove “the”
line 45: “algorithms” to “algorithm”
line 141: “trees” to “tree”
line 170: “computed” to “compute”
line 180: “hte” to “the”
line 224: what does (7) mean?
line 362: “less” to “little”
line 521: “method” to “methods”
line 601: “appproximation” to “approximation”
Citation: https://doi.org/10.5194/egusphere20221362RC2  AC2: 'Reply on RC2', Valérian JacquesDumas, 20 Mar 2023
Status: closed

RC1: 'Comment on egusphere20221362', Anonymous Referee #1, 03 Feb 2023
In the paper "Datadriven methods to estimate the committor function in conceptual ocean models", the authors compare a broad spectrum of datadriven methods for estimating the committor, i.e. the probability to transition conditioned on a starting point. The methods are compared for two climaterelevant models, namely an ocean box model for the AMOC and a model for winddriven ocean circulation, both of which feature bistability, with correspondingly rare noiseinduced transitions between the system's fixed points. The main result of the paper consists of a detailed introduction of all four considered methods (feedforward neural networks, reservoir computing, analogue Markov chain, dynamical Galerkin approximation) and a comparison of their skill in predicting the committor as a function of the amount of provided training data (in the form of seen transitions).
The paper treats a very important problem, in that datadriven methods are clearly the way forward towards more complex models to predict (noiseinduced) tipping point behavior in climate models, and push the boundary of methodology for implementing rare event algorithms in such complex models. The manuscript is very readable, cires the relevant literature, and explains a good amount of detail. Concretely, the detailed comparison of the large number of datadriven approaches makes the presented result applicable to a wide range of problems within climate science and beyond.
As a consequence, I am in favor of publication after minor revisions and clarifications as indicated below.
Minor points:
 p10: Equation (21) is unclear to me: u and v are timeseries. The inner product <,> is, by context, to be interpreted over the statespace and not time? If so, the index i on the rhs is the temporal index. Why is there a sum over the time index? Is this a timeintegral? If so, the rhs is a vector and the lhs a scalar? Fortunately, it seems that equation (21) is not used anywhere later on, but arguably the notation in the section could be clarified.
 p22: The explanation around lines 555 to 565 about 'aborted transitions' is not very satisfactory. In particular, the amount of 'aborted trajectories' is quantified by the committor itself by definition. There cannot be a large fraction of trajectories that reach q=0.5 and then abort. Instead, the fraction of trajectories reaching q=0.5 but not transitioning is exactly 0.5, and the same is true for any other value of the committor (for example, 10% of all trajectories reaching q=0.9 eventually abort). It is therefore unclear how one model can show more aborted transitions than another, or what an aborted transition even is. Maybe I am misunderstanding the intention of this paragraph. line 180: change "hte" to "the"
 line 362: "using as less data as" to "using as little data as"
 line 543: "this phenomena" to "this phenomenon"
Citation: https://doi.org/10.5194/egusphere20221362RC1  AC1: 'Reply on RC1', Valérian JacquesDumas, 20 Mar 2023

RC2: 'Comment on egusphere20221362', Anonymous Referee #2, 05 Feb 2023
The primary objective of the paper is to provide a thorough empirical study of the popular datadriven methods, i.e. methods that require either simulated or observed transition trajectories, for solving the committor functions. The authors focus on four approaches: analogue methods, neural networks, reservoir computing and dynamical Galerkin approximation and utilizes the AMOC and the doublegyre model to demonstrate their performances. The paper provides a concise review of the four approaches mentioned, followed by detailed numerical evidence and discussions about their accuracies and scalings. On the whole, I found the exposition and goal of the paper to be clear. The numerical results, though I am uncertain of their generality, seem to be rather reasonable benchmarks and good starting points for applications to rare event algorithms. Therefore I'm inclined to accept the publication after minor revisions.
Minor comments:
 Figure 3: as described in the caption, there should be “confidence intervals” for FFNN training time in subplot (c). Or is it just too narrow to see?
 It would be helpful to provide more details about the scaling issue in the AMG method. Overall AMG method seems to be rather competitive especially when the number of available data is limited (which is typically the case in real application). Does the computational bottleneck come from the increasingly expensive KDtree search or the eigenvalue algorithms for the large analogue matrix? For the latter case, one can exploit the sparsity structure in the analogue matrix and use iterative eigensolvers. Matrices of size 1e4 by 1e4 seem still efficient to work with.
 The authors focus on the scaling issue from one perspective– the data dimension. However when applying the algorithm to other models, the variable dimension typically plays a critical role and efficient algorithms with moderate complexity growth with the model dimension is indeed the research target in the committor function community. I don’t think it is necessary to provide numerical evidence in this angle but I would suggest the author include a brief discussion and clarify this different definition about “highdimensional” problems.
 The discussion about computational time is rather long for all models. I would suggest adding theoretical scalings of, say O(N), O(N^2), etc. to figure 3 and 5 to make a clear visualization and comparison.
Typos:
line 19: remove “the”
line 45: “algorithms” to “algorithm”
line 141: “trees” to “tree”
line 170: “computed” to “compute”
line 180: “hte” to “the”
line 224: what does (7) mean?
line 362: “less” to “little”
line 521: “method” to “methods”
line 601: “appproximation” to “approximation”
Citation: https://doi.org/10.5194/egusphere20221362RC2  AC2: 'Reply on RC2', Valérian JacquesDumas, 20 Mar 2023
Valérian JacquesDumas et al.
Valérian JacquesDumas et al.
Viewed
HTML  XML  Total  BibTeX  EndNote  

245  132  15  392  8  3 
 HTML: 245
 PDF: 132
 XML: 15
 Total: 392
 BibTeX: 8
 EndNote: 3
Viewed (geographical distribution)
Country  #  Views  % 

Total:  0 
HTML:  0 
PDF:  0 
XML:  0 
 1