the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 07 Mar 2024
Inverse modelling for surface methane flux estimation with 4DVar: impact of a computationally efficient representation of a non-diagonal B-matrix in INVICAT v4
Abstract. Prior information is essential to most inverse problems and the surface flux estimation problem is no exception. The uncertainties of the prior fields, and their inter-correlations, should ideally be reflected in the a-priori error covariance matrix, often called B. The B-matrix, is however, difficult to quantify partly because it is typically a large matrix and partly because its numerical values are unknown.
We present a highly efficient method of representing the B-matrix to represent prior errors in the initial concentration and in the time sequence of surface fluxes for the 4DVar-based inverse modelling system (INVICAT) used to estimate the surface fluxes of methane. Our formulation is based on a spectral formulation of the square-root of B, which we believe has not been used in any such inverse modelling system before. It allows horizontal and vertical error correlations of the initial concentration, and horizontal and temporal error correlations of the flux to be represented. We provide full mathematical details. Our scheme allows the various correlation components to be switched on/off and for the respective length and timescales to be set in a way that is much more computationally efficient than representing such a B-matrix explicitly.
We test 14 configurations of the B-matrix (including the diagonal configuration) in a 100 day test assimilation of surface flask measurements of methane. We measure the performance of each by comparing the analysis to unassimilated observations held back for evaluation purposes. We find that the diagonal configuration is amongst the poorest performing choices of B. The best performing choice uses the spectral method. It does not include correlations for the initial concentration field, but does account for spatio-temporal correlations for the fluxes. These have the form of a SOAR (second order auto-regressive) function with a correlation length-scale of 600 km and a timescale of 3 months. Our results demonstrate the effectiveness of our method, which is applicable to very high resolution inverse modelling systems. We propose that potential biases in the prior initial condition field may be the reason for the poor performance when correlations in the prior initial concentration field are used.
- Preprint
(2070 KB) - Metadata XML
- BibTeX
- EndNote
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2024-655', Anonymous Referee #1, 21 Mar 2024
General comments
The authors discuss the calculations associated with the a priori error covariance matrix in a variational inversion system. They propose a spectral formulation of the matrix in order to optimize them. Without being new, the article is interesting, stimulating, educational and generally well written. That being said, the presentation is rather paradoxical in several respects and could be misleading. As it stands, some students might invest in the spectral formulation in vain and, therefore, I cannot support the publication of the article yet.
The first paradox of the presentation is the motivation: “The spectral method is very efficient. It is applicable to systems with very high resolutions, where existing methods that explicitly represent the B-matrix would not be feasible (Appendix D)” (l. 377-379 and similar sentences in the rest of the text). Actually, looking at Figure D1 about the cost of the two approaches, one can see that the authors’ illustration is on the lower end (L = 32, small resolution), while current results with an explicit representation reach the high end (L seems to be about 400 in doi:10.22541/au.171052488.85903583/v1). Where is the better efficiency argued in l. 66? The authors seem to ignore that the explicit representation is simplified by numerous zero correlations in the 2D flux errors:
- no space-time correlations (l. 160), allowing the use of a simple Kronecker product between time and space correlation matrices, see, e.g., https://doi.org/10.5194/acp-8-6341-2008, https://doi.org/10.5194/gmd-6-583-2013, or https://doi.org/10.1016/B978-0-12-814952-2.00008-3 – I am accumulating the citations here to show that this is a common method rather than a professional secret
- no correlations between land and ocean for surface flux errors, and, if needed, between continents or between certain ocean basins. Therefore, the effective dimension to manage can be much smaller than for the spectral method which requires the entire globe (hidden cost of the remark made in l. 407-408 that suggests duplicating the control vector when using the spectral method)
One could argue that these two items do not apply for the 3D initial state, which is much more challenging. Actually, the authors find that assigning spatial prior error correlations for the initial state degrades the inversion (see their embarrassed explanation in l. 417-420). Leaving aside this surprising result (to say the least), we see that this part of the control vector is similar to the numerical weather prediction (NWP) context where spectral methods emerged, but is this really the goal of atmospheric inversion? Atmospheric inversions suffer from edge effects and it is usual to cut off both ends: in contrast to NWP, obtaining the optimal initial state is not strategic and therefore the representation of its prior uncertainty can be simplified.
The second paradox of the paper is related. The objective of the method is to facilitate the resolution increase, but the detail of the increments is blurred by the horizontal reconfiguration operator Rh. Errera and Ménard (2012, cited twice in the paper) clearly demonstrated that “The practice of producing analysis increments on a horizontal Gaussian grid and then interpolating to an equally spaced grid is also shown to produce a degradation of the analysis.” In that case, who wants to use it?
Minor comments
- l. 52-53: this is not specific to 4D-Var but concerns all ill-posed inverse problems, explicitly or not
- l. 176: the horizontal reconfiguration operator Rh should be detailed (in particular, the need of local mass conservation and the way to handle mixed land-ocean pixels on one of the grids)
- l. 219: that way of doing renders the test useless. In the diagonal case, variances should be inflated to compensate for missing correlations and conserve a realistic total error budget
- l. 467: the word “unfortunate” is too subjective
- Section A3: the result of a classical adjoint test at the machine epsilon should be given to support the implementation
Citation: https://doi.org/10.5194/egusphere-2024-655-RC1 - AC1: 'Reply on RC1', Ross Bannister, 01 Apr 2024
-
RC2: 'Comment on egusphere-2024-655', Anonymous Referee #2, 28 Apr 2024
This paper presents a spectral method to efficiently account for prior error correlations in 4DVAR inversions with very large state vectors.
The paper poses an important problem regarding the specification of prior error covariance matrices for inversions with large state vectors, and I found the reduced spectral representation to be interesting. Whether it’s actually practical is not clear, and the demo seems like an odd choice since it is so poorly observationally constrained. The work seems original for atmospheric chemistry applications though, and the method appears to be sound in principle though I could not follow all the math. I am supportive of publication because it could be useful to some inversion practitioners.
Specific comments:
- Lines 38-40: the ‘direct inversion’ is usually called analytical inversion in the atmospheric chemistry literature and has been used extensively for methane by the Harvard group. It is not really fair to say that it is limited ‘to a relatively small number of large-area surface regions’ because the Jacobian matrix can be computed as an embarrassingly parallel problem. Maasakkers 2021 cited in the paper used the native 50-km resolution of the transport model, and a more recent application by Nesser et al. uses native 25-km resolution (https://doi.org/10.5194/egusphere-2023-946 ). A big advantage of the analytical inversion is that it provides closed-form posterior error covariance matrices enabling characterization of information content, and it allows immediate generation of inversion ensembles, cf. Jacob et al. https://doi.org/10.5194/acp-22-9617-2022
- Line 42: should also mention the LETKF approach used by Myazaki at JPL, for example https://doi.org/10.5194/acp-17-807-2017, which has similarity to the transform done here.
- Lines 43-46: There should be some mention that the size of the state vector is limited not only by computational resources but also by information content. A problem with 4DVAR and EnKF methods is that information content is not directly characterized.
- Line 60: another problem in empirical construction of B is ensuring that it is positive definite – this typically requires massaging the matrix after empirical construction of the off-diagonals.
- Line 131: why does the change in gridbox size with latitude matter?
- Line 152: will Bsp always be positive definite?
- Lines 169-172: Spectral methods and spherical harmonics don’t work great for atmospheric chemistry problems because the variability of chemical species does not follow wave structures. I’m not sure if this is relevant here, but the spectral approach is most useful when there is global structure in the error to be resolved and that is generally not the case for atmospheric chemistry problems.
- Line 218: ‘weak sink over Antarctica’. Weird. Where does this sink come from? Admittedly it doesn’t matter for your demo.
- Line 226: errors in atmospheric chemistry problems generally do not show ‘homogeneous and isotropic correlations’ .
- Figure 2: The correlation length scale looks more like ten degrees, so ~1000 km. Seems long, particularly for methane fluxes which come from a diversity of uncorrelated source sectors. What is the rationale for a spatial error correlation in methane fluxes?
- Figure 3: weird to have the vertical error correlation plot emphasize the stratosphere in a demo for methane fluxes.
- Line 284: The small information content to be obtained from the 60 NOAA stations is incommensurate to the size of the state vector. That would explain why the results are disappointing,
Citation: https://doi.org/10.5194/egusphere-2024-655-RC2 - AC2: 'Reply on RC2', Ross Bannister, 12 Jul 2024
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2024-655', Anonymous Referee #1, 21 Mar 2024
General comments
The authors discuss the calculations associated with the a priori error covariance matrix in a variational inversion system. They propose a spectral formulation of the matrix in order to optimize them. Without being new, the article is interesting, stimulating, educational and generally well written. That being said, the presentation is rather paradoxical in several respects and could be misleading. As it stands, some students might invest in the spectral formulation in vain and, therefore, I cannot support the publication of the article yet.
The first paradox of the presentation is the motivation: “The spectral method is very efficient. It is applicable to systems with very high resolutions, where existing methods that explicitly represent the B-matrix would not be feasible (Appendix D)” (l. 377-379 and similar sentences in the rest of the text). Actually, looking at Figure D1 about the cost of the two approaches, one can see that the authors’ illustration is on the lower end (L = 32, small resolution), while current results with an explicit representation reach the high end (L seems to be about 400 in doi:10.22541/au.171052488.85903583/v1). Where is the better efficiency argued in l. 66? The authors seem to ignore that the explicit representation is simplified by numerous zero correlations in the 2D flux errors:
- no space-time correlations (l. 160), allowing the use of a simple Kronecker product between time and space correlation matrices, see, e.g., https://doi.org/10.5194/acp-8-6341-2008, https://doi.org/10.5194/gmd-6-583-2013, or https://doi.org/10.1016/B978-0-12-814952-2.00008-3 – I am accumulating the citations here to show that this is a common method rather than a professional secret
- no correlations between land and ocean for surface flux errors, and, if needed, between continents or between certain ocean basins. Therefore, the effective dimension to manage can be much smaller than for the spectral method which requires the entire globe (hidden cost of the remark made in l. 407-408 that suggests duplicating the control vector when using the spectral method)
One could argue that these two items do not apply for the 3D initial state, which is much more challenging. Actually, the authors find that assigning spatial prior error correlations for the initial state degrades the inversion (see their embarrassed explanation in l. 417-420). Leaving aside this surprising result (to say the least), we see that this part of the control vector is similar to the numerical weather prediction (NWP) context where spectral methods emerged, but is this really the goal of atmospheric inversion? Atmospheric inversions suffer from edge effects and it is usual to cut off both ends: in contrast to NWP, obtaining the optimal initial state is not strategic and therefore the representation of its prior uncertainty can be simplified.
The second paradox of the paper is related. The objective of the method is to facilitate the resolution increase, but the detail of the increments is blurred by the horizontal reconfiguration operator Rh. Errera and Ménard (2012, cited twice in the paper) clearly demonstrated that “The practice of producing analysis increments on a horizontal Gaussian grid and then interpolating to an equally spaced grid is also shown to produce a degradation of the analysis.” In that case, who wants to use it?
Minor comments
- l. 52-53: this is not specific to 4D-Var but concerns all ill-posed inverse problems, explicitly or not
- l. 176: the horizontal reconfiguration operator Rh should be detailed (in particular, the need of local mass conservation and the way to handle mixed land-ocean pixels on one of the grids)
- l. 219: that way of doing renders the test useless. In the diagonal case, variances should be inflated to compensate for missing correlations and conserve a realistic total error budget
- l. 467: the word “unfortunate” is too subjective
- Section A3: the result of a classical adjoint test at the machine epsilon should be given to support the implementation
Citation: https://doi.org/10.5194/egusphere-2024-655-RC1 - AC1: 'Reply on RC1', Ross Bannister, 01 Apr 2024
-
RC2: 'Comment on egusphere-2024-655', Anonymous Referee #2, 28 Apr 2024
This paper presents a spectral method to efficiently account for prior error correlations in 4DVAR inversions with very large state vectors.
The paper poses an important problem regarding the specification of prior error covariance matrices for inversions with large state vectors, and I found the reduced spectral representation to be interesting. Whether it’s actually practical is not clear, and the demo seems like an odd choice since it is so poorly observationally constrained. The work seems original for atmospheric chemistry applications though, and the method appears to be sound in principle though I could not follow all the math. I am supportive of publication because it could be useful to some inversion practitioners.
Specific comments:
- Lines 38-40: the ‘direct inversion’ is usually called analytical inversion in the atmospheric chemistry literature and has been used extensively for methane by the Harvard group. It is not really fair to say that it is limited ‘to a relatively small number of large-area surface regions’ because the Jacobian matrix can be computed as an embarrassingly parallel problem. Maasakkers 2021 cited in the paper used the native 50-km resolution of the transport model, and a more recent application by Nesser et al. uses native 25-km resolution (https://doi.org/10.5194/egusphere-2023-946 ). A big advantage of the analytical inversion is that it provides closed-form posterior error covariance matrices enabling characterization of information content, and it allows immediate generation of inversion ensembles, cf. Jacob et al. https://doi.org/10.5194/acp-22-9617-2022
- Line 42: should also mention the LETKF approach used by Myazaki at JPL, for example https://doi.org/10.5194/acp-17-807-2017, which has similarity to the transform done here.
- Lines 43-46: There should be some mention that the size of the state vector is limited not only by computational resources but also by information content. A problem with 4DVAR and EnKF methods is that information content is not directly characterized.
- Line 60: another problem in empirical construction of B is ensuring that it is positive definite – this typically requires massaging the matrix after empirical construction of the off-diagonals.
- Line 131: why does the change in gridbox size with latitude matter?
- Line 152: will Bsp always be positive definite?
- Lines 169-172: Spectral methods and spherical harmonics don’t work great for atmospheric chemistry problems because the variability of chemical species does not follow wave structures. I’m not sure if this is relevant here, but the spectral approach is most useful when there is global structure in the error to be resolved and that is generally not the case for atmospheric chemistry problems.
- Line 218: ‘weak sink over Antarctica’. Weird. Where does this sink come from? Admittedly it doesn’t matter for your demo.
- Line 226: errors in atmospheric chemistry problems generally do not show ‘homogeneous and isotropic correlations’ .
- Figure 2: The correlation length scale looks more like ten degrees, so ~1000 km. Seems long, particularly for methane fluxes which come from a diversity of uncorrelated source sectors. What is the rationale for a spatial error correlation in methane fluxes?
- Figure 3: weird to have the vertical error correlation plot emphasize the stratosphere in a demo for methane fluxes.
- Line 284: The small information content to be obtained from the 60 NOAA stations is incommensurate to the size of the state vector. That would explain why the results are disappointing,
Citation: https://doi.org/10.5194/egusphere-2024-655-RC2 - AC2: 'Reply on RC2', Ross Bannister, 12 Jul 2024
Model code and software
Inverse modelling for surface methane flux estimation with 4DVar: impact of a computationally efficient representation of a non-diagonal B-matrix Ross Bannister and Chris Wilson https://doi.org/10.5281/zenodo.10777737
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 335 | 174 | 31 | 540 | 27 | 24 |
- HTML: 335
- PDF: 174
- XML: 31
- Total: 540
- BibTeX: 27
- EndNote: 24
Viewed (geographical distribution)
| Country | # | Views | % |
|---|---|---|---|
| United States of America | 1 | 204 | 38 |
| Germany | 2 | 61 | 11 |
| China | 3 | 48 | 9 |
| France | 4 | 42 | 7 |
| United Kingdom | 5 | 31 | 5 |
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
- 204