Optimization of the Fast Layer Transmittance Algorithm in RTTOV v13.1 for Strong Water Vapor Absorption Channels of the FY-3F HIRAS-II Instrument Using LBLRTM v12.11

Zhang, Panxiang; Zhang, Peng; Ma, Gang; Li, Rui; Lee, Lu; Bai, Wenguang; Qi, Chengli

doi:10.5194/egusphere-2025-5121

Preprints

https://doi.org/10.5194/egusphere-2025-5121

Preprints

07 Apr 2026

| 07 Apr 2026

Optimization of the Fast Layer Transmittance Algorithm in RTTOV v13.1 for Strong Water Vapor Absorption Channels of the FY-3F HIRAS-II Instrument Using LBLRTM v12.11

Panxiang Zhang, Peng Zhang, Gang Ma, Rui Li, Lu Lee, Wenguang Bai, and Chengli Qi

Abstract. Fast and accurate calculation of atmospheric transmittance is essential for infrared atmospheric remote sensing and satellite data assimilation. However, fast radiative transfer models show significant errors in strong water vapour absorption channels (e.g., near 6.7 μm). An important reason is that the numerical instability encountered during the regression of transmittance coefficients when dealing with lower and middle atmosphere. To address this, this study proposes an optimized scheme to calculate atmospheric transmittance vertical profiles for the RTTOV (Radiative Transfer for TOVS) fast transmittance algorithm. The method introduces a physically motivated transmittance threshold to sub-select training samples and employs cumulative transmittance-based weighting factor within a weighted least squares regression to recalibrate the transmittance coefficients. It aims to optimize the calculation scheme for transmittance coefficients of the Hyperspectral Infrared Atmospheric Sounder-II (HIRAS-II) instrument onboard Fengyun 3F satellite (FY-3F). The method is assessed by calculations on the training profile datasets provided within the RTTOV model framework. By comparing transmittance and brightness temperature calculations at 6.7 microns from this method with those from a line-by-line model and observations from HIRAS-II, the results show that the accuracy of the forward model for the 6.7 μm absorption channel is significantly enhanced by applying a threshold-based noise reduction method. This improvement enhances the stability and reliability of the transmittance calculations for this strong absorption band. Further accuracy enhancements are obtained by incorporating weighting corrections into the calculations of transmittance coefficients. The root mean square error (RMSE) and bias of the observation minus background (OMB) time series for FY-3F HIRAS-II demonstrate that the transmittance coefficient calculation scheme with weighting factor correction improves the forward model accuracy, which is more consistent with RTTOV simulation results. The OMB bias at the 6.7 μm absorption peak channel performs better than that of RTTOV, while the OMB bias on both sides of the 6.7 μm absorption peak channel remains consistent with RTTOV.

Received: 17 Oct 2025 – Discussion started: 07 Apr 2026

Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this paper. While Copernicus Publications makes every effort to include appropriate place names, the final responsibility lies with the authors. Views expressed in the text are those of the authors and do not necessarily reflect the views of the publisher.

Download & links

Panxiang Zhang, Peng Zhang, Gang Ma, Rui Li, Lu Lee, Wenguang Bai, and Chengli Qi

Status: final response (author comments only)

RC1:
'Comment on egusphere-2025-5121', Anonymous Referee #1, 11 Apr 2026

1. The authors state that LSM3 slightly outperforms LSM2. However, from a theoretical perspective, weighted ordinary least squares (LSM2) is mathematically equivalent to ordinary least squares with weighted variables (LSM3). Could you please explain why a performance difference is observed in practice?
2. Page 5, Line 156 & Page 6, Line 168: Maybe change ‘.when ...’ to ‘.When...’.
3. Page 6, Eq.(6): The weight factor is defined as a function of transmittance (i.e., W ∝ τ). Is this derived from an iterative optimization procedure, or other physical assumptions? Could you please add a description to clarify the principle behind this specific choice?
4. Page7, Lines 202-214 & Lines 215-220: This part needs to be carefully revised. It appears to be a repetition of content, where two consecutive paragraphs convey essentially the same meaning with slightly different wording.
5. Page 13, Line 331-332: The authors used a semi-empirical threshold of 10^-4as the screening criterion. However, it is not entirely clear whether this threshold is specifically selected for the FY-3F HIRAS-II instrument based on dedicated analysis, or adopted from previous studies as a generally applicable value. Could you please provide more quantitative evidence to support this choice if possible?
6. Figure quality: Most of the comparison figures (e.g., Figs. 5, 6, etc.) appear noticeably blurred and contain more than 15*3 lines in a single image, making it difficult for readers to distinguish between different schemes and symbols. To improve readability, I suggest select a limited number of representative profiles (e.g., Profiles 81, 82, 83) to illustrate the key findings, and move others to the appendix or supplementary material.
7. Figure caption: Different methods are described using words in Fig. 5 (e.g., ‘pentagrams’, ‘circles’, ‘triangles’), while symbols in Fig. 9 (e.g., ‘✯’, ‘○’, ‘△’). For reasons of consistency, please unify the notation style (i.e., either textual descriptions or symbols).

Citation: https://doi.org/10.5194/egusphere-2025-5121-RC1
- AC1: 'Reply on RC1', zhang panxiang, 03 Jun 2026
  
  Thank you for your detailed and constructive comments. Below we provide a point-by-point summary of our responses.
  1 (Theoretical equivalence vs. practical difference between LSM2 and LSM3):
  
  We fully agree that the weighted least squares method (LSM2) and the ordinary least squares method applied to weighted variables (LSM3) are mathematically equivalent under fixed positive weights and ideal exact arithmetic. Nevertheless, the minor performance difference observed in practical applications originates purely from numerical and procedural factors under finite-precision computations, rather than theoretical discrepancies. Three key factors lead to such numerical deviations. First, this study adopts an iterative reweighting strategy dependent on atmospheric transmittance. The substantial variation of transmittance across multiple orders of magnitude results in a severely ill-conditioned matrix in the normal equation of LSM2, while LSM3 effectively reduces the matrix condition number and improves numerical stability. Second, LSM2 is solved via normal equations, which tends to amplify rounding errors for ill-conditioned matrices. In contrast, LSM3 first implements data transformation and then adopts QR/SVD decomposition solvers with stronger robustness. Third, under extreme conditions with extremely low transmittance, rounding errors in the calculated weights are squared and magnified in LSM2, whereas they are only linearly scaled in LSM3, ultimately causing slight deviations in the iterative convergence results. In summary, the marginal superiority of LSM3 is a pure numerical effect rather than a statistical difference. We have elaborated on this mechanism in detail in the revised manuscript.
  2 (Capitalization of “When”):
  
  Corrected. On page 5, line 156 and page 6, line 168, “.when” has been changed to “. When”.
  3 (Derivation of weight factor W ∝ τ):
  
  We have added a physical explanation: transmittance decreases from TOA to surface; lower transmittance implies smaller absorption contribution and lower signal reliability, thus lower weight. After comparing candidate functions, we adopted W_j = |τ_j|. This explanation has been added to the revised manuscript (page 7, lines 184–196).
  4 (Repetitive paragraphs on page 7):
  
  We have consolidated the two repetitive paragraphs into a single concise paragraph (page 7, lines 215–221 in the revised version). Redundant content has been removed.
  5 (Threshold 10⁻⁴ for FY-3F HIRAS-II):
  
  We have added quantitative evidence: (1) convolution artifacts from HIRAS-II SRF cause negative transmittance below 10⁻⁴; (2) comparison with LBLRTM shows 10⁻⁴ suppresses errors better than 10⁻⁵; (3) a larger threshold would remove too many atmospheric samples. This threshold is instrument-specific, not a general empirical value. (Revised manuscript page 14, lines 345–352)
  6 (Figure blurriness and too many profiles):
  
  Figure blurriness was due to resolution loss during file saving. We have upgraded the resolution. Following your suggestion, only three representative profiles (81, 82, 83) are retained in the main figures; the remaining profiles are moved to Appendix A (page 24).
  7 (Inconsistent figure caption notation):
  
  We have unified all figure captions to use textual descriptions (e.g., “pentagrams”, “circles”) instead of graphic symbols (e.g., “✯”, “○”) throughout the manuscript.
  All corresponding changes have been clearly marked in the revised manuscript. We appreciate your valuable comments and guidance.
  
  Citation: https://doi.org/10.5194/egusphere-2025-5121-AC1
RC2:
'Comment on egusphere-2025-5121', Anonymous Referee #2, 09 Jun 2026

This manuscript presents an optimization of the fast layer transmittance coefficient calculation for strong water vapor absorption channels of FY-3F HIRAS-II, using LBLRTM v12.11 as a reference and evaluating threshold-based sample selection and weighted least-squares regression schemes. The topic is important for hyperspectral infrared radiance simulation, water-vapor retrieval, and satellite data assimilation, particularly for the 6.7 micrometre band where fast transmittance models can be affected by small-transmittance numerical instability.
The study is technically useful and contains a substantial amount of work, including coefficient training, comparison with line-by-line calculations, and evaluation using HIRAS-II observations. The proposed threshold and weighting approaches appear to improve the forward simulation in the strongest water-vapor channels. I think the manuscript can be strengthened mainly through clearer methodological description, modest sensitivity tests, and a more careful discussion of the observational validation uncertainty.
1. The manuscript would benefit from a clearer explanation of the physical and numerical basis for the 10^-4 transmittance threshold. The current description is plausible, but a compact sensitivity test using a few nearby threshold values would make the choice more convincing. It would be useful to show whether the retained sample numbers, transmittance errors, and brightness-temperature errors remain stable when the threshold is modestly varied.
2. The weighted least-squares schemes LSM2 and LSM3 are promising, but the mathematical description could be made easier to reproduce. Please clarify exactly how the weight factor is defined and whether it is applied to the predictors, the dependent variable, or both before solving the regression. A short algorithmic summary of LSM0-LSM3 would help readers follow the differences among the schemes.
3. Please clarify the RTTOV baseline used for comparison. The title refers to RTTOV v13.1, whereas the validation section mentions RTTOV v13.2. It would be helpful to state the coefficient set, predictors, spectroscopy assumptions, and surface-emissivity treatment used in the baseline comparison so that readers can interpret the magnitude of the reported improvement.
4. The observational comparison with FY-3F HIRAS-II is valuable. Because OMB statistics can also be influenced by clear-sky screening, ERA5 profile errors, surface emissivity, calibration uncertainty, and residual cloud contamination, I suggest adding a short uncertainty discussion. This would make the interpretation of the approximately 0.1 K improvement near the absorption peak more balanced.
5. The clear-sky selection method based on collocated MERSI cloud mask products is a sensible choice. Please provide a little more information on the number of retained HIRAS-II observations, land-ocean distribution, and whether the main conclusions are sensitive to the strict requirement that all MERSI pixels within the HIRAS-II FOV be clear.
6. AI-based approaches are increasingly being explored in fast radiative-transfer and gas-absorption calculations (e.g., https://doi.org/10.1016/j.jqsrt.2026.109920). It would be interesting if the authors could briefly discuss whether the threshold and weighting strategy proposed here might be combined with AI-based approaches in future work to further improve computational efficiency, robustness, or adaptability for strong-absorption-channel simulations.
7. Please correct grammatical and typographical issues throughout. Examples include 'poorly profiles', repeated sentences around the weighting function, and missing spaces before some section headings.
8. Please ensure consistent terminology for HIRAS, HIRAS-II, FY-3E, FY-3F, RTTOV v13.1, and RTTOV v13.2.
9. The rows in Table 1 are not visually well separated, which makes it somewhat difficult to distinguish the different experimental schemes at a glance. The authors may consider improving the table formatting, for example by adding clearer horizontal rules, alternating row shading, or slightly more spacing between rows.

Citation: https://doi.org/10.5194/egusphere-2025-5121-RC2
- AC2: 'Reply on RC2', zhang panxiang, 07 Jul 2026
  
  1、 The manuscript would benefit from a clearer explanation of the physical and numerical basis for the 10^-4 transmittance threshold.
  We compared thresholds of 10⁻⁵, 5×10⁻⁵, and 10⁻⁴, and found that 10⁻⁵ leads to significant deviations, while 10⁻⁴ ensures regression stability. Using 10⁻³ would overly raise the cutoff altitude, causing insufficient samples for lower water‑vapor channels and inducing simulation errors. Thus, 10⁻⁴ is the optimal balance between numerical reliability and sample validity, and it is specifically optimized for FY‑3F HIRAS‑II. Relevant discussions have been added to the manuscript.
  2、 Please clarify exactly how the weight factor is defined and whether it is applied to the predictors, the dependent variable, or both before solving the regression. A short algorithmic summary of LSM0-LSM3 would help readers follow the differences among the schemes.
  In response to the reviewer's comments, we have thoroughly revised and strengthened the mathematical descriptions of the weighted least-squares schemes LSM2 and LSM3 in Section 2.3 of the revised manuscript. Specifically, we have clearly delineated the differentiated weighting principles of LSM2 and LSM3, rigorously defined the exact formulas for their weight factors, and explicitly stated that the weighting strategy in our study is applied to both the independent and dependent variables, thereby ensuring the physical consistency of the regression fitting.
  Furthermore, to help readers better understand the differences among the four schemes, we have added a detailed algorithmic summary of LSM0–LSM3 in lines 241–248 of the revised manuscript. Concurrently, we have comprehensively revised Table 1. The combination of this unified text description and the detailed table presentation enables readers to intuitively grasp the specific implementation steps and distinctive features of each algorithmic scheme.
  3、Please clarify the RTTOV baseline used for comparison. The title refers to RTTOV v13.1, whereas the validation section mentions RTTOV v13.2. It would be helpful to state the coefficient set, predictors, surface-emissivity treatment used in the baseline comparison so that readers can interpret the magnitude of the reported improvement.
  We apologize for the version inconsistency caused by manuscript revision negligence. We have now unified the RTTOV baseline version throughout the manuscript as v13.1. Meanwhile, we have supplemented detailed baseline configuration parameters in the revised manuscript at lines 471–477, including the official standard coefficient set matching HIRAS‑II, the unified predictor variable version, and the fixed surface emissivity processing scheme adopted in the experiment. All baseline parameters have been explicitly stated to ensure the comparability and interpretability of the experimental results. We thank the reviewer again for the correction.
  4、I suggest adding a short uncertainty discussion. This would make the interpretation of the approximately 0.1 K improvement near the absorption peak more balanced.
  We sincerely thank the reviewer for this insightful suggestion. In response, we have added a systematic uncertainty discussion in the revised manuscript to provide a more balanced interpretation of the ~0.1 K improvement achieved by LSM3. We quantitatively assessed major uncertainty sources: (1) clear‑sky screening limitations, (2) ERA5 reanalysis profile errors, (3) surface emissivity uncertainties (with notably higher standard deviation over land than over ocean, as shown in Table 2), (4) thin cloud contamination (which tends to introduce a cold bias). The corresponding revisions have been incorporated on page 23, lines 526–556. We appreciate your help in enhancing the clarity and rigor of our validation.
  5、Please provide a little more information on the number of retained HIRAS-II observations, land-ocean distribution, and whether the main conclusions are sensitive to the strict requirement that all MERSI pixels within the HIRAS-II FOV be clear.
  We thank the reviewer for the suggestion. In the revised manuscript, we have supplemented detailed statistics on the retained clear‑sky samples in Table 2: a total of approximately 120,000 valid HIRAS‑II clear‑sky observations were retained, with RTTOV and LSM3 using exactly the same sample set. The ocean and land samples are nearly balanced in August 2024, while in February 2015 the land samples (82,102) significantly outnumber the ocean samples (37,766). Furthermore, a sensitivity test that relaxed the screening criterion to allow up to 5% cloudy pixels showed that both schemes exhibit slightly larger simulation biases, but the relative improvement of LSM3 over RTTOV remains unchanged, indicating that our core conclusions are robust to the strictness of the clear‑sky selection. Under clear‑sky, oceanic, and low‑to‑mid latitude conditions, the accuracy of the simulation bias would be further improved. The corresponding discussion has been included in lines 557–567. Thanks again for your valuable comments.
  6、It would be interesting if the authors could briefly discuss whether the threshold and weighting strategy proposed here might be combined with AI-based approaches in future work to further improve computational efficiency, robustness, or adaptability for strong-absorption-channel simulations.
  We have added a discussion on future interdisciplinary research in the conclusion section of the revised manuscript. We pointed out that the physical threshold screening and adaptive weighted regression strategies proposed in this study can be perfectly combined with AI algorithms at lines 626-634.
  7、Please correct grammatical and typographical issues throughout. Examples include 'poorly profiles', repeated sentences around the weighting function, and missing spaces before some section headings.
  We corrected all grammatical errors, typos and inappropriate expressions including the incorrect phrase "poorly profiles" at lines 153-154. We deleted duplicate sentences related to the weighting function derivation and description at lines 234-235, and standardized the formatting of all section headings to fix the missing space problem.
  8、Please ensure consistent terminology for HIRAS, HIRAS-II, FY-3E, FY-3F, RTTOV v13.1, and RTTOV v13.2.
  We have comprehensively sorted and unified all professional terminology and instrument/software version names throughout the manuscript. We completely eliminated the confusion of RTTOV, and fully unified the RTTOV version as v13.1 in all sections to ensure consistent and standardized terminology in the full text. Because FY‑3D uses HIRAS‑I whereas FY‑3E/F use HIRAS‑II, inconsistencies exist between generations. However, since our method is applicable to the entire FY‑3 HIRAS series, we have not differentiated between the two generations in some places.
  9、 The rows in Table 1 are not visually well separated, which makes it somewhat difficult to distinguish the different experimental schemes at a glance. The authors may consider improving the table formatting, for example by adding clearer horizontal rules, alternating row shading, or slightly more spacing between rows.
  We have fully optimized the typesetting format of Table 1. We added standard horizontal dividing lines and appropriately increased the row spacing. The optimized table has a clear hierarchical structure and intuitive visual effect, which facilitates readers to quickly distinguish and compare different LSM scheme parameters and characteristics.
  
  Citation: https://doi.org/10.5194/egusphere-2025-5121-AC2

Panxiang Zhang, Peng Zhang, Gang Ma, Rui Li, Lu Lee, Wenguang Bai, and Chengli Qi

Viewed

Total article views: 415 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	BibTeX	EndNote
272	115	28	415	16	24

HTML: 272
PDF: 115
XML: 28
Total: 415
BibTeX: 16
EndNote: 24

Views and downloads (calculated since 07 Apr 2026)

Month	HTML	PDF	XML	Total
Apr 2026	161	37	17	215
May 2026	69	63	3	135
Jun 2026	35	9	6	50
Jul 2026	7	6	2	15

Cumulative views and downloads (calculated since 07 Apr 2026)

Month	HTML	PDF	XML	Total
Apr 2026	161	37	17	215
May 2026	69	63	3	135
Jun 2026	35	9	6	50
Jul 2026	7	6	2	15

Viewed (geographical distribution)

Total article views: 403 (including HTML, PDF, and XML) Thereof 403 with geography defined and 0 with unknown origin.

Country	#	Views	%

Latest update: 18 Jul 2026

Short summary

To enhance atmospheric transmittance accuracy in strong water vapor absorption bands, this study proposes an optimized scheme for the fast transmittance algorithm applied to the Hyperspectral Infrared Atmospheric Sounder-II. It introduces a transmittance threshold for sample selection and a weighted least squares regression with transmittance weighting. Validation against line-by-line models and observations shows significant improvements in forward model accuracy and stability at 6.7 μm.


Total:	0
HTML:	0
PDF:	0
XML:	0