the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 10 Nov 2023
Improving the predictions of black carbon (BC) optical properties at various aging stages using a machine-learning-based approach
Abstract. It is necessary to accurately determine the optical properties of highly absorbing black carbon (BC) aerosols to estimate their climate impact. In the past, there has been hesitation about using realistic fractal morphologies when simulating BC optical properties due to the complexity involved in the simulations and the cost of the computations. In this work, we demonstrate that the predictions of optical properties like single scattering albedo (ω) and mass absorption cross-section (MAC) can be improved compared to the conventional Mie-based predictions using a highly accurate benchmark machine learning algorithm. Unlike the computationally intensive simulations of complex scattering models, the ML-based approach accurately predicts optical properties in a fraction of a second. There has been an extensive evaluation procedure carried out in this study. Based on comparisons with laboratory measurements, it was demonstrated that incorporating realistic morphologies of BC significantly improved their optical properties. The results indicate that it is possible to generate optical properties in the visible spectrum using BC fractal aggregates with any desired physicochemical properties, such as size, morphology, or organic coating. Based on these findings, climate models can improve their radiative forcing estimates using such comprehensive parameterizations for the optical properties of BC based on their aging stages.
- Preprint
(4634 KB) - Metadata XML
- BibTeX
- EndNote
Status: closed
-
RC1: 'Comment on egusphere-2023-2400', Anonymous Referee #1, 29 Dec 2023
In this manuscript, Romshoo et al. introduce two machine learning (ML) models designed to predict the optical properties of black carbon (BC) particles, utilizing various physical parameters as inputs. These models are trained on data from Multi Sphere T-Matrix (MSTM) simulations, accommodating a wide range of BC particle attributes across different stages of atmospheric aging. The research holds considerable importance in atmospheric studies, particularly in enhancing the accuracy of radiative forcing estimates for BC particles by adopting more complex and realistic morphologies over the conventional spherical assumption. By employing ML models, the authors effectively mitigate the computational intensity associated with MSTM simulations, using them primarily for the preparation of the training dataset.
The manuscript will be eligible for publication in Atmospheric Chemistry and Physics (ACP) once the authors have addressed the following comments:
Major Comments:
- Enhanced Contextualization with Previous Studies: The manuscript should provide a more comprehensive contextualization in relation to prior research. It is mentioned that Luo et al. (2018) and Lamb and Gentine (2021) have previously developed machine learning (ML) algorithms to predict the optical properties of bare black carbon (BC) particles. A more detailed introduction and a comparative analysis with these studies is recommended. Clarify, for example, the range of fractal dimensions and primary sphere sizes used in, as well as the wavelengths for which they trained their models. If possible, employ similar performance metrics as those used in the referenced studies for a direct comparison. Additionally, discuss how the results of your predictions for bare BC compare with previous findings.
- Detailed Description and Justification of Model Selection: Provide a more detailed account of the selected models among those tested, including the rationale for choosing the final two. While Tables B1 and B2 detail different hyperparameters tested, a clear indication of the best and final values or methods selected is necessary. Explain which metric was used to compare the different models and select the best one. Elucidating these choices will significantly enhance the reader's understanding of the research process and the robustness of the final models.
- Model Testing and Regularization Considerations: Discuss whether regularization techniques, such as dropout or data augmentation, were employed to improve the generalization of your models during testing for both interpolation and extrapolation. As the ultimate goal is to apply these algorithms to ambient data, please clarify if any steps were taken to incorporate noise into the training data to simulate the typical real-world measurement errors.
- Expansion of Section 5 - Instrumentation and Error Analysis: Section 5 should be expanded with a brief description of the measurement methods. Although these are described elsewhere, detailing the instruments used, including their associated errors, will introduce the reader to the uncertainties associated with the measurements. How were the input parameters (e.g., the number of primary spheres and fractal dimension) for the ML models and Mie calculations determined? How the measurement errors influence the predicted values? Discussing the potential impact on results if input parameters were varied in line with measurement or estimation errors would be beneficial.
Minor comments:
Abstract: Please provide a detailed and explicit list of the input parameters used in the developed machine learning algorithms.
L5: To substantiate the claim of 'highly accurate,' please provide a specific, quantifiable metric or set of metrics that define and measure the accuracy level being referred to.
L8-12: In my opinion, the focus of this paper is not demonstrating the superiority of MSTM simulations over Mie theory, as was already shown in Romshoo et al. (2022). Instead, it illustrates how machine learning methods can be employed to predict optical properties of black carbon particles based on MSTM simulations.
L10: Change “any desired physicochemical properties” to “any desired physicochemical properties within the range of the training dataset”.
L32: Consider adding an estimate to quantify “good accuracy”.
L39: Correct the reference of Lamb and Gentine to https://doi.org/10.1038/s41598-023-45235-8
L43: Please specify the following details regarding the previous machine learning models: the optical properties predicted, the input parameters utilized, the metric employed for model evaluation, the method used to test the model's extrapolation ability, whether the results were compared with actual measurements, and the size of the training dataset. (See major comment 1).
L68: Please explain the rationale for reporting 35 parameters, even if they are not independent.
L69: In Table A1, add a column detailing the step size for the independent parameters.
Section 2.1: It may be more appropriate to move this section to the introduction for better flow and context.
Sections 2.2.1, 2.2.2, and 2.2.3: I suggest to re-arrange these sections. For example: (i) the fractal dimension (and morphology in general) is related also to size, and not only to mixing state; (ii) the wavelength is not an “optical property” of the BC particles (please change also the label in Figure 1).
L74: Replace “formation” with atmospheric aging or processing.
L103: Please specify the step size used for varying the outer radius of the primary particle.
L112: Please confirm if you intend “mobility size spectrometers”.
L113: Update the citation from Sorensen (2001) to the correct Sorensen (2011) reference:
M. Sorensen (2011) The Mobility of Fractal Aggregates: A Review, Aerosol Science and Technology, 45:7, 765-779, DOI: 10.1080/02786826.2011.560909
L117: Add “in the atmosphere” to “increasing residence time”.
L124: Add the equation for the coating fraction in Appendix A1 for clarity.
L141: Discuss in more detail the limitations of the MSTM model, especially how the coated-BC are simulated and how this relates to atmospheric BC.
L147: Discuss the relevance of geometric cross-section to the optical model or consider moving it to the “size” or “Morphology” section for relevance.
L172: As already pointed out, the choice of the ML input parameters should be described more meticulously. Future users of your models should know which input parameters are needed and their validity range. For example, is the wavelength a free parameter or can it only be chosen among the 3 values used during the training? Furthermore, why do you use the fraction of coating and volume equivalent radii, although they are not independent. Does the input parameter “primary particle size” refer to both ao and ai?
L176: include the equation describing "g" in appendix A1 for reference.
L178: Clarify the use of lambda or consider removing it if it is causing confusion with the Box-Cox transformation equation.
L187: Specify in the text that “Fro” stands for Frobenius norm.
L191: Mention if the polynomial kernel was part of the experimentation and provide insights or remove (it is difficult to clearly identify which components you tried or not, and which ones resulted in the best model).
L193: Specify that it is the L2-norm in the main text for clarity.
L232: Please clarify the distribution of data among training, testing, and validation sets for the three experiments. Specifically, what percentage of the data is excluded from training and is instead used for validation in the interpolation and extrapolation experiments? It is unclear whether the extrapolation was performed on one or two sets: Table 1 suggests that the model was trained once using the combined ranges for Df, while the caption of Figure 3 indicates a Df training range of [1.5, 2.5). Conversely, Table B4 implies that the model underwent training twice, each time with a different Df range.
L236: Discuss if the data was divided into batches during training to improve generalization.
L238: Briefly mention the ranges of the parameters used or excluded from the training also in the main text (in addition to Table B3 and B4).
L243: Consider reporting (in the Appendix would be sufficient) the Mean Absolute Percentage Error (MAPE) in addition to the Mean Absolute Error (MAE), as MAPE provides a more sensitive measure compared to the relative percentage error derived from MAE.
Figure 3: Could you provide a similar boxplot for the Mean Absolute Percentage Error (MAPE)? Additionally, please include a comprehensive description of the boxplot's features. Also, consider reassessing the inclusion and visual representation of outliers in the plots.
Figure 3: How would it look like for the MAPE? Please include a detailed description of the features in the boxplot, such as the representation of the whiskers, median, quartiles, etc. Reassess the inclusion of outliers in plots, their number and distribution are important to evaluate the model performance (e.g., what is the probability that the prediction is an outlier?).
Figure 4: Add the blue line to the legend for clarity.
Figure 5: Why don’t you plot MAE or MAPE here?
Section 5: Consider changing the section’s title. This is a comparison to laboratory measurements and not “Atmospheric implications”.
Figure 6: Please include a legend for the black line and shaded area. Additionally, consider evaluating and reporting the errors associated with the measured Single Scattering Albedo and incorporate error bars (x-axis) in the plot. Can you add the exact results of the MSTM simulations obtained with the same parameters used for the ML model?
L310: Specify which ML algorithm is being used at this point in the text for clarity.
Section 5.1: The limited number of measurements used for comparison with the model should be acknowledged as a limitation. Additionally, future work should aim to increase the dataset size for more robust validation.
L332: What considerations have been made regarding the potential for extrapolating additional parameters? Are there specific constraints on allowable input values? For example, how does the model perform when using more than 1000 primary spheres or a primary particle size different from 15 nm? Demonstrating the model's ability to extrapolate (or not) with these parameters would be advantageous. Alternatively, please specify if the model is designed to accept only those parameters that fall within the training data range, as opposed to 'any reasonable inputs'.
Table A1: The wavelength shouldn’t be presented as a continuous parameter since test at intermediate values are not tested.
Figure C2: I couldn’t find any reference to this figure in the main text.
Figure C7: Does it contain the same number of measurements as presented in Figure 6?
Citation: https://doi.org/10.5194/egusphere-2023-2400-RC1 - AC1: 'Reply on RC1', baseerat romshoo, 08 Mar 2024
-
RC2: 'Review of egusphere-2023-2400', Anonymous Referee #2, 20 Jan 2024
Romshoo et al. (egusphere-2023-2400) simulated bare soot aggregates by diffusion-limited cluster aggregation (DLCA), and coated soot by adding spherical coatings around the spherical monomers of the DLCA aggregates. The authors then applied machine-learning models to interpolate numerically accurate MSTM calculations of the aggregates' optical properties.
The approach taken by the present authors was taken by two different studies previously. Luo et al. (2018) considered soot aggregates but not coatings. Lamb and Gentine (2021) considered uncoated and coated soot, using a coating model almost identical to the present authors. The authors erroneously wrote that Lamb and Gentine "do not consider coating", but the only difference in their approach is an insignificant change in the machine-learning model. For this reason, I have to recommend rejection of the present manuscript.
Independently, I also cannot recommend publication of this manuscript as the coating model is completely unrealistic. No experiment has ever observed coated soot to retain its original shape while adding spherical coatings to the monomers. Romshoo et al., and several others, have already published this model, but it contradicts dozens of smog chamber and field studies using electron microscopy, which all observed restructuring. There is no value in using machine-learning algorithms to interpolate the results of an inaccurate model.
Citation: https://doi.org/10.5194/egusphere-2023-2400-RC2 - AC2: 'Reply on RC2', baseerat romshoo, 08 Mar 2024
- AC3: 'Reply on RC2', baseerat romshoo, 08 Mar 2024
Status: closed
-
RC1: 'Comment on egusphere-2023-2400', Anonymous Referee #1, 29 Dec 2023
In this manuscript, Romshoo et al. introduce two machine learning (ML) models designed to predict the optical properties of black carbon (BC) particles, utilizing various physical parameters as inputs. These models are trained on data from Multi Sphere T-Matrix (MSTM) simulations, accommodating a wide range of BC particle attributes across different stages of atmospheric aging. The research holds considerable importance in atmospheric studies, particularly in enhancing the accuracy of radiative forcing estimates for BC particles by adopting more complex and realistic morphologies over the conventional spherical assumption. By employing ML models, the authors effectively mitigate the computational intensity associated with MSTM simulations, using them primarily for the preparation of the training dataset.
The manuscript will be eligible for publication in Atmospheric Chemistry and Physics (ACP) once the authors have addressed the following comments:
Major Comments:
- Enhanced Contextualization with Previous Studies: The manuscript should provide a more comprehensive contextualization in relation to prior research. It is mentioned that Luo et al. (2018) and Lamb and Gentine (2021) have previously developed machine learning (ML) algorithms to predict the optical properties of bare black carbon (BC) particles. A more detailed introduction and a comparative analysis with these studies is recommended. Clarify, for example, the range of fractal dimensions and primary sphere sizes used in, as well as the wavelengths for which they trained their models. If possible, employ similar performance metrics as those used in the referenced studies for a direct comparison. Additionally, discuss how the results of your predictions for bare BC compare with previous findings.
- Detailed Description and Justification of Model Selection: Provide a more detailed account of the selected models among those tested, including the rationale for choosing the final two. While Tables B1 and B2 detail different hyperparameters tested, a clear indication of the best and final values or methods selected is necessary. Explain which metric was used to compare the different models and select the best one. Elucidating these choices will significantly enhance the reader's understanding of the research process and the robustness of the final models.
- Model Testing and Regularization Considerations: Discuss whether regularization techniques, such as dropout or data augmentation, were employed to improve the generalization of your models during testing for both interpolation and extrapolation. As the ultimate goal is to apply these algorithms to ambient data, please clarify if any steps were taken to incorporate noise into the training data to simulate the typical real-world measurement errors.
- Expansion of Section 5 - Instrumentation and Error Analysis: Section 5 should be expanded with a brief description of the measurement methods. Although these are described elsewhere, detailing the instruments used, including their associated errors, will introduce the reader to the uncertainties associated with the measurements. How were the input parameters (e.g., the number of primary spheres and fractal dimension) for the ML models and Mie calculations determined? How the measurement errors influence the predicted values? Discussing the potential impact on results if input parameters were varied in line with measurement or estimation errors would be beneficial.
Minor comments:
Abstract: Please provide a detailed and explicit list of the input parameters used in the developed machine learning algorithms.
L5: To substantiate the claim of 'highly accurate,' please provide a specific, quantifiable metric or set of metrics that define and measure the accuracy level being referred to.
L8-12: In my opinion, the focus of this paper is not demonstrating the superiority of MSTM simulations over Mie theory, as was already shown in Romshoo et al. (2022). Instead, it illustrates how machine learning methods can be employed to predict optical properties of black carbon particles based on MSTM simulations.
L10: Change “any desired physicochemical properties” to “any desired physicochemical properties within the range of the training dataset”.
L32: Consider adding an estimate to quantify “good accuracy”.
L39: Correct the reference of Lamb and Gentine to https://doi.org/10.1038/s41598-023-45235-8
L43: Please specify the following details regarding the previous machine learning models: the optical properties predicted, the input parameters utilized, the metric employed for model evaluation, the method used to test the model's extrapolation ability, whether the results were compared with actual measurements, and the size of the training dataset. (See major comment 1).
L68: Please explain the rationale for reporting 35 parameters, even if they are not independent.
L69: In Table A1, add a column detailing the step size for the independent parameters.
Section 2.1: It may be more appropriate to move this section to the introduction for better flow and context.
Sections 2.2.1, 2.2.2, and 2.2.3: I suggest to re-arrange these sections. For example: (i) the fractal dimension (and morphology in general) is related also to size, and not only to mixing state; (ii) the wavelength is not an “optical property” of the BC particles (please change also the label in Figure 1).
L74: Replace “formation” with atmospheric aging or processing.
L103: Please specify the step size used for varying the outer radius of the primary particle.
L112: Please confirm if you intend “mobility size spectrometers”.
L113: Update the citation from Sorensen (2001) to the correct Sorensen (2011) reference:
M. Sorensen (2011) The Mobility of Fractal Aggregates: A Review, Aerosol Science and Technology, 45:7, 765-779, DOI: 10.1080/02786826.2011.560909
L117: Add “in the atmosphere” to “increasing residence time”.
L124: Add the equation for the coating fraction in Appendix A1 for clarity.
L141: Discuss in more detail the limitations of the MSTM model, especially how the coated-BC are simulated and how this relates to atmospheric BC.
L147: Discuss the relevance of geometric cross-section to the optical model or consider moving it to the “size” or “Morphology” section for relevance.
L172: As already pointed out, the choice of the ML input parameters should be described more meticulously. Future users of your models should know which input parameters are needed and their validity range. For example, is the wavelength a free parameter or can it only be chosen among the 3 values used during the training? Furthermore, why do you use the fraction of coating and volume equivalent radii, although they are not independent. Does the input parameter “primary particle size” refer to both ao and ai?
L176: include the equation describing "g" in appendix A1 for reference.
L178: Clarify the use of lambda or consider removing it if it is causing confusion with the Box-Cox transformation equation.
L187: Specify in the text that “Fro” stands for Frobenius norm.
L191: Mention if the polynomial kernel was part of the experimentation and provide insights or remove (it is difficult to clearly identify which components you tried or not, and which ones resulted in the best model).
L193: Specify that it is the L2-norm in the main text for clarity.
L232: Please clarify the distribution of data among training, testing, and validation sets for the three experiments. Specifically, what percentage of the data is excluded from training and is instead used for validation in the interpolation and extrapolation experiments? It is unclear whether the extrapolation was performed on one or two sets: Table 1 suggests that the model was trained once using the combined ranges for Df, while the caption of Figure 3 indicates a Df training range of [1.5, 2.5). Conversely, Table B4 implies that the model underwent training twice, each time with a different Df range.
L236: Discuss if the data was divided into batches during training to improve generalization.
L238: Briefly mention the ranges of the parameters used or excluded from the training also in the main text (in addition to Table B3 and B4).
L243: Consider reporting (in the Appendix would be sufficient) the Mean Absolute Percentage Error (MAPE) in addition to the Mean Absolute Error (MAE), as MAPE provides a more sensitive measure compared to the relative percentage error derived from MAE.
Figure 3: Could you provide a similar boxplot for the Mean Absolute Percentage Error (MAPE)? Additionally, please include a comprehensive description of the boxplot's features. Also, consider reassessing the inclusion and visual representation of outliers in the plots.
Figure 3: How would it look like for the MAPE? Please include a detailed description of the features in the boxplot, such as the representation of the whiskers, median, quartiles, etc. Reassess the inclusion of outliers in plots, their number and distribution are important to evaluate the model performance (e.g., what is the probability that the prediction is an outlier?).
Figure 4: Add the blue line to the legend for clarity.
Figure 5: Why don’t you plot MAE or MAPE here?
Section 5: Consider changing the section’s title. This is a comparison to laboratory measurements and not “Atmospheric implications”.
Figure 6: Please include a legend for the black line and shaded area. Additionally, consider evaluating and reporting the errors associated with the measured Single Scattering Albedo and incorporate error bars (x-axis) in the plot. Can you add the exact results of the MSTM simulations obtained with the same parameters used for the ML model?
L310: Specify which ML algorithm is being used at this point in the text for clarity.
Section 5.1: The limited number of measurements used for comparison with the model should be acknowledged as a limitation. Additionally, future work should aim to increase the dataset size for more robust validation.
L332: What considerations have been made regarding the potential for extrapolating additional parameters? Are there specific constraints on allowable input values? For example, how does the model perform when using more than 1000 primary spheres or a primary particle size different from 15 nm? Demonstrating the model's ability to extrapolate (or not) with these parameters would be advantageous. Alternatively, please specify if the model is designed to accept only those parameters that fall within the training data range, as opposed to 'any reasonable inputs'.
Table A1: The wavelength shouldn’t be presented as a continuous parameter since test at intermediate values are not tested.
Figure C2: I couldn’t find any reference to this figure in the main text.
Figure C7: Does it contain the same number of measurements as presented in Figure 6?
Citation: https://doi.org/10.5194/egusphere-2023-2400-RC1 - AC1: 'Reply on RC1', baseerat romshoo, 08 Mar 2024
-
RC2: 'Review of egusphere-2023-2400', Anonymous Referee #2, 20 Jan 2024
Romshoo et al. (egusphere-2023-2400) simulated bare soot aggregates by diffusion-limited cluster aggregation (DLCA), and coated soot by adding spherical coatings around the spherical monomers of the DLCA aggregates. The authors then applied machine-learning models to interpolate numerically accurate MSTM calculations of the aggregates' optical properties.
The approach taken by the present authors was taken by two different studies previously. Luo et al. (2018) considered soot aggregates but not coatings. Lamb and Gentine (2021) considered uncoated and coated soot, using a coating model almost identical to the present authors. The authors erroneously wrote that Lamb and Gentine "do not consider coating", but the only difference in their approach is an insignificant change in the machine-learning model. For this reason, I have to recommend rejection of the present manuscript.
Independently, I also cannot recommend publication of this manuscript as the coating model is completely unrealistic. No experiment has ever observed coated soot to retain its original shape while adding spherical coatings to the monomers. Romshoo et al., and several others, have already published this model, but it contradicts dozens of smog chamber and field studies using electron microscopy, which all observed restructuring. There is no value in using machine-learning algorithms to interpolate the results of an inaccurate model.
Citation: https://doi.org/10.5194/egusphere-2023-2400-RC2 - AC2: 'Reply on RC2', baseerat romshoo, 08 Mar 2024
- AC3: 'Reply on RC2', baseerat romshoo, 08 Mar 2024
Data sets
Database of physicochemical and optical properties of black carbon fractal aggregates B. Romshoo, T. Müller, J. Patil, T. Michels, M. Kloft, and M. Pöhlker https://doi.org/10.5281/zenodo.7523058
Model code and software
Machine-learning-based approach to predict optical properties of black carbon (BC) at various aging stages Baseerat Romshoo, Jaikrishna Patil, Tobias Michels, Thomas Müller, Marius Kloft, & Mira Pöhlker https://doi.org/10.5281/zenodo.8060207
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 311 | 105 | 30 | 446 | 23 | 17 |
- HTML: 311
- PDF: 105
- XML: 30
- Total: 446
- BibTeX: 23
- EndNote: 17
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1