the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 18 Jan 2024
IceDetectNet: A rotated object detection algorithm for classifying components of aggregated ice crystals with a multi-label classification scheme
Abstract. The shape of ice crystals affects their radiative properties, growth rate, fall speed, and collision efficiency and thus, plays a significant role in cloud optical properties and precipitation formation. Ambient conditions like temperature and humidity determine the basic habit of ice crystals, while microphysical processes such as riming and aggregation further shape them, resulting in a diverse set of ice crystal shapes and densities. Current classification algorithms face two major challenges: (1) ice crystals are often classified as a whole (on the image scale), necessitating identification of the dominant component of aggregated ice crystals, and (2) single-label classifications lead to information loss because of the compromise between basic habit and microphysical process information. To address these limitations, here we present a two-pronged solution: a rotated object detection algorithm (IceDetectNet) that classifies each component of an aggregated ice crystal individually, and a multi-label classification scheme that considers both basic habits and physical processes simultaneously. IceDetectNet was trained and tested on two independent datasets obtained by a holographic imager during the NASCENT campaign in Ny-Ålesund, Svalbard, in November 2019 and April 2020. The algorithm correctly classifies 92 % of the ice crystals as either aggregate or non-aggregate and achieved an overall accuracy of 86 % for basic habits and 82 % for microphysical processes classification. On the component scale, IceDetectNet demonstrated high detection and classification accuracy across all sizes, indicating its ability to effectively classify individual components of aggregated ice crystals. Furthermore, the algorithm demonstrated good generalization ability by classifying ice crystals from an independent test dataset with overall accuracies above 70 %. IceDetectNet can provide a deeper understanding of ice crystal shapes, leading to better estimates of ice crystal mass, fall velocity, and radiative properties and thus, has the potential to improve precipitation forecasts and climate projections.
- Preprint
(6173 KB) - Metadata XML
- BibTeX
- EndNote
Status: final response (author comments only)
-
RC1: 'Comment on egusphere-2023-2770', Anonymous Referee #1, 08 Mar 2024
IceDetectNet: A rotated object detection algorithm for classifying components of aggregated ice crystals with a multi-label classification scheme by Zhang et al.
The authors present the IceDetectNet, a new deep learning algorithm aimed at refining the classification of ice crystals, which is significant for understanding cloud properties and precipitation processes. The need for improved classification methods arises from the challenges associated with current deep learning approaches, such as the difficulty in distinguishing individual components within aggregated ice crystals and the compromise between classifying basic habits and microphysical processes. IceDetectNet attempts to address these issues by integrating a rotated object detection technique for component-specific analysis and a multi-label classification framework. The approach is innovative and classification of individual components of aggregated crystals and the related microphysical processes is an important upgrade to traditional machine learning approaches. Yet, the study contains severe limitations, which affects its generalization ability. The main shortcoming is the choice of ice crystal habit and microphysical process categories, which include specific habit categories while omitting other frequently observed ice crystal habits. Since the habit classification scheme is the backbone of this study, this should be well thought of and justified in order for the algorithm to be generazible. It is recommended that the authors refine the the habit classification scheme and address other major concerns related to the structure of IceDetectNet before the paper can be recommended for publication.
Major comments related to habit classification scheme
- The habit classes that the authors are proposing contains seven basic habits. Yet, the choice of these basic habit categories is not justified or based on classification schemes presented in the literature (e.g. Kikuchi et al., 2013; Hallett & Bailey, 2009). Some habit classes, like “CPC” or “lollipop” are rather specialized, whereas other key habit classes (especially for mixed-phase clouds), like needles, are missing. Since the manuscript is focused on identification of microphysical processes and crystal growth regimes, it would be justified to classify the crystals either as single crystals (typically encountered below -20°C; see Hallett & Bailey, 2009) and polycrystalline habits (typically encountered in colder temperatures) and/or as plate-like (colder than -10°C or warmer than -2°C), column-like (warmer than -10°C) or mixed-habits. More generalized habit classification scheme would enhance the usability of the algorithm for other campaigns.
Additional comments on the habit and microphysical classes: - Small: why discriminate between small and irregular habits? Why not use an objective criteria for small, like measured maximum dimension?
- Pristine: The definition of pristine, that the authors are presenting, is too broad. According to this definition any ice crystal that is not rimed, sublimated or aggregated is classified as pristine, including polycrystalline habits or other highly complex shapes. Korolev et al. (1999) defined pristine ice as “faceted single ice crystals”. The authors should adapt a similar narrower definition.
- Aged: It is more precise to categorize ice crystals based on their physical state as "rimed" or "sublimated" rather than using the term "aged," which does not directly correlate with a specific physical transformation.
- The manuscript prompts concerns regarding the precision of manual classification processes employed. Ensuring the accuracy of training and test datasets is crucial for the success of deep learning methodologies. A case in point is the classification of a crystal in Figure 3 as CPC, despite the absence of plate-like features, and its closer resemblance to a needle structure. Given that CPC crystals suggest a transitional phase between columnar and plate growth regimes, as discussed by Pasquier et al. (2023), and needle growth typically occurs at warmer temperatures around -5°C, such misclassification could potentially lead to incorrect assumptions about the environmental conditions experienced by the crystal. To address these concerns, the authors are advised to discuss the classification criteria for each ice crystal habit more comprehensively and provide additional examples from various habit categories. This approach would significantly enhance confidence in the accuracy of the manual labeling process.
- The manuscript introduces three distinct classification frameworks: multi-label classification, basic habit identification, and microphysical process categorization. In addition, it explores a rotated object detection algorithm for discerning aggregates and their respective components. However, the integration and mutual dependencies among these classification strategies are not clearly articulated, leaving the reader uncertain about how the results from these different schemes are interrelated. A particularly crucial element of the research is the aggregate detection capability, where the method for identifying aggregates through bounding boxes is noted to be quite effective, boasting an accuracy rate of 92% as detailed in Section 4.1.1. Conversely, Table 2 reveals a significantly lower accuracy rate of only 50.3% for the multi-label classification scheme in recognizing aggregate classes, indicating suboptimal performance in this aspect. To address these discrepancies, it is recommended that the authors consider implementing a sequential classification approach. This would entail initially employing the rotated object detection algorithm to differentiate between aggregate and non-aggregate forms, followed by the classification of the individual crystals' habits, and culminating with the identification of the microphysical processes involved.
Major comments related to deep learning
- The authors have decided to separate the training and test data sets to different temperature regimes. The reasoning behind this is that the test and training datasets will have different characteristics. Consequently, it is challenging to assess the algorithm's generalization capability across unrepresented habits in the test dataset, such as frozen droplet, CPC, lollipop, and their aged and aggregated forms. Typically, it is advisable for both the training and test datasets to encompass all classes to ensure a comprehensive evaluation of the algorithm's performance. The authors might want to reconsider their approach regarding the selection of training and test datasets or reevaluate the defined habit classes in light of these concerns. Additionally, the representation of some classes within the training dataset, such as column aggregate, is insufficiently robust for effective training. It may be beneficial to exclude classes with inadequate sample sizes to improve the reliability and validity of the classification outcomes.
- The performance of the training dataset is well discussed and shown in Figs. 4-7. The performance of the test dataset is only shortly discussed in Sec. 4.3. Figure 8c shows the overall performance of basic habits, microphysical processes, and all-classes but no detailed results for the different classes are shown similar the training dataset (Figs. 5 and 6).
- Table 2: since the classes are highly imbalanced, overall accuracy is not necessarily the best performance metric. Other performance metrics, such as balanced accuracy or F1-score could work better.
Minor comments
Line 4: can you change “density” to “effective density”
Lines 30-32: it is also possible that ice crystals change environment in convective systems or by precipitation, which can lead to formation of mixed habits.
Lines 35-36: Schmitt & Heymsfield (2014) defined a complexity parameter to discriminate between single crystals and aggregates. This work could be also cited.
Lines 49-52: There are multiple studies investigating the habits (pristine or single habits vs irregular habits) of mixed-phase or cirrus clouds besides that of Korolev et al. (1999). It is advised to give a broader overview of these observations.
Line 273: How did the different folds perform?
References
Hallett, John, and Matthew P. Bailey. “A Comprehensive Habit Diagram for Atmospheric Ice Crystals : Confirmation from the Laboratory , AIRS II , and Other Field Studies.” Journal of Atmospheric Sciences 66 (2009): 2888–99. https://doi.org/10.1175/2009JAS2883.1.
Kikuchi, Katsuhiro, Takao Kameda, Keiji Higuchi, and Akira Yamashita. “A Global Classification of Snow Crystals, Ice Crystals, and Solid Precipitation Based on Observations from Middle Latitudes to Polar Regions.” Atmospheric Research 132–133, no. January 1969 (2013): 460–72. https://doi.org/10.1016/j.atmosres.2013.06.006.
Schmitt, C. G., and A. J. Heymsfield (2014), Observational quantification of the separation of simple and complex atmospheric ice particles, Geophys. Res. Lett., 41, 1301–1307, doi:10.1002/2013GL058781.
Citation: https://doi.org/10.5194/egusphere-2023-2770-RC1 - The habit classes that the authors are proposing contains seven basic habits. Yet, the choice of these basic habit categories is not justified or based on classification schemes presented in the literature (e.g. Kikuchi et al., 2013; Hallett & Bailey, 2009). Some habit classes, like “CPC” or “lollipop” are rather specialized, whereas other key habit classes (especially for mixed-phase clouds), like needles, are missing. Since the manuscript is focused on identification of microphysical processes and crystal growth regimes, it would be justified to classify the crystals either as single crystals (typically encountered below -20°C; see Hallett & Bailey, 2009) and polycrystalline habits (typically encountered in colder temperatures) and/or as plate-like (colder than -10°C or warmer than -2°C), column-like (warmer than -10°C) or mixed-habits. More generalized habit classification scheme would enhance the usability of the algorithm for other campaigns.
-
RC2: 'Comment on egusphere-2023-2770', Anonymous Referee #2, 26 Mar 2024
The manuscript “IceDetectNet: A rotated object detection algorithm for classifying components of aggregated ice crystals with a multi-label classification scheme” by Zhang et al. presents a new deep learning algorithm that classifies each component of aggregated ice crystals based on their basic habit and physical processes. This algorithm enables a more detailed classification of the ice crystals than conventional algorithms and thus provides an innovative and improved tool for identifying atmospheric ice crystals. The algorithm and its evaluation are well described, individual steps are listed in detail, which makes the content of the manuscript very comprehensible and easy to follow. I recommend this paper for publication after major revision.
Major comments:
- L. 75: “Following this initial categorization, each ice crystal was classified into one of seven basic habits: ’column’, ’plate’, ’lollipop’ (Pasquier et al., 2022a), ’Columns on Capped-Columns’ (CPC, Pasquier et al. 2023), ’irregular’, ’frozen droplets’, and ’small’.“
Can the authors explain and justify why this ice particle classification is used? What about other ice crystals occurring in the atmosphere, e.g., needles, droxtals, rosettes, …? Why are the ice particles classified in specific (e.g. lollipop) and unspecific shapes (e.g. irregular)? This shape categorization results in a strong unbalanced training data set. I see, the authors consider this imbalance in their analyses. However, a better classification of particles will counteract the imbalance and might increase the classification performance of IceDetectNet and its applicability to other independent test datasets.
- While the evaluation of the model performance based on the training data set is extensive (Sect. 4.1-4.2), the application of the IceDetectNet algorithm to the test data set is far too short. A more detailed analysis of the generalization ability of IceDetectNet would be desirable. How accurate is the basic habit classification and physical process classification for the test data set? Why do the individual values of evaluation parameters change when the test data set is used instead of the training data set? What are the reasons? Where are (no) challenges, issues? What conclusions can the authors draw from the evaluation of IceDetectNet using the independent test dataset? How well could the algorithm be applied to other test data sets from different seasons, locations, … ? Do the authors expect any limitations here?
Minor comments:
- L. 38: “generalization abilities“: This term is too broad to be understandable. Although this term will be explained later it would be useful to use a more specific formulation here.
- L.34: Square area of the image, particle maximum dimension, and area ratio do not classify the shape of ice crystals, they define the size of ice crystals. Please correct.
- L. 85: Please provide more information about the training and test data set (location, meteorological conditions, ...). How representative are both data sets for the occurrence of generally possible ice particles in the atmosphere (in all seasons and locations)?
- L. 156: “every bounding box was visually classified in an ice category following the multi-label classification scheme”. In Fig. 1, some ice particles of different categories look similar. How does a visual, i.e., subjective, classification scheme influence the training of the data set? How is it decided that an ice particle that could apparently fit into two classes is assigned to one class?
- L. 162: “The initial image needs to be enlarged by 15 %”. What is the reference? Area? Length, width?
- L. 165: “All images are then uniformly resized to 512x512 pixels.” How this is done?
- L. 168: “We replicate the single dimension three times to emulate the three-dimensional structure of RGB images”. I understand why the authors are doing this. However, how do results change, if zero-arrays are used in the second and third dimension? Won’t a triple replication as the authors do lead to a loss of contrast in the resulting RGB image?
- Fig. 3: Why do duplicate bounding boxes have to be removed in step 6, when the duplicate bounding boxes should have already been removed in step 3?
- L. 208-213: Can the authors briefly explain why the learning rate and epochs are chosen in this way?
- Equation 2: What are correctly predicted positive and negative instances when ice crystal classes are predicted?
Citation: https://doi.org/10.5194/egusphere-2023-2770-RC2
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 253 | 67 | 11 | 331 | 7 | 7 |
- HTML: 253
- PDF: 67
- XML: 11
- Total: 331
- BibTeX: 7
- EndNote: 7
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1