Automatic classification and segmentation of Snow Micro Penetrometer profiles with machine learning algorithms

Kaltenborn, Julia; Macfarlane, Amy R.; Clay, Viviane; Schneebeli, Martin

doi:https://doi.org/10.5194/egusphere-2022-938

Preprints

https://doi.org/10.5194/egusphere-2022-938

Preprints

06 Dec 2022

| 06 Dec 2022

Automatic classification and segmentation of Snow Micro Penetrometer profiles with machine learning algorithms

Julia Kaltenborn, Amy R. Macfarlane, Viviane Clay, and Martin Schneebeli

Abstract. Snow-layer segmentation and classification is an essential diagnostic task for a wide variety of cryospheric applications. The SnowMicroPen (SMP) measures the snowpack's penetration force at submillimetre resolution against the snow depth. The resulting depth-force profile can be parameterized for density and specific surface area. However, no information on traditional snow types is currently extracted automatically. The labeling of snow types is a time-intensive task that requires practice and becomes infeasible for large datasets. Previous work showed that automated segmentation and classification is in theory possible, but can either not be applied to data straight from the field or needs additional time-costly information, such as from classified snow pits. To address this gap, we evaluate how well machine learning models can automatically segment and classify SMP profiles. We trained fourteen different models, among them semi-supervised models and artificial neural networks (ANNs), on the MOSAiC SMP dataset, a large collection of snow profiles on Arctic sea ice. We found that SMP profiles can be successfully segmented and classified into snow classes, based solely on the SMP's signal. The model comparison provided in this study enables practitioners to choose a model that is suitable for their task and dataset. The findings presented will facilitate and accelerate snow type identification through SMP profiles. Overall, snowdragon creates a link between traditional snow classification and high-resolution force-depth profiles. With such a tool, traditional snow profile observations can be compared to SMP profiles.

Received: 28 Sep 2022 – Discussion started: 06 Dec 2022

Julia Kaltenborn et al.

Status: final response (author comments only)

RC1:
'Referee Comment on egusphere-2022-938', Anonymous Referee #1, 24 Dec 2022

The manuscript presents the compilation of several machine learning algorithms for the classification of snow types of snow on Arctic sea ice from SMP penetration force profiles. The authors describe the functionality, pros and cons of each model set-up, and determine that ANNs outperform other types of supervised and semi-supervised learners at this task. An exploration of this manuscript was to determine if semi-supervised learners could accurately classify snow type from a subset of the 164 snow-type labeled SMP profiles. I found the computer science aspect of this work to be quite complete and presented at a high enough level for the comprehension of a general reader after some additional clarification on the language. As it is presented, the hypothesis is developed and tested, and a best model is determined. Discussion regarding the practitioner’s choice of model set-up within the snowdragon software package is provided for adapting this work to other SMP datasets and archived for reproducibility. I am recommending this manuscript for minor revision prior to acceptance as some aspects of the experimental design need further explanation or clarification. Addressing the provided comments may help increase the scientific quality of the manuscript, as much of the evaluation of the results remains qualitative. I do find that additional snow-related analyses and uncertainty analyses of these data and model predictions could be incorporated in this manuscript to increase the scientific significance. For this reason, I have scored Scientific Significance as “Good”. However, as the scope of the manuscript is defined, it is complete in the analysis of machine learning classification of snow type.

General and Specific comments to the authors are provided in the attached .pdf document.

Citation: https://doi.org/10.5194/egusphere-2022-938-RC1
- AC2: 'Reply on RC1', Julia Kaltenborn, 05 Mar 2023
  
  Thank you very much for your in-depth feedback and for providing us with such helpful comments. All of your comments are very much appreciated and have helped us to improve the manuscript. To summarize the most important responses: The profiles have been labeled with additional in-situ observations at hand (Micro-CT and NIR) – we added more information on the complete labeling process in the manuscript and provide a comprehensive overview in the additional complementary material. We will also include a more detailed discussion about the micro-mechanical properties of the different snow types and the relation between classification difficulty and micro-mechanical properties. In our responses, you can find an explanation about the “qualitative nature” of the validation process, why we find it important to include such an evaluation, and why this study cannot solve the subjectivity of snow grain classification in general. We hope that you find all your other suggestions addressed in the below responses. We found them all very helpful and will include them in our revised manuscript. Thank you for your time and for helping us improve the manuscript significantly.
  
  Citation: https://doi.org/10.5194/egusphere-2022-938-AC2
RC2:
'Comment on egusphere-2022-938', Pascal Hagenmuller, 26 Jan 2023
Review of Automatic classification and segmentation of Snow Micro Penetrometer profiles with machine learning algorithms by Julia Kaltenborn et al.
Summary :
The authors use machine learning approaches to learn the relation between the penetration profile of the snowpack and grain shape class. The final goal is to have a quick and easy way (measurement of penetration profile with SMP) to capture one key descriptor of snow stratigraphy (grain type) which is usually time-consuming to measure. The predictors comprise different quantities derived from SnowMicroPen (SMP) measurements, including mean force, standard deviation, position in the snowpack (height), and probably noise features obtained with a statistical model (shot noise model). The goal variable is the grain shape class estimated by an expert solely from the SMP profile signature. The data comprises 164 SMP profiles measured on Arctic sea ice during winter 2019-2020 and manually labeled by a single expert. This classification leads to a segmentation of the snowpack profile into distinct layers by assuming that contiguous points of the same grain shape class belong to the same layer. The authors show that the different algorithms are able to reproduce the choice of the expert (max ROC AUC of 0.94).

Main comments:

I am not fully getting the final objective of the paper. What is the scientific question we want to address by automatically reproducing the grain shape class inferred from the penetration profile based on undescribed expert analysis?

By definition, the grain shape class or snow types (Fierz et al., 2009) is related to the shape of the grains and is traditionally derived from the observation of single grains on a crystal card with a magnification lens. This measurement remains manual, is very time-consuming, inevitably contains some subjectivity, and the use of classes is limited to capture the continuous nature of snow types. Trying to overcome some of the two first limitations by automatic classification is of great interest. Different attempts exist to relate the SMP signal to scalar microstructural features of snow based on the physical interpretation of the penetration process (e.g., Löwe & van Herwijnen (2012), Lin et al. (2022)) or with direct statistical / machine learning approaches (e.g., Proksch et al. (2015)). In particular, King et al. (2020) and Satyawali et al. (2009) used the latter approach to relate MEASURED grain shape class to SMP profiles. Here the ground truth is not the measured grain shape on independent data but corresponds to the interpretation of solely the SMP signal signature.

This direct identification has never been documented so far. The description in the text is elusive, with a reference (l.76, Schneebeli et al. 1999) that does not describe the procedure. Besides, the data presented here relies on the interpretation of a single expert (l. 75-77). One cannot evaluate any reproducibility of the procedure or agreement with ground truth based on manual observation in snow pit data. Moreover, it is highly likely that the estimation is subjective. For instance, in Fig. 1, one may wonder why only the specific layer at a depth between 98 and 102 mm is labeled as « Depth hoar wind packed » and not other layers below that show similar features. In addition, there are obviously « inconsistencies in their ground truth labeling » (l. 324) and the results are linked to « different classification styles of experts » (l.332) and the evaluation is qualitative (« classification patterns […] were satisfying to domain experts » l.368). The discussion is not convincing based only on the feeling that «in the view of the authors, a temporally consistent classification is more relevant to the interpretation of the development of the snowpack, even if there is a certain, but unknown, bias to an expert interpretation » (l. 255-257). To me, it appears, in the end, that the presented algorithms are able to reproduce one analysis of one single expert on specific snowpack types. In my opinion, this limits a lot of the interest and generalization to the snow community.

The authors refer to Fierz et al. (2009) for describing the different snow types referenced in the paper (l. 74). However, it is not very clear how the different classes presented in the paper (see legend of Fig. 1) are defined as they are not present in the international classification described in Fierz et al. (2009).

Grain shape class has been used since the beginning of snow science and was first motivated by avalanche forecasting. It remains the most common descriptor in snowpit observations but has many known limitations. It is a discrete class whose evolution cannot be described by differential equations in models. It cannot be quickly and objectively described. Currently, the international classification is not necessarily adapted to any snow on Earth (e.g., here, the authors added classes that are not in the classification). Therefore, one may wonder why, in general, we want to stick to this description of snow.

The interest of the algorithm is described in grandiose terms: they make « training of interdisciplinary scientists in snow type categorization obsolete » (l. 31), «can be directly employed by practitioners for their own SMP datasets in the field » (l. 250), « These findings will enable SMP practitioners to automatically analyze their SMP measurements. To that end, an SMP user must simply decide on one of the fourteen models provided » (l. 369-370). However, I do not understand these sentences. I understood that everything relies on a single expert analysis, that the model must be retrained on other data (e.g., snow data in other places around the world) and that without this expert, no model can be retrained. In contrast, the limits of previous studies are somehow presented unfairly. For instance, it is indicated that « This [generalization] would not have been possible in previous works such as Satyawali et al. (2009) since knowledge rules for one snow region and season do not transfer to other regions or seasons » (l. 335), but the exact same applies to their work as the model must be retrained in any case to be used on other snowpack climate or expert analysis in the end (the model of Satyawali et al. (2009) could be retrained too).

The authors positively present the work as both « automatic classification and segmentation » (title and in the text) of snow profiles. It appears that no segmentation procedure is present in the paper. Indeed, the segmentation consists of saying that connected (i.e., neighboring) points with the same label belong to the same segment.

2. On the form, the description of the work is sometimes vague and incomplete.

The objective of the paper described in l23-31 seems rather unclear to me. It took me several reads to understand that the goal is to reproduce the classification of one expert on SMP data.

There is a welcome short bibliography on previous attempts to classify SMP profiles automatically. The description of the selected articles (Satyawali et al.,(2009), Havens et al., (2012), King et al., (2020)) would benefit from more detailed statements to capture what was really done in these papers. For instance, what is « too small to be representative » (of what?) (l. 34), « including knowledge-based rules » (l. 35), « good accuracy » (l. 42), and « additional snowpit information » (l. 42)?

Fig. 1, the international classification (Fierz et al.,2009) provides a color code. Is there a specific reason for not using it?

One key piece of information about the procedure is the list of predictors used as input for the ML model. They are very shortly described l. 79-86. But the description is too elusive to understand which variables are used. What are « added additional features», « time-dependent information » (where is time here ???), and « including variables of the shot noise model » (which variables?)?

Overall, I did not understand the overarching objective of the work and its applicability to different data. In addition, the presentation of previous and present work is too vague to capture the key ingredients of the methodology. I, however, acknowledge an important work to test numerous machine learning models (14) and the effort to provide the code source directly on a git repository.

I do not feel that the raised issues can be solved before publication.

Pascal Hagenmuller
Citation: https://doi.org/10.5194/egusphere-2022-938-RC2
- AC3:
  'Reply on RC2', Julia Kaltenborn, 05 Mar 2023
  Thank you for your comments and for pointing out which parts raised unclarities or further questions for you. We hope that our additional explanations provided below make the main objective of the paper clearer for you. We also hope that our suggested changes will make sure that those concerns do not arise anymore for the future reader.
  
  The most important changes to address your comments at a glance:
  We will include a detailed description of the labeling process and explain the context of the labeling in more detail.
  
  We will include a comprehensive usage guide for the snowdragon repository.
  
  We will extend the description of the previous work.
  
  We will include a more detailed description of the input data (“features”).
  
  We will make it transparent early on that we are only leaning toward the international classification of seasonal snow on the ground and why.
  
  We will change over-ambitious wording.
  
  We will change the title to communicate the “classification” part more strongly than the “segmentation” part.
  
  Thank you for your time and your feedback.
  
  Citation: https://doi.org/10.5194/egusphere-2022-938-AC3
AC1:
'Comment on egusphere-2022-938', Julia Kaltenborn, 05 Mar 2023
We thank both reviewers for their valuable and insightful comments. Here, we would like to summarize our main responses and changes to provide the editor with an overview. All those topics are addressed in more detail in the point-by-point responses for each reviewer.
Main responses:
The SMP profile labeling has actually been fine-tuned with Micro-CTs and NIR where possible. However, we do not have such measurements for each profile. Arctic conditions (25 m/s wind, -30°C) make in situ observations such as snow grain categorization on a metal plate very difficult, and changing personnel means that those assessments would not remain consistent. Under such conditions, the SMP can provide us with a large number of consistent profiles. Up-scaling consistent labeling of those profiles is exactly the type of task that ML algorithms can tackle.

We are only leaning toward the international classification of seasonal snow on the ground (Fierz et al., 2009). We are not using solely the classes provided there because it contains mainly alpine snow types and does not cover all typical Arctic snow types. However, we still use snow classes in general - instead of abandoning this concept completely - since we strive to use a common language of the cryospheric community.

The work presented here cannot and does not aim to remove the subjectivity of snow grain type classification. The models provided here generate a labeling that is consistent with the training data. As long as the training data is subjective, the predicted profiles will remain subjective. The subjectivity of snow grain type classification is not a problem confined to the SMP - in situ grain type classification on metal plates, snow pit stratigraphy analysis, NIR stratigraphy analysis, etc. - all of them are of subjective nature.

The work presented here is a methodological contribution, evaluated on numerical metrics. We also evaluated from a qualitative perspective if the predictions on an out-of-distribution dataset behave in a way that a practitioner (who labeled the training data!) would: 1) accept those predictions, 2) find them consistent with their own labeling, 3) and subsequently work with these predictions. We find this qualitative assessment important since it decides if the tools provided here can be used in practice.

The main objective of this study is to provide a way to up-scale manual SMP labeling and make the life of practitioners easier this way. We want to simplify the complex and longish procedure of SMP classification by providing ML algorithms that can reproduce a trained labeling pattern on new profiles. This way larger datasets can be processed and analyzed.

Main changes:
We are going to provide a detailed description of the labeling process (partially in the manuscript, in-depth in the supplementary materials) and the environmental context in which the data has been collected.

We will provide a description of the different micromechanical properties of different snow grain types. We will describe how they influence the SMP signals and thus the overall classification through ML models.

We would like to sharpen our manuscript regarding the main objective of this study.

We will provide a more detailed description of the included input features, the metrics, and some ML termini.

We also want to change our wording at several points, e.g. we will not use the word “ground truth” anymore, use the wording “depth-dependent” information, we will make clear that we are only leaning towards Fierz et al. (2009), and we will delete the “segmentation” part from our title.

We will add a detailed user guide to the supplementary material.

You can find more changes we plan to make in the point-by-point responses.

We were pleased to hear that we provide important work to test numerous machine learning models on this task and that the computer science aspect of our study was complete.
We would be very glad if we could present a revised version of our manuscript that includes those changes and several more as suggested by our reviewers. We thank everyone for their time and support!
Citation: https://doi.org/10.5194/egusphere-2022-938-AC1
EC1: 'Editor comment on egusphere-2022-938', Fabien Maussion, 07 Mar 2023

Dear authors, dear reviewers,
I have carefully read the reviews, the answers to the review, and the extended discussion resulting from the comments from Reviewer #2.
I think that both the review and the answer to the review make valid points. Here is my suggestion: I would like to invite the authors to submit a revised version of the manuscript according to the plan they outline in their response, if they feel confident enough to have addressed the most important issues raised by Reviewer #2. I will then ask for a third opinion on the revised paper.

Best regards,
Fabien Maussion

Citation: https://doi.org/10.5194/egusphere-2022-938-EC1

Julia Kaltenborn et al.

Data sets

Snowpit SnowMicroPen (SMP) force profiles collected during the MOSAiC expedition Macfarlane, Amy R.; Schneebeli, Martin; Dadic, Ruzica; Wagner, David N.; Arndt, Stefanie; Clemens-Sewall, David; Hämmerle, Stefan; Hannula, Henna-Reetta; Jaggi, Matthias; Kolabutin, Nikolai; Krampe, Daniela; Lehning, Michael; Matero, Ilkka; Nicolaus, Marcel; Oggier, Marc; Pirazzini, Roberta; Polashenski, Chris; Raphael, Ian; Regnery, Julia; Shimanchuck, Egor; Smith, Madison M.; Tavri, Aikaterini https://doi.pangaea.de/10.1594/PANGAEA.935554

Model code and software

snowdragon Kaltenborn, Julia https://doi.org/10.5281/zenodo.7335813

Pre-trained Models for SMP Classification and Segmentation Kaltenborn, Julia; Macfarlane, Amy R.; Clay, Viviane; Schneebeli, Martin https://zenodo.org/record/7063521

Julia Kaltenborn et al.

Viewed

Total article views: 531 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	BibTeX	EndNote
394	116	21	531	4	3

HTML: 394
PDF: 116
XML: 21
Total: 531
BibTeX: 4
EndNote: 3

Views and downloads (calculated since 06 Dec 2022)

Month	HTML	PDF	XML	Total
Dec 2022	119	32	9	160
Jan 2023	65	25	2	92
Feb 2023	58	14	2	74
Mar 2023	79	23	8	110
Apr 2023	58	8	0	66
May 2023	14	12	0	26
Jun 2023	1	2	0	3

Cumulative views and downloads (calculated since 06 Dec 2022)

Month	HTML	PDF	XML	Total
Dec 2022	119	32	9	160
Jan 2023	65	25	2	92
Feb 2023	58	14	2	74
Mar 2023	79	23	8	110
Apr 2023	58	8	0	66
May 2023	14	12	0	26
Jun 2023	1	2	0	3

Viewed (geographical distribution)

Total article views: 537 (including HTML, PDF, and XML) Thereof 537 with geography defined and 0 with unknown origin.

Country	#	Views	%

Latest update: 03 Jun 2023

Short summary

Snow-layer segmentation and classification is an essential diagnostic task for cryospheric applications. A SnowMicroPen (SMP) can be used to that end, however, the manual classification of its profiles becomes infeasible for large datasets. Here, we evaluate how well machine learning models can automate this task. Of the 14 different models trained on the MOSAiC SMP dataset, the long short-term memory performed the best. The findings presented here facilitate and accelerate SMP data analysis.


Total:	0
HTML:	0
PDF:	0
XML:	0