the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 12 Dec 2023
The AutoICE Challenge
Abstract. Mapping sea ice in the Arctic is essential for maritime navigation, and growing vessel traffic highlights the necessity of timeliness and accuracy of sea ice charts. In addition, with the increased availability of satellite imagery, automation is becoming more important. The aim of the AutoICE Challenge was to encourage the creation of models capable of mapping sea ice automatically from spaceborne Synthetic Aperture Radar (SAR) imagery using deep learning while inspiring participants to move towards multiple sea ice parameter model retrieval instead of the current focus on a single sea ice parameter, such as concentration. Participants were tasked with the development of machine learning algorithms mapping the total sea ice concentration, stage of development and floe size using a state-of-the-art sea ice dataset with dual-polarised Sentinel-1 SAR images and 22 other relevant variables while using professionally labelled sea ice charts from multiple national ice services as reference data. The challenge had 129 teams representing a total of 179 participants, with 34 teams delivering 494 submissions, resulting in a participation rate of 26.4 %, and was won by a team from the University of Waterloo. Participants were successful in training models capable of retrieving multiple ice parameters with convolutional neural network and vision transformer models. The top participants scored best on the total sea ice concentration and stage of development, while the floe size was more difficult. Furthermore, participants offered intriguing approaches and ideas that could help propel future research within automatic sea ice mapping, such as applying high downsampling of SAR data to improve model efficiency and produce better results.
-
Notice on discussion status
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
-
Preprint
(17894 KB)
-
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(17894 KB) - Metadata XML
- BibTeX
- EndNote
- Final revised paper
Journal article(s) based on this preprint
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2023-2648', Anonymous Referee #1, 07 Jan 2024
Review of egusphere-2023-2648
"The AutoICE Challenge"General Comment:
- This article reads extremely well, with no issues with the English language use and with good clear content.
- However, there is just no scientific content or progress. The article would be a very nice summary for the competition web-pages, but it does not achieve any scientific goals.
- There are some hints at scientific concepts, some summarising of the state-of-the-art, but they are only suggestions with no backup or supporting evidence.
- I conclude that this work could possibly be made valuable, but a total overhaul is needed to emphasise the scientific messages.
Specific Comments:- Abstract: No scientific goals are mentioned. The competition may very well have encouraged novel development, but that is not explicitly explored. The organisers are in an unique position to inter-compare and condense the information contained in the choices of solutions and those that achieved best results. They do make some hints at categorising this, but it needs to go much further.
- I note that lines 94-95 and 101-102 do contain some aims to inter-compare and discuss, but these need to be made the formal goal and emphasis in the article. A "literature review" type paper is only worthwhile if the authors can add some value in the summary of the material and opinions on directions to take.
- Section 7, the "Discussion" does attempt to suggest some good ideas and reasons for the results, but it is mostly speculation. Do the authors have any further evidence, or can they actually perform any experiments to demonstrate these ideas, perhaps with the assistance of the competition teams who are listed as co-authors? The value would be hugely more meaningful with further logical or experimental support.
- Lines 160-167: When introducing the sea ice charts, it may be worth emphasising the very coarsely drawn polygons, as this is a well known aspect of the manually drawn charts. The limitations of using known coarse and mixed regions should probably have further discussion, since it is being discussed in the community. Do we want to mimic these coarse charts, or create something more detailed. The egg-codes do account for partial mixtures, but machine-learning training does not usually manage this. Further discussion of this in the motivation and discussions would be valuable.
Technical Comments:- Line 159: This line appears to start a list, but the list is lost in the following sub-headings, with no clear termination. I suggest two options: Firstly, since the sub-sections are quite long, I suggest that this leading sentence actually lists them by name and completes the list with a comment like "explained below"; Secondly, add sub-section numbering or indentation to make it clear which items are in the list, and when the list stops and does not continue into the next sub-section.
- Line 169: I believe that the description of the colours should be "lighter colour" and not "brighter colour". Brighter usually implies stronger/richer/saturated colour. While "lighter" can mean with more white added (which is used in the figure), and the converse "darker" would be towards black.
- The sentence on lines 178-179 is very unclear. Please explain more clearly the relationship between the values, sub-categories of ice, and the polygon.
- Lines 259-261: These sentences should be either better joined, or thematically separated. At the moment, the first sentence says that it incorporates both spatial and temporal information by adding lat-lon coordinates only. The temporal is in the separate sentence. Please get make the logic correct.
- Lines 292-294: You are adding weights p and q. Do these act like prior probabilities, abundances? Are they likely to be image, region or season dependent?
- Line 296: Probably need a "The" in front of "Following data augmentations", and subsequent capitalisation change.
- Lines 299-300: There is something grammatically wrong with "This approach enabled us to make SIC, SOD and FLOE maps simultaneous predictions." Are you meaning that you make the predicted three image "maps" simultaneously?
- Line 303: Something is grammatically wrong with "the checkpoint used loss function". Is the word "used" necessary? Or, if this a specific phrase, then add hyphens or quote or emphasis to indicate this.
- Line 317: It seems that CE is listed twice, or are some list items compound (with two pieces)? Perhaps it needs further commas to separate the compound elements, or say how many "best" results you are listing, or clarify the meaning some other way.
- Line 445: Suggest "...was the only one of the top-5 teams...".
Citation: https://doi.org/10.5194/egusphere-2023-2648-RC1 - AC1: 'Reply on RC1', Andreas Stokholm, 05 Apr 2024
-
RC2: 'Comment on egusphere-2023-2648', Anonymous Referee #2, 16 Feb 2024
The authors present a good summary of the “AutoICE Challenge” competition. All the relevant points about the competition, including the objective, dataset, good solutions, and future work, are clearly and well written. Spaceborne SAR has demonstrated great potential for operational monitoring of sea ice in polar regions. The AutoICE challenge competition is an interesting initiative. The presented top solutions use various deep learning methods, from U-net to DeepLabV3+, as well as computer vision technique. They all yield good results on sea ice concentration, sea ice types and ice floes. This suggests that combination of SAR observations and machine learning is a promising solution for retrieving various sea ice parameters in much higher spatial resolution compared with other conventional remote sensing data.
However, the current manuscript is a project summary or report, instead of scientific paper, which is not suitable for the journal. If the authors would like to summarize state of the art of SAR observations of sea ice based on deep learning methods, together with the AutoICE challenge results, which might be more interesting to the scientific community. In fact, even before the AutoICE challenge starting, there are quite a few studies have applied various deep learning methods to derive sea ice information by spaceborne SAR.
Citation: https://doi.org/10.5194/egusphere-2023-2648-RC2 - AC1: 'Reply on RC1', Andreas Stokholm, 05 Apr 2024
-
RC3: 'Comment on egusphere-2023-2648', Anonymous Referee #3, 18 Feb 2024
General comment:
This article gives a comprehensive overview of the AutoICE challenge, including the data set, evaluation metrics, and summaries of the top contributions. It is very well written and easy to follow, there are almost no language or formatting issues. While I think that the manuscript is worthwhile reading for anyone working in the field of sea ice remote sensing and ice charting, it is in its current form unfortunately not a scientific. The authors do not formulate a research question and do not provide a detailed evaluation of any scientific novelty.
While some novelty and scientific description (although more technical than scientific) is given in the summary of the top submissions to the competition in Section 5, the authors mention that there will be individual contributions of these solutions to a Cryosphere special issue on the AutoICE challenge. Simply repeating a summary of those contributions in this manuscript does not present any additional value. I note that the data set itself is of course a valuable scientific contribution - however publication of the data set does not warrant publication of this manuscript in its current form.
The authors hint at several topics that could be investigated in detail (such as for example the influence of spatial resolution on the results for floe size distribution or the inherent uncertainty in the ice charts that are used for training and evaluation). I think that, with further work, the results and contributions of the competition could lead to valuable scientific contributions on these (or similar) research questions.
Technical/specific comments:
I only have very few technical or language related comments, most of which have already been mentioned in RC1. Since I think that the manuscript needs significant changes before it can be published, I will not list wording details here, but only give two general technical comments for any revised version of this manuscript:
- Please be consistent in use (or no use) of Oxford comma.
- I find that the different viewing geometries of the data presented in Fig 2,3,4 make it difficult to compare the content. I'd suggest using the same geometry and projection.
Citation: https://doi.org/10.5194/egusphere-2023-2648-RC3 - AC1: 'Reply on RC1', Andreas Stokholm, 05 Apr 2024
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2023-2648', Anonymous Referee #1, 07 Jan 2024
Review of egusphere-2023-2648
"The AutoICE Challenge"General Comment:
- This article reads extremely well, with no issues with the English language use and with good clear content.
- However, there is just no scientific content or progress. The article would be a very nice summary for the competition web-pages, but it does not achieve any scientific goals.
- There are some hints at scientific concepts, some summarising of the state-of-the-art, but they are only suggestions with no backup or supporting evidence.
- I conclude that this work could possibly be made valuable, but a total overhaul is needed to emphasise the scientific messages.
Specific Comments:- Abstract: No scientific goals are mentioned. The competition may very well have encouraged novel development, but that is not explicitly explored. The organisers are in an unique position to inter-compare and condense the information contained in the choices of solutions and those that achieved best results. They do make some hints at categorising this, but it needs to go much further.
- I note that lines 94-95 and 101-102 do contain some aims to inter-compare and discuss, but these need to be made the formal goal and emphasis in the article. A "literature review" type paper is only worthwhile if the authors can add some value in the summary of the material and opinions on directions to take.
- Section 7, the "Discussion" does attempt to suggest some good ideas and reasons for the results, but it is mostly speculation. Do the authors have any further evidence, or can they actually perform any experiments to demonstrate these ideas, perhaps with the assistance of the competition teams who are listed as co-authors? The value would be hugely more meaningful with further logical or experimental support.
- Lines 160-167: When introducing the sea ice charts, it may be worth emphasising the very coarsely drawn polygons, as this is a well known aspect of the manually drawn charts. The limitations of using known coarse and mixed regions should probably have further discussion, since it is being discussed in the community. Do we want to mimic these coarse charts, or create something more detailed. The egg-codes do account for partial mixtures, but machine-learning training does not usually manage this. Further discussion of this in the motivation and discussions would be valuable.
Technical Comments:- Line 159: This line appears to start a list, but the list is lost in the following sub-headings, with no clear termination. I suggest two options: Firstly, since the sub-sections are quite long, I suggest that this leading sentence actually lists them by name and completes the list with a comment like "explained below"; Secondly, add sub-section numbering or indentation to make it clear which items are in the list, and when the list stops and does not continue into the next sub-section.
- Line 169: I believe that the description of the colours should be "lighter colour" and not "brighter colour". Brighter usually implies stronger/richer/saturated colour. While "lighter" can mean with more white added (which is used in the figure), and the converse "darker" would be towards black.
- The sentence on lines 178-179 is very unclear. Please explain more clearly the relationship between the values, sub-categories of ice, and the polygon.
- Lines 259-261: These sentences should be either better joined, or thematically separated. At the moment, the first sentence says that it incorporates both spatial and temporal information by adding lat-lon coordinates only. The temporal is in the separate sentence. Please get make the logic correct.
- Lines 292-294: You are adding weights p and q. Do these act like prior probabilities, abundances? Are they likely to be image, region or season dependent?
- Line 296: Probably need a "The" in front of "Following data augmentations", and subsequent capitalisation change.
- Lines 299-300: There is something grammatically wrong with "This approach enabled us to make SIC, SOD and FLOE maps simultaneous predictions." Are you meaning that you make the predicted three image "maps" simultaneously?
- Line 303: Something is grammatically wrong with "the checkpoint used loss function". Is the word "used" necessary? Or, if this a specific phrase, then add hyphens or quote or emphasis to indicate this.
- Line 317: It seems that CE is listed twice, or are some list items compound (with two pieces)? Perhaps it needs further commas to separate the compound elements, or say how many "best" results you are listing, or clarify the meaning some other way.
- Line 445: Suggest "...was the only one of the top-5 teams...".
Citation: https://doi.org/10.5194/egusphere-2023-2648-RC1 - AC1: 'Reply on RC1', Andreas Stokholm, 05 Apr 2024
-
RC2: 'Comment on egusphere-2023-2648', Anonymous Referee #2, 16 Feb 2024
The authors present a good summary of the “AutoICE Challenge” competition. All the relevant points about the competition, including the objective, dataset, good solutions, and future work, are clearly and well written. Spaceborne SAR has demonstrated great potential for operational monitoring of sea ice in polar regions. The AutoICE challenge competition is an interesting initiative. The presented top solutions use various deep learning methods, from U-net to DeepLabV3+, as well as computer vision technique. They all yield good results on sea ice concentration, sea ice types and ice floes. This suggests that combination of SAR observations and machine learning is a promising solution for retrieving various sea ice parameters in much higher spatial resolution compared with other conventional remote sensing data.
However, the current manuscript is a project summary or report, instead of scientific paper, which is not suitable for the journal. If the authors would like to summarize state of the art of SAR observations of sea ice based on deep learning methods, together with the AutoICE challenge results, which might be more interesting to the scientific community. In fact, even before the AutoICE challenge starting, there are quite a few studies have applied various deep learning methods to derive sea ice information by spaceborne SAR.
Citation: https://doi.org/10.5194/egusphere-2023-2648-RC2 - AC1: 'Reply on RC1', Andreas Stokholm, 05 Apr 2024
-
RC3: 'Comment on egusphere-2023-2648', Anonymous Referee #3, 18 Feb 2024
General comment:
This article gives a comprehensive overview of the AutoICE challenge, including the data set, evaluation metrics, and summaries of the top contributions. It is very well written and easy to follow, there are almost no language or formatting issues. While I think that the manuscript is worthwhile reading for anyone working in the field of sea ice remote sensing and ice charting, it is in its current form unfortunately not a scientific. The authors do not formulate a research question and do not provide a detailed evaluation of any scientific novelty.
While some novelty and scientific description (although more technical than scientific) is given in the summary of the top submissions to the competition in Section 5, the authors mention that there will be individual contributions of these solutions to a Cryosphere special issue on the AutoICE challenge. Simply repeating a summary of those contributions in this manuscript does not present any additional value. I note that the data set itself is of course a valuable scientific contribution - however publication of the data set does not warrant publication of this manuscript in its current form.
The authors hint at several topics that could be investigated in detail (such as for example the influence of spatial resolution on the results for floe size distribution or the inherent uncertainty in the ice charts that are used for training and evaluation). I think that, with further work, the results and contributions of the competition could lead to valuable scientific contributions on these (or similar) research questions.
Technical/specific comments:
I only have very few technical or language related comments, most of which have already been mentioned in RC1. Since I think that the manuscript needs significant changes before it can be published, I will not list wording details here, but only give two general technical comments for any revised version of this manuscript:
- Please be consistent in use (or no use) of Oxford comma.
- I find that the different viewing geometries of the data presented in Fig 2,3,4 make it difficult to compare the content. I'd suggest using the same geometry and projection.
Citation: https://doi.org/10.5194/egusphere-2023-2648-RC3 - AC1: 'Reply on RC1', Andreas Stokholm, 05 Apr 2024
Peer review completion
Journal article(s) based on this preprint
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 578 | 229 | 39 | 846 | 56 | 28 |
- HTML: 578
- PDF: 229
- XML: 39
- Total: 846
- BibTeX: 56
- EndNote: 28
Viewed (geographical distribution)
| Country | # | Views | % |
|---|---|---|---|
| United States of America | 1 | 294 | 35 |
| Canada | 2 | 79 | 9 |
| Norway | 3 | 52 | 6 |
| China | 4 | 50 | 6 |
| Germany | 5 | 48 | 5 |
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
- 294
Cited
1 citations as recorded by crossref.
Andreas R. Stokholm
Jørgen Buus-Hinkler
Tore Wulf
Anton Korosov
Roberto Saldo
Leif T. Pedersen
David Arthurs
Ionut Dragan
Iacopo Modica
Juan Pedro
Annekatrien Debien
Xinwei Chen
Muhammed Patel
Fernando J. P. Cantu
Javier N. Turnes
Jinman Park
Linlin Xu
Andrea K. Scott
David A. Clausi
Yuan Fang
Mingzhe Jiang
Saeid Taleghanidoozdoozan
Neil C. Brubacher
Armina Soleymani
Zacharie Gousseau
Michał Smaczny
Patryk Kowalski
Jacek Komorowski
David Rijlaarsdam
Jan N. van Rijn
Jens Jakobsen
Martin S. J. Rogers
Nick Hughes
Tom Zagon
Rune Solberg
Nicolas Longépé
Matilde B. Kreiner
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(17894 KB) - Metadata XML