Bayesian evaluation of deep learning architectures and sensor modalities for remote sensing-based driftwood segmentation in the Mackenzie Delta, Arctic Canada

Stadie, Carl; Nitze, Ingmar; Demir, Begüm; Brandt, Martin Stefan; Grosse, Guido

doi:10.5194/egusphere-2026-4074

Preprints

https://doi.org/10.5194/egusphere-2026-4074

Preprints

20 Aug 2026

| 20 Aug 2026

Bayesian evaluation of deep learning architectures and sensor modalities for remote sensing-based driftwood segmentation in the Mackenzie Delta, Arctic Canada

Carl Stadie, Ingmar Nitze, Begüm Demir, Martin Stefan Brandt, and Guido Grosse

Abstract. Automated mapping of driftwood deposits along Arctic coastlines is a challenging task due to spectral ambiguity, strong depositional heterogeneity, and limited training data. Systematic comparisons of sensor modality and deep learning architecture choices remain absent for this application, leaving practitioners without evidence-based guidance on which combination to deploy. Here we present a statistical evaluation of three architectures, U-Net, Swin-U-Net, and the TerraMind foundation model, across aerial (15 cm), PlanetScope (3 m), and Sentinel-2 (10 m) imagery acquired over ten target areas in the Mackenzie Delta, Arctic Canada. Each combination was trained ten times and evaluated within a Bayesian hierarchical framework to account for run-to-run variability inherent to stochastic training. Choice of the sensor is the dominant performance driver, having an effect approximately four times larger than the choice of architecture in Intersection over Union and a total sensor spread of 0.40 IoU between aerial and Sentinel-2 imagery. Architecture choice is of limited practical consequence at sub-metre and intermediate resolution, but becomes a first-order concern when constrained to coarse imagery: U-Net performs poorly on Sentinel-2 with a posterior mean IoU of 0.127, while transformer-based architectures show a more gradual performance decline. Swin-U-Net paired with PlanetScope imagery is the most competitive accessible alternative to aerial acquisition, with a 94 % posterior probability of practical equivalence to the top-ranked configuration. Conventional single-run evaluation missed this combination as a practical alternative, typically placing it at ranks 4–5. Probabilistic multi-run evaluation is therefore a necessary condition for reliable model selection in spectrally ambiguous remote sensing benchmarks, and the framework presented here could be directly transferable to similar Arctic mapping targets.

Received: 08 Jul 2026 – Discussion started: 20 Aug 2026

Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this paper. While Copernicus Publications makes every effort to include appropriate place names, the final responsibility lies with the authors. Views expressed in the text are those of the authors and do not necessarily reflect the views of the publisher.

Download & links

Preprint (PDF, 15048 KB)

Supplement (1960 KB)

Download & links

Preprint (15048 KB)
Metadata XML
Supplement (1960 KB)
BibTeX
EndNote

Carl Stadie, Ingmar Nitze, Begüm Demir, Martin Stefan Brandt, and Guido Grosse

Status: final response (author comments only)

RC1:
'Comment on egusphere-2026-4074', Anonymous Referee #1, 30 Sep 2026
GENERAL COMMENTS: The paper proposes a Bayesian hierarchical framework to examine the key factors that affect performance when trying to segment driftwood. They consider three model architectures (U-Net, Swin-U-NET, and TerraMind) and data from three sensors (Planet Scope, Sentinel-2, and aerial imagery). The proposed approach is aimed at guiding practitioners in selecting sensor-architecture combinations. The rationale of considering the spectral ambiguity between driftwood and background as a source of stochasticity was interesting. Stochasticity of neural network training is a relevant and valuable topic for the community, which is often overlooked. The inclusion of code and data with the paper will strengthen reproducibility. The emphasis on using multiple runs for evaluation is also a strong point. As such, this paper could be a timely addition to the earth observation community. However, some of the claims made need to be softened as they are not fully supported by the findings in the paper. Some additional analysis is also required to address limitations in the comparison.
RECOMMENDATION: Major revisions
MAJOR POINTS:
There are existing approaches for handling stochasticity in training, such as bootstrapping the mean across multiple seeds. Please clarify why these approaches are not sufficient. What additional value does the proposed approach applying Bayesian hierarchical framework provide over these existing techniques? Please also compare results from the proposed approach to another existing method beyond the single-run evaluation, such as bootstrapping over the 10 runs.

Paper states: “A practitioner relying on single-run evaluation would therefore overlook this combination as a practical alternative entirely”. Also, it states: “Conventional single-run evaluation missed this combination as a practical alternative, typically placing it at ranks 4–5.” Please state which metrics the ranks refer to. If “rank 2” relates to the composite ranking while “rank 4 or 5” relates to IoU and Dice metrics, then these claims are not supported by the comparison. Please compare single-run results to the posterior ranks on the same metric instead.

A key finding made in the abstract depends on the composite utility score. However, this score is computed based on weights that have been “pre-specified”, and the justification for this selection could be seen as subjective. Section 4.3.2 states that the chosen weighting is “illustrative”, but it is being used to support a key finding in the abstract. This is especially important because, as acknowledged in section 4.2, NSD is relative to sensor resolution rather than absolute. With only 1 pixel used for τ, aerial sensor results have a much lower tolerance than Sentinel-2, which could explain why the NSD results generally trend in the opposite direction to IoU for each model. Please provide results from sensitivity analysis on the composite score weight. Do the key findings in the abstract still hold? Please provide results with a sweep on each weight. Also provide results with each component dropped one at a time.

The evaluations made in the paper have several confounding variables. How much of the differences are actually due to the differences in training set-up and not architecture (e.g. unfreezing strategy used for TerraMind with pretrained weights, U-Net is trained from scratch, Swin-U-Net uses pre-trained weights but is seemingly not frozen). How much of the difference in sensor results is actually due to differences in how training data was curated for different sensors such as different dates, different bands, different targets, etc? New experiments are not needed to address this but please add a caveat explicitly noting the differences in training and data curation strategies, and how these can affect the interpretation of results. Ideally, claims should be softened in the abstract and conclusion.

Some claims are not supported based on the evidence given in the paper. For example, stating that this work could be “directly transferable” even though all data comes from a single region. Please qualify these findings as region-specific.

The use of pre-trained weights for Swin-U-Net is not a minor detail, neither is the frozen backbone/unfreezing strategy used for TerraMind. It affects how readers will interpret the results. Please clarify if pre-trained weights were used for all tuned configurations with Swin-U-Net and whether any portion of the model was frozen. Also include these considerations in how results are interpreted. For example, the paper states: “TerraMind configurations show near-zero epistemic uncertainty across all sensors, several orders of magnitude below U-Net and Swin-U-Net, accompanied by elevated aleatoric uncertainty most prominently on Sentinel-2.” Is this possibly due to using a (mostly) frozen pre-trained encoder which would have reduced variance? Please consider stating how the training strategy affected the findings.

MINOR POINTS:
Nice work doing such detailed hyperparameter optimisation for each sensor-architecture combination. Please add info on the budget used to tune each combination. Since there are so many parameters being optimised, it is important that a sufficient budget was used. In addition, some of the parameters being optimised are confounding. To address this please add a note saying the parameters chosen were from the best-effort tuning.

Paper states: “The TerraMind backbone weights are frozen throughout fine tuning and only unfrozen every n epochs (as determined during hyperparameter tuning).” This sounds ambiguous. Please clarify what is meant here. Does this mean the backbone is unfrozen once after n epochs, or does it mean repeatedly unfreezing and refreezing after n epochs?

Paper states: “epistemic uncertainty estimates for TerraMind configurations mostly reflect parameter uncertainty in the decoder only”. If the backbone is unfrozen during part of the training, then it is misleading to refer to the uncertainty in the “decoder only”. Please rephrase or remove.

Paper states: “The TerraMind foundation model did not outperform task-specific architectures on any sensor, including Sentinel-2, its native pretraining modality.” This statement is not accurate according to Table 1. Results show that TerraMind beats U-Net on Sentinel-2 for IoU and Dice metrics. Please clarify this claim.

TECHNICAL POINTS
Please list which bands were used for Sentinel-2.

Broken citation on line 296 (?).

For each table of results, please put the best result in each block of the table in bold to make it easier to read.
Citation: https://doi.org/10.5194/egusphere-2026-4074-RC1
- RC2: 'Additional comments on prioritising revisions', Anonymous Referee #1, 30 Sep 2026
  
  To provide some clarity for any revision effort, please note the following:
  
  Major point 1 is the most important as it is concerned with justifying the paper’s core methodological contribution (using a Bayesian hierarchical framework to handle stochasticity in model evaluation). Specifically, the framework should be compared against at least one other existing approach for analysing multi-run results (e.g. bootstrapping across the 10 runs), with a discussion about what additional value the Bayesian approach provides.
  Major points 2-3 must also be addressed as they are directly related to key conclusions made in the paper. For point 2, this would mean comparing single-run ranks to posterior ranks on the same metric. For point 3, this would require sensitivity analysis on the weights used in the composite score.
  Major points 1-3 can be handled with additional post hoc analysis on the existing results; retraining the models is not necessarily required. The findings mentioned in the abstract, conclusion, and discussion should be updated depending on what the additional post-analysis shows, especially the conclusions on Swin-U-Net with Planet Scope.
  Major point 4-6 are also important but can be addressed by making clarifications in the text. This includes adding caveats about the potential effects of differences in training setup (pre-training, freezing/unfreezing, etc), reducing the scope of some claims, and clarifying the interpretation of architecture effects.
  The minor points and technical points should be straightforward and can be handled with simple edits.
  
  Citation: https://doi.org/10.5194/egusphere-2026-4074-RC2
RC3:
'Comment on egusphere-2026-4074', Anonymous Referee #2, 05 Oct 2026
This manuscript addresses a relevant Earth observation problem through a systematic comparison of three model architectures across three sensor modalities for driftwood mapping. Its strengths include the use of real Arctic imagery, repeated training runs, and a Bayesian framework for assessing performance differences and ranking uncertainty. However, concerns regarding the consistency of ranking comparisons, cross-sensor evaluation criteria, reference annotation comparability, and the interpretation of uncertainty require substantial clarification and additional analysis. I therefore recommend major revision before the manuscript can be considered for publication. My specific comments are as follows:
Please clarify whether the posterior ranking in each panel of Figure S8 is calculated using that panel’s metric or the composite score. If single-run IoU rankings are compared with composite posterior rankings, the differences may reflect the scoring criteria rather than training variability.

Using a one-pixel NSD tolerance corresponds to substantially different physical distances across sensors, which may influence the composite ranking and the reported practical equivalence. Please assess sensitivity to common physical tolerances and alternative metric weights, and clarify that equivalence is conditional on the chosen composite score.

Independently annotated reference masks may represent different object boundaries and levels of aggregation across sensors, confounding sensor effects with differences in the evaluation targets. Please provide an evaluation against a common reference or explicitly discuss the limitations of comparing performance against sensor-specific annotations.

Decoder-only Monte Carlo dropout in TerraMind limits direct comparison with uncertainty estimates obtained using different stochastic sampling schemes, and low predictive variance alone does not demonstrate reliable calibration. Similarly, differences in pretraining and training configurations should be controlled or acknowledged before attributing performance differences primarily to architecture.

Please reconcile the reported sampled area of 177 km² with 400 tiles measuring 500 × 500 m, which imply a gross nominal area of 100 km². The claim of substantial overlap among all architecture-specific IoU HDIs should also be revised, as several intervals in Table 1 are disjoint.
Citation: https://doi.org/10.5194/egusphere-2026-4074-RC3
EC1: 'Comment on egusphere-2026-4074', Peng Liu, 14 Oct 2026

The authors may wish to consider whether the Bayesian aspect should remain the central emphasis of the title, or whether highlighting the broader objectives of uncertainty-aware model evaluation and sensor–architecture selection could more fully convey the scope and relevance of the manuscript to the Earth observation community.

Citation: https://doi.org/10.5194/egusphere-2026-4074-EC1

Carl Stadie, Ingmar Nitze, Begüm Demir, Martin Stefan Brandt, and Guido Grosse

Supplement

https://doi.org/10.5194/egusphere-2026-4074-supplement

Carl Stadie, Ingmar Nitze, Begüm Demir, Martin Stefan Brandt, and Guido Grosse

Viewed

Total article views: 389 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	Supplement	BibTeX	EndNote
199	96	94	389	73	82	97

HTML: 199
PDF: 96
XML: 94
Total: 389
Supplement: 73
BibTeX: 82
EndNote: 97

Views and downloads (calculated since 20 Aug 2026)

Month	HTML	PDF	XML	Total
Aug 2026	41	12	8	61
Sep 2026	88	45	64	197
Oct 2026	70	39	22	131

Cumulative views and downloads (calculated since 20 Aug 2026)

Month	HTML	PDF	XML	Total
Aug 2026	41	12	8	61
Sep 2026	88	45	64	197
Oct 2026	70	39	22	131

Viewed (geographical distribution)

Total article views: 381 (including HTML, PDF, and XML) Thereof 381 with geography defined and 0 with unknown origin.

Country	#	Views	%

Latest update: 22 Oct 2026

Short summary

Driftwood piles up along Arctic coasts, storing carbon and shaping habitats, but mapping it automatically is difficult. We compared three artificial intelligence models on aerial photos and satellite images, repeating each test many times to capture natural training variation. Image resolution mattered far more than which model was used, and testing only once, as is common, gave misleading results. Our approach offers a more reliable way to choose mapping tools for similar Arctic features.


Total:	0
HTML:	0
PDF:	0
XML:	0