05 Dec 2023
 | 05 Dec 2023
Status: this preprint is open for discussion.

From simple labels to semantic image segmentation: Leveraging citizen science plant photographs for tree species mapping in drone imagery

Salim Soltani, Olga Ferlian, Nico Eisenhauer, Hannes Feilhauer, and Teja Kattenborn

Abstract. Knowledge of plant species distributions is essential for various applications, such as nature conservation, agriculture, and forestry. Remote sensing data, especially high-resolution orthoimages from Unoccupied Aerial Vehicles (UAVs), were demonstrated to be an effective data source for plant species mapping. Particularly, in concert with novel pattern recognition methods, such as Convolutional Neural Networks (CNNs), plant species can be accurately segmented in such high-resolution UAV images. Training such pattern recognition models for species segmentation that are transferable across various landscapes and remote sensing data characteristics often requires excessive training data. Training data are usually derived in the form of segmentation masks from field surveys or visual interpretation of the target species in remote sensing images. Still, both methods are laborious and constrain the training of transferable pattern recognition models. Alternatively, pattern recognition models could be trained on the open knowledge of how plants look as available from smartphone-based species identification apps, that is, millions of citizen science-based smartphone photographs and the corresponding species label. However, these pairs of citizen science-based photographs and simple species labels (one label for the entire image) cannot be used directly for training state-of-the-art segmentation models used for UAV image analysis, which require per-pixel labels for training (also called masks). Here, we overcome the limitation of simple labels of citizen science plant observations with a two-step approach: In the first step, we train CNN-based image classification models using the simple labels and apply them in a moving-window approach over UAV orthoimagery to create segmentation masks. In the second phase, these segmentation masks are used to train state-of-the-art CNN-based image segmentation models with an encoder-decoder structure. We tested the approach on UAV orthoimages acquired in summer and autumn on a test site comprising ten temperate deciduous tree species in varying mixtures. Several tree species could be mapped with surprising accuracy (mean F1-score = 0.47). In homogenous species assemblages, the accuracy increased considerably (mean F1-score 0.55). The results indicate that many tree species can be mapped without generating training data and by integrating pre-existing knowledge from citizen science. Moreover, our analysis revealed that citizen science photographs’ variability in acquisition data and context facilitates the generation of models that are transferable through the vegetation season. Thus, citizen science data may greatly advance our capacity to monitor hundreds of plant species and, thus, Earth's biodiversity across space and time.

Salim Soltani, Olga Ferlian, Nico Eisenhauer, Hannes Feilhauer, and Teja Kattenborn

Status: open (until 24 Mar 2024)

Comment types: AC – author | RC – referee | CC – community | EC – editor | CEC – chief editor | : Report abuse
  • RC1: 'Comment on egusphere-2023-2576', Anonymous Referee #1, 12 Jan 2024 reply
Salim Soltani, Olga Ferlian, Nico Eisenhauer, Hannes Feilhauer, and Teja Kattenborn
Salim Soltani, Olga Ferlian, Nico Eisenhauer, Hannes Feilhauer, and Teja Kattenborn


Total article views: 792 (including HTML, PDF, and XML)
HTML PDF XML Total BibTeX EndNote
582 166 44 792 15 15
  • HTML: 582
  • PDF: 166
  • XML: 44
  • Total: 792
  • BibTeX: 15
  • EndNote: 15
Views and downloads (calculated since 05 Dec 2023)
Cumulative views and downloads (calculated since 05 Dec 2023)

Viewed (geographical distribution)

Total article views: 757 (including HTML, PDF, and XML) Thereof 757 with geography defined and 0 with unknown origin.
Country # Views %
  • 1
Latest update: 25 Feb 2024
Short summary
In this research, we developed a novel method using citizen science data as alternative training data for computer vision models to map plant species in Unoccupied Aerial Vehicles (UAVs) images. We use citizen science plant photographs to train models and apply them to UAV images. We tested our approach on UAV images of a test site with ten different tree species, yielding accurate results. This research shows the potential of citizen science data to advance our ability to monitor plant species.