Thesis subject

MSc thesis topic: Temporal Vegetation Modelling with Masked Auto Encoding

Filling masked gaps in images or time series is a common technique in natural language processing or computer vision to pre-train a deep learning model without labels. In computer vision, Masked Image Modelling (MIM) or Masked Auto Encoding (MAE) are common methodologies that have proven extremely effective in natural language processing (see Bert model Devlin et al., 2019). This pre-training methodology is less explored remote sensing time series. Examples for single images involves SatMAE (Cong et al.,2022) or for time series Presto (Tseng et al., 2023).

In this thesis, you will train and explore masked time series modelling methods for remote sensing time series for one of the following objectives (to be determined based on your interest)

  1. Pre-train a deep learning model for a series of benchmark tasks in the Netherlands (crop type mapping, tree species identification, land cover classification, liveability mapping)
  2. Analyse the reconstructed time series for outliers that may correspond to unexpected time series behaviours and may hint, for instance, to the use of glyphosate in crop land
  3. Model the phenology of vegetation to obtain a distribution of “normal” vegetation states based of which breaks in the season can be identified (link to BFAST) or extreme events can be detected
  4. Analyse the performance of masked auto encoders in gap filling time series (in time and space) to improve land cover change detection accuracy. The method would be compared to classical gap filling methods, such as linear interpolation, STLplus, Markov models or Random Forest imputing etc.

Relevance to research/projects at GRS or other groups

This thesis is highly relevant for remote sensing analysis by developing and utilizing pre-trained deep learning models for time series analysis for a variety of remote sensing problems

Objectives and Research questions

  • Modify the PRESTO model on time series data from the Netherlands (or train your own MAE-based model)
  • Evaluate the model either on 1. Downstream tasks 2. Analyze the reconstructed time series for outliers (glyphosate) or 3) Identify changes in the regular vegetation phenology to identify changes in the vegetation cycle.

Requirements

  • Geoscripting course (required)
  • Deep Learning Course (required)

Literature and information

  • Tseng, G., Cartuyvels, R., Zvonkov, I., Purohit, M., Rolnick, D., & Kerner, H. (2023). Lightweight, pre-trained transformers for remote sensing timeseries. arXiv preprint arXiv:2304.14065.
  • Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019, June). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) (pp. 4171-4186).
  • He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2022). Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 16000-16009).
  • Cong, Y., Khanna, S., Meng, C., Liu, P., Rozi, E., He, Y., ... & Ermon, S. (2022). Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery. Advances in Neural Information Processing Systems, 35, 197-211.

Theme(s): Modelling & visualisation