This project aims to segment palsas (a type of permafrost) from aerial images. Specifically, this research used training data from Sweden. Lantmateriet has aerial images (RGB and height information) publically available. There were no segmentation labels to train a model with, however previous work by Backe et al. provides us with weak labels. Therefore, to identify the locations of palsas, a weakly supervised learning approach was used. This project is two-step:
The weak labels were used to train a CNN. The goal of this is to generate class activation maps (CAMs) from the last feature map which contain localization information of the palsas. The dataset by Backe indicated both (1) whether images contained palsa, and provided (2) an estimate of the % of the image which contained palsa. Therefore, we pretrained a CNN classifier on the binary labels. Then, we finetuned the same network with the % information as a loss. Particularily, the classification head was removed. The model was adjusted such that the output was the % of cells activated in the last feature map. This was compared to the % estimate determined by Backe, and used as a loss to further finetune the model. The resulting model was used to generate the CAMs.
CAMs contain good localization information, but make little use of low level features - such as border information. Therefore, to generate the pseudomasks we also used superpixels, generated by the SNIC algorithm. This is a graph-based algorithm which essentially clusters pixels. The localization information contained in the CAM output was used to select superpixels, similar to methods described in [1, 2]. This output is the pseudomask which is used to train the supervised segmentation model.
Currently we are planning on using the pseudomasks to finetune a Segformer. This might still change.