This model was developed to explore the applicability of CGANs for cloud cover nowcasting. The idea is based on Deepmind's success with precipitation nowcasting using a CGAN, their DGMR. The model takes as input EUMETSAT's cloud cover images, which are derived from their satellite images. These images thus already have cloud cover masks.
In a GAN, learning happens through two neural networks that are in competition (adversarial): the generator and discriminator. The generator must learn to generate 'realistic' images. The discriminator must learn to distinguish generated images from real images. Thus, they teach each other. In a vanilla GAN, the goal is not to produce any specific image. However in our case, we wanted to use GANs for prediction. Therefore, we used a CGAN.
In a conditional GAN, the generator receives data on which to base the image it generates. During training, our satellite images were grouped into sequences of 5 consequtive images (5x15 minutes). The generator then received the first 4 images, with the task of generating the 5th image. The discriminator then received the original sequences of 5 images, as well as the sequence generated by the generator (4 original images, with the generated image concatenated on). As such, the CGAN can be used for the task of image prediction.
This version of the CGAN is somewhat unconventional, as it has 1 generator and two discriminators: the spatial and temporal discriminators (based on Deepmind's DGMR). The spatial discriminator takes as input only 1 image, meaning that it gives losses that correspond to 'how real does this single image look'? The temporal discriminator on the other hand takes as input satellite image sequences, such that losses correspond to 'how real does this sequence look?'.
Additionally, there is a conditioning stack, which entails that context images are fed to the generator at multiple resolutions throughout the upsampling process.
In this version with the conditioning stack, it appeared that the generator was quite strong and the discriminator needed to become stronger. Furthermore, in later epochs the images became blurry, so we wanted to implement some losses that would improve learning. Therefore, the binary crossentropy loss that was previously used was exchanged for the hinge loss, which has the possibility of having a higher accuracy than a logistic based loss by being less prone to outliers. So, the generator and discriminator both learn from hinge loss. Additionally, the generator receives a grid cell regularizing term, which is added to the loss before these are backpropagated.