In the final set of the SimCLR-S2 process we perform knowledge distillation to attempt to gain similar performance to the big ResNet152 model with fewer parameters. While it is possible to use some labeled data during distillation we experimented with a completely self-supervised approach.

We leverage a similar approach as the baseline models to fine tune our pretrained contrastive learning model. We train multipe data splits on our largest CNN architecture ResNet152, to maximize performance of the fine-tuned model.

The SimCLR-S2 pretraining uses completely unlabeled data and as the starting point to train out computer vision model. In addition to the Big Earth Net data, we will run pretraining against sentinel-2 images captured of the California central valley.

The goal of this project is to identify if self-supervised learning is a viable option when training on satellite imagery with limited access to labeled data. Our baseline models are trained on a variety of neural network architectures and fractions of labeled data to quantify the performance impact as our fraction of labeled data changes. We will used the results of our baseline models to assess the performance of our self-supervised models with the constraints.