Week 5 | New Baseline Results
After adjusting the loss function, we were able to get some promising new results.
Published on October 14, 2023 by Bronte Sihan Li
Wildfire Deep Learning machine learning AI
2 min READ
With the results from last week, it was clear that we were not approaching the problem correctly. For example, through examining the fire mask prediction results from the trained UNet model, we discovered that by using a loss function that was a combination of binary cross entropy and dice loss, the model was not able to predict the fire masks correctly - in fact, the model was not predicting any positive fire pixels at all. This week, we will present results from training the model using only the weighted binary cross entropy loss function.
Identifying the Issue
From the previous week, we saw that the model was not predicting any positive fire pixels. Through some investigation, we hypothesized that this could be due to the loss function being not appropriate for this particular problem. To test this hypothesis, we decided to train the model using only the weighted binary cross entropy loss function, and use the same metrics as the original paper to evaluate the model so that we have a better reference point.
Results
The training results for Unet and the model details are as follows:
| Checkpoint/Description | Dice Score | PR AUC | Precision | Recall | Weighted F1 | Benchmark PR AUC |
|---|---|---|---|---|---|---|
| Epoch 7, BCE loss with pos_weight=3, pos threshold 0.6 | 0.33 | 0.293 | 0.41 | 0.22 | 0.28 | 0.284 |
| Epoch 7, BCE loss with pos_weight=3, pos threshold 0.5 | 0.31 | 0.293 | 0.34 | 0.37 | 0.34 | 0.284 |
| Epoch 7, BCE loss with pos_weight=3, pos threshold 0.4 | 0.29 | 0.293 | 0.28 | 0.50 | 0.35 | 0.284 |
| Epoch 3, BCE loss with pos_weight=100, pos threshold 0.5 | 0.17 | 0.20 | 0.11 | 0.79 | 0.19 | 0.284 |
We can see that by using a more appropriate loss function, not only were we able to predict positive fire pixels, but we were also able to achieve a PR AUC score of 0.293, which is slightly higher than the benchmark PR AUC score of 0.284.
Some example predictions:
| Predicted Fire Mask | True Fire Mask |
|---|---|
 |  |
 |  |
 |  |
 |  |
What’s Next
This is exciting progress as we were able to get a higher PR-AUC than the original methods in the Next Day Wildfire paper. To further improve the results and extract additional potential from the UNet architecture, we will try the following:
- Add batch norm or layer norm
- Add more features as input
- Try different loss functions
- Increase width of Unet by increasing number of kernels in each block
- Increase depth of Unet by adding more pairs of encoder/decoder blocks
- Try augmenting the Unet with this paper
- Try different architectures
Photo by Riccardo Annandale on Unsplash