Week 7 | What We Have Found So Far

Useful Findings and Insights So Far

After another week of exploring different ways to improve the performance of our model, we have found some useful insights and findings.

• So far, we have been able to achieve a PR-AUC of 0.31 by using UNet with 5 input features: precipitation, specific humidity, wind direction, NDVI, elevation, and previous day fire mask.
• While the models are able to converge very fast with RMSprop optimizer, we achieve better overall validation and test results with ADAM on all models. However, ADAM takes up more memory which may be a limiting factor in our training.
• Interestingly, by resetting the learning rate and optimizer every 6-7 epochs, we were able to continuously improve the validation dice and AUC scores with more epochs trained.
• Despite the demonstrated advantages in using Focal Tversky loss function in studies performing similar prediction on wildfire spread[1], we implemented this loss function and conducted experiments, but found limited success in training with it - further investigation may be required.
• Due to the size of Unet, CUDA out of memory errors are still a challenge given limited computing resources.

Other Improvements

While in the search for optimal parameters including batch size, optimizer to use, number of features, and loss function, we have also made continuous improvements on the project source library:

• Better logging: additional logging on useful parameters on wandb to allow for easier comparisons
• Improved CUDA memory knowledge: conducted additional research on different ways to resolve CUDA out of memory errors and memory optimization with lower precision, offloading tensors from GPU, etc.
• Improvements to allow for easier hyperparameter tuning and continued training by customizing training functions and checkpoint-ing

Thinking About Performance in Terms of Utility and Human Centered AI

Taking inspiration from the flood forecasting tool set from Google, we are thinking about how to evaluate the performance of our model in terms of utility. In the case of the flood forecasting tool set, the utility is defined as the number of people that are able to take action to protect themselves and their property. In our case, we are thinking about how to optimize our model in terms of the number of lives and properties that can benefit from the spread forecasting and what would be the most useful parameters to optimize for. We want to think about 1.the live public datasets available that can be used to perform real-time inference, 2. minimizing false negatives in the predictions, and 3. distance from fire. We will continue to explore this idea and think about how to better formulate the problem under the context of utility.

Looking ahead, we will continue to explore the potential of the UNet architecture by adding more layers or width, though it is worth mentioning that computational resources are limited and we will need to be careful given the large dataset size and the significant increase in the number of parameters in the model. We will also continue to explore other more advanced networks including attention UNet with residual connections as well as the potential of using other attention based architectures such as vision transformers. In the meantime, it may worth considering the real-world application of this model in context of developing a wildfire prediction tool for predicting the spread of wildfires in real-time using Google earth data, for example, using live wildfire boundary maps described in this blog[2].

References

1. Zou, Yufei et al. “Attention-Based Wildland Fire Spread Modeling Using Fire-Tracking Satellite Observations.” Fire (2023): n. pag.
2. https://blog.research.google/2023/02/real-time-tracking-of-wildfire.html

Photo by Martin Sanchez on Unsplash