- Background
- What is Overfitting?
- Lasso (L1) and Ridge (L2) Regularisation
- Early Stopping
- Dropout
- Other Methods
- Summary
So far in this neural networks 101 series we have discussed two ways to improve the performance of neural networks: hyperparameter tuning and faster gradient descent optimisers. You can check those posts below:
There is one other set of techniques that aid in performance and that is regularisation. This helps prevent the model from overfitting to the training dataset to have more accurate and consistent predictions.
In this article, we will cover a wide range of methods to regularise your neural network and how you can do it in PyTorch!
Let’s quickly recap of what we mean by overfitting in machine learning and statistics.
Wikipedia describes overfitting as:
“The production of an analysis that corresponds too closely or exactly to a particular set of data, and may therefore fail to fit to additional data or predict future observations reliably”
In layman’s terms, this is saying that the model is learning the data it is training on, but is failing to generalise. Therefore, it will have poor predictions…