August 03, 2019 · 1 mins read

Lesson 2 part 3 - Common Issues with Neural Networks

This post outlines 4 common issues that cause your neural network to perform poorly along with symptoms and explanations for the behaviour.

This is the last part of a three part series on lesson 2. Here’s the first and second part.

The learning rate is too high

Symptoms: there’s a high validation error.

A neural network approaches optimal parameters during training by taking steps in the right direction every epoch. The learning rate determines the size of each step. When this learning rate is too high, you might miss the best parameters and end up with worse parameters instead.

The learning rate is too low

Symptoms: the error is decreasing, but very slowly.

Similar to the above explanation, having a learning rate that is too low results in taking very small steps toward the optimal solution.

Too few epochs

Remember: an epoch is looking at each of the examples in the dataset once.

Symptoms: there is a high difference between the training error and validation error.

When you train too little, your model doesn’t get a chance to learn about the data.

Too many epochs

Symptoms: training error is much lower than the validation error.

When you train for too long, the neural network starts to learn what the examples in the dataset look like, but not what the data looks like in general. This is called overfitting and it is a huge problem in machine learning. Fortunately, neural networks are less sensitive to overfitting than other machine learning techniques.


Hopefully, you’ll be able to recognize these problems in your own datasets and make adjustments accordingly to get better results.

A huge thanks to Sam Miserendino for proofreading this post!