Bias and variance tradeoff
This very nice post clearly explains the bias and variance tradeoff
http://www.learnopencv.com/bias-variance-tradeoff-in-machine-learning/
This tradeoff can be mathematically stated by the following equations
Another interesting post on the same topic can be found here
http://scott.fortmann-roe.com/docs/BiasVariance.html
The first aforementioned post nicely summarizes some of the main recommendations which Andrew NG gave in many of his talks
https://www.youtube.com/watch?v=F1ka6a13S9I