Overfitting and underfitting are two different failures of generalization. An underfit model has not learned enough useful structure to perform well, even on its training data. An overfit model performs very well on training examples but substantially worse on relevant unseen data.
The goal is not perfect training performance. It is reliable performance on new cases that resemble the problem the model is expected to solve.
Overfitting vs Underfitting
On a phone: swipe horizontally to view every column.
| Signal | Underfitting | Useful fit | Overfitting |
|---|---|---|---|
| Training performance | Poor | Good enough for the task | Very strong |
| Validation performance | Poor | Close enough to training and operationally useful | Meaningfully weaker than training |
| Generalization gap | Often small because both results are poor | Controlled and interpreted in context | Often large or increasing |
| Common cause | Insufficient capacity, weak features, excessive regularization, or inadequate optimization | Capacity and controls match the data and task | Excess capacity relative to useful data, noise, leakage, or excessive tuning |
| Possible response | Improve features, optimization, or model capacity | Validate, stress-test, and monitor | Improve data, regularize, simplify, or stop training earlier |
A small generalization gap does not prove a model is good. A model with 55% training accuracy and 54% validation accuracy may be consistently poor rather than well generalized. Always consider absolute performance and whether the metric matches the real decision.
Worked Example: How the Pattern Changes During Training
Early training
Training accuracy: 62%
Validation accuracy: 61%
Both results are poor and close together. The model currently underfits.
Middle stage
Training accuracy: 86%
Validation accuracy: 83%
Both results are useful and the gap is controlled. This may be a reasonable checkpoint.
Later training
Training accuracy: 97%
Validation accuracy: 77%
Training improves while validation deteriorates. This is evidence of overfitting.
The exact percentages are illustrative. Real diagnosis depends on the metric, sample size, validation design, task difficulty, and cost of errors. For iterative models, early stopping can preserve the middle-stage checkpoint instead of the final training state.
A Practical Diagnostic Process
- Confirm the split is valid. Keep related records, future information, and preprocessing leakage from crossing between training and validation data. Read Training vs Testing Data.
- Choose a task-relevant metric. Accuracy may be misleading for imbalanced data. Compare results using the same metric and threshold. See Model Evaluation Metrics Explained.
- Compare absolute performance and the gap. Poor results on both sets suggest underfitting; a strong training result with a weaker validation result suggests overfitting.
- Inspect learning or validation curves. Track performance as training time, model complexity, or data volume changes.
- Check alternative explanations. Look for label errors, duplicate examples, unrepresentative validation data, leakage, or distribution shift.
- Change one important factor and reevaluate. Diagnose with evidence rather than applying every remedy at once.
What Causes These Patterns?
Common causes of overfitting
- Capacity relative to data: the model can represent noise and accidental correlations in a limited sample.
- Noisy or biased data: incorrect labels, measurement errors, duplicates, and sampling problems encourage unreliable patterns.
- Leakage: information unavailable at prediction time makes evaluation look artificially strong.
- Validation overuse: repeated decisions based on one validation set can overfit the development process to that sample.
- Excessive training: iterative optimization continues improving the training objective after validation performance has peaked.
A large or highly parameterized model is not automatically overfit. Complexity is meaningful only relative to the data, objective, regularization, pretraining, and evaluation design.
Common causes of underfitting
- Insufficient model capacity: the model cannot represent the relationship needed for the task.
- Weak inputs: features or representations do not contain enough useful signal.
- Excessive regularization: constraints prevent the model from learning important structure.
- Insufficient optimization: the model needs better settings, more iterations, or a more suitable optimization process.
- Task mismatch: the algorithm, objective, or feature representation does not fit the problem.
How to Respond to the Evidence
When the model appears overfit
- Collect more relevant and representative examples when limited coverage is the problem.
- Correct labels, duplicates, leakage, and sampling problems.
- Reduce unnecessary features or model complexity.
- Use suitable regularization, such as L1/L2 penalties, weight decay, dropout, or architecture-specific constraints.
- Use early stopping for iterative training.
- Use more reliable resampling or validation when one split is unstable.
- Improve preprocessing and representations without leaking validation information.
When the model appears underfit
- Use a more expressive model when evidence shows insufficient capacity.
- Improve features or representations.
- Reduce excessive regularization.
- Train longer or improve optimization.
- Reconsider whether the algorithm and objective match the task.
How Bias and Variance Relate
High bias is often associated with models that are too constrained to capture useful structure. High variance is often associated with models that are too sensitive to the training sample. This provides a useful conceptual lens, but it is not a universal diagnostic formula for every modern system.
Continue to Bias vs Variance Tradeoff for the deeper explanation.
Overfitting in Deep Learning
Deep neural networks can have very high capacity, but parameter count alone does not determine generalization. Large-scale pretraining, data augmentation, transfer learning, weight decay, dropout, early stopping, architecture choices, and validation-based model selection all affect the outcome.
The same diagnostic principle still applies: compare performance on appropriate unseen data, inspect the gap and absolute results, and rule out leakage or shift before selecting a remedy. See What Is Deep Learning?
What Training and Validation Results Cannot Prove
A convincing validation result does not guarantee future production performance. The validation population may differ from real users, conditions may change, subgroup performance may vary, and the evaluation sample may be too small to estimate performance precisely.
Use an untouched test set when appropriate, report uncertainty, test important subgroups and edge cases, and monitor after deployment. Offline validation and production monitoring answer different questions.
Sources and Further Reading
- Google Machine Learning Crash Course: Overfitting
- Google Machine Learning Crash Course: L2 regularization
- scikit-learn User Guide: Validation and learning curves