Overfitting and underfitting are two different failures of generalization. An underfit model has not learned enough useful structure to perform well, even on its training data. An overfit model performs very well on training examples but substantially worse on relevant unseen data.
The goal is not perfect training performance. It is reliable performance on new cases that resemble the problem the model is expected to solve.
Overfitting vs Underfitting
On a phone: swipe horizontally to view every column.
| Signal | Underfitting | Useful fit | Overfitting |
|---|---|---|---|
| Training performance | Poor | Good enough for the task | Very strong |
| Validation performance | Poor | Close enough to training and operationally useful | Meaningfully weaker than training |
| Generalization gap | Often small because both results are poor | Controlled and interpreted in context | Often large or increasing |
| Common cause | Insufficient capacity, weak features, excessive regularization, or inadequate optimization | Capacity and controls match the data and task | Excess effective capacity relative to useful data, weak regularization, excessive training, or repeated model tailoring |
| Possible response | Improve features, optimization, or model capacity | Validate, stress-test, and monitor | Improve data, regularize, simplify, or stop training earlier |
A small generalization gap does not prove a model is good. A model with 55% training accuracy and 54% validation accuracy may be consistently poor rather than well generalized. Always consider absolute performance and whether the metric matches the real decision.
Worked Example: How the Pattern Changes During Training
Early training
Training accuracy: 62%
Validation accuracy: 61%
Both results are poor and close together. The model currently underfits.
Middle stage
Training accuracy: 86%
Validation accuracy: 83%
Both results are useful and the gap is controlled. This may be a reasonable checkpoint.
Later training
Training accuracy: 97%
Validation accuracy: 77%
Training improves while validation deteriorates. This is evidence of overfitting.
The exact percentages are illustrative. Real diagnosis depends on the metric, sample size, validation design, task difficulty, and cost of errors. For iterative models, early stopping can preserve the middle-stage checkpoint instead of the final training state.
Reading Training and Validation Curves
Curves make the direction of learning visible. Plot the same task-relevant score for training and validation data as training time or model complexity changes. Higher is assumed to be better in the patterns below; loss curves reverse the vertical interpretation.
Underfitting
Training and validation performance both remain poor and often close together. The model has not captured enough useful structure.
Useful fit
Validation reaches a task-useful level without an uncontrolled gap. The best checkpoint may occur before training ends.
Overfitting
Training keeps improving while validation plateaus or worsens. The growing gap is evidence that generalization is deteriorating.
Exact shapes are not universal. Curves support a diagnosis; they do not replace checks for split quality, sample uncertainty, metric choice, or distribution mismatch. See the scikit-learn guide to validation and learning curves for practical examples.
A Practical Diagnostic Process
- Confirm the split is valid. Keep related records, future information, and preprocessing leakage from crossing between training and validation data. Read Training vs Testing Data.
- Choose a task-relevant metric. Accuracy may be misleading for imbalanced data. Compare results using the same metric and threshold. See Model Evaluation Metrics Explained.
- Compare absolute performance and the gap. Poor results on both sets suggest underfitting; a strong training result with a weaker validation result suggests overfitting.
- Inspect learning or validation curves. Track performance as training time, model complexity, or data volume changes.
- Check alternative explanations. Look for label errors, duplicate examples, unrepresentative validation data, leakage, or distribution shift.
- Change one important factor and reevaluate. Diagnose with evidence rather than applying every remedy at once.
What Causes These Patterns?
Factors that can drive genuine overfitting
- Capacity relative to useful data: the model can represent noise and accidental correlations in a limited sample.
- Noisy observations: the learner adapts to label errors, measurement error, or incidental sample details that do not repeat.
- Weak regularization: the learning process lacks appropriate constraints for the data and task.
- Excessive training: iterative optimization continues improving the training objective after validation performance has peaked.
- Repeated tailoring: features, hyperparameters, or model choices become too adapted to the same development data.
Problems that can mimic, hide, or complicate overfitting
- Leakage: information unavailable at prediction time can make evaluation artificially strong and may hide a generalization problem.
- Related records across splits: duplicates or near-duplicates make validation less independent than it appears.
- Unrepresentative validation data: a distribution mismatch can create a gap that model fit alone does not explain.
- Small evaluation samples: unstable estimates can exaggerate or conceal a performance difference.
- Validation-set overuse: repeated decisions against one validation set can overfit the development process to that sample.
A large or highly parameterized model is not automatically overfit. Complexity is meaningful only relative to the data, objective, regularization, pretraining, and evaluation design.
Common causes of underfitting
- Insufficient model capacity: the model cannot represent the relationship needed for the task.
- Weak inputs: features or representations do not contain enough useful signal.
- Excessive regularization: constraints prevent the model from learning important structure.
- Insufficient optimization: the model needs better settings, more iterations, or a more suitable optimization process.
- Task mismatch: the algorithm, objective, or feature representation does not fit the problem.
How to Respond to the Evidence
When the model appears overfit
- Collect more relevant and representative examples when limited coverage is the problem.
- Correct labels, duplicates, leakage, and sampling problems.
- Reduce unnecessary features or model complexity.
- Use suitable regularization—intentional constraints or learning modifications intended to improve generalization—such as L1/L2 penalties, weight decay, dropout, or architecture-specific controls. See Google’s regularization overview.
- Use early stopping when validation performance stops improving during iterative training.
- Use more reliable resampling or validation when one split is unstable.
- Improve preprocessing and representations without leaking validation information.
When the model appears underfit
- Use a more expressive model when evidence shows insufficient capacity.
- Improve features or representations.
- Reduce excessive regularization.
- Train longer or improve optimization.
- Reconsider whether the algorithm and objective match the task.
How Bias and Variance Relate
High bias is often associated with models that are too constrained to capture useful structure. High variance is often associated with models that are too sensitive to the training sample. This provides a useful conceptual lens, but it is not a universal diagnostic formula for every modern system.
Continue to Bias vs Variance Tradeoff for the deeper explanation.
Overfitting in Deep Learning
Deep neural networks can have very high capacity, but parameter count alone does not determine generalization. Large-scale pretraining, data augmentation, transfer learning, weight decay, dropout, early stopping, architecture choices, and validation-based model selection all affect the outcome.
The same diagnostic principle still applies: compare performance on appropriate unseen data, inspect the gap and absolute results, and rule out leakage or shift before selecting a remedy. See What Is Deep Learning?
What Training and Validation Results Cannot Prove
A convincing validation result does not guarantee future production performance. The validation population may differ from real users, conditions may change, subgroup performance may vary, and the evaluation sample may be too small to estimate performance precisely.
Use an untouched test set when appropriate, report uncertainty, test important subgroups and edge cases, and monitor after deployment. Offline validation and production monitoring answer different questions.
Sources and Further Reading
- Google Machine Learning Crash Course: Overfitting
- Google Machine Learning Crash Course: L2 regularization
- scikit-learn User Guide: Validation and learning curves