What Is Ensemble Learning?
Ensemble learning is a model-combination strategy, not a separate machine learning paradigm. Supervised, unsupervised, and reinforcement learning describe different learning settings. Ensemble methods describe how predictions from two or more models are combined within a suitable setting.
The individual models are called base learners. They may be repeated versions of the same algorithm, such as the decision trees in a Random Forest, or different algorithms trained to solve the same prediction task.
An ensemble can outperform a single model when its members are individually useful and make non-identical errors. It does not automatically improve accuracy, reduce overfitting, or generalize better. Those outcomes must be demonstrated on validation and test data.
If you are new to this area, first review Machine Learning Algorithms Overview and Supervised Learning Explained.
Why Combine Models?
Models can fail in different ways. A shallow decision tree may miss important patterns, while a deep tree may fit noise in its training sample. Combining models can sometimes make the final prediction less dependent on one model’s particular errors.
Diversity matters
Diversity does not simply mean using algorithms with different names. Useful diversity means competent models make errors that are not perfectly correlated. Ensembles can create this variation by changing training samples, features, model settings, algorithms, or the order in which models are fitted. If every member repeats the same mistake, combining them offers little benefit.
Bias and variance
| Problem | What it means | How ensembles may help |
|---|---|---|
| High bias | The model is too rigid or makes overly simple assumptions, so it underfits. | Boosting can build a more expressive predictor by adding learners sequentially, although it is not guaranteed to solve underfitting. |
| High variance | The model changes too much with the training sample and may fit noise, so it overfits. | Bagging can reduce variance by averaging unstable learners such as deep decision trees. |
Learn the underlying concepts in Bias vs Variance and Overfitting vs Underfitting.
Three Ensemble Structures
There is no universal step-by-step ensemble workflow. The training relationship between models depends on the method.
1. Parallel aggregation: bagging
Bagging, short for bootstrap aggregating, creates multiple training sets by sampling the original training data with replacement. Each bootstrap sample can contain repeated rows and omit others. A base learner is fitted independently to each sample, so the models can usually be trained in parallel. Their outputs are then averaged for regression or voted on for classification.
Bagging is mainly used to reduce the variance of an unstable learner. Its effect depends on the base model, the data, and how correlated the fitted models remain.
Random Forest: the primary example
Random Forest is the clearest concrete example of a bagging-style ensemble. It combines many decision trees trained with bootstrap samples and additional feature randomness. Classification typically uses a vote; regression typically averages tree predictions.
The feature-selection mechanics, out-of-bag evaluation, and tuning details belong in the dedicated Random Forest guide. Here, the important relationship is simple: a decision tree is a base learner, while Random Forest combines many varied trees.
2. Sequential addition: boosting
Boosting builds an additive model sequentially. Later learners are fitted in response to the current ensemble, but different boosting algorithms define that response differently.
- AdaBoost increases the influence of training examples that earlier learners handled poorly and combines learners using performance-based weights.
- Gradient boosting fits each new learner to a loss-based correction—often described as the negative gradient or, for squared-error regression, residual-like errors. XGBoost and related libraries are optimized gradient-boosting implementations.
Boosting does not deterministically “improve over time.” Additional rounds can help, do little, or overfit, depending on the data, loss, learning rate, tree complexity, regularization, and stopping rule. Its performance must be monitored on validation data.
Bagging vs Boosting
| Feature | Bagging | Boosting |
|---|---|---|
| Training relationship | Base learners are trained independently and can usually run in parallel. | Base learners are added sequentially in response to the current ensemble. |
| Data or error handling | Uses bootstrap samples drawn with replacement. | AdaBoost reweights examples; gradient boosting fits loss-based corrections. |
| Typical strength | Can reduce variance and improve stability. | Can reduce predictive error by building a flexible additive model. |
| Main risk | Added computation and limited benefit when learners are too correlated. | Sensitivity to noise, settings, and overfitting without validation or regularization. |
| Primary example | Random Forest | AdaBoost or gradient-boosted trees |
3. Learned combination: stacking
Stacking trains a second-level model, called a meta-model, to combine predictions from multiple base learners. The base learners may be different algorithms, such as a tree model, logistic regression, and a support vector machine.
The meta-model must be trained with out-of-fold predictions, not predictions made by base models on the same rows they were fitted on. A common leakage-safe process is:
- Split the training data into cross-validation folds.
- For each fold, fit every base learner on the other folds and predict the held-out fold.
- Join those held-out predictions so every training row has predictions from models that did not train on it.
- Train the meta-model on those out-of-fold predictions.
- Refit the base learners on the full training set, then use their predictions as inputs to the trained meta-model for new data.
Using in-sample base-model predictions can leak training information into the meta-model and produce an unrealistically optimistic result. See Training, Validation, and Test Sets for the evaluation foundation.
Voting and Averaging
Voting is a simple parallel ensemble for classification. Hard voting chooses the class predicted by the most models. Soft voting averages or weights predicted class probabilities and selects the class with the highest combined probability.
| Method | Combines | Important condition |
|---|---|---|
| Hard voting | Predicted class labels | Each model contributes a vote unless explicit weights are used. |
| Soft voting | Predicted class probabilities | Probabilities should be reasonably calibrated and refer to the same class order. |
| Regression averaging | Numeric predictions | Simple or validation-chosen weights may be used. |
Soft voting is not automatically better than hard voting. Poorly calibrated probabilities or an overconfident weak model can make it worse.
When an Ensemble Is Worth Using
Consider an ensemble when a well-tuned single model leaves meaningful, repeatable validation error and the added complexity is justified. Ensembles can be useful for tabular classification or regression, unstable tree models, and settings where several complementary models already perform competently.
A simpler model may be the better choice when interpretability, latency, memory, energy use, maintenance, or debugging matters more than a small score increase. Domain claims such as “banks use ensembles” or “ensembles improve diagnoses” are too broad without a specific source and context; the method alone does not establish safety or suitability.
Validate Against a Simpler Baseline
Do not assume that more models mean a better system. Test the ensemble against a credible single-model baseline under the same data split, preprocessing, metric, and tuning budget.
- Choose a metric that matches the real cost of errors.
- Keep a final test set untouched while selecting models.
- Use cross-validation or a validation set for tuning and ensemble weights.
- Compare the ensemble with a simple baseline and the strongest individual member.
- Check performance across important subgroups, time periods, or operating thresholds—not only one average score.
- Measure inference time, memory, model size, calibration, and maintenance cost.
- Evaluate once on the untouched test set after decisions are complete.
If the improvement is small or unstable, keep the simpler model. See Model Evaluation for guidance on choosing and interpreting metrics.
Taxonomy Clarification
Ensemble learning should not be positioned as a peer alternative to deep learning or reinforcement learning. Deep learning describes models built from multi-layer neural networks. Reinforcement learning describes a learning setting in which an agent learns from interaction and rewards. Ensemble strategies can, in principle, combine deep models or models used within a reinforcement-learning system. These concepts answer different questions and can overlap.
Advantages and Limitations
Potential advantages
- Reduced variance through averaging
- Flexible nonlinear predictions through boosting
- Use of complementary model strengths
- Improved robustness when errors are sufficiently diverse
Costs and risks
- More training and inference computation
- Harder explanation and debugging
- More complex deployment and monitoring
- Leakage risk in stacking
- No guarantee of improvement
Frequently Asked Questions
Is Random Forest an ensemble method?
Yes. It combines many decision trees using bootstrap sampling, feature randomness, and voting or averaging.
Does ensemble learning always improve accuracy?
No. The members must be useful and sufficiently complementary, and the comparison must be made on unseen data. Correlated errors, leakage, overfitting, or poor tuning can remove the benefit.
What is the difference between bagging and boosting?
Bagging trains learners independently on bootstrap samples and aggregates them. Boosting adds learners sequentially in response to the current ensemble.
Can neural networks be ensembled?
Yes. Predictions from multiple neural networks can be averaged, voted, or combined by a learned model. Deep learning and ensemble learning are overlapping concepts, not competing taxonomy categories.
Next Step: Learn Random Forest
Random Forest is the best next example because it makes bagging, base learners, diversity, voting, and averaging concrete without requiring advanced mathematics.
Then revisit Decision Trees, Training, Validation, and Test Sets, and Model Evaluation to connect the method to the full learning workflow.