Random Forest is a supervised machine learning algorithm that combines many randomized decision trees. Its central idea is simple: build trees that make somewhat different errors, then combine their predictions so that tree-specific mistakes can cancel out.
If you have not met a decision tree yet, start with Decision Trees Explained. Random Forest adds two sources of randomness to that foundation: rows sampled with replacement for each tree and random candidate features considered at each split.
What Is Random Forest?
A Random Forest is an ensemble of decision trees used for classification or regression. Each tree is trained separately on a randomized version of the training data. The forest then aggregates the trees’ predictions.
For classification, implementations may combine hard class votes or average class probabilities. For example, scikit-learn’s classifier averages the class probabilities produced by its trees. For regression, forests generally average numerical predictions. The exact behavior belongs to the implementation, so “majority vote” is a useful intuition, not a universal rule.
Why One Decision Tree Can Be Unstable
A deep decision tree can fit complicated patterns, but it can also have high variance: a modest change in the training data may produce a different tree and different predictions. Random Forest does not simply copy one tree many times. It deliberately diversifies the trees so their errors are less correlated, then averages across them.
In the original Random Forests paper by Leo Breiman (2001), forest performance is connected to both the strength of individual trees and the correlation between them. The trees are not guaranteed to be statistically independent; “diversified” or “less correlated” is more precise.
The Two Sources of Randomness
1. Random rows for each tree
Each tree usually receives a bootstrap sample: observations drawn from the training set with replacement. One observation may appear more than once while another may be left out.
2. Random features at each split
When a tree chooses a split, it considers only a randomly selected set of candidate features. A new subset can be considered at the next split. This discourages every tree from repeatedly following the same dominant predictors.
The scikit-learn ensemble guide describes these two mechanisms—bootstrap samples and random feature selection at each split—as ways to reduce an estimator’s variance by decoupling tree errors.
How Random Forest Works, Step by Step
Training data → bootstrap samples → randomized decision trees → aggregate predictions
- Start with labeled training data.
- Draw a bootstrap sample with replacement for each tree.
- Grow a decision tree. At each split, evaluate only a random subset of candidate features.
- Repeat to create many different trees.
- Combine their outputs: classification through voting or probability aggregation, depending on the implementation; regression generally through averaging.
Worked Example: Predicting Subscription Renewal
Suppose a subscription service wants to predict whether a customer will renew. Its training table contains months subscribed, product usage, support interactions, plan type, and the known outcome: renewed or did not renew.
- Tree 1 trains on one bootstrap sample and may split first on product usage.
- Tree 2 sees a different sample and may split first on months subscribed.
- Tree 3 may rely more on support interactions and plan type.
The forest combines the trees’ outputs to predict renewal. This example illustrates the mechanism, not a guarantee of business performance. Real deployment still requires representative data, suitable metrics, leakage checks, monitoring, and attention to fairness and changing customer behavior.
Bagging and Out-of-Bag Evaluation
Bagging means bootstrap aggregating: create bootstrap samples, train models separately, and aggregate their predictions. Random Forest builds on bagged trees by also randomizing the candidate features at each split.
Because sampling is done with replacement, some training observations are absent from a particular tree’s sample. These are that tree’s out-of-bag (OOB) observations. An observation can be evaluated using predictions from trees that did not train on it, providing an internal performance estimate. Scikit-learn documents this process in its OOB error example.
OOB evaluation can be useful, but it does not make the final test set optional. Keep untouched test data for the final assessment and use appropriate validation or cross-validation when the problem calls for it. See Training vs Testing Data and Model Evaluation Metrics Explained.
Why Aggregation Can Reduce Variance
If several trees make different errors, averaging can soften unusually high or low predictions, and classification aggregation can prevent one tree from deciding the result. This often makes the forest more stable than one unconstrained tree. The benefit depends on having reasonably useful trees whose errors are not too strongly correlated.
Random Forest therefore often reduces variance and overfitting risk relative to a single high-variance decision tree. It does not eliminate leakage, mislabeled data, class imbalance, distribution shift, poor evaluation, or inappropriate settings. Learn more in Overfitting vs Underfitting Explained.
Key Random Forest Hyperparameters
| Control | What it changes | Trade-off |
|---|---|---|
| Number of trees | How many trees contribute | More trees can stabilize results until gains plateau, but increase training, prediction, and memory costs. |
| Features per split | How many candidate features each split can consider | Fewer candidates can diversify trees; too few may weaken them. |
| Maximum depth | How complex each tree can become | Deeper trees capture more detail but cost more and can fit noise. |
| Minimum leaf size | How many observations must remain in a leaf | Larger leaves smooth predictions; smaller leaves allow finer patterns. |
| Bootstrap sampling | Whether trees train on resampled observations | Needed for the classic bootstrap/OOB workflow; alternatives are implementation-specific. |
Names and defaults vary by library. In scikit-learn, examples include n_estimators, max_features, max_depth, min_samples_leaf, and bootstrap. Tune settings using validation evidence rather than assuming that a larger forest is always better.
Feature Importance—and Its Limits
A fitted forest can help inspect which features it used for prediction, but “importance” depends on both the fitted model and the measurement method.
- Impurity-based importance summarizes how much a feature reduced the tree-splitting criterion. It is fast and commonly available, but can favor high-cardinality features and can reflect training-set overfitting.
- Permutation importance shuffles one feature and measures the resulting change in model performance. It can be computed on unseen data, although correlated features and the chosen metric still affect interpretation.
The scikit-learn permutation-importance guide explains these limitations and warns that importance is specific to a particular model. Feature importance is evidence of predictive usefulness to that model—not proof of causation, fairness, business significance, or real-world impact.
When Random Forest Is a Good Fit
Random Forest is often a useful baseline for tabular classification and regression. It can model nonlinear relationships and feature interactions with less manual specification than a linear model. Google’s TensorFlow Decision Forests materials position decision forests as tools for structured and tabular problems and provide implementation-focused learning resources.
Random Forest methods have been applied in business analytics, security, healthcare, finance, science, and other domains. Those are possible application areas, not automatic endorsements. High-stakes uses require domain expertise, suitable governance, careful validation, and human oversight.
Limitations and Practical Cautions
- A forest is harder to inspect as one simple decision path than a single tree.
- Many deep trees increase model size, training time, prediction time, and memory use.
- Random Forest is not automatically best for every tabular dataset; compare it with suitable baselines.
- Probability quality, class imbalance, and decision thresholds still require evaluation.
- Missing-value handling is implementation-specific. Some modern libraries support missing values natively; others require imputation or different preprocessing. Check the exact version and estimator you use.
Random Forest vs Decision Tree
| Question | Decision Tree | Random Forest |
|---|---|---|
| Model structure | One tree | Many randomized trees |
| Prediction stability | A deep tree can have high variance | Aggregation often reduces variance |
| Interpretation | One path can be inspected directly | The ensemble is harder to summarize as one path |
| Compute and memory | Usually lower | Usually higher |
| Performance | Depends on the task and settings | Often improves stability and predictive performance, but not universally |
Random Forest vs Gradient Boosting
Random Forest diversifies trees and then aggregates them. Its trees can be trained separately. Gradient boosting builds trees sequentially, with each new tree contributing to improvement of the current ensemble according to the optimization objective. Neither is universally more accurate; results depend on the dataset, tuning, metric, constraints, and evaluation design.
How to Evaluate a Random Forest
Choose metrics that match the real decision. Classification may require precision, recall, F1, ROC AUC, calibration, or cost-aware evaluation—not accuracy alone. Regression may require MAE, RMSE, or another error measure. Compare against a simple baseline, validate hyperparameters without touching the test set, check important subgroups, and monitor performance after deployment.
Random Forest and Ensemble Learning
Random Forest is one specific ensemble method. This guide teaches bagging only as far as needed to understand the algorithm. For the wider family—including bagging, boosting, voting, and stacking—use Ensemble Learning Explained.
Frequently Asked Questions
What is Random Forest in simple terms?
It is a group of randomized decision trees whose predictions are combined to produce one result.
Why sample with replacement?
Sampling with replacement creates different training samples for different trees. Some observations repeat and others are left out, helping diversify the trees and enabling OOB evaluation.
Does Random Forest prevent overfitting?
No. It often reduces variance and overfitting risk compared with one deep tree, but it can still fail because of leakage, poor data, distribution shift, unsuitable hyperparameters, or weak evaluation.
Can Random Forest predict numbers?
Yes. Random Forest regression generally averages numerical predictions from its trees.
Does Random Forest handle missing values?
That depends on the implementation and version. Some support missing values natively; others require imputation or another preprocessing strategy.
Conclusion
Random Forest extends the decision-tree idea with bootstrap samples, random candidate features at each split, and aggregated predictions. Those choices aim to create useful trees with less-correlated errors, allowing aggregation to reduce variance. It is a strong baseline for many tabular prediction problems, but it still requires disciplined evaluation and careful interpretation.
Continue the algorithm sequence
Next lesson: Support Vector Machines Explained
Secondary deep-dive: How Ensemble Learning Works