Feature selection keeps a useful subset of the original inputs. Feature extraction transforms inputs into a new representation. Both can reduce complexity, but neither automatically improves a model. The right choice must be tested with a leakage-safe validation process.
Published April 30, 2026 · Reviewed and updated August 27, 2026
This lesson builds on Feature Engineering Explained. You will learn what each approach changes, when it is useful, and how to evaluate it without letting validation or test information influence training. Here, a representation means the numerical form of the data that a model actually receives.
The difference in one example
Suppose a home-price dataset contains 50 columns.
- Feature selection might keep 12 of those original columns, such as floor area, neighborhood, age, and number of bedrooms.
- Feature extraction might transform the 50 columns into 8 new numerical components that summarize patterns across the original data.
The selected features retain their original meaning. Extracted features may be more compact, but they are often harder to explain.
| Question | Feature selection | Feature extraction |
|---|---|---|
| What changes? | Some original features are kept; others are excluded. | Original features are transformed into new features. |
| Interpretability | Usually higher because original column meanings remain. | Often lower because the new representation may be abstract. |
| Common examples | Statistical filters, recursive feature elimination, L1-based selection. | PCA, supervised LDA, autoencoder bottlenecks, learned embeddings. |
| Typical reason to use it | Simplify a model and retain understandable inputs. | Represent high-dimensional or unstructured data more compactly. |
What is feature selection?
Feature selection chooses a subset of the variables already present in a dataset. It does not combine or mathematically transform the selected columns.
Reasons to test feature selection include reducing training cost, simplifying a model, excluding irrelevant, redundant, costly, unstable, unusable, or potentially leaky inputs, and making the final system easier to explain. These are possible benefits—not guarantees. Removing information can also make performance worse.
Filter methods
Filter methods score features without repeatedly training the final predictive model. Examples include variance thresholds, correlation-based checks, chi-square tests, mutual information, and other statistical measures appropriate to the data type. Correlation checks need an explicit rule for redundancy or target association, and common chi-square implementations require suitable non-negative inputs. See the scikit-learn feature-selection guide for method requirements and examples.
They are often computationally efficient, but a one-feature-at-a-time score can miss interactions. A feature with a weak individual relationship may still become useful when combined with another feature.
Wrapper methods
Wrapper methods compare candidate feature subsets by training and evaluating a model. Forward selection, backward elimination, and recursive feature elimination are common examples.
These methods can reflect the behavior of a chosen model, but they may be expensive and can overfit the validation process when too many subsets are tried.
Embedded methods
Embedded methods perform selection as part of model training. L1 regularization can drive some coefficients to zero, producing a sparse model, but the result depends on scaling, regularization strength, correlated predictors, and the fitted model. Tree-based models can provide importance estimates, but an importance score is not automatically a selector: a threshold or selection procedure still has to decide what to retain. These estimates are model-dependent and can be biased or unstable, so they should not be treated as universal proof that a feature matters.
What is feature extraction?
Feature extraction maps the original data into a new representation. The new features may combine information from many original variables or may be learned directly from images, text, or audio.
Extraction often reduces dimensionality, but it does not have to. Its defining property is transformation—not simply having fewer columns.
Principal component analysis (PCA)
Principal component analysis (PCA) creates new, uncorrelated components that capture directions of high variance in the input data. High variance is not the same as high predictive value, so retaining more variance does not guarantee a better model.
Because PCA is sensitive to scale, numerical features are often standardized first when their units differ. The scaler and PCA transformation must both be fitted using training data only.
Linear discriminant analysis (LDA)
Here, LDA means Linear Discriminant Analysis. It is a supervised technique: it uses class labels to create directions that separate classes under its modeling assumptions. That makes it different from unsupervised PCA.
Learned representations
Autoencoders can learn a compressed bottleneck representation. Embedding models convert items such as words, products, or users into numerical vectors. Deep neural networks also learn intermediate representations from images, text, and audio during training.
A manually designed value such as “price per square foot” is usually described as feature engineering. A learned projection such as PCA or an embedding is a clearer example of feature extraction.
The rule that prevents data leakage
Split first. Fit second. Any selector, scaler, PCA transformation, or learned representation that uses information from the dataset must be fitted on training data—not on the full dataset.
- Reserve the test set before exploring feature choices.
- Inside each cross-validation fold, fit preprocessing and the selector or extractor on that fold’s training portion.
- Apply the already-fitted steps to the validation portion without refitting.
- Compare the complete pipeline with an appropriate baseline.
- Use the untouched test set once for the final estimate.
If selection or extraction is fitted before cross-validation, information from validation rows can influence the representation. That can make the reported result look better than the system will perform on new data. Scikit-learn’s data-leakage guidance recommends fitting preprocessing and feature-selection steps only on training data, preferably inside a pipeline.
For the broader evaluation sequence, continue with Training vs Testing Data, Data Preprocessing Explained, and Cross-Validation Explained.
How to choose
Start with feature selection when…
- Stakeholders need to understand the original inputs.
- Your data is tabular and contains clearly unusable, duplicate, or costly variables.
- You want to test whether a smaller input set reduces cost without harming validation performance.
Start with feature extraction when…
- Your inputs are high-dimensional, strongly correlated, or unstructured.
- A compact representation is more important than preserving every original variable’s meaning.
- You are working with images, text, audio, or another domain where learned representations are common.
You may combine both approaches, but there is no universal order. Every added step should earn its place by improving the outcome you care about—such as validation quality, speed, memory use, stability, interpretability, or deployment cost.
A practical comparison experiment
Instead of assuming one method is better, compare several leakage-safe pipelines:
- Baseline: preprocessing plus the model, with no selection or extraction.
- Selection pipeline: preprocessing, feature selection, then the same model.
- Extraction pipeline: preprocessing, feature extraction, then the same model.
Use the same cross-validation splits and evaluation metric for each pipeline. Also compare training time, prediction time, stability across folds, and interpretability. The winner is the pipeline that best satisfies the project’s actual constraints—not simply the one with the most sophisticated transformation.
Common mistakes
- Selecting on all available data: this leaks information into validation or testing.
- Assuming importance means causation: a model association does not prove that changing a feature changes the outcome.
- Using one correlation score as the final answer: interactions and nonlinear relationships may be missed.
- Calling PCA components “the most important information”: PCA prioritizes variance, not necessarily prediction.
- Tuning against the test set: repeated test-set checks turn the test set into another validation set.
Frequently asked questions
Is PCA feature selection or feature extraction?
PCA is feature extraction. Each principal component is a new combination of the original numerical features.
Does feature extraction always reduce dimensionality?
No. It often does, but extraction is defined by transforming the representation. The new representation can be smaller, equal in size, or occasionally larger.
Can feature selection improve accuracy?
It can, particularly when irrelevant or unstable inputs encourage overfitting. It can also reduce accuracy by discarding useful information. Cross-validation is needed to determine the effect.
What is the curse of dimensionality?
As the number of dimensions grows, data can become sparse and distance-based relationships can become less informative. The practical effect depends on the dataset, sample size, representation, and model; a high feature count is not automatically a problem.
Key takeaway
Feature selection keeps original inputs. Feature extraction creates a new representation. Choose between them by testing complete pipelines with leakage-safe cross-validation. Preserve interpretability when it matters, and never assume that fewer or more sophisticated features automatically produce a better model.