Dictionary of Applied Machine Learning: 107 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

feature importance

Updated on 2026-09-14

Readiness 98/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Cite this entry

See also feature loss

An importance score assigns a number to each feature of a learned hypothesis. With standardized features, a linear model's coefficient magnitude equals prediction change per standard deviation. Permutation importance is the average-loss increase when a feature is permuted across data points. Scores are computed for the hypothesis and scoring dataset.

Figure produced by the Python demo of the entry feature importance
Figure written by the entry's Python demo featureimportance.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

B-modelAn importance score assigns a number to each feature of a learned hypothesis $\learnthypothesis$. Predict tomorrow's daytime peak at a weather station from twelve features: today's and the previous day's high and low temperature, air pressure, humidity, sunshine duration and precipitation. A hypothesis learned by linear regression from one year predicts the next day's high with average squared error loss $8.8$, versus label variance $84.6$ (in squared degrees Celsius; Fig. 1).

Figure 1 of the entry featureimportance
Figure 1: Actual daytime high at the weather station Krems an der Donau for June 2024, along with the fitted values of a hypothesis learned by linear regression from the twelve features of the two preceding days. Data generated by pythondemos/featureimportance.py
For a linear model with standardized features, $\learnthypothesis(\featurevec) = \widehat{\weights}^{\top} \featurevec = \sum_{\featureidx=1}^{12} \widehat{\weight}_{\featureidx} \feature_{\featureidx}$ and $|\widehat{\weight}_{\featureidx}|$ is the prediction change per standard deviation of feature $\featureidx$. Current-day high coefficient: $8.6$; low coefficient: $0.02$ (Fig. 2).

Variable-importance scores appear in random forests (Breiman, 2001). Permutation importance uses predictions from a fitted hypothesis (Fisher et al., 2019; Molnar, 2025). On the scoring dataset, the $\featureidx$-th feature's score is the average-loss increase after shuffling its column. Denote it $I_{\featureidx}$.

B-permuteWith standardized features and squared error loss, replacing $\feature_{\featureidx}$ by an independent draw $\feature_{\featureidx}^{\prime}$ changes the prediction by $\delta_{\featureidx}=\widehat{\weight}_{\featureidx} (\feature_{\featureidx}^{\prime}-\feature_{\featureidx})$. The squared error loss increase is $\delta_{\featureidx}^{2} -2(\truelabel-\learnthypothesis(\featurevec)) \delta_{\featureidx}$, where $\truelabel$ is the label. Standardization gives unit variance, and zero covariance between $\feature_{\featureidx}$ and $\feature_{\featureidx}^{\prime}$ implies $\expect\{(\feature_{\featureidx}^{\prime} -\feature_{\featureidx})^{2}\}=2$. If the prediction error is uncorrelated with $\delta_{\featureidx}$, the cross-term averages out: $I_{\featureidx} \approx \expect\{\delta_{\featureidx}^{2}\} = 2\widehat{\weight}_{\featureidx}^{2}$ (Gregorutti et al., 2017). Fig. 2 plots $\sqrt{I_{\featureidx}/2}$ next to $|\widehat{\weight}_{\featureidx}|$; the two scores rank the features almost identically.

Figure 2 of the entry featureimportance
Figure 2: Two importance scores for the twelve features of the temperature forecast of Fig. 1: the coefficient magnitudes of the learned linear model and the values $\sqrt{I_{\featureidx}/2}$ obtained from permutation importances $I_{\featureidx}$. Both agree that the predictions rest almost entirely on the current-day high; permuting the corresponding low barely changes the loss, so for this fitted $\learnthypothesis$ and dataset, its importance is near zero. Data generated by pythondemos/featureimportance.py
The score measures the reliance of the learned hypothesis $\learnthypothesis$ on a feature, not the relevance of the feature itself: a feature that is predictive of the label obtains the score zero whenever $\learnthypothesis$ ignores it. For jointly Gaussian features and label, the score of a feature combines its covariance with the label and its covariances with the other features (Gregorutti et al., 2017).

Synonyms: variable importance.

See also: feature, linear model, loss.

References

  1. Breiman (2001). Random Forests. Machine Learning. doi.org/10.1023/A:1010933404324
  2. Fisher et al. (2019). All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously. Journal of Machine Learning Research.
  3. Molnar (2025). Interpretable Machine Learning: A Guide for Making Black Box Models Explainable. Ebook. christophm.github.io/interpretable-ml-book/
  4. Gregorutti et al. (2017). Correlation and variable importance in random forests. Statistics and Computing. doi.org/10.1007/s11222-016-9646-1

Cite this entry

@misc{dictml_featureimportance,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {feature importance},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-10-27},
  url = {https://dictionaryofml.org/terms/featureimportance.html}
}