# k-fold cross-validation (k-fold CV)

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-08. [HTML](https://dictionaryofml.org/terms/kfoldcv.html) · [typeset PDF](https://dictionaryofml.org/terms/kfoldcv.pdf) · [LaTeX](https://dictionaryofml.org/terms/kfoldcv.tex) · [Python demo](https://dictionaryofml.org/terms/kfoldcv.py) · [notebook](https://dictionaryofml.org/terms/kfoldcv.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

$k$-fold cross-validation ($k$-fold CV) divides a dataset into $k$ folds of equal size and uses each fold once as the validation set while the other $k-1$ folds form the training set of a given machine learning method. The $k$ resulting validation errors are averaged into a single number, an indicator of generalization beyond the training set. While a single validation step estimates the risk of a single hypothesis, $k$-fold CV estimates the risk of a training method. Leave-one-out cross-validation is the extreme case with one data point per fold, and stratified k-fold cross-validation preserves class proportions of the full dataset in every fold.

## Definition

Four days at Krems an der Donau are the only recorded data points. A day is characterized by its morning minimum temperature as the feature $x$ and its maximum daytime temperature as the label $y$ (Fig. 1). Three of the days form the training set of a polynomial regression of degree three, and the remaining day is the validation set. A cubic has four coefficients and the training set holds three days, so the empirical risk minimization problem is underdetermined and its solution passes exactly through those three days: the training error is zero whichever day is held back.

That makes the held-back day the only evidence about the learned hypothesis, and a single split can give a small validation error to a hypothesis with a large risk. Holding back the first day gives a validation error of $1.37$, which suggests a reliable hypothesis. This changes when another day is held back for the validation set, as panels (b) to (d) of Fig. 1 illustrate. Using different splits of a dataset into training set and validation set offers a more reliable assessment of a machine learning method.

![Figure 1 of the entry kfoldcv](https://dictionaryofml.org/terms/fig/kfoldcv-fig1.svg)

*Figure 1: (a) Four days at Krems an der Donau, each a data point with the morning minimum temperature as feature $x$ and the maximum daytime temperature as label $y$, and the degree-three hypothesis that empirical risk minimization delivers from three of them; the red open square is the first day, held back as the validation set (fold $b = 1$). (b) to (d) The hypothesis obtained when the second, third or fourth day is held back instead. Data generated by pythondemos/kfoldcv.py*

Consider a machine learning method $\mathcal{A}$ that maps a training set to a learned hypothesis $\widehat{h} =
\mathcal{A}(\mathcal{D}^{(\rm train)})$. $k$-fold CV estimates the expected loss (or risk) of the method $\mathcal{A}$ as follows (Hastie et al., 2009, Sect. 7.10; Stone, 1974). It divides a dataset $\mathcal{D}$ evenly into $k$ subsets, the folds $\mathcal{D}^{(1)},\,\ldots,\,
\mathcal{D}^{(k)}$ (Fig. 2). For each fold $b = 1, \,\ldots, \,k$, the model is trained on the union of all folds except $\mathcal{D}^{(b)}$ and the resulting hypothesis is evaluated on the held-out fold $\mathcal{D}^{(b)}$. The $k$ per-fold validation errors are then averaged to obtain the CV error $E_{\rm CV}$, the CV estimate of the expected loss. Algorithm 1 summarizes this procedure.

![Algorithm 1 of the entry kfoldcv](https://dictionaryofml.org/terms/fig/kfoldcv-alg1.svg)

*Algorithm 1: $k$-fold CV*

A single validation set assesses one hypothesis, e.g., the one that $\mathcal{A}$ delivers from a training set (Bates et al., 2023, Sect. 3.3.1). In contrast, $k$-fold CV has no single hypothesis to assess, since every iteration learns a potentially different hypothesis. Thus, the average validation error $E_{\rm CV}$ that $k$-fold CV delivers measures the method $\mathcal{A}$ rather than one hypothesis: the average risk of the hypotheses obtained from training sets holding the fraction $(k-1)/k$ of the dataset (Bates et al., 2023, Sect. 3; Hastie et al., 2009, Sect. 7.10.1). Compared to a single split into training set and validation set, $k$-fold CV uses each data point for both training and validation. In Fig. 2, for example, each data point has been used four times for training and once for validation after all five iterations. A bootstrap estimate is an average of the same kind (Bates et al., 2023, Sect. 3.5).

![Figure 2 of the entry kfoldcv](https://dictionaryofml.org/terms/fig/kfoldcv-fig2.svg)

*Figure 2: An instance of $k$-fold CV with $k=5$: the available dataset $\mathcal{D}$ is evenly divided into five folds $\mathcal{D}^{(1)},\,\ldots,\,\mathcal{D}^{(5)}$. Each row corresponds to one iteration; the shaded cell marks the fold used as validation set, while the remaining $k-1$ folds form the training set*

Some variants and special cases of $k$-fold CV are:

- *Stratified k-fold cross-validation* preserves the class proportions of the full dataset in every fold, important for classification with imbalanced data (Kohavi, 1995).
- *Leave-one-out cross-validation* is the extreme case $k = m$, with $m$ the number of data points in the dataset, where each fold contains a single data point (Stone, 1974).
- *Repeated $k$-fold CV* averages the results of several runs with different random fold assignments, which reduces the variance contributed by the partitioning (Kim, 2009).

**See also:** validation, [validation error](https://dictionaryofml.org/terms/valerr.html), generalization gap, learning curve, leave-one-out cross-validation, stratified $k$-fold cross-validation, [model selection](https://dictionaryofml.org/terms/modelsel.html), [bootstrap](https://dictionaryofml.org/terms/bootstrap.html).

## References

- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>
- Stone (1974). Cross-Validatory Choice and Assessment of Statistical Predictions. *Journal of the Royal Statistical Society. Series B (Methodological)*. <https://doi.org/10.1111/j.2517-6161.1974.tb00994.x>
- Bates et al. (2023). Cross-Validation: What Does It Estimate and How Well Does It Do It?. *Journal of the American Statistical Association*. <https://doi.org/10.1080/01621459.2023.2197686>
- Kohavi (1995). A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection. *Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI)*.
- Kim (2009). Estimating Classification Error Rate: Repeated Cross-Validation, Repeated Hold-Out and Bootstrap. *Computational Statistics & Data Analysis*.

## Cite this entry

```bibtex
@misc{dictml_kfoldcv,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {$k$-fold cross-validation ($k$-fold CV)},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11},
  url = {https://dictionaryofml.org/terms/kfoldcv.html}
}
```