$k$-fold cross-validation ($k$-fold CV) divides a
dataset into $k$ folds of equal size and uses each fold once
as the validation set while the other $k-1$ folds form the
training set of a given machine learning (ML) method. The $k$ resulting
validation errors are averaged into a single number, an indicator of
generalization beyond the training set. While a single
validation step estimates the risk of a single
hypothesis, $k$-fold CV estimates the risk of an
ML method. Leave-one-out cross-validation (LOO-CV) is the extreme case of $k$-fold
CV with one data point per fold.
B-daysFig. 1 depicts
four
data points, representing daily weather conditions at Krems
an der Donau (Austria). Each data point is characterized by the
minimum temperature of a day as the feature $x$ and
its maximum temperature as the label $y$. Three of
the four data points are used as the training set for
polynomial regression with degree three. The remaining data point
serves as the validation set. A polynomial of degree three (a cubic)
has four coefficients and the training set holds three
data points. Thus, unless two data points have the same
feature,
polynomial regression delivers a hypothesis that passes exactly
through those three data points and, in turn, the
training error is zero.
B-splitHowever, as depicted in Fig. 1, the
learned hypothesis $\widehat{h}$ does not generalize
well beyond the training set, i.e., polynomial regression with degree
three is overfitting. The validation error on the held-back
data point can reveal this overfitting, but a single
split does so unreliably. Holding back the first data point
gives a validation error of
$1.37$, which suggests a reliable hypothesis. Holding back
the second, third or fourth data point instead gives
validation errors of $8.05$, $7.02$ and $72.42$, as panels (b) to
(d) of Fig. 1 illustrate. Their
average $22.22$ is the $4$-fold CV estimate. Using different
splits of a dataset into training set and validation set
offers a more reliable assessment of a machine learning (ML) method.
Figure 1: (a) Four data points, each a day at Krems an der
Donau (Austria) with its minimum temperature as feature
$\feature$ and its maximum temperature as label
$\truelabel$. The filled circles indicate one
possible choice of a training set of three
data points. The red open square indicates the resulting
held-out data point, which serves as the validation set.
The curve is the hypothesis learned by polynomial regression
with degree three. The fold index $\foldidx$ numbers the
held-back data point. (b) to (d) The same for
$\foldidx = 2, 3, 4$: holding back a different
data point changes the learned hypothesis and its
validation error. Data generated by
pythondemos/kfoldcv.py
Consider an ML method $\mathcal{A}$ that maps a training set to
a learned hypothesis $\widehat{h} =
\mathcal{A}(\mathcal{D}^{(\rm train)})$. $k$-fold CV estimates the expected
loss (or risk) of the method $\mathcal{A}$ as follows
(Hastie et al., 2009, Sect. 7.10;
Stone, 1974). It divides a dataset $\mathcal{D}$
evenly into $k$ subsets, the folds $\mathcal{D}^{(1)},\,\ldots,\,
\mathcal{D}^{(k)}$ (Fig. 2). For each fold
$b = 1, \,\ldots, \,k$, one iteration trains the
model on the union of all folds except
$\mathcal{D}^{(b)}$ and evaluates the resulting
hypothesis on the held-out fold $\mathcal{D}^{(b)}$.
The $k$ per-fold validation errors are then averaged to obtain the
CV error $E_{\rm CV}$, the CV estimate of the expected loss.
Algorithm 1 summarizes this procedure.
Figure 2: An instance of $k$-fold CV with $k=5$: the available
dataset $\dataset$ is evenly divided into five folds
$\dataset^{(1)},\,\ldots,\,\dataset^{(5)}$. Each row
corresponds to one iteration; the shaded cell marks the fold
used as validation set, while the remaining $k-1$ folds form
the training setAlgorithm 1: $k$-fold CV
A single validation set assesses one hypothesis, e.g., the one
that $\mathcal{A}$ delivers from a training set
(Bates et al., 2023, Sect. 3.3.1). In contrast,
$k$-fold CV does not assess a single hypothesis. Instead,
every iteration learns a potentially different
hypothesis; panels (a) to (d) of
Fig. 1 show four such
hypotheses. Thus, the average validation error $E_{\rm CV}$
measures the method $\mathcal{A}$ rather than one hypothesis.
In Fig. 1, $\mathcal{A}$ is polynomial regression
with degree three and $E_{\rm CV} = 22.22$. In particular, $E_{\rm CV}$
measures the average
risk of the hypotheses obtained from
training sets holding the fraction $(k-1)/k$ of the
dataset (Bates et al., 2023, Sect. 3;
Hastie et al., 2009, Sect. 7.10.1).
B-cvCompared to a single split into training set and validation set,
$k$-fold CV uses each data point for both training and
validation. In Fig. 2, for example,
each data point has been used four times for training
and once for validation after all five iterations.
Like $k$-fold CV, the bootstrap estimates the risk of
a method by averaging over several hypotheses, each
learned from a different resample of the dataset; unlike
the folds of $k$-fold CV, these resamples overlap
(Bates et al., 2023, Sect. 3.5).
Important variants and special cases of $k$-fold CV include the
following:
Stratified $k$-fold cross-validation (stratified $k$-fold CV): It preserves the class
proportions of the full dataset in every fold, important
for classification with imbalanced data
(Kohavi, 1995).
Leave-one-out cross-validation (LOO-CV): It is the extreme case
$k = m$, with $m$ the number of
data points in the dataset, where each fold
contains a single data point (Stone, 1974). It
trains on almost the full dataset in every iteration,
at the cost of $m$ training runs and a high
variance of the estimate.
Repeated $k$-fold CV: It averages the results of
several runs with different random fold assignments
(Kim, 2009). This reduces the variance
contributed by the partitioning and is useful for small
datasets, where a single partitioning strongly affects the
estimate.
See also: validation, validation error, generalization gap,
learning curve, leave-one-out cross-validation, stratified $k$-fold cross-validation,
model selection, bootstrap.
References
Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7
Stone (1974). Cross-Validatory Choice and Assessment of Statistical Predictions. Journal of the Royal Statistical Society. Series B (Methodological). doi.org/10.1111/j.2517-6161.1974.tb00994.x
Bates et al. (2023). Cross-Validation: What Does It Estimate and How Well Does It Do It?. Journal of the American Statistical Association. doi.org/10.1080/01621459.2023.2197686
Kohavi (1995). A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection. Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI).
Kim (2009). Estimating Classification Error Rate: Repeated Cross-Validation, Repeated Hold-Out and Bootstrap. Computational Statistics & Data Analysis.
Cite this entry
@misc{dictml_kfoldcv,
author = {Jung, Alexander},
editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {$k$-fold cross-validation ($k$-fold CV)},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-13},
url = {https://dictionaryofml.org/terms/kfoldcv.html}
}
