# hyperparameter

*Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-10-29. [HTML](https://dictionaryofml.org/terms/hyperparameter.html) · [typeset PDF](https://dictionaryofml.org/terms/hyperparameter.pdf) · [LaTeX](https://dictionaryofml.org/terms/hyperparameter.tex) · [Python demo](https://dictionaryofml.org/terms/hyperparameter.py) · [notebook](https://dictionaryofml.org/terms/hyperparameter.ipynb)*

Math is LaTeX with the book's macros expanded, so no notation package is needed to read it.

## Abstract

A hyperparameter is any quantity that specifies the components of a machine learning method. Examples are the maximum degree in polynomial regression, the regularization strength in ridge regression, and the number of data points used to construct a training set. Hyperparameters can also determine the behavior of the optimization method used to solve empirical risk minimization, that is, to train a model; one example is the learning rate in gradient descent. A hyperparameter selects one machine learning method out of several alternatives and can therefore be chosen by model selection techniques, such as those based on the validation error.

## Definition

Predicting the maximum temperature of a day at Krems an der Donau from its minimum temperature calls for decisions before any training happens: which maps the method may return, how many data points to collect, and how to search those maps. Fig. 1 shows twelve days as data points, each with the minimum temperature as its feature $x$ and the maximum temperature as its label $y$. Three straight lines depict hypotheses of the underlying hypothesis space, or model, of the machine learning method. A loss function is used to measure the prediction error incurred on a data point.

A hyperparameter is any quantity that specifies one of these components of a machine learning method. The maximum degree in polynomial regression specifies the model, the regularization strength in ridge regression affects the loss function (Bishop, 2006, Sect. 1.3), and the number of data points gathered specifies the training set.

![Figure 1 of the entry hyperparameter](https://dictionaryofml.org/terms/fig/hyperparameter-fig1.svg)

*Figure 1: Twelve days of 2020 at Krems an der Donau (filled circles), with the minimum temperature of a day as feature $x$ and its maximum temperature as label $y$. The three straight lines depict hypotheses of the model: the solid line is one hypothesis $h$, the dashed and dotted lines are two others; the dashed loop encircling them marks the hypothesis space $\mathcal{H}$. The vertical arrow marks the prediction error of a single data point under $h$, measured by the loss function $L\left((x,y),h \right)$. Data generated by pythondemos/hyperparameter.py*

The third decision of the opening, how to search the model, is specified by hyperparameters of the optimization method that solves empirical risk minimization. The learning rate of gradient descent, for example, bounds the distance between the iterate of gradient descent, minimizing the training error, and its starting point after a fixed number of gradient steps, and a larger learning rate raises that bound. Thus, the set of model parameters it can reach, and with them the effective hypothesis space, grows. Fig. 2 depicts this for five gradient steps on one training set, starting from the zero initialization ${\mathbf w}^{(0)} = (0,0)^{\top}$, at three values $\eta_{1} < \eta_{2} < \eta_{3}$ of the learning rate. The effective hypothesis spaces $\mathcal{H}^{(\eta_{1})} \subset
\mathcal{H}^{(\eta_{2})} \subset \mathcal{H}^{(\eta_{3})}$ are nested, so the learning rate indexes a family of models ordered by size, as the maximum degree does in polynomial regression.

![Figure 2 of the entry hyperparameter](https://dictionaryofml.org/terms/fig/hyperparameter-fig2.svg)

*Figure 2: Gradient descent on one training set of $30$ data points with two features, starting from the zero initialization ${\mathbf w}^{(0)} = (0,0)^{\top}$ (cross) and stopped after five gradient steps, for a small, a medium and a large learning rate, $\eta_{1} = 0.1/L$, $\eta_{2} = 0.4/L$ and $\eta_{3} = 1.4/L$, with $L$ the largest curvature of the training error. The dots are the iterates, drawn in the style of their learning rate: dashed for $\eta_{1}$, dash-dotted for $\eta_{2}$, solid for $\eta_{3}$. The ellipse $\mathcal{H}^{(\eta_{k})}$ passes through the iterate of learning rate $\eta_{k}$ farthest from ${\mathbf w}^{(0)}$ and has the shape of the level sets of the training error; it is the effective hypothesis space that learning rate allows and contains every gradient step taken at it. The three ellipses are nested, and each contains every iterate of every smaller learning rate. Data generated by pythondemos/hyperparameter.py*

Whichever component it specifies, a hyperparameter selects one machine learning method out of a family of alternatives, and that is what makes a single rule for choosing it possible.

The usefulness of a specific hyperparameter value can be assessed by model selection techniques such as validation (Hastie et al., 2009, Sect. 7.10). As with learning the model parameters by empirical risk minimization on a training set, a hyperparameter is chosen by minimizing the validation error, with the validation error in place of the training error (Bishop, 2006, Sect. 1.3). The search often runs over a grid or a randomly drawn set of candidate values, keeping the one with the smallest validation error on the validation set.

Fig. 3 shows this choice for the degree of a polynomial model learned by polynomial regression from $20$ data points. The training error falls with every increase of the degree and would select the largest one. The validation error on $100$ further data points is smallest at degree $3$ and rises steeply beyond it, so validation selects degree $3$.

![Figure 3 of the entry hyperparameter](https://dictionaryofml.org/terms/fig/hyperparameter-fig3.svg)

*Figure 3: Training error (solid line, filled circles) and validation error (dashed line, open squares) of a polynomial model against its degree, the hyperparameter. The training error never increases with the degree and would select degree $9$; the validation error is smallest at degree $3$, the value validation selects. Data generated by pythondemos/hyperparameter.py*

**See also:** [model](https://dictionaryofml.org/terms/model.html), validation, model parameter, [validation error](https://dictionaryofml.org/terms/valerr.html), [training error](https://dictionaryofml.org/terms/trainerr.html), [model selection](https://dictionaryofml.org/terms/modelsel.html), [learning rate](https://dictionaryofml.org/terms/learnrate.html), [gradient descent](https://dictionaryofml.org/terms/gd.html), polynomial regression.

## References

- Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-45528-0>
- Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. *Springer Science+Business Media*. <https://doi.org/10.1007/978-0-387-84858-7>

## Cite this entry

```bibtex
@misc{dictml_hyperparameter,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {hyperparameter},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-03},
  url = {https://dictionaryofml.org/terms/hyperparameter.html}
}
```