Dictionary of Applied Machine Learning: 114 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

hyperparameter

Updated on 2026-10-29

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

See also model validation error training error model selection learning rate gradient descent

A hyperparameter is any quantity that specifies the components of a machine learning (ML) method. Examples are the maximum degree in polynomial regression, the regularization strength in ridge regression, and the number of data points used to construct a training set. Hyperparameters can also determine the behavior of the optimization method used to solve empirical risk minimization (ERM), that is, to train a model; one example is the learning rate in gradient descent (GD). A hyperparameter selects one ML method out of several alternatives and can therefore be chosen by model selection techniques, such as those based on the validation error.

Figure produced by the Python demo of the entry hyperparameter
Figure written by the entry's Python demo hyperparameter.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

B-kremsPredicting the maximum temperature of a day at Krems an der Donau from its minimum temperature calls for decisions before any training happens: which maps the method may return, how many data points to collect, and how to search those maps. Fig. 1 shows twelve days as data points, each with the minimum temperature as its feature $x$ and the maximum temperature as its label $y$. Three straight lines depict hypotheses of the underlying hypothesis space, or model, of the machine learning (ML) method. A loss function is used to measure the prediction error incurred on a data point.

B-reachA hyperparameter is any quantity that specifies one of these components of an ML method. The maximum degree in polynomial regression specifies the model, the regularization strength in ridge regression affects the loss function (Bishop, 2006, Sect. 1.3), and the number of data points gathered specifies the training set.

Figure 1 of the entry hyperparameter
Figure 1: Twelve days of 2020 at Krems an der Donau (filled circles), with the minimum temperature of a day as feature $\feature$ and its maximum temperature as label $\truelabel$. The three straight lines depict hypotheses of the model: the solid line is one hypothesis $\hypothesis$, the dashed and dotted lines are two others; the dashed loop encircling them marks the hypothesis space $\hypospace$. The vertical arrow marks the prediction error of a single data point under $\hypothesis$, measured by the loss function $\lossfunc{(\feature,\truelabel)}{\hypothesis}$. Data generated by pythondemos/hyperparameter.py
The third decision of the opening, how to search the model, is specified by hyperparameters of the optimization method that solves empirical risk minimization (ERM). The learning rate of gradient descent (GD), for example, bounds the distance between the iterate of GD, minimizing the training error, and its starting point after a fixed number of gradient steps, and a larger learning rate raises that bound. Thus, the set of model parameters it can reach, and with them the effective hypothesis space, grows. Fig. 2 depicts this for five gradient steps on one training set, starting from the zero initialization ${\mathbf w}^{(0)} = (0,0)^{\top}$, at three values $\eta_{1} < \eta_{2} < \eta_{3}$ of the learning rate. The effective hypothesis spaces $\mathcal{H}^{(\eta_{1})} \subset \mathcal{H}^{(\eta_{2})} \subset \mathcal{H}^{(\eta_{3})}$ are nested, so the learning rate indexes a family of models ordered by size, as the maximum degree does in polynomial regression.
Figure 2 of the entry hyperparameter
Figure 2: GD on one training set of $30$ data points with two features, starting from the zero initialization $\weights^{(0)} = (0,0)^{\top}$ (cross) and stopped after five gradient steps, for a small, a medium and a large learning rate, $\lrate_{1} = 0.1/L$, $\lrate_{2} = 0.4/L$ and $\lrate_{3} = 1.4/L$, with $L$ the largest curvature of the training error. The dots are the iterates, drawn in the style of their learning rate: dashed for $\lrate_{1}$, dash-dotted for $\lrate_{2}$, solid for $\lrate_{3}$. The ellipse $\hypospace^{(\lrate_{k})}$ passes through the iterate of learning rate $\lrate_{k}$ farthest from $\weights^{(0)}$ and has the shape of the level sets of the training error; it is the effective hypothesis space that learning rate allows and contains every gradient step taken at it. The three ellipses are nested, and each contains every iterate of every smaller learning rate. Data generated by pythondemos/hyperparameter.py
Whichever component it specifies, a hyperparameter selects one ML method out of a family of alternatives, and that is what makes a single rule for choosing it possible.

B-dataThe usefulness of a specific hyperparameter value can be assessed by model selection techniques such as validation (Hastie et al., 2009, Sect. 7.10). As with learning the model parameters by ERM on a training set, a hyperparameter is chosen by minimizing the validation error, with the validation error in place of the training error (Bishop, 2006, Sect. 1.3). The search often runs over a grid or a randomly drawn set of candidate values, keeping the one with the smallest validation error on the validation set.

B-sweepFig. 3 shows this choice for the degree of a polynomial model learned by polynomial regression from $20$ data points. The training error falls with every increase of the degree and would select the largest one. The validation error on $100$ further data points is smallest at degree $3$ and rises steeply beyond it, so validation selects degree $3$.

Figure 3 of the entry hyperparameter
Figure 3: Training error (solid line, filled circles) and validation error (dashed line, open squares) of a polynomial model against its degree, the hyperparameter. The training error never increases with the degree and would select degree $9$; the validation error is smallest at degree $3$, the value validation selects. Data generated by pythondemos/hyperparameter.py

See also: model, validation, model parameter, validation error, training error, model selection, learning rate, gradient descent, polynomial regression.

References

  1. Bishop (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media. doi.org/10.1007/978-0-387-45528-0
  2. Hastie et al. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Science+Business Media. doi.org/10.1007/978-0-387-84858-7

Cite this entry

@misc{dictml_hyperparameter,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {hyperparameter},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-03},
  url = {https://dictionaryofml.org/terms/hyperparameter.html}
}