Dictionary of Applied Machine Learning: 114 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

learning rate

Updated on 2026-10-29

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

See also hypothesis gradient descent stochastic gradient descent empirical risk minimization hyperparameter hypothesis space

A learning rate is a hyperparameter of an iterative machine learning (ML) method that controls the extent of a single update of the current hypothesis. A prime example for a learning rate is the step size of gradient descent (GD). A learning rate that is too small makes little progress within the available number of iterations, and one that is too large can result in divergent iterations. For a fixed number of iterations, the learning rate bounds the effective hypothesis space of an ML method, which makes it a means of regularization. The learning rate can be chosen by the validation error of the learned hypothesis.

Figure produced by the Python demo of the entry learning rate
Figure written by the entry's Python demo learnrate.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

Learning a linear model by gradient descent (GD) on the average squared error loss over a training set, as in Fig. 1, is an iterative machine learning (ML) method for learning a useful hypothesis $h \in \mathcal{H}$. Such a method repeats similar update steps that modify the current hypothesis to obtain an improved one. A learning rate is a parameter of such a method that controls the extent to which the current hypothesis can be modified during a single iteration.

B-stepThe main example of a learning rate $\eta$ is the step size of GD (Boyd and Vandenberghe, 2004, Sect. 9.3), \begin{equation} \label{equ_def_basic_gradstep_lrate_dict} {\mathbf w}^{(t\!+\!1)} = {\mathbf w}^{(t)} - \eta \nabla f({\mathbf w}^{(t)}) \text{.} \end{equation} Here, the objective function $f({\mathbf w})$ is the empirical risk incurred by a hypothesis $h^{({\mathbf w})}$ on a training set, and each gradient step moves the model parameters against its gradient $\nabla f({\mathbf w}^{(t)})$ by a distance proportional to $\eta$. Fig. 1 illustrates one gradient step \(\eqref{equ_def_basic_gradstep_lrate_dict}\) of GD for a linear model, started from the same hypothesis with three choices of the learning rate. The learning rates are given as multiples of $1/L$, with $L$ the largest eigenvalue of the Hessian of $f$. For the squared error loss, a gradient step lowers $f$ for every learning rate below $2/L$ (Nesterov, 2004, Thm. 2.1.14); the largest learning rate of Fig. 1 lies above that threshold, and its step raises $f$.

Figure 1 of the entry learnrate
Figure 1: One gradient step of GD on the average squared error loss of a linear model on a training set. The solid line is the hypothesis map before the step. The dashed line is the hypothesis map after one step with the learning rate $\lrate = 0.1/L$, the dotted line after one step with $\lrate = 1.5/L$, and the gray dash-dotted line after one step with $\lrate = 2.5/L$, where $L$ is the largest eigenvalue of the Hessian of the objective function. Data generated by pythondemos/learnrate.py
Each iteration of an ML method should make large progress towards a useful hypothesis, so the learning rate should not be too small. The step with $\eta = 2.5/L$ in Fig. 1 shows the other failure: a too large step size of GD makes the gradient steps overshoot a minimum of the objective function. Fig. 2 illustrates both cases.
Figure 2 of the entry learnrate
Figure 2: Effect of using an inadequate step size $\lrate$ in the gradient step \(\eqref{equ_def_basic_gradstep_lrate_dict}\). (a) If $\lrate$ is too large, the gradient steps can overshoot such that the iterates $\weights^{(\iteridx)}$ move away from a minimum of $f$, i.e., $f(\weights^{(\iteridx\!+\!1)}) > f(\weights^{(\iteridx)})$. (b) If $\lrate$ is too small, the gradient steps make too little progress towards a minimum within the available number of iterations
Consider an iterative ML method that learns a hypothesis from some underlying hypothesis space $\mathcal{H}$ by running a fixed number $T$ of iterations. Varying the learning rate implements regularization by pruning the effective $\mathcal{H}$. For example, each gradient step above \(\eqref{equ_def_basic_gradstep_lrate_dict}\) moves the model parameters by $\eta \Vert {\nabla f({\mathbf w}^{(t)})} \Vert$. By the triangle inequality, \begin{equation} \label{equ_learnrate_reach_dict} \Vert {{\mathbf w}^{(T)} - {\mathbf w}^{(0)}} \Vert \leq \eta \sum_{t=0}^{T-1} \Vert {\nabla f({\mathbf w}^{(t)})} \Vert \text{.} \end{equation} Together with $T$, the learning rate thus determines the region of the parameter space that the method can reach, and thereby the effective hypothesis space: the hypotheses $h^{({\mathbf w})}$ whose model parameters ${\mathbf w}$ lie in that region.

B-reachFig. 3 draws the balls around ${\mathbf w}^{(0)}$ with the radius \(\eqref{equ_learnrate_reach_dict}\) for $T = 10$ gradient steps of GD solving the empirical risk minimization (ERM) problem of linear regression, and three learning rates. Reducing the learning rate shrinks the effective hypothesis space that can be reached by the gradient steps. Choosing the learning rate too small or too large can then result in underfitting or overfitting.

Figure 3 of the entry learnrate
Figure 3: The model parameters that $\nriter = 10$ gradient steps for linear regression can reach from the initial $\weights^{(0)}$ (filled square); each ball is the effective hypothesis space $\hypospace^{(\lrate)}$ of its learning rate. Each curve is the ball around $\weights^{(0)}$ with the radius \(\eqref{equ_learnrate_reach_dict}\), computed along the iterations with the learning rate $\lrate = 0.1/L$ (solid curve), $0.5/L$ (dashed curve) and $1.5/L$ (dotted curve). The ERM solution $\widehat{\weights}$ (star) lies outside the two smaller balls and inside the largest one. Data generated by pythondemos/learnrate.py
As a hyperparameter, the learning rate can be chosen by validation: the hypotheses learned with different learning rates are compared by their validation error on a validation set.

Synonyms: step size.

See also: machine learning, hypothesis, parameter, gradient descent, stochastic gradient descent, projected gradient descent, step size, empirical risk minimization, linear model, hyperparameter, parameter space, hypothesis space.

References

  1. Boyd and Vandenberghe (2004). Convex Optimization. Cambridge Univ. Press. doi.org/10.1017/CBO9780511804441
  2. Nesterov (2004). Introductory Lectures on Convex Optimization: A Basic Course. Kluwer Academic. doi.org/10.1007/978-1-4419-8853-9

Cite this entry

@misc{dictml_learnrate,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {learning rate},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-03},
  url = {https://dictionaryofml.org/terms/learnrate.html}
}