Dictionary of Applied Machine Learning

learning rate

Updated on 2026-09-18

Typeset PDF version — the authoritative form of this entry

A learning rate is the parameter of an iterative machine learning (ML) method that scales the change applied to the current model parameters in one iteration. In the gradient step of gradient descent (GD), the learning rate $\lrate$ multiplies the negative gradient of the objective function, so the update is $\weights^{(\iteridx+1)} = \weights^{(\iteridx)} - \lrate \nabla f(\weights^{(\iteridx)})$. A learning rate that is too large lets the iterates overshoot, so the objective function value increases, while one that is too small makes too little progress within the available number of iterations. The learning rate is therefore a hyperparameter, chosen by validation or by a schedule that decreases it over the iterations.

Definition

Consider an iterative machine learning (ML) method for finding or learning a useful hypothesis $\hypothesis \in \hypospace$. Such an iterative method repeats similar computational (update) steps that adjust or modify the current hypothesis to obtain an improved hypothesis. A key parameter of an iterative method is the learning rate. The learning rate controls the extent to which the current hypothesis can be modified during a single iteration. Consider, for example, the gradient step (Jung, 2022, Ch. 5) \begin{equation} \label{equ_def_basic_gradstep_lrate_dict} \weights^{(\iteridx\!+\!1)} = \weights^{(\iteridx)} - \lrate \nabla f(\weights^{(\iteridx)}) \end{equation} of a gradient-based method for empirical risk minimization (ERM), where the objective function $f(\weights)$ is the empirical risk incurred by $\hypothesis^{(\weights)}$ on a training set. Given the current model parameters $\weights^{(\iteridx)}$ at iteration $\iteridx$, the gradient step produces updated model parameters $\weights^{(\iteridx\!+\!1)}$ by moving in the opposite direction of the gradient $\nabla f(\weights^{(\iteridx)})$.

Figure 1 of the entry learnrate
Figure 1: Effect of using an inadequate learning rate $\lrate$ in the gradient step \(\eqref{equ_def_basic_gradstep_lrate_dict}\). (a) If $\lrate$ is too large, the gradient steps can "overshoot" such that the iterates $\weights^{(\iteridx)}$ diverge away from the optimum, i.e., $f(\weights^{(\iteridx\!+\!1)}) > f(\weights^{(\iteridx)})$. (b) If $\lrate$ is too small, the gradient steps make too little progress towards the optimum within the available number of iterations (due to limited computational budget).
See also: ML, hypothesis, parameter, gradient descent (GD), stochastic gradient descent (SGD), projected gradient descent (projected GD), step size.

References

  1. Jung (2022). Machine Learning: The Basics. Springer Nature. doi.org/10.1007/978-981-16-8193-6

Cite this entry

@misc{dictml_learnrate,
  author = {Jung, Alexander and Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {learning rate},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-19},
  url = {https://dictionaryofml.org/terms/learnrate.html}
}