# backpropagation *Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-10. [HTML](https://dictionaryofml.org/terms/backpropagation.html) · [typeset PDF](https://dictionaryofml.org/terms/backpropagation.pdf) · [LaTeX](https://dictionaryofml.org/terms/backpropagation.tex) · [Python demo](https://dictionaryofml.org/terms/backpropagation.py) · [notebook](https://dictionaryofml.org/terms/backpropagation.ipynb)* Math is LaTeX with the book's macros expanded, so no notation package is needed to read it. ## Abstract Backpropagation is an algorithm that computes the gradient of the loss with respect to the model parameters of an artificial neural network. The algorithm consists of two consecutive phases, referred to as forward and backward pass. The forward pass evaluates, for the current model parameters, the activations and the resulting prediction errors on a given dataset. The backward pass then applies the chain rule to compute the partial derivatives of the average loss with respect to the model parameters. These partial derivatives constitute a gradient, which can be used in a gradient step to update the model parameters. ## Definition Consider an artificial neural network with a billion model parameters and its prediction for a new data point. If this prediction is wrong, the model parameters can be updated by a gradient step. That update requires the partial derivatives of the loss on the new data point with respect to all model parameters. Backpropagation is an algorithm for computing these partial derivatives with as little duplicated computation as possible (Goodfellow et al., 2016, Sect. 6.5.3; Rumelhart et al., 1986). More generally, it computes the gradient $\nabla f({\mathbf w})$ of an objective function $f({\mathbf w})$ with respect to the model parameters ${\mathbf w}$ of the artificial neural network. The model parameters consist of the edge weights and the offset terms at the nodes of the artificial neural network. The objective function is typically the average loss $$ f({\mathbf w}) = \frac{1}{m} \sum_{r=1}^{m} L\left(\left( {\mathbf x}^{(r)},y^{(r)} \right),h^{({\mathbf w})} \right) \text{,} $$ incurred on a dataset of $m$ data points $\left( {\mathbf x}^{(1)},y^{(1)} \right), \ldots, \left( {\mathbf x}^{(m)},y^{(m)} \right)$. Backpropagation computes the partial derivatives of $f({\mathbf w})$ one layer at a time (Fig. 1). Its execution consists of two consecutive phases, the forward pass and the backward pass. In the forward pass, the artificial neural network is applied to each data point of the dataset: the neurons of each layer evaluate their activations from the activations of the previous layer. The final layer then produces the output, which is the prediction for the label of a data point with features ${\mathbf x}$. The quality of this prediction is measured by a loss function $L\left(\left( {\mathbf x},y \right),h^{({\mathbf w})} \right)$, where $h^{({\mathbf w})}$ is the hypothesis map represented by the artificial neural network with model parameters ${\mathbf w}$. In the backward pass, the partial derivatives of $f({\mathbf w})$ with respect to the edge weights of each layer are computed recursively, from the output back to the input. These partial derivatives form the whole gradient $\nabla f({\mathbf w})$. The backward pass requires about as much computation as the forward pass: its cost grows in proportion to the number of edges of the artificial neural network, one partial derivative, one multiplication, and one addition per edge (Goodfellow et al., 2016, Sect. 6.5.3). ![Figure 1 of the entry backpropagation](https://dictionaryofml.org/terms/fig/backpropagation-fig1.svg) *Figure 1: Backpropagation through an artificial neural network with one hidden layer. The solid arrows indicate the forward pass: the features $x_{1},x_{2},x_{3}$ reach the neurons $a_{1},a_{2},a_{3}$ through the edge weights ${\mathbf W}^{(1)}$, then the output $\widehat{y} = h^{({\mathbf w})}({\mathbf x})$ through ${\mathbf W}^{(2)}$. The output is a prediction for the label of a data point and incurs the loss $f({\mathbf w})$. The dashed arrow indicates the backward pass, which runs the other way and yields the partial derivatives of $f({\mathbf w})$ with respect to the edge weights of both layers. The model parameters ${\mathbf w}$ collect the edge weights of both layers; the offset terms of the neurons are omitted here* To illustrate the details of backpropagation, consider a simple artificial neural network: it consists of three layers with one neuron each. Its model parameters are the three edge weights $w^{(1)}, w^{(2)}, w^{(3)}$. The gradient of an average is the average of the gradients of its summands, so a dataset with a single data point already shows the recursion. The objective function $f({\mathbf w})$ is then the loss of that data point. Write $a^{(0)} = x$ for its feature. Layer $\ell = 1, 2, 3$ computes $$ z^{(\ell)} = w^{(\ell)} a^{(\ell-1)} \text{,} \qquad a^{(\ell)} = \sigma\big(z^{(\ell)}\big) \text{,} $$ with the activation function $\sigma$ and no offset terms. The output is $h^{({\mathbf w})}(x) = a^{(3)}$. The edge weight $w^{(3)}$ enters $f({\mathbf w})$ only through $z^{(3)}$, $w^{(2)}$ through $z^{(2)}$ and then $z^{(3)}$, and $w^{(1)}$ through all three. The chain rule therefore gives $$ \begin{aligned} \frac{\partial f}{\partial w^{(3)}} &= \frac{\partial f}{\partial a^{(3)}}\, \sigma'\big(z^{(3)}\big)\, a^{(2)} \text{,} \\ \frac{\partial f}{\partial w^{(2)}} &= \frac{\partial f}{\partial a^{(3)}}\, \sigma'\big(z^{(3)}\big)\, w^{(3)}\, \sigma'\big(z^{(2)}\big)\, a^{(1)} \text{,} \\ \frac{\partial f}{\partial w^{(1)}} &= \frac{\partial f}{\partial a^{(3)}}\, \sigma'\big(z^{(3)}\big)\, w^{(3)}\, \sigma'\big(z^{(2)}\big)\, w^{(2)}\, \sigma'\big(z^{(1)}\big)\, x \text{.} \end{aligned} $$ Each line repeats the one above it and appends two factors. Evaluating the three lines separately would recompute the repeated factors twice over. Backpropagation computes them once, from the output backwards. The repeated factors form the partial derivative of $f({\mathbf w})$ with respect to $z^{(\ell)}$. This partial derivative obeys its own recursion, $$ \frac{\partial f}{\partial z^{(3)}} = \frac{\partial f}{\partial a^{(3)}} \sigma'\big(z^{(3)}\big) \text{,} \qquad \frac{\partial f}{\partial z^{(\ell)}} = \frac{\partial f}{\partial z^{(\ell+1)}} w^{(\ell+1)} \sigma'\big(z^{(\ell)}\big) \quad \text{for } \ell = 2, 1 \text{.} $$ Each edge weight then costs one multiplication, $$ \frac{\partial f}{\partial w^{(\ell)}} = \frac{\partial f}{\partial z^{(\ell)}} a^{(\ell-1)} \text{.} $$ The backward pass is this recursion, and the activations $a^{(\ell)}$ it needs are the ones the forward pass already computed. The partial derivatives delivered by backpropagation constitute the gradient used in a gradient step: $$ {\mathbf w}^{(t+1)} = {\mathbf w}^{(t)} - \eta \nabla f({\mathbf w}^{(t)}) \text{, for } t = 0,1,\ldots \text{,} $$ with step size $\eta$. Each gradient step requires a new run of backpropagation, since the model parameters change after each step. These partial derivatives are not only used for a gradient step. A class activation map explains a prediction of a convolutional neural network by weighting its feature maps with partial derivatives of the output (Selvaraju et al., 2017). **See also:** [artificial neural network](https://dictionaryofml.org/terms/ann.html), deep net, [gradient](https://dictionaryofml.org/terms/gradient.html), [gradient descent](https://dictionaryofml.org/terms/gd.html), gradient step, layer, edge weight, model parameter, activation function, loss function, neuron. ## References - Goodfellow et al. (2016). Deep Learning. *MIT Press*. - Rumelhart et al. (1986). Learning Representations by Back-Propagating Errors. *Nature*. - Selvaraju et al. (2017). Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. *2017 IEEE Int. Conf. Comput. Vis.*. ## Cite this entry ```bibtex @misc{dictml_backpropagation, author = {Jung, Alexander}, editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard}, title = {backpropagation}, howpublished = {Dictionary of Applied Machine Learning (course edition)}, year = {2026}, doi = {10.5281/zenodo.21569296}, note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11}, url = {https://dictionaryofml.org/terms/backpropagation.html} } ```