Dictionary of Applied Machine Learning: 147 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here
Updated on 2026-11-20
Readiness 90/100
See also artificial neural network backpropagation feature learning feature transformation overfitting hyperparameter interpretable machine learning large language model
A deep net is an artificial neural network (ANN) whose hypothesis maps are compositions of several layers, each of which applies an affine map and then an activation function to every entry of the result. The number of layers that carry learnable model parameters is the depth of the net, and deep learning is the name of the machine learning (ML) methods that use a deep net as their model. The layers before the last one compute a feature transformation learned from the data, and the last layer applies a linear hypothesis to the activations they deliver. A deep net therefore makes the choice of features part of training instead of a design decision taken beforehand. Its model parameters, the weights and offset terms of every layer, are learned by gradient descent (GD) on the training error, with the gradient computed by backpropagation. Depth is a property of the net, not of a single hypothesis: adding layers enlarges the hypothesis space and lowers the training error, while beyond some depth the validation error rises again, an instance of overfitting. Depth is therefore a hyperparameter, and a feature transformation chosen beforehand can reach a lower validation error with far fewer model parameters.

deepnet.py, which recomputes what the entry states: the script block by block · open in ColabB-curvesThe daily maximum temperature at Krems an
der Donau (Austria) rises from January to July and falls again
towards December. The $365$ days of $2023$ form a training set
$\mathcal{D}^{(\rm train)}$ and the $366$ days of $2024$ a validation set $\mathcal{D}^{(\rm val)}$,
with the day of the year as the single feature of a day and
its maximum temperature as the label. A hypothesis that
is a linear function of the date can only rise or only fall over
the year. Its training error, the average squared error loss over
$\mathcal{D}^{(\rm train)}$, is $79.4$ squared degrees, barely below the $83.4$ of
the constant hypothesis that always predicts the
sample mean of the labels. A deep net replaces that
single raw feature by features that it computes
itself. A hidden layer of eight artificial neurons
turns the date into eight activations, and a linear
hypothesis of those eight numbers has a training error of
$16.9$ and a validation error of $23.7$ on $\mathcal{D}^{(\rm val)}$
(Fig. 1(a)).
pythondemos/deepnet.py
The model parameters ${\mathbf w}$ collect the weights in every matrix ${\mathbf W}^{(\ell)}$ and the offset terms in every vector ${\mathbf b}^{(\ell)}$, and a choice of ${\mathbf w}$ selects one hypothesis $h^{({\mathbf w})}$ of the net. They are learned by empirical risk minimization (ERM): gradient descent (GD) with a fixed learning rate lowers the training error $\widehat{L}\big(h^{({\mathbf w})}|\mathcal{D}^{(\rm train)}\big)$ by one gradient step after another. Each of them needs the gradient of the training error with respect to ${\mathbf w}$, and backpropagation computes it one layer at a time (Rumelhart et al., 1986).
Depth is a property of the net, not of a single hypothesis in it, and what a larger depth changes is the size of the hypothesis space. Fig. 1(b) shows the training error at Krems falling from $79.4$ at depth $1$ to $12.0$ at depth $5$, every hidden layer eight artificial neurons wide. The validation error does not follow it down. It is smallest at depth $2$, at $23.7$, and rises to $29.5$ at depth $5$, an instance of overfitting. Depth is therefore a hyperparameter, chosen on $\mathcal{D}^{(\rm val)}$ rather than fixed beforehand.
B-limitsThe depth-$2$ net lowers the validation error to $23.7$ from the $88.2$ of least squares on the raw feature. A feature transformation chosen beforehand reaches a lower validation error with a fifth of the model parameters. Least squares on the sine and cosine of the date at the annual period and at half of it needs five model parameters and has a validation error of $22.6$, against the $23.7$ of the net with its $25$. Two further costs come with the net. Its training error is not a convex function of ${\mathbf w}$ and has many local minima (Goodfellow et al., 2016, Sect. 8.2.2). GD therefore returns a different hypothesis from every initialization: five initializations of the depth-$2$ net give training errors between $16.9$ and $19.1$, and four of the five end above the smallest of them. The learned features also carry no reading of their own, where the sine and cosine name the annual cycle. An explainable artificial intelligence (XAI) method explains the trained net only by an approximation that cannot be fully faithful to it (Rudin, 2019, Sect. 2). A deep net is the answer where no such feature transformation can be written down beforehand: in image classification the first hidden layer delivers edges, the second corners and contours, and the third parts of objects (Goodfellow et al., 2016, Fig. 1.2).
Synonyms: deep neural network.
See also: artificial neural network, layer, activation, activation function, rectified linear unit, backpropagation, deep learning, feature learning, feature transformation, overfitting, hyperparameter, interpretable machine learning, convolutional neural network, large language model.
@misc{dictml_deepnet,
author = {Jung, Alexander},
editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {deep net},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-20},
url = {https://dictionaryofml.org/terms/deepnet.html}
}