Dictionary of Applied Machine Learning
Updated on 2026-09-18
Typeset PDF version — the authoritative form of this entry
Python demo — a script that recomputes what this entry states and prints one line per check
The projection of a vector onto a closed non-empty subset of a Euclidean space is a point in the subset that is closest to the vector in the Euclidean norm. If the subset is convex, this closest point is unique and the solution of a convex optimization problem. If the subset is a subspace, the projection is a linear map, namely the orthogonal projection onto the subspace. Every projection is idempotent, and a linear map of a Hilbert space into itself is an orthogonal projection if and only if the map is idempotent and self-adjoint. In machine learning (ML), projections enforce constraints on model parameters during training: projected gradient descent (projected GD) alternates gradient steps with projections onto the constraint set, e.g., the $\ell_{1}$-ball in an equivalent formulation of the least absolute shrinkage and selection operator (Lasso).
Training a machine learning (ML) model must sometimes keep the model parameters $\weights$ inside a constraint set, such as the set of vectors whose $\ell_{1}$-norm does not exceed a prescribed bound. A gradient step can move $\weights$ outside this set, and the result must then be replaced by a closest admissible vector. The projection formalizes this replacement.
P-defConsider a non-empty closed set $\paramspace \subseteq \reals^{\featuredim}$ in the $\featuredim$-dimensional Euclidean space. The projection $\projection{\paramspace}{\weights}$ of a vector $\weights \in \reals^{\featuredim}$ onto $\paramspace$ is \begin{equation} \nonumber \label{equ_def_proj_generic_dict} \projection{\paramspace}{\weights} = \argmin_{\weights' \in \paramspace} \normgeneric{\weights - \weights'}{2}\text{,} \end{equation} a vector in $\paramspace$ closest to $\weights$ in the Euclidean norm. The minimum exists for every non-empty closed set $\paramspace$, but several closest vectors can exist: for the two-point set $\paramspace = \big\{ (-1,0)^{\top}, (1,0)^{\top} \big\}$, both elements are closest to $(0,0.7)^{\top}$, and likewise to every other point on the vertical axis, which consists of exactly the points equidistant from the two elements. If $\paramspace$ is closed and convex, the closest point in the Euclidean norm is unique (Boyd and Vandenberghe, 2004, Sect. 8.1). The construction extends beyond the Euclidean space: a non-empty closed convex subset of a Hilbert space contains, for each point of the space, a unique closest point (Bauschke and Combettes, 2011, Thm. 3.14). If $\paramspace$ is a subspace, that is, a subset that contains all sums and scalar multiples of its elements (such as a line through the origin), the map $\weights \mapsto \projection{\paramspace}{\weights}$ is a linear map: it is the orthogonal projection onto $\paramspace$. The projection is distinct from the proximal operator: the proximal operator minimizes, over all of $\reals^{\featuredim}$, the sum of a convex function $f(\weights')$ and half the squared Euclidean distance to $\weights$, while the projection minimizes the distance alone, over the given set $\paramspace$.
P-fundEvery projection is idempotent: the vector $\projection{\paramspace}{\weights}$ lies in $\paramspace$, so its closest point in $\paramspace$ is itself and $\projection{\paramspace}{\projection{\paramspace}{\weights}} = \projection{\paramspace}{\weights}$. A linear map $P$ of a Hilbert space into itself is the orthogonal projection onto the range of $P$ if and only if $P$ is idempotent and self-adjoint (Axler, 2024, Sect. 7A, Exercise 20). For a matrix $\mP \in \reals^{\featuredim \times \featuredim}$ acting on the Euclidean space, idempotence reads as $\mP^{2} = \mP$ and self-adjointness amounts to the condition $\mP^{\top} = \mP$. Idempotence alone is not sufficient for a linear map to be an orthogonal projection: an idempotent linear map that is not self-adjoint is an oblique projection (Meyer, 2000, Sect. 5.9). For example, the matrix $\mP = \begin{pmatrix} 1 & 1 \\ 0 & 0 \end{pmatrix}$ on $\reals^{2}$ satisfies $\mP^{2} = \mP$ but $\mP^{\top} \neq \mP$. Its range is the horizontal axis, and it maps $\weights = (0,1)^{\top}$ to $\mP \weights = (1,0)^{\top}$, while the point of the axis closest to $\weights$ is the origin $(0,0)^{\top}$.
P-projgdIn ML, projections enforce constraints on model parameters
during training. Linear regression with the
constraint $\normgeneric{\weights}{1} \leq \tau$, for a prescribed
constant $\tau > 0$, is an equivalent
formulation of the least absolute shrinkage and selection operator (Lasso) (Tibshirani, 1996). This
constrained problem can be solved by projected gradient descent (projected GD)
(Bertsekas, 2016, Sect. 3.3). Each iteration
takes a gradient step on the training error
and then projects the result onto the $\ell_{1}$-ball
$\paramspace = \big\{ \weights' \in \reals^{\featuredim} : \normgeneric{\weights'}{1} \leq \tau \big\}$.
The projection step is what enforces the constraint: without it,
basic gradient descent (GD) on the training error can produce iterates with
$\normgeneric{\weights}{1} > \tau$, and the result solves the
unconstrained problem instead of the constrained one.
Since $\paramspace$ is not a subspace (scaling the vector
$(\tau,0)^{\top} \in \paramspace$ by $2$ yields a vector
outside the ball), this projection is not a
linear map but amounts to solving a convex
optimization problem (see Fig. 1).
@misc{dictml_projection,
author = {Jung, Alexander},
editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {projection},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-21},
url = {https://dictionaryofml.org/terms/projection.html}
}