The gradient of a real-valued function is a vector that determines
the local linear approximation of the function.
The entries of the gradient are the partial derivatives of
the function. However, the existence of partial derivatives does
not imply the existence of a gradient, unless the function is convex.
Geometrically, a nonzero gradient is orthogonal to the level sets and points in the
direction of steepest ascent. In machine learning (ML), gradients of the empirical risk minimization (ERM)
objective drive gradient descent (GD) methods and are computed for deep networks
by backpropagation.
Definition
The gradient of a real-valued function
$f: \reals^{\featuredim} \rightarrow \reals: \weights \mapsto f(\weights)$
determines a local linear approximation of $f$. Formally, the gradient
of $f$ at a point $\weights' \in \reals^{\featuredim}$ is a vector
$\vg \in \reals^{\featuredim}$ such that
\[
\lim_{\weights \rightarrow \weights'}
\frac{f(\weights) - \big( f(\weights') + \vg^{\top} (\weights - \weights') \big)}{\normgeneric{\weights - \weights'}{2}}
= 0 \text{.}
\]
If such a vector exists, it is unique and denoted by
$\nabla f(\weights')$ or $\nabla f(\weights) \big|_{\weights'}$
(Rudin, 1976, Ch. 9). The function
$f(\weights') + \big(\nabla f(\weights')\big)^{\top} (\weights - \weights')$
is the local linear approximation of $f$ at $\weights'$
(see Fig. 1). The entries of
the gradient are the partial derivatives of $f$,
$\nabla f(\weights) = \big( {\partial f}/{\partial \weight_{1}}, \ldots, {\partial f}/{\partial \weight_{\featuredim}} \big)^{\top}$.
Figure 1: A differentiable function
$f: \reals \rightarrow \reals$ and its local
linear approximation at a point $\weights'$.
The slope triangle shows the displacement
$\weights - \weights'$ and the resulting change
$\big(\nabla f(\weights')\big)^{\top} (\weights - \weights')$
of the local linear approximation
In general, the existence of all partial derivatives of $f$
at a point does not guarantee that the gradient exists there
(Rudin, 1976, Ch. 9). For a
convex function $f: \reals^{\featuredim} \rightarrow \reals$, however, it does:
if the partial derivatives
$\partial f / \partial \weight_{\featureidx}$, for
$\featureidx = 1, \ldots, \featuredim$, exist at a point
$\weights'$, then $f$ is differentiable at $\weights'$, and
the gradient $\nabla f(\weights')$ is the vector of these
partial derivatives (Rockafellar, 1970, Sect. 25).
The definition carries over to a real-valued function
$f: \hilbertspace \rightarrow \reals$ on a Hilbert space
$\hilbertspace$: the inner product
$\innerprod{\vg}{\weights - \weights'}$ of $\hilbertspace$
replaces the term $\vg^{\top} (\weights - \weights')$, and the
norm of $\hilbertspace$ replaces the Euclidean norm
(Bauschke and Combettes, 2017).
The gradient has a geometric interpretation. At every point where
$f$ is differentiable and $\nabla f \neq \mathbf{0}$, the
gradient is orthogonal to the level
set of $f$ through that point, and it points in the direction of steepest
ascent of $f$. The negative gradient $-\nabla f(\weights')$ points in
the direction of steepest descent (see Fig. 2;
Boyd and Vandenberghe, 2004, Sect. 9.4). At a local minimum of a
differentiable function $f: \reals^{\featuredim} \rightarrow \reals$,
there cannot be any direction of descent and consequently the gradient
must vanish (see zero-gradient condition).
Figure 2: Level sets $f(\weights) = \text{const.}$ of a
differentiable function $f$ of two variables, with a
minimum at $\widehat{\weights}$. The gradient
$\nabla f(\weights')$ (solid) is orthogonal to the level set
through $\weights'$ and points in the direction of steepest
ascent. The negative gradient $-\nabla f(\weights')$ (dashed)
points in the direction of steepest descent, toward smaller
values of $f$. Gradient descent (GD) repeatedly steps along this direction.
Data generated by pythondemos/gradient.py
Gradients are instrumental for the training of machine learning (ML) models.
They guide the update of model parameters in order to minimize
the incurred loss on a training set
$\trainset = \big\{ \big(\featurevec^{(\sampleidx)}, \truelabel^{(\sampleidx)}\big) \big\}_{\sampleidx=1}^{\samplesize}$.
As a case in point, linear regression minimizes the objective function
\[
f(\weights) \defeq \frac{1}{\samplesize}
\sum_{\sampleidx=1}^{\samplesize}
\big( \truelabel^{(\sampleidx)} -
\weights^{\top} \featurevec^{(\sampleidx)} \big)^{2} \text{.}
\]
This function is convex and differentiable, with gradient
\[
\nabla f(\weights) = - \frac{2}{\samplesize}
\sum_{\sampleidx=1}^{\samplesize}
\big( \truelabel^{(\sampleidx)} -
\weights^{\top} \featurevec^{(\sampleidx)} \big)
\featurevec^{(\sampleidx)} \text{.}
\]
For a deep artificial neural network (ANN) with differentiable activation functions,
the empirical risk minimization (ERM) objective function is also differentiable. The
gradient of this objective function can be computed by backpropagation
(Goodfellow et al., 2016, Sect. 6.5; Rumelhart et al., 1986).
