Dictionary of Applied Machine Learning: 114 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here

eigenvalue

Updated on 2026-10-29

Readiness 100/100

▶ Run the Python demo Open in Colab Typeset PDF LaTeX source Markdown Cite this entry

See also matrix eigenvalue decomposition gradient descent linear regression

An eigenvalue of a square matrix ${\mathbf A}$ is a number $\lambda$ for which some nonzero vector ${\mathbf x}$ satisfies ${\mathbf A}{\mathbf x} = \lambda{\mathbf x}$. Such a vector is an eigenvector of ${\mathbf A}$. Multiplying an eigenvector by the matrix rescales it without moving it off the line it spans. The eigenvalues of a diagonalizable matrix form the diagonal factor of its eigenvalue decomposition (EVD). An iterative method that applies an affine update, such as gradient descent (GD) for linear regression, converges to the unique fixed point of the update, for every initialization, exactly when each eigenvalue of the update matrix has magnitude less than one. The second-smallest eigenvalue of the Laplacian matrix of an undirected graph measures how well the graph is connected and underlies spectral clustering.

Figure produced by the Python demo of the entry eigenvalue
Figure written by the entry's Python demo eigenvalue.py, which recomputes what the entry states: the script block by block · open in Colab

Definition

P-defWhether gradient descent (GD) for linear regression converges is decided by the eigenvalues of a single square matrix. A square matrix ${\mathbf A}$ represents a linear map from $\mathbb{R}^{d}$ to $\mathbb{R}^{d}$; the image of a vector ${\mathbf x}$ under this map is the product ${\mathbf A}{\mathbf x}$. Multiplying a vector by a square matrix normally moves it off the line it spans. Some nonzero vectors are exceptions: each stays on the line it spans and is only rescaled. The factor by which such a vector is rescaled is an eigenvalue.

P-convFormally, a number $\lambda \in \mathbb{R}$ is an eigenvalue of a square matrix ${\mathbf A} \in \mathbb{R}^{d \times d}$, one with $d$ rows and $d$ columns, if ${\mathbf A} {\mathbf x} = \lambda {\mathbf x}$ for some nonzero vector ${\mathbf x} \in \mathbb{R}^{d}$; such a vector is an eigenvector of ${\mathbf A}$ (Strang, 2016). For an eigenvector, the image ${\mathbf A}{\mathbf x}$ stays on the line ${\mathbf x}$ spans, pointing the same way when $\lambda > 0$ and the opposite way when $\lambda < 0$ (see Fig. 1). Arranging $d$ linearly independent eigenvectors of a diagonalizable matrix as the columns of $\mathbf{V}$, and the corresponding eigenvalues in a diagonal matrix ${\boldsymbol \Lambda}$, yields the eigenvalue decomposition (EVD) ${\mathbf A} = \mathbf{V} {\boldsymbol \Lambda} \mathbf{V}^{-1}$.

Figure 1 of the entry eigenvalue
Figure 1: An eigenvector $\vx$ of $\mA$ (solid arrow) and its image $\mA \vx = \eigvalgen \vx$ (dashed arrow), here with eigenvalue $\eigvalgen = 1.8$. The image stays on the dotted line that $\vx$ spans, rescaled by $\eigvalgen$. A negative $\eigvalgen$ would point the image the other way along the same line
Eigenvalues can be used to study iterative machine learning (ML) methods with an update of the form \[ {\mathbf w}^{(t+1)} = {\mathbf M} {\mathbf w}^{(t)} + {\mathbf b} \text{.} \] An example is GD for linear regression, whose update operator $\mathcal{F}^{(\eta)} : {\mathbf w} \mapsto {\mathbf M} {\mathbf w} + {\mathbf b}$ is affine with iteration matrix ${\mathbf M} = \mathbf{I} - (2\eta/m) {\mathbf X}^{\top}{\mathbf X}$. Here, $\mathbf{I}$ is the identity matrix, $\eta$ the learning rate, $m$ the number of data points, and ${\mathbf X}$ the feature matrix of the training set (see linear regression). Such an iteration converges to the unique fixed point $\widehat{{\mathbf w}}$ of $\mathcal{F}^{(\eta)}$, for every initialization, if and only if the spectral radius $\rho$ of ${\mathbf M}$, the largest magnitude of an eigenvalue of ${\mathbf M}$, satisfies $\rho < 1$ (Horn and Johnson, 2013, Thm. 5.6.12). For a diagonalizable ${\mathbf M}$, the EVD shows why: in the basis of its eigenvectors, each iteration rescales the components of the error ${\mathbf w}^{(t)} - \widehat{{\mathbf w}}$ by the eigenvalues, so the error shrinks exactly when every eigenvalue has magnitude below one.

P-graphBeyond the convergence of iterative methods, eigenvalues are also used to measure the connectivity of an undirected graph $\mathcal{G}$ with the number of nodes $n$. The Laplacian matrix ${\mathbf L}^{(\mathcal{G})}$ of $\mathcal{G}$ is positive semi-definite (psd), with eigenvalues $0 = \lambda_{1} \leq \lambda_{2} \leq \ldots \leq \lambda_{n}$. The second-smallest eigenvalue $\lambda_{2}$, the algebraic connectivity of $\mathcal{G}$, quantifies connectivity: $\lambda_{2} > 0$ if and only if $\mathcal{G}$ is connected, and $\lambda_{2}$ close to zero indicates two clusters of nodes with few edges between them (Chung, 1997; Fiedler, 1973; Luxburg, 2007). Fig. 2 shows three edge sets on the same six nodes: adding edges increases $\lambda_{2}$ from $0$, for two clusters without a connecting edge, to $6$, for the complete graph.

Figure 2 of the entry eigenvalue
Figure 2: Three edge sets on the same six nodes, with increasing connectivity from left to right, and the second-smallest eigenvalue $\eigval{2}$ of the corresponding Laplacian matrix. (a) Two clusters without a connecting edge, so the graph is disconnected and $\eigval{2} = 0$. (b) One edge connects the clusters, so the graph is connected and $\eigval{2} \approx 0.44$ is positive but close to zero. (c) The complete graph, where every pair of nodes is connected, so the graph is connected and $\eigval{2} = 6$ attains the largest possible value, the number of nodes $\nrnodes = 6$, since no eigenvalue of the Laplacian matrix of a graph with $\nrnodes$ nodes exceeds $\nrnodes$. Data generated by pythondemos/eigenvalue.py
The eigenvector corresponding to $\lambda_{2}$, referred to as the Fiedler vector, can be used to partition $\mathcal{G}$ into two clusters: each node is assigned according to the sign of its own component of the Fiedler vector (Fiedler, 1975; Spielman, 2025).

See also: matrix, eigenvector, eigenvalue decomposition, spectral radius, gradient descent, linear regression, contractive operator, fixed-point iteration, Laplacian matrix, algebraic connectivity, Fiedler vector, spectral clustering.

References

  1. Strang (2016). Introduction to Linear Algebra. Wellesley-Cambridge Press.
  2. Horn and Johnson (2013). Matrix Analysis. Cambridge Univ. Press. doi.org/10.1017/cbo9781139020411
  3. Chung (1997). Spectral Graph Theory. American Mathematical Society.
  4. Fiedler (1973). Algebraic connectivity of graphs. Czechoslovak Mathematical Journal.
  5. Luxburg (2007). A tutorial on spectral clustering. Statist. Comput.. doi.org/10.1007/s11222-007-9033-z
  6. Fiedler (1975). A property of eigenvectors of nonnegative symmetric matrices and its application to graph theory. Czechoslovak Mathematical Journal.
  7. Spielman (2025). Spectral and Algebraic Graph Theory (Incomplete Draft). Ebook. cs-www.cs.yale.edu/homes/spielman/sagt

Cite this entry

@misc{dictml_eigenvalue,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {eigenvalue},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-04},
  url = {https://dictionaryofml.org/terms/eigenvalue.html}
}