\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amssymb}
\usepackage{algorithm,algpseudocode}
\usepackage{bm}
\usepackage{hyperref}
\usepackage{float}
\usepackage{subcaption}
\usepackage{enumitem}
\usepackage{xcolor}
\definecolor{darkgreen}{rgb}{0.0, 0.5, 0.0}
\definecolor{aaltoBlue}{RGB}{0, 51, 102}
\definecolor{pinegreen}{cmyk}{0.92,0,0.59,0.25}
\definecolor{royalblue}{cmyk}{1,0.50,0,0}
\definecolor{lavander}{cmyk}{0,0.48,0,0}
\definecolor{violet}{cmyk}{0.79,0.88,0,0}
\definecolor{lightblue}{RGB}{173, 216, 230}
\usepackage{tikz}
\usepackage{pgfplots}
\usetikzlibrary{calc,arrows.meta,positioning,fit,shapes.geometric,shapes.misc,intersections,patterns,decorations.pathreplacing,backgrounds,matrix}
\usepgfplotslibrary{fillbetween,groupplots}
\setcounter{secnumdepth}{-\maxdimen}
% Long unbreakable tokens (the ISO/IEC author names in citations, term names
% in the title) otherwise overflow the margin by a few points; a modest
% emergency stretch lets TeX loosen the paragraph instead.
\emergencystretch=2em
\usepackage{mleftright}
% Project macros used by this entry (auto-extracted).
\newcommand{\eigval}[1]{\lambda_{#1}}
\newcommand{\eigvalgen}{\lambda}
\newcommand{\eigvecCov}{\mathbf{u}}
\newcommand{\featuredim}{d}
\newcommand{\mA}[0]{{\bf A}}
\newcommand{\mC}[0]{{\bf C}}
\newcommand{\mI}{\mathbf{I}}
\newcommand{\reals}{\mathbb{R}}
\newcommand{\samplecovmtxgeneric}{\widehat{\mC}}
\newcommand{\samplemeanvecgeneric}{\widehat{\bm \mu}}
\newcommand{\samplesize}{m}
\newcommand{\vx}[0]{{\bf x}}

\begin{document}

\section{eigenvector}

\subsection*{Author and editors}
Alexander Jung\textsuperscript{*}, Department of Computer Science, Aalto University, Espoo, Finland (ORCID \mbox{\href{https://orcid.org/0000-0001-7538-0990}{0000-0001-7538-0990}}) \\
Editors: Konstantina Olioumtsevits (ORCID \mbox{\href{https://orcid.org/0000-0003-2697-8659}{0000-0003-2697-8659}}), Ekkehard Schnoor (ORCID \mbox{\href{https://orcid.org/0000-0001-8252-1499}{0000-0001-8252-1499}}), Aalto University, Espoo, Finland \\
\textsuperscript{*}Corresponding author: \href{mailto:alex.jung@aalto.fi}{alex.jung@aalto.fi}



\subsection{Abstract}
An eigenvector of a square matrix $\mA$ is a nonzero vector $\vx$ that $\mA$ maps to a multiple of itself, $\mA \vx = \eigvalgen \vx$. The number $\eigvalgen$ is the eigenvalue belonging to $\vx$, so multiplying by $\mA$ keeps $\vx$ on the line it spans and only rescales it. An eigenvector is determined only up to a nonzero factor: the eigenvectors belonging to one eigenvalue, together with the zero vector, form the nullspace of $\mA - \eigvalgen \mI$, a subspace of $\reals^{\featuredim}$. For a symmetric matrix, the eigenvectors can be chosen to form an orthonormal basis, and collecting them as the columns of one matrix gives the eigenvalue decomposition (EVD). The eigenvectors of the sample covariance matrix of a dataset are the directions that principal component analysis (PCA) keeps and discards. Keeping those belonging to the largest eigenvalues leaves a reconstruction error equal to the sum of the eigenvalues left out.

\subsection{Definition}
Every day of 2024 at the weather
		station Krems an der Donau (Austria) is a data point whose
		two features are the lowest and the highest temperature
		of that day in degrees Celsius. The sample covariance matrix
		$\samplecovmtxgeneric \in \reals^{2 \times 2}$ of these
		$\samplesize = 366$ days sends most directions of the plane to a
		different direction. Two directions are exceptions: the
		matrix returns each of them rescaled and otherwise
		unchanged, and these two are its eigenvectors. The days scatter
		most widely along the line spanned by the eigenvector with the
		larger scaling factor, and
		Fig.~\ref{fig_eigenvector_krems_dict} draws that line through
		the sample mean of the days together with the distance of
		each drawn day from it.
		\begin{figure}[H]
			\centering
			\begin{tikzpicture}
			\begin{axis}[
				width=0.7\linewidth, axis equal image,
				xlabel={lowest temperature of the day in $^{\circ}$C},
				ylabel={highest temperature of the day in $^{\circ}$C},
				axis lines=left, clip=false,
				unbounded coords=jump,
				enlarge x limits=0.12, enlarge y limits=0.08,
				tick label style={font=\small}, label style={font=\small},
				]
				\addplot[black, thick]
					table[col sep=comma, x=x1, y=x2]{eigenvector_line1.csv};
				\addplot[black, thick, dashed]
					table[col sep=comma, x=x1, y=x2]{eigenvector_line2.csv};
				\addplot[black, dotted, thick]
					table[col sep=comma, x=x1, y=x2]{eigenvector_residuals.csv};
				\addplot[only marks, mark=*, mark size=1.8pt, black]
					table[col sep=comma, x=x1, y=x2]{eigenvector_points_plot.csv};
				\addplot[only marks, mark=x, mark size=3pt, black, very thick]
					coordinates {(7.176,17.907)};
				\node[anchor=east, font=\small] at (axis cs:4.7,19.0)
					{$\samplemeanvecgeneric$};
				\node[anchor=north west, font=\small] at (axis cs:23.4,38.6)
					{$\eigvecCov^{(1)}$};
				\node[anchor=east, font=\small] at (axis cs:1.1,22.6)
					{$\eigvecCov^{(2)}$};
			\end{axis}
			\end{tikzpicture}
		\caption{Every $19$th of the $366$ days of 2024 at Krems an der
			Donau, $20$ days in all, drawn as filled circles. Each day
			is a data point with two features, its lowest
			and its highest temperature. The solid line runs through the
			sample mean $\samplemeanvecgeneric$ (cross) along the
			eigenvector $\eigvecCov^{(1)}$ of the
			sample covariance matrix of all $366$ days, and the dashed line
			runs through the same point along the eigenvector
			$\eigvecCov^{(2)}$. The dotted segments join the $20$ drawn
			days to the closest point of the solid line. Data generated
			by \texttt{pythondemos/eigenvector.py}
			\label{fig_eigenvector_krems_dict}}
		\end{figure}
		Formally, a nonzero vector $\vx \in \reals^{\featuredim}$
		is an eigenvector of a square matrix
		$\mA \in \reals^{\featuredim \times \featuredim}$ if there is a
		number $\eigvalgen \in \reals$ with
		\begin{equation}
			\label{equ_eigenvector_defining_dict}
			\mA \vx = \eigvalgen \vx \text{,}
		\end{equation}
		and that number is then the eigenvalue belonging to $\vx$
		(Strang, 2016, Sect.~6.1). The image $\mA \vx$ is a
		multiple of $\vx$, so it stays on the line that $\vx$ spans. The
		zero vector is excluded because it satisfies the defining
		equation \eqref{equ_eigenvector_defining_dict} for every
		$\eigvalgen \in \reals$ and so singles out no direction.
\par
		An eigenvector is determined only up to a nonzero factor: when
		$\vx$ satisfies the defining equation
		\eqref{equ_eigenvector_defining_dict}, so does $c \vx$ for every
		nonzero $c \in \reals$, with the same $\eigvalgen$. The
		eigenvectors belonging to one eigenvalue $\eigvalgen$,
		together with the zero vector, form the \emph{eigenspace}
		of $\mA$ for $\eigvalgen$, which is the nullspace of
		$\mA - \eigvalgen \mI$ and therefore a subspace of
		$\reals^{\featuredim}$ (Horn and Johnson, 2013, Def.~1.4.2).
		Here, $\mI$ is the identity matrix. Each eigenspace of the
		sample covariance matrix at Krems is a line through the origin, while
		the matrix $2 \mI$ has the single eigenvalue $2$,
		whose eigenspace is the whole plane. An implementation therefore
		returns eigenvectors of unit norm, and even then their
		sign stays arbitrary.
\par
		Symmetry restricts the eigenvectors further. For a real
		symmetric matrix, every eigenvalue is real and the
		eigenvectors can be chosen to form an orthonormal
		basis of $\reals^{\featuredim}$
		(Horn and Johnson, 2013, Thm.~4.1.5;
		Strang, 2016, Sect.~6.4). Collecting such a
		basis as the columns of one matrix gives the
		eigenvalue decomposition (EVD) of the symmetric matrix, which is then a
		spectral decomposition. The sample covariance matrix
		$\samplecovmtxgeneric$ is symmetric, which is why the two lines
		of Fig.~\ref{fig_eigenvector_krems_dict} meet at a right angle.
		Without symmetry the eigenvectors need not be orthogonal:
		the matrix with rows $(1.3, 0.8)$ and $(0.4, 0.9)$ has the
		eigenvectors $(2, 1)^{\top}$ and $(-1, 1)^{\top}$, whose
		inner product is $-1$.
\par
		Eigenvectors are what principal component analysis (PCA) computes. PCA, a method
		for dimensionality reduction, replaces the $\featuredim$ features of
		a data point by $\featuredim' < \featuredim$ numbers,
		chosen so that a linear map rebuilds the feature vector
		from them with the smallest average squared error loss over a
		dataset. That reconstruction error is the empirical risk of
		an instance of empirical risk minimization (ERM), and the eigenvectors of the
		sample covariance matrix belonging to its $\featuredim'$ largest
		eigenvalues attain its minimum, which equals the
		sum of the eigenvalues left out
		(Bishop, 2006, Sect.~12.1.2). For the two temperatures at
		Krems with $\featuredim' = 1$, the retained direction is the
		eigenvector $\eigvecCov^{(1)}$ drawn in
		Fig.~\ref{fig_eigenvector_krems_dict} and the discarded
		eigenvalue is $\eigval{2} = 7.9$ squared degrees, the
		average squared length of the dotted segments over all $366$
		days. The two eigenvalues add up to $138.9$ squared
		degrees, so the retained direction keeps all but $5.7$ percent
		of that total, while a line through the sample mean turned
		by $30$ degrees away from $\eigvecCov^{(1)}$ would leave $38.7$
		squared degrees behind.

\subsection{Cross References}
eigenvalue, EVD, spectral decomposition, matrix, vector, nullspace, subspace, symmetric matrix, orthonormal, basis, linear map, sample covariance matrix, PCA, dimensionality reduction

\subsection{References}
\begin{enumerate}
  \item Bishop (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media, New York, NY, USA.
  \item Horn and Johnson (2013). Matrix Analysis. 2nd ed., Cambridge Univ. Press, New York, NY, USA.
  \item Strang (2016). Introduction to Linear Algebra. 5th ed., Wellesley-Cambridge Press, Wellesley, MA, USA.
\end{enumerate}

\end{document}
