\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amssymb}
\usepackage{algorithm,algpseudocode}
\usepackage{bm}
\usepackage{hyperref}
\usepackage{float}
\usepackage{subcaption}
\usepackage{enumitem}
\usepackage{xcolor}
\definecolor{darkgreen}{rgb}{0.0, 0.5, 0.0}
\definecolor{aaltoBlue}{RGB}{0, 51, 102}
\definecolor{pinegreen}{cmyk}{0.92,0,0.59,0.25}
\definecolor{royalblue}{cmyk}{1,0.50,0,0}
\definecolor{lavander}{cmyk}{0,0.48,0,0}
\definecolor{violet}{cmyk}{0.79,0.88,0,0}
\definecolor{lightblue}{RGB}{173, 216, 230}
\usepackage{tikz}
\usepackage{pgfplots}
\usetikzlibrary{calc,arrows.meta,positioning,fit,shapes.geometric,shapes.misc,intersections,patterns,decorations.pathreplacing,backgrounds,matrix}
\usepgfplotslibrary{fillbetween,groupplots}
\setcounter{secnumdepth}{-\maxdimen}
% Long unbreakable tokens (the ISO/IEC author names in citations, term names
% in the title) otherwise overflow the margin by a few points; a modest
% emergency stretch lets TeX loosen the paragraph instead.
\emergencystretch=2em
\usepackage{mleftright}
% Project macros used by this entry (auto-extracted).
\newcommand{\covmtxgeneric}{\mC}
\newcommand{\eigval}[1]{\lambda_{#1}}
\newcommand{\eigvecCov}{\mathbf{u}}
\newcommand{\expect}{\mathbb{E} }
\newcommand{\featuredim}{d}
\newcommand{\featureidx}{j}
\newcommand{\featurevec}{\vx}
\newcommand{\innerprod}[2]{\langle #1, #2 \rangle}
\newcommand{\iteridx}{t}
\newcommand{\lossfunc}[2]{L\left(#1,#2 \right)}
\newcommand{\mC}[0]{{\bf C}}
\newcommand{\mI}{\mathbf{I}}
\newcommand{\mW}[0]{{\bf W}}
\newcommand{\meanvecgeneric}{{\bm \mu}}
\newcommand{\mvneval}[3]{\mathcal{N}\left(#1 \,;\, #2, #3\right)}
\newcommand{\mvnormal}[2]{\mathcal{N}\left(#1,#2\right)}
\newcommand{\perturbation}[1]{{\bm \varepsilon}^{(#1)}}
\newcommand{\probdist}{\mathbb{P}}
\newcommand{\reals}{\mathbb{R}}
\newcommand{\samplecovmtxgeneric}{\widehat{\mC}}
\newcommand{\sampleidx}{r}
\newcommand{\samplemeanvecgeneric}{\widehat{\bm \mu}}
\newcommand{\samplesize}{m}
\newcommand{\vw}[0]{{\bf w}}
\newcommand{\vx}[0]{{\bf x}}
\newcommand{\vz}[0]{{\bf z}}
\newcommand{\weights}{\vw}

\begin{document}

\section{probabilistic principal component analysis (probabilistic PCA)}

\subsection*{Author and editors}
Alexander Jung\textsuperscript{*}, Department of Computer Science, Aalto University, Espoo, Finland (ORCID \mbox{\href{https://orcid.org/0000-0001-7538-0990}{0000-0001-7538-0990}}) \\
Editors: Konstantina Olioumtsevits (ORCID \mbox{\href{https://orcid.org/0000-0003-2697-8659}{0000-0003-2697-8659}}), Ekkehard Schnoor (ORCID \mbox{\href{https://orcid.org/0000-0001-8252-1499}{0000-0001-8252-1499}}), Aalto University, Espoo, Finland \\
\textsuperscript{*}Corresponding author: \href{mailto:alex.jung@aalto.fi}{alex.jung@aalto.fi}



\subsection{Abstract}
Probabilistic principal component analysis (probabilistic
		PCA) is a probabilistic model for the feature vectors of a
		dataset: a low-dimensional Gaussian random vector,
		the code, is mapped into the feature space by a
		matrix, and isotropic Gaussian noise is added. The
		probability distribution of the feature vector is then a multivariate normal distribution
		whose covariance matrix is a matrix of rank
		$\featuredim'$ plus a multiple of the identity matrix, so
		the model gives a probability density function (pdf) value to every point of the
		feature space. That value is what principal component analysis (PCA) does not
		deliver: a data point outside the training set can be
		ranked by how probable it is, and features missing from
		a feature vector are filled in by an expectation. The
		maximum likelihood estimate of the model parameters has a
		closed form in the eigenvalues and eigenvectors
		of the sample covariance matrix: the columns of the matrix span
		the subspace that PCA keeps, and the noise
		variance is the average of the discarded
		eigenvalues. The expectation–maximization (EM) algorithm computes the
		same estimate as a fixed-point iteration, without an eigenvalue decomposition (EVD).
		The posterior mean of the code pulls the PCA coordinates
		toward the origin, and becomes the orthogonal projection
		on the subspace as the noise variance tends to
		zero, which recovers PCA. The isotropic noise is the
		model's main restriction: it makes the learned subspace
		depend on the units of the features, and factor
		analysis gives each measurement its own noise variance
		instead.

\subsection{Definition}
A weather
		station at Krems an der Donau (Austria) records eight numbers
		for each day of 2024: the lowest, highest, and mean air
		temperature, precipitation, sunshine duration, humidity, air
		pressure, and wind speed. Principal component analysis (PCA) replaces the eight by two,
		few enough for a scatterplot, and says nothing about how
		probable a day is: 14 September 2024, which brought $81.7$ mm
		of rain, is placed in the plane like any other day. Probabilistic
		PCA keeps the subspace that PCA delivers and adds a
		probability distribution over the eight numbers, given by a probability density function (pdf) on
		the feature space. The rainy day then carries a density
		value far below that of every other day of the year.
\par
		Probabilistic PCA is a latent variable probabilistic model.
		Each data point of a dataset carries a feature vector
		$\featurevec^{(\sampleidx)} \in \reals^{\featuredim}$, for
		$\sampleidx = 1, \ldots, \samplesize$, with $\featuredim = 8$
		and $\samplesize = 366$ for the days of 2024. The model builds
		it from a low-dimensional vector $\vz^{(\sampleidx)} \in
		\reals^{\featuredim'}$, the code, whose length
		$\featuredim' < \featuredim$ is a hyperparameter:
		\begin{equation}
			\label{equ_ppca_model_dict}
			\featurevec^{(\sampleidx)} = \mW \vz^{(\sampleidx)} +
			\meanvecgeneric + \perturbation{\sampleidx} \text{, for }
			\sampleidx = 1, \ldots, \samplesize \text{.}
		\end{equation}
		Here, $\mW \in \reals^{\featuredim \times \featuredim'}$ is a
		matrix and $\meanvecgeneric \in \reals^{\featuredim}$ a
		mean vector. The codes and the noise vectors
		of the model \eqref{equ_ppca_model_dict} are realizations
		of independent random vectors, $\vz^{(\sampleidx)} \sim
		\mvnormal{\mathbf{0}}{\mI}$ and $\perturbation{\sampleidx} \sim
		\mvnormal{\mathbf{0}}{\sigma^{2} \mI}$, with the identity
		matrix $\mI$ and a noise variance $\sigma^{2} > 0$.
		The noise is isotropic: it has the same variance in every
		direction and no covariance between features. The
		matrix $\mW$ maps a code to a feature vector, the role
		the decoder plays in PCA, and the
		model parameters are $\mW$, $\meanvecgeneric$, and
		$\sigma^{2}$.
\par
		Integrating the code out of the model
		\eqref{equ_ppca_model_dict} leaves a multivariate normal distribution for the
		feature vector itself,
		\begin{equation}
			\label{equ_ppca_marginal_dict}
			\featurevec^{(\sampleidx)} \sim
			\mvnormal{\meanvecgeneric}{\covmtxgeneric} \text{, where }
			\covmtxgeneric = \mW \mW^{\top} + \sigma^{2} \mI \text{.}
		\end{equation}
		The covariance matrix $\covmtxgeneric$ of the multivariate normal distribution
		\eqref{equ_ppca_marginal_dict} is a matrix of rank
		at most $\featuredim'$ plus a multiple of the identity
		matrix. That is the one restriction separating
		probabilistic PCA from an unconstrained multivariate normal distribution: the
		variance along each of the $\featuredim'$ retained
		directions is modeled on its own, while every direction
		orthogonal to all of them is given the same
		variance $\sigma^{2}$. Because the model
		\eqref{equ_ppca_model_dict} states how a data point comes
		about, it is a generative model: drawing a code and adding
		noise produces a new day. Fig.~\ref{fig_ppca_density_dict}
		draws the probability distribution \eqref{equ_ppca_marginal_dict} learned
		from the days at Krems taken with two features, the
		lowest and the highest temperature of the day, as a level curve
		one standard deviation out from the mean vector. For a
		code of one number, the subspace that PCA keeps is
		a straight line, and the probability distribution fills out an ellipse
		around it.
		\begin{figure}[H]
			\centering
			\begin{tikzpicture}
			\begin{axis}[
				width=0.62\linewidth, axis equal image,
				xlabel={lowest temperature of the day in
					$^{\circ}$C, centered},
				ylabel={highest temperature in $^{\circ}$C, centered},
				axis lines=left, clip=false,
				enlarge x limits=0.14, enlarge y limits=0.10,
				tick label style={font=\small}, label style={font=\small},
				legend style={draw=none, fill=none, font=\footnotesize,
					at={(0.5,-0.26)}, anchor=north},
				legend cell align=left,
				]
				\addplot[only marks, mark=*, mark size=1.6pt, black]
					table[col sep=comma, x=x1, y=x2]{ppca_points_plot.csv};
				\addlegendentry{twenty days of 2023 (filled circles)}
				\addplot[only marks, mark=triangle, mark size=2.6pt, black]
					table[col sep=comma, x=x1, y=x2]{ppca_draws_plot.csv};
				\addlegendentry{twenty draws from the model (open triangles)}
				\addplot[black, thick, no marks]
					table[col sep=comma, x=x1, y=x2]{ppca_ellipse.csv};
				\addlegendentry{level curve of the density (solid curve)}
				\addplot[black, dashed, no marks]
					table[col sep=comma, x=x1, y=x2]{ppca_axis.csv};
				\addlegendentry{subspace kept by PCA (dashed line)}
			\end{axis}
			\end{tikzpicture}
		\caption{Twenty of the $365$ days of 2023 at Krems (every
			$19$th day), each a data point with the lowest and the
			highest temperature of the day as its two features,
			centered (filled circles). The solid curve is the level
			curve one standard deviation out of the pdf
			\eqref{equ_ppca_marginal_dict} learned from all $365$ days
			with $\featuredim' = 1$, and the dashed line marks the
			subspace that PCA keeps. The open triangles are
			twenty feature vectors drawn from the learned model.
			Data generated by \texttt{pythondemos/ppca.py}
			\label{fig_ppca_density_dict}}
		\end{figure}
		What the probability distribution \eqref{equ_ppca_marginal_dict} delivers
		and PCA does not is a number for every point of the
		feature space. A day that did not belong to the
		training set receives a density value, so days can be ranked
		by how probable the model finds them. With the
		model parameters learned from the $365$ days of 2023 and
		$\featuredim' = 3$, the least probable day of 2024 at Krems is
		14 September, where the logarithm of the pdf
		\eqref{equ_ppca_marginal_dict} is $-440.8$, against $-8.5$ for
		the median day and $-105.6$ for the next lowest
		(Fig.~\ref{fig_ppca_year_dict}). Its $81.7$ mm of rain are
		$20.4$ sample standard deviations above the 2023 average of $1.5$
		mm. Ranking data points by their density value is one
		route to anomaly detection, and it is closed to PCA,
		which returns coordinates and no probability.
		\begin{figure}[H]
			\centering
			\begin{tikzpicture}
			\begin{axis}[
				width=0.72\linewidth, height=0.40\linewidth,
				xlabel={day of the year 2024},
				ylabel={logarithm of the density},
				axis lines=left, xmin=0, xmax=372,
				ymin=-480, ymax=40,
				tick label style={font=\small}, label style={font=\small},
				legend style={draw=none, fill=none, font=\footnotesize,
					at={(0.5,-0.30)}, anchor=north},
				legend cell align=left,
				]
				\addplot[black, no marks]
					table[col sep=comma, x=day, y=logdens]{ppca_logdens.csv};
				\addlegendentry{each day of 2024 (solid curve)}
				\addplot[only marks, mark=square*, mark size=2.2pt, black]
					coordinates {(258,-440.8)};
				\addlegendentry{14 September 2024 (filled square)}
			\end{axis}
			\end{tikzpicture}
		\caption{The logarithm of the pdf
			\eqref{equ_ppca_marginal_dict} at each of the $366$ days of
			2024 at Krems, under the model parameters learned from
			the $365$ days of 2023 with $\featuredim' = 3$ and all
			eight measurements (solid curve). The filled square marks
			14 September 2024, the day of $81.7$ mm of rain, at
			$-440.8$, while the median day of the year is at
			$-8.5$, where the solid curve runs for most of the year.
			Data generated by \texttt{pythondemos/ppca.py}
			\label{fig_ppca_year_dict}}
		\end{figure}
		The model parameters are learned by maximum likelihood, and
		the estimate has a closed form. The eigenvalues of the
		sample covariance matrix $\samplecovmtxgeneric$ of the
		training set, ordered as $\eigval{1} \geq \cdots \geq
		\eigval{\featuredim} \geq 0$, come with orthonormal
		eigenvectors $\eigvecCov^{(1)}, \ldots,
		\eigvecCov^{(\featuredim)}$. The estimate of the mean
		vector is the sample mean $\samplemeanvecgeneric$,
		the estimate of the noise variance is
		\begin{equation}
			\label{equ_ppca_sigma_dict}
			\widehat{\sigma}^{2} = \frac{1}{\featuredim - \featuredim'}
			\sum_{\featureidx = \featuredim'+1}^{\featuredim}
			\eigval{\featureidx} \text{,}
		\end{equation}
		and the estimate $\widehat{\mW}$ has the columns
		$\sqrt{\eigval{\featureidx} - \widehat{\sigma}^{2}} \,
		\eigvecCov^{(\featureidx)}$, for $\featureidx = 1, \ldots,
		\featuredim'$ (Bishop, 2006, Sect.~12.2.1, Eqs.~(12.45)
		and~(12.46); Tipping and Bishop, 1999, Sect.~3.2, Eqs.~(7)
		and~(8)). The average
		\eqref{equ_ppca_sigma_dict} of the discarded eigenvalues
		is the variance the retained directions leave
		unexplained, spread evenly over the $\featuredim -
		\featuredim'$ directions that carry it. The columns of
		$\widehat{\mW}$ span the subspace that PCA keeps,
		the one spanned by the first $\featuredim'$
		eigenvectors of $\samplecovmtxgeneric$. Rotating those
		columns among themselves leaves the covariance matrix
		$\covmtxgeneric$ unchanged, so maximum likelihood
		determines the subspace and not the individual columns.
\par
		The expectation–maximization (EM) algorithm reaches the same estimate without
		an eigenvalue decomposition (EVD) of $\samplecovmtxgeneric$, which is the cheaper
		route when $\featuredim$ is large and few directions are
		wanted (Bishop, 2006, Sect.~12.2.2). Collecting the
		model parameters $\mW$ and $\sigma^{2}$ in $\weights$, one
		iteration is a fixed-point iteration
		$$\weights^{(\iteridx+1)} = T \big( \weights^{(\iteridx)} \big)
		\text{,}$$
		whose operator $T$ re-estimates $\mW$ from
		$\samplecovmtxgeneric$ and the posterior distribution of the
		codes, then sets $\sigma^{2}$ to the variance the new
		$\mW$ leaves unexplained (Tipping and Bishop, 1999, Appendix~B, Eqs.~(29)
		and~(30)). No iteration decreases the
		likelihood function, and the fixed points of $T$ are the
		stationary points of that likelihood function. The closed-form
		estimate is the one among them at which the likelihood function
		attains its maximum; the stationary points built from
		eigenvectors other than the leading $\featuredim'$ are
		saddle points (Tipping and Bishop, 1999, Appendix~A.3).
\par
		In the vocabulary of empirical risk minimization (ERM), probabilistic PCA changes two
		of the three parts of PCA and keeps the third. The
		hypothesis space gains one number. PCA searches over the
		products of an encoder and a decoder, which are
		the matrices of rank at most $\featuredim'$.
		Probabilistic PCA searches over the multivariate normal distributions
		\eqref{equ_ppca_marginal_dict}, whose covariance matrices are such
		a matrix plus $\sigma^{2} \mI$. The columns of $\mW$
		need not be orthonormal: those of $\widehat{\mW}$ have
		squared lengths $\eigval{\featureidx} - \widehat{\sigma}^{2}$,
		for $\featureidx = 1, \ldots, \featuredim'$, so they fix the
		variance along each retained direction and not only the
		subspace. The loss function is the negative logarithm
		of the pdf of the probability distribution $\probdist$
		\eqref{equ_ppca_marginal_dict}, in place of the
		squared error loss of PCA,
		$$\lossfunc{\featurevec}{\probdist} = - \ln
		\mvneval{\featurevec}{\meanvecgeneric}{\covmtxgeneric}
		\text{,}$$
		so minimizing the empirical risk is maximum likelihood. The
		third part is unchanged: the EVD of
		$\samplecovmtxgeneric$ that solves PCA delivers this
		minimum in closed form as well, and the EM
		algorithm is the route that avoids it. The two
		loss functions meet as the noise variance tends to
		zero: $2 \sigma^{2}$ times the empirical risk tends to the
		minimum reconstruction error
		$\sum_{\featureidx = \featuredim' + 1}^{\featuredim}
		\eigval{\featureidx}$ of PCA
		(Tipping and Bishop, 1999, Appendix~A.2, Eq.~(17)). That sum is
		$1.636$ for the eight standardized measurements of the $365$
		days of 2023 with $\featuredim' = 3$. Multiplied by
		$2 \sigma^{2}$, the empirical risk reaches $0.979$ of it at
		$\sigma^{2} = 10^{-2}$, $0.998$ at $10^{-4}$, and $1.000$ at
		$10^{-6}$.
\par
		A data point is placed in the subspace through the
		posterior distribution of its code. The posterior mean of the code
		is
		\begin{equation}
			\label{equ_ppca_postmean_dict}
			\expect \big\{ \vz \mid \featurevec \big\} = \big(
			\widehat{\mW}^{\top} \widehat{\mW} + \widehat{\sigma}^{2}
			\mI \big)^{-1} \widehat{\mW}^{\top} \big( \featurevec -
			\samplemeanvecgeneric \big) \text{,}
		\end{equation}
		whose $\featureidx$-th entry, for the closed-form estimate, is
		the $\featureidx$-th PCA coordinate
		$\innerprod{\eigvecCov^{(\featureidx)}}{\featurevec -
		\samplemeanvecgeneric}$ multiplied by
		$\sqrt{\eigval{\featureidx} - \widehat{\sigma}^{2}} \big/
		\eigval{\featureidx}$, a factor below one, for $\featureidx =
		1, \ldots, \featuredim'$. The code is therefore pulled toward
		the origin, by the factors $0.50$, $0.71$, and $0.78$ for the
		three retained directions at Krems. As $\widehat{\sigma}^{2}$
		tends to zero each factor tends to one, and the posterior mean
		\eqref{equ_ppca_postmean_dict} becomes the
		orthogonal projection of the feature vector on the
		subspace spanned by the first $\featuredim'$
		eigenvectors, which is what PCA computes
		(Bishop, 2006, Sect.~12.2.1, Eq.~(12.50);
		Tipping and Bishop, 1999, Sect.~3.4, Eq.~(9)). At that limit
		the covariance matrix $\covmtxgeneric$ is singular
		and the pdf ceases to exist, so the limit that recovers
		PCA is the one that gives up the probability distribution.
\par
		A probability distribution over the feature space also handles
		missing data. Conditioned on the entries of a
		feature vector that were recorded, the probability distribution
		\eqref{equ_ppca_marginal_dict} is again Gaussian in the entries
		that were not, and its expectation is the filled-in
		value. Deleting the highest temperature of each of the $366$
		days of 2024 and filling it in this way, from the
		model parameters learned on 2023, leaves a mean squared error (MSE) whose
		square root is $2.28$ degrees, against $9.21$ degrees for
		filling in the 2023 average instead. The projection that
		PCA applies needs every entry of the feature vector and
		offers no such rule, while the same conditional reading lets
		the model parameters be learned from feature vectors with
		entries missing (Tipping and Bishop, 1999, Sect.~4.1).
\par
		The model makes two assumptions, and the days at Krems show
		what each one costs. The noise is isotropic, so every direction
		orthogonal to the retained subspace is given the
		same variance $\widehat{\sigma}^{2} = 0.33$ for the eight
		standardized measurements with $\featuredim' = 3$. The mean air
		temperature of a day is the average of its lowest and its
		highest to within $0.05$ degrees on every day of 2023. The
		sample standard deviation of the days along the last
		eigenvector of $\samplecovmtxgeneric$ is therefore
		$0.0027$, where the model expects $\widehat{\sigma} = 0.57$.
		The isotropic noise also makes the learned subspace
		depend on the units of the measurements: multiplying the
		precipitation reading by ten turns it by $51$ degrees, the
		largest angle between the subspace before and after the
		rescaling. That is why the eight measurements
		are standardized before the model parameters are learned.
		Factor analysis differs from probabilistic PCA only in giving
		the noise a covariance matrix that is a diagonal matrix, one
		variance per measurement, and a rescaling of one
		measurement leaves it unchanged
		(Bishop, 2006, Sect.~12.2.4;
		Tipping and Bishop, 1999, Sects.~2.1 and~3.3). With its
		model parameters learned from the same $365$ days of 2023, it
		raises the median logarithm of the pdf over the
		$366$ days of 2024 from $-8.5$ to $-3.2$, and what it gives up
		is the closed form (Tipping and Bishop, 1999, Sect.~2.1).
\par
		The second assumption is the Gaussian shape of the pdf
		\eqref{equ_ppca_marginal_dict}, on which the ranking of days by
		density value depends. If the days of 2024 were
		realizations of i.i.d. random vectors with the
		learned multivariate normal distribution, the quadratic form in the exponent of
		that pdf would be a realization of a chi-squared
		random variable (RV) with eight degrees of freedom. Its $99$th percentile
		is $20.1$, which $3.7$ of the $366$ days would exceed on
		average. Twenty-six do, as do $16$ of the $365$ days of 2023
		the model parameters were learned from. For 14 September
		2024, the quadratic form is $870.6$, which the same chi-squared
		RV exceeds with probability $1.2 \cdot 10^{-182}$,
		and that day occurred. The density value therefore ranks days,
		and it does not say how often a day of that kind occurs.
		Where one multivariate normal distribution is too simple, the standard extension
		is a mixture of probabilistic PCA models, a Gaussian mixture model (GMM) whose
		component covariance matrices are each of the form $\mW \mW^{\top} +
		\sigma^{2} \mI$ (Tipping and Bishop, 1999, Sect.~4.2). What the two
		assumptions buy in return is a probability distribution over the whole
		feature space in closed form: it gives a density value to
		a day outside the training set and a filled-in value to a
		feature that was not recorded.

\subsection{Cross References}
PCA, robust PCA, sparse PCA, dimensionality reduction, ERM, probabilistic model, multivariate normal distribution, maximum likelihood, EM, latent variable, generative model, sample covariance matrix, missing data, anomaly detection, GMM

\subsection{References}
\begin{enumerate}
  \item Bishop (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media, New York, NY, USA.
  \item Tipping and Bishop (1999). Probabilistic Principal Component Analysis. J. Roy. Statist. Soc.: Ser. B (Statist. Methodology), 61(3), pp.~611--622.
\end{enumerate}

\end{document}
