# Gaussian mixture model (GMM) *Entry of the [Dictionary of Applied Machine Learning](https://dictionaryofml.org) (course edition), by Alexander Jung, Aalto University. CC BY 4.0. Updated 2026-11-10. [HTML](https://dictionaryofml.org/terms/gmm.html) · [typeset PDF](https://dictionaryofml.org/terms/gmm.pdf) · [LaTeX](https://dictionaryofml.org/terms/gmm.tex) · [Python demo](https://dictionaryofml.org/terms/gmm.py) · [notebook](https://dictionaryofml.org/terms/gmm.ipynb)* Math is LaTeX with the book's macros expanded, so no notation package is needed to read it. ## Abstract A Gaussian mixture model (GMM) is a probabilistic model for the generation of data points with numeric feature vectors. Each data point is generated by first drawing a latent cluster index and then drawing the feature vector from the multivariate normal distribution of that cluster. The resulting marginal distribution is a weighted sum of multivariate normal distributions. The model parameters are chosen to maximize the likelihood function of a dataset. The resulting maximum likelihood optimization problem is an instance of empirical risk minimization using a specific choice for the loss function. This optimization problem is non-convex and typically solved using the expectation–maximization algorithm. This algorithm consists of a fixed-point iteration that never increases the empirical risk. ## Definition A photograph of a mountain is cut into small square patches, and each patch is described by two numbers: how green and how blue it is on average (Fig. 1). Vegetation, mountain and haze, and bright sky form three clusters of such patches, which are data points characterized by a feature vector ${\mathbf x} \in \mathbb{R}^{d}$. A GMM is a probabilistic model for the generation of such data points (Bishop, 2006; Murphy, 2012). A GMM assumes that each data point is generated by first drawing a latent cluster index $I \in \{1,\,\ldots,\,k\}$ according to cluster probabilities $$ \mathbb{P}\left(I=c\right) = p_{c}, \qquad \sum_{c=1}^{k} p_{c}=1\text{.} $$ Given the choice $I=c$, the feature vector ${\mathbf x}$ is drawn from a multivariate normal distribution $\mathbb{P}^{(c)}= \mathcal{N}\left({\boldsymbol \mu}^{(c)},\mathbf{C}^{(c)}\right)$, with mean ${\boldsymbol \mu}^{(c)}$ and covariance matrix $\mathbf{C}^{(c)}$. The resulting marginal distribution of ${\mathbf x}$ is therefore a weighted sum of multivariate normal distributions (Fig. 1), i.e., $$ \mathbb{P} =\sum_{c=1}^{k} p_{c} \mathcal{N}\left({\boldsymbol \mu}^{(c)},\mathbf{C}^{(c)}\right)\text{.} $$ ![Figure 1 of the entry gmm](https://dictionaryofml.org/terms/fig/gmm-fig1.svg) *Figure 1: Top: the photograph of the Oetscher massif with the patch grid drawn on it; each patch is one data point. Bottom: the $972$ patches, each described by its average greenness and its average blueness, both on a scale from $0$ to $1$. The three labeled curves, vegetation (solid curve), mountain and haze (dashed curve), and bright sky (dotted curve), are contours of the multivariate normal distributions of a GMM, so each covariance matrix shows as an ellipse, elongated because greenness and blueness are correlated within each cluster. Data generated by pythondemos/gmm.py* A GMM is parameterized by the cluster-specific probability $p_{c} = \mathbb{P}\left(I=c\right)$, mean ${\boldsymbol \mu}^{(c)}$, and covariance matrix $\mathbf{C}^{(c)}$ for $c=1,\,\ldots,\,k$. These model parameters can be learned from a dataset $\mathcal{D} = \{{\mathbf x}^{(1)}, \ldots, {\mathbf x}^{(m)}\}$ by maximum likelihood. The resulting optimization problem has no closed-form solution, and the standard method for solving it approximately is the expectation–maximization algorithm (Dempster et al., 1977), which alternates between two steps. The E-step computes, for every data point ${\mathbf x}^{(r)}$ and every cluster index $c$, the posterior distribution under the current model parameters, \begin{equation} \tag{1} u_{c}^{(r)} := \mathbb{P}\left(I = c \mid {\mathbf x}^{(r)}\right) = \frac{p_{c}\, \mathcal{N}\left({\boldsymbol \mu}^{(c)},\mathbf{C}^{(c)}\right)({\mathbf x}^{(r)})} {\sum_{c'=1}^{k} p_{c'}\, \mathcal{N}\left({\boldsymbol \mu}^{(c')},\mathbf{C}^{(c')}\right)({\mathbf x}^{(r)})}\text{,} \end{equation} and the M-step re-fits the model parameters as averages weighted by these posterior distributions, \begin{equation} \tag{2} \begin{aligned} p_{c} &:= \frac{1}{m} \sum_{r=1}^{m} u_{c}^{(r)}, \qquad {\boldsymbol \mu}^{(c)} := \frac{\sum_{r=1}^{m} u_{c}^{(r)}\, {\mathbf x}^{(r)}} {\sum_{r=1}^{m} u_{c}^{(r)}}, \\ \mathbf{C}^{(c)} &:= \frac{\sum_{r=1}^{m} u_{c}^{(r)}\, ({\mathbf x}^{(r)} - {\boldsymbol \mu}^{(c)}) ({\mathbf x}^{(r)} - {\boldsymbol \mu}^{(c)})^{\top}} {\sum_{r=1}^{m} u_{c}^{(r)}} \quad \text{for } c = 1, \ldots, k\text{.} \end{aligned} \end{equation} The M-step formulas Eq. (2) are not a closed-form solution of the maximum likelihood optimization problem, because the posterior distributions on their right-hand side depend on the model parameters through the E-step Eq. (1) (Bishop, 2006, Sect. 9.2.2). The two steps are repeated until the model parameters stop changing (Bishop, 2006, Sect. 9.2.2). The maximum likelihood optimization problem for the model parameters of a GMM is an instance of empirical risk minimization with a particular choice for the loss function. Assume that every covariance matrix $\mathbf{C}^{(c)}$ is invertible, so that the GMM has a probability density function on $\mathbb{R}^{d}$. The loss incurred by a feature vector ${\mathbf x}$ is the negative logarithm of this probability density function at ${\mathbf x}$, $$ L\left({\mathbf x},\mathbb{P} \right) = - \ln \sum_{c=1}^{k} p_{c} \mathcal{N}\left({\boldsymbol \mu}^{(c)},\mathbf{C}^{(c)}\right) ({\mathbf x}) \text{.} $$ The empirical risk of a GMM is then obtained as the average loss over the entire dataset. Minimizing it maximizes the likelihood function, since the logarithm is increasing and the sign is flipped. Fitting a GMM is therefore empirical risk minimization over the hypothesis space of mixtures of $k$ multivariate normal distributions. The loss function is a non-convex function of the model parameters (Murphy, 2012, Sect. 11.3.2). The expectation–maximization algorithm might therefore converge to a local minimum of the empirical risk (Bishop, 2006, Sect. 9.2.2), so it is typically run several times with different initializations, keeping the run with the smallest empirical risk (Murphy, 2012, Sect. 11.3.2). A common construction of the initial model parameters runs k-means first and starts from its cluster centroids (Bishop, 2006, Sect. 9.2.2). Whatever its initialization, the expectation–maximization algorithm is, like k-means, a fixed-point iteration. Collect the model parameters of a GMM in one vector ${\mathbf w}$. One iteration, the E-step Eq. (1) followed by the M-step Eq. (2), applies an operator $\mathcal{F}$ to the current model parameters, so the method is the fixed-point iteration ${\mathbf w}^{(t+1)} = \mathcal{F}\big({\mathbf w}^{(t)}\big)$ for $t = 0,1,\ldots$. The fixed points of $\mathcal{F}$ are model parameters at which the gradient of the empirical risk vanishes, i.e., the stationary points of the empirical risk (see zero-gradient condition) (Wu, 1983, Sect. 2.1). Convergence follows from monotone descent rather than from a contractive operator property: applying $\mathcal{F}$ never increases the empirical risk (Dempster et al., 1977, Thm. 1), so the sequence of empirical risk values is non-increasing. It converges if the empirical risk is bounded below. This bound is not automatic for a GMM: a component whose mean coincides with one data point and whose variance shrinks to zero drives the empirical risk to $-\infty$ (Bishop, 2006, Sect. 9.2.1). The bound holds once the eigenvalues of every covariance matrix $\mathbf{C}^{(c)}$ are at least some $\delta > 0$, since then the probability density function of the GMM nowhere exceeds $(2\pi\delta)^{-d/2}$. Under the conditions of Wu (1983, Thm. 2), which imply this bound, the sequence of empirical risk values converges and every limit point of the iterates ${\mathbf w}^{(t)}$ is a fixed point. On the patches of Fig. 1, the empirical risk never increased in any of the $300$ iterations of expectation–maximization. Trained by expectation–maximization on the $972$ patches of Fig. 1, a three-component GMM separates vegetation, mountain and haze, and bright sky. Every patch it assigns to the bluest component lies in the upper half of the photograph. The learned model parameters define a soft clustering: the posterior distribution $\mathbb{P}\left(I = c \mid {\mathbf x}\right)$ is the probability that a data point belongs to cluster $c$, and it quantifies the degree of belonging to cluster $c$. In contrast, hard clustering methods assign each data point to exactly one cluster. Fig. 2 shows these degrees of belonging on the photograph itself. ![Figure 2 of the entry gmm](https://dictionaryofml.org/terms/fig/gmm-fig2.svg) *Figure 2: (a) The photograph from Fig. 1 at patch resolution, each patch one data point. The same photograph three times, each patch dimmed by its posterior distribution for one cluster: vegetation (b), mountain and haze (c), and bright sky (d). A patch that belongs to one cluster alone is bright in one copy and black in the other two, while a shared patch is dim in several. Because the three probabilities of a patch sum to one, the three copies add back up to (a). Data generated by pythondemos/gmm.py* The expectation–maximization algorithm for learning the model parameters of a GMM reduces to Lloyd's algorithm for k-means under additional constraints. Fix every cluster probability at $1/k$ and every covariance matrix at $\sigma^{2} \mathbf{I}$, a multiple of the identity matrix with one common $\sigma^{2}$. The posterior distribution then depends on ${\mathbf x}$ only through its Euclidean distances to the means, and as $\sigma^{2}$ shrinks towards $0$ it concentrates on the closest one. In the limit ($\sigma^{2} \to 0$), the E-step Eq. (1) becomes an assignment of every data point to its nearest mean, and the M-step Eq. (2) re-centers the means on the data points assigned to them (Bishop, 2006, Sect. 9.3.2, Eq. (9.42)). That is Lloyd's algorithm for k-means. Fig. 3 shows the effect of the common $\sigma^{2}$ on the degree of belonging to the vegetation cluster. With $\sigma^{2} = 10^{-4}$, every patch obtains a degree of belonging of $0$ or $1$, and the grading coincides with the k-means assignment on more than $99\%$ of the patches. With $\sigma^{2} = 2 \cdot 10^{-2}$, $63\%$ of the patches obtain a degree of belonging between $0.2$ and $0.8$. On the patches of Fig. 1, with $\sigma^{2} = 1$ the three means collapse onto the average of all patches and every patch obtains the same degree of belonging $1/3$. Learning the covariance matrices and the cluster probabilities instead of fixing them lets the grading adapt to the data, where k-means assigns each data point to one cluster. ![Figure 3 of the entry gmm](https://dictionaryofml.org/terms/fig/gmm-fig3.svg) *Figure 3: The photograph at patch resolution, each patch dimmed by its degree of belonging to the vegetation cluster under the constrained GMM with cluster probabilities $1/k$ and covariance matrices $\sigma^{2} \mathbf{I}$. (a) The k-means assignment: a patch is bright if it is assigned to the vegetation cluster and black otherwise. (b) $\sigma^{2} = 10^{-4}$: the grading is again $0$ or $1$ and coincides with (a). (c) $\sigma^{2} = 2 \cdot 10^{-2}$: the meadow is graded highest, the foliage and the horizon in between, the sky lowest. (d) $\sigma^{2} = 1$: every patch is graded $1/3$. Data generated by pythondemos/gmm.py* **See also:** probabilistic model, multivariate normal distribution, [clustering](https://dictionaryofml.org/terms/clustering.html), [expectation–maximization](https://dictionaryofml.org/terms/em.html), fixed-point iteration, maximum likelihood, soft clustering, degree of belonging, hard clustering, [$k$-means](https://dictionaryofml.org/terms/kmeans.html), Lloyd's algorithm. ## References - Bishop (2006). Pattern Recognition and Machine Learning. *Springer Science+Business Media*. - Murphy (2012). Machine Learning: A Probabilistic Perspective. *MIT Press*. - Dempster et al. (1977). Maximum likelihood from incomplete data via the EM algorithm. *J. Roy. Statist. Soc.: Ser. B (Methodological)*. - Wu (1983). On the convergence properties of the EM algorithm. *Ann. Statist.*. ## Cite this entry ```bibtex @misc{dictml_gmm, author = {Jung, Alexander}, editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard}, title = {Gaussian mixture model (GMM)}, howpublished = {Dictionary of Applied Machine Learning (course edition)}, year = {2026}, doi = {10.5281/zenodo.21569296}, note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-11-11}, url = {https://dictionaryofml.org/terms/gmm.html} } ```