gmm.py, which recomputes what the entry states: the script block by block · open in ColabDictionary of Applied Machine Learning: 107 machine-learning terms, each with a definition, a figure, citations and a Python demo · All terms · Demos · Start here
Updated on 2026-10-22
Readiness 100/100
See also clustering expectation–maximization $k$-means
A Gaussian mixture model (GMM) is a probabilistic model for the generation of data points with numeric feature vectors. Each data point is generated by first drawing a latent cluster index and then drawing the feature vector from the multivariate normal distribution of that cluster. The resulting marginal distribution is a weighted sum of multivariate normal distributions. The model parameters are chosen to maximize the likelihood function of a dataset. The resulting maximum likelihood optimization problem is an instance of empirical risk minimization (ERM) using a specific choice for the loss function. This optimization problem is non-convex and typically solved using the expectation–maximization (EM) algorithm. This algorithm consists of a fixed-point iteration that never increases the empirical risk.
gmm.py, which recomputes what the entry states: the script block by block · open in ColabB-patchesFig. 1 shows a photograph of a mountain that is cut into small square patches, and each patch is described by two numbers: how green and how blue it is on average. Vegetation, mountain and haze, and bright sky form three clusters of such patches, which are data points characterized by a feature vector $\featurevec \in \reals^{\nrfeatures}$. A GMM is a probabilistic model for the generation of such data points (Bishop, 2006; Murphy, 2012).
B-emThe idea of a GMM is to assume that each data point is
generated by first drawing
a latent cluster index $I \in \{1,\,\ldots,\,\nrcluster\}$
according to cluster probabilities
\[\prob{I=\clusteridx} = p_{\clusteridx},
\qquad
\sum_{\clusteridx=1}^{\nrcluster} p_{\clusteridx}=1\text{.}\]
Given the choice $I=\clusteridx$, the feature vector $\featurevec$
is drawn from a multivariate normal distribution
$\probdist^{(\clusteridx)}= \mvnormal{\meanvec{\clusteridx}}{\covmtx{\clusteridx}}$,
with mean $\meanvec{\clusteridx}$ and covariance matrix
$\covmtx{\clusteridx}$.
The resulting marginal distribution of $\featurevec$ is therefore
a weighted sum of multivariate normal distributions (Fig. 1), i.e.,
\[\probdist
=\sum_{\clusteridx=1}^{\nrcluster}
p_{\clusteridx} \mvnormal{\meanvec{\clusteridx}}{\covmtx{\clusteridx}}\text{.}\]
pythondemos/gmm.py
B-imagesTrained by EM on the $972$ patches of
Fig. 1, a three-component GMM separates
vegetation, mountain and haze, and bright sky. Every patch it
assigns to the bluest component lies in the upper half of the
photograph. The learned model parameters define a
soft clustering: the posterior distribution
$\prob{I = \clusteridx \mid \featurevec}$ is the
probability that a data point belongs to
cluster $\clusteridx$, and it provides a principled way to
quantify the degree of belonging to cluster $\clusteridx$. In
contrast, hard clustering methods assign each
data point to exactly one cluster.
Fig. 2 shows these degrees of belonging on
the photograph itself.
pythondemos/gmm.py
The loss function is a non-convex function of the model parameters, so no closed form solves the optimization problem (Murphy, 2012, Sect. 11.3.2). The EM algorithm is an approximate method for the maximum likelihood optimization problem: it never increases the empirical risk. Since the EM algorithm might converge to a local minimum of the empirical risk (Bishop, 2006, Sect. 9.2.2), it is typically run several times with different initializations, keeping the run with the smallest empirical risk (Murphy, 2012, Sect. 11.3.2). A common construction of the initial model parameters runs $k$-means first and starts from its cluster centroids (Bishop, 2006, Sect. 9.2.2).
Like $k$-means, the EM algorithm is a fixed-point iteration. Collect the model parameters of a GMM in one vector $\weights$. One iteration, the E-step \(\eqref{equ_gmm_estep_dict}\) followed by the M-step \(\eqref{equ_gmm_mstep_dict}\), applies an operator $\fixedpointop$ to the current model parameters, so the method is the fixed-point iteration $\weights^{(\iteridx+1)} = \fixedpointop\big(\weights^{(\iteridx)}\big)$. The fixed points of $\fixedpointop$ are model parameters at which the gradient of the empirical risk vanishes, i.e., the stationary points of the empirical risk (see zero-gradient condition) (Wu, 1983). Convergence follows from monotone descent rather than from a contractive operator property: applying $\fixedpointop$ never increases the empirical risk (Dempster et al., 1977, Thm. 1), so the sequence of empirical risk values converges, and under mild continuity conditions the iterates $\weights^{(\iteridx)}$ approach a fixed point (Wu, 1983). On the patches of Fig. 1, the empirical risk never increased in any of the $300$ iterations of EM.
B-kmeansThe EM algorithm for learning the model parameters of a GMM reduces to Lloyd's algorithm for $k$-means under additional constraints. Fix every cluster probability at $1/\nrcluster$ and every covariance matrix at $\sigma^{2} \mI$, a multiple of the identity matrix with one common $\sigma^{2}$. The posterior distribution then depends on $\featurevec$ only through its Euclidean distances to the means, and as $\sigma^{2}$ shrinks towards $0$ it concentrates on the closest one. In the limit ($\sigma^{2} \to 0$), the E-step \(\eqref{equ_gmm_estep_dict}\) becomes an assignment of every data point to its nearest mean, and the M-step \(\eqref{equ_gmm_mstep_dict}\) re-centers the means on the data points assigned to them. That is Lloyd's algorithm for $k$-means.
B-sigmaFig. 3 shows the effect of the common
$\sigma^{2}$ on the degree of belonging to the vegetation cluster.
With $\sigma^{2} = 10^{-4}$, every patch obtains a degree of belonging
of $0$ or $1$, and the grading coincides with the $k$-means
assignment on more than $99\%$ of the patches. With
$\sigma^{2} = 2 \cdot 10^{-2}$, $63\%$ of the patches obtain a
degree of belonging between $0.2$ and $0.8$. On the patches of
Fig. 1, with $\sigma^{2} = 1$ the
three means collapse onto the average of all patches and
every patch obtains the same degree of belonging $1/3$. Leaving the
covariance matrices and the cluster probabilities free
is what lets a GMM provide a degree of belonging where $k$-means
assigns each data point to one cluster.
pythondemos/gmm.py
See also: probabilistic model, multivariate normal distribution, clustering, expectation–maximization, fixed-point iteration, maximum likelihood, soft clustering, degree of belonging, hard clustering, $k$-means, Lloyd's algorithm.
@misc{dictml_gmm,
author = {Jung, Alexander},
editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
title = {Gaussian mixture model (GMM)},
howpublished = {Dictionary of Applied Machine Learning (course edition)},
year = {2026},
doi = {10.5281/zenodo.21569296},
note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-10-27},
url = {https://dictionaryofml.org/terms/gmm.html}
}