{ "name": "Dictionary of Applied Machine Learning (course edition)", "url": "https://dictionaryofml.org", "license": "CC BY 4.0", "doi": "10.5281/zenodo.21569296", "isbn": "978-952-64-3013-3", "authors": [ "Jung, Alexander", "Olioumtsevits, Konstantina", "Schnoor, Ekkehard" ], "note": "Definitions are lightly cleaned LaTeX: inline math is transliterated, display math kept as TeX, glossary links replaced by the linked term's name. The typeset PDF is the authoritative form of each entry.", "retrieved_format_docs": "https://dictionaryofml.org/llms.txt", "terms": [ { "key": "accuracy", "name": "accuracy", "url": "https://dictionaryofml.org/terms/accuracy.html", "pdf": "https://dictionaryofml.org/terms/accuracy.pdf", "abstract": "Accuracy is the fraction of correct predictions made by a hypothesis on a dataset with a finite label space, equal to one minus the average 0/1 loss. It is widely used as a single-number metric for classification methods. Accuracy is not suitable as an objective function for training, since the 0/1 loss is non-smooth and non-convex. empirical risk minimization uses a smooth surrogate loss instead, and accuracy is reported during validation. On imbalanced data, a large accuracy can hide failure on the minority class, which calls for other means of evaluation such as the confusion matrix.", "description": "Accuracy is the fraction of correct predictions made by a hypothesis h: X → Y on a dataset D = (x(r), y(r) ) r=1m with a finite label space Y : \\begin{align*} \\operatorname{acc}(\\hypothesis|\\dataset) & \\defeq 1 - \\frac{1}{\\samplesize} \\sum_{\\sampleidx=1}^{\\samplesize} \\lossfunczo{\\big(\\featurevec^{(\\sampleidx)}, \\truelabel^{(\\sampleidx)}\\big)}{\\hypothesis} & = \\frac{1}{\\samplesize} \\Big| \\Big\\{ \\sampleidx \\in \\{1, \\,\\ldots, \\,\\samplesize\\} : \\hypothesis\\big(\\featurevec^{(\\sampleidx)}\\big) = \\truelabel^{(\\sampleidx)} \\Big\\} \\Big| \\text{.} \\end{align*} Accuracy ranges from 0 (no prediction is correct) to 1 (every prediction is correct). Equivalently, accuracy equals one minus the average 0/1 loss on the dataset. For example, in image classification, accuracy is the fraction of images assigned the correct category. Fig.\\ \\ref{fig_acc_dict} compares two classifiers on the same dataset. The linear classifier h misclassifies three data points near its decision boundary. The nonlinear classifier h' curves around each of these data points and classifies them correctly. % Data generated by pythondemos/accuracy.py The classifier h' in Fig.\\ \\ref{fig_acc_dict} achieves the optimal accuracy of 1 on D: its decision boundary closely follows the positions of the individual data points that the linear classifier misclassifies. As a result, h' is sensitive to small perturbations of the feature vectors, such as measurement noise. A slight shift in the feature vector of a data point near the decision boundary can flip the prediction. Optimizing accuracy on the training set therefore does not by itself ensure good generalization. While accuracy is often used to compare learned classifiers on a validation set or a test set, it is not suitable as an objective function for training. Indeed, accuracy is defined via the 0/1 loss, which is non-smooth and non-convex. gradient-based optimization methods cannot maximize accuracy directly. Instead, training methods for classifiers, e.g., via empirical risk minimization, typically use a surrogate loss that approximates the 0/1 loss. Two widely used examples for such a surrogate loss are the logistic loss, which is convex and differentiable, and the hinge loss, which is convex but not differentiable. Fig.\\ \\ref{fig_acc_losses_dict} shows the three loss functions for binary classification with Y = -1, 1, where a real-valued hypothesis h predicts via the sign of h(x). Each loss function depends on the data point only through the product y · h(x). Accuracy can also be misleading for imbalanced data. Consider a frost warning for Krems an der Donau (Austria): the features of a data point are the minimum and the maximum air temperature of one day, and its label indicates whether the minimum temperature of the following day falls below 0 C. Frost days are the minority class: only 6 of 40 recorded days are followed by frost. The constant prediction ``above zero'' (a baseline that ignores the features) is therefore correct on 34 of the 40 days, an accuracy of 0.85, and a classifier learned by logistic regression from the 40 data points reaches the same accuracy of 0.85. Accuracy cannot tell the two apart, which calls for other means of evaluation such as the confusion matrix. The confusion matrices in Fig.\\ \\ref{fig_acc_cm_dict} show that the learned classifier detects one of the six frost days while the baseline detects none. See also: 0/1 loss, loss, classification, F1 score, confusion matrix, imbalanced data.", "synonyms": [], "see_also": [], "demo": "https://dictionaryofml.org/terms/accuracy.py", "notebook": "https://dictionaryofml.org/terms/accuracy.ipynb" }, { "key": "ann", "name": "artificial neural network", "url": "https://dictionaryofml.org/terms/ann.html", "pdf": "https://dictionaryofml.org/terms/ann.pdf", "abstract": "An artificial neural network (ANN) is a graphical (signal-flow) representation of a hypothesis that maps the features of a data point at its input to a prediction for the label at its output. Its computational unit is the artificial neuron, which applies an activation function to the weighted sum of its inputs; the edge weights are the tunable model parameters. An ANN can be represented as a directed acyclic graph, and its connectivity structure — the architecture — is a central design choice. deep nets arrange neurons in consecutive layers; varying the connectivity yields feedforward networks, CNNs, recurrent networks, and transformer-based architectures.", "description": "The face recognition of a smartphone camera and the speech transcription of a voice assistant are both computed by networks of simple computing units with tuned connection strengths. Such a network is an ANN: a graphical (signal-flow) representation of a hypothesis that maps features of a data point at its input to a prediction for the corresponding label at its output. The fundamental computational unit of an ANN is the artificial neuron, which applies an activation function σ(·) to the weighted sum of its inputs plus an offset term (Fig.~\\ref{fig_ann_neuron_dict}); the edge weights of the network are its tunable model parameters. The output of a neuron can be used either as the final output of the ANN or as an input to other neurons. One important design aspect of an ANN is its connectivity structure (or architecture), i.e., which outputs are connected to which downstream neuron's inputs . As illustrated in Fig.\\ \\ref{fig_ANN_DAG_dict}, an ANN can be represented as a directed acyclic graph. One widely used type of ANN is deep nets where neurons form consecutive layers (Fig.~\\ref{fig_ann_layers_dict}). In a deep net, the outputs of neurons in a given layer are typically only connected to the inputs of the neurons in a consecutive layer. Each layer l then applies a feature transformation Φ(l) whose components are computed by the neurons of that layer (each as in Fig.~\\ref{fig_ann_neuron_dict}), and the deep net computes the concatenation of these layer-wise maps. The network of Fig.~\\ref{fig_ann_layers_dict}, with its dashed edge removed, delivers the prediction \\hypothesis^{(\\weights)}(\\featurevec) = \\featuretrafovec^{(3)}\\Big(\\featuretrafovec^{(2)} \\big(\\featuretrafovec^{(1)}(\\featurevec)\\big)\\Big) \\text{,} starting from the input feature vector x = (x1, x2, x3): the two hidden layers compute Φ(1), Φ(2): R3 → R3 and the output layer computes Φ(3): R3 → R. Sometimes it is useful to add shortcut or skip connections that directly connect the outputs of neurons in one layer to the inputs of neurons in a nonconsecutive layer (; ). Varying the connectivity pattern yields different ANN architectures: feedforward networks connect layers strictly in sequence, CNNs share weights across spatial positions, recurrent networks include feedback connections for sequence processing, and transformer-based architectures underlie modern LLMs.", "synonyms": [], "see_also": [ "label", "artificialneuron", "actfun", "deepnet", "layer", "cnn", "transformer" ], "demo": "https://dictionaryofml.org/terms/ann.py", "notebook": "https://dictionaryofml.org/terms/ann.ipynb" }, { "key": "attention", "name": "attention", "url": "https://dictionaryofml.org/terms/attention.html", "pdf": "https://dictionaryofml.org/terms/attention.pdf", "abstract": "Attention is a mechanism that models the dependencies between the tokens that make up a data point, such as the words in a sentence or the pixel patches of an image. The idea is to represent the relationship between two tokens by a parameterized weight function whose model parameters are learned from a training set. An attention head then computes a new vector representation for each token as a weighted combination of the value vectors of all tokens. Like the weight function, the value vectors are learned. An attention head acts as a differentiable associative memory: a token uses its query vector to retrieve, through the key vectors of the other tokens, the values most relevant to it. Attention heads capture long-range dependencies between tokens regardless of their positions within a data point, and they are a core component of modern LLMs.", "description": "Some machine learning applications involve data points composed of smaller units, referred to as tokens. For example, a sentence consists of words, an image of pixel patches, and a network of nodes. In general, the tokens that constitute a single data point are not independent of one another. Instead, each token of a data point depends on specific other tokens of the same data point. The attention mechanism is a building block of ANNs that captures long-range dependencies between tokens regardless of their positions within a data point. probabilistic models provide a principled way of representing and analyzing such dependencies . Attention mechanisms instead represent these dependencies directly, without specifying a probability distribution over the tokens. They represent the relationship between two tokens i and i' by an attention weight f(w)(i,i'), a parameterized function whose model parameters w are learned. The attention weight measures how strongly token i attends to token i'. Practical attention mechanisms differ in their choice of the function f(w)(i,i') and in the empirical risk minimization variant used to learn w. The most widely used choice, described below, is scaled dot-product attention . Scaled dot-product attention derives three vectors from the embedding vector x(i) ∈ Rd of each token i via learned matrices: a query (vector) q(i) = WQ x(i), a key (vector) k(i) = WK x(i), and a value (vector) v(i) = WV x(i), where \\mW_Q, \\mW_K, \\mW_V \\in \\reals^{\\featuredim \\times \\featuredim} are model parameters learned during training. Intuitively, the query determines what a token ``asks for,'' the key of a token determines what it ``advertises,'' and the value carries the information that is aggregated (cf.\\ ; ). The output of the attention mechanism for token i is a weighted sum of the values of all tokens, \\vz^{(\\nodeidx)} = \\sum_{\\nodeidx'=1}^{\\contextlen} \\alpha_{\\nodeidx,\\nodeidx'}\\, \\vv^{(\\nodeidx')} \\text{.} Here, n denotes the number of tokens of the data point, and the coefficient αi,i' is the attention weight, i.e., the scaled dot-product form of f(w)(i,i'). The attention weights are computed from the queries and keys in two steps. First, the attention score between tokens i and i' is s_{\\nodeidx,\\nodeidx'} = \\frac{\\big(\\vq^{(\\nodeidx)}\\big)^{\\top} \\vk^{(\\nodeidx')}} {\\sqrt{\\featuredim}} \\text{.} The inner product (also called dot product) (q(i)) k(i') measures how well the key of token i' matches the query of token i. Second, the attention weight αi,i' follows by applying the softmax function to the scores si,i' across all tokens i': \\alpha_{\\nodeidx,\\nodeidx'} = \\frac{\\exp(s_{\\nodeidx,\\nodeidx'})} {\\sum_{j=1}^{\\contextlen} \\exp(s_{\\nodeidx,j})} \\text{.} The softmax function normalizes the scores: the attention weights αi,i' are nonnegative and sum to one across i' = 1, …, n, forming a probability distribution over the tokens. Dividing the inner product by d in the attention score counteracts its growth with the dimension d . Indeed, (q(i)) k(i') is a sum of d terms, so without the scaling, a large d would push the softmax function into a region where its gradients nearly vanish. Fig.~\\ref{fig_attention_dict} illustrates scaled dot-product attention for an eight-token sentence. Moving from a single token to the whole sequence of tokens, an attention head is a parameterized function h(w) that maps the input embeddings to the output vectors, \\hypothesis^{(\\weights)}\\big(\\featurevec^{(1)}, \\ldots, \\featurevec^{(\\contextlen)}\\big) = \\big(\\vz^{(1)}, \\ldots, \\vz^{(\\contextlen)}\\big) \\text{,} with model parameters w comprising the three matrices WQ, WK, WV. Collecting the output vectors as the rows of a matrix Z, and the queries, keys, and values as the rows of Q, K, V, h(w) computes them in the compact form \\mZ = \\operatorname{softmax}\\!\\!\\left(\\frac{\\mQ\\,\\mK^{\\top}}{\\sqrt{\\featuredim}}\\right) \\mV \\text{.} Here, the softmax function is applied independently to each row of Q K/d, so that each row of the resulting matrix is a probability distribution over the n tokens. Since the queries, keys, and values are all derived from the same sequence of tokens, this construction is called self-attention. In practice, several attention heads, each with its own learned matrices WQ, WK, WV, operate in parallel; this is called multi-head attention, and each head can specialize to a different relation between tokens . Computing all pairwise scores si,i' requires n2 query--key inner products, one for each ordered pair of tokens. Attention can be read as a differentiable associative memory, i.e., a content-addressable store : the keys k(i') act as addresses, the values v(i') as stored contents, and a token i searches the other tokens with its query q(i). The inner product (q(i))k(i') measures how well the address i' matches the query, the softmax function turns the match scores into retrieval weights, and the output z(i) is the retrieved content. Because q(i) and k(i') are obtained from the different matrices WQ and WK, the attention weights are \\emph{directed}: in general, αi,i' αi',i, so that token i attending strongly to token i' does not imply the reverse. Directionality arises because asking for information (the query) and advertising it (the key) are distinct roles. A classical dictionary data structure or hash table returns the single value stored at an exactly matching address . Attention instead returns a weighted average. Moreover, its output z(i) is a differentiable function of the matrices WQ, WK, and WV. In particular, the queries, keys, and values can themselves be learned via gradient-based methods for solving empirical risk minimization. The same retrieval principle, softened into a weighted average, underlies k-nearest neighbors methods. It also guides efficient implementations: locality-sensitive hashing restricts each query to nearby keys, reducing the cost of one attention head from n2 query--key comparisons to approximately n n, with n again the number of tokens . Fig.~\\ref{fig_attention_assoc} visualizes the attention weights αi,i' that a single head learns on sentences from the Universal Declaration of Human Rights . Trained to reconstruct each masked token from the others, the head gives many query tokens a weight concentrated on a few keys rather than spread uniformly, illustrating the associative-memory reading on this small corpus. The heat map also shows the directed nature of the weights discussed above: the query token ``rights'' places its full weight on the key token ``beings'' (α12,3 = 1.0), while the query token ``beings'' places no weight on the key token ``rights'' (α3,12 = 0.0). Synonyms: attention mechanism, attention head.", "synonyms": [], "see_also": [ "token", "embedding", "transformer", "softmax", "sequence", "nlp", "llm", "modelparam", "erm", "knn", "differentiable" ], "demo": "https://dictionaryofml.org/terms/attention.py", "notebook": "https://dictionaryofml.org/terms/attention.ipynb" }, { "key": "bagging", "name": "bootstrap aggregating", "url": "https://dictionaryofml.org/terms/bagging.html", "pdf": "https://dictionaryofml.org/terms/bagging.pdf", "abstract": "Bagging is an ensemble technique that trains each base learner on a resampled copy of the training set, typically drawn by bootstrap sampling. The learned hypotheses are aggregated by a majority vote in classification or by averaging in regression. Averaging over many base learners reduces the variance of the final prediction: fluctuations caused by the particular training set tend to cancel. A random forest applies bagging to decision trees.", "description": "A single decision tree, fit to a bank's customer records to predict loan defaults, changes its predictions noticeably when a handful of records are replaced. Bagging suppresses this sensitivity: it is an ensemble technique in which each base learner is trained on a resampled copy of the training set, and their predictions are aggregated . Given the original training set D, bagging generates perturbed copies D(1), …, D(M) (typically via bootstrap sampling) and trains one base learner on each, producing hypotheses h(1), …, h(M). The final hypothesis aggregates these by majority vote for classification (i.e., the class predicted by the most base learners) or averaging for regression (see Fig.~\\ref{fig_bagging_dict}). A random forest, for example, applies bagging to decision trees: each tree is trained on a different bootstrap sample, reducing the variance of the overall prediction.", "synonyms": [], "see_also": [ "ensemble", "bootstrap", "randomforest", "baselearner", "variance" ] }, { "key": "baseline", "name": "baseline", "url": "https://dictionaryofml.org/terms/baseline.html", "pdf": "https://dictionaryofml.org/terms/baseline.pdf", "abstract": "A baseline is a reference level against which the performance of a trained model is compared. This comparison allows verifying whether the achieved average loss is satisfactory, i.e., whether the learned hypothesis is already close to optimal. Baselines can be obtained from human performance, from an existing machine learning method, or from an accurate probabilistic model for the data generation. Given such a probabilistic model, the smallest achievable risk is the Bayes risk, incurred by the Bayes estimator of the label given the features. In practice, however, the Bayes estimator is infeasible for at least two reasons. First, the probability distribution of the data generation process might be impossible to determine. Second, computing the Bayes estimator might be computationally too expensive.", "description": "Consider some machine learning method that produces a learned hypothesis (or trained model) h ∈ H. Its quality is typically evaluated by computing the average loss on a test set. But how can it be verified that the measured performance is satisfactory? In other words, is the learned hypothesis already so close to optimal that there is little point in investing more resources (for data collection or computation) to improve it? To answer such questions, a baseline (or reference level) for the smallest loss achievable by any method is needed. \\hspace*{\\parindent}Sometimes a baseline can be read off a scatterplot such as Fig.~\\ref{fig_baseline_krems}, which shows weather records of the station Krems an der Donau: each of the 366 days of 2024 is a data point whose feature x is the minimum air temperature of the day and whose label y is its maximum air temperature. On the 15 days with x between 11.5 and 12.5 degrees Celsius, the label ranged from 13.2 to 33.4 degrees Celsius. Two of these days even share the same feature value x = 11.8, with labels 13.2 and 29.9. A hypothesis is a map, so it assigns both days the same prediction and incurs, on these two days, an average squared error loss of at least (16.7/2)2 69.7. Averaging the variance of the label within all such one-degree bands yields 19.34 (in squared degrees Celsius) as an estimate for the smallest achievable average squared error loss. A hypothesis learned by linear regression incurs an average squared error loss of 19.17 on the 366 days, within one percent of that estimate: this hypothesis is already close to optimal. A baseline might be obtained from human performance, e.g., the misclassification rate of dermatologists who diagnose cancer from visual inspection of skin . For LLMs, baselines are often obtained from a benchmark, i.e., a fixed collection of test problems together with an evaluation protocol. As an example, the benchmark GSM8K measures the ability of an large language model to solve grade-school mathematics word problems . \\hspace*{\\parindent}Another source for a baseline is an existing, but for some reason unsuitable, machine learning method: it might be computationally too expensive for the intended machine learning application, yet its test set error can still serve as a baseline. A more principled source of a baseline is a probabilistic model. In many cases, given a probabilistic model p(x,y), it is possible to precisely determine the minimum achievable risk among any hypotheses (not even required to belong to the hypothesis space H) . \\hspace*{\\parindent}This minimum achievable risk (referred to as the Bayes risk) is the risk of the Bayes estimator of the label y of a data point, given its features x. For the squared error loss, the Bayes risk equals the variance of the label around its posterior mean, averaged over the features; the band-wise estimate in Fig.~\\ref{fig_baseline_krems} approximates exactly this quantity. \\hspace*{\\parindent}Note that, for a given choice of loss function, the Bayes estimator (if it exists) is fully determined by the probability distribution P . However, computing the Bayes estimator %need to recheck if indeed chap.4 - and if possible to cite sth more specific e.g. Sect. or Example and Bayes risk presents two main challenges. First, the probability distribution P is unknown and must be estimated from observed data. Second, even if P were known, computing the Bayes risk exactly may be computationally infeasible . \\hspace*{\\parindent}A widely used probabilistic model is the multivariate normal distribution xy ~ μC for data points characterized by numeric features and labels. Here, for the squared error loss, the Bayes estimator is given by the posterior mean μy|x of the label y, given the features x (; ). The corresponding Bayes risk is given by the posterior variance σ2y|x (see Fig. \\ref{fig_post_baseline_dict}). The posterior variance can be read off the inverse of the covariance matrix: σ2y|x is the reciprocal of the diagonal entry of C-1 that corresponds to the label y . In practice, the mean vector μ and the covariance matrix C can be estimated from a dataset by the sample mean and the sample covariance matrix . See also: Bayes risk, Bayes estimator, test set, probabilistic model, risk, accuracy, benchmark.", "synonyms": [], "see_also": [], "demo": "https://dictionaryofml.org/terms/baseline.py", "notebook": "https://dictionaryofml.org/terms/baseline.ipynb" }, { "key": "boosting", "name": "boosting", "url": "https://dictionaryofml.org/terms/boosting.html", "pdf": "https://dictionaryofml.org/terms/boosting.pdf", "abstract": "Boosting is an iterative optimization method that learns an accurate hypothesis by sequentially combining less accurate base learners, referred to as weak learners. Starting from an initialization, each iteration adds the output of a base learner, scaled by a learning rate, to the current hypothesis. This update generalizes the gradient step of gradient-based methods for empirical risk minimization: the output of the base learner acts as a negative generalized gradient. The fixed points of the update are hypotheses at which no base learner can reduce the training error further. gradient boosting instantiates this scheme with decision trees as base learners.", "description": "An online advertiser must predict, for each impression, whether the user will click on the shown advertisement. Widely deployed predictors for this task are built by boosting: an iterative optimization method that learns an accurate hypothesis map (or strong learner) by sequentially combining less accurate base learners (referred to as weak learners) (; ; ; ). Boosting can be understood as a generalization of gradient-based methods for empirical risk minimization using parametric models and smooth loss functions . In particular, starting from an initialization h, boosting methods construct a sequence of hypotheses h(t), t=1, …, via a generalized gradient step \\widetilde{\\hypothesis}^{(\\iteridx)} = \\widetilde{\\hypothesis}^{(\\iteridx-1)}+ \\lrate^{(\\iteridx)} \\learnthypothesis^{(\\iteridx)}\\text{.} Here, η(t) denotes a learning rate and h(t) is provided by the tth base learner (see Fig.~\\ref{fig_boosting_dict}). Comparing the above update with the plain gradient step suggests viewing h(t) as a (negative) generalized gradient. %wondering whether this sentence (e.g., comparison with 'plain gradient step' or view as a 'negative' gradient) could become clearer to a reader by further elaboration or illustration The update is a fixed-point iteration on the hypothesis space: its fixed points are hypotheses at which the base learners provide no further descent direction, so the sequence stalls exactly where no base learner can reduce the training error further. Boosting methods differ in their choice of base learners for computing the generalized gradients h(t). For instance, gradient boosting — behind many deployed click-through predictors — most commonly uses decision trees as base learners.", "synonyms": [], "see_also": [ "ensemble", "adaboost", "gradientboosting", "baselearner", "gdmethod" ] }, { "key": "cav", "name": "concept activation vector", "url": "https://dictionaryofml.org/terms/cav.html", "pdf": "https://dictionaryofml.org/terms/cav.pdf", "abstract": "A concept activation vector (CAV) represents a concept named by a user as a direction in the activations of one hidden layer of a deep net. The user supplies data points that carry the concept and data points that do not, and a linear classifier is fitted to tell the two apart from those activations. The CAV is the normal vector of the resulting decision boundary. Whether the deep net responds to that direction is measured by the directional derivative of its score along the CAV, and the fraction of data points of one label with a positive derivative is the TCAV score. Both steps read the activations of a hidden layer, so a CAV is unavailable for a system that returns predictions and nothing else, and it concerns the interpretability of the deep net rather than its explainability. The resulting explanation, the TCAV score, belongs to one layer and one label; it aggregates the per-data point sensitivities and thereby describes a whole class of data points rather than a single prediction.", "description": "A deep net labels photographs, and one of its labels is zebra. Whether it uses stripes to arrive at that label cannot be read off a single neuron: stripes are not a feature the network was built with, and need not be represented by any one neuron . The concept is specified by examples instead: photographs that show stripes, and photographs that do not . Consider such a deep net, with several hidden layers, trained to predict the label of a data point from its feature vector. Fix one hidden layer and write z = f(x) for its activations, with f the map delivered by the layers up to that point. The deep net is applied to data points that carry a concept C and to data points that do not, and a binary linear classifier g(z) is fitted to tell the two apart from those activations. Its decision boundary is a hyperplane; the CAV for C is the normal vector w of that hyperplane . A hyperplane has two normal vectors, one pointing to each side. By convention, the CAV is the one pointing towards the side that g(z) assigns to the concept class: fitting g(z) with the concept data points as the positive class delivers this orientation, and when the two sets of activations are linearly separable, it is the side of the concept activations. In the terminology of mechanistic interpretability, this direction is a feature of the trained deep net: a human-interpretable concept encoded as a direction in the space of activations. Fig.~\\ref{fig_cav_plane_dict} draws both objects in the plane spanned by the activations of two neurons of a hidden layer. Two classification problems appear in the figure: the task the deep net is trained for, and the concept classification that g(z) solves on the activations, built on top of the trained deep net. The decision boundary of the first is the solid curve; it is nonlinear by assumption, the case a deep net is built for, and a straight line fitted to the same labels classifies 82\\% of the 400 data points correctly. The decision boundary of the second is the dashed hyperplane separating three concept from three non-concept activations; the CAV is its normal vector, pointing towards the concept side. Fitting g(z) to all 400 data points instead turns the CAV by four degrees. Like any good explanation, a CAV has to be understandable and faithful (see explanation). It is understandable by construction: the user supplies the examples that define the concept. Faithfulness is a separate question, because the concept is a direction in the activations whether or not the deep net responds to it. Testing with CAVs (TCAV) decides it. Write the trained deep net as a concatenation of two maps, the activations z = f(x) of the chosen hidden layer and a score s(z) that the remaining layers assign to one label. The conceptual sensitivity of a data point is the directional derivative of that score along the unit CAV v := w / w, which for differentiable s equals the inner product of the gradient of s with v , \\begin{equation} S(\\featurevec) \\defeq \\innerprod{\\nabla s\\big(f(\\featurevec)\\big)}{\\vv} \\text{.} \\end{equation} It measures how much the score for that label moves when the activations move a little towards the concept . The orientation convention gives the sign its meaning: S(x) > 0 says the score rises towards the concept, and the opposite normal vector would flip every sensitivity. Equation \\eqref{equ_cav_sensitivity} delivers one number per data point. TCAV collects these numbers over one label: the TCAV score is the fraction of the data points carrying that label for which S(x) > 0 . A random set of non-concept data points also yields a CAV, so a single such fraction is not evidence on its own. The CAV is therefore refitted against many fresh draws of non-concept data points, 500 of them in the original experiments, and the resulting TCAV scores are compared against 1/2 by a two-sided t-test; a concept the deep net does not respond to produces scores that scatter around one half . The explanation delivered is a single number attached to a concept, a label and a layer. Unlike the sensitivity \\eqref{equ_cav_sensitivity}, which is defined for a single data point, it describes a whole class of data points rather than a single prediction . In the plane of Fig.~\\ref{fig_cav_plane_dict}, moving along the CAV raises the score at 65\\% of the 182 class-1 data points, against 58\\% for a concept planted orthogonal to the direction the deep net varies in and 50\\% for random directions. Fitting the linear classifier g(z) and evaluating Eq.~\\eqref{equ_cav_sensitivity} both read quantities inside the deep net: the activations f(x) of the chosen layer, and the gradient s(z) of the score with respect to them. A CAV is therefore unavailable for a system that returns predictions for submitted feature vectors and nothing else, such as a deep net served behind an interface or supplied by a vendor. An explanation produced by local interpretable model-agnostic explanations, by contrast, needs predictions and nothing else: its local approximation is fitted to predictions obtained by querying the learned hypothesis h . A second consequence of reading the internals is that a CAV is not a property of h alone: the split into f and s is a choice of layer, and two deep nets that deliver the same h through different layers have different CAVs. A CAV therefore does not depend on the input--output behavior of h alone, but on the internal computation as well. Both functions it involves are defined on the space of activations: g(z) approximates concept membership there, and Eq.~\\eqref{equ_cav_sensitivity} is a directional derivative of the score s on that same space. A CAV thus concerns the interpretability of the deep net rather than its explainability, and it is the kind of quantity mechanistic interpretability studies. It does not deliver predictability: that stripes raise the zebra score for most photographs of zebras does not determine what the deep net predicts for one photograph.", "synonyms": [], "see_also": [ "deepnet", "feature", "linclass", "trustAI", "interpretability", "mechanisticinterpretability", "transparency", "explanation", "explainability", "neuron", "decisionboundary", "hyperplane", "lime" ], "demo": "https://dictionaryofml.org/terms/cav.py", "notebook": "https://dictionaryofml.org/terms/cav.ipynb" }, { "key": "classification", "name": "classification", "url": "https://dictionaryofml.org/terms/classification.html", "pdf": "https://dictionaryofml.org/terms/classification.pdf", "abstract": "Classification is the task of predicting a discrete-valued label for a given data point, based solely on its features. The label belongs to a finite label space: binary classification uses two label values, and multi-class classification more than two. A hypothesis whose predictions take values in the finite label space is a classifier; it partitions the feature space into decision regions separated by the decision boundary. A widely used construction compares a real-valued hypothesis against a threshold to obtain the predicted label. A natural quality measure is the 0/1 loss; since it is neither convex nor differentiable, training methods minimize surrogates such as the logistic loss or the hinge loss, and the learned classifier is judged by the accuracy and the confusion matrix on a test set. When the label space is continuous, the task is regression.", "description": "A weather station records the minimum and the maximum air temperature of each day. From these two numbers, it must be decided whether the following day brings frost, that is, whether its minimum temperature falls below 0 C. A library must file each incoming text into one of the categories math, novel, or engineering. Both are classification problems: the task of predicting a discrete-valued label y of a given data point, based solely on its feature vector x, the vector of its features (; ; ). Classification assigns each feature vector a label from a finite set. The label takes values in a finite label space Y and names the category of the data point. binary classification uses a label space with two label values, such as Y = -1, 1 with y = 1 marking a frost day; multi-class classification uses more than two label values, such as Y = math, novel, engineering for the texts. When the label space is continuous instead, the task is regression: predicting tomorrow's minimum temperature is regression, predicting whether tomorrow brings frost is classification. Classification is also distinct from clustering: as a form of supervised learning, classification learns from data points whose labels are known, while clustering groups data points without any given labels. A hypothesis whose predictions take values in a finite label space is referred to as a classifier. A classifier partitions the feature space into decision regions, one region per label value, separated by the decision boundary (Fig.~\\ref{fig_classification_dict}). Each classifier is fully determined by its decision regions: all feature vectors within the same decision region obtain the same predicted label. A widely used construction of a classifier proceeds in two steps. First, a real-valued hypothesis h: X → R quantifies the confidence in one particular label value. Second, the confidence h(x) is compared against a threshold to obtain the prediction y ∈ Y. logistic regression, for the label space Y = -1, 1, uses a linear map h(x) = w x as the confidence in the label value 1 and compares it against the threshold 0, \\predictedlabel \\defeq \\begin{cases} 1 & \\text{if } \\hypothesis(\\featurevec) \\geq 0\\text{,} -1 & \\text{otherwise.} \\end{cases} The real-valued hypothesis h is not itself the classifier: the classifier is the thresholded map x y, and calling h the classifier is a slight abuse of language. The two decision regions are then the halfspaces on either side of the hyperplane w x = 0. A natural quality measure for a classifier is the 0/1 loss: its average over a dataset is the fraction of misclassified data points, and one minus that fraction is the accuracy. Even the best classifier cannot always achieve zero error: when data points with the same feature vector carry different labels, some misclassifications are unavoidable, and the smallest achievable risk is the Bayes risk. The 0/1 loss is rarely used as the objective function for training: it is neither convex nor differentiable as a function of the model parameters, so its minimization is computationally hard. Practical methods therefore minimize the average of a surrogate loss over the training set: a loss function that approximates the 0/1 loss while having more convenient properties, such as convexity or differentiability. logistic regression uses the logistic loss, which is convex and differentiable ; the support vector machine uses the hinge loss, which is convex but not differentiable (; ). Fig.~\\ref{fig_classification_losses_dict} compares the three as functions of the margin y · h(x). The learned classifier is then judged by the accuracy and the confusion matrix on a test set. See also: label, label space, classifier, binary classification, decision boundary, decision region, 0/1 loss, logistic loss, hinge loss, logistic regression, support vector machine, accuracy, confusion matrix, regression, supervised learning, clustering, Bayes risk.", "synonyms": [], "see_also": [], "demo": "https://dictionaryofml.org/terms/classification.py", "notebook": "https://dictionaryofml.org/terms/classification.ipynb" }, { "key": "clustering", "name": "clustering", "url": "https://dictionaryofml.org/terms/clustering.html", "pdf": "https://dictionaryofml.org/terms/clustering.pdf", "abstract": "Clustering decomposes a dataset without labels into a small number of subsets, called clusters, such that data points within the same cluster are more similar to each other than to data points in other clusters. It is a prototypical instance of unsupervised learning. Clustering methods differ in the measure of similarity between data points and in the representation of a cluster: a cluster centroid in k-means, a multivariate normal distribution in a Gaussian mixture model. They also differ in whether each data point is assigned to exactly one cluster (hard clustering) or to several with varying degrees (soft clustering). When the dataset is a graph, graph clustering partitions its nodes into densely connected subsets.", "description": "A retailer with millions of customer records has no predefined categories but still wants to group customers with similar purchasing behavior so that marketing campaigns can be targeted. Clustering addresses this type of problem: given a dataset without labels, decompose it into a small number of subsets, called clusters, such that data points within the same cluster are more similar to each other than to data points in other clusters. Clustering is a prototypical instance of unsupervised learning. Different clustering algorithms differ in three design choices: \\begin{enumerate}[label=\\arabic*)] \\item the measure of similarity between data points (e.g., Euclidean distance between feature vectors, edge weights in a graph); \\item the representation of a cluster (e.g., a cluster centroid in k-means , a multivariate normal distribution in a Gaussian mixture model ); and \\item whether each data point is assigned to exactly one cluster (hard clustering) or to several with varying degrees (soft clustering). \\end{enumerate} When the dataset is a graph rather than a set of feature vectors, graph clustering partitions the nodes into densely connected subsets with few edges between subsets . Fig.~\\ref{fig_clustering_modes} illustrates the application of clustering to the pixels of a smartphone snapshot (Fig.~\\ref{fig_clustering_modes}-(a)) and to a toy graph (Fig.~\\ref{fig_clustering_modes}-(d)). Fig.~\\ref{fig_clustering_modes}-(b) shows the result of hard clustering, which partitions the pixels into disjoint regions. Fig.~\\ref{fig_clustering_modes}-(c) illustrates soft clustering, which assigns each pixel a vector of membership probabilities and produces graded transitions. Fig.~\\ref{fig_clustering_modes}-(d) depicts how graph clustering partitions a graph into densely connected subsets of nodes. Beyond image segmentation and customer segmentation, clustering is used for document grouping in natural language processing, community detection in social networks, and as a preprocessing step to reduce a dataset to a smaller set of prototypes before supervised learning.", "synonyms": [ "segmentation" ], "see_also": [ "cluster", "clustercentroid", "kmeans", "hardclustering", "softclustering", "gmm", "graphclustering", "unsupervisedlearning" ], "demo": "https://dictionaryofml.org/terms/clustering.py", "notebook": "https://dictionaryofml.org/terms/clustering.ipynb" }, { "key": "cm", "name": "confusion matrix", "url": "https://dictionaryofml.org/terms/cm.html", "pdf": "https://dictionaryofml.org/terms/cm.pdf", "abstract": "The confusion matrix of a hypothesis on a finite dataset with k label values is the k × k matrix whose entry (c, c') counts the data points with true label c and prediction c'. Its diagonal counts the correct predictions, and each off-diagonal entry counts one kind of misclassification. The accuracy, the precision, and the recall are read off the matrix by normalizing the diagonal, a column, or a row. On a dataset with label skewness, the confusion matrix exposes failure on the rare class that the accuracy alone hides.", "description": "The weather station in Krems an der Donau (Austria) records the minimum and the maximum air temperature of each day. A linear classifier is learned by logistic regression from 40 such data points from February and April 2024 (Fig.~\\ref{fig_cm_data_dict}). The features of a data point are the two temperatures of one day; its label indicates whether the minimum temperature of the following day stays above 0 C or falls below it — a frost warning. Most days pose no risk: only 6 of the 40 following days bring frost. The constant prediction ``above zero'' — a baseline that ignores the features — is therefore correct on 34 of the 40 days, an accuracy of 0.85, and the learned classifier reaches the same accuracy of 0.85. The single number cannot tell the two apart, nor reveal which mistakes either makes: does it miss the frost days, or does it raise false alarms? The confusion matrix breaks the count of predictions down by what was true and what was predicted (Fig.~\\ref{fig_cm_dict}): the learned classifier detects one of the six frost days at the cost of one false alarm, while the baseline detects none. Consider a finite dataset with m data points, each characterized by a feature vector x and a label y from a finite label space Y = 1, …, k. For a given hypothesis h, the confusion matrix is a k × k matrix whose row c collects the data points with true label y = c and whose column c' collects those with prediction h(x) = c' : the entry at position (c, c') is the number of data points with y = c and h(x) = c'. The diagonal therefore counts the correctly classified data points, and the off-diagonal entries count each kind of misclassification separately. The standard classification metrics are read off the confusion matrix. The accuracy is the sum of the diagonal divided by m. Normalizing a row by its sum yields the fraction of data points of that class that are detected — for the positive class, the recall — while normalizing a column yields the fraction of predictions of that class that are right, the precision. On a dataset with label skewness, the confusion matrix exposes what a large accuracy can hide: a classifier that never predicts the rare class fills the row of that class with misclassifications while keeping most predictions correct, as the always-above-zero baseline in Fig.~\\ref{fig_cm_dict} does (see baseline).", "synonyms": [], "see_also": [ "classification", "accuracy", "precision", "recall", "labelskewness", "baseline", "label", "labelspace", "matrix" ], "demo": "https://dictionaryofml.org/terms/cm.py", "notebook": "https://dictionaryofml.org/terms/cm.ipynb" }, { "key": "convex", "name": "convex", "url": "https://dictionaryofml.org/terms/convex.html", "pdf": "https://dictionaryofml.org/terms/convex.pdf", "abstract": "A subset of the Euclidean space is convex if it contains the line segment between any two of its points. A function is convex if its epigraph is a convex set. A convex optimization problem is the minimization of a convex function over a convex set. empirical risk minimization is a convex optimization problem whenever the hypothesis space is parameterized by a convex set of model parameters and the loss function is convex in those model parameters. Any local minimizer of a convex objective function is necessarily also a global minimizer, so a method that reaches a local minimum has solved the optimization problem.", "description": "Many machine learning methods learn model parameters by an iterative optimization method: each iteration updates the current model parameters to reduce an objective function within a neighborhood of these model parameters. Widely used examples are gradient descent and its variants. These optimization methods work particularly well when the objective function is a convex function of the model parameters. Indeed, for a convex function every local minimum is necessarily a global minimum: any choice of model parameters at which no small change reduces the objective function already attains the smallest value that the objective function takes anywhere. The average squared error loss of linear regression is a convex function of the model parameters, and so is the objective function of the support vector machine. For differentiable convex objective functions and suitable step size choices, gradient descent converges to such a minimum . Convexity is defined both for sets and for functions. For sets, the definition is geometric. A subset C Rd of the Euclidean space Rd is referred to as convex if it contains the line segment between any two points w, w' ∈ C in that set, i.e., \\expcoeff \\weights + (1-\\expcoeff) \\weights' \\in \\mathcal{C} \\quad \\text{for all } \\expcoeff \\in [0,1] \\text{.} Similarly, a function f: Rd → R is convex if its epigraph ( w,t ) ∈ Rd+1 : t ≥ f(w) is a convex set . Examples of a convex set and of a convex function are illustrated in Fig.\\ \\ref{fig_convex_set_function_dict}. Two families of convex sets appear throughout machine learning. The first is the halfspace w ∈ Rd : a w ≤ b. It is determined by a nonzero normal vector a ∈ Rd 0, which fixes the direction of the bounding hyperplane, and an offset b ∈ R, which fixes its position. If two points satisfy the inequality, so does every point of the segment between them (). The second is the convex hull of finitely many vectors x(1), …, x(m), defined as the set of all their convex combinations, \\[ \\operatorname{conv}\\big\\{ \\featurevec^{(1)}, \\ldots, \\featurevec^{(\\samplesize)} \\big\\} \\defeq \\Big\\{ \\sum_{\\sampleidx=1}^{\\samplesize} \\expcoeff_{\\sampleidx} \\featurevec^{(\\sampleidx)} \\; : \\; \\expcoeff_{\\sampleidx} \\geq 0 \\text{ for } \\sampleidx = 1, \\ldots, \\samplesize, \\; \\sum_{\\sampleidx=1}^{\\samplesize} \\expcoeff_{\\sampleidx} = 1 \\Big\\} \\text{.} \\] It is the smallest convex set that contains these vectors (). halfspaces are not merely one example among others: every closed convex set is given by the intersection of all halfspaces that contain it (; ) \\begin{equation} \\mathcal{C} = \\bigcap \\big\\{ \\mathcal{H} \\subseteq \\reals^{\\featuredim} \\;:\\; \\mathcal{H} \\text{ is a halfspace}, \\; \\mathcal{C} \\subseteq \\mathcal{H} \\big\\} \\text{.} \\end{equation} In general, intersecting a subset of the halfspaces containing a convex set yields a convex set that contains the original one, i.e., an outer approximation (see Fig.~\\ref{fig_convex_outer_dict}). Approximate inference in probabilistic models defined over a graph uses this construction: the set of achievable marginal distributions is a convex set described by a large number of linear inequalities, and message-passing methods optimize over an outer approximation of it obtained from a subset of those inequalities (; ). Returning to \\eqref{equ_halfspace_intersection_dict}, that intersection can be narrowed: it needs to consider only those halfspaces which are bounded by a supporting hyperplane. Every boundary point of a non-empty convex set carries such a hyperplane (), and each point outside the set is already excluded by one of the halfspaces that these supporting hyperplanes bound: for a point x C and C closed, there is a unique nearest point p ∈ C to x. That nearest point is a boundary point of C, and therefore one of the points that carry a supporting hyperplane: a point of the interior could be moved a little toward x and would still lie in C, so it would not be the nearest one. The hyperplane through p with normal vector x - p supports C at p, and the halfspace it bounds contains C but not x (see Fig.~\\ref{fig_convex_nearest_dict}). Fig.~\\ref{fig_convex_halfspace_dict} shows eight supporting hyperplanes of an ellipse. For the epigraph of a differentiable convex function f, the supporting hyperplane is read off the gradient: at the boundary point (w, f(w)), its normal vector is (f(w), -1), since \\[ \\big(\\nabla f(\\weights)^{\\top}, -1\\big) \\left[ \\begin{pmatrix} \\weights' t \\end{pmatrix} - \\begin{pmatrix} \\weights f(\\weights) \\end{pmatrix} \\right] \\leq 0 \\quad \\text{for every } \\big(\\weights'^{\\top}, t\\big)^{\\top} \\in \\operatorname{epi} f \\text{.} \\] This is the first-order condition f(w') ≥ f(w) + f(w) (w' - w) written as a statement about the epigraph . Of the two families of convex sets introduced above, the convex hull is the one that characterizes the linear separability of data points with binary label values. Consider m data points with feature vectors x(r) ∈ Rd and labels y(r) ∈ -1,+1, for r = 1, …, m. These data points are linearly separable if some linear classifier predicts every label correctly, i.e., there are w ∈ Rd 0 and b ∈ R with y(r) ( w x(r) + b ) > 0 for r = 1, …, m. This holds precisely when the convex hull of the feature vectors labeled +1 and the convex hull of those labeled -1 do not intersect (). Separability is therefore a property of two convex sets built from the data points. Convexity also appears in the study of probabilistic models, where it singles out a widely used family. In particular, an exponential family is constituted by the probability distributions whose probability density function has the form \\[ p(\\featurevec; \\weights) = h(\\featurevec) \\exp\\big(\\weights^{\\top} \\suffstat(\\featurevec) - A(\\weights)\\big) \\text{.} \\] This probability density function is parameterized by the vector w and the sufficient statistics t(·). The log-partition function A is fully determined by requiring p(x; w) to integrate to one. Its Hessian is the covariance matrix of the sufficient statistics t(x) under p(x; w), \\[ \\nabla^{2} A(\\weights) = \\E \\big\\{ \\suffstat(\\featurevec) \\suffstat(\\featurevec)^{\\top} \\big\\} - \\E \\big\\{ \\suffstat(\\featurevec) \\big\\} \\E \\big\\{ \\suffstat(\\featurevec) \\big\\}^{\\top} \\text{,} \\] which is positive semi-definite. A twice-differentiable function whose Hessian is positive semi-definite is convex, so A is a convex function on its domain (). The convexity of A implies that the negative log-likelihood function of a training set, up to an additive constant that does not depend on w, \\[ - \\sum_{\\sampleidx=1}^{\\samplesize} \\weights^{\\top} \\suffstat\\big(\\featurevec^{(\\sampleidx)}\\big) + \\samplesize A(\\weights) \\text{,} \\] is a convex function of w. Thus, for an exponential family, maximum likelihood estimation is a convex optimization problem.", "synonyms": [], "see_also": [ "euclidspace", "function", "epigraph", "convexopt", "minimum", "objfunc", "halfspace", "supportinghyperplane", "hessian" ], "demo": "https://dictionaryofml.org/terms/convex.py", "notebook": "https://dictionaryofml.org/terms/convex.ipynb" }, { "key": "covmtx", "name": "covariance matrix", "url": "https://dictionaryofml.org/terms/covmtx.html", "pdf": "https://dictionaryofml.org/terms/covmtx.pdf", "abstract": "The covariance matrix of a random vector consists of the covariances between its entries. In particular, the diagonal entries are the variances of the individual entries. Together with the mean vector, the covariance matrix collects the second-order statistics of a random vector. For a Gaussian random vector, the mean vector and the covariance matrix provide full information about the optimal method for linear regression: the model parameters of the optimal linear hypothesis map, along with the incurred risk, can be read off the covariance matrix.", "description": "The covariance matrix of a random vector x ∈ Rd is defined as the expectation (if it exists): \\covmtx{\\featurevec} \\defeq \\expect \\bigg \\{ \\big( \\featurevec - \\expect \\big\\{ \\featurevec \\big\\} \\big) \\big(\\featurevec - \\expect \\big\\{ \\featurevec \\big\\} \\big)^{\\top} \\bigg\\} \\text{.} The entry in row j and column j' is the covariance of the entries xj and xj', with the diagonal entries being the variances of the individual entries. The empirical counterpart computed from a dataset is the sample covariance matrix. Fig.~\\ref{fig_covmtx_ellipse} reads the matrix off the cloud of realizations it summarizes. Fig.~\\ref{fig_covmtx_mlpicture} illustrates the role of the covariance matrix for a simple regression task. The data points of a training set, a validation set and a test set are modeled as realizations of i.i.d. Gaussian random vectors z(r) = ( x(r), y(r) ) ~ μC (see multivariate normal distribution). The ellipse in Fig.~\\ref{fig_covmtx_mlpicture} is the constant-density contour z : (z - μ) C-1 (z - μ) = 4 of this probability distribution. Its principal axes point along the eigenvectors v(1), v(2) of the covariance matrix C, with lengths proportional to the square roots 1, 2 of the corresponding eigenvalues (see eigenvalue decomposition). The long axis of the ellipse is the direction in which feature and label vary together. The hypothesis map with the smallest risk under the squared error loss is linear for this probabilistic model, with slope given by the ratio of the covariance between x and y to the variance of x (see linear regression). Predicting one feature of a data point from the others is a common task in health records : each data point is a patient, the entries x1, …, xd of x are biomarker measurements, and the biomarker xj of a new patient, being expensive to measure, is to be predicted from the remaining ones. The smallest risk achievable by such a prediction can be read off the inverse ( x )-1 of an invertible covariance matrix, referred to as the precision matrix . Collect the remaining features in the vector x-j := ( x1, …, xj-1, xj+1, …, xd ). For a Gaussian random vector x ~ μx (see multivariate normal distribution), the conditional probability distribution of xj, given x-j, is Gaussian with a variance that does not depend on the value of x-j. This conditional variance is 1 / ( ( x )-1 )j,j . For data points modeled as realizations of i.i.d. random vectors x(r) ~ μx, this conditional variance is also the smallest risk that any hypothesis can achieve under the squared error loss, \\[ \\min_{\\hypothesis: \\reals^{\\featuredim-1} \\to \\reals} \\expect \\Big\\{ \\big( \\feature_{\\featureidx} - \\hypothesis\\big(\\featurevec_{-\\featureidx}\\big) \\big)^{2} \\Big\\} = 1 \\big/ \\big( \\big( \\covmtx{\\featurevec} \\big)^{-1} \\big)_{\\featureidx,\\featureidx} \\text{,} \\] i.e., the Bayes risk of this regression problem. It is attained by the Bayes estimator E xj x-j, which depends linearly on x-j in the Gaussian case . The diagonal of ( x )-1 therefore reads off a baseline, one entry per feature: a large diagonal entry marks a biomarker that the remaining biomarkers predict accurately, and no machine learning method can push the risk of predicting that biomarker below the reciprocal of the entry.", "synonyms": [], "see_also": [ "covariance", "matrix", "rv", "randomvector", "variance", "samplecovmtx", "mvndist", "evd", "eigenvalue", "linreg", "bayesrisk", "baseline" ], "demo": "https://dictionaryofml.org/terms/covmtx.py", "notebook": "https://dictionaryofml.org/terms/covmtx.ipynb" }, { "key": "data", "name": "data", "url": "https://dictionaryofml.org/terms/data.html", "pdf": "https://dictionaryofml.org/terms/data.pdf", "abstract": "This entry distinguishes three main uses of the term \\emph{data}. In a general sense, data refer to an abstract raw material that is used (or consumed) by machine learning methods. As a data point, it is an elementary information-carrying unit. As a dataset, it is an indexed collection of data points. A related regulatory term, personal data, is defined in the general data protection regulation and the EU AI Act.", "description": "A weather station measures the temperature at its location and stores the value 23.4 C together with a timestamp (see Fig.~\\ref{fig_data_dict}(a)). Such recorded values are data: representations of information, recorded in a form suitable for storage, communication, and processing by ML systems . Every machine learning method fits a model to data, and the usefulness of the hypothesis the method learns is limited by the quality and quantity of the available data. The word \\emph{data} is used in three technical senses, summarized in Fig.~\\ref{fig_data_dict}, and additionally in a legal sense. In its broadest use, data refer to recorded information. This sense applies when the distinction between unit and collection is irrelevant. Examples are raw sensor measurements such as the temperature reading in Fig.~\\ref{fig_data_dict}(a) and compound terms such as data parallelism, networked data, and data augmentation. As a formal unit, a data point is the fundamental information-carrying unit (Fig.~\\ref{fig_data_dict}(b)). Its information is contained in two types of properties: features (easy to measure or compute) and labels (difficult to measure or compute, or in need of human annotation) . As a formal collection, a dataset D = z(1), …, z(m) is the indexed collection of data points on which model training and validation are performed (Fig.~\\ref{fig_data_dict}(c)) . The word \\emph{data} also carries a regulatory meaning. The general data protection regulation and the EU AI Act use personal data and biometric data as defined legal terms. Here, data denote any recorded information about an identifiable natural person. Several regulatory obligations on ML systems, including the data minimization principle, the right to erasure, and the right to explanation, refer to data in this legal sense (; ).", "synonyms": [], "see_also": [ "datapoint", "dataset", "model", "hypothesis", "gdpr", "euaiact", "personaldata" ], "demo": "https://dictionaryofml.org/terms/data.py", "notebook": "https://dictionaryofml.org/terms/data.ipynb" }, { "key": "dataleakage", "name": "data leakage", "url": "https://dictionaryofml.org/terms/dataleakage.html", "pdf": "https://dictionaryofml.org/terms/dataleakage.pdf", "abstract": "Data leakage is the use, during the training or the evaluation of a machine learning method, of information that is unavailable at the time a prediction must be delivered. The most direct route is a feature computed from the label or from quantities that become observable only together with it. Leakage also enters through the machine learning pipeline, as when a preprocessing step is computed from the entire dataset before the split into training set, validation set, and test set, or when time-ordered data points are assigned to them by random shuffling. The average loss on the validation set or test set then no longer estimates the risk of the learned hypothesis, since the evaluation grants access to information that the deployed method lacks. An error far below a stated baseline is a symptom that warrants a per-feature audit of availability at prediction time.", "description": "Consider a classifier trained to predict, from the weather measurements available on a given morning, whether the coming day will be wet. One of its features is the day's total rainfall, copied from the same table column that defines the label. The accuracy on the training set and on the test set is then close to one, yet the deployed classifier is useless: on the morning of a new day, that day's total rainfall has not been measured yet. Data leakage is the use, during the training or the evaluation of a machine learning method, of information that is unavailable at the time a prediction must be delivered . Fig.~\\ref{fig_dataleakage_dict} depicts the most direct route: a leaked feature is computed from the label, xleak = g(y) for some map g, or from quantities that become observable only together with the label — in the opening example, the rainfall total that defines wetness. Leakage also enters through the machine learning pipeline: a preprocessing step such as data normalization computed from the entire dataset before the split passes summary statistics of the validation set and test set into the training, and splitting time-ordered data points by random shuffling lets the training read the future of the data points it is later evaluated on . The presence of test set data points themselves in the training set is the sibling defect of test set contamination. The reported error misleads because the average loss on a validation set or test set estimates the risk of the learned hypothesis h only if the evaluation reproduces the conditions of prediction. Leakage breaks this premise: the evaluated hypothesis reads a feature that is missing at prediction time, so the reported error refers to a different prediction task than the one the method faces after deployment. An error far below a stated baseline is a symptom of leakage — near-perfect accuracy on a task known to be hard warrants an audit before the result is trusted . The audit asks, for each feature, whether its value is computable at prediction time. Pipeline leakage is avoided by computing every preprocessing step from the training set only and by splitting time-ordered data points by time .", "synonyms": [], "see_also": [ "testsetcontamination", "trainset", "valset", "testset", "risk", "generalization", "mlpipeline", "preprocessing", "baseline" ] }, { "key": "datapoint", "name": "data point", "url": "https://dictionaryofml.org/terms/datapoint.html", "pdf": "https://dictionaryofml.org/terms/datapoint.pdf", "abstract": "A data point is the elementary information-carrying unit on which machine learning methods operate. The properties of a data point fall into two categories: features, which are easily measurable or computable, and labels, which are higher-level facts that typically can only be determined using human expertise. Whether a given property is treated as a feature or a label is a design choice that depends on the machine learning application.", "description": "A data point is an object that conveys information~; such objects are the elementary units on which machine learning methods operate. Examples include students, radio signals, trees, images, RVs, real numbers, or proteins. data points of the same type are described by two categories of properties. The first category includes features that are measurable or computable properties of a data point. They can be automatically extracted or computed using sensors, computers, or other data collection systems. The second category includes labels that are higher-level facts, or quantities of interest, that typically require human expertise or domain knowledge to determine rather than being directly measurable. Whether a given property serves as a feature or a label is ultimately a design choice that depends on the resources available in a given machine learning application. Fig.\\ \\ref{fig:datapoint_cowherd_dict} shows an image as an example of a data point. Several properties of the image can serve as features: the color intensities x1, …, xd of all image pixels, the timestamp xd+1 of the image capture, and the spatial location xd+2 of the image capture. Higher-level facts can serve as labels: the number of cows y1, the number of wolves y2, and the condition of the pasture y3 (e.g., healthy or overgrazed). For a data point that represents a patient, a feature could be the body weight, while the label could be the presence of cancer, the ground-truth fact of interest. Determining this label requires a medical expert (or even a committee of experts) to examine the patient. A property treated as a label in one setting (e.g., a cancer diagnosis) may serve as a feature in another, where reliable automation (e.g., image analysis) allows the property to be computed without human intervention. machine learning aims to predict the label of a data point from its features. Both kinds of properties are error-prone. For many machine learning applications, it is rarely possible to access the true labels. For example, a medical expert's diagnosis can be wrong, rendering it a noisy proxy for the patient's true condition. Errors of this kind, called label noise, arise whenever a label is only a proxy for the true quantity of interest, whether that proxy is produced by human judgment or by an automated system. features are also a source of error within an machine learning method. A feature is obtained by measuring or computing a property of a data point. Physical and chemical sensing devices have a finite measurement uncertainty (), and computing a feature on a finite computer introduces rounding errors (). These imperfections make the recorded value deviate from the true feature value. This discrepancy is feature noise. machine learning methods should be robust to both feature and label noise. Their predictions on new data points should not degrade sharply when some features or labels in the training set are corrupted. The EU AI Act reflects this concern: training data must, to the best extent possible, be ``free of errors'' (), and high-risk AI systems must attain an appropriate level of robustness (). Robustness to feature and label noise can be improved by data augmentation, which deliberately perturbs the features or the label of a training data point (see Fig.~\\ref{fig_datapoint_augmentation}). Training on such perturbed data points acts as regularization and reduces the influence of corrupted feature or label values on the learned hypothesis. See also: data, feature, label, dataset, robustness, EU AI Act, data augmentation.", "synonyms": [], "see_also": [], "demo": "https://dictionaryofml.org/terms/datapoint.py", "notebook": "https://dictionaryofml.org/terms/datapoint.ipynb" }, { "key": "dataset", "name": "dataset", "url": "https://dictionaryofml.org/terms/dataset.html", "pdf": "https://dictionaryofml.org/terms/dataset.pdf", "abstract": "A dataset is a finite collection of data points on which model training and validation are performed. Strictly speaking, a dataset is a set of distinct data points. With slight abuse of terminology, the term is also used interchangeably with sample, a finite sequence that may contain repetitions. In practice, machine learning methods often do not have access to the actual dataset. Instead, they must use approximate representations such as tables in the relational model.", "description": "A dataset is a collection D = z(1), …, z(m) of data points. machine learning methods use a dataset for model training and validation. Strictly speaking, a dataset is an unordered collection of distinct data points, i.e., a set with no repetitions. In machine learning literature, however, the term is often used for a sample: a sequence of m data points, indexed by r = 1, …, m, that may contain repetitions (see Fig.~\\ref{fig_dataset_set_dict}). The notion of a dataset is broad: data points may represent concrete physical entities (such as humans or animals) or abstract objects (such as numbers). For illustration, Fig.~\\ref{fig_cows_dataset_dict} depicts a dataset whose data points are cows. In machine learning applications, it is often not possible to directly access the underlying dataset. For instance, accessing the dataset in Fig.~\\ref{fig_cows_dataset_dict} would require visiting the cow herd. In practice, a dataset is represented within some data model, a rigorous formalism for representing and processing data (; ; ; ). For example, the relational model organizes data as a collection of tables, or relations (; ). A single table consists of rows and columns, where each row corresponds to a single data point and each column represents a specific attribute of a data point. The order of rows is immaterial, and each attribute is associated with a domain that specifies its set of admissible values. machine learning methods use these attributes as the features or the label of a data point, so the attribute domains correspond to the feature space and the label space. Table~\\ref{tab:cowdata_dict} shows an example obtained from the dataset in Fig.~\\ref{fig_cows_dataset_dict}. \\begin{table}[H] \\refstepcounter{table} \\caption*{ \\centering \\scshape TABLE \\thetable [0.5ex] \\scshape Relation (or Table) Representing a Dataset of Cows } \\centering \\begin{tabular}{lcccc} \\hline \\textbf{Name} & \\textbf{Weight} & \\textbf{Age} & \\textbf{Height} & \\textbf{Stomach temperature} \\hline Zenzi & 100 & 4 & 100 & 25 Berta & 140 & 3 & 130 & 23 Resi & 120 & 4 & 120 & 31 \\hline \\end{tabular} \\end{table} A table captures the content of a dataset, but not whether that content is appropriate for a given purpose. A dataset used in ethical and lawful ML systems is required to satisfy further properties. For example, the EU AI Act requires that datasets on which high-risk ML systems are trained be ``sufficiently representative'' of the intended use cases (). trustworthy artificial intelligence also requires documenting how a dataset was assembled, for example through a datasheet that records the dataset's motivation, composition, and collection process . When the data points are personal data, this becomes a legal duty: under the purpose limitation principle, the general data protection regulation requires that such data be collected for ``specified, explicit and legitimate purposes'' that are fixed at the time of collection ().", "synonyms": [], "see_also": [ "datapoint", "sample", "data", "relationalmodel", "feature", "featurespace", "labelspace", "euaiact" ], "demo": "https://dictionaryofml.org/terms/dataset.py", "notebook": "https://dictionaryofml.org/terms/dataset.ipynb" }, { "key": "decisiontree", "name": "decision tree", "url": "https://dictionaryofml.org/terms/decisiontree.html", "pdf": "https://dictionaryofml.org/terms/decisiontree.pdf", "abstract": "A decision tree is a flowchart-like representation of a hypothesis map: a root node reads in the feature vector of a data point and evaluates a basic test on it. Depending on the test result, the feature vector is forwarded to one of the child nodes. The child nodes apply further tests and forward the feature vector accordingly. This forwarding continues until the feature vector reaches a leaf node. A leaf node has no children and represents one specific function value, i.e., a prediction. The tree partitions the feature space into decision regions, one per leaf node, on which the represented hypothesis is constant. Decision trees serve as base models of the random forest and of gradient-boosted decision tree methods. Small decision trees yield interpretable machine learning if the user can comprehend the tests executed at their nodes.", "description": "A bank's loan approval can be written as a chain of simple tests on the applicant's features that ends in an ``approve'' or ``reject'' prediction. For example, is the income above a threshold? Is the credit score large enough? A decision tree is such a chain in general form: a flowchart-like representation of a hypothesis map h. More formally, a decision tree is a directed acyclic graph containing a root node that reads in the feature vector x of a data point. The root node then forwards the data point to one of its child nodes based on some elementary test on the features x. If the receiving child node is not a leaf node, i.e., it has child nodes itself, it represents another test. Based on the test result, the data point is forwarded to one of its descendants. This testing and forwarding of the data point is continued until the data point ends up in a leaf node without any children, which carries the prediction (Fig.~\\ref{fig_decision_tree_dict}). The leaf nodes partition the feature space into decision regions; in Fig.~\\ref{fig_decision_tree_dict}, each test compares an entry of x with a threshold, so the decision regions are rectangles. Next to each node, Fig.~\\ref{fig_decision_tree_dict} highlights the part of the feature space that contains the feature vectors reaching that node. training a decision tree amounts to extending a leaf node by attaching another decision tree. In the simplest case, a leaf node is replaced with a test node whose children are new leaf nodes (Fig.~\\ref{fig_decision_tree_grow_dict}). Widely used training methods for decision trees include CART , ID3 , and C4.5 . Starting from a single leaf node, CART repeatedly replaces a leaf node with a test node: among candidate tests that compare a single feature with a threshold, it greedily selects the one that most reduces an impurity measure of the label values routed to the new children. Common impurity measures are the Gini index and the entropy for classification, and the variance for regression . This selection is a combinatorial search over candidate tests rather than a gradient-based update of continuous model parameters. The growth stops when a leaf node contains too few data points or no candidate test reduces the impurity substantially . Fig.~\\ref{fig_decisiontree_picture_dict} shows the result of this training on real data: each day of 2024 at the Finnish Meteorological Institute weather station Helsinki Kaisaniemi is a data point, with the minimum temperature of the day as its feature and the maximum temperature as its label. The learned depth-2 tree is a piecewise constant hypothesis with four pieces, one per leaf node. Decision trees serve as base models of the random forest and of gradient-boosted decision tree methods, and their explicit tests make each prediction traceable. A decision tree with a small number of nodes is often considered interpretable: its computation can be traced by following the data point from the root node to a leaf node . This makes small decision trees a common choice in interpretable machine learning. See also: decision region, classification, random forest, gradient-boosted decision tree, feature importance, interpretable machine learning.", "synonyms": [], "see_also": [], "demo": "https://dictionaryofml.org/terms/decisiontree.py", "notebook": "https://dictionaryofml.org/terms/decisiontree.ipynb" }, { "key": "distshift", "name": "distribution shift", "url": "https://dictionaryofml.org/terms/distshift.html", "pdf": "https://dictionaryofml.org/terms/distshift.pdf", "abstract": "Distribution shift is a mismatch between the probability distribution underlying the training set and test set of a machine learning method and the probability distribution of the data points that the learned hypothesis faces after deployment. The average loss on the test set estimates the risk under the training probability distribution only, so a small test error is no evidence for a small loss on data points outside the test set. Factoring the joint probability distribution of features and labels separates the named special cases of covariate shift and label shift.", "description": "The Finnish Meteorological Institute (FMI) station Helsinki Kaisaniemi records the minimum and the maximum air temperature of each day. Every summer day is a data point: its feature is the day's minimum temperature, its label the day's maximum. A linear hypothesis h(x) = 0.481 x + 14.5, learned from 62 days of June--August 2026 by minimizing the average squared error loss, reaches a training error of 5.0 and a test error of 6.5 on a test set of the 30 remaining summer days. Deployed at the northernmost FMI station, Utsjoki Nuorgam (70.1\\,N), on the 90 days of December 2025--February 2026, the same hypothesis incurs an average error of 242.1, which is 37 times its test error (Fig.~\\ref{fig_distshift_ml_dict}). Every winter morning at Nuorgam is colder than each summer morning in the training set, and the extrapolation overshoots: the mean winter day at Nuorgam has a minimum of -19.8C and a maximum of -9.6C, while h assigns such a morning a maximum of +5.0C. Distribution shift is a mismatch between the probability distribution p underlying the training set and test set of a machine learning method and the probability distribution p' of the data points the learned hypothesis faces after deployment . The average loss on the test set estimates the risk under p, since the independent and identically distributed assumption covers draws from p, not from p'. A small test error is therefore no evidence about the loss incurred under p' p: the densities in Fig.~\\ref{fig_distshift_dict} show how a threshold placed for p fails under p', and the two stations of Fig.~\\ref{fig_distshift_ml_dict} realize the failure with measured temperatures. The named special cases of distribution shift correspond to the two factorizations of the joint probability distribution of a data point (x, y) . Under the factorization p(x, y) = p(y x) p(x), covariate shift changes only the marginal p(x) of the covariates (the features) and leaves the conditional p(y x) untouched, as when a second weather station sits in a colder climate but cold mornings precede rain in the same way; concept drift changes that conditional itself over time, as when a warming climate alters which morning temperatures precede rain. Under the reverse factorization p(x, y) = p(x y) p(y), label shift changes only the frequency p(y) of the label values and leaves the class-conditional p(x y) untouched. The two-station example is a pure case of none of the three: the marginal of the feature moves, since winter mornings are colder, and the conditional moves as well, since a line fitted to the winter days has slope 0.77 against the summer slope 0.48. Distribution shift is detected by comparing summary statistics of the features between the training set and the data points arriving after deployment, or by monitoring the incurred loss whenever labels become available . One response is sample weighting: each data point of the training set is weighted by the ratio p'(x)/p(x), which corrects covariate shift provided that the conditional p(y x) is unchanged and that every feature vector occurring under p' also occurs under p, so that the ratio is defined . Both conditions fail in the two-station example: no reweighting of summer days produces a winter morning at -20C. Responses that remain applicable are transfer learning and fine-tuning on data points from p', and online learning, which updates the hypothesis as data points from the drifting probability distribution arrive .", "synonyms": [ "dataset shift", "domain shift" ], "see_also": [ "generalization", "iidasspt", "probdist", "risk", "trainset", "testset", "transferlearning", "sampleweighting", "onlinelearning" ], "demo": "https://dictionaryofml.org/terms/distshift.py", "notebook": "https://dictionaryofml.org/terms/distshift.ipynb" }, { "key": "eigenvalue", "name": "eigenvalue", "url": "https://dictionaryofml.org/terms/eigenvalue.html", "pdf": "https://dictionaryofml.org/terms/eigenvalue.pdf", "abstract": "An eigenvalue of a square matrix A is a number λ for which some nonzero vector x satisfies Ax = λx. Such a vector is an eigenvector of A. Multiplying an eigenvector by the matrix rescales it without moving it off the line it spans. The eigenvalues of a diagonalizable matrix form the diagonal factor of its eigenvalue decomposition. An iterative method that applies an affine update, such as gradient descent for linear regression, converges to the unique fixed point of the update, for every initialization, exactly when each eigenvalue of the update matrix has magnitude less than one. The second-smallest eigenvalue of the Laplacian matrix of an undirected graph measures how well the graph is connected and underlies spectral clustering.", "description": "Multiplying a vector by a square matrix normally moves it off the line it spans. A few nonzero vectors stay on that line, and the factor by which one of them is scaled is an eigenvalue. Formally, a number λ ∈ R is an eigenvalue of a square matrix A ∈ Rd × d, one with d rows and d columns, if A x = λ x for some nonzero vector x ∈ Rd; such a vector is an eigenvector of A (see Fig.\\ \\ref{fig_eigenvalue_dict}). Eigenvalues can be used to study iterative machine learning methods with an update of the form \\[ \\weights^{(\\iteridx+1)} = \\mM \\weights^{(\\iteridx)} + \\vb \\text{.} \\] An example is gradient descent for linear regression, whose update operator η : w M w + b is affine with M = I - (2η/m) XX, where η is the learning rate, m the number of data points and X the feature matrix of the training set (see linear regression). Such an iteration converges to the unique fixed point w of η, for every initialization, if and only if the spectral radius ρ of M, the largest magnitude of an eigenvalue of M, satisfies ρ < 1 . Beyond the convergence of iterative methods, eigenvalues are also used to measure the connectivity of an undirected graph G with n nodes. The Laplacian matrix G of G is positive semi-definite, with eigenvalues 0 = 1 ≤ 2 ≤ … ≤ n. The second-smallest eigenvalue 2, the algebraic connectivity of G, quantifies connectivity: 2 > 0 if and only if G is connected, and 2 close to zero indicates two clusters of nodes with few edges between them (; ; ). Fig.~\\ref{fig_eigenvalue_conn_dict} shows three edge sets on the same six nodes: adding edges increases 2 from 0, for two clusters without a connecting edge, to 6, for the complete graph (\\texttt{pythondemos/eigenvalue.py}). The eigenvector corresponding to 2, referred to as the Fiedler vector, can be used to partition the graph into two clusters according to the sign of its entry in that eigenvector (; ). See also: matrix, eigenvector, eigenvalue decomposition, spectral radius, gradient descent, linear regression, contractive operator, fixed-point iteration, Laplacian matrix, algebraic connectivity, Fiedler vector, spectral clustering.", "synonyms": [], "see_also": [], "demo": "https://dictionaryofml.org/terms/eigenvalue.py", "notebook": "https://dictionaryofml.org/terms/eigenvalue.ipynb" }, { "key": "em", "name": "expectation–maximization", "url": "https://dictionaryofml.org/terms/em.html", "pdf": "https://dictionaryofml.org/terms/em.pdf", "abstract": "The expectation–maximization (EM) algorithm is an iterative optimization method for approximately solving maximum likelihood optimization problems that are difficult to solve directly, such as fitting a Gaussian mixture model. Each iteration alternates an E-step, which computes the posterior distribution of an unobserved auxiliary attribute under the current model parameters, and an M-step, which minimizes a surrogate objective function derived from this posterior distribution. The surrogate upper-bounds the negative log-likelihood and is tight at the current iterate, so each iteration never increases the negative log-likelihood: EM is a majorize–minimize method. The method is a fixed-point iteration whose fixed points are model parameters that minimize their own surrogate.", "description": "The nightly minimum temperatures recorded over one year at the weather station of Krems an der Donau (Austria) scatter around two regimes — cold-season and warm-season nights. A Gaussian mixture model with two components captures such a two-regime distribution, but maximizing its likelihood has no closed-form solution. The standard method for fitting it is the EM algorithm : an iterative optimization method for approximately solving certain maximum likelihood optimization problems that are difficult to solve directly (; ). To motivate the EM algorithm and explain its construction, consider a machine learning application involving a single observed data point with feature x ∈ X, where X is a finite feature space. The data generation is modeled via a probabilistic model that consists of a random variable x' with a probability mass function x'·;w. Here, the actual model parameters w ∈ W—used for the data generation via sampling from x'·;w—are unknown. A widely used approach for estimating these model parameters is via the solutions of the following maximum likelihood problem: \\begin{equation} \\min_{\\weights \\in \\paramspace} - \\log \\pmf{\\feature'}{\\feature;\\weights}. \\end{equation} For some probabilistic models, such as a Gaussian mixture model, this optimization problem can be difficult to solve directly. As a work-around, one can often introduce an auxiliary attribute y ∈ Y, generated via some random variable y'. For a suitable choice of this attribute, the corresponding probabilistic model x',y'·,·;w yields the following, much easier maximum likelihood problem: \\begin{equation} \\min_{\\weights \\in \\paramspace} - \\log \\pmf{\\feature',\\truelabel'}{\\feature,\\truelabel;\\weights}. \\end{equation} The attribute y is introduced solely to simplify \\eqref{equ_def_complete_EM_dict}, but it is not observed in practice—only the feature x is available. Thus, \\eqref{equ_def_complete_EM_dict} cannot be solved directly, as the value of y to plug into the probability mass function is unknown x',y'x,y;w. The EM method resolves this dilemma by alternating between two steps. The E-step computes a “soft’’ estimate of the auxiliary attribute y: the posterior distribution y'|x'·;w, i.e., the probability of each value of y given the observed feature, under the current choice w for the model parameters. The M-step minimizes a surrogate objective function derived from this posterior distribution. Together, one E-step and one M-step constitute one full iteration of the EM method. In more detail, the E-step produces the following function: \\[ Q(\\weights \\mid \\widehat{\\weights}) \\defeq - \\sum_{\\truelabel \\in \\labelspace} \\pmf{\\truelabel'|\\feature'}{\\truelabel;\\widehat{\\weights}} \\log\\!\\Big( \\pmf{\\feature',\\truelabel'}{\\feature,\\truelabel;\\weights} /\\pmf{\\truelabel'|\\feature'}{\\truelabel;\\widehat{\\weights}} \\Big), \\] and the M-step minimizes Q(w w) over w ∈ W. This function satisfies the following two key properties (; ): 1) upper bound Q(ww) ≥ - x'x;w for all w ∈ W; and 2) tightness Q(ww) =- x'x;w. To summarize, during each iteration, EM minimizes an upper-bounding surrogate objective function that is tight at the current iterate w. Thus, EM is a majorize–minimize method for approximately solving \\eqref{equ_def_ML_EM_dict}. The EM method is also a fixed-point iteration w(t+1) = F(w(t)) with the operator \\fixedpointop(\\widehat{\\weights}) \\defeq \\argmin_{\\weights \\in \\paramspace} Q(\\weights \\mid \\widehat{\\weights})\\text{.} By the upper-bound and tightness properties above, each application of F never increases the objective function of \\eqref{equ_def_ML_EM_dict} — the negative log-likelihood — so monotone descent, rather than a contractive operator property, justifies convergence. The fixed points of F are model parameters that minimize their own surrogate: there, the surrogate touches the negative log-likelihood from above, so no further descent step is available to the method. The above construction and analysis of EM can be extended to more general settings involving multiple data points and infinite feature spaces such as Rd (see for further details). Fig.~\\ref{fig_em_dict} shows EM at work on real data: a Gaussian mixture model with two components, fitted to the 366 nightly minimum temperatures of 2024 at Krems, recovers the cold-season and warm-season regimes, and the negative log-likelihood descends monotonically to a fixed point. See also: maximum likelihood, Gaussian mixture model, majorize–minimize, fixed point, posterior distribution, optimization problem, probabilistic model.", "synonyms": [], "see_also": [], "demo": "https://dictionaryofml.org/terms/em.py", "notebook": "https://dictionaryofml.org/terms/em.ipynb" }, { "key": "ensemble", "name": "ensemble", "url": "https://dictionaryofml.org/terms/ensemble.html", "pdf": "https://dictionaryofml.org/terms/ensemble.pdf", "abstract": "An ensemble method combines several machine learning methods, each referred to as a base learner, into one predictor. The predictions of the base learners are aggregated by averaging in regression or by a majority vote in classification. The combination often predicts more reliably than any single base learner. Ensemble methods differ in how they construct the base learners: bootstrap aggregating trains each on a resampled copy of the training set, boosting runs them sequentially so that each corrects the errors of its predecessors, and stacking trains different models on the same training set.", "description": "For each incoming email, three different spam filters cast a vote, and the message is moved to the spam folder when at least two of them say spam. Such a combination of machine learning methods is an ensemble method: each combined method is a base learner, and the combination often predicts more reliably than any single base learner . The base learners can be empirical risk minimization-based, using different choices for the loss, model, and training set. The aggregation of their predictions can amount to averaging (in regression) or to a majority vote (in classification), as in the spam example (Fig.~\\ref{fig_ensemble_dict}). Different ensemble methods use different constructions for the base learners. For example, bootstrap aggregating methods (such as a random forest) use random sampling to construct slightly different training sets for each base learner. On the other hand, boosting methods run the base learners sequentially, i.e., each base learner tries to correct the prediction errors of the previous ones. A third family of ensemble methods is stacking, where base learners are trained on the same training set but with different models.", "synonyms": [], "see_also": [ "bagging", "boosting", "stacking", "baselearner", "randomforest" ] }, { "key": "erm", "name": "empirical risk minimization", "url": "https://dictionaryofml.org/terms/erm.html", "pdf": "https://dictionaryofml.org/terms/erm.pdf", "abstract": "machine learning methods aim to find a hypothesis that yields accurate predictions, reflected by a small loss. empirical risk minimization (ERM) formalizes this: it selects a hypothesis with minimal empirical risk, i.e., minimal average loss on a training set D(train). Different machine learning methods arise from different choices of the model H and the loss function L, both of which depend on the feature space X and the label space Y of a given method. The entry also covers the online form of ERM (Follow-The-Leader) and contrasts ERM with reinforcement learning methods that use a different access mechanism for the loss values.", "description": "A weather forecaster predicting tomorrow's maximum daytime temperature is useful only if its predictions are accurate on most days. This forecaster is a hypothesis: it maps a day's features to a prediction of tomorrow's temperature, and its loss quantifies how far that prediction lies from the measured value. More generally, machine learning methods aim to find a hypothesis that incurs a small loss for any data point. But what does \\emph{any} data point mean? One way to make this notion precise is to use a probabilistic model for the generation of data points. If data points are assumed to be drawn from a probability distribution, then the risk of a hypothesis is defined as the expectation of its loss under that probability distribution. The ideal choice is then the hypothesis with minimal risk. However, in most machine learning applications, the underlying probability distribution is not known. Instead, only a finite training set D(train) of data points is available. ERM replaces the intractable risk with its sample-average surrogate, the empirical risk on D(train), and picks the hypothesis that minimizes this surrogate (; ; ; ). For a fixed choice of loss function and hypothesis space, ERM is a map A that reads in a training set D(train) = z(r)r=1m of data points z(r) = x(r)y(r) and returns a learned hypothesis h = A(D(train)) ∈ H that minimizes the empirical risk on D(train), \\[ \\learnthypothesis \\in \\argmin_{\\hypothesis \\in \\hypospace} \\frac{1}{\\samplesize} \\sum_{\\sampleidx=1}^{\\samplesize} \\lossfunc{\\datapoint^{(\\sampleidx)}}{\\hypothesis} \\text{.} \\] The hypothesis h is chosen from the underlying hypothesis space (or model) H. For an ERM-based method, model training is to compute A. In practice, the map A is computed by an iterative optimization method. Many of these optimization methods can be represented as a fixed-point iteration that starts with an initial hypothesis h(0) and then repeatedly applies an update operator F, \\hypothesis^{(\\iteridx+1)} = \\fixedpointop\\big( \\hypothesis^{(\\iteridx)} \\big) \\text{, for } \\iteridx=0,1,\\ldots \\text{.} The update operator F depends on the training set, the loss function, and the hypothesis space. It is chosen such that its fixed points are minimizers of the empirical risk. For ERM using a parametric model H, the fixed-point iteration can be formulated directly in terms of the model parameters w instead of the hypothesis h. In particular, starting with initial model parameters w(0), the update operator F is applied repeatedly to the model parameters, \\weights^{(\\iteridx+1)} = \\fixedpointop\\big( \\weights^{(\\iteridx)} \\big) \\text{, for } \\iteridx=0,1,\\ldots \\text{.} One important example of such a fixed-point iteration is gradient descent, which updates the model parameters in the direction of the negative gradient of the empirical risk (which needs to be differentiable). ERM presupposes that the loss z(r)h can be evaluated for every hypothesis h ∈ H and every training data point z(r). This full-feedback requirement is what distinguishes ERM from reinforcement learning algorithms, which use partial feedback. In particular, at each time instant t, an reinforcement learning algorithm chooses a hypothesis h(t) delivering a prediction (or action) and only measures the loss incurred by this hypothesis. It has no information about the loss that would have been incurred by any other hypothesis h ∈ H h(t). ERM can also be implemented as an online algorithm, which is useful when the data points in D(train) can only be accessed sequentially. Such sequential access arises, for instance, when memory limits prevent loading the entire training set at once. The canonical online form of ERM is the Follow-The-Leader algorithm, which, at every round, solves a partial ERM problem on all data points seen so far. Assuming that the data point z(t) arrives at time instant t=1,2,…, Follow-The-Leader updates the learned hypothesis h(t) by \\learnthypothesis^{(\\timeidx)} \\in \\argmin_{\\hypothesis \\in \\hypospace} \\frac{1}{\\timeidx} \\sum_{\\sampleidx=1}^{\\timeidx} \\lossfunc{\\datapoint^{(\\sampleidx)}}{\\hypothesis} \\text{.} Implementing this partial ERM separately for each time instant can become computationally expensive (and wasteful), as it does not exploit the similarity between consecutive partial ERM problems. This similarity can be exploited by online gradient descent, which performs a single gradient descent step on the new data point z(t) at every time instant t . Different machine learning methods arise from different design choices for the two ingredients of the map A: the model H and the loss function L . Both depend on the nature of the data points, i.e., on the feature space X and the label space Y. For example, image data points with a high-dimensional feature space and a finite label space typically call for an artificial neural network together with the logistic loss. Conversely, data points with few real-valued features and a real-valued label often allow for a linear model together with the squared error loss. Fig.~\\ref{fig_erm_dict} illustrates ERM for a linear model on data points with a single feature x and label y. Each hypothesis is a linear map h(x) = w1 x + w0 with model parameters w0, w1. ERM picks the model parameters that minimize the empirical risk on D(train). Synonyms: empirical loss minimization.", "synonyms": [], "see_also": [ "optproblem", "loss", "lossfunc", "emprisk", "risk", "trainset", "hypospace", "model", "hypothesis", "algorithm", "training", "generalization", "optmethod", "gd", "onlinealgorithm", "onlineGD", "ftl", "reinforcementlearning" ], "demo": "https://dictionaryofml.org/terms/erm.py", "notebook": "https://dictionaryofml.org/terms/erm.ipynb" }, { "key": "evd", "name": "eigenvalue decomposition", "url": "https://dictionaryofml.org/terms/evd.html", "pdf": "https://dictionaryofml.org/terms/evd.pdf", "abstract": "An eigenvalue decomposition (EVD) is a factorization of a square matrix of the form A = V Λ V-1. The columns of the matrix V are eigenvectors of A. The diagonal matrix Λ contains the eigenvalues corresponding to these eigenvectors. A matrix that admits an EVD is referred to as diagonalizable; symmetric matrices and matrices with distinct eigenvalues are diagonalizable. The EVD can speed up computations: given an EVD of the matrix XX, each iteration of gradient descent for linear regression reduces to element-wise operations on vectors. The EVD of the Laplacian matrix of an undirected graph underlies spectral clustering: the second-smallest eigenvalue measures how well the graph is connected, and the signs of the entries of the corresponding eigenvector split the nodes into two clusters.", "description": "An EVD for a square matrix A ∈ Rd × d is a factorization of the form \\[ \\mA = \\mV {\\bm \\Lambda} \\mV^{-1} \\text{.} \\] The columns of the matrix V = ( v(1), …, v(d) ) are the eigenvectors of the matrix A. The diagonal matrix Λ = diag 1, …, d contains the eigenvalues j corresponding to the eigenvectors v(j). Multiplying the factorization by v(j) gives A v(j) = j v(j): each eigenvector is a direction that A only scales, by the factor j. Fig.~\\ref{fig_evd_directions} shows the two directions of the 2 × 2 matrix \\[ \\mA = \\begin{pmatrix} 1.3 & 0.8 0.4 & 0.9 \\end{pmatrix} \\text{,} \\] alongside a vector that A does turn. matrices that allow for an EVD are referred to as diagonalizable. Two sufficient conditions for a square matrix to be diagonalizable are that it is symmetric, or that its d eigenvalues are distinct . Not every square matrix is diagonalizable: the matrix \\[ \\mN = \\begin{pmatrix} 0 & 1 0 & 0 \\end{pmatrix} \\] has the single eigenvalue 0, and its eigenvectors span only the line (a, 0) : a ∈ R. Hence, there is no invertible matrix V whose columns are eigenvectors of N (\\texttt{pythondemos/evd.py}). The EVD can also speed up computations. Consider gradient descent for linear regression, with the update w(t+1) = w(t) - (2η/m) X ( X w(t) - y ) for the feature matrix X and label vector y of the training set (see linear regression). The matrix XX is symmetric, so it has an EVD XX = V Λ V with an orthogonal matrix V. In the transformed coordinates w(t) := V w(t), the update decouples into \\[ \\widetilde{w}_{\\featureidx}^{(\\iteridx+1)} = \\big( 1 - (2\\lrate/\\samplesize) \\eigval{\\featureidx} \\big) \\widetilde{w}_{\\featureidx}^{(\\iteridx)} + (2\\lrate/\\samplesize) \\big( \\mV^{\\top} \\featuremtx^{\\top} \\labelvec \\big)_{\\featureidx} \\text{.} \\] After the one-time computation of the EVD, each iteration amounts to element-wise multiplication and addition of vectors of length d (\\texttt{pythondemos/evd.py}). A concrete application of the EVD in machine learning is spectral clustering . Consider data points represented as the nodes i = 1, …, n of an undirected graph G, e.g., the users of a social network with edges given by friendships. The Laplacian matrix G of G is symmetric and positive semi-definite. Its EVD therefore uses an orthogonal matrix of eigenvectors, V-1 = V, and real nonnegative eigenvalues, ordered as 0 = 1 ≤ 2 ≤ … ≤ n. The second-smallest eigenvalue 2 is the algebraic connectivity of G: 2 > 0 if and only if G is connected, and 2 close to zero indicates two subsets of nodes joined by few edges . The corresponding eigenvector v(2), referred to as the Fiedler vector, assigns a number v(2)i to each node i. Grouping the nodes by the sign of v(2)i partitions the nodes of G into two clusters. Fig.~\\ref{fig_evd_fiedler_dict} shows this for a graph of six nodes, two clusters of three nodes each joined by a single edge: the entries of the Fiedler vector switch sign exactly between the two clusters. See also: matrix, eigenvector, eigenvalue, diagonalizable, invertible, orthogonal, Laplacian matrix, algebraic connectivity, Fiedler vector, spectral clustering, gradient descent, linear regression.", "synonyms": [], "see_also": [], "demo": "https://dictionaryofml.org/terms/evd.py", "notebook": "https://dictionaryofml.org/terms/evd.ipynb" }, { "key": "explainability", "name": "explainability", "url": "https://dictionaryofml.org/terms/explainability.html", "pdf": "https://dictionaryofml.org/terms/explainability.pdf", "abstract": "A machine learning method is explainable if there is an effective way to explain its predictions. The method delivers an explanation along with every prediction, and that explanation is effective if it lets a human user comprehend how the features of a data point drive the prediction made for it. Explainability is always relative to a specific user or group of users: an explanation can be effective for one and useless for another. Explainability can be measured by comparing the predictions of a learned hypothesis to the anticipations of a user before and after they are provided with an explanation.", "description": "A machine learning method is explainable for a human user if the explanations it provides let the user anticipate the predictions it delivers (; ). Explainability thus includes the notion of an explanation: each prediction is delivered with an explanation for this specific prediction (see Fig.~\\ref{fig_explainability_delivery_dict}), such as the feature values that drove it or, for image data, the relevant pixels. The definition asks for the existence of an explanation, not for a particular one: explainability is certified by exhibiting one explanation that works, and a single explanation the user cannot follow refutes nothing. Explainability is relative to a specific user or group of users: the same learned hypothesis can be explainable for one user and inscrutable for another. The usefulness of an explanation can be measured by how much it enables the user to anticipate the predictions on a curated test set: the user states an anticipation for each data point in the test set. If the user comprehends the explanations, these anticipations should be close to the predictions of the learned hypothesis (; ). A measure of this form is referred to as predictability . As with interpretability, predictability is the weaker notion: anticipating the predictions does not require comprehending the provided explanations. A probabilistic model for data generation provides a second measure of this predictability: the conditional differential entropy of the predictions given the anticipations (; ). The conditional differential entropy quantifies the uncertainty that remains about the predictions once the anticipations are known, so a smaller value means that the anticipations determine the predictions more tightly. In practice, this conditional differential entropy is unknown and must be replaced by an estimator, e.g., an estimate computed from the empirical frequencies of discretized predictions and anticipations on a test set. An explanation raises explainability when the user can reason with it. Fig.~\\ref{fig_explainability_dict} illustrates this for a user who reasons in terms of linear maps and anticipates the predictions of an opaque hypothesis (a kernel method). Without explanations, the anticipations deviate strongly from the predictions. Given a local linear approximation of the hypothesis around each data point (cf.\\ local interpretable model-agnostic explanations), the anticipations match the predictions almost exactly. The example presupposes that the user knows how to apply a linear approximation: the anticipations improve only because the user can evaluate the explanation for a data point. A user who cannot is left where they started. The international terminology standard for artificial intelligence defines explainability as the property of an artificial intelligence system to express important factors influencing its results in a way that humans can understand. An explanation is intended to answer the question of why the artificial intelligence system produced a result, without arguing that the result was optimal . The AI Risk Management Framework of the US National Institute of Standards and Technology (NIST) ties explainability to a representation of the mechanisms underlying the operation of an artificial intelligence system, and reserves interpretability for the meaning of the output in the context of the designed purpose . The two frameworks thus divide the terms differently. The expressed factors influencing a result, which the terminology standard places under explainability, are close to the NIST notion of interpretability; the NIST notion of explainability is instead close to interpretability as comprehension of the computation of the method. Regulation treats explainability as an ingredient of transparency. The EU AI Act requires that high-risk AI systems are sufficiently transparent to enable deployers to interpret their outputs . For individual automated decisions, the right to explanation entitles an affected person to a clear and meaningful account of the role that an artificial intelligence system played in the decision . The Colorado Automated Decision-Making Technology Act of 2026 requires that the deployer give the affected consumer a plain-language description of the decision and of the role the technology played in it, together with the means to request further information . What counts as an automated decision is drawn widely: the Court of Justice of the European Union held that a credit information agency computing a person's ability to meet future payments already makes one, when a third party draws strongly on that value .", "synonyms": [], "see_also": [ "explanation", "interpretability", "xaiterm", "eerm", "lime", "transparency", "righttoexplanation", "trustAI", "regularization" ], "demo": "https://dictionaryofml.org/terms/explainability.py", "notebook": "https://dictionaryofml.org/terms/explainability.ipynb" }, { "key": "explanation", "name": "explanation", "url": "https://dictionaryofml.org/terms/explanation.html", "pdf": "https://dictionaryofml.org/terms/explanation.pdf", "abstract": "An explanation accompanies a prediction delivered by a machine learning method and says what about the data point drove it. It can be text, a score per feature, a simple hypothesis that approximates the learned one near the data point, or a heat map over the regions of an image. Two requirements pull against each other: an explanation must be faithful, reflecting the computation the learned hypothesis carries out, and effective, letting the user it is written for anticipate the predictions it accompanies. An explanation that agreed with the learned hypothesis everywhere would be that hypothesis again, so a simpler one may have to give up faithfulness somewhere.", "description": "One approach to enhance the transparency of a machine learning method for its human user is to provide an explanation alongside the predictions delivered by the method. Explanations can take different forms. For instance, they may consist of human-readable text or quantitative indicators, such as feature importance scores for the individual features of a given data point~. One construction of such scores is SHapley Additive exPlanations, which assigns each feature its Shapley value . Fig.\\ \\ref{fig_explanation_dict} illustrates two types of explanations. The first is a local linear approximation g(x) of a nonlinear learned hypothesis h(x) around a specific feature vector x', as used in the method local interpretable model-agnostic explanations . The second form of explanation depicted in the figure is a sparse set of predictions h(x(1)), h(x(2)), h(x(3)) at selected feature vectors, offering concrete reference points for the user. For a differentiable h, the local linear approximation is the one determined by the gradient h(x'), and the reference points are values of the same function. A widely used form of explanation is a heat map: an intensity map that scores each region of an image by how much it drove the prediction, drawn over the image itself . For a convolutional neural network, those scores are read off the network's own activations, which is what a class activation map does. Explanations differ in scope. The local linear approximation of local interpretable model-agnostic explanations concerns a single prediction; a global explanation, such as a decision tree fitted to the predictions of the learned hypothesis, describes its behavior across the whole feature space . A counterfactual answers a different question: not which features drove the prediction, but what change of the feature vector would have changed it. Whatever form it takes, an explanation carries two requirements: it must be (i) understandable and (ii) faithful. Understandable means that the user can work with the explanation on their own. When the explanation is a local linear approximation g, this is concrete: the user must be able to evaluate g at a feature vector of their choosing and read off what it delivers for the data points of a test set. An explanation the user cannot evaluate leaves the predictions as unanticipated as no explanation at all, which is what explainability measures. Faithful means that the explanation reflects the computation the learned hypothesis actually carries out, rather than one that is easier to present. A map over the pixels of an image is faithful when manipulating a few of the pixels it highlights changes the prediction while changing as many dark ones does not (see explainable artificial intelligence). The two requirements are separate, and neither follows from the other. An unfaithful explanation can still be understandable and still let the user anticipate well, whenever it tracks the prediction without carrying anything about the computation: maps have been found that are independent of both the model parameters and the training set, yet look like the ones that are not . Such an explanation predicts by correlation, so it fails when the correlation does, and it cannot support a user who acts on the features it highlights. The two requirements also pull against each other. An explanation that agreed with the learned hypothesis everywhere would be that hypothesis again, so a simpler explanation may have to give up faithfulness somewhere; where the hypothesis is itself simple enough to follow, it serves as its own explanation and none has to be constructed (see interpretable machine learning). Fig.~\\ref{fig_explanation_radar_dict} shows both requirements at work on a prediction made from weather radar. The features are the hourly precipitation over a 48\\,km box around Krems an der Donau, and the prediction answers whether it will rain at Krems two hours later. A convolutional neural network fitted to these images is explained by a class activation map, which scores each cell of the image by how much it contributed to the prediction . For the hour drawn, the network is certain of rain, and the map puts its weight on the band of precipitation south-east of the town rather than on the rain already overhead. Setting the precipitation to zero in the 160 cells the map scores highest moves the predicted score by 3.32, against 0.64 for the 160 it scores lowest, and the map-guided cells move it further on 58 of the 72 held-out hours. See also: machine learning, prediction, feature, data point, classification, explainability, explainable artificial intelligence, interpretable machine learning, local interpretable model-agnostic explanations, SHapley Additive exPlanations, counterfactual, class activation map.", "synonyms": [], "see_also": [], "demo": "https://dictionaryofml.org/terms/explanation.py", "notebook": "https://dictionaryofml.org/terms/explanation.ipynb" }, { "key": "feature", "name": "feature", "url": "https://dictionaryofml.org/terms/feature.html", "pdf": "https://dictionaryofml.org/terms/feature.pdf", "abstract": "A feature of a data point is one of its attributes that can be measured or computed easily, without human supervision. The features of a data point are collected into a feature vector, which a hypothesis map reads to predict the label of the data point. The red-green-blue (RGB) pixel intensities of a digital image or the signal values of an audio recording are typical features. Which attributes of a data point serve as features, rather than as labels, is a design choice within a machine learning application.", "description": "A feature of a data point z is one of its attributes that is measurable, or can be computed easily, without the need for human supervision (; ; ). The features of a data point are assembled into a feature vector x = Φ(z) by a feature transformation, a function Φ: Z → X, viewing the data point itself as its raw features from which the computed features are obtained. For example, if a data point is a digital image, then the red-green-blue (RGB) intensities of its pixels can serve as features. Another example is shown in Fig.~\\ref{fig:audio_features_dict}, where the signal values of a finite-duration audio signal are used as its features. New features can be constructed by transforming existing features. Such a transformation can be an arithmetic computation applied to the existing features. For example, if a data point has a numeric feature x ∈ R, the quantities x2, (-x), and 3x can also be used as features of the data point. Instead of its raw signal values, the audio signal in Fig.~\\ref{fig:audio_features_dict} can be described by the magnitudes of its discrete-time Fourier transform (DTFT) (). These features capture the frequency content of the signal and are unchanged by time shifts, which is useful for a stationary signal, one whose statistical properties do not change over time. Another example is the activation of a neuron within an artificial neural network: it is a new feature derived from the input features by a sequence of basic computations encoded by the artificial neural network. Using activations as features underlies a second, narrower use of the word \\emph{feature} in mechanistic interpretability: a feature is a human-interpretable concept encoded inside a trained artificial neural network, such as a curve detector or the presence of a wheel. The activations of a subset of d neurons (for example, a whole layer, or the neurons that an analyst selects for study) form a vector in a Euclidean space Rd. Whereas the broader sense above treats a single neuron activation as one feature, here a feature is obtained as a function of this whole activation vector. One such function is the projection of the activation vector onto a one-dimensional subspace spanned by a unit vector (or direction) (; ). This unit vector can be obtained from a hyperplane separating data points that express the concept from those that do not () (see Fig.~\\ref{fig_feature_activation_dict}). Another construction of an activation-based feature is based on regions instead of directions in the activation space (). In either case, constructing the function that maps the activation vector to the feature requires a set of data points labeled as representing the concept or not. Whether a given attribute is treated as a feature or a label is not inherent to the data point. It is more a design choice that depends on the machine learning application and the resources available for determining each attribute. An attribute should be used as a feature only if its value can be determined with sufficient accuracy. The accuracy of features also matters for regulation. The EU AI Act requires the training data of high-risk ML systems to be relevant, representative, and as error-free as possible (). When features are personal data, the general data protection regulation adds the data minimization principle and the accuracy principle (). Conversely, deliberately perturbing the features can be useful: adding small perturbations to the features of data points during training is a form of data augmentation that acts as regularization. For example, the ridge regression penalty equals the average loss under Gaussian feature perturbations.", "synonyms": [ "covariate", "explanatory variable", "independent variable", "input", "predictor", "regressor" ], "see_also": [ "datapoint", "label", "featurevec", "featurespace", "dataset", "mechanisticinterpretability", "cav", "euaiact", "dataaug", "regularization", "ridgeregression", "fourier" ], "demo": "https://dictionaryofml.org/terms/feature.py", "notebook": "https://dictionaryofml.org/terms/feature.ipynb" }, { "key": "featureimportance", "name": "feature importance", "url": "https://dictionaryofml.org/terms/featureimportance.html", "pdf": "https://dictionaryofml.org/terms/featureimportance.pdf", "abstract": "A feature importance score quantifies, for each feature, how much it contributes to the predictions of a learned hypothesis or to their quality. The permutation importance of a feature is the increase of the average loss on a dataset when the values of that feature are randomly permuted across the data points. A decision tree additionally offers an internal score by summing the improvement of its splitting criterion over the splits that use the feature, and averaged SHapley Additive exPlanations contributions yield another global score. Importance scores are properties of the learned hypothesis and the scoring dataset: correlated features share their credit, and an implausibly large importance is a symptom of data leakage.", "description": "A random forest serves as a classifier that predicts whether a day will be wet from a weather station's morning measurements. Whether its predictions deserve trust depends on which of the measurements they rest on — and on whether a column that should carry no information receives a large share of the credit. A feature importance score quantifies, for each feature, how much that feature contributes to the predictions of a learned hypothesis h, or to their quality. The permutation importance of the j-th feature is the increase of the average loss on a dataset when the values of that feature are randomly permuted across the data points, which breaks their association with the label while leaving the feature's marginal distribution intact (; ). A feature whose permutation leaves the average loss unchanged played no measurable role for h on that dataset (Fig.~\\ref{fig_featureimportance_dict}). Permutation importance treats h as a black box; a decision tree additionally offers an internal score, obtained by summing the improvement of its splitting criterion over all splits that use the feature . SHapley Additive exPlanations assigns each feature a contribution to one single prediction ; averaging the magnitudes of these contributions over a dataset yields another global importance score. Importance scores are properties of the learned h and the dataset used for scoring, not of the world: two strongly correlated features share their credit, so permuting either one alone understates how much the pair carries . The scores also serve as an audit: a feature that could not plausibly matter but receives a large importance points to data leakage, and a feature whose removal leaves the importance ranking and the average loss unchanged was not needed. As a global summary of which features matter, feature importance is one of the tools of explainability for otherwise opaque methods such as a random forest.", "synonyms": [ "variable importance" ], "see_also": [ "feature", "randomforest", "decisiontree", "shap", "explainability", "dataleakage", "loss" ] }, { "key": "gaussrv", "name": "Gaussian random variable", "url": "https://dictionaryofml.org/terms/gaussrv.html", "pdf": "https://dictionaryofml.org/terms/gaussrv.pdf", "abstract": "A standard Gaussian random variable (Gaussian RV) is a real-valued random variable with probability density function proportional to (-η2/2). A general Gaussian random variable with mean μ and variance σ2 is obtained by scaling and shifting a standard Gaussian random variable. A Gaussian random vector is an affine transformation of independent standard Gaussian RVs. Gaussian RVs arise naturally as limits of averages of many independent RVs through the central limit theorem. Among all random vectors with a given covariance matrix, the Gaussian random vector maximizes the differential entropy.", "description": "A standard Gaussian random variable is a real-valued random variable x with a probability density function (; ; ) \\begin{equation} \\nonumber \\pdf{\\feature}{\\eta} = \\frac{1}{\\sqrt{2\\pi}} \\exp\\,(-\\eta^2/2). \\end{equation} Given a standard Gaussian random variable x, a general Gaussian random variable x' with mean μ and variance σ2 can be constructed via x' := σ x + μ. The probability distribution of a Gaussian random variable is referred to as the normal distribution, denoted by μσ2. Two methods generate Gaussian RVs. For exact generation, let U be uniformly distributed on (0,1) and let Φ be the cumulative distribution function of a standard Gaussian random variable. Then x' := μ + σΦ-1(U) has distribution μσ2. For approximate generation, if x(i) are independent RVs with common mean 0 and finite, nonzero variance τ2, then \\[ \\frac{1}{\\tau\\sqrt{n}}\\sum_{i=1}^{n}\\feature^{(i)} \\] converges in distribution to a standard Gaussian random variable as n increases. Thus, the central limit theorem makes a Gaussian approximation appropriate for averages of many independent contributions, although a finite sum need not be Gaussian . The two parameters act separately on the probability density function, as shown in Fig.~\\ref{fig_gaussrv_pdf}. Adding μ shifts the probability density function along the horizontal axis without changing its shape. For σ>0, multiplying the random variable by σ stretches the probability density function horizontally by a factor of σ and lowers its peak by the same factor, so its area remains 1. A Gaussian random vector x ∈ Rd with covariance matrix C and mean μ can be constructed as (; ; ) \\[ \\featurevec \\defeq \\mA \\vz + \\meanvecgeneric, \\] where z := ( z1, …, zd ) is a vector of independent and identically distributed standard Gaussian RVs, and A ∈ Rd × d satisfies A A = C. The probability distribution of a Gaussian random vector is referred to as the multivariate normal distribution, denoted by μC. A Gaussian random vector need not have independent components; its components are independent exactly when its covariance matrix is diagonal. A Gaussian random vector x=(x1, …, xd) can be interpreted as a stochastic process indexed by the finite set 1, …, d. More generally, a Gaussian process is a stochastic process over an index set I whose restriction to every finite subset of I is a Gaussian random vector . Gaussian RVs are widely used probabilistic models in the statistical analysis of machine learning methods, including regression with additive Gaussian measurement noise. For example, a regression method can use y = wx + ε with ε ~ 0σ2. Maximizing the resulting likelihood is equivalent to minimizing squared loss, connecting the Gaussian noise assumption to empirical risk minimization. Among all random vectors with a given covariance matrix C, the random vector x ~ μC maximizes differential entropy . It is therefore the least-committal choice of probability distribution when only the first two moments are fixed.", "synonyms": [], "see_also": [ "mvndist", "GaussProc", "probmodel", "clt", "diffentropy" ], "demo": "https://dictionaryofml.org/terms/gaussrv.py", "notebook": "https://dictionaryofml.org/terms/gaussrv.ipynb" }, { "key": "gd", "name": "gradient descent", "url": "https://dictionaryofml.org/terms/gd.html", "pdf": "https://dictionaryofml.org/terms/gd.pdf", "abstract": "Gradient descent (GD) is an iterative algorithm for minimizing a differentiable function f: Rd → R. Each iteration updates the current estimate by a step along the negative gradient, scaled by a step size η. Such a gradient step can be interpreted as the application of an operator that is parameterized by the step size and the underlying function. GD and its variants are widely used to train parametric models in deep learning.", "description": "Many machine learning applications amount to solving an optimization problem: an objective function f: Rd → R measures the quality of the vector w ∈ Rd of model parameters, which is the optimization variable. When training a deep net for image recognition, for example, f(w) is the average loss incurred by the network on a training set of labeled images. In many important cases, the objective function is differentiable: at every point w there exists a gradient f(w) ∈ Rd, the vector of partial derivatives of f at w, which determines a local linear approximation of f (see differentiable). GD uses this local linear approximation to iteratively improve the current choice of the model parameters: starting from an initialization w(0), GD generates a sequence of estimates w(0), w(1), w(2), … that ideally converge to a minimum of f. GD and its variants are the standard solvers for model training in deep learning . At each iteration t, GD refines the current estimate w(t) by stepping in the direction of steepest descent of the local linear approximation to f at w(t). This direction is the negative gradient f(w(t)), giving the update \\begin{equation} \\weights^{(\\iteridx+1)} = \\weights^{(\\iteridx)} - \\lrate \\nabla f(\\weights^{(\\iteridx)}) \\text{,} \\end{equation} where η > 0 is a step size. For a sufficiently small η, each step decreases the function value, f(w(t+1)) f(w(t)), with strict decrease whenever f(w(t)) 0 . Fig.~\\ref{fig_basic_GD_step_dict} illustrates a single GD step. With a constant step size η, held fixed across iterations, a single step is described by the GD step operator η(w) = w - η f(w), so GD is the fixed-point iteration w(t+1) = η(w(t)). Its fixed points, where η(w) = w, are exactly the stationary points f(w) = 0. For convex f, these are the minima according to the zero-gradient condition . Consider shrinking the step size η in the GD step \\eqref{equ_def_GD_step_dict} toward zero. This results in iterates w(t) becoming approximately the values of a continuous-time trajectory w(τ). This trajectory satisfies the ordinary differential equation w(τ) = -f(w(τ)), which is known as gradient flow. In particular, GD is the explicit Euler discretization of this gradient flow with step η . Indeed, replacing the time derivative w(τ) by the difference quotient (w(t+1) - w(t))/η recovers the update \\eqref{equ_def_GD_step_dict}. Fig.~\\ref{fig_gd_flow_dict} illustrates how, for a small step size, the GD iterates closely follow the gradient flow trajectory. Returning to the fixed-point iteration w(t+1) = η(w(t)), its convergence is governed by the properties of the GD step operator η. Assume f is convex and its gradient is L-Lipschitz continuity continuous, i.e., \\norm{\\nabla f(\\weights) - \\nabla f(\\weights')} \\le L\\,\\norm{\\weights - \\weights'} \\quad \\text{for all } \\weights, \\weights' \\in \\reals^{\\nrfeatures} \\text{.} For 0 < η 1/L, the operator η is then a non-expansive operator with respect to the Euclidean norm ·: it does not increase distances between any two points w, w', \\norm{\\gdstep{\\lrate}(\\weights) - \\gdstep{\\lrate}(\\weights')} \\le \\norm{\\weights - \\weights'} \\text{.} Non-expansiveness alone does not guarantee a unique fixed point . Assume, moreover, that f is μ-strongly convex, i.e., the function f(w) - (μ/2) w2 is still convex, so the curvature of f is at least μ > 0 in every direction. Then η is a contractive operator with respect to the same norm, \\norm{\\gdstep{\\lrate}(\\weights) - \\gdstep{\\lrate}(\\weights')} \\le (1 - \\mu\\lrate)\\,\\norm{\\weights - \\weights'} \\text{.} In this case, Banach's fixed-point theorem gives a unique fixed point w (the minimizer of f) to which GD converges at a geometric rate . For linear regression, these properties are explicit: the gradient of the average squared error loss is an affine function of w, and the contraction factor of η is governed by the eigenvalues of the matrix XX. Here, X denotes the feature matrix, whose rows are the feature vectors of the data points in the training set (see linear regression). How fast GD converges depends on the curvature of f, as quantified by the constants L and μ above. Let w be a minimizer and f = f(w). For convex f with L-Lipschitz continuity gradient and the constant step size η = 1/L (; ), f\\big(\\weights^{(\\nriter)}\\big) - f^{\\star} \\le \\frac{L\\,\\norm{\\weights^{(0)} - \\widehat{\\weights}}^{2}}{2\\,\\nriter} \\text{,} so the suboptimality after T steps falls in proportion to 1/T: halving the suboptimality requires doubling the number of iterations. If f is in addition μ-strongly convex, the bound improves to the geometric rate f\\big(\\weights^{(\\nriter)}\\big) - f^{\\star} \\le \\left(1 - \\tfrac{\\mu}{L}\\right)^{\\nriter} \\big(f(\\weights^{(0)}) - f^{\\star}\\big) \\text{,} consistent with the contractive operator factor 1 - μη above (; ). A geometric rate means that the suboptimality shrinks by at least the constant factor 1 - μ/L in every single iteration. Thus, halving the suboptimality now costs a fixed number of iterations, however small the current suboptimality already is. Momentum methods, discussed below, sharpen both bounds. In two common settings that motivate variants of GD, the assumptions behind these guarantees fail or the gradient becomes too costly. First, when f is non-smooth, its gradient may fail to exist and GD is replaced by subgradient descent, which steps along a subgradient in place of the gradient. A concrete case is the least absolute shrinkage and selection operator objective, whose penalty term α w1 is non-smooth. Second, in empirical risk minimization-based methods, f is an average of m loss functions, one for each data point, so evaluating the full gradient sums m gradients and its cost grows in proportion to the training set size m. GD is thus replaced by stochastic gradient descent, which uses a cheap gradient estimate from a random subset of the training set at each step. Momentum methods provide another generalization of plain GD: each update combines gradients from several past steps rather than the current one alone. The name stems from a physical analogy: the iterates trace a particle whose momentum accumulates the force exerted by the negative gradient of the objective function. One prototypical example of a momentum method is Polyak's heavy-ball method. It adds a fraction β ∈ [0, 1) of the previous increment, \\weights^{(\\iteridx+1)} = \\weights^{(\\iteridx)} - \\lrate \\nabla f(\\weights^{(\\iteridx)}) + \\beta\\big(\\weights^{(\\iteridx)} - \\weights^{(\\iteridx-1)}\\big) \\text{, for } \\iteridx=1,\\ldots. For a strongly convex quadratic f, this iteration converges faster than plain GD . Fig.~\\ref{fig_GD_momentum_dict} compares plain GD with the heavy-ball method on a strongly convex quadratic function. A related momentum algorithm is Nesterov's accelerated variant of GD, which improves the 1/T bound above for convex f with L-Lipschitz continuity gradient to f\\big(\\weights^{(\\nriter)}\\big) - f^{\\star} \\le \\frac{2\\,L\\,\\norm{\\weights^{(0)} - \\widehat{\\weights}}^{2}} {(\\nriter+1)^{2}} \\text{.} In contrast to the bound for plain gradient descent, which falls in proportion to 1/T, this bound falls in proportion to 1/T2 (; ). For μ-strongly convex f, acceleration improves the contraction factor from 1 - μ/L to 1 - μ/L (; ). See also: gradient step, gradient, step size, stochastic gradient descent, subgradient descent, subgradient, gradient flow, convex, strongly convex, minimum, differentiable, empirical risk minimization, fixed point, Banach's fixed-point theorem.", "synonyms": [], "see_also": [], "demo": "https://dictionaryofml.org/terms/gd.py", "notebook": "https://dictionaryofml.org/terms/gd.ipynb" }, { "key": "generalization", "name": "generalization", "url": "https://dictionaryofml.org/terms/generalization.html", "pdf": "https://dictionaryofml.org/terms/generalization.pdf", "abstract": "Generalization is the ability of a machine learning method to make accurate predictions on data points that have not been used during training. Many machine learning methods implement training via empirical risk minimization, i.e., learning a hypothesis h that minimizes the empirical risk on the training set D(train). The learned h is then used to compute predictions on new data points outside D(train). A probabilistic model for data generation quantifies the prediction errors beyond D(train) via the risk of h, i.e., the expected loss incurred on a new randomly chosen data point. The difference between the risk and the empirical risk is the generalization gap. The stability of the machine learning method bounds the expected generalization gap without reference to the size of the hypothesis space. A complementary deterministic view characterizes generalization via the robustness of h to small perturbations of the features of a data point.", "description": "A weather forecaster trained on past data is useful only if its predictions are accurate on future unseen data points. Generalization refers to the ability of a learned hypothesis h to make similarly accurate predictions on data points that have not been used during training. It is the central goal of machine learning: learning patterns in the features and labels of data points that extend beyond the training set. The canonical learning technique is empirical risk minimization: the machine learning system obtains h ∈ H by minimizing the empirical risk on the training set D(train). A low empirical risk on D(train) does not guarantee accurate predictions on data points outside D(train). online learning and Bayesian inference face the same generalization challenge. Studying generalization mathematically requires formalizing the notion of \\emph{unseen} data points. A widely used approach is to assume a probabilistic model for the generation of data points, such as the independent and identically distributed assumption. Here, data points are interpreted as independent RVs with an identical, fixed but unknown, probability distribution P. Fig.~\\ref{fig_generalization_dict} sketches this setup: the shaded ellipse depicts the region of typical data points under P, and the two large filled circles are the data points of the training set that empirical risk minimization fits with the learned hypothesis h. The risk of h is the expected loss: \\[ \\risk{\\learnthypothesis} = \\expect \\left\\{ \\lossfunc{\\datapoint}{\\learnthypothesis} \\right\\}. \\] The difference h - hD(train) is the generalization gap. For any fixed hypothesis h ∈ H, h is a deterministic number that depends only on h and P, whereas hD(train) is an random variable that depends on D(train), which, in turn, is a realization of an independent and identically distributed sequence of RVs z(1), …, z(m) ~ P. Bounding the generalization gap therefore means controlling the probability of the (undesirable) event \\genericevent^{(\\hypothesis)} \\defeq \\left\\{ \\trainset : \\big| \\emprisk{\\hypothesis}{\\trainset} - \\risk{\\hypothesis} \\big| > \\varepsilon \\right\\}, where ε > 0 is a user-specified tolerance for the generalization gap. Each h ∈ H defines a different event A(h) (see Fig.~\\ref{fig_generalization_events_dict}), and generalization requires low probability of A(h) uniformly over H (; ). Controlling every event A(h) uniformly over H is not the only probabilistic route. If the machine learning method is stable — replacing a single data point of the training set changes the loss of the learned hypothesis only slightly on average — then the expected generalization gap is bounded by that change, regardless of the size of the hypothesis space (see stability; ). probability theory is not the only route to generalization. A complementary deterministic approach uses robustness: a good hypothesis h should not change its prediction h(x) much if the features x of a data point z are slightly perturbed. For example, an object detection system trained on smartphone photos should still detect the object even if a single pixel is changed , and it should deliver the same result if the object in the image is rotated . The arrows in Fig.~\\ref{fig_generalization_dict} indicate small perturbations of each data point in the training set of an machine learning method.", "synonyms": [], "see_also": [ "erm", "emprisk", "risk", "gengap", "iidasspt", "trainset", "valerr", "testset", "overfitting", "validation", "regularization", "hypothesis", "hypospace", "onlinelearning", "bayesianinference", "stability" ], "demo": "https://dictionaryofml.org/terms/generalization.py", "notebook": "https://dictionaryofml.org/terms/generalization.ipynb" }, { "key": "gmm", "name": "Gaussian mixture model", "url": "https://dictionaryofml.org/terms/gmm.html", "pdf": "https://dictionaryofml.org/terms/gmm.pdf", "abstract": "A Gaussian mixture model (GMM) is a probabilistic model for data points with numeric feature vectors. Each data point is generated by first drawing a latent cluster index according to cluster probabilities and then drawing the feature vector from the multivariate normal distribution of that cluster; the marginal distribution is a weighted sum of multivariate normal distributions. The model parameters — cluster probabilities, means, and covariance matrices — are learned by maximum likelihood, typically via the expectation–maximization algorithm. A fitted GMM delivers soft clustering: the posterior distribution of the latent index grades the membership of each data point in every cluster.", "description": "The nightly minimum temperatures recorded over one year at a weather station do not scatter around a single typical value: cold-season and warm-season nights form two distinct regimes. A GMM is a probabilistic model that captures such multi-regime data: it models the generation of data points with numeric feature vectors x ∈ Rd (; ). It assumes that each data point is generated by first drawing a latent cluster index I ∈ 1, …, k according to cluster probabilities \\prob{I=\\clusteridx} = p_{\\clusteridx}, \\qquad \\sum_{\\clusteridx=1}^{\\nrcluster} p_{\\clusteridx}=1\\text{.} Conditioned on I=c, the feature vector x is drawn from a multivariate normal distribution P(c)= cc. The resulting marginal distribution of x is therefore a weighted sum of multivariate normal distributions (Fig.~\\ref{fig_gmm_dict}), i.e., \\probdist =\\sum_{\\clusteridx=1}^{\\nrcluster} p_{\\clusteridx} \\mvnormal{\\meanvec{\\clusteridx}}{\\covmtx{\\clusteridx}}\\text{.} % \\begin{figure}[H] % \\begin{tikzpicture}[scale=0.4] % \\draw [thick] \\boundellipse{0,0}{10}{5} node[right] {1}; % \\fill (0,0) circle (2pt) ; % \\node [right] at (0,5.5) {1} ; % \\draw [thick] \\boundellipse{11,1}{-2}{4} node[right] {2}; % \\fill (11,1) circle (2pt) ; % \\node [right] at (11,5.5) {2} ; % \\draw [thick] \\boundellipse{-9,4}{2}{3} % node[left,xshift=3mm,yshift=3mm] {3}; % \\fill (-9,4) circle (2pt) ; % \\node [right] at (-9,8) {3} ; % \\end{tikzpicture} % \\caption{Illustration of a GMM with three components.} %Each component is a multivariate normal distributions % cc with mean % vector c and covariance matrix c. % The overall probability distribution is a convex combination of these % components, weighted by the cluster probabilities pc.", "synonyms": [], "see_also": [] }, { "key": "gradient", "name": "gradient", "url": "https://dictionaryofml.org/terms/gradient.html", "pdf": "https://dictionaryofml.org/terms/gradient.pdf", "abstract": "The gradient of a real-valued function is a vector that determines the local linear approximation of the function. The entries of the gradient are the partial derivatives of the function. Their existence does not by itself make the function differentiable: the vector they form need not give a local linear approximation. For a convex function, it does. Geometrically, a nonzero gradient is orthogonal to the level sets and points in the direction of steepest ascent. In machine learning, gradients of the empirical risk minimization objective function drive gradient descent methods and are computed for deep networks by backpropagation.", "description": "Training a machine learning model adjusts its model parameters w to reduce a loss f(w), which requires knowing how f changes as w moves. The gradient of a real-valued function f: Rd → R: w f(w) answers this by determining a local linear approximation of f. Formally, the gradient of f at a point w' ∈ Rd is a vector g ∈ Rd such that \\begin{equation} \\lim_{\\weights \\rightarrow \\weights'} \\frac{f(\\weights) - \\big( f(\\weights') + \\vg^{\\top} (\\weights - \\weights') \\big)}{\\normgeneric{\\weights - \\weights'}{2}} = 0 \\text{.} \\end{equation} If such a vector exists, it is unique, the function f is differentiable at w', and the vector is denoted by f(w') or f(w) |w' . The function f(w') + (f(w')) (w - w') is the local linear approximation of f at w' (see Fig.~\\ref{fig_gradient_tangent_dict}). The entries of the gradient are the partial derivatives of f, f(w) = ( f/w1, …, f/wd ); the partial derivative f/wj is the rate of change of f when only the j-th entry of w varies. In general, the existence of all partial derivatives of f at a point does not guarantee that the gradient exists there . Additional assumptions on f restore the implication; convexity is one sufficient condition. If a convex function f: Rd → R has partial derivatives f / wj, for j = 1, …, d, at a point w', then f is differentiable at w', and the gradient f(w') is the vector of these partial derivatives . The definition carries over to a real-valued function f: H → R on a real Hilbert space H (a Hilbert space over R; the complex case needs a separate treatment of the sesquilinear inner product): the inner product gw - w' of H replaces the term g (w - w') in \\eqref{eq_gradient_def}, and the norm of H replaces the Euclidean norm w - w'2 . The gradient has a geometric interpretation. At every point where f is differentiable and f 0, the gradient is orthogonal to the level set of f through that point --- the set w : f(w) = c of points where f takes a common value c --- and it points in the direction of steepest ascent of f. The negative gradient -f(w') points in the direction of steepest descent, and gradient descent repeatedly steps along it (see Fig.~\\ref{fig_gradient_dict}; ). At a local minimum of a differentiable function f: Rd → R, there cannot be any direction of descent and consequently the gradient must vanish (see zero-gradient condition). Gradients are instrumental for the training of machine learning models. They guide the update of model parameters to minimize the incurred loss on a training set D(train) = (x(r), y(r)) r=1m, which is the goal of empirical risk minimization. As a case in point, linear regression minimizes the objective function defined by \\[ f(\\weights) \\defeq \\frac{1}{\\samplesize} \\sum_{\\sampleidx=1}^{\\samplesize} \\big( \\truelabel^{(\\sampleidx)} - \\weights^{\\top} \\featurevec^{(\\sampleidx)} \\big)^{2} \\text{.} \\] This function is convex and differentiable, with gradient \\[ \\nabla f(\\weights) = - \\frac{2}{\\samplesize} \\sum_{\\sampleidx=1}^{\\samplesize} \\big( \\truelabel^{(\\sampleidx)} - \\weights^{\\top} \\featurevec^{(\\sampleidx)} \\big) \\featurevec^{(\\sampleidx)} \\text{.} \\] For a deep artificial neural network whose activation functions, loss, and every other operation it is composed of are differentiable, the empirical risk minimization objective function is also differentiable. The gradient of this objective function can be computed by backpropagation (; ).", "synonyms": [], "see_also": [ "function", "vector", "differentiable", "partialderivative", "gd", "zerogradientcondition", "convex", "hilbertspace" ], "demo": "https://dictionaryofml.org/terms/gradient.py", "notebook": "https://dictionaryofml.org/terms/gradient.ipynb" }, { "key": "hilbertspace", "name": "Hilbert space", "url": "https://dictionaryofml.org/terms/hilbertspace.html", "pdf": "https://dictionaryofml.org/terms/hilbertspace.pdf", "abstract": "A Hilbert space is a complete inner product space: every Cauchy sequence of its elements has a limit in the space. Three examples in machine learning are: a finite-dimensional Euclidean space, a space of finite-variance RVs on a common probability space, and a reproducing kernel Hilbert space. linear regression uses a Euclidean space as its feature space and to parameterize its hypothesis space. Optimal estimation amounts to a projection in a Hilbert space consisting of RVs. A kernel method uses an reproducing kernel Hilbert space as its transformed feature space and as its hypothesis space.", "description": "Consider a machine learning method whose hypothesis space is a metric space. An iterative training method produces the sequence h(1), h(2), … of hypotheses. For the sequence to converge to an optimal hypothesis h, the hypotheses must form a Cauchy sequence. Being a Cauchy sequence is not sufficient on its own. If the underlying metric space is complete, then every Cauchy sequence in that space converges to a point in the space. A prime example of a complete metric space is a Euclidean space Rd of finite dimension d. A Hilbert space is a generalization of a Euclidean space to possibly infinite dimensions. In particular, a Hilbert space (H, ··) is an inner product space that is complete, i.e., in which every Cauchy sequence of its elements has a limit that again belongs to H (see Fig.~\\ref{fig_hilbertspace_dict}). The elements of H are called vectors, whether they are arrays of numbers, RVs, or functions: what makes them vectors is that they can be added and scaled (see vector space). The inner product induces a norm uH := uu and, in turn, a metric uv := u - vH (see inner product). A sequence u(1), u(2), … of vectors of H is a Cauchy sequence if its vectors eventually lie within any prescribed distance of each other: for every distance ε > 0, however small, there is an index N such that u(r) - u(r')H < ε for all r, r' ≥ N . Three examples of Hilbert spaces are used throughout machine learning: a Euclidean space Rd as feature space, a space of RVs with finite variance on a common probability space, and a reproducing kernel Hilbert space of functions. The first of them carries the standard inner product uv = u v, and its completeness follows from that of the real numbers . linear regression uses the Euclidean space Rd in two roles. The feature vectors x ∈ Rd of the data points it is applied to are vectors of this space, and so are the weights w ∈ Rd that parameterize its hypothesis space h(x) = wx : w ∈ Rd. gradient descent searching for those weights produces a sequence w(1), w(2), … of model parameters, the sequence which is depicted in Fig.~\\ref{fig_hilbertspace_dict}. These are vectors in the second role, so the condition of the opening paragraph is met here: Rd is complete, and a Cauchy sequence of model parameters converges to model parameters. The second example is the set of all RVs x with finite variance, defined on a common probability space . Here, two RVs are identified whenever the expectation of their squared difference is zero, E (x - x')2 = 0, i.e., whenever x = x' with probability one. With this identification, the expectation xx' := E x x' is an inner product, and the resulting space is complete . In the Hilbert space of finite-variance RVs on a common probability space, optimal linear estimation is an orthogonal projection (see Fig.~\\ref{fig_hilbertspace_projection_dict}). A linear estimator y of an random variable y from an observed random variable x is an element of the subspace a x : a ∈ R spanned by x. The corresponding estimation error y - y is measured by the induced norm y - yy - y. The smallest error is obtained by the linear estimator whose error is orthogonal to x , \\begin{align} 0 & = \\innerprod{\\truelabel - \\predictedlabel}{\\feature} \\nonumber & = \\expect \\big\\{ \\big(\\truelabel - \\predictedlabel \\big) \\feature \\big\\} \\text{.} \\nonumber \\end{align} When the RVs y,x are zero-mean, this orthogonality means that the error y - y is uncorrelated with x. Optimal nonlinear estimation can also be represented as an orthogonal projection, but on a larger subspace. Projecting y onto the RVs that are functions of x, rather than onto the multiples a x alone, gives the conditional expectation E y x: it has the smallest squared error among all estimators computed from x . The third example is an reproducing kernel Hilbert space. It is a Hilbert space H of functions h: X → R whose inner product reproduces point evaluations. In particular, each reproducing kernel Hilbert space is associated with a kernel ·· such that x· ∈ H for every x ∈ X and \\[ \\hypothesis(\\featurevec) = \\innerprod{\\hypothesis}{\\kernelmap{\\featurevec}{\\cdot}} \\quad \\text{for every } \\hypothesis \\in \\hilbertspace \\text{.} \\] Thus, each vector h ∈ H is itself a hypothesis map X → R, evaluated by taking an inner product with x·. This construction of a hypothesis map generalizes the linear model, which uses the standard inner product of the Euclidean space. The map x x· is a feature transformation, carrying the feature vector of a data point into H. A kernel method uses an reproducing kernel Hilbert space in two roles. It is the transformed feature space, in which a data point is represented by x·, and it is the hypothesis space from which a hypothesis is learned (; ). These are the two roles of Rd in linear regression, with one difference: there a vector parameterizes a hypothesis, here a vector is one. Any two Hilbert spaces whose orthonormal bases have the same number of elements are copies of one another: mapping one basis onto the other extends to a linear map that preserves inner products and, in turn, distances. Such a linear map is therefore an isometry .", "synonyms": [], "see_also": [ "innerproduct", "vectorspace", "norm", "cauchysequence", "euclidspace", "rv", "variance", "expectation", "conditionalexpect", "rkhs", "kernelmethod", "orthogonalprojection", "isometry" ], "demo": "https://dictionaryofml.org/terms/hilbertspace.py", "notebook": "https://dictionaryofml.org/terms/hilbertspace.ipynb" }, { "key": "hypospace", "name": "hypothesis space", "url": "https://dictionaryofml.org/terms/hypospace.html", "pdf": "https://dictionaryofml.org/terms/hypospace.pdf", "abstract": "A hypothesis space H is a set of hypothesis maps h: X → Y from a feature space into a label space. Every machine learning method uses an underlying hypothesis space, which is a subset of the set YX of all possible maps from X into Y. Available computational resources limit the size of H, which in turn shapes the method's computational cost and predictive behavior. The hypothesis space typically carries geometric structure: real-valued hypotheses can be compared via the deviation in their predictions on a reference dataset, and a parametric model inherits a geometric structure from the underlying parameter space W.", "description": "A weather service predicting tomorrow's maximum daytime temperature from today's morning temperature must commit to a set of candidate prediction maps: for example, all linear functions of the morning temperature. This set of candidates is the hypothesis space of the method; more generally, every machine learning method commits to a specific hypothesis space H, the set from which it learns a single hypothesis h. Formally, H is a subset of the set YX of all possible maps from the feature space into the label space (see Fig.~\\ref{fig_hypospace_dict}). linear regression and other linear methods use the linear model as H, i.e., the set of all linear maps Rd → R . Another canonical example is %need to check citation again - maybe Sect. 2.3 for the first sentences? and linear regression is mentioned in Sect. 9.2 the set of input--output maps realizable by an artificial neural network of fixed architecture, as the model parameters vary. The choice of H is a central design decision for an machine learning method. From an machine learning engineering perspective, it is guided by two factors. The first is the available computational resources, such as memory, processing time, and communication bandwidth. The second factor is the number of available data points, which limits the maximum size of H that can be effectively trained without overfitting. The size of the underlying H is an important characteristic of an machine learning method. In principle, H could be as large as YX itself. In practice, finite computational resources restrict an machine learning method to a much smaller subset H. The size of H allows the anticipation of generalization before any training is carried out. A small H typically incurs a small generalization gap, while a large H is more prone to overfitting. For a finite H, the effective size is its cardinality. For an infinite H, an effective size can be quantified by the Vapnik–Chervonenkis dimension or the Rademacher complexity. The same H can be used by different machine learning algorithms, such as empirical risk minimization on a fixed training set, online learning on a stream of data points, or Bayesian inference that returns a posterior distribution over H rather than a single hypothesis . The word \\emph{space} (rather than \\emph{set}) reflects the fact that H typically carries some geometric structure. In general, H is not a vector space: the sum of two hypotheses in H need not belong to H. What is more, there are hypothesis spaces that are not equipped with an algebraic structure at all, i.e., there is no meaningful way to add two hypotheses in H or to multiply a hypothesis by a scalar. For real-valued hypotheses, two h, h' ∈ H can be compared by their average squared prediction deviation on a reference dataset D = x(1), …, x(m), \\[ \\frac{1}{\\samplesize} \\sum_{\\sampleidx=1}^{\\samplesize} \\big(\\hypothesis(\\featurevec^{(\\sampleidx)}) - \\hypothesis'(\\featurevec^{(\\sampleidx)})\\big)^{2} \\text{.} \\] Fig.~\\ref{fig_hypospace_compare_dict} illustrates this comparison: the two hypotheses are evaluated on the data points in D and the squared differences in their predictions are averaged. This construction is not meaningful for hypotheses with a finite label space, as used in classification methods. Instead, two such hypotheses can be compared by their disagreement rate on D. A parametric model H = h(w) : w ∈ W inherits a geometric structure from the underlying parameter space W Rd. A norm on Rd induces the candidate distance h(w)h(w') := w - w' on H. When the parameterization w h(w) is injective, distinct weight vectors give distinct hypotheses, and this candidate distance is a genuine metric. The linear model H = h(w)(x) = w x : w ∈ Rd is a canonical example of an injective parameterization: distinct weight vectors w produce distinct linear maps, so w - w' is a metric on H. When the parameterization is not injective, two distinct weight vectors w w' can represent the same hypothesis h(w) = h(w'). The distance w - w' is then strictly positive even though the two hypotheses coincide, so it is only a pseudo-metric. Permuting the hidden units of an artificial neural network, for instance, changes w but leaves h(w) unchanged.", "synonyms": [ "hypothesis class", "model" ], "see_also": [ "hypothesis", "model", "map", "linmodel", "parammodel", "paramspace", "metric", "norm", "erm", "onlinelearning", "bayesianinference", "posteriordist", "vcdim", "rademachercomplexity" ], "demo": "https://dictionaryofml.org/terms/hypospace.py", "notebook": "https://dictionaryofml.org/terms/hypospace.ipynb" }, { "key": "hypothesis", "name": "hypothesis", "url": "https://dictionaryofml.org/terms/hypothesis.html", "pdf": "https://dictionaryofml.org/terms/hypothesis.pdf", "abstract": "A hypothesis is a map h: X → Y from the feature space X to the label space Y. Given a data point with features x, the hypothesis returns a prediction h(x) for the label y. A machine learning method selects a single learned hypothesis h from a restricted hypothesis space H YX. This restriction reflects finite computational resources of a practical machine learning method. As a case in point, an empirical risk minimization-based method must be able to find a hypothesis minimizing the empirical risk on a training set. It does not examine the hypotheses one by one: gradient descent moves through a parametrized H by following a gradient, which is possible only because H carries that structure.", "description": "Predicting tomorrow's maximum daytime temperature from today's morning temperature is a typical machine learning task. A hypothesis is the map that performs this prediction. Formally, a hypothesis is a map (also called a function) h: X → Y from the feature space X to the label space Y . Given a data point with features x, the hypothesis h returns an estimate y = h(x) of the true label y. Fig.~\\ref{fig:hypothesis_dict} illustrates this for an audio-based application where the hypothesis predicts a Freddie-likeness score from an audio recording. An machine learning method aims to find (or learn) a hypothesis h such that y h(x) for any data point with features x and label y. Finite computational resources require restricting the search to a subset of \\[ \\labelspace^{\\featurespace} = \\big\\{ \\hypothesis: \\featurespace \\rightarrow \\labelspace \\big\\} \\text{,} \\] the set of all functions from X to Y. This subset is the hypothesis space H (also called the model) underlying the method. The output of an machine learning algorithm acting on a training set is a single learned hypothesis h ∈ H obtained via empirical risk minimization, i.e., the element of H that minimizes the empirical risk on the training set. Different hypothesis spaces admit different representations of their hypothesis maps. polynomial regression (with a single scalar feature x) represents each h ∈ H as an algebraic expression h(x) = j=0r wj xj, with coefficients wj as the model parameters. A hypothesis space of computer programs represents h as executable source code, e.g., a Python function with a prescribed signature. A decision tree or an artificial neural network represents h as a signal-flow chart whose nodes apply simple operations (e.g., comparisons, weighted sums, nonlinear activations) and whose edges define the order in which these operations are composed. These representations differ in which functions they can express and in how efficiently the empirical risk minimization problem can be solved over the corresponding H. Fig.~\\ref{fig:hypothesis_reps_dict} shows the three representations side by side. Synonyms: predictor.", "synonyms": [], "see_also": [ "map", "function", "prediction", "predictor", "model", "hypospace", "featurevec", "label", "datapoint", "erm", "training", "generalization" ], "demo": "https://dictionaryofml.org/terms/hypothesis.py", "notebook": "https://dictionaryofml.org/terms/hypothesis.ipynb" }, { "key": "innerproduct", "name": "inner product", "url": "https://dictionaryofml.org/terms/innerproduct.html", "pdf": "https://dictionaryofml.org/terms/innerproduct.pdf", "abstract": "Many machine learning applications involve data points whose features form numeric arrays. Examples include the color intensities of image pixels, the amplitudes at regular intervals of a sensor signal, and the embeddings of tokens used by LLMs. These numeric arrays can be naturally represented by vectors in some vector space. Many machine learning methods rely on measuring the similarity between vectors that represent data points. Each neuron of an artificial neural network matches its input, for example the color intensities of an image, against a learned template. The attention unit of an large language model compares the query vector of a token with the key vector of another token, both formed from the embeddings of those tokens. Both comparisons compute an inner product: a number assigned to a pair of vectors in a vector space that measures their similarity.", "description": "data points are often characterized by features that form a numeric array: the color intensities of the pixels of an image, the amplitudes of a sensor signal at regular intervals, or the embedding of a token in a large language model. Such an array can be represented by a vector, i.e., an element of a vector space. Many machine learning methods compare two data points by measuring how similar the vectors that represent them are. One principled measure of this similarity is an inner product. Consider a vector space V over the field of real numbers R. An inner product in V is a function \\[ \\innerprod{\\cdot}{\\cdot}: \\vecspace \\times \\vecspace \\to \\reals \\] that satisfies the following properties for all vectors u, v, w ∈ V and all scalars β ∈ R : \\begin{itemize} \\item Symmetry: uv = vu; \\item Linearity in the first argument: β u + wv = β uv + wv; \\item Positive-definiteness: uu ≥ 0, with equality if and only if u = 0. \\end{itemize} For a vector space over the field of complex numbers C, symmetry is replaced by conjugate symmetry, uv = vu, while linearity in the first argument is retained; the inner product is then conjugate-linear in the second argument, uβ v + w = β uv + uw . Positive-definiteness needs no change, because conjugate symmetry makes uu real: applied to v = u, it gives uu = uu, and a number equal to its own conjugate is real, so the inequality uu ≥ 0 is meaningful in the complex case as well. Two vectors u, v ∈ V are called orthogonal if their inner product is zero, uv = 0. The pair (V, ··) is called an inner product space. Each inner product induces a norm via u := uu for all u ∈ V, which in turn induces a metric via uv := u - v for all u, v ∈ V. For the Euclidean space V = Rd, the standard inner product, also called the dot product, is xx' = x x' = j=1d xj x'j . Geometrically, the inner product determines orthogonal projections: the orthogonal projection of x' onto the direction of a nonzero vector x has the signed length xx' / x (see Fig.~\\ref{fig_innerproduct_dict}). The inner product also defines the angle θ between two nonzero vectors x, x' ∈ V, via θ := xx' / ( x x' ). The Cauchy-Schwarz inequality |xx'| ≤ x x' ensures that this ratio lies in [-1, 1] . Here is the cosine function, which decreases strictly from 0 = 1 to π = -1 and therefore takes each value of [-1,1] exactly once on the interval [0, π]. For the Euclidean plane R2, as illustrated by Fig.~\\ref{fig_innerproduct_dict}, the angle defined in this way agrees with the elementary geometric angle between two vectors . For two vectors, the sign and magnitude of their inner product can be read off the angle between these vectors. The inner product is positive for θ < π/2, zero for θ = π/2, where the two vectors are orthogonal, and negative for θ > π/2; and θ is the inner product divided by the two norms, so it grades alignment on [-1,1] without regard to length. The angle drawn in Fig.~\\ref{fig_innerproduct_dict} is smaller than π/2: the two vectors have a positive inner product, and the orthogonal projection therefore falls on the same side of the origin as x. Fig.~\\ref{fig_innerproduct_dict} projects one vector onto the direction of another, that is, onto the line it spans. The same characterization holds for a whole subspace, and it is what turns approximation problems, with respect to the induced norm, into systems of linear equations. Consider a subspace U V and a vector v ∈ V. In general, the vector of U closest to v with respect to some norm is called the projection of v onto U. If the norm is induced by some inner product, the projection becomes an orthogonal projection: a vector v ∈ U minimizes v - u over all u ∈ U if and only if the approximation error v - v is orthogonal to every vector in U, \\begin{equation} \\innerprod{\\vv - \\widehat{\\vv}}{\\vu} = 0 \\quad \\text{for all } \\vu \\in \\mathcal{U} \\text{.} \\end{equation} The reason is the Pythagorean identity \\[ \\norm{\\vv - \\vu}^{2} = \\norm{\\vv - \\widehat{\\vv}}^{2} + \\norm{\\widehat{\\vv} - \\vu}^{2} \\text{,} \\] which holds for every u ∈ U whenever \\eqref{equ_orthogonality_innerproduct_dict} does. The second term is non-negative, so no vector of U is closer to v than v is: orthogonality of the error and being closest are the same condition . The orthogonality condition \\eqref{equ_orthogonality_innerproduct_dict} is linear in v, so a minimizer can be computed by solving linear equations. The characterization extends beyond subspaces. For a non-empty closed convex set C Rd, a vector v ∈ C is the projection of v onto C if and only if \\begin{equation} \\innerprod{\\vv - \\widehat{\\vv}}{\\vu - \\widehat{\\vv}} \\leq 0 \\quad \\text{for all } \\vu \\in \\mathcal{C} \\text{.} \\end{equation} Such a projection exists and is unique . One application of the subspace case \\eqref{equ_orthogonality_innerproduct_dict} is a characterization of linear regression: the learned model parameters are those for which the prediction error is orthogonal to every column of the feature matrix (see normal equations), so the resulting predictions are the orthogonal projection of the label vector onto the column space of the feature matrix. The convex case \\eqref{equ_variational_inequality_dict} becomes relevant when linear regression is modified by adding a convex constraint on the model parameters. The Euclidean space Rd carries other inner products besides the standard one . As a case in point, each symmetric and positive definite matrix A ∈ Rd × d induces an inner product xx' := x A x'. Such a matrix weighs the coordinate directions against each other, and thereby fixes a different notion of similarity between data points. Take A = diag(4, 1/4), which counts the first feature sixteen times as heavily as the second, and the query x = (1, 1). Of the two candidates (0.9, 1.2) and (1.3, 0.4), the standard inner product prefers the first (2.1 against 1.7) and the weighted one prefers the second (5.3 against 3.9): the same query and the same candidates, but a different answer to which is more similar. Since an inner product also induces a norm and a metric, the choice of A fixes which data points count as nearest neighbors as well. An inner product also gives rise to the notion of an orthonormal basis: a basis b(1), …, b(d) of a vector space V of dimension d is orthonormal if its elements are pairwise orthogonal and have unit norm, b(j)b(j') = 0 for j j' and b(j)b(j) = 1 . The coordinates of a vector with respect to an orthonormal basis are delivered by inner products: u = j=1d βj b(j) with expansion coefficients βj = ub(j). Conversely, an inner product can be defined by declaring a basis orthonormal: given any basis b(1), …, b(d) of V, setting \\[ \\innerprod{\\vu}{\\vv} \\defeq \\sum_{\\featureidx=1}^{\\featuredim} \\expcoeff_{\\featureidx} \\expcoeff'_{\\featureidx} \\quad \\text{for } \\vu = \\sum_{\\featureidx=1}^{\\featuredim} \\expcoeff_{\\featureidx} \\vb^{(\\featureidx)} \\text{ and } \\vv = \\sum_{\\featureidx=1}^{\\featuredim} \\expcoeff'_{\\featureidx} \\vb^{(\\featureidx)} \\] yields the unique inner product for which this basis is orthonormal. The standard inner product of the Euclidean space arises in this way from the standard basis given by the columns of the identity matrix Id. Fig.~\\ref{fig_innerproduct_cities_dict} illustrates inner products between feature vectors in R2 that represent five European cities. Each city is characterized by its latitude and longitude as its two features. The similarity measure of the opening paragraph is what several machine learning constructions are built on. An embedding maps a text block to a vector in a vector space. The alignment of two text blocks is measured by an inner product of their corresponding embeddings . Each neuron of an artificial neural network matches its input vector against a vector of learned model parameters . attention scores in an large language model are scaled inner products between query and key vectors . Many machine learning methods access the feature vectors of data points only through inner products between feature vectors. A linear model uses hypotheses h(x) = wx. Consider regularized empirical risk minimization over a training set D(train) = (x(r), y(r)) r=1m, with a penalty that increases strictly with w. It returns model parameters that are a linear combination of the feature vectors in the training set, w = r=1m βr x(r). This is the representer theorem, and the support-vector expansion of the support vector machine is a special case of it . The prediction for a new data point then reads \\[ \\learnthypothesis(\\featurevec) = \\sum_{\\sampleidx=1}^{\\samplesize} \\expcoeff_{\\sampleidx} \\innerprod{\\featurevec^{(\\sampleidx)}}{\\featurevec} \\text{,} \\] so training and prediction touch the feature vectors only through the inner products x(r)x(r') within the training set and x(r)x between the training set and the new data point. This is exploited by kernel methods that use a kernel k(x, x') to compute inner products without ever forming feature vectors (see kernel method). Methods built on Euclidean distances access the feature vectors only through inner products as well, even when they are defined without reference to one. The squared Euclidean distance between two feature vectors is a sum of three inner products, x - x'22 = xx - 2 xx' + x'x'. A method that uses the feature vectors only through their Euclidean distances therefore also uses them only through inner products. k-nearest neighbors is one example: it ranks the data points in the training set by the distance of their feature vectors to the feature vector of the new data point . The same kernel substitution therefore reaches it: by the identity above, each Euclidean distance is computed from three kernel evaluations, and no feature vector is ever formed.", "synonyms": [ "scalar product", "dot product" ], "see_also": [ "field", "norm", "vector", "hilbertspace", "orthogonalitycondition", "metricspace", "euclidspace", "cauchyschwarzinequ", "orthogonalprojection", "kernel", "kernelmethod", "neuron", "attention", "pca", "knn" ], "demo": "https://dictionaryofml.org/terms/innerproduct.py", "notebook": "https://dictionaryofml.org/terms/innerproduct.ipynb" }, { "key": "interpretability", "name": "interpretability", "url": "https://dictionaryofml.org/terms/interpretability.html", "pdf": "https://dictionaryofml.org/terms/interpretability.pdf", "abstract": "The interpretability of a machine learning method is the extent to which a human user can comprehend the computational process it carries out. Interpretability is closely related to predictability: the user should be able to anticipate the predictions of the method on a test set. Interpretability differs from explainability, which concerns understanding specific predictions with the help of provided explanations.", "description": "Consider a training set of monthly sales and a straight line fitted through it. Reading next month's prediction off that line takes a slope, an intercept, and one multiplication, and a user can carry the computation out on paper. The same prediction delivered by a deep net arrives with no account a user could follow. Interpretability is what separates the two. A machine learning method is interpretable for a human user if they can comprehend the computational process of the method. The international terminology standard for artificial intelligence does not define interpretability . Its closest notion is predictability, the property of an artificial intelligence system that enables reliable assumptions by users about the predictions delivered by an machine learning method . Comprehension of a computation cannot be observed directly, so interpretability is judged through predictability: the user anticipates the predictions the method delivers, and the evaluations proposed in the literature measure how often those anticipations are right (; ; ; ). Predictability is weaker than interpretability: a method whose computation the user can follow is predictable to that user, but a method whose predictions the user anticipates need not be one whose computation they could carry out. The terminology standard makes the same point: a user may rely on the predictions of an artificial intelligence system without being able to comprehend how the method produced them . Fig.~\\ref{fig_interpretability_predict_dict} depicts a test of predictability. A training set is used by two different methods that learn hypotheses h and h'. The user anticipates the predictions on a test set by visually extrapolating the linear trend of the training set. The predictions of h agree with these anticipations: the method producing h is predictable to the user. The predictions of h' deviate from them: the method producing h' fails the test of predictability and therefore cannot be one whose computation the user comprehends. The agreement alone does not establish interpretability, since the user anticipated the predictions of h without carrying out its computation. The method producing h is nevertheless also interpretable to a user familiar with the concept of a linear map: such a user could comprehend the computation of h. Interpretability can also be obtained by decomposing a learned hypothesis into components that are themselves interpretable . The hypothesis h in Fig.~\\ref{fig_interpretability_predict_dict} decomposes into a slope and an intercept: the slope states how the prediction changes with the feature. A trained deep net lacks such a decomposition, since the activation of an individual neuron typically does not correspond to a human-interpretable concept. mechanistic interpretability aims to recover interpretable components from a trained deep net (; ) by decomposition methods such as sparse coding of activations . The AI Risk Management Framework of the US National Institute of Standards and Technology distinguishes interpretability from explainability , the property pursued by explainable artificial intelligence . In contrast to interpretability, explainability requires that an explanation is provided along with each prediction. These explanations may take the form of saliency maps or reference examples from the training set. A challenge for explainable artificial intelligence methods is that a computed explanation can be unfaithful to the learned hypothesis and thereby mislead the user .", "synonyms": [], "see_also": [ "explainability", "xaiterm", "interpretableml", "trustAI", "regularization", "lime", "mechanisticinterpretability", "transparency" ], "demo": "https://dictionaryofml.org/terms/interpretability.py", "notebook": "https://dictionaryofml.org/terms/interpretability.ipynb" }, { "key": "interpretableml", "name": "interpretable machine learning", "url": "https://dictionaryofml.org/terms/interpretableml.html", "pdf": "https://dictionaryofml.org/terms/interpretableml.pdf", "abstract": "Interpretable machine learning (interpretable ML) refers to machine learning methods whose hypothesis space contains only hypotheses that a human user can comprehend directly, such as sparse linear models or shallow decision trees. Such methods let a user see how a prediction changes when the features of a data point change, and how the predictions change when a different training set is used. An interpretable hypothesis is its own explanation; post hoc explanations of the predictions of an opaque learned hypothesis can instead be unfaithful. Restricting the hypothesis space to interpretable hypotheses also acts as a form of regularization. A penalty term can pick a comprehensible hypothesis out of a hypothesis space that mostly holds others: least absolute shrinkage and selection operator drives most weights of a linear model to zero, leaving a hypothesis a user can read off, and explainable empirical risk minimization penalizes deviations from the predictions one user supplies during training.", "description": "Interpretable ML refers to machine learning methods whose hypothesis space contains only hypotheses that a human user can comprehend directly (see interpretability). Such methods let a user follow both what a learned hypothesis does to its input and how training produced it. Ultimately, an interpretable machine learning method allows the user to understand how a prediction changes when the features of a data point change, and how the predictions change when a different training set is used. Thus, interpretable machine learning methods require suitable choices for the hypothesis space and the training algorithm. Examples of interpretable machine learning include empirical risk minimization with linear models that combine a small number of meaningful features, shallow decision trees whose predictions follow from a few explicit tests, GAMs, and lists of decision rules . Fig.~\\ref{fig_interpretableml_dict} depicts a decision tree for medical triage in a hospital. This system involves two tests of vital signs, allowing the staff to trace every prediction by reading the tree. An interpretable hypothesis makes the effect of feature changes explicit. For a linear model h(w)(x) = wx, changing the feature xj by an amount Δ changes the prediction by exactly wj Δ. Each weight states how strongly one feature affects the prediction. For the decision tree of Fig.~\\ref{fig_interpretableml_dict}, a prediction changes only when a feature crosses one of the explicit test thresholds. A patient at 38.5 C with a heart rate of 110/min is routine; a rise to 38.9 C leaves that prediction unchanged, while a rise to 39.5 C turns it urgent. An interpretable machine learning method also makes the effect of training set changes traceable. This dependence is studied under stability, which formalizes an machine learning method as a map from a training set to a learned hypothesis. For linear regression, this map is available in closed form through the normal equations, so the user can compute how a perturbed label in the training set shifts the predictions (see linear regression). An interpretable hypothesis is its own explanation: no separate explanation is constructed for it. The explanation delivered with a prediction can then be a description of the learned hypothesis itself: the weights of a linear model that combines few features, or the flow chart of a shallow decision tree as in Fig.~\\ref{fig_interpretableml_dict}. A hypothesis space containing only such hypotheses is called intrinsically interpretable . This contrasts with explainable artificial intelligence, which constructs post hoc explanations for the predictions of a possibly opaque learned hypothesis. For high-stakes applications, such as the medical triage of Fig.~\\ref{fig_interpretableml_dict}, using an interpretable model can be preferable to explaining an opaque model, since post hoc explanations can be unfaithful to the hypothesis they explain . An interpretable learned hypothesis can also be obtained without restricting the hypothesis space beforehand. A penalty term added to the objective function of empirical risk minimization de-incentivizes the hypotheses a user cannot follow, so training picks a comprehensible one out of a hypothesis space that mostly holds the others. least absolute shrinkage and selection operator does this for linear models, which are no more comprehensible than a deep net once they weigh hundreds of features. least absolute shrinkage and selection operator uses the ℓ1-norm w1 of the weights w as the penalty term, which drives most weights to exactly zero . The learned hypothesis is then a weighted sum of a few features, so a user can name the features the prediction rests on and how much each one moves it. explainable empirical risk minimization extends this idea to the training of an arbitrary model. Its penalty term is built from the predictions that one specific user supplies for the data points of the training set (see explainable empirical risk minimization). It therefore delivers explainability to that user rather than interpretability, and it leaves the hypothesis space as it is. The user predictions need not be elicited one data point at a time. They might be obtained as the predictions of a simpler proxy model that the user considers interpretable, so that the penalty term measures the deviation from that proxy. In Fig.~\\ref{fig_interpretableml_eerm_dict}, the user predictions are obtained from a simple linear model fitted to the training set. See also: interpretability, explainability, explainable artificial intelligence, decision tree, linear model, generalized additive model, least absolute shrinkage and selection operator, regularization, stability, local interpretable model-agnostic explanations, explainable empirical risk minimization, transparency, EU AI Act, trustworthy artificial intelligence.", "synonyms": [], "see_also": [], "demo": "https://dictionaryofml.org/terms/interpretableml.py", "notebook": "https://dictionaryofml.org/terms/interpretableml.ipynb" }, { "key": "kernelmethod", "name": "kernel method", "url": "https://dictionaryofml.org/terms/kernelmethod.html", "pdf": "https://dictionaryofml.org/terms/kernelmethod.pdf", "abstract": "A kernel method applies a linear method, such as a linear model or a linear classifier, to transformed feature vectors. The transformation is constructed from a kernel: each feature vector is mapped to a function with domain equal to the original feature space, and inner products between transformed feature vectors reduce to evaluations of a kernel. Kernel methods offer great flexibility in the choice of the feature space, which can consist of text documents or discrete structures such as graphs.", "description": "A kernel method is a machine learning method that applies a linear method, such as a linear model for regression or a linear classifier for classification, to transformed feature vectors that are constructed from a kernel (; ). Kernel methods are used in computer vision to decide whether an image shows a specific object , and in natural language processing to classify text documents by topic . Linear methods learn a hypothesis that combines the features of a data point linearly, e.g., h(x) = w x for the feature space X = Rd. Such a hypothesis predicts well only if the relation between the feature vectors and the labels of data points is approximately linear. Moreover, a linear method requires numeric feature vectors and is not directly applicable to data points from an arbitrary feature space X, such as text documents or graphs. Both limitations can be mitigated by a feature transformation φ: X → H from the original feature space X to a transformed feature space H, chosen as a Hilbert space. Fig.~\\ref{fig_linsep_kernel_dict} shows a binary classification problem with the first limitation: no straight line separates the two classes in the original feature space. A suitable feature transformation makes a dataset ``more linear'': the relation between the transformed feature vectors φ(x) and the labels is closer to linear in H than the relation between the original feature vectors x and the labels is in X. A useful feature transformation φ often delivers transformed feature vectors in a Hilbert space H of high (possibly infinite) dimension . Computing and storing the transformed feature vectors φ(x) explicitly can then be infeasible. However, linear methods access the feature vectors of data points only through inner products between pairs of feature vectors (see linear model). For example, the prediction delivered by a trained support vector machine is a weighted sum of inner products between the feature vector of a new data point and the feature vectors in the training set. To apply a linear method to the transformed feature vectors, it is therefore enough to know the inner products φ(x)φ(x') for every possible pair of feature vectors x, x' ∈ X. These inner products are captured by a single function of two arguments, the kernel defined by \\[ \\kernelmap{\\featurevec}{\\featurevec'} \\defeq \\innerprod{\\featuretrafo(\\featurevec)}{\\featuretrafo(\\featurevec')} \\text{, for } \\featurevec, \\featurevec' \\in \\featurespace \\text{.} \\] The value xx' quantifies the similarity of the feature vectors x and x'. As an inner product of transformed feature vectors, the kernel is symmetric, and every matrix of pairwise kernel values is positive semi-definite; these two properties characterize kernels. A linear method that is applied to the transformed feature vectors thus requires only the kernel k; the feature transformation φ itself is never evaluated. Kernel methods reverse this construction: they start from a kernel k: X × X → R and construct the feature transformation from it. The feature transformation sends a feature vector x ∈ X to the function x·, i.e., the transformed feature vector z := φ(x) = x· is itself a function with domain X, for every x ∈ X. These functions belong to the reproducing kernel Hilbert space Hk associated with the kernel k . The inner product between the transformed feature vectors of two original feature vectors x, x' ∈ X is a single kernel evaluation, \\[ \\innerprod{\\kernelmap{\\featurevec}{\\cdot}}{\\kernelmap{\\featurevec'}{\\cdot}} = \\kernelmap{\\featurevec}{\\featurevec'} \\text{.} \\] More generally, the inner product between the transformed feature vector x· and any function h ∈ Hk is a point evaluation, \\[ \\innerprod{\\hypothesis}{\\kernelmap{\\featurevec}{\\cdot}} = \\hypothesis(\\featurevec) \\text{,} \\] which is referred to as the reproducing property of the reproducing kernel Hilbert space Hk . This reproducing property contains x·x'· = xx' as a special case which is obtained for the choice h = x'·. The (possibly infinite-dimensional) transformed feature vectors therefore never need to be computed explicitly. Fig.~\\ref{fig_kernelmethod_rkhs_dict} depicts the reproducing kernel Hilbert space Hk for a training set with two data points: a hypothesis h is a single vector in Hk. The value hx· = h(x) is a linear function of the transformed feature vector x· ∈ Hk, but the function x h(x) induced on the original feature space X is nonlinear in general. Linearity in the original feature vector x may not even be defined, since the feature space X need not be a vector space, e.g., when it consists of text documents or graphs. Kernel methods can be formulated as regularized empirical risk minimization over the reproducing kernel Hilbert space Hk, \\[ \\min_{\\hypothesis \\in \\hilbertspace_{\\kernel}} \\frac{1}{\\samplesize} \\sum_{\\sampleidx=1}^{\\samplesize} \\lossfunc{\\big(\\featurevec^{(\\sampleidx)}, \\truelabel^{(\\sampleidx)}\\big)}{\\hypothesis} + \\regparam \\normgeneric{\\hypothesis}{\\hilbertspace_{\\kernel}}^{2} \\text{,} \\] over a training set with feature vectors x(r) and labels y(r), for r = 1, …, m, and with a regularization parameter α > 0 that weights the penalty term hHk2. The loss function (x(r), y(r))h depends on h only through the prediction h(x(r)), i.e., it is a function of the prediction h(x(r)) and the label y(r) only. Since h ∈ Hk, the reproducing property delivers this prediction as an inner product, h(x(r)) = hx(r)·. By the representer theorem, this optimization problem has a minimizer of the form h = r=1m βr x(r)· with expansion coefficients β1, …, βm ∈ R . Fig.~\\ref{fig_kernelmethod_rkhs_dict} illustrates the underlying projection argument: replacing a hypothesis h by its orthogonal projection h onto the subspace spanned by x(1)·, …, x(m)· leaves the predictions on the training set unchanged and never increases the penalty term; hence some minimizer h lies in this subspace and has the above form. training thus reduces to a convex optimization problem in these m coefficients, and the prediction h(x) = r=1m βr x(r)x requires only kernel evaluations. Examples of such kernel methods are the kernel support vector machine, obtained for the hinge loss, and kernel ridge regression, obtained for the squared error loss (; ). For X = Rd, a widely used choice of the kernel k is the Gaussian kernel xx' = (- x - x'22 / (2 σ2) ) with bandwidth σ > 0. kernels are also available for data points without numeric features, such as strings and graphs . Fig.~\\ref{fig_kernelmethod_demo_dict} shows the nonlinear decision boundary h(x) = 0 of a hypothesis h learned by a Gaussian-kernel method with squared error loss from a training set of two concentric rings that no linear classifier separates. Here kernel ridge regression fits the labels y(r) ∈ -1, +1 and the prediction is the sign of the fitted value, a construction referred to as regularized least-squares classification . Kernel methods impose the smoothness assumption through the reproducing kernel Hilbert space norm. That norm hHk quantifies the smoothness of a hypothesis h: it is large for a rapidly varying h and small for a smooth h. Moreover, different kernels impose different kinds of smoothness. The Gaussian kernel gives infinitely differentiable functions and penalizes high-frequency components. The Mat\\'ern kernels are a family of kernels indexed by a smoothness parameter ν > 0: their reproducing kernel Hilbert space coincides, with equivalent norms, with the Sobolev space of order determined by ν, whose functions have derivatives up to that order in the weak sense (; ). Because the penalty term α hHk2 in the above regularized empirical risk minimization is the squared reproducing kernel Hilbert space norm, it implements smoothness as quantified by hHk: among hypotheses with the same training error, the regularized empirical risk minimization objective function is smallest for the hypothesis with the smallest reproducing kernel Hilbert space norm. The choice of kernel shapes not only training but also the predictions of the learned hypothesis: for localized kernels, such as the Gaussian kernel, the prediction h(x) is dominated by the data points in the training set whose feature vectors are close to x. In this sense, kernel methods implement the smoothness assumption: data points with nearby feature vectors obtain similar predictions. The locality of the predictions also makes kernel methods a soft-weighted analogue of k-nearest neighbors: instead of averaging the labels of a fixed number of nearest data points, the prediction weights all data points in the training set by the kernel value x(r)x .", "synonyms": [ "kernel machine" ], "see_also": [ "kernel", "featuretransformation", "hilbertspace", "rkhs", "linmodel", "linclass", "svm", "kernelridgeregression", "ridgeregression", "smoothnessassumption", "knn" ], "demo": "https://dictionaryofml.org/terms/kernelmethod.py", "notebook": "https://dictionaryofml.org/terms/kernelmethod.ipynb" }, { "key": "kmeans", "name": "k-means", "url": "https://dictionaryofml.org/terms/kmeans.html", "pdf": "https://dictionaryofml.org/terms/kmeans.pdf", "abstract": "The k-means method is a hard clustering method for data points with numeric feature vectors. It partitions a dataset into k clusters. Each cluster is represented in the feature space by a cluster centroid. The quality of a clustering is measured by the clustering error: the average squared Euclidean distance between a data point and the nearest cluster centroid. Minimizing the clustering error is a non-convex and NP-hard optimization problem. Practical k-means methods use an approximate iterative optimization method known as Lloyd's algorithm: a fixed-point iteration that alternates between assigning each data point to its nearest cluster centroid and re-averaging. No iteration increases the clustering error, and the iterates reach a fixed point after finitely many iterations.", "description": "Consider the task of grouping the days of a year by their weather at some location, say the town Krems in Austria. Each day yields a data point whose feature vector is obtained by stacking temperature measurements recorded during that day, e.g., the minimum and the maximum air temperature. Days in the same group should have similar feature vectors. k-means produces such a grouping: with k = 2, it partitions the days into a cold-season and a warm-season cluster. k-means is an optimization-based hard clustering method for data points with a numeric feature vector . It partitions a dataset D = x(1), …, x(m) with x(r) ∈ Rd into k disjoint clusters indexed by c = 1, …, k. Each cluster is represented by a cluster centroid c ∈ Rd, obtained by averaging the feature vectors of the data points assigned to that cluster (see Fig.~\\ref{fig_kmeans_dict} for an example with k = 2 cluster centroids). The quality of a given choice of cluster centroids is measured by the clustering error: the average squared Euclidean distance between the feature vector of a data point and the nearest cluster centroid. The k-means principle is to choose cluster centroids that minimize the clustering error, \\min_{\\clustercentroid{1}, \\ldots, \\clustercentroid{\\nrcluster}} \\;\\frac{1}{\\samplesize}\\sum_{\\sampleidx=1}^{\\samplesize} \\min_{\\clusteridx = 1, \\ldots, \\nrcluster} \\normgeneric{\\featurevec^{(\\sampleidx)} - \\clustercentroid{\\clusteridx}}{2}^{2}\\text{.} The inner minimization assigns each data point to its nearest cluster centroid. The outer minimization chooses the cluster centroids so that the average squared Euclidean distance (between each data point and its nearest cluster centroid) is small. Solving the k-means optimization problem exactly is NP-hard . The difficulty of the optimization problem is also reflected by the fact that the objective function is non-convex in the cluster centroids. With a single data point at x = 0 and two scalar cluster centroids w(1), w(2) ∈ R, the objective is f(w(1), w(2)) = ((w(1))2, (w(2))2). Both (w(1), w(2)) = (1, 0) and (0, 1) are global optima with value 0, meaning at least one centroid perfectly coincides with the data point x, yet their midpoint (0.5, 0.5) gives f = 0.25 > 0. A convex combination of optima is therefore worse than either optimum, which violates the defining property of a convex function. Practical implementations use iterative alternating-minimization methods that converge to a local optimum. A widely used choice is Lloyd's algorithm, which alternates between assigning each data point to its closest cluster centroid and recomputing each cluster centroid as the mean of the currently assigned data points. One iteration of Lloyd's algorithm is the application of an operator F to the stacked cluster centroids w := (1, …, k), so the method is the fixed-point iteration w(t+1) = F(w(t)). Its fixed points are cluster centroids that each coincide with the mean of the data points assigned to them. Applying F never increases the clustering error, and since a finite dataset admits only finitely many partitions, the iterates reach a fixed point after finitely many iterations . The k-means principle is a special case of the empirical risk minimization principle. Indeed, k-means uses a hypothesis space Hk that consists of all maps h: X → X of the piecewise-constant form \\hypothesis(\\featurevec) = \\clustercentroid{\\,\\clusteridx^{\\star}(\\featurevec)}, \\qquad \\clusteridx^{\\star}(\\featurevec) = \\argmin_{\\clusteridx = 1, \\ldots, \\nrcluster} \\normgeneric{\\featurevec - \\clustercentroid{\\clusteridx}}{2}\\text{,} parameterized by the k cluster centroids 1, …, k ∈ X. Each such hypothesis maps every feature vector to its nearest cluster centroid. The loss function is the squared Euclidean distance between the feature vector of each data point and its nearest cluster centroid, xh = x - h(x)22. With these choices, empirical risk minimization on Hk recovers the k-means optimization problem above . Applications of k-means include customer segmentation (grouping the customers of a retailer by feature vectors such as monthly spending and number of purchases), outlier detection, compression, and image segmentation. Fig.~\\ref{fig_kmeans_radar_dict} illustrates the latter two on a subsampled photo of the \\\"{O}tscher massif: k-means on the pixel colors with k = 4 replaces each pixel's color by the nearest of k palette colors, compressing 24 bits per pixel to 2 bits per pixel plus a small palette; with k = 2, the cluster assignments form a segmentation mask that separates the sky and the summit from the vegetation. Synonyms: k-means clustering.", "synonyms": [], "see_also": [ "hardclustering", "cluster", "clustercentroid", "clusteringerror", "lloydalgorithm", "kmeanspp", "eucliddist", "optproblem", "erm", "hypospace", "lossfunc" ], "demo": "https://dictionaryofml.org/terms/kmeans.py", "notebook": "https://dictionaryofml.org/terms/kmeans.ipynb" }, { "key": "label", "name": "label", "url": "https://dictionaryofml.org/terms/label.html", "pdf": "https://dictionaryofml.org/terms/label.pdf", "abstract": "The ultimate goal of machine learning is the accurate prediction of the label of a data point from its features. A label is an attribute of a data point that represents a quantity of interest. Determining the label of a data point is error-prone and often involves human annotation. The choice of label determines the learning task: whether the task is classification or regression depends on the label space together with the loss function. In self-supervised learning, the label is constructed from the features themselves, so that no manual annotation is required.", "description": "A label of a data point z is one of its attributes that represents a higher-level fact, or quantity of interest, and that typically requires human expertise or domain knowledge to determine (; ; ). Unlike a feature, which can be computed or measured easily, a label is usually harder to obtain. For a data point that represents a patient, a feature could be the body weight, while the label could be the presence of a disease. Whether a given attribute serves as a feature or a label is often a design choice that depends on the resources available in a given machine learning application. If an oncologist is readily available, a cancer prognosis can serve as a feature. On the other hand, if no oncologist is available, the cancer prognosis is a label that needs to be predicted from patient features. One way to make the notion of a label precise is by introducing a labeling function h: Z → Y. The domain of this function is the data point space Z, the set of all possible data points that can occur in a given machine learning application. Its values lie in the label space Y, the set of all possible values the label of a data point can take on. The function assigns each data point z its label y = h(z) (). Note that the labeling function h acts on the data point z itself, that is, on the whole patient. A trained model (or learned hypothesis) h instead acts on the feature vector x, a fixed list of recorded features such as the patient's body weight and age. The feature vector usually captures only part of a data point. For example, in a health-care application, the data point space Z contains patients, whereas the feature space X contains only their recorded feature vectors. Two patients with the same feature vector may therefore carry different labels, since they can differ in information that was never recorded as a feature. No hypothesis h that reads only x can then deliver perfect predictions. In practice, the true label h(z) of a data point is rarely known exactly. Even with full access to the data point, its label must be determined by human judgment or measurement, which is imperfect, so the available label is only a noisy version of h(z) (). For instance, an object category assigned by a human annotator or a diagnosis obtained from a clinician may be wrong, even though the annotator or clinician examines the data point directly. The discrepancy between such an observed label and the true label is called label noise and can degrade training and the resulting generalization. This source of error differs from the incompleteness of the recorded features discussed above: here the data point is accessible, but its label is determined imperfectly. Label noise also matters for regulation. The EU AI Act requires the minimization of label noise and its impact on high-risk ML systems (). When labels are personal data, the general data protection regulation requires them to be accurate (). One way to reduce the effect of label noise is label smoothing, which slightly perturbs the labels of data points used for model training (). It is a form of data augmentation that acts as regularization. The choice of label (or labeling function h) determines a specific learning task. For the image of a cow herd in Fig.~\\ref{fig_label_choices}, the label could be a binary indicator of whether the image contains cows (binary classification), the number of cows or the average green level (regression), or a masked center pixel predicted from the surrounding pixels (self-supervised learning). Whether a learning task is a classification problem or a regression problem depends not only on the label space Y but also on the loss function. For example, the label space Y = 0,1,2,3,… can be used for a classification method that uses the 0/1 loss as the loss function. The same label space can also be used for a regression method that uses the squared error loss as the loss function. In self-supervised learning, the label is constructed from the features themselves, so no manual annotation is required. A prominent example is large language model training: consider the sentence ``All human beings are born free and equal,'' tokenized as the sequence (``All'', ``human'', ``beings'', ``are'', ``born'', ``free'', ``and'', ``equal''). The label for the subsequence (``All'', ``human'', ``beings'') is the next token ``are,'' the label for (``All'', ``human'', ``beings'', ``are'') is ``born,'' and so on. By sliding through the sequence, a single sentence produces many labeled data points from plain text alone.", "synonyms": [ "target", "response", "ground truth", "quantity of interest" ], "see_also": [ "datapoint", "feature", "labelspace", "learningtask", "classification", "regression", "selfsupervisedlearning", "llm", "euaiact", "robustness", "dataaug", "regularization" ], "demo": "https://dictionaryofml.org/terms/label.py", "notebook": "https://dictionaryofml.org/terms/label.ipynb" }, { "key": "linreg", "name": "linear regression", "url": "https://dictionaryofml.org/terms/linreg.html", "pdf": "https://dictionaryofml.org/terms/linreg.pdf", "abstract": "Linear regression is a regression method that learns a linear hypothesis map for predicting the numeric label of a data point from its features. Its least squares variant chooses a map that minimizes the average squared error loss on a training set. Linear regression is an instance of empirical risk minimization that is obtained by using the linear model and the squared error loss. The optimal model parameters are determined by the normal equations. This system can be solved in closed form or iteratively, for example, by gradient descent. Adding a penalty term to the average squared error loss yields regularized variants such as ridge regression and least absolute shrinkage and selection operator.", "description": "Linear regression methods learn a linear hypothesis map that delivers a prediction of the numeric label of a data point. The prediction is based solely on the features of the data point, which are fed as input to the learned hypothesis map. Linear regression can be used to predict tomorrow's temperature from weather measurements recorded over the last five days, using them as features and tomorrow's temperature as the label . Formally, linear regression learns a linear hypothesis map h(w)(x) = wx to predict the numeric label y ∈ R of a data point from its feature vector x = (x1, …, xd) ∈ Rd. Here, d denotes the number of features of a data point. The model parameters w are learned from a training set D(train) = x(r)y(r) r=1m of data points. Appending a constant feature to x lets one entry of w act as an intercept, so wx represents an affine function of the original measurements. In what follows, x denotes the feature vector that is fed to the linear hypothesis map. It can be either the original or the augmented feature vector. The least squares variant of linear regression measures the quality of a linear hypothesis map by the average squared error loss on the training set. As an instance of empirical risk minimization, it learns the model parameters w by solving the optimization problem \\[ \\min_{\\weights \\in \\reals^{\\nrfeatures}} \\frac{1}{\\samplesize} \\sum_{\\sampleidx=1}^{\\samplesize} \\big( \\truelabel^{(\\sampleidx)} - \\weights^{\\top} \\featurevec^{(\\sampleidx)} \\big)^{2} \\text{.} \\] Fig.~\\ref{fig_linreg_dict} illustrates this linear regression problem for a training set with a constant scalar feature x(r)=1 for r=1,…,m. The optimization problem can be written more compactly using the feature matrix X and the label vector y, \\[ \\featuremtx = \\big(\\featurevec^{(1)},\\,\\ldots,\\,\\featurevec^{(\\samplesize)}\\big)^{\\top} \\in \\reals^{\\samplesize \\times \\nrfeatures} \\text{,} \\qquad \\labelvec = \\big( \\truelabel^{(1)},\\,\\ldots,\\,\\truelabel^{(\\samplesize)} \\big)^{\\top} \\in \\reals^{\\samplesize} \\text{.} \\] In terms of X and y, the optimization problem reads \\[ \\min_{\\weights \\in \\reals^{\\nrfeatures}} \\underbrace{\\frac{1}{\\samplesize} \\normgeneric{\\labelvec - \\featuremtx \\weights}{2}^{2}}_{f(\\weights)} \\text{.} \\] The average squared error loss f(w) is a convex and differentiable function of w. By the zero-gradient condition (), a vector w solves this optimization problem if and only if it satisfies the system of linear equations \\begin{equation} \\featuremtx^{\\top}\\featuremtx \\widehat{\\weights} = \\featuremtx^{\\top} \\labelvec \\text{.} \\end{equation} The normal equations \\eqref{eq_linreg_normal_eq_dict} always have a solution because Xy lies in the column space of XX (). However, the solution of \\eqref{eq_linreg_normal_eq_dict} is unique only if the feature matrix X has full column rank. This requires at least as many data points as features, m ≥ d. When this holds, and the columns of X are linearly independent, the matrix XX is invertible and \\eqref{eq_linreg_normal_eq_dict} has the unique closed-form solution w = (XX)-1 X y. If instead m < d (more features than data points), X cannot have full column rank, XX is singular, and the optimization problem has infinitely many solutions. All of them incur the same minimal average squared error loss on the training set, but their predictions for data points outside the training set can differ arbitrarily. Choosing among them is therefore a matter of generalization (see overfitting). A unique solution can be selected, for example, by picking the minimum-norm solution X y given by the pseudoinverse X. A common remedy is regularization: adding a penalty term to the average squared error loss. The penalty term can be interpreted as an estimate of how much higher the loss is on data points outside the training set than on it . This yields regularized variants of linear regression, with ridge regression using the penalty term α w22 and the least absolute shrinkage and selection operator using the penalty term α w1. Linear regression also has a statistical interpretation. Consider a probabilistic model with a joint probability distribution P(x, y) over the features and the label. Under the squared error loss, the Bayes estimator (i.e., the hypothesis with minimum risk) is the conditional expectation Ey x (; ). In this statistical interpretation, x denotes the original feature vector. When the features and the label are jointly Gaussian RVs with zero mean and invertible covariance matrix x = Exx, this Bayes estimator is \\[ \\bayeshypothesis(\\featurevec) = \\big(\\weights^{\\star}\\big)^{\\top}\\featurevec \\text{, with } \\weights^{\\star} = \\big(\\covmtx{\\featurevec}\\big)^{-1}\\, \\covvec{\\featurevec,\\truelabel} \\text{.} \\] Here, x,y = E is the covariance between the features x and the label y. This Bayes estimator is linear in x. Without the zero-mean assumption, it is an affine function of x. However, as for the linear hypothesis map above, the intercept can be absorbed by appending a constant feature. Least-squares linear regression is the empirical risk minimization counterpart of this predictor: it replaces x and x,y with the sample-based estimates (1/m)XX and (1/m)Xy, recovering, for invertible XX, the unique solution w = (XX)-1Xy of \\eqref{eq_linreg_normal_eq_dict}. Instead of solving the normal equations \\eqref{eq_linreg_normal_eq_dict} directly (via the inverse matrix or pseudoinverse), machine learning methods often solve it by an iterative optimization method, as implemented in widely used machine learning software libraries such as scikit-learn () and PyTorch (). Starting from initial model parameters w(0), such a method repeatedly applies an update operator F, \\[ \\weights^{(\\iteridx+1)} = \\fixedpointop\\big(\\weights^{(\\iteridx)}\\big) \\text{, for } \\iteridx = 0,1,\\ldots \\text{.} \\] The operator F is designed such that the sequence of model parameters w(t) converges to a solution of \\eqref{eq_linreg_normal_eq_dict}. One important example of such an iterative optimization method is gradient descent, which uses the gradient descent step operator \\[ \\gdstep{\\lrate}(\\weights) = \\weights - \\lrate \\nabla f(\\weights) \\text{.} \\] Here, η > 0 is a step size that must be chosen small enough to ensure convergence (). The superscript in η makes the dependence of the operator on η explicit. Inserting the average training error into the general form of the gradient descent step yields the explicit update \\[ \\begin{aligned} \\gdstep{\\lrate}(\\weights) &= \\weights + \\frac{2\\lrate}{\\samplesize} \\sum_{\\sampleidx=1}^{\\samplesize} \\big( \\truelabel^{(\\sampleidx)} - \\weights^{\\top}\\featurevec^{(\\sampleidx)} \\big) \\featurevec^{(\\sampleidx)} &= \\Big(\\mI - \\frac{2\\lrate}{\\samplesize}\\featuremtx^{\\top}\\featuremtx\\Big)\\weights + \\frac{2\\lrate}{\\samplesize}\\featuremtx^{\\top}\\labelvec \\text{.} \\end{aligned} \\] The fixed points of η are exactly the solutions of the normal equations \\eqref{eq_linreg_normal_eq_dict}, since η(w) = w holds if and only if XXw = Xy. The update operator is affine, \\gdstep{\\lrate}(\\weights) = \\mM \\weights + \\vb \\text{,} with linear part M = I - (2η/m)XX and offset b = (2η/m) Xy. When X has full column rank, so that XX is positive definite, η is a contractive operator with respect to the Euclidean norm for every step size 0 < η < m/. Here, is the largest eigenvalue of XX: the eigenvalues of the linear part M are 1 - (2η/m) j, with the positive eigenvalues j of XX, and they lie strictly between -1 and 1 exactly for such step sizes (cf.\\ , which covers gradient descent for any smooth, strongly convex objective function and step sizes η ≤ m/( + ), with the smallest eigenvalue of XX). The fixed-point iteration converges to the unique solution w. Moreover, the convergence speed of gradient descent is governed by the ratio of the largest to the smallest eigenvalue of XX, that is, by its condition number XX = / . The gradient descent step above uses the entire training set to compute the gradient. When the data points instead arrive sequentially at time instants t=1,2,…, or the training set is too large to fit in memory, the model parameters can be learned by online gradient descent. At each time step t, it applies a single gradient descent step using only the arriving data point x(t)y(t), \\[ \\weights^{(\\iteridx+1)} = \\weights^{(\\iteridx)} + 2\\lrate \\big( \\truelabel^{(\\iteridx)} - \\big(\\weights^{(\\iteridx)}\\big)^{\\top} \\featurevec^{(\\iteridx)} \\big) \\featurevec^{(\\iteridx)} \\text{.} \\] This is the gradient descent step on the loss function (y(t) - wx(t))2 of the arriving data point alone, rather than on the average squared error loss f(w). It is the least mean squares update (). It avoids storing the entire training set, and its per-iteration complexity is independent of the training set size m. The optimality condition \\eqref{eq_linreg_normal_eq_dict} allows one to study the stability of linear regression. Ideally, the learned model parameters are insensitive to limited perturbations of the training set. A label-only perturbation, for example, replaces a single label of the training set with an outlier or another corrupted value. A perturbed training set yields a feature matrix X = X + Δ X and a label vector y = y + Δ y, with perturbation matrix Δ X and vector Δ y. This gives the perturbed normal equations \\begin{equation} \\widetilde{\\featuremtx}^{\\top} \\widetilde{\\featuremtx} \\widetilde{\\weights} = \\widetilde{\\featuremtx}^{\\top} \\widetilde{\\labelvec} \\text{.} \\end{equation} matrix perturbation theory quantifies how much w deviates from a solution w of the clean normal equations \\eqref{eq_linreg_normal_eq_dict} . For a label-only perturbation, Δ X = 0, the perturbed and clean normal equations share the coefficient matrix XX. Assume X has full column rank, so XX is invertible and the solution is unique. Subtracting the clean normal equations \\eqref{eq_linreg_normal_eq_dict} from the perturbed normal equations \\eqref{eq_linreg_perturbed_normal_eq_dict} yields \\[ \\widetilde{\\weights} - \\widehat{\\weights} = \\big(\\featuremtx^{\\top}\\featuremtx\\big)^{-1} \\featuremtx^{\\top} \\Delta\\labelvec = \\pinv{\\featuremtx} \\Delta\\labelvec \\text{,} \\] with the pseudoinverse X. This in turn allows one to quantify the effect of perturbing the training set as \\[ \\normgeneric{\\widetilde{\\weights} - \\widehat{\\weights}}{2} \\leq \\normgeneric{\\pinv{\\featuremtx}}{2}\\, \\normgeneric{\\Delta\\labelvec}{2} \\text{.} \\] When X has full column rank, X2 = 1/(XX) = XX/X2, where XX = / is the condition number of XX. In the setting of Fig.~\\ref{fig_linreg_dict}, X = 1 (the all-ones vector), so X Δy = (1/m) r=1m Δy(r). Fig.~\\ref{fig_linreg_outlier_dict} illustrates such a label-only perturbation of the training set from Fig.~\\ref{fig_linreg_dict}: the single label perturbation Δy(3) = 6 gives w - w = 6/3 = 2, i.e., the outlier pulls the average of the labels upward, from w = 3 to w = 5. The effect of perturbations can also be studied on the level of a specific optimization method. For example, the update of online gradient descent becomes \\[ \\begin{aligned} \\weights^{(\\iteridx+1)} &= \\weights^{(\\iteridx)} + 2\\lrate \\big( \\widetilde{\\truelabel}^{(\\iteridx)} - \\big(\\weights^{(\\iteridx)}\\big)^{\\top} \\widetilde{\\featurevec}^{(\\iteridx)} \\big) \\widetilde{\\featurevec}^{(\\iteridx)} &= \\weights^{(\\iteridx)} + 2\\lrate \\big( \\truelabel^{(\\iteridx)} - \\big(\\weights^{(\\iteridx)}\\big)^{\\top} \\featurevec^{(\\iteridx)} \\big) \\featurevec^{(\\iteridx)} + \\perturbation{\\iteridx} \\text{.} \\end{aligned} \\] Here, t is a perturbation term that depends on the data point perturbation Δ x(t)Δ y(t) and the current model parameters w(t).", "synonyms": [], "see_also": [ "regression", "linmodel", "leastsquares", "erm", "sqerrloss", "gd", "onlineGD", "onlinealgorithm", "ridgeregression", "lasso", "pseudoinverse", "datapoint", "feature", "label", "overfitting", "bayesestimator", "risk", "lms", "stochGD" ], "demo": "https://dictionaryofml.org/terms/linreg.py", "notebook": "https://dictionaryofml.org/terms/linreg.ipynb" }, { "key": "llm", "name": "large language model", "url": "https://dictionaryofml.org/terms/llm.html", "pdf": "https://dictionaryofml.org/terms/llm.pdf", "abstract": "A large language model (LLM) is an artificial neural network, typically with billions of model parameters, that implements a hypothesis map from a prompt to a response. The training of an LLM often proceeds in stages: pretraining uses self-supervised learning, splitting raw text into overlapping fragments that yield data points without manual annotation, and fine-tuning uses a small dataset of manually curated prompt--response pairs. LLM agents extend LLMs beyond text: a program interprets the response, for example as code to run or as a tool to execute, and can append the result to the next prompt, so the LLM chooses actions that the surrounding program carries out.", "description": "Consider a chat-based artificial intelligence system such as ChatGPT, Claude, Gemini, DeepSeek, or Mistral: a user types a question, the prompt, and receives a reply, the response. The system that produces the reply is an LLM: an artificial neural network, typically with billions of model parameters, trained to output text given input text. Each data point of this learning task is a pair of texts, and the trained LLM implements a hypothesis map from the prompt to the response (Fig.~\\ref{fig_llm_dict}). In contrast, a text classifier that maps a document to one of a fixed set of labels is not an LLM: it returns a label, not text. Most LLMs implement h autoregressively: the response is generated one token at a time, each drawn from a probability distribution over the vocabulary that the network computes from the prompt and all tokens generated so far (; ). Alternative constructions exist: masked probabilistic models predict removed tokens and serve text analysis rather than generation , and diffusion language models generate many tokens in parallel . The training of an LLM consists of several stages. One stage typically uses self-supervised learning: a given text is split into overlapping fragments, and each fragment amounts to a data point whose label is one of its tokens and whose features are the remaining tokens. training sets built from text fragments require no manual annotation and are used for pretraining the LLM via empirical risk minimization (; ). Another stage is fine-tuning, which uses a small dataset of manually curated prompt--response pairs . An LLM whose model parameters are reused across many learning tasks is a foundation model . LLM inference, i.e., computing the response for a given prompt, is computationally much cheaper than its training. Still, the inference itself is computationally costly and is often implemented using cloud computing services. Local execution is catching up, however: open-weight LLMs, whose model parameters are published , can be compressed by quantization. Quantization stores each model parameter in a few bits instead of a 16-bit floating-point number . An LLM with billions of model parameters then runs on a laptop or phone (see edge computing). The output of an LLM by itself is nothing but a sequence of letters that constitute the response. An LLM (or coding) agent interprets the response of an LLM as instructions for computations to be executed . For example, the prompt can steer the LLM to produce a response that is valid source code in a programming language such as Python, as in the spelled-out data point of Fig.~\\ref{fig_llm_dict}. The delivered source code can then be executed, and the result can be appended to the next prompt.", "synonyms": [], "see_also": [ "token", "vocabulary", "transformer", "attention", "ann", "wordembedding", "softmax", "nlp", "foundationmodel", "pretraining", "finetuning", "agent", "cloudcomputing", "edgecomputing" ], "demo": "https://dictionaryofml.org/terms/llm.py", "notebook": "https://dictionaryofml.org/terms/llm.ipynb" }, { "key": "logreg", "name": "logistic regression", "url": "https://dictionaryofml.org/terms/logreg.html", "pdf": "https://dictionaryofml.org/terms/logreg.pdf", "abstract": "Logistic regression learns a linear hypothesis map h(x) = w x for a binary classification problem with label y ∈ -1, 1. The sign of h(x) is the predicted label, and the sigmoid function transform σ(h(x)) estimates the probability of the label y = 1. The parameters are learned by empirical risk minimization with the average logistic loss on the training set, which is equivalent to maximum likelihood estimation under the sigmoid function probability model. The resulting objective is convex and smooth, so gradient descent converges to a minimizer whenever one exists; on a linearly separable training set a penalty term restores a unique minimizer. Thresholding the learned hypothesis yields a linear classifier whose decision boundary is a hyperplane.", "description": "An email service must decide, for every incoming message, whether to move it to the spam folder. features extracted from the message — word frequencies, sender information — form its feature vector x ∈ Rd, and the label y ∈ -1, 1 records whether the message is spam. Logistic regression learns, for such a binary classification problem, a linear hypothesis map h(x) = w x (; ). Despite its name, the method solves a classification problem — the regression in the name refers to fitting the real-valued map h, not to a numeric label. The value h(x) is read in two ways. Its sign is the predicted label, which makes the learned h a linear classifier whose decision boundary is the hyperplane w x = 0. Its sigmoid function transform σ(h(x)) is an estimate of the probability of the label y = 1 given the feature vector. The quality of a candidate w is measured by the average logistic loss on the training set D(train) = (x(r), y(r))r=1m of m data points, and empirical risk minimization delivers \\[ \\widehat{\\weights} \\in \\argmin_{\\weights \\in \\reals^{\\featuredim}} f(\\weights) \\text{, with } f(\\weights) \\defeq \\frac{1}{\\samplesize} \\sum_{\\sampleidx=1}^{\\samplesize} \\log\\Big(1 + \\exp\\big(-\\truelabel^{(\\sampleidx)} \\weights^{\\top} \\featurevec^{(\\sampleidx)}\\big)\\Big) \\text{.} \\] Minimizing f is equivalent to maximum likelihood estimation under the model that assigns the label y = 1 with probability σ(w x) . Fig.~\\ref{fig_logreg_dict} shows a learned probability curve for a training set with a single feature. Unlike linear regression with the squared error loss, the minimization admits no closed-form solution, but f is convex and smooth, so gradient descent applies. The iteration is \\[ \\weights^{(\\iteridx+1)} = T\\big(\\weights^{(\\iteridx)}\\big) \\text{, with } T(\\weights) \\defeq \\weights - \\lrate \\nabla f(\\weights) \\text{,} \\] with learning rate η and the gradient f(w). A vector w is a fixed point of the operator T exactly when f(w) = 0, which by convexity of f is exactly when w is a minimizer. For a sufficiently small learning rate, no update increases f, and the iterates converge to a minimizer whenever one exists . On a linearly separable training set no minimizer exists — the norm of the iterates grows without bound while the training error tends to zero — and adding a penalty term to f, one of the three routes that regularization distinguishes, restores a unique minimizer. Applying linear regression with the squared error loss to the binary labels is a near miss: the squared error loss penalizes a prediction wx that is large, positive, and correct, while the logistic loss decreases as this margin grows. For more than two label values, replacing the sigmoid function by the softmax function yields multinomial logistic regression .", "synonyms": [], "see_also": [ "classification", "binclass", "linclass", "decisionboundary", "sigmoid", "logloss", "erm", "gd", "maxlikelihood", "linreg", "softmax", "regularization" ], "demo": "https://dictionaryofml.org/terms/logreg.py", "notebook": "https://dictionaryofml.org/terms/logreg.ipynb" }, { "key": "loss", "name": "loss", "url": "https://dictionaryofml.org/terms/loss.html", "pdf": "https://dictionaryofml.org/terms/loss.pdf", "abstract": "The loss incurred by a hypothesis on a single data point is a scalar that quantifies the error of the hypothesis when evaluated on that data point. The ultimate goal of machine learning methods is to train a model such that the learned hypothesis delivers predictions with minimum loss. A key characteristic of machine learning applications and methods is the accessibility of loss values. In supervised learning, the loss can be evaluated for every hypothesis. In reinforcement learning, the loss is observed only for the action actually taken.", "description": "In temperature forecasting, the quality of a predicted temperature is naturally measured by how far it deviates from the temperature actually measured the next day. The loss formalizes this idea. The loss incurred by a hypothesis h ∈ H on a single data point is a scalar that quantifies the error of h when evaluated on that data point . A smaller loss value indicates a smaller error. In supervised learning, this error is the discrepancy between the prediction h(x) delivered for the features x and the label y of that data point. Some losses (e.g., the squared error loss) take a continuum of values, while others (e.g., the 0/1 loss) are binary, indicating only whether the prediction matches the label. For the temperature forecast, the loss on a given day can be the squared difference between the predicted temperature (the prediction) and the measured one (the label); see Fig.~\\ref{fig_loss_singlepoint_dict}. Loss values are typically nonnegative: many losses are built from a norm or metric and are therefore bounded below by zero. Nonnegativity is not required, however: the logarithmic loss (negative log-likelihood) underlying maximum-likelihood methods can take negative values, since a probability density can exceed one. The loss is also not specific to supervised learning. In unsupervised learning there is no label: a clustering loss measures how far a data point lies from its assigned cluster centroid, and the loss of an autoencoder is the reconstruction error between a data point and its reconstruction. machine learning methods use observed loss values to construct an objective function for model training. The average loss across a training set of data points is the empirical risk that empirical risk minimization minimizes. machine learning applications can be categorized by how the loss values are accessible. In supervised learning, the loss can, in principle, be evaluated for every hypothesis in the hypothesis space, since the true label of each training data point is observed. In reinforcement learning, the situation differs . At each time step, the machine learning method (implemented by an agent) selects an action via the currently used hypothesis. The agent then observes the loss incurred by the action actually taken. The loss that would have been incurred by the other actions is not revealed. This limited accessibility of loss values is known as bandit (or partial) feedback . For example, a route-planning system learns the actual travel time only for the route the driver took, not for any alternative route. Similarly, a self-driving car selects a steering direction, and the incurred loss can be measured from onboard sensors such as collision detectors and lane-departure warnings, but only for the trajectory actually driven.", "synonyms": [ "cost" ], "see_also": [ "lossfunc", "logarithmicloss", "datapoint", "hypothesis", "emprisk", "erm", "norm", "metric", "reinforcementlearning", "supervisedlearning", "unsupervisedlearning" ], "demo": "https://dictionaryofml.org/terms/loss.py", "notebook": "https://dictionaryofml.org/terms/loss.ipynb" }, { "key": "matrix", "name": "matrix", "url": "https://dictionaryofml.org/terms/matrix.html", "pdf": "https://dictionaryofml.org/terms/matrix.pdf", "abstract": "A matrix is a rectangular array of numbers arranged in rows and columns, the two-dimensional special case of an array. The entry Ar,j of a matrix A ∈ Rm × d sits in row r and column j. A prominent example in machine learning is the feature matrix of a dataset, obtained by stacking the feature vectors of m data points row-wise. A matrix represents several distinct mathematical objects: a system of linear equations, such as the normal equations of linear regression; a linear map between two vector spaces, after fixing a basis for each; or a tabular dataset with one row per data point and one column per feature.", "description": "Consider a spreadsheet of daily weather records at one station: each row holds a single day, each column one measurement such as temperature or humidity. Such a tabular dataset holds m data points, each described by d numerical features. The features of the r-th data point are collected in its feature vector x(r) = (x(r)1, …, x(r)d) ∈ Rd, one entry per column of the spreadsheet. Stacking these m feature vectors row-wise, so that the r-th row is (x(r)), produces an m × d rectangular array of numbers. Its entry in row r and column j is the j-th feature of the r-th data point. This two-dimensional numeric array is referred to as the feature matrix X of the dataset; Fig.~\\ref{fig_matrix_featuremtx_dict} carries out the construction on three days of the weather example. More generally, a matrix of size m × d is a two-dimensional array of numbers denoted by \\mA = \\begin{pmatrix} A_{1,1} & A_{1,2} & \\dots & A_{1,\\nrfeatures} A_{2,1} & A_{2,2} & \\dots & A_{2,\\nrfeatures} \\vdots & \\vdots & \\ddots & \\vdots A_{\\samplesize,1} & A_{\\samplesize,2} & \\dots & A_{\\samplesize,\\nrfeatures} \\end{pmatrix} \\in \\reals^{\\samplesize \\times \\nrfeatures}\\text{.} Here, Ar,j denotes the matrix entry in the r-th row and the j-th column. A digital image is a matrix of exactly this kind. Recording how green each pixel is turns a 16 × 16 image into a 16 × 16 matrix whose two indices are pixel positions (Fig.~\\ref{fig_matrix_image_dict}). Matrices represent several distinct mathematical objects , including the following: \\begin{itemize} \\item Systems of linear equations: A matrix collects the coefficients of a system of linear equations, \\begin{pmatrix} A_{1,1} & A_{1,2} A_{2,1} & A_{2,2} \\end{pmatrix} \\begin{pmatrix} x_1 x_2 \\end{pmatrix} =\\begin{pmatrix} b_1 b_2 \\end{pmatrix} \\text{, written compactly as } \\mA \\vx = \\vb \\text{.} One important example is the normal equations XX w = Xy, whose solutions are the model parameters that minimize the training error in linear regression. \\item linear maps: Consider two vector spaces U and V, of dimension d and m, respectively. Fixing a basis u(1), …, u(d) for U and a basis v(1), …, v(m) for V, each matrix A ∈ Rm × d defines a linear map f: U → V (see Fig.~\\ref{fig_matrix_dict}) via \\vu^{(\\featureidx)} \\mapsto \\sum_{\\sampleidx=1}^{\\samplesize} A_{\\sampleidx,\\featureidx} \\vv^{(\\sampleidx)}\\text{.} \\item datasets: As in the opening example, a matrix can represent a dataset with each row holding a single data point and each column a specific feature or label. \\end{itemize} \\hspace*{\\parindent}Written with the feature matrix, the training error that empirical risk minimization-based linear regression minimizes becomes a succinct algebraic expression. Collecting the labels of the training set in y ∈ Rm, the training error of the hypothesis h(w)(x) := wx is (1/\\samplesize)\\,\\normgeneric{\\featuremtx \\weights - \\labelvec}{2}^{2} \\text{,} with gradient (2/m) X(X w - y). Each gradient descent step is therefore one multiplication by X followed by one by X, and equating the gradient to zero leads back to the normal equations from the first example above. Matrices recur across machine learning beyond the feature matrix. The Jacobian matrix and the Hessian collect the first- and second-order partial derivatives of a multivariate function and drive gradient-based training. The covariance matrix records the pairwise covariances of a probability distribution over features. principal component analysis, singular value decomposition, and a random projection factor or compress the feature matrix to lower its dimension. A kernel method replaces explicit feature vectors by a matrix of pairwise inner products, and a tabular Markov decision process stores its transition probabilities as a matrix. A graph is carried by matrices as well: its adjacency matrix records which nodes are joined and with what edge weight, and the Laplacian matrix assembled from those weights is the matrix whose eigenvectors graph clustering uses to split the nodes into clusters .", "synonyms": [], "see_also": [ "linearmap", "dataset", "linmodel", "vector", "featuremtx", "normalequations", "array", "transpose" ], "demo": "https://dictionaryofml.org/terms/matrix.py", "notebook": "https://dictionaryofml.org/terms/matrix.ipynb" }, { "key": "mean", "name": "mean", "url": "https://dictionaryofml.org/terms/mean.html", "pdf": "https://dictionaryofml.org/terms/mean.pdf", "abstract": "The mean of a random vector x is its expectation, the Lebesgue integral of x with respect to its probability distribution. The term also refers to the sample mean, i.e., the average of the data points in a dataset. The two usages are consistent: the sample mean is the mean of the random vector obtained by drawing a data point uniformly from the dataset. Both usages share a further characterization: for a random vector with a finite second moment, the mean is the unique solution of a risk minimization problem under the squared error loss. For the random vector from a dataset, this optimization problem reduces to empirical risk minimization in the simplest regression setting: predicting a numeric label without features, for which the learned hypothesis is the sample mean of the labels.", "description": "A weather station records the daily maximum temperature for a year. Adding the 365 numbers and dividing by 365 returns a single value that summarizes the whole record. This is the mean of the recorded numbers, and the same word names the corresponding quantity for a probability distribution. The mean of a random vector x, which takes on values in a Euclidean space Rd, is its expectation Ex. It is the Lebesgue integral of x with respect to its probability distribution P (e.g., see or ), i.e., \\[ \\expect\\{\\featurevec\\} = \\int_{\\reals^{\\featuredim}} \\featurevec \\, \\mathrm{d}\\probdist(\\featurevec) \\text{.} \\] Here, P is the probability distribution that x induces on Rd, so the integral runs over the values x takes rather than over the underlying sample space. No probability density function need exist for the mean to be defined; when one does, denoted by p, it is Rd x p(x) dx. The term is also used to refer to the sample mean of a finite dataset D = x(1), …, x(m) ∈ Rd. These two usages are consistent. A dataset defines a discrete random variable x(D)=x(I) on the sample space 1, …, m. Here, the index I is chosen uniformly at random, i.e., I=r=1/m for all r=1,…,m. The mean of x(D) is precisely the sample mean \\[ ({1}/{\\samplesize}) \\sum_{\\sampleidx=1}^{\\samplesize} \\featurevec^{(\\sampleidx)} \\text{.} \\] The probability distribution of x(D), which places mass 1/m on each data point, is the empirical distribution of the dataset. Fig.~\\ref{fig_mean_empirical_dict} shows both readings for a real-valued random variable and a dataset of seven numbers, with the empirical distribution drawn as its cumulative distribution function. The two readings are further connected by a shared characterization: for any random vector with a finite second moment, i.e., E x22 < ∞, the mean is the unique solution of the following risk minimization problem, whose objective function is a strictly convex function of the optimization variable c ∈ Rd : \\[ \\expect\\{\\featurevec\\} = \\argmin_{\\vc \\in \\reals^{\\featuredim}} \\expect \\big\\{\\normgeneric{\\featurevec - \\vc}{2}^{2}\\big \\} \\text{.} \\] This risk-minimization characterization applies directly to the simplest machine learning problem, predicting a numeric label without any features. Consider a training set D(train) = y(1), …, y(m) that consists of labels y(r) ∈ R only. A hypothesis is then a single number h ∈ R that serves as the prediction for every data point. empirical risk minimization with the squared error loss searches over candidate hypotheses h' ∈ R and learns \\[ \\learnthypothesis = \\argmin_{\\hypothesis' \\in \\reals} \\frac{1}{\\samplesize} \\sum_{\\sampleidx=1}^{\\samplesize} \\big(\\truelabel^{(\\sampleidx)} - \\hypothesis'\\big)^{2} = \\frac{1}{\\samplesize} \\sum_{\\sampleidx=1}^{\\samplesize} \\truelabel^{(\\sampleidx)} \\text{,} \\] i.e., the learned hypothesis is the sample mean of the labels (see Fig.~\\ref{fig_mean_dict}). This setting coincides with linear regression for data points that carry the constant scalar feature x = 1. Squaring the deviations also makes the mean sensitive to outliers. Moving one of m data points by an amount δ shifts the sample mean by δ / m, so a single corrupted data point can move the mean arbitrarily far. The trimmed mean that discards a fraction γ at each end stays bounded however far at most γ m data points are moved, and the median stays bounded however far up to half of the m data points are moved. Both are therefore preferred when features or labels may be unreliable (; \\texttt{pythondemos/mean.py}).", "synonyms": [ "expectation", "expected value" ], "see_also": [ "rv", "randomvector", "expectation", "samplemean", "probdist", "LebesgueIntegral", "erm", "sqerrloss", "linreg", "median", "trimmedmean" ], "demo": "https://dictionaryofml.org/terms/mean.py", "notebook": "https://dictionaryofml.org/terms/mean.ipynb" }, { "key": "model", "name": "model", "url": "https://dictionaryofml.org/terms/model.html", "pdf": "https://dictionaryofml.org/terms/model.pdf", "abstract": "In machine learning, the word \\emph{model} is used in different ways. Formally, a model is a hypothesis space H: the set of candidate hypothesis maps from which an machine learning method picks (or learns) one. In applied machine learning literature and in software libraries, the word \\emph{model} also refers to the trained predictor, i.e., to the learned hypothesis returned by an machine learning algorithm; the EU AI Act uses this sense in the compound term general-purpose AI model. A different type of mathematical model is a probabilistic model, which specifies a family of probability distributions that characterize a data generation process.", "description": "Consider the problem of predicting tomorrow's maximum daytime temperature from today's morning temperature. machine learning methods learn a hypothesis map h that reads in today's morning temperature and delivers an accurate prediction of tomorrow's maximum temperature. This learned hypothesis is chosen from a (typically large) set of candidate hypothesis maps, i.e., a hypothesis space H. The hypothesis space underlying an machine learning application is often referred to as the model of the method . For example, linear regression and logistic regression use the linear model as their hypothesis space. A decision tree method uses a hypothesis space that consists of hypothesis maps generated by a flow chart. large language model systems use a hypothesis space consisting of non-linear maps represented by an artificial neural network. model training refers to the process of finding a hypothesis in the hypothesis space that yields accurate predictions. Strictly speaking, a trained (or fitted) model represents a learned hypothesis h. In applied machine learning literature and in software libraries, the word \\emph{model} typically refers to a trained model, i.e., to a specific hypothesis map h (; ; ). The EU AI Act uses the word \\emph{model} in the same sense (i.e., a learned hypothesis) in the compound term general-purpose AI model . A different type of mathematical model is a probabilistic model, which specifies a family of probability distributions. Each such probability distribution defines a data generator: data points are realizations of independent and identically distributed RVs with the given probability distribution. Fig.~\\ref{fig_model_dict} illustrates these three uses of the word \\emph{model}: the hypothesis space, the trained model, and the probabilistic model. Synonyms: hypothesis space, hypothesis class, model class.", "synonyms": [], "see_also": [ "hypospace", "hypothesis", "modelparam", "parammodel", "training", "erm", "probmodel", "probdist", "decisiontree", "neuron", "ann", "transformer", "llm", "generalaimodel", "aisystem" ], "demo": "https://dictionaryofml.org/terms/model.py", "notebook": "https://dictionaryofml.org/terms/model.ipynb" }, { "key": "norm", "name": "norm", "url": "https://dictionaryofml.org/terms/norm.html", "pdf": "https://dictionaryofml.org/terms/norm.pdf", "abstract": "A norm on a vector space is a function that assigns each vector a nonnegative number, its length. A norm is definite and homogeneous, and it satisfies the triangle inequality. Every norm defines a metric via the norm of the difference of two vectors, and every inner product induces a norm. The ℓp-norms on Rd include the ℓ1-norm, the Euclidean norm, and the ℓ∞-norm. In machine learning, norms are used to construct loss functions and regularizers.", "description": "Consider a data point that is represented by a feature vector x ∈ Rd, e.g., the hourly temperature recordings of one day at a weather station. Many machine learning methods use a measure for the size of such a vector or for the distance between two of them. A norm provides this measure: a norm · on a vector space V over the field R is a function · : V → R+ that satisfies the following conditions for all u, v ∈ V and β ∈ R, with |β| the absolute value: \\begin{itemize} \\item Definiteness: u = 0 u = 0; \\item Homogeneity: β u = |β| · u; \\item Triangle inequality: u + v ≤ u + v. \\end{itemize} The same conditions define a norm on a vector space over the complex numbers, with |β| the modulus. A function that satisfies homogeneity and the triangle inequality but assigns the value zero to some non-zero vector is a seminorm, not a norm: on R2, the map u |u1| is a seminorm that vanishes on (0,1). Every norm defines a metric by uv := u - v, making V·· a metric space. On an inner product space, the inner product induces a norm via u := uu; the norm of a Hilbert space is of this form. For 1 ≤ p < ∞, the ℓp-norm of a vector x ∈ Rd is defined as xp = ( j=1d |xj|p )1/p. Its instances include the ℓ1-norm and the ℓ2-norm (or Euclidean norm); the limit p → ∞ yields the ℓ∞-norm x∞ = j = 1, …, d |xj|. Not every norm is induced by an inner product. For an induced norm, expanding u vu v yields the parallelogram identity u + v2 + u - v2 = 2 u2 + 2 v2. The ℓ1-norm on R2 violates this identity for u = (1, 0) and v = (0, 1): the left-hand side equals 8, the right-hand side 4. The unit sphere 1 := u ∈ R2 : u = 1 of a norm on R2 consists of the vectors of unit length; Fig.\\ \\ref{fig:norms_unit_balls_dict} shows the unit spheres of these three norms. machine learning methods use norms to construct loss functions and regularizers. In linear regression, the average squared error loss of a linear hypothesis with model parameters w on a training set of size m is the scaled squared Euclidean norm (1/m) y - X w22 of the residual y - X w. ridge regression adds the penalty term α w22 to this training error, while the least absolute shrinkage and selection operator adds α w1 . The choice of the norm determines the learned hypothesis by empirical risk minimization. Consider the two optimization problems \\begin{equation} \\min_{\\weights \\in \\reals^{\\featuredim}} \\normgeneric{\\labelvec - \\featuremtx \\weights}{2} \\quad \\text{ and } \\quad \\min_{\\weights \\in \\reals^{\\featuredim}} \\normgeneric{\\labelvec - \\featuremtx \\weights}{1} \\text{,} \\end{equation} which differ only in the norm of the residual. Squaring the ℓ2-norm does not change the set of minimizers, so minimizing the ℓ2-norm of the residual is equivalent to minimizing the average squared error loss. Fig.~\\ref{fig:norm_mlfit_dict} shows a training set of four data points, one of which is an outlier, and two linear hypotheses h(x) = w x learned from it. Minimizing the squared ℓ2-norm of the residual yields the slope w = 46/30. This solution is pulled toward the outlier because squaring makes the largest residual dominate the training error. Minimizing the ℓ1-norm yields w = 1, which fits the three remaining data points exactly and leaves a single large residual at the outlier. See also: vector space, metric, metric space, Euclidean space, Hilbert space, inner product, Euclidean norm, regularizer, linear regression, least absolute shrinkage and selection operator.", "synonyms": [], "see_also": [], "demo": "https://dictionaryofml.org/terms/norm.py", "notebook": "https://dictionaryofml.org/terms/norm.ipynb" }, { "key": "overfitting", "name": "overfitting", "url": "https://dictionaryofml.org/terms/overfitting.html", "pdf": "https://dictionaryofml.org/terms/overfitting.pdf", "abstract": "Overfitting is the failure mode of a machine learning method that fits its training set too closely: the learned hypothesis incurs a small empirical risk on the training set but a large risk, i.e., the expected loss on data points drawn from the underlying probability distribution. Consequently, the learned hypothesis has a large generalization gap. validation detects overfitting by comparing the training error with the validation error obtained on a validation set. Overfitting typically arises when the hypothesis space is too large for the number of data points in the training set; the Vapnik–Chervonenkis dimension and the Rademacher complexity measure that size. regularization counteracts overfitting by pruning the model, adding a penalty term to the empirical risk, or augmenting the training set.", "description": "An image classifier that memorizes every photograph in its training set but fails to correctly classify unseen images is overfitting. In general, consider a machine learning method that uses empirical risk minimization to learn a hypothesis h ∈ H with minimum empirical risk hD(train) on a given training set D(train). The method overfits D(train) if the empirical risk hD(train) is small while the risk h, i.e., the expected loss on data points drawn, independently of D(train), from the underlying probability distribution, is large. Consequently, an overfitting machine learning method learns a hypothesis with a large generalization gap (; ). A large generalization gap alone does not imply overfitting: a linear model, fitted to a small training set whose labels are noisy and depend nonlinearly on the features, can incur a large training error and a still larger risk, so the generalization gap is large while the training error is not small. The opposite failure is underfitting, where a hypothesis space holding no hypothesis that represents the relation between the features and the label of a data point leaves both quantities large. Fig.~\\ref{fig_overfitting_three_dict} shows why a small training error is on its own no evidence against overfitting. Three data points lie on a straight line. None of the three curves is drawn by hand: each is a hypothesis h delivered by empirical risk minimization with the squared error loss on those same three data points. What separates them is the hypothesis space that empirical risk minimization searched, and the three are nested, H(1) H(2) H(3): H(1) is the constants h(x) = b, H(2) the linear model, whose hypotheses are h(x) = w x + b, and H(3) all continuous functions on R. On H(1), empirical risk minimization has a unique solution: the constant whose value is the average of the three labels. That constant meets the middle data point and misses the other two, and no member of H(1) does better. The hypothesis space is too small here, and the method underfits. On H(2), the solution is unique as well and attains a training error of zero; the hypothesis space matches the relation between feature and label, and the method neither underfits nor overfits. On H(3), every continuous function through the three data points attains a training error of zero, and there are infinitely many of them: empirical risk minimization has infinitely many solutions there, and Fig.~\\ref{fig_overfitting_three_dict} draws one of them. The training error cannot separate that curve from the solution on H(2), the straight line on which the three data points lie, and it cannot choose among the solutions on H(3) either. The curve and the straight line agree on the training set but differ at feature values between those of the training data points, and the risk registers this difference, since data points drawn from the probability distribution also fall in that region. The hypothesis space is too large here, and the method overfits. validation detects overfitting, because the generalization gap that defines it can be estimated: an machine learning method that overfits delivers a small training error but a large validation error on a validation set held out from training. Fig.~\\ref{fig_overfitting_dict} illustrates overfitting for polynomial regression. Each data point carries a single feature x and a noisy label y, drawn from a probability distribution described in the caption. Polynomials of degree r = 0, …, 9 are fitted, using empirical risk minimization with the squared error loss, to training sets of m = 5, 10, and 20 data points; each panel of Fig.~\\ref{fig_overfitting_dict} shows the training error and the validation error for one size. The training error decreases with increasing degree in every panel: the polynomial of degree 9 has 10 model parameters and interpolates the m = 10 data points; for m = 5, every degree from 4 on interpolates; for m = 20, no degree shown interpolates. The validation error, obtained on a shared validation set of 100 data points, is in contrast smallest at a moderate degree and grows by several orders of magnitude for larger degrees: the high-degree polynomials overfit the training set, and the gap between validation error and training error estimates their large generalization gap. Comparing the three training set sizes shows that the smaller the training set, the lower the degree at which the validation error departs from the training error: for m = 5, the validation error jumps by nearly three orders of magnitude already at degree 4, the lowest degree whose 5 model parameters suffice to interpolate the training set; for m = 20, the validation error varies by less than one order of magnitude across all degrees shown and, at degree 9, is smaller than the m = 10 validation error by a factor of more than 100. % Data generated by pythondemos/overfitting.py Overfitting typically occurs when the hypothesis space H is too large relative to the number of data points in the training set (as for H(3) in Fig.~\\ref{fig_overfitting_three_dict}). Counting model parameters is the obvious way to measure that size, but not always the right one: the hypotheses h(w)(x) = 0.5 ( w x ) carry a single model parameter yet form a hypothesis space with infinite Vapnik–Chervonenkis dimension . The count is exactly right, in contrast, for the classifiers h(x) = sign( w0 + w1 x + … + wr xr ) obtained by thresholding the polynomials of maximum degree r in a single feature: their hypothesis space has Vapnik–Chervonenkis dimension r + 1, the number of model parameters and one more than the maximum degree. These thresholded classifiers concern a different case than Fig.~\\ref{fig_overfitting_dict}, which fits polynomials of the same degrees to a regression task. The feature map x (x, x2, …, xr) turns these classifiers into linear classifiers on Rr, whose hypothesis space has Vapnik–Chervonenkis dimension r + 1 ; conversely, any r + 1 data points with distinct features are shattered: the classifiers realize every possible assignment of binary labels to these data points, since a polynomial of degree at most r interpolates arbitrary labels on them. The Vapnik–Chervonenkis dimension of H and its Rademacher complexity measure the size of H without counting model parameters. Both enter the upper bounds on the generalization gap, which grow with them and shrink with the number of data points in the training set (; ). regularization counteracts overfitting in three elementary forms: pruning the model, adding a penalty term to the empirical risk, or augmenting the training set via data augmentation.", "synonyms": [], "see_also": [ "erm", "gengap", "generalization", "regularization", "underfitting", "validation", "vcdim" ], "demo": "https://dictionaryofml.org/terms/overfitting.py", "notebook": "https://dictionaryofml.org/terms/overfitting.ipynb" }, { "key": "probdist", "name": "probability distribution", "url": "https://dictionaryofml.org/terms/probdist.html", "pdf": "https://dictionaryofml.org/terms/probdist.pdf", "abstract": "A machine learning method reads in a dataset and delivers an output, such as a learned hypothesis or the prediction for a given data point. Feeding it a different dataset can deliver a different output. In general, a machine learning method should work well for any dataset that is typical for the machine learning application at hand. One widely used construction of a typical dataset is as realizations of independent and identically distributed RVs with a common probability distribution. The probability distribution can be postulated based on domain knowledge or estimated from a dataset. The probability distribution of a binary random variable is fully specified by a single probability, that of a continuous real-valued random variable may be specified by a probability density function, and in the most general case by a probability measure.", "description": "Consider a machine learning method that reads in a dataset D = z(1), …, z(m) and delivers an output, such as a learned hypothesis h or the prediction for a given data point. Feeding a different dataset into the same method can deliver a different output. In general, the method should work well for any dataset that is typical for the machine learning application at hand. One widely used construction of a typical dataset is as independent and identically distributed RVs z(1), …, z(m) ~ P with a common probability distribution P (see Fig.~\\ref{fig_probdist_method_dict}). The probability distribution P decides which sequences of data points (or datasets) are typical. P is never observed; only the dataset is. In practice, P can be postulated based on domain knowledge or estimated from a dataset, e.g., by its empirical distribution or histogram, or by maximum likelihood within a parametric family. How a probability distribution is specified depends on the values the random variable takes. The probability distribution of a binary random variable y ∈ 0,1 is fully specified by the single probability y = 0, since y=1 = 1-y=0; more generally, an random variable with finitely many values is specified by a probability mass function. The probability distribution of a continuous real-valued random variable x ∈ R might be specified by a probability density function x·, for which x ∈ [a,b] xa |b-a| holds for a short interval [a,b]. In the most general case, an random variable is a map z: Ω → Z from a probability space (Ω, Σ, ·) to its value space Z. The probability distribution of z is the measure that z induces on Z: it assigns to a measurable subset B Z the probability z ∈ B (; ; ). For data points z = (x, y) with a single numeric feature x and a numeric label y, a probability distribution can be visualized in the plane R2. Fig.~\\ref{fig_probdist_picture_dict} shows one as a grayscale over the feature--label plane: the darker a region, the larger its probability density function, and the more typical the data points in it. The depicted probability density function is a weighted sum of two Gaussian pdfs, i.e., a Gaussian mixture model. Paralleling Fig.~\\ref{fig_probdist_method_dict}, three datasets D1, D2, D3, each consisting of m = 40 realizations of independent and identically distributed RVs with this distribution, are shown together with the three hypothesis maps h1, h2, h3 learned from them by the same polynomial regression method. The three learned maps differ most where the probability density function is small: a region that the datasets rarely cover constrains the learned hypotheses only weakly. Synonyms: distribution, law.", "synonyms": [], "see_also": [ "iid", "realization", "rv", "probability", "pdf", "pmf", "probmeasure", "probspace", "empiricaldistribution", "expectation", "generalization" ], "demo": "https://dictionaryofml.org/terms/probdist.py", "notebook": "https://dictionaryofml.org/terms/probdist.ipynb" }, { "key": "projection", "name": "projection", "url": "https://dictionaryofml.org/terms/projection.html", "pdf": "https://dictionaryofml.org/terms/projection.pdf", "abstract": "The projection of a vector onto a closed non-empty subset of a Euclidean space is a point in the subset that is closest to the vector in the Euclidean norm. If the subset is convex, this closest point is unique and the solution of a convex optimization problem. If the subset is a subspace, the projection is a linear map, namely the orthogonal projection onto the subspace. Every projection is idempotent, and a linear map of a Hilbert space into itself is an orthogonal projection if and only if it is idempotent and self-adjoint. In machine learning, projections enforce constraints on model parameters during training: projected gradient descent alternates gradient steps with projections onto the constraint set, e.g., the ℓ1-ball in an equivalent formulation of the least absolute shrinkage and selection operator.", "description": "Consider a non-empty closed set W Rd in the d-dimensional Euclidean space. The projection Ww of a vector w ∈ Rd onto W is \\begin{equation} \\nonumber \\projection{\\paramspace}{\\weights} = \\argmin_{\\weights' \\in \\paramspace} \\normgeneric{\\weights - \\weights'}{2}\\text{,} \\end{equation} a vector in W closest to w in the Euclidean norm. The minimum exists for every non-empty closed set W, but several closest vectors can exist: for the two-point set W = (-1,0), (1,0), both elements are closest to (0,0.7). If W is closed and convex, the closest point in the Euclidean norm is unique . The construction extends beyond the Euclidean space: a non-empty closed convex subset of a Hilbert space contains, for each point of the space, a unique closest point . If W is a subspace, the map w Ww is a linear map: it is the orthogonal projection onto W. Every projection is idempotent: the vector Ww lies in W, so its closest point in W is itself and WWw = Ww. A linear map P of a Hilbert space into itself is the orthogonal projection onto its range if and only if it is idempotent and self-adjoint . For a matrix P ∈ Rd × d acting on the Euclidean space, idempotence reads as P2 = P and self-adjointness amounts to the condition P = P. Idempotence alone is not sufficient: an idempotent linear map that is not self-adjoint is an oblique projection, whose output is not a closest point of its range. For example, the matrix P = pmatrix 1 & 1 0 & 0 pmatrix on R2 satisfies P2 = P but P P. Its range is the horizontal axis, and it maps w = (0,1) to P w = (1,0), while the point of the axis closest to w is the origin (0,0). In machine learning, projections enforce constraints on model parameters during training: linear regression with the constraint w1 ≤ τ is an equivalent formulation of the least absolute shrinkage and selection operator , and this constrained problem can be solved by projected gradient descent. Each iteration takes a gradient step on the training error and then projects the result onto the ℓ1-ball W = w' ∈ Rd : w'1 ≤ τ. Since W is not a subspace, this projection is not a linear map but amounts to solving a convex optimization problem (see Fig.~\\ref{fig_projection_l1_dict}). See also: Euclidean space, vector, minimum, convex, orthogonal projection, projected gradient descent.", "synonyms": [], "see_also": [], "demo": "https://dictionaryofml.org/terms/projection.py", "notebook": "https://dictionaryofml.org/terms/projection.ipynb" }, { "key": "randomforest", "name": "random forest", "url": "https://dictionaryofml.org/terms/randomforest.html", "pdf": "https://dictionaryofml.org/terms/randomforest.pdf", "abstract": "A random forest is an ensemble of decision trees. Each tree is trained on a bootstrap sample of the training set, and each split may choose only among a small random subset of the features. The predictions of the trees are aggregated by a majority vote in classification or by averaging in regression. The two sources of randomness decorrelate the trees, so that averaging reduces the variance of the aggregated prediction. Adding trees does not lead to overfitting: the generalization error converges to a limit that is bounded via the strength of the trees and the correlation between them. A random forest also yields a widely used form of feature importance, read off from the features its trees split on.", "description": "A bank must decide, for each loan application, whether the applicant is creditworthy. A random forest answers by asking many different decision trees and following their majority: it is an ensemble of decision trees, each trained on a bootstrap sample of the training set (as in bootstrap aggregating) and further randomized by allowing each split to choose only among a small random subset of the features . The predictions of the trees are aggregated by a majority vote in classification or by averaging in regression. Fig.~\\ref{fig_randomforest_ml_dict} shows a regression example: each data point is one day at the weather station Krems, its feature the minimum and its label the maximum air temperature of the day. Three decision trees, each trained on its own bootstrap sample of the days, yield three different piecewise constant hypotheses; the random forest averages them. The ensemble idea behind this construction is that averaging many individually unreliable predictions yields a more reliable one, provided the errors of the individual predictions do not align. A deep decision tree is an ideal base learner for this purpose: it can fit the training set closely (small bias), but its prediction varies strongly with the training set (large variance). Averaging attacks precisely this variance: for B identically distributed predictions, each with variance σ2 and with pairwise correlation ρ, the averaged prediction has variance \\rho \\sigma^{2} + \\frac{1-\\rho}{B}\\, \\sigma^{2} \\text{,} which decreases with the number B of trees toward the floor ρ σ2 set by the correlation . Averaging leaves the bias unchanged, since each tree has the same expected prediction. In Fig.~\\ref{fig_randomforest_ml_dict}, the averaged curve is visibly less ragged than each single tree, and its squared-error risk on the depicted days is smaller than the average of the trees' risks. The two sources of randomness serve one purpose: they make the trees less alike (Fig.~\\ref{fig_randomforest_dict}). Averaging reduces the variance of a prediction most when the averaged predictions are weakly correlated, and restricting each split to a random subset of features prevents all trees from reusing the same dominant features . A random forest also reports which features its trees split on most productively — a widely used form of feature importance. A random forest also admits precise guarantees. As trees are added, the generalization error of the forest converges almost surely to a limiting value: a random forest does not suffer from overfitting as the number of trees grows . The limiting error is at most ρ (1-s2)/s2, where the strength s is the expected vote margin (the expected gap between the fraction of trees that vote for the correct label of a data point and the largest fraction that votes for any wrong one) and ρ is the mean correlation between the raw margin functions of two independently drawn trees . The bound is loose, but it names the two levers that the randomization operates on: strong trees and weak correlation. The averaging also yields stability: the learned hypothesis depends only weakly on any single data point of the training set. Consider the subbagged version of any machine learning method with predictions in [0,1]: each base learner is trained on a share p of the m data points, drawn without replacement, and the predictions are averaged over all such subsamples. Then, removing a single data point from the training set changes the prediction at a test point by more than ε for at most a δ-fraction of the removed data points, for every pair (ε, δ) with δ ε2 ≥ 14(m-1) · p1-p . This guarantee holds for every training set and requires nothing of the base learner beyond the bounded predictions — the stability is created by the averaging itself. In the credit-scoring application, the majority vote over hundreds of trees is far less sensitive to a few unusual customer records than any single decision tree, and the feature importance readout indicates which applicant features drive the decisions.", "synonyms": [], "see_also": [ "ensemble", "bagging", "decisiontree", "bootstrap", "featureimportance", "variance", "stability" ], "demo": "https://dictionaryofml.org/terms/randomforest.py", "notebook": "https://dictionaryofml.org/terms/randomforest.ipynb" }, { "key": "regularization", "name": "regularization", "url": "https://dictionaryofml.org/terms/regularization.html", "pdf": "https://dictionaryofml.org/terms/regularization.pdf", "abstract": "Regularization refers to modifications to a machine learning method that improve its generalization. Three routes are distinguished by which component of the method is modified: pruning the hypothesis space, adding a penalty term to the loss function, and augmenting the training set with perturbed copies of its data points. The three routes are often interchangeable. As a case in point, applying data augmentation to an empirical risk minimization-based method is equivalent to adding a penalty term to the training error.", "description": "training a large deep net (such as a large language model) on a fixed training set via plain empirical risk minimization typically leads to overfitting: the learned hypothesis h performs well on the training set but poorly outside it. Regularization refers to modifications to a machine learning method to ensure the learned hypothesis performs nearly as well on unseen data points as on the data points of the training set . For empirical risk minimization-based methods, regularization can be done in three ways: \\begin{enumerate}[label=\\arabic*)] \\item {model pruning:} shrink the hypothesis space H (the set of candidate hypotheses h) to a smaller H'. For a parametric model, the shrinkage can be implemented via constraints on the model parameters (e.g., w1 ∈ [0.4,0.6] on the weight of feature x1 in linear regression). \\item {loss penalization:} add a penalty term to the training error of empirical risk minimization. The penalty estimates how much higher the risk is than the average loss on the training set. \\item {data augmentation:} enlarge the training set D(train) with perturbed copies of its data points (e.g., adding the realizations of independent and identically distributed RVs to the feature vector of each data point). \\end{enumerate} Fig.~\\ref{fig_equiv_dataaug_penal_dict} illustrates the three routes. These routes can yield the same learned hypothesis. As an example, consider data augmentation for linear regression that adds zero-mean perturbations with covariance matrix σ2 I to the feature vector of each data point. Asymptotically in the number of perturbations, the resulting learned hypothesis coincides with the one obtained from ridge regression. ridge regression adds the penalty term σ2 w22 to the training error of linear regression (; ). According to Lagrangian duality, adding this penalty term is equivalent to shrinking the hypothesis space to the set of hypotheses with w22 ≤ C for some constant C (; ). The above regularization techniques are not limited to empirical risk minimization-based methods but can also be applied to other types of machine learning algorithms. For example, in online learning, the Follow-The-Leader method can be regularized by adding a regularizer to the per-round objective function of Follow-The-Leader. The resulting Follow-The-Regularized-Leader method achieves lower regret than plain Follow-The-Leader . Another instance of regularization is the use of a prior distribution over H in Bayesian inference. The prior distribution effectively acts as a regularizer by steering the resulting posterior distribution .", "synonyms": [], "see_also": [ "overfitting", "generalization", "erm", "modelpruning", "penaltyterm", "dataaug", "ridgeregression", "lasso", "ftrl", "priordist" ], "demo": "https://dictionaryofml.org/terms/regularization.py", "notebook": "https://dictionaryofml.org/terms/regularization.ipynb" }, { "key": "rv", "name": "random variable", "url": "https://dictionaryofml.org/terms/rv.html", "pdf": "https://dictionaryofml.org/terms/rv.pdf", "abstract": "A random variable (RV) is a function that maps each outcome of a random experiment to an element of a measurable space. Important types of RVs include binary RVs, discrete RVs, real-valued RVs, random vectors, and random matrices. In machine learning, data points are often interpreted as realizations of independent and identically distributed RVs, whose probability distribution then governs their typical properties.", "description": "An RV is a function that maps the outcomes of a random experiment to elements of a measurable space (; ). Mathematically, an RV is a function x: Ω → X whose domain is the sample space Ω of a probability space and whose co-domain is a measurable space X (see Fig.~\\ref{fig_rv_map}). Different types of RVs include \\begin{itemize} \\item {binary RVs}, which map each outcome to an element of a binary set (e.g., -1,1 or cat, no cat); \\item {discrete RVs}, which take on values in a countable set (which can be finite or countably infinite); \\item {real-valued RVs}, which take on values in the real numbers R; \\item {random vectors}, which map outcomes to the Euclidean space Rd; \\item {random matrices}, which map outcomes to a space of matrices Rm × d. \\end{itemize} In machine learning, the data points of a dataset are commonly modeled as realizations of independent and identically distributed RVs. An observed data point is typically not itself an RV but a single realization: the value x(ω) that the RV x takes for one outcome ω ∈ Ω. The RV is the entire map x, whereas a recorded data point is its image under a single ω. Their shared probability distribution then governs typical properties, such as the expected loss that empirical risk minimization approximates by the average loss on a training set. probability theory relies on measurable spaces to define collections of RVs and study their properties .", "synonyms": [], "see_also": [ "function", "randomexperiment", "samplespace", "probspace", "vector", "euclidspace", "probability", "measurable", "randomvector", "randommatrix", "subgaussianrv" ], "demo": "https://dictionaryofml.org/terms/rv.py", "notebook": "https://dictionaryofml.org/terms/rv.ipynb" }, { "key": "selfsupervisedlearning", "name": "self-supervised learning", "url": "https://dictionaryofml.org/terms/selfsupervisedlearning.html", "pdf": "https://dictionaryofml.org/terms/selfsupervisedlearning.pdf", "abstract": "Self-supervised learning constructs the labels of a training set from the data points themselves. Some features of a data point are withheld and used as its label while the rest remain features, so what results is an ordinary supervised learning problem, solved by empirical risk minimization on data that no annotator has touched. In natural language processing the withheld feature is the next token, which is how a large language model is trained. In computer vision it is a set of pixels, deleted from an image and predicted from the patches left visible.", "description": "An open image database returns thousands of photographs for a keyword such as \\emph{cat}, and a web crawl returns billions of sentences. Neither comes with a label saying what should be predicted. A prediction task can be made from any one of them: hide the last word of a sentence, and the words before it become a question whose answer is already known. Self-supervised learning constructs labels this way, from the data point itself. uses the term for a classifier whose label for one sensory modality is derived from a co-occurring input in another. Some of the features of a data point are withheld and used as the label; the remaining ones stay as the features. The result is an ordinary supervised learning problem, solved by empirical risk minimization, and its training set needs nothing beyond the raw data. A single unlabeled collection yields as many training sets as there are ways of hiding part of a data point (see Fig.~\\ref{fig_selfsup_dict}). In natural language processing a data point is a stretch of text, its features are the tokens v(1), …, v(n) and its label is the token v(n+1) that follows. Every position in every document is one labeled data point, so a corpus of m tokens gives on the order of m of them. This is how a large language model is trained . Masking a token in the middle and predicting it from both sides withholds a different feature of the same data point . In computer vision the withheld features are pixels: an image is split into patches, most of them are deleted, and the method predicts the missing pixel values from those that remain. delete 75% of the patches and still recover recognizable content. A deep net solving either task computes a composition h = s φ, where the feature transformation φ X → Rd sends a data point to a vector of activations and the map s sends that vector to the withheld feature. The withheld part can be predicted only from what φ has kept: a next token cannot be recovered from a vector that has discarded the context, nor a deleted patch from one that has discarded its surroundings. Fitting the composition is therefore what shapes φ, even though φ appears nowhere in the loss. name the two halves: their encoder is φ and sees only the patches left visible, while their decoder is s and rebuilds the pixels from its output. What restricts φ there is the deletion of most of the input, not a narrow layer. In both domains it is φ that is carried over. The map s is dropped, and a small replacement for it is fitted on the output of φ using the few labels the later task does have (see transfer learning, foundation model). Self-supervised learning is therefore one answer to the same shortage that motivates semi-supervised learning: there, unlabeled data points assist a label-scarce problem directly; here, they are turned into a labeled problem of their own first. See also: feature, label, supervised learning, semi-supervised learning, large language model, natural language processing, token, transfer learning, foundation model.", "synonyms": [], "see_also": [], "demo": "https://dictionaryofml.org/terms/selfsupervisedlearning.py", "notebook": "https://dictionaryofml.org/terms/selfsupervisedlearning.ipynb" }, { "key": "sobolevspace", "name": "Sobolev space", "url": "https://dictionaryofml.org/terms/sobolevspace.html", "pdf": "https://dictionaryofml.org/terms/sobolevspace.pdf", "abstract": "A Sobolev space consists of the functions whose derivatives, in a generalized sense, have bounded size. The generalization is needed because the functions arising in machine learning are often not differentiable everywhere: a weak partial derivative is defined by an integration-by-parts identity that moves every derivative onto a smooth test function. For an integer k ≥ 1 and p ∈ [1,∞], the space Wk,p(Ω) collects the functions on an open set Ω Rd whose weak derivatives up to order k exist and have finite ·p. The case W1,∞ matters most for machine learning: a function f belongs to it, with x ∈ Ω f(x)2 ≤ L, exactly when it is Lipschitz continuity continuous with constant L. That bound turns robustness into a number a supplier can declare: a classifier taking the sign of such an f keeps its prediction under every perturbation of Euclidean norm smaller than |f(x)|/L. Sobolev norms also serve as regularizers, and they describe the reproducing kernel Hilbert space of a Mat\\'ern kernel.", "description": "A supplier of a high-risk artificial intelligence system, such as a classifier that screens medical images, must be able to state how much a prediction can change when the feature vector of a data point is perturbed slightly. Sobolev spaces make such a statement precise: they consist of the functions whose derivatives, in a generalized sense, have bounded size . The generalization is needed because the functions of interest are typically not differentiable everywhere. A function gj is a weak partial derivative of f: Ω → R, on an open set Ω Rd, if \\[ \\int_{\\Omega} f(\\featurevec) \\frac{\\partial \\varphi(\\featurevec)}{\\partial \\feature_{\\featureidx}} \\, \\mathrm{d}\\featurevec = - \\int_{\\Omega} g_{\\featureidx}(\\featurevec) \\varphi(\\featurevec) \\, \\mathrm{d}\\featurevec \\] holds for every infinitely differentiable that vanishes outside a bounded subset of Ω . The identity is integration by parts, with all derivatives moved onto . Both sides are Lebesgue integrals, so f and gj need only be integrable on bounded subsets of Ω. For an integer k ≥ 1 and p ∈ [1,∞], the Sobolev space Wk,p(Ω) consists of the functions whose weak derivatives up to order k exist and have finite ·p; the case p=2 gives the Hilbert space Hk(Ω) = Wk,2(Ω). The case that matters most for machine learning is W1,∞, the functions with a bounded weak gradient. A function f belongs to it, with \\[ \\sup_{\\featurevec \\in \\Omega} \\normgeneric{\\nabla f(\\featurevec)}{2} \\leq L \\text{,} \\] precisely when it is Lipschitz continuity continuous with constant L . The bound therefore reads as a robustness guarantee: perturbing a feature vector by δ changes the value of f by at most L δ2. Fig.~\\ref{fig_sobolev_dict} contrasts a function with a bounded weak gradient and one without. The bound is what turns robustness into a number that can be computed and declared. Consider a binary classification hypothesis h(x) = sign(f(x)) with f ∈ W1,∞ and constant L. If |f(x)| = γ, then every perturbation δ with δ2 < γ / L leaves sign(f(x + δ)) unchanged, since the value of f moves by less than γ. The radius γ / L is a certified robustness guarantee against an adversarial attack . For a linear classifier f(x) = w x + b the weak gradient is the constant w, and γ / L is the distance of the feature vector from the decision boundary (see support vector machine). An artificial neural network with rectified linear unit activation functions is the case that makes the weak derivative indispensable: such a hypothesis is piecewise linear, so its classical gradient does not exist along the kinks, while its weak gradient exists and is bounded by the product of the largest singular values of the weight matrices. For a two-layer rectified linear unit artificial neural network on R2, that product gives L ≤ 4.94 while the smallest valid constant is 2.12: the bound holds but is loose. Drawing 60000 perturbations inside the certified radius γ / L produced no change of the prediction (\\texttt{pythondemos/sobolevspace.py}). By contrast, a decision tree is piecewise constant with jumps and has no bounded weak gradient at all: in the same experiment, a stump changes its prediction across a perturbation of Euclidean norm 2 · 10-9, so no certified radius exists for it, however large the margin. A certified radius is also of regulatory interest. Article~15 of the EU AI Act requires a high-risk artificial intelligence system to reach an appropriate level of robustness and to perform consistently, to be as resilient as possible to errors and inconsistencies, and to have its accuracy levels and metrics declared in the instructions for use . A certified radius is such a metric: it states, as a single number, how large a perturbation of a feature vector the prediction provably survives. Sobolev spaces also underlie two regularizers used in machine learning. Penalizing the squared H1 seminorm \\[ \\int_{\\Omega} \\normgeneric{\\nabla f(\\featurevec)}{2}^{2} \\, \\mathrm{d}\\featurevec \\] expresses the smoothness assumption; its counterpart for a function defined on the nodes of a graph replaces the gradient by differences across edges, which is the form used by generalized total variation and generalized total variation minimization . Penalizing the gradient in L1 instead gives total variation regularization, which allows jumps: as a transition of width ε sharpens, its squared H1 seminorm grows without bound, while its total variation stays at the height of the limiting jump (\\texttt{pythondemos/sobolevspace.py}). This is why total variation regularization is used in image denoising . kernels connect to Sobolev spaces as well. For a Mat\\'ern kernel with smoothness parameter ν > 0 on a domain Ω Rd with Lipschitz boundary, and for s := ν + d/2 an integer, the reproducing kernel Hilbert space of the kernel is Ws,2(Ω) as a set of functions, and the two norms are equivalent: there are c1, c2 > 0 with c1 fWs,2(Ω) ≤ fHk ≤ c2 fWs,2(Ω) for every f in the reproducing kernel Hilbert space (; ; ). The penalty term of the corresponding regularized empirical risk minimization is therefore a Sobolev norm, and the smoothness it enforces is again weak: the functions of this reproducing kernel Hilbert space have weak derivatives up to order s, while they are guaranteed to be differentiable in the classical sense only up to order ν .", "synonyms": [], "see_also": [ "Lipschitz", "robustness", "smoothnessassumption", "gtv", "gtvmin", "rkhs", "hilbertspace", "differentiable", "LebesgueIntegral" ], "demo": "https://dictionaryofml.org/terms/sobolevspace.py", "notebook": "https://dictionaryofml.org/terms/sobolevspace.ipynb" }, { "key": "stochGD", "name": "stochastic gradient descent", "url": "https://dictionaryofml.org/terms/stochGD.html", "pdf": "https://dictionaryofml.org/terms/stochGD.pdf", "abstract": "Stochastic gradient descent (SGD) is a variant of gradient descent in which the gradient of the objective function is replaced by a computationally cheaper stochastic approximation. Its main application in machine learning is empirical risk minimization on a training set that is large or stored in a distributed database: the gradient of the empirical risk requires an average over the entire training set. SGD approximates this average by the average over a randomly drawn batch of data points. The batch size trades the computational cost of a single update against the accuracy of the gradient approximation.", "description": "Consider an artificial intelligence system that uses logistic regression to learn an image classifier. logistic regression amounts to finding the minimum of a differentiable convex objective function, the empirical risk \\[ f(\\weights) \\defeq \\frac{1}{\\samplesize} \\sum_{\\sampleidx=1}^{\\samplesize} \\lossfunc{\\datapoint^{(\\sampleidx)}}{\\weights} \\text{.} \\] Here, z(r)w denotes the loss incurred on the data point z(r) of a training set D(train) = z(1), …, z(m) by the hypothesis with model parameters w, and m is the number of data points in the training set. gradient descent minimizes a differentiable convex function by taking a sufficient number of gradient steps, \\[ \\weights^{(\\iteridx+1)} = \\weights^{(\\iteridx)} - \\lrate^{(\\iteridx)} \\nabla f(\\weights^{(\\iteridx)}) \\text{,} \\] with a step size η(t) > 0. The gradient of the above objective function is f(w) = 1m r=1m w z(r)w, an average with one term per data point, so its exact evaluation requires a pass over the entire training set. SGD instead uses the estimate \\[ g(\\weights) \\defeq \\frac{1}{\\batchsize} \\sum_{\\sampleidx \\in \\batch} \\nabla_{\\weights} \\lossfunc{\\datapoint^{(\\sampleidx)}}{\\weights} \\text{,} \\] the same average over a batch B 1, …, m of B = |B| indices chosen uniformly at random (see Fig.~\\ref{fig_sgd_mlpicture_dict}). Each SGD iteration generates a new batch B(t) and performs the gradient descent update with the estimate in place of the gradient, \\[ \\weights^{(\\iteridx+1)} = \\weights^{(\\iteridx)} - \\lrate^{(\\iteridx)} \\frac{1}{\\batchsize} \\sum_{\\sampleidx \\in \\batch^{(\\iteridx)}} \\nabla_{\\weights} \\lossfunc{\\datapoint^{(\\sampleidx)}}{\\weights^{(\\iteridx)}} \\text{,} \\] with a step size η(t) > 0; only the gradient estimate differs from gradient descent (see Fig.~\\ref{fig_sgd_approx_dict}). The update is a random instance of the gradient descent operator: for a uniformly drawn batch, its expectation is the gradient descent step w w - η(t) f(w), whose fixed points are the points satisfying the zero-gradient condition f(w) = 0. Conditions under which such random iterations converge go back to stochastic approximation : the step sizes must diminish such that t=1∞ η(t) = ∞ while t=1∞ (η(t))2 < ∞ . The batch size B is an important parameter of SGD: B = m recovers gradient descent, while B = 1 updates with the gradient of a single data point and yields the noisiest estimate. SGD with 1 < B < m is mini-batch SGD . Each update costs B gradient evaluations instead of m, which is what makes empirical risk minimization on large training sets feasible; the price is noise in the update, whose variance shrinks as the batch grows . See also: gradient descent, gradient-based method, gradient, empirical risk minimization, batch, step size, objective function, zero-gradient condition, online gradient descent.", "synonyms": [], "see_also": [], "demo": "https://dictionaryofml.org/terms/stochGD.py", "notebook": "https://dictionaryofml.org/terms/stochGD.ipynb" }, { "key": "svd", "name": "singular value decomposition", "url": "https://dictionaryofml.org/terms/svd.html", "pdf": "https://dictionaryofml.org/terms/svd.pdf", "abstract": "The singular value decomposition (SVD) is a factorization of a matrix A ∈ Rm × d of the form A = V Λ U with orthonormal matrices V and U. The matrix Λ is nonzero only along its main diagonal, whose entries are nonnegative and referred to as singular values. In contrast to an eigenvalue decomposition, which only exists for a square diagonalizable matrix, an SVD exists for every matrix. Truncating the SVD after the k largest singular values yields the best approximation by a matrix of rank k, which underlies dimensionality reduction. The SVD delivers the pseudoinverse, the spectral norm, and the condition number of a matrix.", "description": "The SVD of a matrix A ∈ Rm × d is a factorization of the form \\[ \\mA = \\mV {\\bm \\Lambda} \\mU^{\\top} \\] with orthogonal matrices \\[ \\mV = \\big(\\vv^{(1)},\\,\\ldots,\\,\\vv^{(\\samplesize)}\\big) \\in \\reals^{\\samplesize \\times \\samplesize}, \\qquad \\mU = \\big( \\vu^{(1)},\\,\\ldots,\\,\\vu^{(\\featuredim)} \\big) \\in \\reals^{\\featuredim \\times \\featuredim} \\text{.} \\] The matrix Λ ∈ Rm × d is only nonzero along the main diagonal, whose entries Λj,j = j ≥ 0 are referred to as singular values and ordered as 1 ≥ 2 ≥ … ≥ 0. Multiplying the factorization by u(j) gives A u(j) = j v(j): A maps the orthonormal vectors u(j) to the scaled orthogonal vectors j v(j) (see Fig.\\ \\ref{fig_svd_dict}). In contrast to an eigenvalue decomposition, which only exists for square diagonalizable matrices, an SVD exists for every matrix . The existence of the SVD (and its computation) builds on the spectral decomposition of the symmetric positive semi-definite matrix AA: it has an orthonormal basis of eigenvectors u(j) with eigenvalues j2 ≥ 0, the squares of the singular values, and setting the left singular vectors v(j) := A u(j) / j (for j > 0) recovers A = V Λ U. If A is itself symmetric and positive semi-definite, its SVD coincides with its eigenvalue decomposition. The largest singular value 1 equals the spectral norm A2, and the ratio of the largest to the smallest nonzero singular value is the condition number A. Retaining only the k largest singular values yields the truncated SVD \\[ \\widehat{\\mA} \\defeq \\sum_{\\featureidx=1}^{k} \\eigval{\\featureidx} \\vv^{(\\featureidx)} \\big(\\vu^{(\\featureidx)}\\big)^{\\top} \\text{,} \\] a matrix of rank at most k. Among all matrices B of rank at most k, the truncation A minimizes the approximation error A - BF . The SVD can be used to compress data. A grayscale image, stored as a matrix of m × d pixel intensities, is compressed by keeping only the k largest singular values: the truncation requires only k (m + d + 1) numbers (\\texttt{pythondemos/svd.py}). The SVD can be used to implement principal component analysis. For a centered feature matrix X ∈ Rm × d, the right singular vectors u(j) are the eigenvectors of the sample covariance matrix 1m X X, whose eigenvalues are j2/m. principal component analysis projects each data point onto the leading u(1), …, u(k), the k directions of largest variance, giving a linear dimensionality reduction. The SVD can also be used to analyze and to implement linear regression methods. Given a feature matrix X ∈ Rm × d, whose r-th row is the feature vector of the r-th data point, and a label vector y ∈ Rm, linear regression minimizes the squared error X w - y22 over the model parameters w, the empirical risk minimization objective for the squared loss (the least squares problem). Let k be the number of nonzero singular values (the rank of X). A component of w along a right singular vector u(j) with j > k leaves X w unchanged, so expand w = j=1k w(j) u(j) in the leading right singular vectors. Using X u(j) = jv(j) and the orthonormality of the left singular vectors v(j), \\[ \\begin{aligned} \\normgeneric{\\featuremtx \\weights - \\labelvec}{2}^{2} &= \\normgeneric{\\textstyle\\sum_{\\featureidx=1}^{k} \\eigval{\\featureidx}\\widetilde{w}^{(\\featureidx)} \\vv^{(\\featureidx)} - \\labelvec}{2}^{2} &= \\sum_{\\featureidx=1}^{k} \\big(\\eigval{\\featureidx}\\widetilde{w}^{(\\featureidx)} - (\\vv^{(\\featureidx)})^{\\top}\\labelvec\\big)^{2} + \\sum_{\\featureidx=k+1}^{\\samplesize} \\big((\\vv^{(\\featureidx)})^{\\top}\\labelvec\\big)^{2} \\text{.} \\end{aligned} \\] Only the components (v(j))y of y along the left singular vectors enter; the second sum is the constant, irreducible error from the part of y outside the column space of X. The first sum is minimized termwise at w(j) = (v(j))y / j for j ≤ k, the minimum-Euclidean norm (pseudoinverse) solution.", "synonyms": [], "see_also": [ "matrix", "orthogonal", "orthonormal", "singularvalue", "evd", "spectralnorm", "condnr", "rank", "pseudoinverse", "pca", "dimred", "linreg" ], "demo": "https://dictionaryofml.org/terms/svd.py", "notebook": "https://dictionaryofml.org/terms/svd.ipynb" }, { "key": "svm", "name": "support vector machine", "url": "https://dictionaryofml.org/terms/svm.html", "pdf": "https://dictionaryofml.org/terms/svm.pdf", "abstract": "The support vector machine (SVM) is a binary classification method that learns a linear classifier by regularized empirical risk minimization, combining the hinge loss with a squared-norm penalty term. For a linearly separable training set and sufficiently weak regularization, the solution is the maximum-margin separating hyperplane. This hyperplane is fully determined by the feature vectors closest to it, which are called support vectors. The kernel SVM is obtained by combining the basic SVM with a feature transformation derived from a kernel function.", "description": "The SVM is a binary classification method for data points with feature space X=Rd. In its simplest form, the SVM learns a linear classifier with decision boundary \\{ \\featurevec \\in \\reals^{\\nrfeatures} : \\weights^{\\top} \\featurevec + \\offset = 0 \\}. The linear classifier is parameterized by a nonzero vector w ∈ Rd 0 and an offset b ∈ R. The vector w is the normal vector to the decision boundary and the offset b shifts the decision boundary away from the origin. For a data point with feature vector x and label y ∈ -1, +1, the SVM delivers the prediction y = sign(w x + b). SVMs have been applied to text classification, where each document is first represented by a numeric feature vector of word frequencies and then categorized by topic. Another application is handwritten digit recognition, where digit images are classified from pixel-level features . The SVM can be formulated as an instance of regularized empirical risk minimization for learning a linear classifier from a training set that consists of m data points. The model parameters (w, b) of the linear classifier are obtained as \\begin{equation} (\\widehat{\\weights}, \\widehat{\\offset}) = \\argmin_{\\weights \\in \\reals^{\\nrfeatures},\\, \\offset \\in \\reals} \\underbrace{\\frac{1}{\\samplesize} \\sum_{\\sampleidx=1}^{\\samplesize} \\max\\{0,\\, 1 - \\truelabel^{(\\sampleidx)} (\\weights^{\\top} \\featurevec^{(\\sampleidx)} + \\offset)\\} + \\regparam \\normgeneric{\\weights}{2}^{2}}_{\\defeq f(\\weights, \\offset)}\\text{.} \\end{equation} The SVM objective function consists of the average hinge loss on the training set plus a penalty term α w22 with regularization parameter α > 0 (; ). The penalty term in \\eqref{eq:svm_rerm} is the scaled squared Euclidean norm α w22, the same penalty term as in ridge regression. It does not involve the model parameter b, which is therefore left unpenalized. In linear regression, the penalty term can be interpreted as an estimate of how much higher the loss is on data points outside the training set than on it (; see linear regression). This interpretation rests on the quadratic structure of the squared error loss and does not carry over to the hinge loss. For the SVM, the penalty term instead controls the generalization gap: the hinge loss is Lipschitz continuity continuous in w, with a constant given by the Euclidean norm of the feature vector. Adding the penalty term α w22 to the average hinge loss has two effects: First, it makes the overall objective function in \\eqref{eq:svm_rerm} strongly convex in w, so that its solution w is unique. Second, it controls how much that solution varies if the data points in the training set change. If the feature vectors in the training set have bounded Euclidean norm, the expectation of the difference between the risk and the empirical hinge loss of the learned linear classifier decreases in proportion to 1/(α m) (). The penalty term also has a geometric role: a small norm w2 corresponds to a large margin 1/w2 of the learned linear classifier, as discussed below. Like logistic regression and linear regression, the SVM learns the model parameters (w, b) of a linear model. In contrast to logistic regression and linear regression, the SVM objective function f(w, b) is non-smooth because of the hinge loss. However, since the SVM objective function is convex, the optimization problem \\eqref{eq:svm_rerm} can be solved by convex optimization methods. One such method is subgradient descent, which is obtained from gradient descent by replacing the gradient of the objective function with a subgradient (; ; ). Starting from an initial choice w(0), b(0) of the model parameters, subgradient descent repeatedly applies the update \\[ \\big(\\weights^{(\\iteridx+1)}, \\offset^{(\\iteridx+1)}\\big) = \\big(\\weights^{(\\iteridx)}, \\offset^{(\\iteridx)}\\big) - \\lrate^{(\\iteridx)} \\vg^{(\\iteridx)} \\text{, for } \\iteridx = 0,1,\\ldots \\text{,} \\] with a step size η(t) > 0 and a subgradient g(t) ∈ f(w(t), b(t)) of the SVM objective function \\eqref{eq:svm_rerm}. Inserting the SVM objective function \\eqref{eq:svm_rerm} into the generic update yields the explicit update \\[ \\begin{aligned} \\weights^{(\\iteridx+1)} &= \\big(1 - 2 \\lrate^{(\\iteridx)} \\regparam\\big) \\weights^{(\\iteridx)} + \\frac{\\lrate^{(\\iteridx)}}{\\samplesize} \\sum_{\\sampleidx=1}^{\\samplesize} \\expcoeff^{(\\iteridx)}_{\\sampleidx} \\truelabel^{(\\sampleidx)} \\featurevec^{(\\sampleidx)} \\text{,} \\offset^{(\\iteridx+1)} &= \\offset^{(\\iteridx)} + \\frac{\\lrate^{(\\iteridx)}}{\\samplesize} \\sum_{\\sampleidx=1}^{\\samplesize} \\expcoeff^{(\\iteridx)}_{\\sampleidx} \\truelabel^{(\\sampleidx)} \\text{,} \\end{aligned} \\] with expansion coefficients β(t)r ∈ [0, 1], for r = 1, …, m. The coefficient β(t)r is nonzero only for data points that satisfy y(r) ( (w(t)) x(r) + b(t) ) ≤ 1, i.e., data points on which the hinge loss is nonzero or non-differentiable at the current model parameters. Each iteration scales w(t) by the factor 1 - 2 η(t) α (the effect of the penalty term) and adds corrections y(r) x(r) only from these data points. In particular, for the initialization w(0) = 0, every iterate w(t) is a weighted sum r=1m βr y(r) x(r) of the feature vectors in the training set. The solution w of \\eqref{eq:svm_rerm} admits the same expansion, with coefficients that are nonzero only for the data points satisfying the same condition y(r) ( w x(r) + b ) ≤ 1 (see the discussion of support vectors below). For a suitably diminishing step size, e.g., η(t) = 1/(t+1), the iterates (w(t), b(t)) converge to a solution of \\eqref{eq:svm_rerm} . The solution of \\eqref{eq:svm_rerm} has a clear geometric meaning when the training set is linearly separable, i.e., there is some choice for (w, b) such that sign(w x(r) + b) y(r)=1 for r=1, …, m. For a sufficiently small α, the solution of \\eqref{eq:svm_rerm} determines the separating hyperplane that is farthest from the feature vectors in the training set (see Fig.~\\ref{fig_svm_margin_dict}). The distance between this hyperplane and the nearest feature vectors, which are precisely the support vectors discussed below, is referred to as the margin and is given by 1/w2 (; ; ). The margin of the learned hyperplane measures robustness against perturbations of the features of a data point. Perturbing a feature vector x by a vector δ changes the score wx + b by wδ, whose magnitude is at most w2 δ2 by the Cauchy-Schwarz inequality. A data point therefore stays correctly classified under every feature perturbation with δ2 < γ, as illustrated by the dashed circle around a support vector in Fig.~\\ref{fig_svm_margin_dict}. data points whose feature vectors lie farther from the decision boundary tolerate even larger perturbations: the tolerated radius equals the distance of the feature vector from the decision boundary, which is at least γ and equals γ precisely for the support vectors. Maximizing the margin maximizes this worst-case tolerated radius, so the maximum-margin classifier is the separating classifier that is robust against the largest feature perturbations . While the above discussion only applies when the SVM training set is linearly separable, every solution of the SVM problem \\eqref{eq:svm_rerm} has a structure that holds in general. In particular, \\widehat{\\weights} = \\sum_{\\sampleidx=1}^{\\samplesize} \\expcoeff_{\\sampleidx} \\truelabel^{(\\sampleidx)} \\featurevec^{(\\sampleidx)} with expansion coefficients 0 ≤ βr ≤ 1/(2 α m) that are nonzero only for the feature vectors with \\truelabel^{(\\sampleidx)} (\\widehat{\\weights}^{\\top} \\featurevec^{(\\sampleidx)} + \\widehat{\\offset}) \\leq 1. These feature vectors are referred to as support vectors since they entirely determine the SVM solution (w, b) . Applying a perturbation to any feature vector which is not a support vector leaves w, b unchanged, provided the perturbation is small enough to preserve the strict inequality y (w x + b) > 1 (see Fig.~\\ref{fig_svm_support_dict}). An overly large regularization parameter α makes the SVM underfit: the above expansion of w implies w2 ≤ ρ/(2α), with ρ the largest Euclidean norm of a feature vector in the training set, so w shrinks toward 0. Consider a training set with unequal class sizes. For sufficiently large α, the predictions approach the constant majority-class prediction sign(b) and every data point of the minority class is a misclassified support vector. The training set of Fig.~\\ref{fig_svm_support_dict} also has unequal class sizes, four data points labeled y = +1 and three labeled y = -1, but with the value α = 1/14 used there, the SVM misclassifies one data point which can be considered an outlier. Increasing the value of α to 50 for that training set shrinks w2, makes every prediction on the training set equal to +1, and misclassifies all three data points of the minority class. The basic SVM \\eqref{eq:svm_rerm} learns a linear classifier on the feature space X = Rd. It can be generalized to a classification method for data points whose feature vectors lie in an arbitrary feature space X by first applying a feature transformation φ: X → X' with X' = Rd and then learning a linear classifier on the transformed feature vectors. By choosing φ (and d) appropriately, any given training set can be made linearly separable in X' (; ). A principled construction of a feature transformation φ is via a kernel k: X × X → R. In this construction, the transformed feature space is, in general, not Rd but a Hilbert space H, which can be infinite-dimensional (see kernel method). The resulting method is then referred to as a kernel SVM .", "synonyms": [ "maximum-margin classifier" ], "see_also": [ "binclass", "linmodel", "classifier", "hingeloss", "margin", "hyperplane", "decisionboundary", "robustness", "ridgeregression", "lasso", "kernel", "kernelmethod", "sgd" ], "demo": "https://dictionaryofml.org/terms/svm.py", "notebook": "https://dictionaryofml.org/terms/svm.ipynb" }, { "key": "tabulardata", "name": "tabular data", "url": "https://dictionaryofml.org/terms/tabulardata.html", "pdf": "https://dictionaryofml.org/terms/tabulardata.pdf", "abstract": "Tabular data consist of data points that share a common, fixed set of attributes. The attributes serve as the features or as the label of a data point. The data points constitute the rows, or records, of a table, and the attributes are its columns. Every row has one cell per column, and the cells within one column hold values of the same attribute. The fixed attribute set separates tabular data from other types of data such as text or networks.", "description": "Tabular data consist of data points that share a common, fixed set of attributes . As the name suggests, tabular data can be stored as a table: the rows of the table are the data points, and the columns of the table are the attributes (Fig.~\\ref{fig_tabulardata_dict}). In database theory, the relational model formalizes such a table as a relation. A relation is a set of tuples of attribute values. The relational model also provides the operations, such as selecting rows or joining tables, that database systems use to manipulate tabular data . The weather measurements at a Finnish Meteorological Institute weather station are typically represented as a table: one row for each day, with columns for the morning minimum temperature, the precipitation, and the maximum daytime temperature . The attributes serve as the features or as the label of a data point. In Fig.~\\ref{fig_tabulardata_dict}, a machine learning method forecasts the maximum daytime temperature of a day from its morning minimum temperature: it reads the minimum-temperature column as the feature and the maximum-temperature column as the label, while the precipitation column is left unused. The distinct characteristic of tabular data is the rigid shape of a table. Every row contains the same number of cells, one per column, and the cells within one column hold values of the same attribute of the data points. The value range for each cell must be clearly defined. Often the value range includes a special value (such as ``N/A'') that indicates an empty cell . An empty cell is how missing data arises in tabular data: an attribute that was not recorded for a data point leaves its cell empty, while the table keeps its shape. The fixed attribute set is what separates tabular data from other data types. A text is a sequence of tokens whose length varies from data point to data point, and a graph varies in its node and edge sets. For such data points there is often no natural choice for a fixed list of attributes. Instead, feature learning methods are developed that map such data points to feature vectors (or embeddings) (; ).", "synonyms": [], "see_also": [ "data", "datapoint", "dataset", "feature", "label", "featuremtx", "relationalmodel", "missingdata", "gbdt", "featlearn" ], "demo": "https://dictionaryofml.org/terms/tabulardata.py", "notebook": "https://dictionaryofml.org/terms/tabulardata.ipynb" }, { "key": "testset", "name": "test set", "url": "https://dictionaryofml.org/terms/testset.html", "pdf": "https://dictionaryofml.org/terms/testset.pdf", "abstract": "A test set is a dataset of data points used neither for training a model nor for choosing between candidate models based on a validation set. For a hypothesis learned and selected without reference to the test set, the average loss on it estimates the risk under the independent and identically distributed assumption. This distinguishes it from the validation set, whose average loss flows back into the choice of the hypothesis and is therefore optimistic for the chosen candidate. The estimate stays valid only as long as no design choice depends on the test set; repeated evaluation with selection turns it into a validation set, and overlap with the training set constitutes test set contamination.", "description": "A straight line and a degree-two polynomial have both been fitted to historic days of weather recordings, and the line was selected because it incurred the smaller average loss on days held back from the fit (model selection via a validation set). Quoting that same average as the line's expected error on future days would understate the error: the line was selected for scoring well on exactly those held-back days. The days reserved to answer the final question — how large a loss to expect on a new day — form the test set. A test set D(test) is a dataset of data points used neither for training a model, e.g., via empirical risk minimization on the training set D(train), nor for choosing between candidate models based on a validation set D(val) (Fig.~\\ref{fig_testset_picture_dict}). For a hypothesis h that was learned and selected without reference to D(test), the test error is the average loss \\[ \\frac{1}{|\\testset|} \\sum_{\\datapoint \\in \\testset} \\lossfunc{\\datapoint}{\\learnthypothesis} \\text{.} \\] It estimates the risk of h under the independent and identically distributed assumption . The distinction from the validation set is the direction of information flow: the validation error flows back into the choice of h, so it is an optimistic estimate for the chosen candidate; the test error is computed once, after all choices are frozen, and flows back into nothing. The estimate stays valid only as long as the test set is used this way. Evaluating many candidate models on D(test) and keeping the best turns the test set into a validation set, and its error into a validation error; data points of D(test) that enter the training set constitute test set contamination. In an ML competition, the labels of the test set are therefore withheld from the participants: no design choice of theirs can then depend on the test data, and the reported error keeps its meaning as a risk estimate.", "synonyms": [], "see_also": [ "trainset", "valset", "modelsel", "risk", "generalization", "testsetcontamination", "dataleakage", "erm" ] }, { "key": "trainset", "name": "training set", "url": "https://dictionaryofml.org/terms/trainset.html", "pdf": "https://dictionaryofml.org/terms/trainset.pdf", "abstract": "The ultimate goal of machine learning is to learn a hypothesis that accurately predicts the label of any data point based on its features. A training set is a dataset used to compare the performance of candidate hypotheses: it consists of data points for which the loss incurred by each candidate can be evaluated. model training methods then pick the hypothesis that performs best on the training set. For example, empirical risk minimization learns the hypothesis minimizing the average loss on the training set, the empirical risk; the minimum value itself is the training error. Since the learned hypothesis is picked to perform well on the training set, a small training error can be misleading and result in overfitting. The validation error on a held-back validation set probes whether the learned hypothesis also predicts well outside the training set.", "description": "The ultimate goal of a machine learning method is to learn a hypothesis (or to train a model) that incurs a small prediction error for any data point. The prediction error is measured by some loss function, and the goal is formalized probabilistically as a small risk: the expected loss under the probability distribution from which the data points are drawn (independent and identically distributed assumption). The method must therefore compare the candidate hypotheses from its hypothesis space on data points for which the loss can be evaluated. For example, in regression or classification, the loss can only be computed for data points with a known label. These data points form a dataset referred to as the training set, D(train) = z(r)r=1m. For example, empirical risk minimization minimizes the average loss on the training set, the empirical risk, to learn a hypothesis \\[ \\learnthypothesis \\in \\argmin_{\\hypothesis \\in \\hypospace} \\frac{1}{\\samplesize} \\sum_{\\sampleidx=1}^{\\samplesize} \\lossfunc{\\datapoint^{(\\sampleidx)}}{\\hypothesis} \\text{.} \\] The empirical risk of the learned hypothesis is the training error. More generally, the training set is the input of the map A that defines a training method: h = A(D(train)), and training is to compute this map. Consider three days of weather recordings: for each day, the morning minimum temperature is the feature and the maximum daytime temperature is the label . Fig.~\\ref{fig_trainset_picture_dict} shows a training set of these three days along with two hypotheses learned from it: the straight line h(1) delivered by empirical risk minimization on the linear model, and a degree-two polynomial h(2) that passes through all three data points and therefore attains a training error of zero. Which data points are contained in D(train) decides what is learned. The choice of the training set is therefore crucial for trustworthy artificial intelligence: a training set that underrepresents a group of data points yields a learned hypothesis that incurs a larger loss on that group, a source of unfairness (see fairness) (; ; ). Since h is chosen to make this particular average small, the training error understates the loss that the same hypothesis incurs on data points outside D(train) . The training error is therefore a poor estimate of the loss incurred outside the training set. Fig.~\\ref{fig_trainset_picture_dict} shows how misleading it can be: the polynomial h(2) has a smaller training error than the straight line, yet the two curves differ sharply away from the training data points, which is where the loss on new data points is incurred (see overfitting). A second subset of data points, the validation set D(val), is therefore held back from model training (Fig.~\\ref{fig_trainset_dict}): the average loss on D(val), the validation error, probes the learned hypothesis on data points it was not trained on. Comparing the two errors diagnoses the machine learning method: if both are small, the learned hypothesis also predicts well outside D(train); a small training error paired with a validation error far above it means the hypothesis space is too large compared to the number of data points in D(train) (; ). See also: training, dataset, validation set, test set, data point, empirical risk minimization, hypothesis, loss, training error, validation error, hypothesis space, overfitting.", "synonyms": [], "see_also": [], "demo": "https://dictionaryofml.org/terms/trainset.py", "notebook": "https://dictionaryofml.org/terms/trainset.ipynb" }, { "key": "transformer", "name": "transformer", "url": "https://dictionaryofml.org/terms/transformer.html", "pdf": "https://dictionaryofml.org/terms/transformer.pdf", "abstract": "Many machine learning applications involve data points that consist of smaller units, so-called tokens, such as the words of a text or the patches of an image. A transformer is an artificial neural network that is built by composing layers, each of which transforms the features of such a token-based data point. The feature vectors of the tokens are stacked into a matrix; each layer maps this matrix to one of the same shape, so that layers can be composed freely into a deep artificial neural network. The layers alternate between token mixing, which combines information across tokens, and a position-wise multilayer perceptron, which transforms each token separately.", "description": "Many machine learning applications involve data points that consist of smaller units, so-called tokens: a text consists of words, and an image consists of patches of pixels (see Fig.~\\ref{fig_transformer_tokens_dict}). A transformer is an artificial neural network that is built by composing layers, each of which transforms the features of such a token-based data point . The feature vectors (or embeddings) x(1), …, x(n) ∈ Rd of the n tokens are stacked into a matrix X = (x(1), …, x(n)) ∈ Rn × d. Each layer of the original transformer maps such a matrix to a transformed matrix of the same shape, so that layers can be composed freely into a deep artificial neural network . Some transformer variants reduce the number of tokens in deeper layers . For example, a transformer that translates a sentence transforms the stacked word embeddings, layer by layer, into feature vectors from which the translated words are predicted . Contemporary LLMs, such as GPT-3, are transformers trained to predict the next token of a text ; transformers also process images by taking patches as tokens . The layers of a transformer alternate between two types of transformations of X (see Fig.~\\ref{fig_transformer_block_dict}). A token-mixing layer maps X to a new matrix X, each row of which is a weighted sum of the rows of X, with weights that can depend on X itself; the tokens thereby interact. In the original transformer, token mixing is implemented by multi-head attention ; attention sets transformers apart from RNNs, which process the tokens of a data point sequentially instead of in parallel . A position-wise multilayer perceptron then transforms each row of X separately . Each of the two transformations is wrapped in a residual connection, and a subsequent layer normalization standardizes each row of the resulting matrix . The matrices produced by successive layers represent the tokens at increasingly abstract levels. This progression is measured by probing: a simple classifier is trained to predict a chosen property from the output of a single layer, and the depth at which the property becomes predictable locates it in the network. Early layers of a language transformer encode word-level properties, such as part-of-speech tags, while relations between tokens far apart in the text become predictable only in deeper layers . Deep CNNs show the same progression, from edges and textures in early layers to object parts in deeper ones , and vision transformers develop it too: their early layers split attention between nearby and distant patches, while the deeper layers attend across the whole image . The most informative representations often arise in the middle layers, whose outputs can outperform those of the final layer as features for other tasks . attention compares every pair of tokens, so its cost grows as n2 with the number n of tokens . This cost motivates cheaper token mixing: any transformation that combines the rows of X can take the place of attention --- a multilayer perceptron applied across tokens , a fixed Fourier transform , or a state-space layer whose cost is linear in n .", "synonyms": [], "see_also": [ "attention", "ann", "layer", "token", "embedding", "wordembedding", "rnn", "nlp", "llm", "multilayerperceptron", "residualconnection", "layernormalization", "cnn" ], "demo": "https://dictionaryofml.org/terms/transformer.py", "notebook": "https://dictionaryofml.org/terms/transformer.ipynb" }, { "key": "transparency", "name": "transparency", "url": "https://dictionaryofml.org/terms/transparency.html", "pdf": "https://dictionaryofml.org/terms/transparency.pdf", "abstract": "Transparency is a key requirement for trustworthy artificial intelligence: it names the duties to close the information gaps between the provider of an artificial intelligence system, its deployer, and the persons affected by its predictions. For machine learning methods, transparency is often used interchangeably with explainability: explanations, delivered along with the predictions of a learned hypothesis, let a human user anticipate those predictions. The EU AI Act makes transparency a binding design requirement. Under the Act, the provider must design a high-risk artificial intelligence system so that its deployer can interpret the delivered predictions, persons interacting with an artificial intelligence system must be informed of that fact, and affected persons may obtain an explanation of decisions based on the predictions delivered by a high-risk artificial intelligence system. The Act also requires documentation of the algorithm design, the training datasets, and the intended use of an artificial intelligence system, as well as a machine-readable marking of synthetic content as artificially generated.", "description": "Consider a bank that uses an artificial intelligence system to score loan applications. The loan officer sees only the score, and the applicant might not even know that an artificial intelligence system was involved. Transparency names the duties to close such information gaps; it is a fundamental requirement for trustworthy artificial intelligence . In the context of machine learning methods, transparency is often used interchangeably with explainability (; ): explanations, delivered along with the predictions of a learned hypothesis, let a human user anticipate those predictions. Some machine learning methods inherently offer this form of transparency. classification methods quantify the confidence in an individual prediction via the distance of the feature vector from the decision boundary. A shallow decision tree does not require a separately constructed explanation: presenting the tree as a flow chart of explicit, human-readable decision rules is an explanation of every prediction it delivers (see interpretable machine learning) . In the broader scope of AI systems, transparency extends beyond explainability: it includes informing persons that an artificial intelligence system is in use and providing information about the limitations, the overall reliability, and the intended use of the system as a whole. The binding transparency obligations discussed below are those of the EU AI Act; other jurisdictions impose related but distinct duties, e.g., the Colorado Automated Decision-Making Technology Act and the court proceedings State v. Loomis on the use of a proprietary recidivism-risk score in sentencing . The Act distributes its obligations among three roles: the provider that develops an artificial intelligence system, the deployer that uses it, and the persons who interact with the artificial intelligence system or are affected by its output (see deployer). The Act's term \\emph{output} denotes the predictions delivered by a learned hypothesis; the machine learning term \\emph{prediction} is used below. Like interpretability and explainability, which are relative to a specific user or group of users, transparency is relative to its addressee: what must be disclosed depends on the role. Concretely, Fig.~\\ref{fig_transparency_mlpicture} locates these obligations in a scatterplot of training set data points together with a learned hypothesis, for the credit-scoring example. The solid curve depicts the learned hypothesis h that the artificial intelligence system applies: it maps the income x of an applicant to a predicted credit score h(x), and a loan is approved when the predicted score exceeds an approval threshold (dotted line). Art.~13 concerns the use of this map: the deployer must be able to interpret the delivered predictions — here, to read off the prediction h(x') for an applicant with income x' (open square) and its distance from the threshold. The filled circles depict the training set D(train) of completed loans, with income as the feature x and credit score as the label y, from which h was learned, e.g., via empirical risk minimization — by minimizing the average loss of the hypothesis over D(train). Art.~11 requires the provider to document both ingredients of this training: the dataset D(train) and the design of the algorithm, including the model and the loss used. Income levels outside the range covered by D(train) (shaded region) are a limitation that the documentation must state. The right to explanation of Art.~86 concerns a single prediction: the applicant with income x', whose predicted score falls below the threshold, may ask why. One answer is a counterfactual: the smallest change of the feature that flips the decision — the income x'' (open diamond, arrow) at which h reaches the threshold. Specifically, toward the deployer, the provider must design and develop a high-risk artificial intelligence system so that its operation is sufficiently transparent to enable the deployers to interpret its predictions and use them appropriately . In medical diagnosis, the provider is, e.g., a company that develops an X-ray analysis tool, and a clinic that uses the tool acts as the deployer: the provider can meet the duty of Art.~13 by designing the tool to disclose to the clinician the confidence level for the predictions delivered by a learned hypothesis. This duty targets the meaning of the predictions in the context of their use; the AI Risk Management Framework of the US National Institute of Standards and Technology reserves the term \\emph{interpretability} for this contextual meaning . This usage differs from interpretability as the comprehension of the computational process of the method; Art.~13 does not require that comprehension. Toward persons who interact directly with an artificial intelligence system, such as an artificial intelligence-powered chatbot, the provider must design the artificial intelligence system so that these persons are informed of that fact . providers of AI systems that generate synthetic audio, image, video, or text content must mark such content in a machine-readable format as artificially generated . This marking duty applies whether or not the artificial intelligence system is a high-risk artificial intelligence system; it is waived, e.g., where the artificial intelligence system only performs an assistive function for standard editing . For a deep fake, the deployer must, in addition, disclose visibly that the content is artificially generated or manipulated . A person can be affected by an artificial intelligence system without directly interacting with it. A patient whose diagnosis is supported by an artificial intelligence-based system deals only with the clinician. Nevertheless, the deployer must inform the patient that a high-risk artificial intelligence system is used concerning them . The EU AI Act also grants the affected person a right to explanation: the deployer must, on request, provide a clear and meaningful explanation of the role of the artificial intelligence system in a decision based on its predictions . In credit scoring, for example, a loan applicant faced with an adverse automated decision may obtain from the deployer, under this right, an explanation of the contributing factors, such as income level or credit history, and use it to contest the decision (see Fig.~\\ref{fig_transparency_mlpicture}). Transparency also encompasses documentation detailing the purpose and design choices underlying the artificial intelligence system. For a high-risk artificial intelligence system, the provider must provide documentation that covers the algorithm design and the datasets used for training . The provider of a general-purpose AI model must additionally publish a sufficiently detailed summary of the content used for training . Datasheets for datasets and model cards help practitioners understand the intended use cases and limitations of an artificial intelligence system. Fig.~\\ref{fig_transparency_dict} summarizes these transparency obligations as information flows between the provider of an artificial intelligence system, its deployer, and the persons exposed to its predictions. See also: trustworthy artificial intelligence, explainability, interpretability, interpretable machine learning, explanation, counterfactual, right to explanation, EU AI Act, high-risk artificial intelligence system, artificial intelligence system, general-purpose AI model, provider, deployer, watermarking, deep fake, content provenance.", "synonyms": [], "see_also": [], "demo": "https://dictionaryofml.org/terms/transparency.py", "notebook": "https://dictionaryofml.org/terms/transparency.ipynb" }, { "key": "valset", "name": "validation set", "url": "https://dictionaryofml.org/terms/valset.html", "pdf": "https://dictionaryofml.org/terms/valset.pdf", "abstract": "A validation set consists of data points which have not been used for model training. For a learned hypothesis, the average loss on the validation set, the validation error, indicates how well the hypothesis predicts the labels of data points outside the training set. The validation error is used for model selection: the hypotheses learned by different machine learning methods are compared, and the one with the smallest validation error is selected. The selected hypothesis depends on the validation set, so assessing it requires a test set whose data points entered neither training nor selection. The size of the validation set determines how reliable the validation error is, and Hoeffding's inequality prescribes the size needed for the risk to lie, with high probability, within a given uncertainty band around the measured validation error. When data points are scarce, k-fold cross-validation averages the validation errors obtained from using different folds of the dataset as validation set.", "description": "Consider three days of weather recordings: for each day, the morning minimum temperature is the feature and the maximum daytime temperature is the label . Fig.~\\ref{fig_valset_picture_dict} shows a training set D(train) of three such days along with two hypotheses learned from it: the straight line h(1) delivered by empirical risk minimization on the linear model, and a degree-two polynomial h(2) that passes through all three data points and therefore attains a training error of zero. Three further days, marked by open triangles, were held back from model training: they form a validation set, on which the loss of each learned hypothesis can be evaluated. A validation set D(val) consists of data points which have not been used for model training. For a hypothesis h learned from the training set, the resulting average loss on the validation set, \\[ \\frac{1}{|\\valset|} \\sum_{\\datapoint \\in \\valset} \\lossfunc{\\datapoint}{\\learnthypothesis} \\text{,} \\] is the validation error: it indicates how well h predicts the labels of data points outside the training set . In Fig.~\\ref{fig_valset_picture_dict}, the polynomial h(2) attains a training error of zero but a validation error of 17.3, while the line h(1) has a training error of 2.9 and a validation error of 3.0. The validation error is used for model selection: the hypotheses learned by different machine learning methods are compared, and the one with the smallest validation error is selected . Fig.~\\ref{fig_valset_modelsel_dict} shows the selection between the two learned hypotheses of the weather example: the line is selected, since 3.0 < 17.3. The selected hypothesis depends on D(val): its validation error is the smallest of the compared validation errors, so the guarantee for a single fixed hypothesis no longer applies, and a valid bound must hold uniformly over all compared candidates, growing with their number . Assessing the selected hypothesis therefore requires a third dataset whose data points entered neither training nor selection: the test set . In Fig.~\\ref{fig_valset_modelsel_dict}, the three days of the test set, marked by open squares, yield a test error of 1.3 for the selected line. The size |D(val)| determines how well the validation error reflects the overall performance of a learned hypothesis: the validation error is an average of |D(val)| terms, and a validation set that is too small results in an unreliable validation error . A concentration inequality quantifies the reliability. For a loss with values in [0,1] and a fixed hypothesis h, Hoeffding's inequality guarantees that, with probability at least 1 - δ, the risk of h lies within an uncertainty band of half-width Δ around the measured validation error, provided that \\[ |\\valset| \\geq \\frac{\\ln(2/\\delta)}{2 \\Delta^{2}} \\text{.} \\] Fig.~\\ref{fig_valset_bound_dict} shows this lower bound as a function of the half-width Δ for three values of δ. Narrowing the band is costly: halving Δ quadruples the required size. To have the validation error within Δ = 0.1 of the risk with δ = 0.05, it suffices to hold back |D(val)| ≥ 185 data points, regardless of the hypothesis space and of the size of D(train). When data points are scarce, holding back a large validation set leaves few data points for the training set. As a remedy, k-fold cross-validation divides the dataset into k folds and uses each fold in turn as validation set: every fold yields a noisy validation error, an average over few data points, and averaging the k per-fold validation errors produces a more reliable estimate of the expected loss (; ). In a complete workflow, the available dataset is thus split three ways: a training set to learn hypotheses, a validation set to select among them, and a test set, used only once, to assess the selected hypothesis.", "synonyms": [], "see_also": [ "validation", "trainset", "testset", "datapoint", "risk", "hypothesis", "loss", "valerr", "trainerr", "kfoldcv", "loo", "modelsel" ], "demo": "https://dictionaryofml.org/terms/valset.py", "notebook": "https://dictionaryofml.org/terms/valset.ipynb" }, { "key": "variance", "name": "variance", "url": "https://dictionaryofml.org/terms/variance.html", "pdf": "https://dictionaryofml.org/terms/variance.pdf", "abstract": "The variance of a real-valued random variable is the expectation of the squared difference between the random variable and its mean. It quantifies the spread of the probability distribution of the random variable around the mean: a small variance means that realizations concentrate near the mean. The term also refers to the sample variance of a dataset; the two usages are consistent, since the sample variance is the variance of the random variable obtained by drawing a data point uniformly from the dataset. Under the squared error loss, the variance is the smallest risk that a constant prediction of a numeric label can achieve. For a random vector, the variance is the expectation of the squared Euclidean norm of the deviation from the mean. This quantity equals the trace of the covariance matrix of the random vector, i.e., the sum of the variances of its entries.", "description": "The variance of a real-valued random variable y is the expectation E ( y - Ey )2 of the squared difference between y and its expectation Ey. Fig.~\\ref{fig_variance_temps} depicts seven realizations of such an random variable: daily maximum temperatures measured at the Helsinki Kaisaniemi weather station , together with their sample mean and the deviation of each measurement from it. The spread of the measurements around the sample mean is what the variance quantifies. The term is also used to refer to the sample variance of a finite dataset D = y(1), …, y(m) ∈ R. These two usages are consistent. A dataset defines a discrete random variable y(D) = y(I) on the sample space 1, …, m. Here, the index I is chosen uniformly at random, i.e., I=r=1/m for all r=1,…,m, so the probability distribution of y(D) is the empirical distribution of the dataset. The variance of y(D) is precisely the sample variance \\[ \\frac{1}{\\samplesize} \\sum_{\\sampleidx=1}^{\\samplesize} \\Big( \\truelabel^{(\\sampleidx)} - \\frac{1}{\\samplesize} \\sum_{\\sampleidx'=1}^{\\samplesize} \\truelabel^{(\\sampleidx')} \\Big)^{2} \\text{,} \\] i.e., the average squared deviation of the data points from their sample mean. For an random variable with a finite second moment, i.e., E y2 < ∞, the variance is the minimum of the risk minimization problem whose unique solution is the mean : \\[ \\expect\\big\\{ \\big( \\truelabel - \\expect\\{\\truelabel\\} \\big)^{2} \\big\\} = \\min_{\\offset \\in \\reals} \\expect\\big\\{ ( \\truelabel - \\offset )^{2} \\big\\} \\text{.} \\] This minimum-risk characterization applies directly to the simplest machine learning problem, predicting a numeric label y without using any features. The hypothesis space for this problem consists of the constant maps h(·) = b with b ∈ R. Each such hypothesis amounts to a single bias term: it delivers the same prediction b for every data point. Given a training set D(train) = y(1), …, y(m) that consists of labels y(r) ∈ R only, empirical risk minimization with the squared error loss amounts to finding the constant that best predicts the label of a data point: the solution b is the sample mean of the labels (see mean), and the training error of the learned hypothesis is the sample variance of the labels (\\texttt{pythondemos/variance.py}). For the temperatures of Fig.~\\ref{fig_variance_temps}, the learned hypothesis is the dashed line: b = 20.5C, with training error 6.34/7 0.91. At the level of the underlying probability distribution, the variance of the label is the smallest risk that a constant prediction can achieve under the squared error loss. It therefore serves as a baseline: an machine learning method that uses features is useful only if its risk falls below the variance of the label. The definition extends to random vectors x as E x - Ex 22 = x, i.e., the sum of the variances of each entry of x, which can be written compactly as the trace of the covariance matrix x of x.", "synonyms": [ "second central moment" ], "see_also": [ "rv", "randomvector", "expectation", "mean", "samplevariance", "stddev", "probdist", "erm", "sqerrloss", "baseline", "covmtx", "trace", "outlier" ], "demo": "https://dictionaryofml.org/terms/variance.py", "notebook": "https://dictionaryofml.org/terms/variance.ipynb" }, { "key": "xaiterm", "name": "explainable artificial intelligence", "url": "https://dictionaryofml.org/terms/xaiterm.html", "pdf": "https://dictionaryofml.org/terms/xaiterm.pdf", "abstract": "Explainable artificial intelligence (XAI) is the subfield of artificial intelligence concerned with making the predictions of machine learning methods understandable to humans. Much of it can be posed as a function approximation problem: a learned hypothesis is explained by a simpler function that a human can comprehend. explanations differ in what they are made of. One kind reports a score for each feature, read off a linear map fitted near the data point (local interpretable model-agnostic explanations) or obtained as additive contributions (SHapley Additive exPlanations). A counterfactual instead names another feature vector, the closest one whose prediction differs. A concept activation vector states the explanation in a concept the user names, located in the activations inside the trained method. All three are post hoc and treat the learned hypothesis as fixed. The alternative is a hypothesis that needs no separate explanation, reached by restricting the hypothesis space to hypotheses a human comprehends or by regularization that favors explainable ones (explainable empirical risk minimization).", "description": "XAI is the subfield of artificial intelligence concerned with making the predictions of machine learning methods understandable to humans. It aims to complement each prediction by an explanation of how it has been obtained. If the machine learning method uses a sufficiently simple hypothesis space, the learned hypothesis may be inherently interpretable and no separate explanation is needed ; see interpretable machine learning, discussed further below. The term XAI was popularized by a program of the US Defense Advanced Research Projects Agency ; in the context of machine learning, the synonymous term explainable machine learning is also used. The underlying property is explainability, which the international terminology standard for artificial intelligence defines for AI systems in general . XAI can be posed as a function approximation problem. A trained hypothesis h: X → Y, e.g., delivered by an opaque deep net, is typically a highly nonlinear function on a potentially high-dimensional feature space X . Explaining h at a data point with feature vector x ∈ X means answering one question about that function in a form a human can comprehend. The examples of explanations below are of three kinds, distinguished by what the explanation is made of. The first is a score for each feature of the data point. The scores are read off a simpler hypothesis that agrees with h near x and comes from a user-parsable hypothesis space. local interpretable model-agnostic explanations fits a linear map as that simpler hypothesis, and its weights are the scores. Fig.~\\ref{fig_explainableml_dict} draws that construction. The second is another feature vector, one whose prediction differs. The third is a concept the user names, located inside the trained method rather than among the features of the data point. All three are post hoc: they treat the learned hypothesis as fixed and construct the explanation after training. The first kind of explanation reports one score per feature, saying how much that feature contributed to the prediction. local interpretable model-agnostic explanations obtains the scores by fitting a linear map to h near x and reading off its weights . SHapley Additive exPlanations obtains them by decomposing the prediction into a base value and one contribution per feature, the contributions being Shapley values (; ). The two deliver the same kind of explanation and differ in how its scores are computed. When features are properties of image pixels, the feature scores are a second image of the same size: one relevance score per pixel (see Fig.~\\ref{fig_xai_saliency_dict}). The class activation map is one construction of such a per-pixel score. The second kind of explanation is a counterfactual. It reports no scores and approximates nothing. It asks instead where h takes a different value, pointing out how the features must be modified for the prediction h(x) to come out differently. A counterfactual identifies the smallest such modification, with closeness measured by a chosen metric (see Fig.~\\ref{fig_xai_approx_dict}). In a loan-approval setting, for example, local interpretable model-agnostic explanations reports which of the applicant features, such as income and credit history, its local linear approximation assigns the largest weight to in a rejection, while a counterfactual states the smallest modification of those features that would turn the prediction into an approval. The third kind of explanation refers not to the features of the data point but to a concept the user specifies. Such a concept is defined by a dataset of examples the user supplies. A concept activation vector locates such a concept as a direction in the space of the activations of one hidden layer of an artificial neural network: the normal vector of a decision boundary that separates the supplied examples carrying the concept from those that do not . In contrast to local interpretable model-agnostic explanations and counterfactual explanations, a concept activation vector needs access to the internal computations of a machine learning system. One such internal computation is the activation z = f(x) of a hidden layer in a deep net. A deep net computes h by feeding the activations z to the final layers, which deliver a score s(z) for each label; the prediction h(x) follows from those scores (see concept activation vector). In an image classifier, for example, a user who suspects that photographs are labeled zebra through the concept ``stripes'' supplies photographs that show stripes and photographs that do not. Each of these photographs results in an activation z in the layer of interest. These activations are then used to fit a linear classifier that separates the two sets, and its normal vector is the concept activation vector for ``stripes''. How much the concept contributed to the prediction of a zebra photograph is then measured by the directional derivative of the score s along that concept activation vector. The above XAI methods construct an explanation after training, for a learned hypothesis that is already fixed. The alternative is to arrive at one that needs no separate explanation. interpretable machine learning does so by admitting only hypotheses a human comprehends directly. This can be achieved by manually choosing a hypothesis space that is simple enough, or by regularization that favors hypotheses with specific properties, such as sparsity or predictability (; ). explainable empirical risk minimization uses regularization but not to that end: it leaves the hypothesis space as it is and delivers explainability to one specific user. That user supplies their own predictions for the data points of a training set, and the penalty term measures the deviation of the predictions of the learned hypothesis from the user predictions . Three aspects of an explanation are studied: whom it serves, how much of the learned hypothesis it covers, and whether it is faithful. How much an explanation achieves depends on who receives it: explainability is measured relative to a specific user (; ). explanations can be local, concerning a single prediction, or global, characterizing the learned hypothesis as a whole . An explanation must be faithful, i.e., reflect the computation that the learned hypothesis actually carries out. When the explanation scores the features, this can be tested rather than asserted. A class activation map is faithful for a prediction if flipping the pixels it scores highest changes that prediction more often than flipping as many of the pixels it scores lowest. Perturbing the highest-scoring regions first and recording how quickly the predicted class score falls is the standard form of the test . For the image of Fig.~\\ref{fig_xai_saliency_dict} and a linear classifier fitted to images of that kind, flipping the three highest-scoring pixels changes the prediction, while flipping the pixels in the opposite order leaves it unchanged through all 36 of them; the map-guided order also changes the prediction sooner on each of 200 noisy variants of the image. Faithfulness also limits what a post hoc explanation can achieve: one that agreed with the learned hypothesis everywhere would be that hypothesis itself, so a simpler explanation deviates from it somewhere . Two legal instruments explicitly state requirements on XAI. Under the general data protection regulation, a person subjected to a decision taken by automated means is entitled to ``meaningful information about the logic involved'' in that decision . Communicating the algorithm itself is not a sufficiently concise and intelligible explanation . counterfactual information can be appropriate: the extent to which a variation in the personal data would have led to a different result. The second instrument is the EU AI Act, which requires for a high-risk artificial intelligence system that affected persons obtain ``clear and meaningful explanations of the role of the AI system in the decision-making procedure'' (see right to explanation).", "synonyms": [ "explainable machine learning (explainable ML)" ], "see_also": [ "explainability", "interpretability", "explanation", "interpretableml", "eerm", "lime", "shap", "counterfactual", "feature", "mechanisticinterpretability", "classactivationmap", "righttoexplanation" ], "demo": "https://dictionaryofml.org/terms/xaiterm.py", "notebook": "https://dictionaryofml.org/terms/xaiterm.ipynb" } ] }