Dictionary of Applied Machine Learning

large language model

Updated on 2026-09-16

Typeset PDF version — the authoritative form of this entry

Python demo — a script that recomputes what this entry states and prints one line per check

A large language model (LLM) is an artificial neural network (ANN), typically with billions of model parameters, that implements a hypothesis map from a prompt to a response. The training of an LLM often proceeds in stages: pretraining uses self-supervised learning, splitting raw text into overlapping fragments that yield data points without manual annotation, and fine-tuning uses a small dataset of manually curated prompt–response pairs. LLM agents extend LLMs beyond text: a program interprets the response, for example as code to run or as a tool to execute, and can append the result to the next prompt, so the LLM chooses actions that the surrounding program carries out.

Definition

B-nexttokenConsider a chat-based artificial intelligence (AI) system such as ChatGPT, Claude, Gemini, DeepSeek, or Mistral: a user types a question, the prompt, and receives a reply, the response. The system that produces the reply is an LLM: an artificial neural network (ANN), typically with billions of model parameters, trained to output text given input text. Each data point of this learning task is a pair of texts, and the trained LLM implements a hypothesis map from the prompt to the response (Fig. 1). In contrast, a text classifier that maps a document to one of a fixed set of labels is not an LLM: it returns a label, not text.

Figure 1 of the entry llm
Figure 1: An LLM as a hypothesis map from prompts to responses. Each filled circle is one data point, a pair of input text and output text; for one of the three, the prompt and the response are spelled out. The dashed curve depicts the hypothesis $\learnthypothesis$ that the trained LLM implements: it assigns a response to any prompt. The axes are schematic, as the underlying feature space and label space consist of token sequences for which there is no natural ordering
Most LLMs implement $\learnthypothesis$ autoregressively: the response is generated one token at a time, each drawn from a probability distribution over the vocabulary that the network computes from the prompt and all tokens generated so far (Bengio et al., 2003; Brown et al., 2020). Alternative constructions exist: masked probabilistic models predict removed tokens and serve text analysis rather than generation (Devlin et al., 2019), and diffusion language models generate many tokens in parallel (Li et al., 2022).

The training of an LLM consists of several stages. One stage typically uses self-supervised learning: a given text is split into overlapping fragments, and each fragment amounts to a data point whose label is one of its tokens and whose features are the remaining tokens. Training sets built from text fragments require no manual annotation and are used for pretraining the LLM via empirical risk minimization (ERM) (Brown et al., 2020; Devlin et al., 2019). Another stage is fine-tuning, which uses a small dataset of manually curated prompt–response pairs (Ouyang et al., 2022). An LLM whose model parameters are reused across many learning tasks is a foundation model (Bommasani et al., 2021).

B-localLLM inference, i.e., computing the response for a given prompt, is computationally much cheaper than its training. Still, the inference itself is computationally costly and is often implemented using cloud computing services. Local execution is catching up, however: open-weight LLMs, whose model parameters are published (Touvron et al., 2023), can be compressed by quantization. Quantization stores each model parameter in a few bits instead of a $16$-bit floating-point number (Dettmers et al., 2022). An LLM with billions of model parameters then runs on a laptop or phone (see edge computing).

B-agentThe output of an LLM by itself is nothing but a sequence of letters that constitute the response. An LLM (or coding) agent interprets the response of an LLM as instructions for computations to be executed (Yao et al., 2023). For example, the prompt can steer the LLM to produce a response that is valid source code in a programming language such as Python, as in the spelled-out data point of Fig. 1. The delivered source code can then be executed, and the result can be appended to the next prompt.

See also: token, vocabulary, transformer, attention, artificial neural network, word embedding, softmax function, natural language processing, foundation model, pretraining, fine-tuning, agent, cloud computing, edge computing.

References

  1. Bengio et al. (2003). A Neural Probabilistic Language Model. J. Mach. Learn. Res.. www.jmlr.org/papers/v3/bengio03a.html
  2. Brown et al. (2020). Language Models are Few-Shot Learners. Adv. Neural Inf. Process. Syst.. proceedings.neurips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
  3. Devlin et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proc. 2019 Conf. North American Chapter Assoc. Computational Linguistics: Human Lang. Technologies, Volume 1 (Long and Short Papers). doi.org/10.18653/v1/N19-1423
  4. Li et al. (2022). Diffusion-LM Improves Controllable Text Generation. Adv. Neural Inf. Process. Syst. (NeurIPS). proceedings.neurips.cc/paper_files/paper/2022/hash/1be5bc25d50895ee656b8c2d9eb89d6a-Abstract.html
  5. Ouyang et al. (2022). Training Language Models to Follow Instructions with Human Feedback. Adv. Neural Inf. Process. Syst. (NeurIPS). proceedings.neurips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract.html
  6. Bommasani et al. (2021). On the Opportunities and Risks of Foundation Models. doi.org/10.48550/arXiv.2108.07258
  7. Touvron et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arxiv.org/abs/2302.13971
  8. Dettmers et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. Adv. Neural Inf. Process. Syst. (NeurIPS). proceedings.neurips.cc/paper_files/paper/2022/hash/c3ba4962c05c49636d4c6206a97e9c8a-Abstract.html
  9. Yao et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. Int. Conf. Learn. Represent. (ICLR). openreview.net/forum?id=WE_vluYUL-X

Cite this entry

@misc{dictml_llm,
  author = {Jung, Alexander},
  editor = {Olioumtsevits, Konstantina and Schnoor, Ekkehard},
  title = {large language model},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-09-21},
  url = {https://dictionaryofml.org/terms/llm.html}
}