transparency

Jung, Alexander

Transparency is a key requirement for trustworthy artificial intelligence (trustworthy AI): it names the duties to close the information gaps between the provider of an artificial intelligence system (AI system), its deployer, and the persons affected by its predictions. For machine learning (ML) methods, transparency is often used interchangeably with explainability: explanations, delivered along with the predictions of a learned hypothesis, let a human user anticipate those predictions. The EU AI Act makes transparency a binding design requirement. Under the Act, the provider must design a high-risk artificial intelligence system (high-risk AI system) so that its deployer can interpret the delivered predictions. Persons interacting with an AI system must be informed of that fact. Affected persons may obtain an explanation of decisions based on the predictions delivered by a high-risk AI system. The Act also requires documentation of the algorithm design, the training datasets, and the intended use of an AI system, as well as a machine-readable marking of synthetic content as artificially generated.

Definition

Consider a bank that uses an artificial intelligence system (AI system) to score loan applications. The loan officer sees only the score, and the applicant might not even know that an AI system was involved. Transparency names the duties to close such information gaps; it is a fundamental requirement for trustworthy artificial intelligence (trustworthy AI) (High-Level Expert Group on Artificial Intelligence, 2019).

P-methodsIn the context of machine learning (ML) methods, transparency is often used interchangeably with explainability (Gallese, 2023): explanations, delivered along with the predictions of a learned hypothesis, let a human user anticipate those predictions (Jung and Nardelli, 2020). Some ML methods inherently offer this form of transparency. Classification methods quantify the confidence in an individual prediction via the distance of the feature vector from the decision boundary. A shallow decision tree does not require a separately constructed explanation: presenting the tree as a flow chart of explicit, human-readable decision rules is an explanation of every prediction it delivers (see interpretable machine learning (interpretable ML)) (Rudin, 2019).

In the broader scope of AI systems, transparency extends beyond explainability: it includes informing persons that an AI system is in use and providing information about the limitations, the overall reliability, and the intended use of the system as a whole. The binding transparency obligations discussed below are those of the EU AI Act; other jurisdictions impose related but distinct duties, e.g., the Colorado Automated Decision-Making Technology Act (Colorado General Assembly, 2026) and the court proceedings State v. Loomis on the use of a proprietary recidivism-risk score in sentencing (Supreme Court of Wisconsin, 2016). The Act distributes its obligations among three roles: the provider that develops an AI system, the deployer that uses it, and the persons who interact with the AI system or are affected by its output (see deployer). The Act's term output denotes the predictions delivered by a learned hypothesis; the ML term prediction is used below. Like interpretability and explainability, which are relative to a specific user or group of users, transparency is relative to its addressee: what must be disclosed depends on the role.

Specifically, toward the deployer, the provider must design and develop a high-risk artificial intelligence system (high-risk AI system) so that its operation is sufficiently transparent to enable the deployers to interpret its predictions and use them appropriately (European Parliament and Council of the European Union, 2024, Art. 13). A second example, medical diagnosis, shows the two roles: the provider is, e.g., a company that develops an X-ray analysis tool, and a clinic that uses the tool acts as the deployer. The provider can meet the duty of Art. 13 by designing the tool to disclose to the clinician the confidence level for the predictions delivered by a learned hypothesis. This duty concerns the meaning of the predictions in the context of their use; the AI Risk Management Framework of the US National Institute of Standards and Technology reserves the term interpretability for this contextual meaning (National Institute of Standards and Technology, 2023). This usage differs from interpretability as the comprehension of the computational process of the method; Art. 13 does not require that comprehension.

Toward persons who interact directly with an AI system, such as an artificial intelligence (AI)-powered chatbot, the provider must design the AI system so that these persons are informed of that fact (European Parliament and Council of the European Union, 2024, Art. 50(1)). Providers of AI systems that generate synthetic audio, image, video, or text content must mark such content in a machine-readable format as artificially generated (European Parliament and Council of the European Union, 2024, Art. 50(2)). This marking duty applies whether or not the AI system is a high-risk AI system; it is waived, e.g., where the AI system only performs an assistive function for standard editing (European Parliament and Council of the European Union, 2024, Art. 50(2)). For a deep fake, the deployer must, in addition, disclose visibly that the content is artificially generated or manipulated (European Parliament and Council of the European Union, 2024, Art. 50(4)).

A person can be affected by an AI system without directly interacting with it. A patient whose diagnosis is supported by an AI-based system deals only with the clinician. Nevertheless, the deployer must inform the patient that a high-risk AI system is used concerning them (European Parliament and Council of the European Union, 2024, Art. 26(11)). The EU AI Act also grants the affected person a right to explanation: the deployer must, on request, provide a clear and meaningful explanation of the role of the AI system in a decision based on its predictions (European Parliament and Council of the European Union, 2024, Art. 86). In credit scoring, for example, a loan applicant faced with an adverse automated decision may obtain from the deployer, under this right, an explanation of the contributing factors, such as income level or credit history, and use it to contest the decision (see Fig. 1).

Transparency also encompasses documentation detailing the purpose and design choices underlying the AI system. For a high-risk AI system, the provider must provide documentation that covers the algorithm design and the datasets used for training (European Parliament and Council of the European Union, 2024, Art. 11). The provider of a general-purpose AI model (GPAI model) must additionally publish a sufficiently detailed summary of the content used for training (European Parliament and Council of the European Union, 2024, Art. 53). Datasheets for datasets (Gebru et al., 2021) and model cards (Mitchell et al., 2019) help practitioners understand the intended use cases and limitations of an AI system.

P-readConcretely, Fig. 1 locates these obligations in a scatterplot of training set data points together with a learned hypothesis, for the credit-scoring example. The solid curve depicts the learned hypothesis $\widehat{h}$ that the AI system applies: it maps the income $x$ of an applicant to a predicted credit score $\widehat{h}(x)$, and a loan is approved when the predicted score exceeds an approval threshold (dotted line). The duty of Art. 13 concerns the use of this map: here, the deployer must be able to state the prediction $\widehat{h}(x')$ for an applicant with income $x'$ (open square) and its distance from the threshold.

P-trainThe filled circles depict the training set $\mathcal{D}^{(\rm train)}$ of completed loans, with income as the feature $x$ and credit score as the label $y$, from which $\widehat{h}$ was learned, e.g., via empirical risk minimization (ERM), by minimizing the average loss of the hypothesis over $\mathcal{D}^{(\rm train)}$. The documentation of Art. 11 covers both components of this training: the dataset $\mathcal{D}^{(\rm train)}$ and the design of the algorithm, including the model and the loss used. Income levels outside the range covered by $\mathcal{D}^{(\rm train)}$ (shaded region) are a limitation that the documentation must state, since no estimate of the risk computed from $\mathcal{D}^{(\rm train)}$ covers them.

P-cfThe right to explanation of Art. 86 is exercised for a single prediction: the applicant with income $x'$, whose predicted score falls below the threshold, may ask why. One answer is a counterfactual: the smallest change of the feature that flips the decision, the income $x''$ (open diamond, arrow) at which $\widehat{h}$ reaches the threshold.

transparency

Definition

References

Cite this entry