\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath,amssymb}
\usepackage{algorithm,algpseudocode}
\usepackage{bm}
\usepackage{hyperref}
\usepackage{float}
\usepackage{subcaption}
\usepackage{enumitem}
\usepackage{xcolor}
\definecolor{darkgreen}{rgb}{0.0, 0.5, 0.0}
\definecolor{aaltoBlue}{RGB}{0, 51, 102}
\definecolor{pinegreen}{cmyk}{0.92,0,0.59,0.25}
\definecolor{royalblue}{cmyk}{1,0.50,0,0}
\definecolor{lavander}{cmyk}{0,0.48,0,0}
\definecolor{violet}{cmyk}{0.79,0.88,0,0}
\definecolor{lightblue}{RGB}{173, 216, 230}
\usepackage{tikz}
\usepackage{pgfplots}
\usetikzlibrary{calc,arrows.meta,positioning,fit,shapes.geometric,shapes.misc,intersections,patterns,decorations.pathreplacing,backgrounds,matrix}
\usepgfplotslibrary{fillbetween,groupplots}
\setcounter{secnumdepth}{-\maxdimen}
% Long unbreakable tokens (the ISO/IEC author names in citations, term names
% in the title) otherwise overflow the margin by a few points; a modest
% emergency stretch lets TeX loosen the paragraph instead.
\emergencystretch=2em
\usepackage{mleftright}
% Project macros used by this entry (auto-extracted).
\newcommand{\dataset}{\mathcal{D}}
\newcommand{\defeq}{:=}
\newcommand{\featuredim}{d}
\newcommand{\featurespace}{\mathcal{X}}
\newcommand{\featuretrafovec}{{\bf \Phi}}
\newcommand{\featurevec}{\vx}
\newcommand{\hypothesis}{h}
\newcommand{\learnthypothesis}{\widehat{\hypothesis}}
\newcommand{\lossfunczo}[2]{L^{(0/1)}\left(#1,#2 \right)}
\newcommand{\prob}[1]{\mathbb{P}\left(#1\right)}
\newcommand{\reals}{\mathbb{R}}
\newcommand{\sensattr}{s}
\newcommand{\testset}{\dataset^{(\rm test)} }
\newcommand{\trainset}{\dataset^{(\rm train)}}
\newcommand{\truelabel}{y}
\newcommand{\vw}[0]{{\bf w}}
\newcommand{\vx}[0]{{\bf x}}
\newcommand{\vz}[0]{{\bf z}}
\newcommand{\weights}{\vw}

\begin{document}

\section{sensitive attribute}

\subsection*{Author and editors}
Alexander Jung\textsuperscript{*}, Department of Computer Science, Aalto University, Espoo, Finland (ORCID \mbox{\href{https://orcid.org/0000-0001-7538-0990}{0000-0001-7538-0990}}) \\
Editors: Konstantina Olioumtsevits (ORCID \mbox{\href{https://orcid.org/0000-0003-2697-8659}{0000-0003-2697-8659}}), Ekkehard Schnoor (ORCID \mbox{\href{https://orcid.org/0000-0001-8252-1499}{0000-0001-8252-1499}}), Aalto University, Espoo, Finland \\
\textsuperscript{*}Corresponding author: \href{mailto:alex.jung@aalto.fi}{alex.jung@aalto.fi}

\subsection{Synonyms}
protected attribute, protected characteristic

\subsection{Abstract}
A sensitive attribute of a data point is an attribute,
		such as gender or ethnic origin, that must not influence the
		prediction a machine learning system (ML system) delivers for that
		data point, whether or not it belongs to the
		feature vector. The word carries two senses that are kept
		apart: a regulatory one, in which data protection law fixes which
		categories of personal data are special, and a machine learning (ML) one,
		in which the application names the attribute $\sensattr$ that a
		fairness criterion compares predictions across.
		Dropping $\sensattr$ from the feature vector does not remove it
		from the data, because the remaining features can be
		dependent on it. Learned features, such as those principal component analysis (PCA)
		delivers, are no exception: the criterion that learns them never
		mentions $\sensattr$, and they can be predictive of it all the
		same. How much of $\sensattr$ they carry is measured by the
		recovery accuracy, the accuracy with which a linear classifier
		predicts $\sensattr$ from them on data points it was not
		learned from. For the weather at Krems an der Donau (Austria), two
		learned features recover the half of the year a day belongs
		to with a recovery accuracy of $0.94$, against a majority-class
		rate of $0.51$. Projection of the learned features
		onto the orthogonal complement of the direction the
		linear classifier uses lowers that accuracy without bringing it to
		the majority-class rate, so a sensitive attribute is easier to
		measure than to remove.

\subsection{Definition}
Consider a bank that decides
		on loan applications from the income, the occupation, and the
		address of an applicant. A supervisor asks whether the decisions
		depend on the gender of the applicant. The bank answers that the
		gender was never recorded as a feature: the feature vector
		$\featurevec$ that the learned hypothesis
		$\learnthypothesis$ reads holds those three numbers and nothing
		else. That answer settles nothing. The occupation and the address
		of an applicant can be predictive of the gender, and a
		prediction computed from $\featurevec$ then depends on the
		gender all the same.
\par
		A sensitive attribute of a data point is an attribute that
		must not influence the prediction delivered for that
		data point, whether or not it belongs to the
		feature vector. A data point then carries three parts: the
		feature vector $\featurevec \in \featurespace$, the label
		$\truelabel$, and the sensitive attribute $\sensattr$, which
		typically takes few values. The gender of a loan applicant is
		written $\sensattr \in \{0, 1\}$ below.
\par
		Which attribute counts as sensitive is settled in two different
		ways, and the two senses are kept apart. In the regulatory sense,
		the answer is given by law. The general data protection regulation (GDPR) prohibits, with
		exceptions, the processing of personal data revealing racial
		or ethnic origin, political opinions, religious or philosophical
		beliefs, or trade union membership, and of genetic data, biometric
		data used to identify a person, health data, and data concerning
		sex life or sexual orientation (European Parliament and Council of the European Union, 2016, Art.~9(1)). The
		EU AI Act adopts that list under the name special categories
		of personal data (European Parliament and Council of the European Union, 2024, Art.~3(37)). In the
		machine learning (ML) sense, the application decides, and its choice need not
		belong to that list: the gender of the loan applicant is a
		sensitive attribute of the example above and is not a special
		category under the GDPR. A fairness criterion compares the
		predictions of $\learnthypothesis$ across the values of
		$\sensattr$ (Barocas et al., 2023), while privacy protection asks
		instead that the output of the machine learning system (ML system) not allow
		inference of $\sensattr$, an inference measured by
		privacy leakage.
\par
		The obvious mitigation is the one the bank applied: leave
		$\sensattr$ out of the feature vector. It removes $\sensattr$
		from the feature vector and not from the data. The
		data point is modeled as a realization of random variables (RVs),
		with the feature vector and the sensitive attribute drawn
		jointly. The attribute is still present whenever the two are
		dependent,
		\begin{equation}
			\label{equ_sensattr_proxy_dict}
			\prob{\sensattr = 1 \mid \featurevec \in \mathcal{A}} \neq
			\prob{\sensattr = 1} \text{ for some } \mathcal{A} \subseteq
			\featurespace \text{.}
		\end{equation}
		A feature for which the dependence
		\eqref{equ_sensattr_proxy_dict} holds is a proxy for $\sensattr$.
		Such redundant encodings are what lets a decision rule that
		ignores $\sensattr$ discriminate all the same
		(Dwork et al., 2012).
\par
		Feature learning does not undo the dependence
		\eqref{equ_sensattr_proxy_dict}. It replaces the raw
		features by learned ones, $\vz =
		\featuretrafovec(\featurevec) \in \reals^{\featuredim'}$,
		delivered by a feature transformation $\featuretrafovec$
		that is itself learned from a dataset. Principal component analysis (PCA) learns a
		linear $\featuretrafovec$ whose rows are the $\featuredim'$
		eigenvectors of the sample covariance matrix with the largest
		eigenvalues, a criterion in which $\sensattr$ never
		appears. The learned features can be predictive of
		$\sensattr$ nonetheless, and how much of it they carry is a
		number that can be measured. A linear classifier
		$\learnthypothesis^{(\sensattr)}$ that predicts $\sensattr$ from
		$\vz$ is learned by least squares on a training set
		$\trainset$, and its recovery accuracy is its accuracy on a
		test set $\testset$ of data points outside
		$\trainset$,
		\begin{equation}
			\label{equ_sensattr_recovery_dict}
			\operatorname{acc}\big(\learnthypothesis^{(\sensattr)} \mid
			\testset\big) \defeq 1 - \frac{1}{\big|\testset\big|}
			\sum_{(\vz, \sensattr) \in \testset}
			\lossfunczo{(\vz, \sensattr)}{\learnthypothesis^{(\sensattr)}}
			\text{,}
		\end{equation}
		with the $0/1$ loss, which is $1$ for a wrong
		prediction and $0$ otherwise. The recovery accuracy
		\eqref{equ_sensattr_recovery_dict} is read against the
		majority-class rate, the accuracy of predicting the more
		frequent value of $\sensattr$ for every data point.
\par
		The features that PCA learns from the weather at
		Krems an der Donau (Austria) show what this measurement reports.
		Each day from January 6 to December 31 of 2024 is a
		data point, and its raw features are the eight daily
		measurements of each of the five previous days, forty numbers.
		Its attribute $\sensattr$ says whether the day lies in the cold
		half of the year, October through March; $178$ of the $361$ days
		do, so the majority-class rate is $0.51$. The half-year is not
		among the forty raw features. Each raw feature is
		scaled to zero sample mean and unit sample variance
		first, because temperatures, precipitation, and pressure carry
		different units. Two learned features
		recover it with a recovery accuracy
		\eqref{equ_sensattr_recovery_dict} of $0.94$, averaged over twenty
		random splits of the days into two halves, $\trainset$ and
		$\testset$, and a single learned feature already gives
		$0.94$. The decision boundary of one such linear classifier is
		a line in the plane of the two learned features
		(Fig.~\ref{fig_sensattr_dict}(a)). Dropping every temperature
		measurement from the raw features, the obvious proxy for
		the season, leaves twenty-five numbers per day. The recovery
		accuracy is then $0.82$ with two learned features and
		$0.89$ with five. The two curves of
		Fig.~\ref{fig_sensattr_dict}(b) compare the two sets of raw
		features for $\featuredim' = 1, \ldots, 10$. The half-year
		of a day is not a sensitive attribute. It
		stands in for one here, and the computation does not depend on
		what the attribute means.
		\begin{figure}[H]
		\centering
		\begin{tikzpicture}
		\begin{groupplot}[
			group style={group size=2 by 1, horizontal sep=1.7cm},
			width=0.5\linewidth, height=0.44\linewidth,
			axis lines=left,
			tick label style={font=\scriptsize},
			label style={font=\scriptsize, align=center},
			legend style={draw=none, fill=none, font=\scriptsize,
				at={(0.5,-0.42)}, anchor=north},
			legend cell align=left,
		]
			\nextgroupplot[
				xlabel={learned feature $z_{1}$\\(a) $21$ of the $361$ days},
				ylabel={learned feature $z_{2}$},
				xmin=-7.5, xmax=6.5, ymin=-5.5, ymax=5.5]
			\addplot[only marks, mark=*, mark size=1.6pt, black,
				restrict expr to domain={\thisrow{s}}{1:1}]
				table[col sep=comma, x=z1, y=z2]{sensattr_points_plot.csv};
			\addlegendentry{cold half of the year (filled circles)}
			\addplot[only marks, mark=square, mark size=1.7pt, black,
				restrict expr to domain={\thisrow{s}}{0:0}]
				table[col sep=comma, x=z1, y=z2]{sensattr_points_plot.csv};
			\addlegendentry{warm half of the year (open squares)}
			\addplot[black, thick]
				table[col sep=comma, x=z1, y=z2]{sensattr_boundary.csv};
			\addlegendentry{decision boundary (solid line)}
			\nextgroupplot[
				xlabel={number $\featuredim'$ of learned features\\(b) recovery accuracy},
				ylabel={recovery accuracy},
				xtick={1,...,10}, xmin=0.5, xmax=10.5, ymin=0.4, ymax=1.02]
			\addplot[black, thick]
				table[col sep=comma, x=kept, y=acc_all]{sensattr_accuracy.csv};
			\addlegendentry{all measurements (solid curve)}
			\addplot[black, dashed]
				table[col sep=comma, x=kept, y=acc_notemp]{sensattr_accuracy.csv};
			\addlegendentry{no temperature (dashed curve)}
			\addplot[black, dotted] coordinates {(0.5,0.507) (10.5,0.507)};
			\addlegendentry{majority-class rate (dotted line)}
		\end{groupplot}
		\end{tikzpicture}
		\caption{An attribute that is no feature, recovered from
			learned features of the weather at Krems an der Donau.
			(a) Every $18$th of the $361$ days of $2024$ in calendar
			order, $21$ days, placed by two learned features and
			marked by the half of the year they belong to, the cold one
			(filled circles) or the warm one (open squares), with the
			decision boundary (solid line) of the linear classifier
			learned from $180$ of the $361$ days. (b) The recovery
			accuracy \eqref{equ_sensattr_recovery_dict} against the number
			$\featuredim'$ of learned features, for all eight daily
			measurements (solid curve) and with every temperature
			measurement dropped (dashed curve), against the
			majority-class rate (dotted line). Data generated by
			\texttt{pythondemos/sensattr.py}
			\label{fig_sensattr_dict}}
		\end{figure}
		A sensitive attribute is easier to measure than to remove. The
		linear classifier $\learnthypothesis^{(\sensattr)}$ uses a single
		direction of the learned feature space, the one given by its
		learned model parameters $\widehat{\weights}$. The
		projection of the learned features onto the
		orthogonal complement of that direction, the
		nullspace of the linear map $\vz \mapsto
		\widehat{\weights}^{\top} \vz$, removes it and no more. For ten
		learned features of the days at Krems, a linear classifier
		learned afresh on the projected
		features still reaches $0.84$ on one of the twenty splits,
		against $0.93$ before the projection, so the other
		directions carry the attribute too.
		Learning a feature transformation under a penalty term for
		the recovery of $\sensattr$ attacks the same problem where the
		features are learned (Zemel et al., 2013). Every one of
		these steps needs $\sensattr$ to be recorded, which is why the
		EU AI Act lets a provider of a high-risk artificial intelligence system (high-risk AI system)
		process special categories of personal data when that is
		strictly necessary for detecting and correcting bias, under
		safeguards (European Parliament and Council of the European Union, 2024, Art.~10(5)).

\subsection{Cross References}
fairness, trustworthy AI, privacy protection, privacy leakage, feature learning, PCA, linear classifier, GDPR, EU AI Act, personal data, nullspace, accuracy

\subsection{References}
\begin{enumerate}
  \item European Parliament and Council of the European Union (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance). Official Journal of the European Union, L series, Jul. 12. URL: https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng.
  \item European Parliament and Council of the European Union (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (Text with EEA relevance). Official Journal of the European Union, L~119/1, May 4. URL: https://eur-lex.europa.eu/eli/reg/2016/679/oj.
  \item Barocas et al. (2023). Fairness and Machine Learning: Limitations and Opportunities. MIT Press.
  \item Dwork et al. (2012). Fairness through awareness. In: Proceedings of the 3rd innovations in theoretical computer science conference, pp.~214--226.
  \item Zemel et al. (2013). Learning fair representations. In: International Conference on Machine Learning, pp.~325--333.
\end{enumerate}

\end{document}
