[Access original publication on EDPB website]

SUPPORT POOL OF EXPERTS PROGRAMME

AI-Complex Algorithms and effective Data Protection Supervision - Bias evaluation by Dr. Kris SHRISHAK

1. STATE OF THE ART FOR BIAS EVALUATION

Artificial intelligence (AI) systems are socio-technical systems whose behaviour and outputs can harm people. Bias in AI systems can harm people in various ways. Bias can result from interconnected factors that may together amplify harms such as discrimination (European Union Agency for Fundamental Rights, 2022; Weerts et al., 2023). Mitigating bias in AI systems is important and identifying the sources of bias is the first step in any bias mitigation strategy.

1.1 Sources of bias

The AI pipeline involves many choices and practices that contribute to biased AI systems. Biased data is just one of the sources of biased AI systems, and understanding its various forms can help to detect and to mitigate the bias. In one application, the lack of representative data might be the source of bias, e.g., medical AI where data from women with heart attacks is less represented than men in the dataset. In another, the proxy variables that embed gender bias might be the problem, e.g., in résumé screening. Increasing the dataset size for women could help in the former case, but not in the latter case.

In addition to bias from data, AI systems can also be biased due to the algorithm and the evaluation. These three sources of bias are discussed next.

1.1.1 Bias from data

1. Historical bias: When AI systems are trained on historical data, they often reflect societal bias which are embedded in the Out-of-date datasets with sensitive attributes and related proxy variables contribute to historical bias. This can be attributed to a combination of factors: how and what data were collected and the labelling of the data, which involves subjectivity and the bias of the labeller. An example of historical bias in AI systems has been shown with word embedding (Garg et al., 2018), which are numerical representations of words and are used in developing text generation AI systems.
2. Representation bias: Representation bias is introduced when defining and sampling from the target population during the data collection process. Representation bias can take the form of availability bias and sampling
   1. Availability bias: Datasets used in developing AI systems should represent the chosen target population. However, datasets are sometimes chosen by virtue of their availability rather than their suitability to the task at hand. Available datasets often underrepresent women and people with disabilities. Furthermore, available datasets are often used out of context for purposes different from their intended purpose (Paullada et al., 2021). This contributes to biased AI
   2. Sampling bias: It is usually not possible to collect data about the entire target population. Instead, a subset of data points related to the target population is collected, selected and used. This subset or sample should be representative of the target population for it to be relevant and of high For instance, data collected from scraping Reddit or other social media sites are not randomized and are not representative of the population that don’t use these sites. Such data are not generalizable for wider population beyond these sites. And yet, the data are used in AI models deployed in other contexts.

When defining the target population, the subgroups with sensitive characteristics should be considered. An AI system built using a dataset collected from a city will only have a small percentage of certain minority groups, say 5%. If the dataset is used as- is, then the outputs of this AI system will be biased against this minority group because they only make up 5% of the dataset and the AI system has relatively less data to learn from about them.

1. Measurement bias: Datasets can be the result of measurement bias. Often, the data that is collected is a proxy for the desired This proxy data is an oversimplification of the reality. Sometimes the proxy variable itself is wrong. Furthermore, the method of measurement, and consequently, the collection of the data may vary across groups. This variation could be due to easier access to the data from certain groups over others.
2. Aggregation bias: False conclusions may be drawn about individuals or small groups when the dataset is drawn from the entire The most common form of this bias is Simpson’s paradox (Blyth, 1972) where patterns observed in the data for small groups disappear when only the aggregate data over the entire population is considered. The most well-known example of this comes from the UC Berkeley admissions in 1973 (Bickel et al., 1975). Based on the aggregate data, it seemed that women applicants were rejected significantly more than men. However, the analysis of the data at the department level revealed that the rejection rates were higher for men in most departments. The aggregate failed to reveal this because a higher proportion of women applied to departments with low overall acceptance rate than they did to departments with high acceptance rate.

1.1.2 Algorithm bias

Although much of the discussion around bias focusses on the bias from data, other sources of bias that contribute to discriminatory decisions should not be overlooked. In fact, AI models reflect biased outputs not only due to the datasets but also due to the model itself (Hooker, 2021). Even when the datasets are not biased and are properly sampled, the algorithmic choices can contribute to biased decisions. This includes the choice of objective functions, regularisations, how long the model is trained, and even the choice of statistically biased estimators (Danks & London, 2017).

The various trade-offs made during the design and development process could result in discriminatory outputs. Such trade-offs can include model size and the choice of privacy protection mechanisms (Ferry et al., 2023; Fioretto et al., 2022; Kulynych et al., 2022). Even with Diversity in Faces (DiF) dataset that has broad coverage of facial images, an AI model trained with certain differential privacy techniques disproportionately degrades performance for darker-skinned faces (Bagdasaryan et al., 2019). Furthermore, techniques to compress AI models can disproportionally affect the performance of AI models for people with underrepresented sensitive attributes (Hooker et al., 2020).

1.1.3 Evaluation bias

The performance of AI systems is evaluated based on many metrics, from accuracy to “fairness”. Such assessments are usually performed against a benchmark, or a test dataset. Evaluation bias arises at this stage because the benchmark itself could contribute to bias.

AI systems can perform extremely well against a specific test dataset, and this test performance may fail to translate into real-world performance due to “overfitting” to the test dataset. This is especially a problem if the test dataset carries over historical, representation or measurement bias.