A few years ago, for example, Amazon introduced an AI-supported tool for the automated evaluation of applications.
The aim of the tool was to screen the CVs of applicants and then assign them a score based on the advertised position. The advantage of such a tool is obvious: instead of laboriously sifting through dozens of applications by hand, the algorithm can be fed with numerous CVs and identify the top candidates for the job within a very short time. [1]
That sounds great so far. But what Amazon had not considered was that the tool's rating system was not neutral: Amazon found that female candidates were systematically rated lower than their male colleagues by the algorithm. Male applicants, on the other hand, were automatically given a higher rating by the system, which led to a gender-discriminatory evaluation at the expense of female applicants. [2]
The question that now arises is: How can this be? After all, a computer has no subjective feelings and should therefore be able to make purely neutral decisions based on the parameters provided.
It's not quite that simple: an AI system does not make fair decisions by definition. In this article, I would like to show you why this may be the case and why AI cannot simply be used as a savior for all problems.
What does fairness mean in the context of AI?
Let's first take a look at what is meant by fairness in the context of AI. In this context, an algorithm can be described as fair if its decisions are free of prejudice against an individual or a group. Conversely, for an unfair algorithm, this means that it makes its decisions at the expense of a certain group of people. In this context, the biases relate to so-called "sensitive attributes" such as gender or religion. [3]
But how does such unfairness arise in AI systems?
In order to investigate the cause of such a bias effect, it is important to look at the training data on which the algorithm is based. Since algorithms use training sets to recognize correlations, the quality of the algorithms' decisions is strongly linked to the test data sets. If there is already a bias in the data used for training, the AI system adopts this pattern in its decisions. This means that existing biases in the training data are adopted by AI systems and lead to deviations. [4]
Using Amazon as an example, there was also a problem with the underlying training data: Here, the applications submitted to Amazon over the past ten years were used. However, since significantly more men than women applied to Amazon and were hired, there was a high imbalance in the training data. This bias to the disadvantage of female applicants led to male applicants being rated higher. Specifically, the algorithm gave lower scores to CVs that contained words such as "women's college" or "women's chess club". The imbalance in the training data therefore meant that the AI system made biased decisions to the detriment of female candidates and therefore rated them unfairly. [5]
According to Amazon, the HR tool was adapted several times after the problems became known, but was ultimately discontinued and discarded. The IT company also emphasized that the project was only tested with internal test data and was never seriously used in the recruiting process [6].
The potential of AI systems is almost limitless. There are numerous use cases in which algorithms can make data-based decisions and thus save a lot of time and money. However, the example of Amazon also shows that the Development of AI systems there are a few factors to consider. An algorithm only works as well as humans have designed and trained it.
More on the use of AI and advanced analytics
Sources
[1] https://t3n.de/news/diskriminierung-deshalb-platzte-amazons-traum-vom-ki-gestuetzten-recruiting-1117076/
[2] https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G)
[3, 4] https://arxiv.org/pdf/1908.09635.pdf
[5] https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G
[6] https://www.theverge.com/2018/10/10/17958784/ai-recruiting-tool-bias-amazon-report
