Lesson 3
AI Bias and Fairness: What You Need to Know
AI-generated
AI models learn patterns from their training data. If that data reflects historical biases - and it does - the model will reproduce and sometimes amplify those biases. This is not theoretical. It has been documented extensively across multiple domains.
Hiring tools: Amazon built an AI resume screening tool that penalized resumes containing the word "women's" (as in "women's chess club") because it learned from historical hiring data where men were disproportionately hired. Amazon scrapped the tool.
Facial recognition: Multiple studies have shown that facial recognition systems have higher error rates for darker-skinned faces and women. An MIT study found error rates of 0.8% for light-skinned men versus 34.7% for dark-skinned women.
Healthcare: An algorithm used by US hospitals to allocate healthcare resources was found to systematically under-refer Black patients for additional care. The algorithm used healthcare spending as a proxy for health needs, but Black patients historically had less spent on their care due to systemic inequities.
Language models: LLMs can reproduce stereotypes, generate biased content about certain demographic groups, and reflect the cultural biases present in internet text.
Training data reflects history, and history includes discrimination. If a model learns from decades of news articles, hiring records, or medical data, it learns the patterns in that data - including discriminatory ones.
Measurement choices matter. The healthcare algorithm used spending as a proxy for health needs. This seemed reasonable but embedded a systemic bias. Choosing what to measure and optimize for is a human decision with real consequences.
Underrepresentation in data means underperformance for underrepresented groups. If 90% of training images are of light-skinned faces, the model will be 90% optimized for light-skinned faces.
AI labs are investing in bias detection and mitigation. Anthropic's Constitutional AI approach explicitly includes fairness principles. OpenAI, Google, and Meta all publish research on reducing bias.
Regulation is emerging. The EU AI Act requires bias assessments for high-risk AI applications. Several US states have passed laws requiring bias audits for AI used in hiring.
Independent audits and benchmarks are helping surface problems before they cause harm.
Be aware that AI output may reflect biases, especially in sensitive areas like hiring, healthcare, criminal justice, and lending.
If you are building AI applications, test for bias across demographic groups. Do not assume neutrality.
Report biased behavior when you encounter it. Most AI providers have feedback mechanisms.
AI bias is especially dangerous because it can appear objective. A human decision-maker's biases can be challenged. An algorithm's biases look like neutral, data-driven facts. Always question AI outputs in high-stakes decisions affecting people.
MIT Media Lab: Gender Shades study (facial recognition bias) - https://www.media.mit.edu/projects/gender-shades/
ProPublica: "Machine Bias" - https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
Nature: "Dissecting racial bias in healthcare algorithm" - https://www.nature.com/articles/d41586-019-03228-6
EU AI Act: High-risk AI requirements - https://artificialintelligenceact.eu/
Anthropic: Constitutional AI approach - https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback