Learning Objectives
- Identify sources of bias in AI training data and explain how they propagate to model outputs
- Articulate the core principles of responsible AI: fairness, accountability, transparency
- Recognize the key AI governance frameworks: EU AI Act, NIST AI RMF, ISO/IEC 42001
Where Bias Comes From
AI systems learn from data — and data reflects the world as it is, including its historical inequalities, underrepresentation, and measurement limitations. Bias enters AI in several ways:
Representation bias: The training data under-represents certain groups. A medical AI trained primarily on data from one demographic will perform less accurately on others.
Measurement bias: The way data is collected or labeled introduces systematic error. If loan default is used as a proxy for creditworthiness, and default rates are affected by discriminatory lending practices, the resulting model inherits those biases.
Aggregation bias: A single model is applied uniformly to subgroups that have different underlying patterns. A one-size-fits-all diabetes prediction model may perform well on average but poorly for specific populations.
Evaluation bias: The benchmarks used to assess AI performance don't reflect real-world diversity. A model that scores highly on benchmarks derived from predominantly English text may perform poorly in other languages.
⚠️Warning
Compounding harms: Bias in AI systems deployed at scale causes outsized harm because decisions are made automatically, at speed, across millions of people. A hiring algorithm that subtly deprioritizes women's resumes affects far more people than a single biased hiring manager.
Responsible AI Principles
The AI industry and government regulators have converged on a set of core principles for responsible AI development and deployment:
| Principle | What It Means |
|---|---|
| Fairness | Systems should not discriminate or produce disparate outcomes across demographic groups |
| Accountability | Clear ownership for AI decisions; humans remain responsible for AI-driven outcomes |
| Transparency | Users should know when AI is making decisions that affect them; explainability for high-stakes domains |
| Privacy | Data minimization; consent; protection of sensitive information |
| Safety | Systems should not cause harm; guardrails against misuse |
| Human oversight | Meaningful human review for high-stakes decisions (hiring, lending, healthcare, law enforcement) |
Key Governance Frameworks
EU AI Act (2024)
The world's first comprehensive AI regulation, passed by the European Union. It takes a risk-based approach:
- Prohibited AI: Social scoring, real-time biometric surveillance in public spaces, manipulative AI
- High-risk AI: Must meet transparency, data governance, and human oversight requirements (e.g., AI in hiring, credit, healthcare, law enforcement)
- General-purpose AI models: Large models (above certain compute thresholds) face transparency and copyright compliance requirements
NIST AI Risk Management Framework (AI RMF)
Developed by the US National Institute of Standards and Technology — a voluntary framework for organizations to identify, assess, and manage AI risks. Organized around four functions: Govern, Map, Measure, Manage.
ISO/IEC 42001
The first international management system standard for AI, providing organizations with a framework to demonstrate responsible AI practices — similar to how ISO 27001 works for information security.
Case Studies in AI Failure
COMPAS recidivism scoring: A risk assessment tool used in US criminal sentencing was found to incorrectly flag Black defendants as higher risk at nearly twice the rate of white defendants with similar histories.
Amazon's hiring tool: Amazon built a resume-screening AI trained on historical hiring decisions — which had been predominantly male. The model learned to penalize resumes that included the word "women's" (as in "women's chess club") and downgraded graduates of all-women's colleges.
Medical AI and skin conditions: Several dermatology AI models trained predominantly on lighter skin tones performed significantly worse at detecting conditions on darker skin tones.
✅Tip
If you are building or evaluating AI systems, ask these questions early: What is the training data? Who is represented and who is missing? What are the failure modes? Who is harmed if the system is wrong? What is the appeals process?
Key Takeaways
- Bias in AI comes from bias in data — representation, measurement, aggregation, and evaluation bias
- Responsible AI is built on fairness, accountability, transparency, privacy, safety, and human oversight
- The EU AI Act is the most comprehensive AI regulation globally; NIST AI RMF provides a voluntary US framework
- Real-world AI failures have caused measurable harm to individuals and groups — the stakes of getting this right are high
