Table of Contents
- 1. Lesson Overview
- 2. Learning Outcomes
- 3. Before You Begin
- 4. Retrieval Practice
- 5. Opening Scenario
- 6. Proxy Variables
- 7. Differential Performance
- 8. Forms of Harm
- 9. Disparate Impact
- 10. Distinguishing Intent from Outcome
- 11. Automation Bias and Amplification
- 12. Cautious Interpretation of Synthetic Evidence
- 13. Process Walkthrough
- 14. Worked Example
- 15. Guided Learning Activities
- 16. Exercises
- 17. Homework Problems
- 18. Mini-Project
- 19. Self-Check Quiz
- 20. Assignment Connections
- 21. Key Takeaways
- 22. Key Terms
- 23. Looking Ahead
- 24. Sources & Further Reading
1. Lesson Overview
This lesson examines how AI systems may perform differently across groups, contexts, or input types—and how these differences can lead to unequal outcomes even when there is no discriminatory intent. You will learn to identify proxy variables that may correlate with protected characteristics, distinguish group representation from system performance, and recognise disparate impact when seemingly neutral processes produce materially unequal effects.
You will also explore multiple forms of harm—allocation, quality-of-service, representational, procedural, relational, and capability—and examine how automation bias can amplify differential performance. The lesson emphasises cautious interpretation of synthetic comparison evidence and the importance of distinguishing intent from outcome.
2. Learning Outcomes
By the end of this lesson, you should be able to:
- Identify proxy variables that may correlate with protected characteristics.
- Distinguish group representation from system performance (i.e., presence vs. quality of service).
- Recognize differential performance and unequal error distribution across groups or contexts.
- Examine allocation, quality-of-service, representational, procedural, relational, and capability harms.
- Identify disparate impact and distinguish it from discriminatory intent.
- Interpret synthetic comparison evidence cautiously, acknowledging limitations.
- Analyze how automation bias can amplify unequal effects.
3. Before You Begin
- Assigned reading: Chapter 9: Bias, Fairness, and Inclusion (sections on differential performance, proxy variables, disparate impact, and forms of harm).
- Prerequisite knowledge: Lessons 7.1 and 7.2 (bias types, representation, fairness criteria, accessibility).
- Optional AI tool: You may use an approved generative AI with synthetic, public information; supplied outputs are available.
- Privacy reminder: Do not enter personal or confidential data. Use only synthetic or public examples.
- Academic integrity: Record your work for Assignments 1 and 3.
- Accessibility: All activities include non-AI options. Contact coordinator for support.
- Estimated preparation time: 20–30 minutes for reading and retrieval.
4. Retrieval Practice: Concepts from Lessons 7.1 and 7.2
These questions draw on Lessons 7.1 and 7.2. Answer each before reading the feedback.
Question 1: Provide a brief example where equal treatment of all users leads to inequitable outcomes, and explain why.
Show answer
Example: An AI-powered learning platform gives all students the same set of practice problems. Students with less prior knowledge or different learning styles may struggle, while others breeze through. Equal treatment does not account for different starting points, leading to unequal learning gains—an inequitable outcome.
Question 2: List three types of accessibility barriers that can affect AI-generated content.
Show answer
- Visual: Lack of alternative text, small fonts, poor contrast.
- Language: Content only in one language, complex vocabulary.
- Cognitive: Dense text, lack of summaries, time-pressured interactions.
Question 3: Why is it important to consider intersectionality when evaluating fairness?
Show answer
Answer: Intersectionality recognises that people have overlapping identities and barriers that interact. Focusing on a single dimension (e.g., gender) may miss compounded disadvantages (e.g., gender + disability + language). Fairness analysis must consider these interactions to avoid overlooking harms.
5. Opening Scenario: "AI-Powered Resume Screening for Internships"
A company uses an AI system to screen internship applications. The system scores each applicant based on a model trained on historical internship data. The tool uses a variety of features, including:
- University name and ranking
- Grade point average
- Previous internship experience
- Extracurricular activities (e.g., leadership roles, volunteer work)
- Geographic location (postal code)
Initial analysis shows that applicants from certain postal codes consistently receive lower scores, even when they have similar qualifications. Additionally, the system seems to favour candidates from elite universities, while applicants from community colleges or foreign institutions receive lower scores.
The company's diversity and inclusion team is concerned that the system may be perpetuating biases, even though the model does not explicitly use race, gender, or other protected characteristics.
Questions to consider:
- What proxy variables might be at play in this system?
- How might the system's performance differ across groups (e.g., by institution type, geographic location)?
- Is there a potential for disparate impact? If so, what form might it take?
- What evidence would be needed to determine whether the system is fair?
Show guided analysis
1. Proxy variables: Postal code may correlate with socioeconomic status and race. University name and ranking may correlate with family background and access to resources. These proxies can indirectly reveal protected characteristics, even though they are not explicit.
2. Differential performance: The system may perform worse for applicants from non-elite institutions, non-Western countries, or lower-income areas—producing higher error rates (i.e., lower scores) for those groups.
3. Disparate impact: If the system consistently filters out applicants from certain backgrounds, it may have a disparate impact on those groups, even without discriminatory intent. This could lead to allocation harm (fewer internships) and representational harm (perpetuating homogeneity).
4. Evidence needed: Analysis of score distributions across demographic groups (anonymised), validation against actual performance, and qualitative review of the features used.
6. Proxy Variables
A proxy variable is a measurable attribute that may correlate with or indirectly reveal another characteristic, often a protected or sensitive one. Proxies can unintentionally encode bias when they are used as stand-ins for characteristics that we wish not to discriminate against.
Common proxy variables in AI systems include:
- Postal code / geographic location — correlates with race, socioeconomic status, and access to resources.
- University attended — may reflect family income, legacy status, or geographic access.
- Employment gaps — may correlate with gender (caregiving) or health conditions.
- Language preference — may indicate national origin or cultural background.
- Device type or browsing patterns — may reflect income level or digital literacy.
- Name — can be associated with race, ethnicity, or gender.
- Schedule availability — may reflect caregiving responsibilities (often gendered).
- Proxy variables correlate with protected characteristics and can encode bias indirectly.
- Common proxies include postal code, university, employment gaps, and language.
- Removing explicit labels does not eliminate proxy-based bias.
7. Differential Performance
Differential performance occurs when an AI system's quality, accuracy, usefulness, or accessibility varies across groups, languages, domains, or input types. This can manifest as:
- Recognition accuracy — speech recognition works worse for certain accents.
- Translation quality — machine translation produces more errors for some language pairs.
- Response completeness — the system provides shorter, less detailed responses for certain query types or user groups.
- Tone — the system may use different tones (e.g., more formal or less supportive) for different users.
- Refusal rate — the system may refuse to answer certain types of questions more often for some groups.
- Hallucination rate — the system may generate more incorrect or fabricated information in certain contexts.
- Accessibility — the system's interface may be less usable for people with disabilities.
- Latency — response times may be slower for users in certain regions or with certain devices.
- Level of detail — explanations may be more thorough for some topics or users than others.
- Recommendation quality — the system may provide better recommendations for some user profiles.
It is important to not invent performance statistics. Use synthetic examples and qualitative descriptions. When evaluating differential performance, consider: Who experiences lower quality? What are the consequences of these differences?
- Differential performance refers to variation in system quality across groups or contexts.
- It can manifest in many ways: accuracy, tone, latency, accessibility, and more.
- Use synthetic examples; do not invent real statistics.
8. Forms of Harm
AI systems can cause harm in multiple ways. Understanding these categories helps in designing effective mitigations.
| Type of harm | Description | Example |
|---|---|---|
| Representational harm | Stereotyping, erasure, demeaning portrayal, or exclusion of certain groups. | AI-generated images of CEOs are predominantly white and male. |
| Allocation harm | Unequal distribution of opportunity, service, benefit, or burden. | An AI screening tool disproportionately rejects applicants from certain postal codes. |
| Quality-of-service harm | Lower-quality performance for some users or contexts. | A voice assistant works less well for users with non-native accents. |
| Procedural harm | Opaque, inconsistent, inaccessible, or non-contestable processes. | No explanation is given for why an application was rejected, and no appeal is possible. |
| Relational harm | Damage to trust, dignity, professional relationships, care, or community authority. | A medical AI undermines the patient-clinician relationship by providing incorrect information. |
| Capability harm | Loss of opportunity to learn, develop expertise, participate, or exercise agency. | Over-reliance on AI recommendations stifles students' critical thinking skills. |
- Six forms of harm: representational, allocation, quality-of-service, procedural, relational, capability.
- Harms often co-occur; classifying them helps design targeted mitigations.
- Not all harms are immediately visible—some manifest over time or through relationships.
9. Disparate Impact
Disparate impact refers to a materially unequal effect that occurs even when a rule, process, or system appears neutral and discriminatory intent is not established. It is an outcome-based measure of fairness.
Key points:
- Intent is not required — a system can have a disparate impact even if the designers had no discriminatory intent.
- Neutral criteria can be problematic — variables that seem relevant (e.g., education level) may disproportionately disadvantage certain groups.
- Disparate impact is context-dependent — what constitutes a "material" difference depends on the domain, the stakes, and the availability of alternatives.
- Legal and ethical considerations — many jurisdictions recognise disparate impact as a form of discrimination, but the specific legal tests vary. In this course, we treat it as an ethical and practical concern, not a legal conclusion.
- Disparate impact is about outcomes, not intent.
- Neutral criteria can still produce unequal effects.
- Assessing disparate impact requires disaggregated data and contextual analysis.
10. Distinguishing Intent from Outcome
It is essential to separate intent (what the designers intended) from outcome (what actually happens). A system can be designed with the best intentions but still produce unequal outcomes due to:
- Historical patterns in the data.
- Unintended correlations between features and protected characteristics.
- Interaction effects that were not anticipated.
- Human interpretation and use of the system.
- Intent and outcome are distinct; good intent can still produce harmful outcomes.
- Outcome-based evaluation is essential for detecting disparate impact.
- Systems should be evaluated on what they actually do, not only on what they were meant to do.
11. Automation Bias and Amplification of Unequal Effects
Automation bias—the tendency to over-rely on automated outputs—can amplify differential performance and disparate impact. If users defer to an AI system without critical evaluation, any biases in the system become magnified in decision-making.
Mitigating automation bias requires:
- Training users on the limitations of AI.
- Providing transparency about how scores are generated.
- Encouraging human review and independent judgment.
- Designing interfaces that reduce over-reliance (e.g., not presenting AI scores as definitive).
- Automation bias can amplify differential performance into larger harms.
- Mitigations include training, transparency, and interface design.
- Human reviewers must be empowered to question and override AI outputs.
12. Cautious Interpretation of Synthetic Comparison Evidence
In this course, you will work with synthetic data and qualitative scenarios. When interpreting such evidence, remember:
- Synthetic data is not real data — it is designed to illustrate concepts, not to prove system-wide bias.
- One output does not prove bias — patterns must be observed across many instances and contexts.
- Aggregate performance can conceal important differences — an overall high accuracy may mask low accuracy for a subgroup.
- Correlation is not causation — observing a difference between groups does not automatically tell us the cause.
- Evidence must be combined with stakeholder input and context — quantitative differences must be interpreted in light of real-world impact.
- Synthetic evidence illustrates concepts; it does not prove real-world bias.
- Aggregate metrics can hide subgroup differences.
- Quantitative evidence must be combined with stakeholder input.
13. Process Walkthrough: Inputs, Proxies, Performance Differences, and Stakeholder Impacts
This walkthrough applies the concepts to the resume-screening scenario. It shows how to systematically examine inputs, proxies, differential performance, and harms.
| Stage | Action | Fairness check | Intervention |
|---|---|---|---|
| 1. Input features | List all features used: university, GPA, experience, extracurriculars, postal code. | Which features could be proxies? (postal code, university ranking). | Conduct a proxy audit; consider removing or reweighting problematic features. |
| 2. Performance differences | Measure score distributions across groups (e.g., by institution type, geographic region). | Are there systematic differences in scores? Are error rates (e.g., false rejections) unequal? | Analyse differential performance; adjust thresholds or model if needed. |
| 3. Identify harms | Classify potential harms: allocation (fewer interviews), procedural (opaque criteria), representational (stereotyping). | Who bears the burden? Who benefits? | Mitigate with transparent explanations and contestability. |
| 4. Assess disparate impact | Compare outcomes (interview invitation rates) across groups. | Is there a materially unequal effect that cannot be justified? | Redesign or adjust criteria to reduce impact. |
| 5. Examine intent vs. outcome | Review design documentation to understand intent, and compare with observed outcomes. | Does good intent mask harmful outcomes? | Focus on outcome-based evaluation, not just intent. |
| 6. Automation bias risk | Consider how recruiters might use the scores. | Do recruiters treat scores as definitive? Is there a human override? | Train recruiters on AI limitations; provide alternative ways to review applicants. |
- Systematically examine inputs, performance, harms, and disparate impact.
- Distinguish intent from outcome.
- Consider how automation bias may amplify effects.
14. Worked Example: AI-Powered Translation Service for Public Health Information
Context: A public health agency uses an AI translation service to convert health advisories into multiple languages. The service is used to reach diverse communities.
Concern: The translation quality is not uniform across languages. Some languages (e.g., Spanish, French) have high accuracy, while others (e.g., Indigenous languages, less-resourced languages) have frequent errors and omissions. This could lead to unequal access to critical health information.
Worked Differential Impact Analysis
- Input features: The translation model is trained on large corpora for some languages, but limited data for others.
- Performance differences: Translation accuracy (BLEU scores, human evaluation) is significantly lower for Indigenous languages compared to major European languages. Error types include mistranslations of medical terms and loss of nuance.
- Proxy variables: Language itself is not a protected characteristic, but it strongly correlates with cultural identity and access to services. Using an AI system that performs worse for certain languages may disproportionately affect those communities.
- Forms of harm:
- Quality-of-service harm: Lower accuracy and completeness for some language groups.
- Allocation harm: Communities with lower-quality translations may miss critical health information, affecting their ability to make informed decisions.
- Representational harm: The system implicitly devalues certain languages and cultures by providing inferior service.
- Disparate impact: Even though the agency intends to serve all communities equally, the unequal translation quality leads to disparate impact on speakers of less-resourced languages.
- Intent vs. outcome: The agency's intent is to provide inclusive information, but the outcome is unequal access.
- Automation bias: Staff may assume that all translations are equally reliable, leading to over-reliance on inaccurate translations.
- Mitigations:
- Invest in improving translation models for under-resourced languages.
- Provide human-reviewed translations for critical health information.
- Clearly label translated content to indicate confidence levels.
- Engage community members to validate translations.
- Recommendation: Proceed with safeguards: use high-confidence translations for critical information, supplement with human review, and communicate limitations to users.
Show key insights from this example
- Differential performance can arise from data imbalances.
- Proxy variables (here, language) can lead to unequal outcomes even without explicit bias.
- Multiple harms (quality, allocation, representational) often co-occur.
- Intent does not negate impact; outcome monitoring is essential.
15. Guided Learning Activities
- Purpose: To practise identifying differential performance across groups or contexts.
- Learning outcome: Recognize differential performance and unequal error distribution.
- Estimated time: 20 minutes.
- Materials: Synthetic comparison table (provided).
- Task: Examine the following synthetic performance data for an AI chatbot that answers customer service questions. Identify at least three differences in performance across user groups, and discuss potential consequences.
| Language | Correct response rate | Average response length (words) | User satisfaction (1–5) |
|---|---|---|---|
| English | 92% | 45 | 4.2 |
| Spanish | 88% | 38 | 3.8 |
| Mandarin | 85% | 30 | 3.5 |
| Arabic | 78% | 25 | 3.0 |
Show sample analysis
- Difference 1: Correct response rate declines for non-English languages, with Arabic being lowest (78%). This means Arabic-speaking users receive incorrect information more often.
- Difference 2: Response length is shorter for non-English languages, suggesting less complete answers.
- Difference 3: User satisfaction is lowest for Arabic, correlating with lower accuracy and shorter responses.
- Consequences: Arabic-speaking users may be underserved, leading to frustration and potential misinformation. This is a quality-of-service harm.
- Purpose: To identify potential proxy variables in a given dataset.
- Learning outcome: Identify proxy variables and assess their appropriateness.
- Estimated time: 15–20 minutes.
- Task: A bank is developing an AI model to assess creditworthiness. The dataset includes: age, income, employment status, home ownership, postal code, number of credit cards, and years at current address. Identify at least three variables that could operate as proxies for protected characteristics, and explain the risk.
Show sample audit
- Postal code: May correlate with race, socioeconomic status, and neighbourhood resources.
- Age: May correlate with health status and life stage, and could proxy for age discrimination (a protected characteristic).
- Home ownership: May correlate with wealth and generational privilege, which could proxy for race and class.
- Risk: Using these variables could lead to disparate impact on certain racial, age, or socioeconomic groups, even if the model does not explicitly use protected categories.
- Purpose: To classify real-world or synthetic scenarios into types of harm.
- Learning outcome: Distinguish representational, allocation, quality-of-service, procedural, relational, and capability harms.
- Estimated time: 15 minutes.
- Task: For each scenario, identify the primary type(s) of harm and explain.
- An AI recruitment tool rejects candidates who attended community colleges more often than those from elite universities, even when qualifications are similar.
- A voice assistant consistently misinterprets the speech of users with non-standard accents, providing incorrect responses.
- An AI system recommends higher insurance premiums for residents of certain postal codes without explanation, and residents cannot appeal.
- An AI-powered learning platform provides fewer challenging problems to students from lower-income backgrounds, based on their inferred socioeconomic status.
Show sample classifications
- 1. Allocation harm (unequal access to opportunities) and representational harm (stereotyping of community college graduates).
- 2. Quality-of-service harm (lower accuracy for certain accents).
- 3. Procedural harm (opaque and non-contestable process) and allocation harm (higher premiums).
- 4. Capability harm (reduced learning opportunities) and allocation harm (unequal educational benefit).
- Purpose: To practice distinguishing intent from outcome.
- Learning outcome: Distinguish intent from outcome and identify disparate impact.
- Estimated time: 15 minutes.
- Scenario: A city uses an AI system to prioritise pothole repairs based on citizen reports and road usage data. The system is designed to allocate resources efficiently. However, the system tends to prioritise repairs in wealthier neighbourhoods because residents there are more likely to use the reporting app. Lower-income neighbourhoods with less app usage get fewer repairs, even though their roads may be in worse condition.
- Task: What was the intent of the system? What is the actual impact? Is there disparate impact? Propose a change to align outcomes with intent.
Show sample analysis
Intent: Efficient allocation of repair resources based on reported problems and usage.
Actual impact: Wealthier neighbourhoods receive more repairs because of higher app usage, while lower-income areas are underserved.
Disparate impact: Yes, the system disproportionately benefits wealthier areas, leading to unequal road quality.
Proposed change: Supplement app-based reports with data from city inspections and community outreach to ensure that all neighbourhoods are considered equitably.
- Purpose: To understand how automation bias can amplify unequal effects.
- Learning outcome: Examine how automation bias may amplify unequal effects.
- Estimated time: 15 minutes.
- Scenario: An AI system provides a "risk score" for parole decisions. The system is known to have higher false-positive rates for certain racial groups. Parole officers are given the score and are encouraged to consider it, but they often treat it as the primary factor because it saves time.
- Task: Explain how automation bias could worsen the disparate impact of the AI system. What safeguards could reduce this?
Show sample analysis
Amplification: Because parole officers over-rely on the score, the existing bias (higher false positives for certain groups) is magnified in the final decisions. Officers may not critically examine the score or consider other factors, leading to disproportionately high denial rates for those groups.
Safeguards: Train officers on the system's limitations; require them to document independent reasons for decisions; provide a clear process for appeal; and monitor decision outcomes for disparate impact.
16. Exercises
- Difficulty: Introductory
- Estimated time: ~10 min
- Learning outcome: Identify differential performance across groups.
Task: Give an example of an AI system (real or synthetic) that exhibits differential performance. Describe the groups or contexts in which performance differs and the nature of the difference.
Show sample answer
Example: An AI-powered automated captioning system for video conferencing. It performs well for speakers with standard North American English accents but has higher error rates for speakers with Indian, African, or Australian accents. This differential performance leads to lower-quality captions for those users, affecting accessibility and communication.
- Difficulty: Intermediate
- Estimated time: ~10 min
- Learning outcome: Interpret synthetic comparison evidence cautiously.
Task: You are given a synthetic report showing that an AI system has a 95% accuracy rate overall, but accuracy drops to 80% for users over 65. What conclusions can you draw? What additional information would you need?
Show sample answer
Conclusions: The system appears to perform less well for older users, indicating a potential differential performance. However, we must be cautious: the synthetic data may not reflect real-world conditions, and the cause of the difference is unknown.
Additional information needed: Real-world validation with actual users; understanding the types of errors (e.g., are they more harmful for older users?); investigation into whether the system was trained on representative data; and qualitative feedback from older users.
- Difficulty: Intermediate
- Estimated time: ~10 min
- Learning outcome: Identify possible proxy variables in a given system.
Task: An AI system for college admissions uses the following features: high school GPA, SAT scores, number of Advanced Placement (AP) courses taken, and zip code of the applicant. Which of these could be proxies for socioeconomic status or race? Explain.
Show sample answer
- Zip code: Strong proxy for socioeconomic status and racial composition of neighbourhoods.
- Number of AP courses: May be a proxy because access to AP courses varies by school district, often correlated with income and race.
- SAT scores: Can be a proxy for coaching and test-prep resources, which are correlated with income.
- High school GPA: May be less of a proxy but can still be influenced by grading practices and school quality.
- Difficulty: Intermediate
- Estimated time: ~10 min
- Learning outcome: Classify cases into forms of harm.
Task: For the college admissions system in Exercise 7.3.3, what forms of harm might arise if the system inadvertently disadvantages applicants from lower-income areas? Provide at least three types.
Show sample answer
- Allocation harm: Fewer admissions offers to students from lower-income areas.
- Representational harm: The system may perpetuate stereotypes about certain groups' academic potential.
- Procedural harm: Applicants from disadvantaged backgrounds may not understand why they were rejected, and there may be no clear appeal process.
- Difficulty: Intermediate
- Estimated time: ~10 min
- Learning outcome: Distinguish intent from impact.
Task: A company implements an AI system to automate hiring, with the stated goal of reducing bias by removing human subjective judgment. The system uses a machine learning model trained on historical hiring data. The system appears to favour candidates from certain universities. Discuss the intent vs. the actual impact, and propose a way to reconcile them.
Show sample answer
Intent: Reduce bias by removing human subjectivity.
Actual impact: The system perpetuates historical biases because the training data reflected past hiring patterns that favoured certain universities.
Reconciliation: Audit the training data and model for bias, retrain with more diverse data, and add human oversight to catch and correct discrepancies. Do not assume that replacing humans with AI eliminates bias.
- Difficulty: Advanced
- Estimated time: ~10 min
- Learning outcome: Identify how automation bias can amplify unequal effects.
Task: A hospital uses an AI triage system that assigns patients to levels of urgency. The system has been found to under-triage (assign lower urgency) to patients from certain ethnic backgrounds. Doctors are given the AI recommendation and usually follow it. Explain how automation bias could worsen the health outcomes for that group.
Show sample answer
Amplification: Because doctors trust the AI recommendation, patients from the affected ethnic group are consistently assigned lower urgency, leading to longer waiting times and potentially worse health outcomes. The automation bias means that the system's bias is propagated into care decisions, and without critical review, the disparity becomes systemic.
17. Homework Problems
- Estimated time: ~8 min
Task: Give an example of an AI system that may have differential performance based on user location (e.g., urban vs. rural). Explain the potential consequences.
Show sample answer
Example: An AI-powered weather forecasting app that provides less accurate predictions for rural areas due to fewer data collection stations. Consequences: Farmers in rural areas may receive unreliable forecasts, affecting crop planning and safety.
- Estimated time: ~8 min
Task: You see a claim that "AI system X is fair because it has a 99% accuracy rate overall." Why is this claim insufficient to ensure fairness? What additional information would you need?
Show sample answer
Insufficient: Overall accuracy can hide unequal performance across subgroups. The system could have 100% accuracy for one group and 0% for another, yet still have 99% overall if that group is small.
Additional information needed: Accuracy rates broken down by demographic groups, error types, and the consequences of errors for each group.
- Estimated time: ~10 min
Task: An AI system for mortgage lending uses "credit history length" as a feature. Explain how this could be a proxy for age or other characteristics, and discuss the fairness implications.
Show sample answer
Proxy for age: Younger individuals tend to have shorter credit histories, even if they are creditworthy. This could lead to disparate impact against younger applicants, which may correlate with age discrimination.
Fairness implications: Using credit history length without adjustment may unfairly penalise young people, especially those from disadvantaged backgrounds who may have had less opportunity to build credit.
- Estimated time: ~8 min
Task: An AI system used by a social services agency to allocate benefits is found to provide less detailed explanations to applicants with limited digital literacy. What type of harm is this primarily? Justify.
Show sample answer
Primary harm: Procedural harm. The process is inaccessible and does not provide adequate explanation, which undermines the applicant's ability to understand and challenge decisions.
Justification: The lack of understandable explanation is a procedural fairness issue, and it may also contribute to allocation harm if applicants cannot effectively navigate the system.
- Estimated time: ~8 min
Task: A tech company develops an AI tool to "democratise" access to job recommendations by providing equal access to all users. However, the tool recommends higher-paid jobs to users from certain zip codes. Discuss the intent vs. the impact, and propose a remedy.
Show sample answer
Intent: Provide equal access to job opportunities.
Impact: The tool perpetuates existing inequalities by steering users from wealthier areas to higher-paid jobs, possibly because the training data reflected historical job distributions.
Remedy: Audit the recommendation algorithm for geographic bias, diversify training data, and add a feature that explicitly asks users about their preferences rather than relying solely on historical patterns.
- Estimated time: ~10 min
Task: A government agency uses an AI system to assess eligibility for a program. The system uses a set of criteria that, on the surface, are neutral, but analysis shows that applicants from certain regions are disproportionately denied. What steps should the agency take to investigate and address this?
Show sample answer
- Conduct a disaggregated analysis of approval rates by region and other relevant characteristics.
- Review the criteria for possible proxies that may be driving the disparity.
- Engage with affected communities to understand their experiences.
- Consider adjusting the criteria or adding alternative pathways to eligibility.
- Implement monitoring and a contestability process.
- Estimated time: ~8 min
Task: A financial advisor uses an AI system to recommend investment portfolios. The system has been found to recommend riskier portfolios to certain demographic groups. How could automation bias exacerbate this issue, and what could be done to mitigate it?
Show sample answer
Exacerbation: If the advisor accepts the AI's recommendations without critical review, the biased recommendations will be implemented, leading to disproportionate financial risk for those groups.
Mitigation: Train advisors to question AI recommendations, provide transparency reports on the system's performance, and require justification for any recommendation that deviates from standard practice.
- Estimated time: ~10 min
Task: For the mortgage lending proxy problem (Problem 7.3.3), propose two concrete mitigations to reduce the disparate impact while still using credit history as a feature.
Show sample answer
- Mitigation 1: Include alternative credit data (e.g., rent payments, utility bills) to build a fuller picture of creditworthiness, especially for young applicants.
- Mitigation 2: Use a model that explicitly adjusts for age or credit history length, or set different thresholds for different age groups, ensuring that the system does not unfairly penalise younger applicants.
- Estimated time: ~10 min
Task: Choose a domain (e.g., criminal justice, education, social media content moderation) and describe how differential performance, proxy variables, or disparate impact could arise. Propose a method to detect and mitigate one of these issues.
Show sample answer
Domain: Social media content moderation. An AI system flags and removes posts based on certain keywords. The system may have higher false-positive rates for posts in African American Vernacular English (AAVE) because the training data did not adequately represent that dialect. Detection: Monitor false-positive rates by dialect; Mitigation: Include diverse dialect data in training and use human review for borderline cases.
18. Mini-Project: Differential Impact Assessment
- Purpose: To apply the concepts of differential performance, proxy variables, and disparate impact to a bounded synthetic case.
- Learning outcomes: Identify differential performance; interpret synthetic evidence; identify proxy variables; classify harms; distinguish intent from impact; assess automation-bias risk.
- Estimated effort: 90–120 minutes.
- Context: A city uses an AI system to allocate public housing assistance. The system scores applicants based on factors such as income, family size, housing condition, and length of time on the waiting list. Early data suggests that applicants from certain neighbourhoods receive lower scores, even when other factors are similar. Additionally, the system's explanations are available only in English and are very technical.
Required deliverable: A structured assessment (approximately 600–900 words) with the following sections:
- Use and purpose — what the system is meant to do.
- Comparison groups or contexts — which groups appear to be affected.
- Basis for comparison — what evidence is available (synthetic).
- Performance measure — what is being measured (e.g., scores, approval rates).
- Observed differences — describe the differences.
- Data limitations — what is unknown about the data.
- Possible alternative explanations — other factors that could cause the differences.
- Proxy variables — identify at least two potential proxies in the system.
- Potential harms — classify at least three types of harm.
- Consequences of errors — what happens if the system makes mistakes.
- Automation-bias risk — how might human decision-makers over-rely on the system?
- Affected stakeholders — who is affected?
- Additional evidence required — what more do you need to know?
- Proposed mitigations — at least three concrete changes.
- Monitoring requirement — what to track over time.
- Human-review requirement — when should human review be triggered?
- Contestability requirement — how can applicants challenge decisions?
- Stop condition — when would you pause or stop the system?
- Preliminary conclusion — what is your initial recommendation?
Privacy and ethical boundaries: Use only synthetic information. Do not use real data. Do not infer individual characteristics from group patterns.
Self-review checklist
- Have I identified at least two proxy variables?
- Have I described observed differences and acknowledged data limitations?
- Have I classified at least three forms of harm?
- Have I considered automation-bias risk?
- Have I proposed at least three concrete mitigations?
- Have I defined monitoring, review, contestability, and stop conditions?
- Have I made a clear preliminary conclusion?
Show solution architecture
- Define the scope — clearly describe the system and its purpose.
- Identify comparison groups — e.g., by neighbourhood, income level, language.
- Analyse performance differences — using the synthetic data provided, note any disparities.
- Investigate proxies — examine which features could be proxies for protected characteristics.
- Classify harms — allocation, quality, procedural, representational, etc.
- Assess automation bias — consider how staff might use the system.
- Propose mitigations — adjust criteria, add transparency, involve stakeholders.
- Design monitoring and contestability — set up feedback loops and appeals.
- Make a recommendation — based on the evidence.
Show illustrative partial example (different context)
Context: AI-powered job matching platform.
Observed differences: Candidates from certain regions receive fewer job recommendations.
Proxy variables: Postal code (socioeconomic), university name (elite vs. non-elite).
Harms: Allocation (fewer opportunities), representational (stereotyping of regions), procedural (lack of explanation).
Mitigations: Add regional diversity weights, provide transparent explanations, allow users to correct information.
19. Self-Check Quiz
Question 1: What is a proxy variable in the context of AI fairness?
Show answer and explanation
Correct answer: b.
Explanation: A proxy variable is a measurable attribute that may correlate with or indirectly reveal another characteristic (often a protected one), and its use can lead to biased outcomes. Not all proxies are intentionally discriminatory (a) or always unlawful (c).
Question 2: Which of the following are examples of differential performance? Select all that apply.
Show answer and explanation
Correct answers: a, b, and c. (d is about timing, not differential performance across user groups or contexts.)
Explanation: Differential performance refers to variation in system quality or outcomes across groups or contexts. a, b, and c all describe such variation.
Question 3: Match each harm to its description.
| Harm | Description |
|---|---|
| 1. Allocation harm | A. Lower-quality service for some users |
| 2. Quality-of-service harm | B. Unequal distribution of opportunities or benefits |
| 3. Representational harm | C. Stereotyping or erasure of groups |
Show answer and explanation
- 1 — B (Allocation harm: unequal distribution)
- 2 — A (Quality-of-service harm: lower quality)
- 3 — C (Representational harm: stereotyping/erasure)
Question 4: What is the difference between disparate impact and discriminatory intent?
Show answer and explanation
Answer: Disparate impact refers to a materially unequal effect that occurs even without discriminatory intent. Discriminatory intent involves a deliberate desire to treat a group unfairly. A system can have a disparate impact even if it was designed with good intentions, because bias can be embedded in data, proxies, or context.
Question 5: True or false: If an AI system does not use explicit demographic information (e.g., race, gender), it cannot produce biased outcomes.
Show answer and explanation
Answer: False.
Explanation: Bias can arise through proxy variables that correlate with protected characteristics, as well as through data imbalances, model design, and human interpretation. Removing explicit labels does not guarantee fairness.
Question 6: How can automation bias amplify the effects of differential performance?
Show answer and explanation
Answer: Automation bias leads users to over-rely on AI outputs without critical evaluation. If the AI has differential performance (e.g., higher error rates for certain groups), automation bias causes those errors to be propagated into decisions, magnifying the unequal impact.
Question 7: Which of the following is an example of a procedural harm?
Show answer and explanation
Correct answer: b.
Explanation: Procedural harm involves opaque, inconsistent, or non-contestable processes. a is quality-of-service, c is allocation, d is representational.
Question 8: When interpreting synthetic comparison evidence, what are two key cautions to keep in mind?
Show answer and explanation
- Caution 1: Synthetic data may not reflect real-world complexity; patterns observed may not generalise.
- Caution 2: Correlation does not imply causation; observed differences may have alternative explanations.
20. Assignment Connections
This lesson provides critical evidence for Assignment 3. Preserve:
- Proxy variable analysis — which variables could be proxies and why.
- Differential performance findings — observed differences and their potential causes.
- Harm classification — allocation, quality, procedural, representational, relational, capability.
- Disparate impact assessment — whether there are unequal effects and how to measure them.
- Automation-bias risk — how human use could amplify issues.
- Mitigations — proposed changes to reduce differential impact.
Reflect on the following prompts for your Unit 7 journal entry.
Show reflection prompts and planning checklist
- How has your understanding of fairness changed after learning about proxy variables and disparate impact?
- Think of an AI system you have used. What proxies might have been operating? Were you aware of them?
- How can good intentions lead to harmful outcomes? Give an example from your experience or a hypothetical.
- What role does evidence play in assessing differential performance? How would you gather evidence responsibly?
- How does automation bias affect your own use of AI? How can you guard against it?
Reflection planning checklist:
- I have identified at least one proxy variable in a system I know.
- I have distinguished intent from impact in an example.
- I have reflected on a form of harm I might not have noticed before.
- I have considered how automation bias could affect my decision-making.
21. Key Takeaways
- Proxies can hide bias: Variables that correlate with protected characteristics can produce unequal outcomes even without explicit demographic data.
- Performance varies: AI systems often perform differently across groups, languages, or contexts—and these differences matter.
- Harm takes many forms: Allocation, quality-of-service, representational, procedural, relational, and capability harms all require attention.
- Disparate impact is about outcomes, not intent: A system can be unfair even if no one intended discrimination.
- Automation bias amplifies: Over-reliance on AI can magnify differential performance and disparate impact.
- Interpret evidence cautiously: Synthetic data and aggregate metrics can mislead; always seek contextual and disaggregated analysis.
22. Key Terms
- Proxy variable
- A measurable attribute that correlates with or indirectly reveals another characteristic, often a protected one.
- Differential performance
- Variation in system quality, accuracy, or usefulness across groups, languages, or contexts.
- Disparate impact
- A materially unequal effect that occurs even when a process appears neutral and discriminatory intent is absent.
- Allocation harm
- Unequal distribution of opportunities, benefits, or burdens.
- Quality-of-service harm
- Lower-quality performance for some users or contexts.
- Representational harm
- Stereotyping, erasure, or demeaning portrayal of groups.
- Procedural harm
- Opaque, inconsistent, or non-contestable processes.
- Relational harm
- Damage to trust, dignity, relationships, or community authority.
- Capability harm
- Loss of opportunity to learn, participate, or exercise agency.
- Automation bias
- The tendency to over-rely on automated system outputs.
23. Looking Ahead: Lesson 7.4
In Lesson 7.3, you have examined differential performance, proxy variables, and disparate impact. The final lesson in this sequence, Lesson 7.4: FAIRER, Participation, Contestability, and Remedy, will bring together all the concepts from Unit 7 into a comprehensive fairness review framework. You will learn to apply FAIRER, design meaningful participation, build contestability pathways, and make justified recommendations.
After completing Lesson 7.4, you will be ready to undertake the Unit 7 cumulative project and prepare your Assignment 3 case analysis.
24. Sources & Further Reading
- Primary reading: Chapter 9: Bias, Fairness, and Inclusion, COMP 3270 textbook.
- Supporting: Appendix D (Responsible-Use Toolkit) and Appendix E (AI Workflow Design Toolkit).
- External: [VERIFICATION REQUIRED] — please confirm any additional sources with the course coordinator.
Some textbook content, definitions, examples, and figures are marked with placeholders. These will be supplied by the course coordinator or subject-matter expert.