Unit 7 Consolidation: Bias, Fairness, and Inclusion

Hint: Consider both the stated purpose (e.g., providing health information) and the broader societal goal (e.g., improving community health equity). Think about who the system is designed for and who might be excluded.

Solution design: The human purpose should include both the functional goal (providing accurate, timely health information) and the fairness goal (ensuring equitable access and outcomes for all residents, regardless of language, location, or ability). A well-defined purpose might include: "To empower all residents with accessible, accurate, and culturally relevant health information to support informed decision-making and improve health outcomes."

Hint: Think beyond direct users. Consider family members, community health workers, health educators, Indigenous communities, rural residents, people with disabilities, and non-English speakers.

Solution design: A comprehensive stakeholder map should include: (1) direct users (residents seeking health information); (2) non-users (those who cannot access the system due to language, disability, or connectivity barriers); (3) indirect stakeholders (family members, community health workers, public health officials); (4) those whose data is used (the training data sources); and (5) those responsible for oversight (health educators, administrators).

Hint: Use the ten-stage bias entry-point map: problem definition, data/sources, prompt, context, model output, retrieval, human interpretation, decision, monitoring, remedy.

Solution design: Possible entry points include: (1) Problem definition — defining health information without considering cultural diversity; (2) Data/sources — training data dominated by Western medical perspectives; (3) Prompt — English-only interface; (4) Context — no consideration of rural connectivity; (5) Model output — responses that omit Indigenous health practices; (6) Human interpretation — health educator reviewing only a random sample; (7) Decision — using postal codes as a proxy; (8) Monitoring — no tracking of satisfaction by subgroup; (9) Remedy — no contestability process.

Hint: Consider cultural, linguistic, geographic, and ability representation. Look at the examples the AI generates — whose experiences are treated as typical?

Solution design: Representation analysis should note that the AI's examples predominantly feature urban, middle-class, non-Indigenous populations. Indigenous perspectives, rural experiences, and non-English speakers are absent or underrepresented. This omission can lead to representational harm and may cause users from these groups to feel excluded or to receive less relevant information.

Hint: Look for assumptions about users' language, literacy, connectivity, cultural background, and health knowledge.

Solution design: Default assumptions include: all users are fluent in English; all users have reliable internet access; all users are comfortable with text-based interfaces; all users share Western health beliefs; all users have similar health literacy levels. Each assumption excludes or disadvantages certain groups.

Hint: Equality = same treatment; equity = different treatment based on need; fairness = justifiable outcomes; inclusion = meaningful participation; accessibility = usability by all.

Solution design: Equality: all users receive the same AI responses. Equity: users with limited English or literacy need simplified or translated responses. Fairness: the system should provide comparable quality of information across groups. Inclusion: rural, Indigenous, and disabled users should be involved in design and testing. Accessibility: the system should work with screen readers and offer audio alternatives.

Hint: Consider equal access, equal treatment, equitable support, comparable error rates, consistent procedures, transparency, individualized consideration, and human-review availability.

Solution design: Important criteria include: equal access (everyone can use the system), equitable support (additional help for those with barriers), comparable error rates (accuracy should not vary by language or location), and transparency (clear explanations). Conflicts may arise: providing equitable support (e.g., translations) may be resource-intensive and could reduce equal access if not implemented carefully.

Hint: Consider a user who is both non-native English speaking and has a visual impairment, or a rural Indigenous user with limited connectivity.

Solution design: An Indigenous user living in a rural area with limited English proficiency faces a triple barrier: cultural relevance (Indigenous health perspectives are absent), language (English-only), and connectivity (rural internet may be slow). These intersect to make the system nearly unusable, creating a compounded disadvantage that is not captured by examining each barrier alone.

Hint: Look at the postal code used for service recommendations. Also consider language preference and device type.

Solution design: Postal code is a strong proxy for socioeconomic status and race. Using it to recommend services may direct users to different quality of services based on their neighbourhood. Language preference may proxy for national origin or cultural background. Device type may proxy for income level (older or cheaper devices may indicate lower income).

Hint: Consider accuracy, completeness, usefulness, and user satisfaction across language, location, and ability groups.

Solution design: Differential performance is evident in user satisfaction surveys (rural and non-native English speakers report lower satisfaction). The AI may produce less accurate or less complete responses for these groups due to training data imbalances or language limitations. The system may also be less useful for users with visual impairments due to the lack of screen-reader compatibility.

Hint: Consider representational, allocation, quality-of-service, procedural, relational, and capability harms.

Solution design: Harms include: representational (Indigenous and rural perspectives are absent); allocation (users from different postal codes may receive different quality of service recommendations); quality-of-service (lower satisfaction and usefulness for non-native speakers); procedural (no clear contestability pathway); relational (erosion of trust in the health authority); capability (users may receive incorrect information, limiting their health decision-making).

Hint: Look at the differential performance and proxy variables. Are certain groups consistently disadvantaged?

Solution design: The system appears to have a disparate impact on rural residents, non-native English speakers, and Indigenous communities. These groups receive lower-quality information and are less satisfied with the service. The use of postal codes as a proxy for service recommendations may further amplify these disparities, even though no discriminatory intent is present.

Hint: Consider health educators, community workers, and users who might treat the AI's responses as authoritative without critical evaluation.

Solution design: Health educators and community workers may rely on the AI's responses without verifying them, especially for routine questions. If the AI provides less accurate or less complete information for certain groups, this bias will be propagated into the guidance given to those communities. Users themselves may also accept the AI's responses as definitive, without questioning their relevance or accuracy.

Hint: Consider affected communities, health workers, accessibility experts, and Indigenous health organizations.

Solution design: Establish a community advisory panel that includes Indigenous health representatives, rural residents, non-native English speakers, and people with disabilities. Conduct co-design workshops to understand their needs and priorities. Involve health educators and community health workers in testing and monitoring. Ensure that participation is compensated and that feedback is acted upon transparently.

Hint: Consider when human review should be triggered (e.g., complaints, complex cases, high-stakes information).

Solution design: Human review should be triggered when: (1) a user requests it; (2) the AI's response is flagged by a community health worker; (3) the system produces a response that involves high-stakes information (e.g., symptoms that could indicate a serious condition); (4) there are patterns of complaints from a particular group. Reviewers should be trained health educators with cultural competency and language skills.

Hint: Include notice, explanation, correction, human review, challenge, response, remedy, and documentation.

Solution design: Users should be able to: (1) know that AI was used; (2) obtain an understandable explanation of the response; (3) correct inaccurate information; (4) request human review; (5) challenge a response through a formal process; (6) receive a timely response; (7) obtain remedy if appropriate. All contestations should be documented for monitoring.

Hint: Consider correction of information, reconsideration, restoration of access, and human reassessment.

Solution design: Remedies should include: (1) correction of inaccurate information in the system; (2) reconsideration of a response by a human health educator; (3) restoration of access for users who were excluded; (4) an apology and explanation when errors are identified; (5) workflow changes to prevent recurrence.

Hint: Include performance differences, error types, complaint patterns, accessibility barriers, and override rates.

Solution design: Key indicators: (1) user satisfaction scores by language, location, and disability status; (2) complaint rates by subgroup; (3) accuracy of responses for different query types; (4) accessibility test results; (5) human review outcomes; (6) remedy effectiveness. Monitor these quarterly and trigger a reassessment if disparities exceed a threshold.

Hint: Use the six operational elements: fairness goals, stakeholders, inclusion/accessibility, differential effects, evidence/evaluation, and redress/reassessment.

Solution design: See the worked example in Lesson 7.4 for the structure. For this case: fairness goals = equitable health information for all residents; stakeholders = users, non-users, health educators, Indigenous communities; inclusion = involve affected communities; differential effects = performance varies by language and location; evidence = collect disaggregated data; redress = contestability and remedy pathways.

Hint: Consider disparities that cannot be explained, accessibility barriers, lack of contestability, and repeated complaints.

Solution design: Stop conditions: (1) if user satisfaction for any group falls below a threshold for two consecutive quarters; (2) if accessibility barriers prevent meaningful use for any group; (3) if contestability is not functional (e.g., complaints are not resolved within 30 days); (4) if the system materially affects health outcomes (e.g., users act on incorrect information). Escalation should occur when disparities are identified and require deeper investigation.

Hint: Options: Proceed, Proceed with safeguards, Pilot with monitoring, Limit the use, Redesign, Require specialist review, Provide a non-AI pathway, Pause, Do not proceed, Retire.

Solution design: Recommendation: Redesign and Pilot with monitoring. The system has significant fairness issues (representation, accessibility, proxy variables, differential performance) that require redesign before full deployment. A pilot with the proposed participation, contestability, and monitoring measures should be conducted, with a commitment to stop if disparities persist. A non-AI pathway (e.g., human-staffed health information line) should be maintained.

Correct answer: b.

Explanation: Equality means sameness; equity means fairness that accounts for different starting points and barriers.

Correct answers: All of the above (a–f).

Explanation: Bias can arise at every stage of the workflow.

Answer: A proxy variable is a measurable attribute that correlates with a protected characteristic. It is relevant because even if explicit demographics are removed, proxies can still produce biased outcomes.

• 1 — b (Allocation: unequal distribution)
• 2 — a (Representational: stereotyping/erasure)
• 3 — c (Procedural: opaque processes)

Answer: False.

Explanation: Disparate impact can occur through proxy variables, even without explicit demographic labels.

Answer: Intersectionality recognizes that people have overlapping social identities and barriers that interact. It is important because focusing on a single dimension can miss compounded disadvantages.

Correct answer: a.

Explanation: Notice that AI was used is a key element of contestability. The other options are not sufficient.

Answer: Disclosure is telling someone about a problem; remedy is taking action to fix it. Disclosure alone does not address the harm.

Correct answer: b.

Explanation: FAIRER is a systematic evaluation framework, not a technical formula or a guarantee of perfection.

Answer: A stop condition is a predefined circumstance under which a system should be paused or withdrawn. It is important because it provides a clear mechanism for responding to serious fairness issues.

Answer: False.

Explanation: Good intentions do not guarantee fair outcomes; systematic evaluation of outcomes is essential.

Correct answer: a.

Explanation: Automation bias is the tendency to over-rely on automated outputs without critical evaluation.

Answer: Meaningful participation ensures that affected stakeholders have a genuine voice and influence, helping to identify issues that designers might miss and building trust and accountability.

Correct answers: a, b, and d.

Explanation: c is about timing, not differential performance across groups or contexts.

Answer: FAIRER builds on these frameworks: ORBIT provides purpose and workflow, SCOPE identifies omissions and evidence gaps, PARTNER guides human-AI role allocation. FAIRER adds a systematic fairness evaluation, stakeholder participation, contestability, remedy, and ongoing monitoring.

Answer: Allowed recommendations: Proceed, Proceed with safeguards, Pilot with monitoring, Limit the use, Redesign, Require specialist/community review, Provide a non-AI pathway, Pause, Do not proceed, Retire. Each represents a different level of confidence and action, from full deployment to complete withdrawal.