Table of Contents
- 1. Lesson Overview
- 2. Learning Outcomes
- 3. Before You Begin
- 4. Retrieval Practice
- 5. Opening Scenario
- 6. Risk and Its Factors
- 7. Consequences of Error
- 8. Detectability and Reversibility
- 9. Proportionality and Safeguards
- 10. Introductory Risk Categories
- 11. When AI is Unsuitable
- 12. Decision‑Process Walkthrough
- 13. Worked Example
- 14. Guided Learning Activities
- 15. Exercises
- 16. Homework Problems
- 17. Mini‑Project
- 18. Self‑Check Quiz
- 19. Assignment 1 Journal Connection
- 20. Key Takeaways
- 21. Key Terms
- 22. Looking Ahead
- 23. Sources & Further Reading
1. Lesson Overview
In Lesson 3.1 you learned to start with a human problem, define a legitimate purpose, and assess preliminary suitability. This lesson moves to the next critical step: analysing risk. Even if a task is technically suitable, it may still carry unacceptable risks if the consequences of error are severe, the information is sensitive, or the effects cannot be detected or reversed.
You will learn to identify possible consequences of error, assess information sensitivity, evaluate detectability and reversibility, consider scale and stakeholder vulnerability, and apply proportionality to select safeguards. You will also learn to distinguish lower‑risk, moderate‑risk, higher‑risk, and unsuitable uses at an introductory level. By the end, you will be able to determine whether an AI use should proceed, proceed with safeguards, be redesigned, or be replaced by a non‑AI alternative.
Generative AI can produce plausible outputs for almost any task, but the risks vary dramatically by context. The same AI capability used to brainstorm ideas (low risk) could be unacceptable when used to make clinical decisions (high risk). Learning to assess risk systematically helps you avoid harm, maintain trust, and use AI only where it is genuinely appropriate.
2. Learning Outcomes
By the end of this lesson, you should be able to:
- Identify possible consequences of inaccurate, incomplete, biased, outdated, or misused AI output.
- Assess risk using consequence, information sensitivity, scale, detectability, reversibility, stakeholder vulnerability, and level of AI autonomy.
- Distinguish lower‑risk, moderate‑risk, higher‑risk, and unsuitable AI uses at an introductory level.
- Explain proportionality in the design of safeguards.
- Match risk characteristics with proportionate safeguards.
- Determine when a non‑AI alternative is preferable to an AI approach.
- Justify a preliminary risk classification and safeguard plan.
3. Before You Begin
- Required reading: Chapter 3 of the course textbook (sections on risk, consequence, proportionality, and safeguards).
- Prior knowledge: You should be familiar with the AI Paradox and ORBIT from Unit 1, and with the problem‑purpose‑suitability analysis from Lesson 3.1.
- Optional: If you have access to an institution‑approved generative AI system, you may use it for the guided activities. A supplied‑output alternative is provided for every activity.
- Privacy reminder: Do not enter personal, confidential, proprietary, or sensitive information into any AI system for these activities. Use only synthetic, public, or authorized information.
- Academic integrity: All work you submit for credit must be your own. Using AI to generate complete answers for graded work without proper attribution is a violation of academic integrity.
- Estimated preparation time: 15–20 minutes for the reading and retrieval practice.
4. Retrieval Practice
These questions activate knowledge from Lesson 3.1. Answer them before expanding the feedback.
Q1. What is the difference between technical capability and contextual suitability?
Show feedback
Sample answer: Technical capability refers to what the AI system can produce (e.g., generate text, summarise, answer questions). Contextual suitability considers whether that capability is appropriate for the specific task, given the purpose, stakeholders, information, consequences, and available safeguards. A capability may exist without being suitable for a particular use.
Q2. Why is it important to consider non‑AI alternatives before selecting an AI approach?
Show feedback
Sample answer: Considering alternatives prevents the "solution in search of a problem" trap and ensures that the chosen approach is genuinely the best fit for the human purpose. Non‑AI approaches may be more reliable, transparent, or cost‑effective in many cases.
Q3. In Lesson 3.1, what factors are used to assess AI suitability?
Show feedback
Sample answer: Clarity of purpose, legitimacy of purpose, information authority, consequence of error, ease of verification, reversibility, scale, affected stakeholders, required expertise, need for human relationships, professional or community authority, and availability of safeguards.
5. Opening Scenario
A large language model (LLM) is capable of generating clear, structured text based on a set of input data. Two different teams in the same organisation propose to use this capability:
- Team A: Use the LLM to draft internal meeting notes from an audio transcript. The draft will be reviewed by a team member before sharing.
- Team B: Use the same LLM to generate a summary of a client's financial records, which will be sent directly to the client without human review (to save time).
The technology is the same, but the risk classification differs significantly.
Questions to consider:
- Why does the same AI capability carry different levels of risk in these two scenarios?
- What factors make Team B's use riskier?
- What safeguards could be added to Team B's use to reduce the risk?
- Would a non‑AI alternative be preferable for either scenario?
Show guided analysis
1. Why does the same AI capability carry different risk levels?
Risk depends on context: the consequence of error, the sensitivity of the information, the scale of impact, the detectability of errors, and the reversibility of actions. Team A's use is internal, low‑stakes, and reversible (review before sharing). Team B's use involves sensitive financial information, a direct external action, no human review, and potentially serious consequences if the summary is inaccurate.
2. Factors making Team B's use riskier:
- Consequence of error: Financial loss, reputational harm, legal liability.
- Information sensitivity: Client financial records are highly sensitive.
- Scale: Could affect many clients if repeated.
- Detectability: The client may not detect errors; the organisation may not review.
- Reversibility: Once sent, it is difficult to retract; may require formal corrections.
- Autonomy: No human review means the AI is acting autonomously in a consequential context.
3. Safeguards for Team B: Require human review before sending; use a verified AI system; include clear disclaimers; provide a way for clients to challenge the summary; restrict the use to low‑risk clients initially.
4. Non‑AI alternatives: For Team B, a human‑prepared summary or a rules‑based report generator might be more reliable and transparent. The AI approach should only be considered if it can match human accuracy and is subject to oversight.
6. Risk and Its Factors
Risk is the uncertainty concerning possible adverse outcomes. In the context of AI, risk arises from the gap between what the AI can do and what is safe, fair, or appropriate in a given context. Risk assessment involves identifying potential harms and evaluating their likelihood and severity.
Key factors to consider include:
- Consequence of error: What is the worst that could happen if the AI output is wrong?
- Information sensitivity: How sensitive is the information being used?
- Scale: How many people or decisions are affected?
- Vulnerability: Are any stakeholders particularly vulnerable to harm?
- Detectability: Can an error be identified before it causes harm?
- Reversibility: Can the effects of an error be corrected or undone?
- AI autonomy: How much decision‑making authority is delegated to the AI?
- Professional authority: Is the AI use constrained by professional or regulatory standards?
- Contestability: Can affected parties challenge the AI's output or decision?
- Remedy: Is there a process to provide redress if harm occurs?
Risk cannot be reduced to a single score. Different factors interact, and the same factor (e.g., consequence of error) may be acceptable in one context but unacceptable in another. A nuanced, qualitative assessment is essential.
7. Consequences of Error
Consequences can range from minor inconvenience to severe harm. When evaluating a proposed AI use, consider both direct and indirect consequences:
| Category | Examples |
|---|---|
| Minor inconvenience | Wasted time, confusion, need to re‑do work |
| Misinformation | Incorrect public statements, misleading internal reports |
| Privacy exposure | Unauthorised disclosure of personal or confidential information |
| Reputational harm | Damage to individual or organisational reputation |
| Financial loss | Loss of money, missed opportunities, penalties |
| Exclusion or discrimination | Unfair treatment of individuals or groups |
| Lost educational opportunity | Students receiving incorrect guidance or feedback |
| Professional error | Mistakes in medicine, law, engineering, finance, etc. |
| Denial of service | Inability to access essential services or benefits |
| Safety risk | Physical harm to people or property |
| Legal or regulatory consequence | Lawsuits, fines, regulatory sanctions |
| Erosion of trust | Loss of confidence in the AI system, the organisation, or the profession |
| Capability loss | Over‑reliance on AI leading to atrophy of human skills |
Direct consequences are immediate (e.g., a wrong recommendation leads to a bad decision). Indirect consequences can spread over time (e.g., a biased AI system perpetuates inequality in hiring). Both must be considered.
8. Detectability and Reversibility
Two key factors in risk assessment are detectability (can an error be identified before it produces an effect?) and reversibility (can the output, decision, or action be corrected or undone?).
- Private brainstorming: errors are easy to spot and correct.
- Draft document: can be revised before finalisation.
- Internal memo: can be retracted and corrected.
- Public communication: errors can spread widely and be hard to retract.
- Record modifications: incorrect entries may persist.
- High‑impact decisions: may require formal appeals and remedies.
Combinations matter:
- Easy to detect, easy to reverse: Low risk (e.g., a draft that is reviewed).
- Easy to detect, difficult to reverse: Moderate risk (e.g., a public statement that can be retracted but still causes reputational damage).
- Difficult to detect, easy to reverse: Risk if detection comes too late (e.g., an error in a data analysis that is later corrected, but decisions were made in the interim).
- Difficult to detect, difficult to reverse: High risk (e.g., a medical diagnosis that is not challenged and leads to incorrect treatment).
They exist on a spectrum. The feasibility of detection depends on the availability of independent evidence, the expertise of the reviewer, and the time available. Reversibility may involve costs, effort, and residual harm even after correction.
9. Proportionality and Safeguards
Proportionality means that the safeguards implemented should reflect the seriousness and nature of the risk. A low‑risk use requires lighter safeguards; a high‑risk use requires stronger, more robust measures.
Examples of proportionate safeguards:
- Low‑risk: Ordinary human review, basic fact‑checking.
- Moderate‑risk: Source verification, specialist review, approval gates, restricted information access.
- Higher‑risk: Independent audit, formal approval processes, logging, appeal mechanisms, rollback capabilities, and in some cases, prohibition.
Even low‑risk uses require some responsibility. The safeguards should be adequate to the risk, but there is no "zero‑responsibility" zone. Every use of generative AI should be governed by clear accountability and transparency.
When designing safeguards, consider:
- Who will review the output? What expertise do they need?
- What evidence is required to verify accuracy?
- What approval gates are needed?
- How will errors be logged and corrected?
- What is the fallback plan if the AI output is unusable?
- How will affected parties be informed and given a chance to challenge?
10. Introductory Risk Categories
For instructional purposes, we use four categories: Lower Risk, Moderate Risk, Higher Risk, and Unsuitable / Prohibited. These are not formal legal classifications but tools to guide thinking.
| Category | Typical characteristics | Example uses |
|---|---|---|
| Lower Risk | Authorised public/synthetic information; bounded task; reversible output; straightforward checking; low consequence; no external action. | Brainstorming ideas, drafting internal memos, generating study questions, organising personal notes. |
| Moderate Risk | Internal information; public communication; meaningful operational effect; interpretation required; moderate financial or reputational consequence; stronger human review required. | Drafting public‑facing communications, summarising internal reports, providing preliminary analysis for professional review. |
| Higher Risk | Significant effect on rights, health, safety, education, employment, finances, legal status, public benefits, or essential services; sensitive information; vulnerable stakeholders; difficult reversibility; professional or regulatory duty. | Medical diagnosis support, legal advice, financial recommendations, hiring decisions, eligibility determinations. |
| Unsuitable / Prohibited | Illegitimate purpose; absent authority; uncontrolled sensitive information; high‑impact decision without qualified human authority; no correction or appeal; irreversible action; unacceptable residual risk. | Autonomous weapons, unverified medical treatment, unauthorised data processing, decisions without human oversight. |
Do not treat these categories as formal legal or regulatory classifications. They are teaching tools to help you think about risk. Actual risk assessment may require more nuanced analysis and may be guided by institutional policies or professional standards.
11. When AI is Unsuitable
There are situations where generative AI should not be used, regardless of technical capability. These include:
- Illegitimate purpose: The use is unlawful, unethical, or violates human rights.
- Absent authority: The person or organisation does not have the right to use the information or make the decision.
- Uncontrolled sensitive information: The information cannot be adequately protected, and the risk of breach is unacceptable.
- High‑impact decision without qualified human authority: The decision requires professional judgment that cannot be delegated.
- No correction or appeal: There is no process to correct errors or challenge outcomes.
- Irreversible action: The action cannot be undone, and the cost of error is too high.
- Unacceptable residual risk: Even with safeguards, the remaining risk is too great.
Declining to use AI in a given context is not a failure; it is a sign of professional judgment. Responsible use includes knowing when not to use AI.
12. Decision‑Process Walkthrough
The following diagram illustrates the risk assessment stage of the responsible‑use decision process.
Stages in this lesson:
- Identify possible errors — What could go wrong with the AI output?
- Assess consequences — How serious could the consequences be?
- Evaluate information sensitivity — Is the information sensitive or protected?
- Consider scale and vulnerability — How many people are affected? Are any vulnerable?
- Assess detectability and reversibility — Can errors be detected and corrected?
- Determine AI autonomy — How much decision‑making power is delegated to the AI?
- Classify risk — Assign a preliminary risk category.
- Design proportionate safeguards — Select controls appropriate to the risk level.
- Evaluate if a non‑AI alternative is preferable — Is the AI approach still the best option?
- Make a revised preliminary recommendation — Proceed with safeguards, redesign, or do not proceed.
Risk assessment is not about eliminating all risk (which is impossible) but about ensuring that risks are understood, managed, and proportionate to the benefits. The goal is to make an informed decision about whether and how to use AI.
13. Worked Example
The following example builds on the meeting‑minutes use case from Lesson 3.1 and adds a risk assessment.
The project manager from Lesson 3.1 is now assessing the risks of using AI to draft meeting minutes. The AI would generate a draft from a meeting recording, and the manager would review and correct it before distribution.
1. Possible errors
- The AI may misinterpret or omit key decisions.
- Action items may be assigned to the wrong person or lack clear deadlines.
- Quotes or statements may be misattributed.
- The tone may be inappropriate for the audience.
- The AI may hallucinate details that were not discussed.
2. Consequences of error
- Minor: Team members may be confused, requiring follow‑up clarifications.
- Moderate: A critical action item may be missed, delaying a project milestone.
- Severe: A misrepresentation could harm stakeholder trust or lead to a wrong decision.
- Direct: Wasted time for the manager to correct errors.
- Indirect: Erosion of team confidence in the manager's communication.
3. Information sensitivity
Meeting content is internal and not highly sensitive (no personal or confidential client data). However, it may contain strategic or performance‑related information that should not be disclosed outside the organisation.
4. Scale and vulnerability
The minutes are shared with a team of 10–15 people. No particular vulnerabilities, though junior team members may rely on the minutes for action items.
5. Detectability
The manager was present at the meeting and can compare the AI draft with their own notes and recollection. This makes errors fairly detectable, provided the manager has time to review carefully.
6. Reversibility
The minutes are reversible: corrections can be made before distribution. Even after distribution, corrections can be sent, though they may cause confusion.
7. AI autonomy
The AI is generating a draft, not making decisions. The manager retains full control over the final content. Autonomy is low.
8. Risk classification
Moderate risk. The consequences of error are moderate (potential project delays, confusion), but detectability and reversibility are good, and autonomy is low. The use is not high‑risk because it does not affect rights, health, or significant financial decisions.
9. Proportionate safeguards
- Mandatory human review: The manager must review the entire draft before distribution.
- Source verification: The manager should check key decisions and action items against their own notes.
- Approval gate: The manager must explicitly approve the minutes before sending.
- Fallback: If the AI draft is unusable, the manager can revert to manual note‑taking.
- Logging: The manager keeps a record of AI use and corrections.
10. Non‑AI alternative comparison
The non‑AI alternative (manual minutes) is reliable but time‑consuming. The AI approach could save time if the manager is willing to invest in review. The safeguards above ensure that the AI approach does not compromise accuracy.
11. Revised preliminary recommendation
Proceed with safeguards. The risk is moderate, but with strong human review and clear fallback procedures, the AI use is acceptable. The manager should pilot the approach with a few meetings and monitor the quality of the drafts.
Show discussion of the worked example
This example illustrates how risk assessment is not a simple "yes/no" but a nuanced evaluation of multiple factors. The moderate‑risk classification led to a set of proportionate safeguards that make the use responsible. The manager also considered the non‑AI alternative and confirmed that the AI approach, with safeguards, offers a net benefit.
Note that if the meeting minutes involved highly sensitive information (e.g., legal strategy) or if the manager did not have time for review, the risk classification might shift to higher risk or unsuitable.
14. Guided Learning Activities
- Purpose: Practise identifying risk factors in different scenarios.
- Learning outcome: Assess risk using consequence, information sensitivity, scale, detectability, reversibility, and autonomy.
- Estimated time: 20 minutes.
Instructions: For each scenario, identify the key risk factors and provide a brief justification.
- A teacher uses AI to generate quiz questions from course material.
- A journalist uses AI to summarise a public government report for an article.
- A financial analyst uses AI to recommend stock trades, which are then executed without human review.
Show model analysis for #1
- Consequence: Low to moderate (students may learn incorrect information).
- Information sensitivity: Low (course material is not sensitive).
- Scale: Could affect many students if used across a course.
- Detectability: Moderate (teacher can review questions for accuracy).
- Reversibility: High (questions can be corrected before use).
- Autonomy: Low (teacher selects and reviews questions).
- Overall: Lower to moderate risk, with safeguards (teacher review).
Show model analysis for #2
- Consequence: Moderate (misinformation in a published article).
- Information sensitivity: Low (public report).
- Scale: Potentially large (public audience).
- Detectability: Moderate (fact‑checking possible but time‑consuming).
- Reversibility: Difficult (public retraction may be needed).
- Autonomy: Low (journalist reviews and edits).
- Overall: Moderate risk; strong editorial review and fact‑checking required.
Show model analysis for #3
- Consequence: High (financial loss, regulatory issues).
- Information sensitivity: High (client financial data).
- Scale: Potentially large (multiple clients).
- Detectability: Low (trades executed quickly, errors may be noticed too late).
- Reversibility: Very difficult (trades may be irreversible).
- Autonomy: High (no human review before execution).
- Overall: Higher risk / unsuitable; human review and approval are essential.
- Purpose: Classify scenarios according to detectability and reversibility.
- Learning outcome: Assess detectability and reversibility in context.
- Estimated time: 15 minutes.
Instructions: For each scenario, determine whether errors are (a) easy/difficult to detect and (b) easy/difficult to reverse. Explain your reasoning.
- AI‑generated draft of a job posting, reviewed by HR before publication.
- AI‑generated translation of a patient's medical history, used directly in treatment without review.
- AI‑generated summary of meeting notes, distributed to the team without review.
Show model analysis
- Job posting draft: Detectability – easy (HR can review); Reversibility – easy (corrections before publication).
- Medical translation: Detectability – difficult (requires medical and language expertise); Reversibility – very difficult (wrong treatment could have serious consequences).
- Meeting notes summary: Detectability – moderate (team members may notice errors); Reversibility – moderate (corrections can be sent, but may cause confusion).
- Purpose: Practise classifying uses into lower, moderate, higher, or unsuitable.
- Learning outcome: Distinguish lower‑risk, moderate‑risk, higher‑risk, and unsuitable AI uses.
- Estimated time: 20 minutes.
Instructions: Classify each proposed use and justify your classification.
- Using AI to generate a list of potential research topics for a student paper.
- Using AI to provide personalised learning recommendations to high school students based on their performance data.
- Using AI to automate the grading of multiple‑choice exams.
- Using AI to screen job applications for a large corporation.
Show model analysis
- Research topics: Lower risk – low consequence, reversible, no sensitive data.
- Learning recommendations: Moderate risk – affects educational outcomes; requires human oversight and validation.
- Multiple‑choice grading: Moderate risk – if errors occur, grades could be wrong; but the task is structured and can be verified.
- Screening job applications: Higher risk – affects employment opportunities; risk of bias; requires careful design, oversight, and appeal mechanisms.
- Purpose: Match risk characteristics with appropriate safeguards.
- Learning outcome: Explain proportionality in the design of safeguards.
- Estimated time: 15 minutes.
Instructions: For the scenarios below, suggest proportionate safeguards.
- AI used to generate a first draft of a blog post (to be edited by the author).
- AI used to draft a response to a customer complaint (to be reviewed by a customer service manager).
- AI used to suggest medication doses for a patient (to be reviewed by a pharmacist).
Show model analysis
- Blog post draft: Light review by author, fact‑checking for key claims.
- Customer complaint response: Manager review to ensure tone and accuracy; logging of AI use.
- Medication doses: Extensive safeguards: mandatory pharmacist review, independent verification, logging, and clear accountability; possibly a non‑AI alternative (e.g., standard dosing protocols) may be preferred.
15. Exercises
Apply the concepts from this lesson to the following scenarios.
- Learning outcome: Identify possible consequences of inaccurate AI output.
- Estimated time: 10 minutes.
- Scenario: A non‑profit uses AI to generate summaries of public policy documents to share with community members.
- Task: List at least five possible consequences if the AI summary is inaccurate. Include both direct and indirect consequences.
Show Hint 1: Think about the audience
Community members may rely on the summaries to make decisions or understand their rights.
Show Hint 2: Consider the organisation's reputation
If the summary is wrong, trust in the non‑profit could be damaged.
Show sample solution
- Community members may misunderstand their rights or eligibility for services.
- Misinformation could spread, leading to poor decisions.
- The non‑profit's reputation could be harmed, reducing future trust.
- Staff may spend additional time correcting misinformation.
- Policymakers may receive incorrect feedback on the policy's impact.
- Legal liability if the summary is misleading.
- Learning outcome: Assess detectability and reversibility.
- Estimated time: 10 minutes.
- Scenario: A marketing team uses AI to generate social media posts for a product launch. The posts are reviewed by a team member before posting.
- Task: How detectable and reversible are errors in this context? What if the posts were posted without review?
Show Hint 1: Compare with and without review
Review adds detectability and reversibility before posting. Without review, errors may go public and be harder to retract.
Show sample answer
With review: Detectability is high (reviewer can spot errors); reversibility is high (corrections before posting).
Without review: Detectability is low (errors may only be noticed by audience); reversibility is difficult (retracting a public post may not undo all harm).
- Learning outcome: Classify risk using the introductory categories.
- Estimated time: 10 minutes.
- Scenario: A software development team uses AI to generate code snippets for a non‑critical internal tool. The code is reviewed by a senior developer before integration.
- Task: Classify the risk as lower, moderate, higher, or unsuitable. Justify your answer.
Show Hint 1: Consider the consequence of a coding error
Internal tool, non‑critical, reviewed by senior developer → low consequence.
Show sample answer
Lower risk. The task is bounded, the output is reviewed by an expert, the consequence of error is low (internal tool), and the information is not sensitive. The safeguard (human review) is proportionate.
- Learning outcome: Match risk characteristics with proportionate safeguards.
- Estimated time: 10 minutes.
- Scenario: A health app uses AI to provide general wellness tips based on user‑inputted activity data. The tips are not medical advice, and the app includes a disclaimer.
- Task: What safeguards would be proportionate for this use?
Show Hint 1: Consider the disclaimer and the nature of the tips
If the tips are general and non‑medical, the risk is lower.
Show sample answer
- Transparency: Clearly state that the tips are AI‑generated and not medical advice.
- Human review: Periodically review the AI's tips for accuracy.
- Feedback mechanism: Allow users to report concerns.
- Data privacy: Ensure user data is protected.
- Fallback: Provide a link to human‑created resources.
These safeguards are proportionate to the low‑risk nature of the use.
- Learning outcome: Determine when a non‑AI alternative is preferable.
- Estimated time: 10 minutes.
- Scenario: A small business wants to use AI to generate customised email marketing campaigns for each customer segment. They have limited data and no dedicated marketing staff.
- Task: Compare the AI approach with a non‑AI alternative (e.g., using a standard template). Which is preferable, and why?
Show Hint 1: Consider the quality and personalisation
AI could personalise, but with limited data, the benefit may be small. A standard template might be more reliable and easier to manage.
Show sample answer
Non‑AI alternative: A standard email template with minor customisation (e.g., using the recipient's name). This is reliable, easy to implement, and does not require AI expertise.
AI approach: Could offer more personalisation, but with limited data, the AI may generate generic or irrelevant content. It also requires human review to avoid errors.
Preferable: The non‑AI alternative is preferable because it is simpler, lower‑risk, and likely sufficient for a small business with limited resources.
16. Homework Problems
- Learning outcome: Identify possible consequences of error.
- Scenario: An AI system is used to generate feedback on student essays, which is then sent directly to students without instructor review.
- Task: List at least five consequences of error, both direct and indirect. Which is the most serious, and why?
Show hint
Consider educational outcomes, student motivation, instructor workload, and fairness.
Show sample solution
- Students may receive incorrect feedback, leading to poor learning.
- Students may become demotivated if feedback is unfair or nonsensical.
- Instructors may have to spend time correcting the AI's feedback.
- Grades may be unfairly influenced if feedback is tied to grading.
- Trust in the educational system may be eroded.
The most serious consequence is likely incorrect learning, as it undermines the educational purpose of the feedback.
- Learning outcome: Assess information sensitivity in context.
- Scenario: A human resources department wants to use AI to analyse employee sentiment from internal survey responses.
- Task: What is the sensitivity of this information? What precautions should be taken?
Show hint
Employee survey responses are personal and may contain sensitive opinions.
Show sample answer
Sensitivity: High – employee opinions are personal and may relate to job satisfaction, management, or workplace issues. Misuse could harm trust and morale.
Precautions: Anonymise data, use a secure AI system, limit access to results, ensure compliance with privacy laws, and be transparent with employees about the analysis.
- Learning outcome: Consider scale and stakeholder vulnerability.
- Scenario: A government agency uses AI to review applications for a public benefit programme.
- Task: Who is vulnerable in this context? How does scale affect the risk?
Show hint
Applicants may be low‑income or have limited digital literacy.
Show sample answer
Vulnerable stakeholders: Applicants who rely on the benefit for basic needs; those with limited ability to appeal or challenge decisions.
Scale: If the AI is used for thousands of applications, even a small error rate can affect many people. This amplifies the risk.
- Learning outcome: Assess detectability.
- Scenario: An AI system generates financial risk assessments for a bank's loan officers, who use them to inform lending decisions.
- Task: How detectable are errors in the AI's risk assessments? What factors affect detectability?
Show hint
Loan officers may not have the expertise or time to verify the AI's analysis.
Show sample answer
Detectability: Moderate to low. Loan officers may not have access to the same data as the AI, or may lack the quantitative skills to validate the assessments. Errors may only become apparent after loans default.
Factors: Availability of independent data, expertise of reviewers, time available for review, and the transparency of the AI's reasoning.
- Learning outcome: Assess reversibility.
- Scenario: A university uses AI to determine which students are at risk of dropping out, and automatically sends intervention emails to those students.
- Task: How reversible are the effects of an incorrect identification? What could be done to improve reversibility?
Show hint
Incorrect intervention could be stigmatising or misdirect resources.
Show sample answer
Reversibility: Moderate. An email can be sent, but if it causes emotional distress or stigma, the harm may not be fully reversible. Resources may be wasted on students who don't need help.
Improvements: Provide a way for students to opt out; allow students to contest the identification; combine AI with human counsellor review.
- Learning outcome: Classify risk using introductory categories.
- Scenario: A legal tech startup uses AI to analyse contracts and flag potentially problematic clauses for lawyers to review.
- Task: Classify this use as lower, moderate, higher, or unsuitable. Justify your answer.
Show hint
Legal implications, but lawyers review the flags.
Show sample answer
Moderate risk. The AI is assisting, not replacing, legal judgment. Errors could lead to missed issues, but the lawyer's review provides a safeguard. However, contract analysis involves sensitive information and potentially significant financial consequences.
- Learning outcome: Explain proportionality in the design of safeguards.
- Scenario: An AI system is used to generate personalised workout plans for a fitness app. The plans are not medically supervised.
- Task: Design a set of proportionate safeguards for this use.
Show hint
Consider the risk of injury, the need for disclaimers, and user feedback.
Show sample answer
- Clear disclaimer that the plans are not medical advice.
- Recommend consulting a physician before starting.
- Allow users to adjust intensity.
- Provide a way to report issues.
- Regularly review the AI's recommendations for safety.
- Learning outcome: Determine when a non‑AI alternative is preferable.
- Scenario: A small museum wants to use AI to generate audio guides for exhibits, based on the museum's collection database.
- Task: Compare the AI approach with a non‑AI alternative (e.g., hiring a writer to create scripts). Which is preferable? Justify.
Show hint
Consider the quality of the guide, the cost, and the museum's reputation.
Show sample answer
AI approach: Low cost, scalable, but may produce generic or inaccurate content that lacks the museum's unique voice.
Non‑AI: Higher cost, but ensures quality, accuracy, and reflects the museum's expertise.
Preferable: For a museum, the non‑AI approach is preferable because the audio guide is a key part of the visitor experience and should reflect the institution's authority and care. The AI approach could be used as a draft, but human curation is essential.
- Learning outcome: Recognise situations in which AI should not be used.
- Scenario: An AI system is proposed to provide real‑time mental health crisis support via chat, without human backup.
- Task: Explain why this use is unsuitable. What alternatives should be considered?
Show hint
Mental health crises require human empathy, judgment, and immediate intervention.
Show sample answer
Unsuitable because: High consequence of error (safety risk), requires empathy and professional judgment, irreversible if wrong, and no qualified human authority is present. AI cannot handle the complexity and urgency of a crisis.
Alternatives: Human‑staffed crisis lines, referral to mental health professionals, or AI‑assisted triage that immediately connects users to a human.
- Learning outcome: Apply risk assessment to a new context.
- Scenario: A real estate agency wants to use AI to generate property descriptions for listings from a database of property features.
- Task: Assess the risk of this use. What factors would you consider? What safeguards would you recommend?
Show hint
Property descriptions affect buyer decisions and may have legal implications.
Show sample answer
Risk assessment: Moderate risk. Consequences of error include misleading buyers, potential legal liability, and reputational damage. Information is not highly sensitive, but accuracy is important. Scale could be large if many listings are generated.
Safeguards: Human review of all descriptions before publication; ensure AI is trained on accurate data; include a disclaimer that descriptions are AI‑assisted; provide a way for buyers to ask for clarification.
17. Mini‑Project: Proportionate Risk and Safeguard Plan
- Purpose: Extend the use case from Lesson 3.1 by adding a risk assessment and proportionate safeguards.
- Learning outcomes: Identify risk factors, classify risk, design proportionate safeguards, and determine if a non‑AI alternative is preferable.
- Estimated effort: 60–90 minutes.
Instructions: Use the same use case you developed in Lesson 3.1 (or a new one with instructor approval). Add the following sections to your brief:
- Error scenarios: What could go wrong with the AI output? List at least three specific types of errors.
- Possible consequences: For each error, describe the direct and indirect consequences.
- Information sensitivity: How sensitive is the information the AI will process?
- Scale: How many people or decisions will be affected?
- Affected stakeholders: Who are the stakeholders, and are any particularly vulnerable?
- Detectability: How easily can errors be detected? Who would detect them?
- Reversibility: Can errors be corrected? What is the process for correction?
- AI autonomy: How much decision‑making authority is delegated to the AI?
- Required expertise: What expertise is needed for human review?
- Preliminary risk category: Lower, moderate, higher, or unsuitable? Justify.
- Proportionate safeguards: List specific safeguards that are proportionate to the risk.
- Residual risk: What risk remains after safeguards? Is it acceptable?
- Alternative approach: Compare the AI approach with a non‑AI alternative. Which is preferable?
- Revised preliminary conclusion: Based on the risk assessment, what is your recommendation?
Deliverable: Submit a written plan (approximately 800–1,000 words) that includes all the sections above. You may use the worked example in this lesson as a model.
- Use only synthetic, public, or authorised information.
- Do not use real personal, confidential, or proprietary information.
- Do not use a use case that involves a high‑impact decision unless you have the appropriate expertise and authority.
Show self‑review checklist
- ☐ I have identified at least three types of errors.
- ☐ I have described consequences for each error.
- ☐ I have assessed information sensitivity.
- ☐ I have considered scale and vulnerability.
- ☐ I have evaluated detectability and reversibility.
- ☐ I have determined the level of AI autonomy.
- ☐ I have classified the risk and justified my classification.
- ☐ I have proposed proportionate safeguards.
- ☐ I have discussed residual risk.
- ☐ I have compared the AI approach with a non‑AI alternative.
- ☐ I have made a revised preliminary recommendation.
Show solution architecture
Structure for a strong plan:
- Brief recap of the use case (human problem, purpose, proposed AI role).
- Error scenarios – list 3–5 specific errors.
- Consequences – use a table or bullet list for each error.
- Risk factor analysis – sensitivity, scale, vulnerability, detectability, reversibility, autonomy.
- Risk classification – assign a category and justify.
- Safeguards – list specific, actionable safeguards.
- Residual risk and alternative comparison – discuss remaining risk and compare with non‑AI.
- Revised recommendation – proceed, proceed with safeguards, redesign, or do not proceed.
Note: The plan should be specific to your use case. Avoid generic statements; tailor the safeguards and analysis to the context.
Show illustrative partial example (different context)
Use case: AI‑generated thank‑you letters for donors (from Lesson 3.1).
- Error scenarios: Incorrect donor name, wrong donation amount, inappropriate tone, factual error about the organisation.
- Consequences: Donor may feel undervalued, may not donate again; reputational damage; staff time to correct.
- Sensitivity: Donor names and amounts are private; must be protected.
- Scale: Hundreds of donors; errors could affect many.
- Detectability: High – staff review each letter.
- Reversibility: High – corrections can be made before sending.
- Risk category: Lower to moderate.
- Safeguards: Human review, template validation, donor data protection, and a process to handle corrections.
- Recommendation: Proceed with safeguards.
Note: This is a partial example. Your plan should include all sections.
18. Self‑Check Quiz
Test your understanding of risk, consequence, and proportionality.
Q1. Which of the following is not a factor in risk assessment?
Show answer and explanation
Correct answer: b) The number of tokens in the AI model
Explanation: The number of tokens (context window size) is a technical detail of the model, not a risk factor. Risk factors include consequence, detectability, reversibility, sensitivity, scale, vulnerability, and autonomy.
Q2. A task with high detectability and high reversibility is generally considered:
Show answer and explanation
Correct answer: c) Lower risk
Explanation: When errors are easy to detect and easy to correct, the risk is reduced because harm can be prevented or mitigated. High detectability and reversibility are favourable characteristics.
Q3. Proportionality means:
Show answer and explanation
Correct answer: b) Designing safeguards that are commensurate with the level of risk.
Explanation: Proportionality means matching the intensity of safeguards to the seriousness of the risk. Low‑risk uses need lighter safeguards; high‑risk uses need stronger ones. It does not mean using identical safeguards for all uses, nor eliminating all risk (which is impossible), nor avoiding safeguards.
Q4. Which of the following is an example of a higher‑risk use?
Show answer and explanation
Correct answer: c) Using AI to recommend treatment options for cancer patients, reviewed by oncologists.
Explanation: This use involves health, high consequence of error, and vulnerable stakeholders. Even with review, the stakes are high, making it a higher‑risk use. The others are lower to moderate risk.
Q5. True or false: If an error is difficult to detect, it is always high risk.
Show answer and explanation
Correct answer: False.
Explanation: Difficulty of detection is one factor; the overall risk also depends on consequence, reversibility, scale, and other factors. A difficult‑to‑detect error with low consequence may still be low risk. Risk is multi‑dimensional.
Q6. Which of the following is a characteristic of an unsuitable AI use?
Show answer and explanation
Correct answer: c) There is no process for appeal or correction.
Explanation: An unsuitable use often lacks accountability, correction, or appeal mechanisms, especially when consequences are significant. The other options are generally favourable characteristics.
Q7. A non‑AI alternative may be preferable when:
Show answer and explanation
Correct answer: c) The non‑AI approach is more reliable and the risk of AI error is unacceptable.
Explanation: Reliability and risk are key considerations. If the AI approach cannot achieve acceptable accuracy or introduces unacceptable risk, the non‑AI alternative should be chosen, even if it is slower or less innovative.
Q8. Which of the following is a proportionate safeguard for a moderate‑risk AI use?
Show answer and explanation
Correct answer: b) Mandatory human review by a subject‑matter expert.
Explanation: For moderate risk, human review by someone with relevant expertise is a proportionate safeguard. No review (a) is insufficient; full audit (c) may be excessive for moderate risk; automatic execution (d) is inappropriate.
19. Assignment 1 Journal Connection
The following prompts are designed to help you reflect on risk assessment and proportionality for your Assignment 1 journal.
- How does considering consequence, detectability, and reversibility change the way you think about risk?
- Describe a situation where you might classify a use as moderate risk. What safeguards would you consider proportionate?
- Why is it important to consider non‑AI alternatives even when an AI approach is technically capable?
- Think of a use case that you initially thought was low risk, but after considering scale and vulnerability, you realise it is higher risk. What changed your assessment?
Show reflection planning checklist
- ☐ I have described a specific use case or scenario.
- ☐ I have identified the risk factors and explained my reasoning.
- ☐ I have considered both direct and indirect consequences.
- ☐ I have discussed proportionality and safeguards.
- ☐ I have considered a non‑AI alternative.
- ☐ I have reflected on what I have learned or how my thinking has evolved.
20. Key Takeaways
- Risk is multi‑factorial: Assess consequence, sensitivity, scale, vulnerability, detectability, reversibility, autonomy, and contestability.
- Consequences can be direct or indirect: Consider both immediate and long‑term effects.
- Detectability and reversibility are critical: Errors that are hard to detect or reverse pose greater risk.
- Proportionality means matching safeguards to risk: Low risk → light safeguards; high risk → strong safeguards.
- Risk categories (lower, moderate, higher, unsuitable) are instructional tools: Use them to guide thinking, not as rigid legal labels.
- Some uses are unsuitable: When the purpose is illegitimate, authority is absent, or risk cannot be managed.
- Non‑AI alternatives should be compared honestly: They may be more reliable, transparent, or cost‑effective.
- A responsible decision may be to not use AI: Declining AI use is a sign of good judgment.
21. Key Terms
- Risk
- Uncertainty concerning possible adverse outcomes; the combination of likelihood and consequence.
- Consequence of error
- The adverse effects that may result from an inaccurate, incomplete, biased, or misused AI output.
- Detectability
- The degree to which an error can be identified before it produces an effect.
- Reversibility
- The degree to which an output, decision, or action can be corrected or undone.
- Proportionality
- The principle that safeguards should be commensurate with the level and nature of the risk.
- Safeguard
- A measure implemented to reduce risk, such as human review, verification, approval gates, or fallback procedures.
- Lower‑risk use
- A use with low consequence, high detectability and reversibility, and minimal sensitivity.
- Moderate‑risk use
- A use with moderate consequence, requiring stronger human review and verification.
- Higher‑risk use
- A use with significant consequences, sensitive information, vulnerable stakeholders, or difficult reversal.
- Unsuitable use
- A use that should be avoided due to illegitimate purpose, absent authority, unacceptable residual risk, or inability to provide safeguards.
- Residual risk
- The risk that remains after safeguards have been applied.
22. Looking Ahead
In Lesson 3.3, you will learn about Human Authority, Oversight, Transparency, and Accountability. You will build on the risk assessment from this lesson by determining what decisions must remain human, how to design meaningful oversight, and how to establish transparency and accountability.
Together, Lessons 3.1–3.3 provide the core components of a responsible‑use decision. In Lesson 3.4, you will integrate all these elements into a complete responsible‑use assessment.
After completing Unit 3, you will move to Module 2: Prompt Engineering and Human‑AI Collaboration, starting with Unit 4: Prompt Design Fundamentals.
23. Sources & Further Reading
- Primary reading: Chapter 3: Effective and Responsible AI Use – sections on risk, consequence, proportionality, and safeguards.
- Prerequisite review: Chapter 1 (AI Paradox, ORBIT) and Chapter 2 (technical limitations).
- Supplementary: Appendix D: Responsible‑Use Toolkit for risk assessment checklists.
- Supplementary: Appendix E: AI Workflow Design Toolkit for designing safeguards.