Safety, Misuse and Sensitive Topics: Health, Money, Self-Harm and Harassment

Up to now this course has mostly asked who is affected by an AI-assisted task. This page asks a sharper question: how bad could the worst realistic outcome be, and what do you do when a person might be hurt? Most uses of AI carry small risks. A few carry serious ones: a message that tricks someone out of money, an answer that someone acts on for their health, a chat with a person in distress, or a tool that helps decide who gets a job, a loan or treatment.

We keep the tone practical and calm. The descriptions are non-graphic, and the focus is on recognising risk and responding well. This page does not give medical, mental-health or crisis instructions, and it does not explain how to do anything harmful. It teaches you where the line sits and who to turn to. You met checking habits in Trust, Verification and Automation Bias: When to Rely on AI Output and fake media in Deepfakes, Provenance and Disclosure: Telling Real from Synthetic. Here we build on both.

What you will learn

• How an ordinary error differs from a safety-critical failure, and how to rate the worst realistic outcome of a use.
• How AI-assisted scams and impersonation work in outline, and a short routine for responding.
• How to recognise an output or a request that could enable harm, and what "dual use" means.
• Safe boundaries for health questions, and why an AI chat is not a therapist or an emergency service.
• Three high-stakes areas (hiring, credit, medical triage), each reduced to a short list of questions to ask.

Ordinary errors and safety-critical failures

Every AI tool makes mistakes, as you learned in How AI Works: LLMs, Tokens and Limitations. What matters is what a mistake costs. An ordinary error is cheap, visible and easy to undo: a clumsy sentence, a wrong date in a draft that you will proofread. A safety-critical failure is an error where someone could be seriously hurt, or lose money, health, liberty or a basic opportunity, and where the damage is hard to undo.

Invented scenario:

Katja, in Berlin, uses an AI tool to draft two things on the same afternoon. First, a friendly
reminder to her sports club that training moves to Thursday. Second, a one-page safety notice
for volunteers who will lift heavy equipment at a charity sale. The tool gets Thursday wrong
(it writes Tuesday). It also writes a lifting instruction that sounds sensible but that Katja
has no way of checking.

The wrong weekday is an ordinary error: people ask, Katja corrects it. The lifting instruction is closer to a safety-critical one, because volunteers may follow it exactly and someone could be injured. Katja's best move is not to avoid the tool. It is to treat the second document differently: check it against an official safety source or ask a person who knows the work, and only then publish it. The same tool, the same afternoon, two different levels of care.

Scams and impersonation

AI makes some old tricks cheaper and more convincing: a message written in your language with no spelling errors, a voice that sounds like a relative, a video call with familiar faces. The goal of the scammer has not changed. It is to create urgency and authority so that you act before you think, and to move money or secrets through a channel that cannot be reversed.

You do not need to detect the fake. You need a routine that works even when the fake is perfect.

The advice in the table follows consumer-protection guidance. A consumer alert from the United States Federal Trade Commission (2023) says scammers can clone a relative's voice from a short audio clip, and recommends that you do not trust the voice, hang up and call the person on a number you know, check with another relative, and be wary of requests for wire transfers, cryptocurrency or gift cards. It also advises reporting the scam to your national consumer-protection or fraud-reporting service. Look for the equivalent service in your own country.

Documented case:

Year: 2024. Source: Hong Kong Free Press, 5 February 2024, reporting what Hong Kong police said.
Reported: an employee of a multinational company was tricked during a video conference in which
everyone on the call except the victim was an impersonation. About HK$200 million was
transferred in 15 payments. Police received the report on 29 January. No arrests had been made.
Established: that the police made these statements and the press reported them.
Not established here: how the fakes were made or any further detail. This tutorial does not
name the company and does not rely on any detail beyond the cited report.

The pattern matters more than the number: an authority figure, a believable video, a large and urgent payment. The defence is also a pattern. Before any large or unusual payment, confirm the request through a second channel that you start yourself, and require a second approver.

Responding if you have been targeted. Stop all contact. Tell your bank or payment provider quickly, because speed often matters. Keep the messages as evidence. Report to the relevant consumer-protection or fraud service and to your employer if it involves work. Tell someone you trust. Being tricked by a good fake is not a sign of foolishness; it is the reason the routine exists.

When an output could enable harm: misuse and dual use

Sometimes the risk is not that the AI is wrong but that it is right, and the result could help someone hurt others. You do not need to be a specialist to notice the signs. Ask three questions about a request or an output: Would the content help someone harm a person, a group or a system? Is the purpose unclear or hidden? Would it be hard to undo once shared?

Dual use means a capability that has good uses and harmful ones. The same ability can help or harm depending on who uses it and why. Many everyday tools are dual use, so the answer is rarely "ban it" or "allow everything".

As a user, if a request makes you hesitate, you can do three things: help with the safe, general part of the task; decline the harmful specifics; or ask the person to explain the purpose. If you are inside a team, raise it with the person who owns the tool or policy instead of deciding alone. You do not have to prove bad intent to decline. "I am not comfortable helping with that part" is enough.

Where reasonable people disagree

How much should powerful AI capabilities be restricted to prevent misuse? People weigh this differently.

Regions differ in practice too: the European Union has a binding, risk-based law, while several other places rely on voluntary guidance together with existing laws.

Both views share the same concern: preventing harm. They differ on where to place the control (on the capability or on the act) and on how to weigh the cost to honest users. In your own practice you do not set world policy. You decide what you will and will not help with, and you follow your workplace and local rules.

Sensitive health questions: safe boundaries

People ask AI tools about symptoms, medicines, test results and diets because the tools are available at any hour. The World Health Organization (WHO) has published ethics and governance guidance on large multimodal models in health (listed on the WHO site in 2025). Its publication page says these models are expected to be used widely in health care and public health, and notes that it is not yet proven whether they can handle a wide range of tasks. A cautious attitude is therefore reasonable.

A practical boundary helps. The line below is about the kind of question, not a medical rule.

Also remember privacy from Privacy, Consent and Data Use: What You Share with an AI Tool: health details are sensitive. Share the least you need, and check how the service uses what you type.

Mental health: when human help is needed

Some people talk to AI chat tools when they feel lonely, stressed or low. A chat can feel patient and private, and for some people it is a way to organise their thoughts. But an AI chat is not a therapist, a doctor or an emergency service. It cannot see you, it cannot take responsibility for your safety, and it may agree with you too easily or sound sure when it is wrong.

Signs that a person needs human support include: feelings that last or get worse; changes in sleep, eating or daily life that worry the person or others; a sense of being unable to cope; any thought of hurting oneself or someone else. These are reasons to involve a qualified professional or a trusted person, not to keep talking only to a tool.

Invented scenario:

Nadia, a student in Dhaka, notices that her cousin Imran has started spending hours at night
talking to an AI chat tool about feeling hopeless. He shows her a few messages. The replies are
kind but generic, and Imran says the chat is "the only one who gets it".

What can Nadia do? She can listen without judging, tell Imran she is glad he told her, and encourage him to speak to a qualified professional or to a trusted adult, and offer to help him find one. If she believes he may be in immediate danger, she contacts her local emergency services. She does not need to take over his care or argue with the tool. She also does not need to say the chat was foolish; she can say that a person who can see him and take responsibility should be involved.

In the same way, a team that builds or deploys a chat product should make it easy to reach a human or an appropriate service when a user shows signs of distress. This tutorial gives no country-specific phone numbers because they differ and change. Look up the services for your own region, and keep that list where you can find it.

AI is not an emergency service

An emergency needs a fast, accountable human response: someone who can send help, who knows the local area, and who is responsible for what happens next. An AI chat tool cannot do that. It may answer slowly, wrongly or not at all, and it has no duty to act.

• If someone may be in immediate danger, contact your local emergency services first. Do that before asking a chat tool.
• Use the tool afterwards, if at all, for calm tasks: writing down what happened, or drafting questions for a professional.
• Prepare in advance. Know your local emergency number from official local sources, and make sure family members or colleagues know where to find it.

The first three high-stakes areas: what to ask

Some decisions change a life: whether you get a job, a loan, or the right care. Using AI in these areas is not automatically wrong, but the care you owe is much higher, and the people affected usually have little say. On this page we reduce each area to questions to ask. The next page covers oversight, appeals and when not to use AI at all. Use the eight steps from the first page (situation, who is affected, benefits and harms, options, evidence, decision with conditions, accountable person, review date) when you write up a real case.

Notice what is shared across the rows. Each asks about evidence, groups who might be treated worse, a person who can override, a way to correct errors, and an accountable owner.

Legal note (as of 2026-10-12; not legal advice): According to the European Commission's pages, the European Union's AI Act is a binding, risk-based law. It describes "high-risk" systems as those that could seriously affect health, safety or fundamental rights, and names areas that include employment, education, essential services, law enforcement, migration and justice. The Commission lists duties for the provider of such systems, including risk management, data quality, logging, documentation, human oversight, accuracy and cybersecurity. Whether a particular tool counts as high-risk depends on the legal text, not on the area alone. The Commission's timeline, which takes into account the 2026 amendments, shows the high-risk rules for these areas applying from 2027-12-02; the Commission's own pages are not fully consistent with each other, so check the current official text. In New York City, the Department of Consumer and Worker Protection says that employers and employment agencies may not use an automated employment decision tool unless it has had a bias audit within one year, information about the audit is public, and notices have been given; enforcement began on 2023-07-05. These are two examples from two places. Many countries have no equivalent rule. Check the current official text for your country and sector.

Legal note (as of 2026-10-12; not legal advice): According to an unofficial online text of Article 22 of the European Union's General Data Protection Regulation, a person has a right not to be subject to a decision based solely on automated processing that produces legal or similarly significant effects, with exceptions. Where an exception applies, the organisation must at minimum offer human intervention, a way to express a view, and a way to contest the decision. The next page returns to this idea in plain words. Check the current official text for your country and sector.

Applying the pattern: a short example

Invented scenario:

Andrés works in HR at a logistics firm in Bogotá. About 400 people applied for 12 warehouse
roles. A vendor offers a tool that ranks applicants and hides the bottom half. Andrés's manager
likes the idea because the team is small.

Worksheet, filled in (short form):

1. SITUATION       Reduce 400 applications to a shortlist for 12 roles.
2. WHO IS AFFECTED Applicants (no say), HR team, hiring managers, the firm.
3. BENEFITS/HARMS  Benefit: time saved. Harm: a good candidate hidden by a ranking error or a
                   proxy; hard for the applicant to detect or contest.
4. OPTIONS         A. Tool ranks and hides the bottom half.
                   B. Tool ranks, but HR reads a sample of the lower half every round.
                   C. No AI: clear written criteria, two people score each application.
5. EVIDENCE        Unknown: how the tool was tested; whether results differ across groups.
6. DECISION        Option B only after asking the vendor for test results and a contact for
                   appeals; stop if the sample shows good candidates ranked low.
7. ACCOUNTABLE     The HR manager, named in the hiring record.
8. REVIEW DATE     After the first round: compare the shortlist with the sample review.

Another team may prefer option C because the risk to applicants outweighs the time saved. That is a defensible choice. What matters is that the reasons are written down.

Where this shows up at work

A finance clerk who receives an urgent payment request with a familiar voice can call back on a known number. A volunteer coordinator can send a safety notice to someone qualified before publishing. A product team can put a visible route to a human in a chat feature. A manager asked to buy a screening tool can bring the three sets of questions above to the vendor meeting. None of this needs technical skill. It needs the habit of asking "what is the worst realistic outcome, and who would carry it?"

Common mistakes

• Treating all errors alike. A wrong weekday and a wrong safety instruction need different care. Fix: rate the worst realistic outcome before you decide how much to check.
• Trusting a voice or face. Fakes can be convincing. Fix: use a second channel you start yourself, and require a second approver for unusual payments.
• Using a chat for personal medical decisions or a crisis. It has no duty of care. Fix: contact a qualified professional or your local emergency services, and use the tool only for general information or preparing questions.
• Deciding alone in a grey area. A dual-use request is not always clear. Fix: help with the safe general part, decline the rest, and tell the person who owns the policy.
• Buying or using a screening tool without asking questions. Fix: ask about evidence, groups, override, correction and ownership before approval.

Try it yourself

1. Easy. Rate each use as ordinary error or safety-critical failure, and give one reason: (a) a wrong spelling of a client's name in a thank-you note; (b) a summary of dosage instructions that a carer will follow; (c) a tool's note that a customer is "low value" that decides how quickly the customer is served.
2. Medium. Invented scenario: Aiko, an accountant in Osaka, receives a video message that looks like her director. The director asks her to send a large payment to a new supplier today and not to tell anyone. List the warning signs, and write the steps she should take, in order. Say what she should not do.
3. Hard. Invented scenario: A credit union in Accra plans to use a tool to approve or reject small loan applications automatically, with staff reviewing only the rejections that applicants complain about. Write five questions you would ask before agreeing, and say which two matter most to you and why. Then name one point where two reasonable people could disagree, and revisit the Andrés worksheet to see which steps your answer changes.

Answers

Easy. (a) Ordinary: easy to see and fix, little harm. (b) Safety-critical: a carer may act on it, and a wrong dose could injure someone; it needs checking against the official instructions or a qualified professional. (c) Closer to safety-critical than it looks: the customer bears the cost of a label they cannot see, and a recommendation that decides service speed may behave like a decision. How serious it is depends on what the customer needs; a defensible answer says so. Pass: you give a reason that mentions who bears the harm or whether it can be undone. Weak: only a label without a reason, or treating (c) as harmless because nobody is physically hurt.

Medium. Strong answer: warning signs are urgency (today), authority (a director), secrecy (tell no one), a new payee and a large amount. Steps: pause and do not pay; do not reply on the same channel; contact the director through a number or channel you already know; tell a colleague or manager and the finance controller; follow the company's payment-approval rule; if it is a scam, report it to the bank and the proper service and keep the evidence. She should not rely on how real the video looks, should not keep it secret, and should not make the payment to "check later". Pass: warning signs, a second channel she controls, a second approver, reporting. Weak: "look for glitches in the video" as the main defence.

Hard. A strong set of questions: What data is used, and is it accurate? How was the tool tested, and on people like our members? Are results compared across groups? Can an applicant get a reason and correct wrong data? Who can overrule it, and do they have time and authority? Can every rejected applicant ask for a person to review, not only those who complain? Who is accountable, and when do we review the results? A sensible priority might be the route for every applicant to a human review (because complaint-only review lets quiet errors stand) and testing on members like those who will apply. A point of disagreement: one person may value speed and low cost for many small loans; another may say any automatic rejection of a person's loan needs a human check. Pass: at least five questions covering evidence, groups, correction, override and ownership; two chosen with reasons; a fair statement of both views. Weak: a verdict without questions, or questions only about speed and cost.

Summary

• Rate the worst realistic outcome, who bears it and whether it can be undone. Check in proportion, and treat safety-critical outputs differently.
• Scams use urgency, authority and secrecy. A second channel that you start yourself and a second approver work even against perfect fakes.
• Many capabilities are dual use. Help with the safe general part, decline harmful specifics, and raise unclear cases with the person who owns the policy.
• For health and mental health, use AI chat for general information and preparing questions only. A qualified professional or your local emergency services decide and respond.
• For hiring, credit and medical triage, ask about evidence, groups, override, correction and ownership. Rules differ by place; check the current official text.

Next: High-Stakes Decisions, Human Oversight and Appeals continues with education and justice, decisions that are hard to reverse, and how to tell meaningful oversight from a rubber stamp.