Misuse and Safety Analysis: Threat Scenarios, Dual Use and Abuse Cases

Earlier pages asked what a feature does when it works as planned. This page asks what else people will do with it. Misuse analysis is a calm, practical craft: list who might use the feature in unintended ways, decide which cases are foreseeable, and attach a control and a test to each. This page and the next one are defensive: they stay at the level of scenarios and controls and give no operational detail for causing harm.

No code this time. Everything is a table, a template or a worksheet you can fill in. The running example continues on a new feature.

Honesty convention. Every company, person, feature and scenario is invented unless labelled Documented case. Nothing here is legal advice.

What you will learn

• How to build a threat and misuse scenario list, and tell intended use from foreseeable misuse.
• How to write abuse cases that go beyond the normal user journey.
• How to assess fraud, impersonation and synthetic voice and video risks, and how disclosure and provenance help and where they stop.
• How to treat non-consensual synthetic likenesses, children and vulnerable users, and sensitive health requests at the policy level.

The running example: a synthetic-voice feature

Invented scenario:

Company:  Harbourline Staffing (invented), the agency from page 1.
Feature:  InviteVoice. A recruiter types an interview invitation and the system reads it aloud
          in the recruiter's own synthetic voice, made from a sample the recruiter recorded.
Purpose:  save recruiters time and give candidates a warmer message than plain text.
Boundary: the recruiter's own voice only; invitations only; sent from the agency's account.
Question: before launch, what could go wrong, for whom, and what do we do about it?

Use the decision pattern from page 1. Here the option "do not build it" is real: a plain text invitation does the job. A feature that makes convincing voices is also a feature that can make convincing false ones.

A threat and misuse scenario list

Build a threat and misuse scenario list. A threat is a way someone could harm the feature, its users or other people. Misuse is use that works against the purpose. Build the list from four questions: who are the actors (outsiders, users, insiders, and the system itself failing), what do they want, what do they have access to, and who is harmed. Describe each scenario in a generic sentence; you do not need technique to know it is a risk.

Template:

ID   Actor           Goal (generic)                    Who is harmed        Existing control?
M1   Outsider        gets a voice made for someone     the person copied;   none yet
                     who did not agree                 people who trust it
M2   Registered user asks for a fake voice message of   the colleague;       none yet
                     a manager approving a payment     the agency
M3   Insider         uses a colleague's voice sample   the colleague        access log only
                     without asking
M4   Outsider        sends the agency's voice to        candidates           none yet
                     candidates in a false job offer
M5   Any user        tries many odd inputs to see what  anyone (unknown)     rate limit
                     the filter lets through
M6   The system      reads a name or number wrongly     a candidate          human check
                     in an invitation

Notice that the list mixes malice (M2, M4), carelessness (M6) and curiosity (M5). A good list does not guess motives; it asks what the feature allows. Score each row with the register from page 2 (likelihood, severity, reversibility) and keep the list in the same place as the risk register, with an owner.

Intended use and foreseeable misuse

Distinguish intended use from foreseeable misuse. Intended use is what the team designed and states in its purpose. Foreseeable misuse is use that is not intended but is easy, likely or already seen with similar tools. It is different from possible misuse: almost anything is possible, so a list of all possibilities helps nobody. A practical test: would a thoughtful colleague who knew the feature say "of course someone will try that"?

How far should a maker go in anticipating misuse? Reasonable people disagree.

The course does not choose a side. It asks you to state which misuses you treat as foreseeable, why, and what you do about them, so that a reviewer can disagree with specifics rather than with a mood.

Abuse cases beyond normal user journeys

Identify abuse cases beyond normal user journeys. A user story says "As a recruiter, I want to send an invitation so that the candidate replies". An abuse case turns it round: "As a person who wants to deceive, I want a voice message that sounds like a manager so that someone acts on it". Teams that only write user stories test the happy path. Prompts that help you find abuse cases, none of which needs technique:

• Wrong person. What if the sample belongs to someone who did not agree?
• Wrong content. What if the text is not what the feature was built for (a payment request, a threat, a false offer)?
• Wrong scale. What if one account sends thousands of messages?
• Wrong channel. What if the output is saved and reused outside the product, where no label travels with it?
• Wrong insider. What if an employee, a contractor or a vendor with access misuses it?
• Manipulating the system. What if the text contains instructions aimed at the system (see prompt injection in Prompt Engineering for AI: Advanced)?
• Your own failure. What if the feature makes an honest mistake at scale?

Fraud and impersonation risks

Assess fraud and impersonation risks. Fraud with synthetic media usually combines three ingredients: authority (someone senior or familiar), urgency (act now), and a realistic voice or face. The defence is also usually a process, not a detector: confirm through a second channel the requester cannot control.

Documented case: in February 2024 Hong Kong Free Press reported, citing police, that an employee of a multinational company had been tricked during a video conference into making about HK$200 million in transfers, in 15 payments. Police said everyone else on the call had been impersonated and that the fake participants appeared to be pre-recorded. At the time of the report no arrests had been made. The company is not named here. The case shows the pattern, not the technique: authority, urgency and a call that looked real.

The US Federal Trade Commission has also published consumer advice (2023) about scammers cloning a relative's voice from a short audio clip. Its guidance is: do not trust the voice; hang up and call back on a number you already know; check with another relative; be wary of any request to pay by wire, cryptocurrency or gift cards; and report the scam to the authority in your country.

Risks from synthetic voice and video

Assess risks from synthetic voice and video. Group them by who is harmed:

• The person copied. Their voice or face is used to say what they never said: reputation, money, safety, dignity.
• The audience. People deceived into paying, voting, trusting or complaining.
• The wider information environment. When fakes are easy, real recordings can be dismissed as fake, and people may stop trusting any media. This is a reasoned concern, not a measured one.
• The maker. Reputation, legal exposure and, if an incident happens, the cost of response.

Video adds two things to voice: faces are strongly tied to identity, and a group call with several fake participants (as in the Hong Kong report) can look like independent confirmation. A "second pair of eyes" on the same channel is not a second channel.

Disclosure for synthetic media

Define disclosure requirements for synthetic media. Disclosure means telling people that media is synthetic or that they are talking to a machine. Your policy has three parts: whom you tell (the person hearing it), how (visible label, spoken notice, metadata, a notice in the interface) and when (before they act on it). Also decide who is responsible: the provider that makes the tool, or the organisation that publishes the content.

Legal note (as of 2026-10-12; not legal advice): The European Commission's page on the EU AI Act lists a "transparency risk" category: users must be told when they interact with AI such as a chatbot, generative output must be identifiable, and deepfakes must be labelled. Article 50, as shown on an unofficial text site, says providers must design systems that interact with people so that people know it is AI (unless obvious) and must mark synthetic audio, image, video or text in a machine-readable way "as far as technically feasible"; deployers must disclose deepfakes (with lighter disclosure for evidently artistic, satirical or fictional works) and AI-generated text published to inform the public on matters of public interest, unless a human has editorial control and a named person is responsible. According to the Commission's AI Service Desk timeline these rules apply from 2 August 2026, with a transitional deadline of 2 December 2026 for certain providers already on the market before that date. A voluntary code of practice on marking and labelling is reported to exist or be in preparation; check the Commission's site. Other countries are reported to require labels on AI-generated content; check the rules where you operate. Check the current official text for your country and sector.

Disclosure also depends on context rules that are not law: a school, a journal or an employer may require it. The International Committee of Medical Journal Editors (ICMJE), for example, asks authors to disclose AI use and does not accept chatbots as authors. Keep the three apart: ethical concern, legal obligation and the policy you set yourself.

Provenance signals and proof of authenticity

Distinguish provenance signals from proof of authenticity. Provenance is a record of where a file came from and what was done to it. C2PA is an open technical standard for this; the implementation is called Content Credentials (the home page describes it as a "nutrition label for digital content", and shows version 2.3 as current). A credential is a signed claim bundled with the file; the structure is tamper-evident, and the specification says it does not make value judgments about whether the data is good or bad. Software checks the signature and shows the result so that a human can decide how much to trust the signer.

The rule to teach: provenance changes how much you believe a file, never whether you must check. Never use a detector score alone to accuse a person. Use process evidence: who sent it, through which channel, what independent record exists, and what the second channel says.

Non-consensual synthetic likenesses

Assess risks of non-consensual synthetic likenesses. A synthetic likeness is made without consent when a person's face, body or voice is imitated without their agreement. The most serious kind involves intimate or sexual imagery, and imagery of children. We keep this section non-graphic and defensive: the harm to the person is real even though the image is synthetic, and a feature should be designed so that it cannot easily be used this way.

Legal note (as of 2026-10-12; not legal advice): The European Commission's AI Service Desk timeline shows that, from 2 December 2026, the EU AI Act adds a prohibition on AI systems that generate non-consensual sexual deepfakes and child sexual abuse material, and the Commission's page says it also covers "nudification" apps. On 2026-10-12 this prohibition is adopted but does not yet apply. Law-firm summaries report that the wording also reaches systems without reasonable safeguards against such content; the exact text was not confirmed. Other countries are reported to have laws or offences on non-consensual intimate images, including synthetic ones; check the rules where you operate. UNICEF's Guidance on AI and Children, version 3.0 (December 2025), adds material on AI-generated child sexual abuse material and non-consensual intimate images. Check the current official text for your country and sector.

Policy-level design questions:

• Consent by design. Make a person's own verified consent a condition for making their likeness; do not rely on a tick box.
• Scope. Does the feature need to create realistic likenesses of real people at all? Limiting it to the account holder's own voice removes most of the risk.
• Prohibited categories. State clearly that sexual imagery of real people and any content involving children's likenesses are never produced, and test it.
• Report and remove. A visible way to report, a named owner, and a promise to act within a stated time.
• Support for the person harmed. The guidance is non-graphic: do not forward the material, report it to the platform and to the appropriate authority where you live, keep a record of where it appeared, and seek support from someone you trust or from a qualified professional. No country-specific contact details are given here because they change; the product team keeps a checked list for each region it serves.

Children and vulnerable users

Identify risks to children and vulnerable users. Vulnerability can be a fixed trait (age) or a situation (grief, illness, debt, isolation, a crisis). UNICEF's guidance names ten requirements for child-centred AI, including safety, data and privacy, non-discrimination, and the child's best interests and well-being. Version 3.0 adds attention to AI companions and to AI-generated abusive imagery.

Legal note (as of 2026-10-12; not legal advice): Under GDPR Article 8 (unofficial text site), for consent-based online services offered directly to a child, lawful consent is possible from age 16, or from a parent below that; Member States may set a lower age, not below 13, and the controller must make reasonable efforts to verify parental consent. Age thresholds vary by country. Check the current official text for your country and sector.

Sensitive health requests

Define safe handling of sensitive health requests. People ask AI systems about their health. A product needs a policy for that, written in advance. The WHO has published ethics and governance guidance on large multi-modal models in health, and its listing notes that wide capability is "not yet proven". The boundary for a general-purpose product is: general information, not diagnosis, not personal treatment decisions, and not dosing; a qualified professional decides. This course gives no medical content; it designs the policy around it.

Where this shows up at work

Trust-and-safety, product and security teams keep a misuse list next to the risk register, review it before launch and after each incident, and give each row an owner. A small team can start with ten rows and one meeting.

Common mistakes

1. Listing only the happy path. If every story is "a user wants...", no abuse case is found. Write the opposite story.
2. Listing everything possible. A thousand rows hide the ten that matter. Keep foreseeable ones, rank them, and say why others are out.
3. Treating a label or credential as proof. It shifts belief; verification through a second channel is still needed.
4. Stopping at "users must be adults". A statement is not a control. Ask what stops a child, and what happens if one gets through.
5. Writing a medical answer into a policy. The policy sets who answers and when; it does not play doctor.

Try it yourself

The exercises revisit the risk register from page 2.

1. Easy. For InviteVoice, write one abuse case that is not on the list above, in the form "As a person who wants to ..., I want ... so that ...", in generic terms.
2. Medium. Take scenario M2 (a fake voice message of a manager approving a payment). Name one control at the feature and one at the organisation, and say which of the two you would trust more and why.
3. Hard. Harbourline is asked to add video: the recruiter appears in a short synthetic video message. List five additional rows for the misuse list, state which are foreseeable, and say whether you would build the feature, build it with conditions, or not build it, using the decision pattern.

Answers

Easy. A model answer: "As a person who wants to pressure a candidate, I want a voice message that sounds like the recruiter's manager so that the candidate accepts a bad offer without checking." Other strong answers: wrong scale, wrong channel or insider misuse. Pass checklist: the actor, the aim and the harm are all named; no technique is included; the case is about the feature. A weak answer is a technical exploit, or a case that needs no feature at all.

Medium. Feature-level: only the verified account holder's own voice can be synthesised; approval-type messages are not produced. Organisation-level: a payment needs a second channel the requester cannot control. Most strong answers trust the organisational control more, because feature controls can be bypassed by tools outside the product, but they still keep both. Pass checklist: one control at each level; a reason; an acknowledgement of what each cannot stop. A weak answer lists "train the staff" only.

Hard. Possible additional rows: a video of a real person who did not agree; a video used to impersonate a leader; a fake endorsement of a job offer; a video saved and reused with the label removed; a candidate being unable to tell the video is synthetic. Foreseeable: all five, similar to the voice risks but stronger because a face adds apparent proof. A defensible decision is "build with conditions": the recruiter's own consented likeness only, visible and spoken disclosure, metadata credentials, a report route and a review date; a defensible alternative is "do not build", because plain text meets the purpose. Pass checklist: five distinct rows; foreseeable versus possible explained; a decision with conditions or a reason; an accountable role; a review date. A weak answer says only "add a watermark".

Summary

• Build a misuse list from actors, goals, access and harm; keep it with the risk register and give each row an owner.
• Separate intended use, foreseeable misuse and out-of-scope use; state which you treat as foreseeable and why.
• Fraud with synthetic media is mostly authority, urgency and realism; the defence is a second channel, not a detector.
• Disclosure, credentials and watermarks help but are not proof; a missing credential does not prove a fake.
• For non-consensual likenesses, children, vulnerable users and health requests, design the policy and the escalation path in advance, without medical or graphic detail.

Next: Safety Controls and Synthetic Media: Layered Defences, Provenance and Response turns these scenarios into controls, tests and escalation.