Trust, Verification and Automation Bias: When to Rely on AI Output

An AI tool answers in smooth, confident sentences whether it is right or wrong. A photo, a voice note or a video clip can look real and still be false. The risk is not only that the tool is sometimes mistaken. The risk is that people stop checking, and then a wrong claim travels on to others who trust us. This page is about the habits that keep your trust in proportion: knowing which claims you have verified, noticing when a review has turned into a rubber stamp (approval without real checking), and recognising false or synthetic content before you pass it on.

You already know why models can produce false statements from How AI Works: LLMs, Tokens and Limitations, and you met everyday checking habits in Prompt Engineering for AI: Beginner. We do not repeat either. Here we ask what happens next: how checking fails inside a team, how false information spreads between people, and what to do when you suspect a clip or a voice is fake.

What you will learn

• The difference between a verified claim and a plausible one, and why a confident tone is not evidence.
• What automation bias is, how it turns a human review into a rubber stamp, and what independent review looks like.
• How false information spreads through repeated sharing, and how satire, fiction, honest error and deliberate deception differ.
• How fabricated quotes, misleading images, synthetic audio and voice cloning are used to mislead, with a hypothetical payment scam.
• How to recognise and respond to deepfake risks in elections, workplaces and personal relationships, without panic.

Verified claims and plausible claims

A plausible claim sounds right: it fits what you expect and nothing in it looks odd. A verified claim has been traced to a source you can open, written by someone in a position to know, and checked by you or by someone independent of the person who made the claim. Plausible is a feeling. Verified is a record.

Invented scenario:

Diego edits a monthly newsletter for a neighbourhood association in Bogotá. He asks an AI
tool for a short history of the local market. The tool answers: "The market opened in 1952
under a charter signed by the mayor, and it has been renovated three times." The paragraph is
fluent and gives dates. Diego has no source for any of it. He plans to print it on Friday.

Diego can verify the paragraph in a few steps: ask the market office or the municipal archive, look at a dated plaque or a local history book, and ask two long-time residents. If he cannot confirm the 1952 date, he can write "the market is believed to date from the early 1950s, according to residents" or leave the date out. How much checking is enough depends on the stakes, as on page 1. A date in a newsletter is low stakes. A claim that a named person broke the law is not.

Confident-sounding answers

People tend to read fluent, specific, well-organised text as reliable. Models write that way by default, so a wrong answer and a right one can look the same on the screen. Over-reliance means accepting an answer because of how it sounds, not because of what supports it. Some signs that call for a check:

• Exact numbers, dates or names with no source attached.
• References you cannot open or find.
• A tidy answer to a question that experts still argue about.
• No mention of what is unknown, even when you asked about something obscure or recent.

Two questions help. "Where could I check this in two minutes?" and "What would show this is wrong?" If you cannot answer either, you have an unchecked claim. A confident tone, a long answer and repeated agreement from the same tool are not extra evidence. Asking the tool "are you sure?" does not verify anything, because it can repeat the same mistake.

Automation bias in a review process

Automation bias is the tendency to accept the suggestion of an automated system and to look less carefully for errors, especially when it is usually right. Research reviews report that people over-rely on automated advice, that this includes experts, and that training alone does not remove it. This page does not need exact figures. The practical message is that "a person looks at it" does not guarantee that errors get caught.

Invented scenario:

A trading company in Lagos receives about 300 supplier invoices a day. A tool compares each
invoice with the purchase order and marks it "Match" or "Check". Hassan approves payments.
He is shown the tool's mark first and has about 90 seconds per invoice. In the last month the
tool marked 97% of invoices "Match", and Hassan approved all of those. Last week one "Match"
invoice was for an amount that was ten times too high. Nobody noticed until the supplier called.

Hassan is formally the reviewer. But the screen shows the answer first, the time is short, the tool is right almost every day, and the work is repetitive. Each of these makes agreement the easy path. When page 1 said a recommendation can behave like a decision, this is what it meant. Look at how the review could be designed so that errors have a chance of being seen.

Independent review and rubber-stamping

The "mixed-in known errors" idea is easy to try. Toy data:

Hassan's manager adds 6 invoices with deliberate errors to one day's batch of 300, without
telling him. Hassan approves 5 of the 6 as "Match" and stops 1.

Seeded errors caught: 1 of 6 (about 17%).

This does not prove that Hassan is careless. It shows that the process, as designed, lets most errors through, which is what the manager needs to know. (The numbers are invented and only show the idea.) Fixes could include showing the evidence before the tool's mark, flagging a random sample of "Match" invoices for a full check, limiting how many items one person reviews per hour, and recording when reviewers override the tool.

How false information spreads

False information rarely stays where it started. Each person who passes it on adds their own trust, and the people they send it to trust them more than they would trust a stranger. Small changes also pile up: caveats are dropped, a guess turns into a fact, and a headline stands in for a story. Toy data:

One person sends a false message to 20 contacts. In each round, 1 in 4 of the people who
see it send it on to 20 new contacts.

Round 1: seen by    20, of whom  5 pass it on
Round 2: seen by   100, of whom 25 pass it on
Round 3: seen by   500, of whom 125 pass it on
Round 4: seen by 2,500, of whom 625 pass it on
Total seen after 4 rounds: 3,120 (overlap between contacts ignored)

One message became about three thousand views in four steps, and the people who passed it on mostly believed they were helping. The wording changes along the way:

Original (a staff member's note): "Bus line 14 may be delayed tomorrow. Not confirmed."
After share 1:                    "Bus line 14 delayed tomorrow."
After share 2:                    "Bus line 14 cancelled tomorrow, avoid it!"

The habit to build is small: before sharing, check who first said it, whether they were in a position to know, and whether the version in front of you has kept its caveats.

Satire, fiction, error and deliberate deception

Not every false statement is the same, and the right response depends on which kind you are looking at.

In this course, misinformation means false content shared without intent to mislead, and disinformation means false content made or shared on purpose to mislead. You often cannot see intent, so first ask whether the content is accurate, then who made it, and only then what they might want. A true warning from a person you trust and a deliberate fake can look equally convincing.

Fabricated quotes and false attribution

A fabricated quote is words a person never said. False attribution is a real quote given to the wrong person, or a real person's words moved to a different time or setting. Both are easy to produce with AI tools, and an image with a name and a quote on it looks more official than it is.

Invented scenario:

A picture card circulates in a school parents' group in Nairobi. It shows a photo of the
head teacher, Ms Odera, and the text: "Homework is a waste of children's time." There is
no date and no link. The card has been forwarded 40 times. A parent asks the group whether
it is true.

A good response is neither "of course she said it" nor "obviously fake". It is a short check: Is there a date, place or event? Can the quote be found in a recording, a letter, a school newsletter or a news report from the real source? Does Ms Odera, or the school office, confirm it? Does the exact wording appear anywhere that existed before the card? If nothing supports it, say "I cannot find a source for this" and ask the sender where it came from. Do not repeat the quote in your reply as if it were true.

Misleading images and synthetic audio

Misleading media comes in several kinds. The first two need no AI at all.

It is tempting to rely on visual clues, such as odd hands or a strange background. Do not depend on them. Tools improve, many fakes show no obvious flaw, and real photos sometimes look odd. A better habit is to check the source and the context, and to look for a second, independent confirmation. The next page covers labels and provenance, which help but are not proof.

Voice cloning and a hypothetical payment scam

Invented scenario:

Aiko works in the accounts team of a printing company in Osaka. At 17:40 on a Friday her phone
rings. The caller sounds exactly like Mr Sato, the owner. He says he is at the airport, a
supplier will cancel an order unless 4,800 is paid in the next 20 minutes, and he gives her new
bank details. He asks her not to mention it to anyone because the deal is private. The call
drops before she can ask a question.

The pattern combines urgency, authority, secrecy and a new payment route. A voice that sounds right is no longer a proof of identity, since a short audio clip can be enough for a tool to copy a voice. The response is calm and procedural:

1. Do not pay because of the call.
2. Hang up and call back on a number you already know (not the one that called, and not one given in the call).
3. Check with a second person, such as another manager.
4. Follow the company's payment rule: new bank details need a second approver and a check through a known channel.
5. Report the attempt to your manager and, if money was involved, to your bank and the proper authority in your country.

A consumer-protection agency in the United States, the Federal Trade Commission (FTC), published a 2023 consumer alert on a related scheme in which scammers clone a relative's voice to fake a family emergency. Its advice matches the steps above: do not trust the voice, hang up and call the number you know, check with another relative, and be wary of requests for wire transfers, cryptocurrency or gift cards. That alert is advice, not a case history. Fraud rules and reporting routes differ by country, so find yours.

Documented case:

Hong Kong video-conference fraud, reported 2024-02-05. Hong Kong Free Press reported what the
police said. An employee of a multinational company was tricked on a video conference into
making 15 transfers worth about HK$200 million. The police said everyone on the call except the
employee had been impersonated using deepfake technology. No arrests had been made at the time
of the report. The company is not named here.
Established: that the report exists and what the police said at the time.
Not covered: later developments, and details that the police or the company have not confirmed.

The pattern is the same as in Aiko's call: urgency, authority, a realistic voice or face, and a payment. The defence is also the same: confirm through a second channel and use a payment procedure that no single person can bypass.

Deepfake risks in elections, workplaces and personal relationships

A deepfake is synthetic or altered audio, image or video that shows a real person doing or saying something they did not. The risks differ by setting, and in each the useful skill is recognising and responding, not alarm.

The harm is real even when the content is synthetic. UNICEF's guidance on AI and children (version 3.0, December 2025) includes material on AI-generated child sexual abuse material and non-consensual intimate images. For anyone affected, the priority is safety and support, not proving the image is fake to everyone. If you are the target, you are not to blame. If you receive such content, do not forward it, even to warn others; report it and support the person.

Where this shows up at work

An accounts team adds a callback rule for changes to bank details. A newsroom volunteer checks a photo's first posting before captioning it. A team lead sets up a monthly test with known errors to see if the review process catches them. A teacher shows pupils how a forwarded quote card was traced to nothing. A manager asks "who checked this, and how?" before approving an AI-written summary for a client. None of these needs special software. Each needs a written habit.

Common mistakes

• Treating a confident answer as a checked one. Fix: ask where it can be verified, and record the source you actually opened.
• Counting "a person reviewed it" as a safeguard without looking at how. Fix: check what the reviewer sees first, how much time they have, and whether anyone tests them.
• Sharing to warn others. A warning about a fake spreads the fake. Fix: share the verified correction or the source, not the content itself.
• Trusting a voice or face as proof of identity. Fix: confirm through a second channel and use approvals that do not depend on one person.
• Relying on visual clues. Fix: check source and context first.

Try it yourself

1. Easy. Label each statement "verified", "plausible" or "unchecked", and say why: (a) a tool says "the bridge was built in 1974" and you have no source; (b) the municipal archive's web page, which you opened, gives the same date; (c) a colleague says the date "sounds right".
2. Medium. Invented scenario: Meera, the finance assistant for a design studio in Pune, receives a voice message from "the director" asking her to pay a new vendor today and keep it confidential. The voice sounds right. Write the steps Meera should take, in order, and one rule the studio should add.
3. Hard. Go back to Omar's email-ranking tool on page 1, where Leila reads the "Urgent" pile first. Design a short review process that avoids a rubber stamp: say what Leila sees first, how known errors are mixed in, how the result is recorded, who is accountable, and when it is reviewed. Name one cost of your design.

Answers

Easy. (a) Unchecked: no source. (b) Verified, as long as you opened it and the page is the archive's own and matches in details. (c) Plausible: it is a feeling, not a source. Pass: each label comes with a reason that mentions a source or the lack of one. Weak: "verified" for (c) because the colleague is trusted.

Medium. A strong answer: do not pay yet; do not reply from the same message; call the director back on a number Meera already has, or ask another partner; check whether the vendor exists and the bank details match a known record; tell the studio manager about the attempt. The rule to add: any payment to a new vendor or any change of bank details needs a second approver and a callback to a known number, however urgent the message says it is. Pass: the steps include a second channel and a procedure that does not rely on one person. Weak: "listen carefully to the voice" as the main check, or paying first and checking later.

Hard. A strong design: Leila first opens a random sample of emails from all three piles, shown without the tool's label, before looking at the Urgent pile; once a week Omar adds 5 test emails he wrote himself with known problems (one marked as a damaged order, one as a refund) to see how they are ranked; Leila records every disagreement with the tool in a shared sheet; Omar is accountable; the review date is 2026-12-15, when they compare caught tests, complaints and time saved. Cost: sampling takes some of the time the tool was supposed to save, and writing test emails takes effort. Pass: the review shows evidence before or without the label, includes seeded or sampled checks, records disagreements, names an owner and a date, and states a cost. Weak: "Leila should be more careful".

For verifying AI output in day-to-day writing, return to the habits in Prompt Engineering for AI: Beginner.

Summary

• Plausible is a feeling and verified is a record. A confident tone is not evidence, and asking the same tool "are you sure?" does not verify anything.
• Automation bias makes human review weaker than it looks. Independent review shows evidence first, allows time, makes disagreement easy and tests the process with known errors.
• False information spreads quickly through repeated sharing and loses its caveats on the way. Check the first source before you pass it on.
• Satire, fiction, honest error and deliberate deception call for different responses. Fabricated quotes, misleading media, synthetic audio and voice cloning rely on urgency and authority; confirm through a second channel.
• For deepfakes in elections, workplaces and personal relationships, do not forward, keep evidence, report, and support the person affected.

Next: Deepfakes, Provenance and Disclosure: Telling Real from Synthetic looks at labels and provenance, what they can and cannot prove, when synthetic content should be disclosed, and how to respond to an incident.