Judgment

How to check if an AI answer is actually true (in about 20 seconds)

By the Keep Your Head desk · About a 6-minute read

AI tools don't lie the way people lie. They do something stranger: they invent confident, specific, plausible details — dates, statistics, names, quotes, medical facts — with exactly the same fluent tone they use when they're right. Here's the quick habit that catches most of it.

Why smart people get fooled

Here's the uncomfortable mechanic underneath every chatbot: it doesn't look facts up. It predicts what a good answer would sound like, word by word, based on patterns in what it read during training. Most of the time the most-plausible-sounding answer is also true. But when it isn't — when the model doesn't actually know — it doesn't say "I don't know." It fills the gap with something that fits. Researchers call these hallucinations. You'll experience them as a perfectly formatted, completely made-up court case, book title, statistic, or dosage.

The trap is that fluency and accuracy feel like the same thing. A hesitant human mumbling "I think it was maybe 2019?" triggers your skepticism. An AI stating "The policy was enacted in March 2019 and affects approximately 2.3 million households" does not — even when it dreamed up both numbers. Your brain reads confidence as competence, and AI is infinitely confident.

So the fix isn't "never trust AI." It's knowing which claims deserve a check, and having a check fast enough that you'll actually do it.

Step 1: Spot the claims worth checking (5 seconds)

You don't need to verify everything — that would make AI useless. Scan the answer for what we call load-bearing facts: the specific, checkable claims that the answer leans on. They're easy to spot because they're the specific bits:

Numbers: statistics, prices, dosages, percentages, dates, deadlines.
Names: people, court cases, studies, books, laws, products.
Quotes: anything in quotation marks attributed to a real person.
Rules: "it's legal to…", "the requirement is…", "your landlord must…".

One useful rule of thumb: the more specific and surprising a detail is, the more it needs checking. "Exercise is good for sleep" — fine. "A 2017 Stanford study of 4,200 adults found 38% improvement" — check it. Hallucinations are usually very specific, because specific is what plausible sounds like.

Step 2: The 20-second check

For each load-bearing fact, do the fastest check that would catch it if it were wrong:

Search the exact detail, not the topic. Copy the specific claim — the study name, the statistic, the case name — into a normal search engine, in quotes if it's a name or phrase. If a real study or case exists, it shows up in seconds. If the only "source" is other AI-generated pages, or nothing at all, you've likely caught an invention.
Ask the AI to cite — then click. Ask: "What's your source for that number?" Two things can happen. It names a source you can open and confirm — good. Or it apologizes and revises the number — which happens remarkably often, and tells you everything. Warning: models sometimes invent the citation too, complete with plausible authors and page numbers. The citation isn't the check. Opening it is.
Cross-examine in a fresh chat. Open a new conversation (so it can't just agree with itself) and ask the question again, differently phrased. Same specific answer twice is weak evidence but evidence. Two different confident answers to the same factual question is a red flag you can get in under 30 seconds.

If you want to build the habit into the conversation itself, end factual questions with this:

For any specific facts in your answer (numbers, dates, names,
studies), tell me your confidence level and whether I should
verify it independently before relying on it.

This doesn't make the model magically accurate — but it reliably surfaces its own uncertainty, and it changes your reading posture from "receiving an answer" to "reviewing a draft." That posture shift is most of the game.

Step 3: Match the effort to the stakes

Twenty seconds is for everyday claims. Scale up or down honestly:

No check needed: brainstorming, rewording an email, summarizing text you provided, creative work. If you can judge the output just by reading it, judgment is the check.
20-second check: facts you'll repeat to others, numbers going into a document, recommendations you'll act on cheaply.
Real verification — a primary source or a professional: anything medical, legal, financial, or safety-related. AI is a decent way to prepare questions for the doctor, lawyer, or accountant. It is not the doctor, lawyer, or accountant. People have submitted AI-invented court cases to actual judges. Don't join them.

The one question that catches the rest

Before you repeat or act on an AI answer, ask yourself: "Is there a fact in here I'd be embarrassed to repeat if it were wrong?" If yes, that's the fact you check. If no, carry on. This single question, applied consistently, filters out the vast majority of AI-fluency accidents — the wrong date in your presentation, the invented book you recommend at dinner, the "law" that isn't one.

Keep your head

Fluent and correct feel identical when you're reading. They are not the same thing. Twenty seconds is what the difference costs.

Get one of these a week. Our free newsletter sends one genuinely useful AI habit and one judgment check every week — no hype, four-minute read. Subscribe on the home page.