This is how AI gets things wrong, and why it sounds so sure
It will invent a plausible answer rather than say "I don’t know," and it will do it in the same confident voice. Here’s the shape of the failure and how to live with it.
This is how AI gets things wrong, and why it sounds so sure while doing it.
Ask an AI about a treaty that doesn’t exist, phrased as if it does, and there’s a real chance you get back a fluent little summary: parties, provisions, historical significance. Not because it wants to deceive you. Because its entire job is producing the most plausible continuation, and a plausible-sounding summary is more statistically likely than "that doesn’t exist."
The industry calls this hallucination. You can call it what it feels like: a well-read friend who would rather improvise than admit they don’t know.
Where it strikes
The failure concentrates predictably. Highest risk: specific facts at the edge of common knowledge. Names, dates, statistics, citations, prices, the contents of specific documents it hasn’t been shown, anything niche enough that it read about it rarely. Lowest risk: reasoning about material you GAVE it, common knowledge, and structure (what to consider, how to compare, what to ask).
Notice that the low-risk zone is exactly what this site teaches. That isn’t luck; the moments and skills here were chosen to live where the tool is strong. Show it your lease and its reading is grounded in your lease. Ask it for the history of lease law in your state, uncited, and you’re in the casino.
The three-question test
Before acting on any AI answer, run these:
- Is this a fact I could check, or a way of thinking? Ways of thinking (checklists, comparisons, questions to ask) are its safe zone. Checkable facts get checked.
- What happens if this is wrong? Wrong dinner suggestion: nothing. Wrong medication interaction: everything. Stakes set the verification effort.
- Did it get this from me, or from nowhere? Answers grounded in what you provided (your photo, your document, your numbers) are far more reliable than answers pulled from its training.
Making it more honest
You can lower (not eliminate) the risk with two habits: give it an out ("if you’re not sure, say so; I prefer an honest gap to a guess"), and ask for its confidence and sources ("which parts of that answer are you least sure about?"). The second question is quietly excellent; it will often correctly identify its own weakest claims when asked, even though it stated them confidently a moment earlier.
What you cannot do is trust tone. The sureness is part of the writing style, not a measure of anything. This is the single most common way smart people get burned.
None of this is a reason to skip the tool; it’s the manual for the tool. A car that needs its mirrors checked is still a car.
This is how to use AI.