How to Tell If an AI Is Hallucinating: Signs, Checks and Tools

You usually cannot tell from the text. A hallucinated answer is, by construction, one that reads exactly like a correct one, so “read it carefully” is not a detection strategy — it is the absence of one. What you can do is look for the signals that correlate with invention, run the checks that expose it, and, if you are responsible for a feature rather than just using one, replace guesswork with a measured rate.

This post covers all three, in that order. For the definition and the mechanism, start with what an AI hallucination is.

Six signs, for anyone reading an AI’s output

• Specificity with no source. Precise names, dates, figures, quotations or identifiers that arrive with no indication of where they came from. Ask where. A model with a source can usually point to it; one without will produce a source-shaped answer, which is the next sign.
• References that do not resolve. A URL that 404s, a case name no database returns, a paper no journal lists, a product code the manufacturer does not recognise. This is the single most reliable tell, and the easiest to check.
• Uniform confidence. The same assured tone for a well-known fact and an obscure one. Real knowledge is uneven. A model that never sounds less sure is not reporting its uncertainty, because it does not have a reliable representation of it to report.
• Instability under rephrasing. Ask the same thing three ways, or ask it again tomorrow. A grounded answer survives; an invented one shifts, because each time the model is sampling a plausible continuation rather than retrieving a fact.
• Suspicious completeness. Every field filled, every question answered, no gaps, no caveats. Reality has gaps. An answer with none is often one where the model filled them.
• In agents, narration that outruns the log. “I have checked the database and updated the record.” The tool log shows no call. This is invisible in the reply and obvious in the trace, which is why agent hallucinations are caught by inspecting what the agent did, not what it said.

The checks a team can run on a live feature

The signs above are for a reader. If you run the feature, you can do better than watch for them, and the checks below are roughly in order of how cheap they are to add.

1. Reference resolution. Anything the output cites — a URL, an identifier, a document, a policy clause — is checked mechanically for existence before the answer is shown. It would have caught most of the documented cases and costs almost nothing.
2. Retrieval inspection. For a RAG system, log what was retrieved alongside what was generated, and measure the two separately. A confident wrong answer built on the wrong document is a retrieval failure that a generation check will pass.
3. Groundedness scoring. A classifier, or a second model, judges whether each claim in the answer is supported by the context that was provided. Open-source faithfulness models and the groundedness checks built into the major AI platforms both do this. It answers “is this supported by the source?” — which is not the same as “is this true?”.
4. Self-consistency sampling. Generate several answers to the same input and measure agreement. Low agreement is a cheap, model-agnostic proxy for invention.
5. Constrained output as a check in itself. When the output must conform to a schema or choose from an enumerated set, whole classes of hallucination become structurally impossible and the rest become detectable as schema violations.
6. Trace inspection for agents. Compare the agent’s narration to its tool calls: was the tool called, with what arguments, and did the result match what the agent then said? This is the only check that catches the failure described in sign six.
7. Calibrated LLM-as-judge. A model can score outputs at volume against a rubric — but only once its verdicts have been checked against human labels on the same cases, and it should be treated as a metric, not a methodology.
8. Human review sampling. Still the ground truth, and the thing every automated check above is calibrated against. The question is not whether to do it but how to sample so it stays affordable.

What “AI hallucination checkers” actually detect — and don’t

Tools marketed as hallucination detectors mostly do one of the things above, usually groundedness scoring: given a source and an answer, they estimate whether the answer is supported by the source. That is genuinely useful and it has three limits worth knowing before you rely on one.

First, supported is not true. A checker can tell you an answer is not backed by the provided context. It cannot tell you the context itself was wrong — the glue-on-pizza problem — or that an unsupported claim is nonetheless correct. Second, detectors have their own error rates, which are rarely stated on the tin and vary with domain; a checker tuned on news summaries will behave differently on engineering reports. Third, they are not the same thing as “AI content detectors” — the tools that try to guess whether a student’s essay was machine-written. That is a different problem with a different, and generally poorer, track record, and the two get confused constantly.

Detection is not measurement

Everything above tells you about instances: this answer looks invented, this citation does not resolve, this trace does not match its narration. None of it tells you your rate — how often the feature does this, on your traffic, against your definition of correct. A rate needs a denominator, a sampling method, a rubric fixed before results are seen, and a re-run on every change. What a hallucination rate actually measures sets that out.

The two jobs are complementary and they live in different places. Detection runs at request time, as a guardrail: catch this one before the user sees it. Measurement runs before release, as a gate: know how often it happens and whether the last change made it better or worse. A team with guardrails and no gate will catch individual failures and never know whether the feature is improving. A team with a gate and no guardrails will know the rate precisely and still ship every instance of it.

The number behind the checks

Once a feature has a measured rate, the checks above stop being a list of things you could do and become choices with a cost and a payoff: reference resolution catches this segment of failures, groundedness scoring that one, and neither touches the agent trace problem, which is a third of the total. That is what LLM evaluation gives you — not a detector, but the measurement that tells you which detectors are worth running and whether they are working.

Frequently asked questions

How can you tell if ChatGPT is hallucinating?

Look for specific claims that arrive with no source, ask where a fact came from and see whether the answer is a real reference or a reference-shaped one, check that any citation, URL or title actually resolves, notice whether the tone is equally confident about obscure facts and well-known ones, and ask the same question two or three different ways to see whether the answer holds. None of these is proof, but a fabricated reference that does not exist when checked is close to it. For anything that matters, verify the specific claim against a source you trust rather than asking the model whether it was right; it will usually say yes.

Is there a tool that detects AI hallucinations?

Yes, several, and most work by groundedness scoring: given the source material and the answer, a classifier or a second model estimates whether each claim is supported by the source. Open-source faithfulness models and the groundedness checks built into the major AI platforms both do this. They are useful with three caveats: supported is not the same as true, so they cannot catch an answer grounded in a wrong source; they have their own error rates, which vary by domain; and they are a different thing from AI content detectors, which try to guess whether text was machine-written and are frequently confused with them.

Can an AI detect its own hallucinations?

Only partially, and not by being asked. A model asked whether its previous answer was correct will usually say yes, because that is the plausible continuation. What works better is structural: sampling several answers and measuring agreement, having a separate model judge whether the answer is supported by the provided context, or constraining the output so that invention becomes a detectable schema violation. Each of these uses a model, but none of them relies on the model knowing what it does not know, which is the capability it lacks.

What is hallucination detection in AI?

Hallucination detection is the set of techniques for identifying, at the level of an individual output, that a generative model has produced a claim not supported by its inputs or by fact. In practice it covers mechanical checks such as verifying that cited references exist, groundedness or faithfulness scoring against retrieved context, self-consistency sampling across repeated generations, schema validation of constrained outputs, and for agents, comparing the model's narration with its actual tool calls. It is distinct from hallucination measurement, which establishes how often a feature does this across a sample of real traffic.

How do you detect hallucinations in a RAG system?

Measure retrieval and generation separately. Log which documents were retrieved for each query and check whether the right one was among them, because a confident wrong answer built on the wrong document is a retrieval failure that no generation check will catch. Then score the generated answer for groundedness against the retrieved context, so that claims not supported by the documents are flagged. Finally, check that any citations the answer makes point to the documents actually retrieved. A RAG system that passes a groundedness check can still be wrong if the retrieval step returned the wrong source, which is what happened in several of the widely reported cases.

How do you catch hallucinations in an AI agent?

By inspecting the trace, not the reply. An agent hallucinates in the middle of a task: it narrates having called a tool it did not call, reports a result different from what the tool returned, or describes an action it never took, and then produces a final answer that reads perfectly well. The check is to compare the agent's stated actions with its actual tool calls and their arguments and results at every step. That is only possible if the trace is logged in full, which is why observability of tool calls is a prerequisite for evaluating agents at all.