LLM Evaluation: How to Know Whether Your AI Is Any Good

LLM evaluation is measuring an AI feature against a definition of what good means for it, so that quality becomes a number you can track rather than an impression formed from spot checks. It covers agents, retrieval pipelines and prompt chains alike, and it is the difference between knowing your AI feature works and hoping it does.

How good is your AI feature right now, compared with the day it launched? Most engineering leaders would say about the same, or a bit better after the prompt work. Ask how they know and the answer is usually a mix of opinion, a handful of screenshots and the absence of complaints. That is not measurement; it is the lack of it. And unlike ordinary software, an AI feature that has got worse looks identical to one that is working: same latency, same formatting, no errors, just a slightly worse answer every time, until a customer says so or stops saying anything.

The cost of that uncertainty is paid in slow releases, model upgrades nobody dares adopt, incidents that cannot be traced to a cause, and a board question ("how good is our AI?") that gets answered with a shrug. In this guide, Elliott Prince, Managing Director at Ferrous Labs, sets out what LLM evaluation actually involves: why buying an eval framework is not the same as having an evaluation, how to score an answer that has no single right version, what a hallucination rate really measures, and what good looks like for a team that wants to ship AI with confidence.

An eval framework is not an evaluation

The first confusion to clear up is the one the tooling market encourages. An eval framework, such as Langfuse, Braintrust, Ragas or DeepEval, gives you somewhere to put measurements and a way to run them. That is genuinely useful. What none of them can give you is the measurement itself: a definition of what "good" means for your particular feature, the cases that test it, and the thresholds that decide whether a change ships. Buying the framework and skipping that work is like buying a set of scales and never deciding what you are weighing.

So the honest first step is decomposition. Take the feature apart into the behaviours that matter to the people who use it. For a support agent that might be whether it cited the right policy, whether it quoted the right figure, whether it asked a clarifying question when it should have, whether it refused when it should have. For a document pipeline it might be field accuracy, completeness and what happens on a scan it has never seen. Each behaviour becomes a variable you can measure, and the set of variables is your definition of good. Everything else in evaluation hangs off that definition, which is why teams that skip it end up with dashboards full of numbers nobody trusts.

Four ways to score an answer with no single right answer

The fair objection at this point is that LLM output is free text. There is no expected value to compare against, so what exactly are you scoring? There are four approaches, and the order matters, because each one costs more and tells you less precisely what went wrong than the one before it.

Check what code can check. Far more of an AI's output is testable with plain code than most teams assume. Is the JSON valid? Do the line items add up to the total? Does every document it cites exist in what it retrieved? Is the date in range, is the answer under the word limit, did it avoid naming a competitor? These checks are cheap, instant and deterministic, and they catch a surprising share of the mistakes that embarrass you in front of customers.

Break written answers into facts you can check. This is the step most teams skip, and it is where the value is. Rather than asking a model "how good is this answer?", extract the specific claims it made (the policy cited, the figure quoted, the recommendation given) and compare each one to what it should have said. A model can do the extracting; "what number did this text state?" is a far easier question than "is this answer good?", and when the extraction is wrong you can see it. What you end up with are numbers that behave like numbers: citation accuracy 94 per cent, figure accuracy 89 per cent, correct recommendation 81 per cent. You can compare them across releases, trace a drop to a specific change and defend them in a meeting. Nobody can pick apart a seven out of ten.

Use a second model to mark what code cannot. Tone, helpfulness, whether an explanation is easy to follow, whether a refusal was right. These matter and no amount of parsing turns them into a field. A second model marking against a rubric can work, provided you know its habits: it is much better at comparing two answers than scoring one out of ten, it leans towards longer answers, and it is swayed by which option it sees first. Shuffle the order, control for length, and check its marking against human reviewers regularly.

Keep humans in, permanently. Not as a fallback but as the check on everything above: a small sample each week, read by someone who knows the domain, marked against the same rubric. It is the only way to catch the problems nobody thought to measure, and to notice when the automated numbers have stopped meaning what you think they mean.

What a hallucination rate actually measures

The most searched question in this whole area is "what is the hallucination rate of model X?", and the honest answer is that there is no such number. A hallucination rate is not a property of a model. It is a property of a model, on a task, on a set of inputs, against a definition of correct, and every published leaderboard fixes all four of those to things that are not your product. Public benchmarks (Vectara's is the best known) typically hand a model a document and score whether its summary stays faithful to it, which is a fine test of summarisation and says little about your agent answering your customers from your knowledge base.

Even the unit of counting changes the answer. A two-hundred-word reply with one invented sentence is a 100 per cent hallucination rate if you count per response and about 5 per cent if you count per claim. Neither is wrong; they measure different things, and you need to decide which one your users care about before you can set a threshold. Our guide to LLM hallucination rate works through how to measure your own, and the companion on what an AI hallucination is explains why models state false things with exactly the fluency they use for true ones: they are completing a shape as much as answering a question, and asked for a reference they produce something reference-shaped.

This is also why "AI hallucination checkers" disappoint. Detection on a single output is possible to a degree (our piece on how to tell if an AI is hallucinating lists the signs and the checks), but detection is not measurement. A rate needs a defined set of cases, run repeatedly, against a definition of correct, and that is an evaluation suite, not a plug-in.

Evaluating a RAG pipeline is three questions, not one

Retrieval-augmented systems add a wrinkle: an answer can sound right while citing the wrong source. So a useful RAG evaluation framework tests three things separately. What did the system find? Did it retrieve the passages a competent person would have pulled? What did it say? Is the answer faithful to what was retrieved, rather than to the model's general knowledge? And what happens when the evidence is missing? A system that confidently answers from nothing is more dangerous than one that says it cannot find the source, and only the third test will show you which you have built. Scoring all three tells you where a failing result came from, which is the whole point: a drop in answer quality caused by retrieval needs a different fix from one caused by the prompt.

What good looks like in production

Measuring once, at launch, is the most common pattern and arguably worse than not measuring at all, because it produces a reassuring flat line while real traffic drifts away from the test set. The teams who get this right share four habits.

Their evaluation set gates every release: change a prompt, a model, a retrieval setting or a tool, and the tests run, and a failing result blocks the release the way a failing unit test blocks a deploy. They treat that set as a living thing, with new cases arriving from production every week and old ones retiring when a workflow disappears, and someone owns it. They version everything together, so "which prompt was live on the 14th?" is a lookup rather than an investigation. And they report quality like they report uptime: one small, stable number shared with the business every month, so there is a regular moment when someone has to look.

None of this requires a large labelled dataset to begin. Existing production traffic usually provides the first cases, and a few dozen well-chosen ones, scored on the variables that matter, beat ten thousand generic benchmark rows. Where a feature is already live and the complaints are about accuracy, start from the complaints: each one is a test case you did not have. Our post on AI hallucinations in production covers where errors enter a live feature and what reduces the rate, in the order that actually pays.

What it looks like when it is done properly

The shape we use, which we call The Science of AI Reliability, runs in four steps: decompose the feature into the variables that matter, quantify each one with the smallest useful metric, evaluate against real cases to set a baseline, and control it by making the tests continuous. It takes four weeks for a typical agent or pipeline, and it is the engine behind our Agent Improvement Sprint. It is also what we are applying to the grounding and context behind Growth Gorilla's AI content assistants, where the question is not whether the assistants write well (they do) but whether they stay faithful to a business's own voice and facts as the suite grows.

Ali Nadian, our Technical Director, has written about why your AI may be getting worse without anyone noticing, and the three things shifting underneath every deployed feature: the model, your own accumulated changes, and your users. Evaluation is how you find out which one moved.

You cannot afford to guess

It is tempting to put evaluation on the "later" pile while there are features to ship. But every week without it is a week your AI could be getting worse with nobody the wiser, and the costs land all at once: on every customer, every day, for as long as nobody notices. So start small. Pick the one behaviour your users would complain about first, write twenty cases for it from real traffic, score them with code where you can and with claim extraction where you cannot, and put the number next to uptime in next month's report.

Once you can measure your AI, the whole posture changes. A new model release stops being a risk and becomes a free upgrade you can adopt in days. A dip in quality gets caught on a Tuesday afternoon rather than in next quarter's churn. And when the board asks how good your AI really is, you give them a number instead of a shrug. Evaluation feels like a luxury until you have it; afterwards it is hard to remember how anyone shipped without it.