Grounded generation: making an AI writing agent dependable without changing the model

An AI agent that writes for businesses must not invent facts about them. This paper describes a method that puts a verified evidence boundary, deterministic checks, an LLM reviewer and a bounded revision loop around an existing writing agent, and evaluates it blind on 19 production blog assignments. With the model held constant, blogs containing a blocking factual error fell from 5 to 0, reviewer passes rose from 3 to 12, and a blind judge preferred the grounded output in 13 of 19 cases. We describe each component in enough detail to reproduce it, report the measures that did not improve, and discuss how the method transfers to other agents.

1. The problem: fluent, confident, unverified

The client operates a growth-marketing platform whose customers are small and mid-sized businesses: recruiters, accountants, consultancies, trades. One of its agents writes blog posts on a customer's behalf from a title, the customer's notes and the customer's onboarding profile. The output read well. That was the difficulty: the writing was fluent enough that nobody could tell, without checking, which statements were true.

An audit of production output found three recurring failure classes:

• Invented figures. Tax rates, office counts, percentages and dates that appeared nowhere in the customer's material. One blog on R&D tax credits contained 22 numeric claims, none of them supplied by the customer, including a specific relief rate and effective date that were simply wrong.
• Scope mixing. A real figure attached to the wrong subject. A recruiter's website says it has offices in 45 countries and has placed candidates in 91; the agent wrote "offices in 91 countries". Both numbers exist; the claim is false.
• Padding and repetition. The same statistic restated in four sections, generic paragraphs any competitor could publish, and 1,600-word posts against a 1,200-word brief.

Prompt instructions such as "use only the provided input data" were already present and did not prevent any of this. The model had no explicit list of what it was allowed to assert, nothing examined its output against such a list, and the team had no measurement that would tell them whether a prompt change had helped or harmed. Each fix was an opinion.

2. Method overview

The method leaves the writer's prompt template and output schema untouched and wraps five components around it. Figure 1 shows the data flow. The components are:

1. Evidence preparation (section 3): crawl the customer's website, extract facts with verbatim quotes and confidence scores, consolidate them into a register with one value per subject, and approve only those above a confidence threshold and free of conflicts.
2. Grounded writer (section 4): the original writer, given a token-budgeted grounding pack of approved facts, excluded facts, previously published material and explicit numeric rules.
3. Deterministic checks (section 5): six code-level checks with no LLM involved, which parse every number in the draft and test it against the allowed inputs.
4. Reviewer (section 6): an LLM that receives the brief, the grounding pack, the check findings and the draft, and returns a pass/fail verdict with typed issues.
5. Revision loop (section 7): if any blocking issue remains, the writer receives every issue and its previous draft and rewrites once. Output that still blocks is withheld, not published.

Accepted output is ingested into a content memory, so each later piece is checked against what the customer has already published. Every run is written to a ledger that records the pack, every round, every issue and every reviewer verdict, which is what makes the evaluation in section 8 possible.

3. Evidence preparation

Everything downstream depends on the quality of the fact list, so this stage is the most heavily engineered. It runs as a background job of one to two minutes per customer, and re-runs only when page content changes.

3.1 Crawl

A breadth-first crawl starts at the customer's website, stays on the same host, honours robots.txt, and stops at 15 pages or depth 2. Links are ranked so that home, about, services and case-study pages are fetched before legal and careers pages. Each page is stored with its cleaned text, word count, a content hash and a page type. A page whose hash has not changed within 30 days is not re-fetched and keeps its previously extracted facts. No model is involved.

3.2 Extract candidate facts

One model call per page (five in parallel) returns a page summary and 5–20 candidate facts. Each candidate carries a text, a category (presence, services, numbers, people, history, claims, clients, other), a subject, a numeric value where one exists, a quote that must be copied verbatim from the page, and a confidence between 0 and 1. The extraction prompt is deliberately literal:

• "Only facts the page actually states. No inference, no industry knowledge, no filler."
• "'80 offices in 45+ countries' is TWO facts: (subject office, value 80) and (subject country, value 45+)."
• Case-study figures "belong to the client … never phrase them as if the company achieved them."
• Placeholder counters ("0+", "£0M") are unrendered page widgets, not facts.

Code guards sanitise the result: a value with no digit is discarded, and zero placeholders have their confidence capped at 0.3. Candidates are stored as pending and de-duplicated on normalised text.

3.3 Consolidate into a fact register

A second, more capable model reads all candidates for a customer and produces a register: at most 30 entries, each citing the candidate IDs it was merged from. The consolidation rules are where most of the scope-mixing protection lives:

• One value per subject. "If two candidates give different values for the same subject (e.g. '70 countries' and '45+ countries'), keep BOTH as separate entries with the same conflict_group and a note explaining the likely scope of each. Never silently pick one."
• Shared nouns must say what they count. An entry such as "Candidates have been placed in 91 countries (placements, not offices; the company has offices in 45 countries)" carries its own scope note into the grounding pack.
• Expect to drop most candidates. A good register for a small company is 10–20 entries.

Code then enforces what the prompt asked for: entries below confidence 0.70 are dropped; calls to action, contact details, headings and enumeration counts ("4 ways to…") are removed as filler; false conflict groups (three or more members, or every subject an enumeration word) are cleared; same-subject entries that share no conflict group keep only the more confident one. The register is sorted by category priority, then confidence, and truncated to 30.

3.4 Approve

A register entry becomes approved when its confidence is at least 0.80 and it belongs to no conflict group. Conflicts are never auto-approved: the customer decides. Raw candidates are never approved once a register exists. A re-run of consolidation deletes only pending register rows, so a customer's earlier approvals and rejections are never discarded by the system.

4. The grounded writer

The writer's system prompt is the original template, byte for byte. What changes is the user turn, which now begins with a grounding pack assembled per assignment within a budget of roughly 1,800 tokens. Its sections, in order:

1. Approved facts ("the only numbers you may use"): the twelve approved entries with the highest keyword overlap with the title and notes, each printed with its scope note.
2. A shared-subject warning, generated automatically when one noun carries several approved values: "Each figure measures something different. Whenever you use one, state in the same sentence what it counts… never combine them into a single claim."
3. Do not use: facts the customer rejected or that are out of scope.
4. Already published: summaries of up to five prior pieces, statistics already used ("use sparingly, at most once") and distinctive sentences ("do not repeat these").
5. Rules: "every number, percentage, date or count must come from the approved facts or the user's notes; if no figure is supplied, make the claim without a number. DO use the approved figures: a piece that avoids them reads as generic. Use each important figure exactly once, in the sentence where it does the most work, and refer back to it afterwards without the number… Do not open more than one section with a figure."

When the budget is exceeded, sections are trimmed in a fixed order: sources first, then recalled phrases, then recalled pieces, then the brand-voice sample. Facts are never trimmed. The pack also exposes an allowed_texts() set: company name, description, industry, key messages, keywords, approved fact texts, source summaries, and the user's own title and notes. That set is the whitelist the checks use.

Two design points deserve emphasis. First, the rules push the writer towards the approved figures, not away from numbers altogether; a grounded piece that refuses to use any figure reads as generic, and the reviewer penalises that too. Second, the pack is in the user turn rather than the system prompt so that the revision request in section 7 can reproduce the original message exactly and append to it.

5. Deterministic checks

Checks run in code, cost nothing, and cannot be talked out of a finding. They begin with a number extractor that handles currency, thousands separators, decimals, percentages, "+" and "×" suffixes, m/bn/k units (normalising "5m" to 5,000,000), spelled-out numbers ("forty-five countries"), and year detection ("founded in 1978"). For each mention it records the raw string, the value, the subject (the singular noun following the figure, skipping articles and possessives) and the sentence.

Only two findings block on their own: a figure with no source, and a figure the customer rejected. Everything else is evidence. A subject mismatch, for instance, is often a parsing artefact ("45 members of staff across our countries"), so it is passed to the reviewer rather than enforced. This division of labour, strict where the parser is reliable and advisory where it is not, is what keeps false positives from making the loop useless.

6. The reviewer

The reviewer is an LLM call with a narrow remit: "No praise, no rewriting." It receives the brief (title and notes), the full grounding pack, the check findings formatted as [code] message — in: "sentence", and the draft. It returns {passed, summary, issues[]}, where each issue has a code, a severity, the offending sentence (at most 15 words quoted) and a reason. At most six issues are reported.

Its fail criteria are written as tests rather than qualities:

• generic_copy: a paragraph could be published by any company in the industry with no edits.
• off_brief: the draft does not deliver what the title promises, or ignores the user's notes.
• unsupported_claim: a specific non-numeric claim about the company (capability, award, client, history) that is not in the grounding pack. A figure supported only for a different subject also lands here.
• repeated_idea: the same point is made three or more times in different words.
• tone: the voice clearly contradicts the tone or key messages.
• needs_sources: the notes ask for a statistic the pack does not contain. Always a warning, never a failure, because the writer cannot fix it.

Two instructions govern how it treats the automated findings: "Findings are evidence, not verdicts: decide each one," and shared-noun figures are acceptable "when each sentence in the draft says what its figure counts." Severity is "block" for "anything that would embarrass the company if published", and passed is true only when there are no blocking issues. If the reviewer itself fails twice, the piece is marked needs_attention rather than passed by default: an unavailable reviewer is not a passing reviewer.

7. The revision loop

The loop is bounded at two rounds: one draft and at most one rewrite. The bound is deliberate. An unbounded loop converts a bad brief into an expensive bad brief, and the ledger showed that a second rewrite rarely fixed what the first had not.

Four terminal states exist. accepted: no issues. accepted_with_warnings: no blocking issues. needs_attention: blocking issues after the final round, or reviewer unavailable; the piece is not published and the user sees the top three blocking issues with the message "We did not publish this draft because it did not pass our accuracy review." failed: the model call itself failed. Every round, with its output, issues and reviewer verdict, is written to a ledger in its own transaction, so failed attempts are preserved even when the surrounding request is rolled back.

8. Evaluation design

A method that cannot be measured cannot be improved, so the evaluation harness was built alongside the pipeline rather than after it. The question it answers is narrow: for the same assignment, is the grounded output better than what the client actually shipped?

Three measures are recorded per assignment and kept separate, because they answer different questions:

1. Blocking findings from the deterministic checks: did the text contain a figure with no source, or a rejected one? This is the factual-risk measure and does not depend on model judgement.
2. Reviewer verdict: did the text pass the reviewer's criteria? This is the workflow-acceptance measure.
3. Blind pairwise preference: which text did a judge prefer, given the brief and the approved facts, with accuracy ranked above repetition, specificity, brief and readability, and the instruction to "prefer the post with fewer accuracy faults even if the other reads better"? Order is randomised per pair with a fixed seed, and the judge's confidence and reasons are stored.

A synthetic stress suite complements the production benchmark: six invented companies with known traps (shared-noun scope mixes, a rejected figure, a founding year that invites "N years in business", a previously published signature sentence), run under three variants (baseline: no pack, no checks; pack_only; full) and scored deterministically. It is used to catch regressions when prompts or thresholds change; the production benchmark is used to claim results.

9. Results

The primary benchmark covers 19 production blog assignments from the client's customers, written once by the original process and once by the grounded process. Both runs used the same writer model (GPT-5), so the difference is attributable to the method.

9.1 Factual risk

The original texts contained 26 blocking findings across 5 of the 19 blogs; 21 of the 26 were in a single tax-credit article. The grounded texts contained none. This is the result the method was built for, and it is also the one that needs the most careful phrasing: zero findings in this sample means that no figure without a source reached the output, not that the checks cannot be evaded. The mechanism is simple enough to trust, though. A figure not in allowed_texts() blocks, the writer is told exactly which sentence to fix, and a second offence withholds the piece.

9.2 Workflow acceptance

Reviewer passes rose from 3 to 12 of 19. The seven grounded blogs that did not pass are instructive: the reviewer's objections were repetition, a missed element of the brief, or a note that asked for a statistic the register did not hold. None was an unsupported claim. In practice these seven need a short human edit rather than a rewrite.

9.3 Blind preference

The judge preferred the grounded text in 13 assignments, the original in 4, and could not separate 2. Judge confidence was higher on the production assignments (80–90%) than on the client's earlier internal tests (42–66%), which were written against thinner fact registers. The four cases where the original won are kept in the report: in three of them the judge preferred the original's tighter structure, and in one the grounded text had misattributed a real figure to the wrong subject, which the judge, correctly, treated as the most serious fault present.

9.4 Problem types, before and after

Figure 6 breaks the findings down by type, summed over all 19 blogs. It shows where the method worked and where it did not.

10. Model choice: GPT-5 against GPT-6

The client also asked whether a newer model would help. Seven of the nineteen assignments were re-run with the grounded pipeline on GPT-6, with the reviewer and judge also on GPT-6. Compared with the original production output, V2 on GPT-6 was preferred in 7 of 7, had no blocking findings, passed the reviewer in 4 of 7, and scored 8.6 against 6.7 out of 10 on a separate five-criterion quality rubric. But that comparison changes the model and the method at once, and most of its gain is the method's. The informative comparison is V2 on both models for the same seven assignments:

The two models are indistinguishable on factual risk, which is the point: safety came from the system, not the model. GPT-6 wrote the text readers preferred, including in the two assignments where the GPT-5 output did not beat the original. GPT-5 was ahead by one blog on the reviewer and hit the length target more closely. We recommended GPT-6 for writing quality, with the caveat that cost per piece was not measured and that each model judged its own output; the next batch uses a single neutral judge across both.

11. Limitations and open problems

This is a 19-assignment evaluation of one agent for one client. The preference measure comes from an LLM judge, not from readers. "Blocking" and "pass" mean what the checks and the reviewer define them to mean. The numbers describe this evaluation, not a guaranteed error rate.

Three problems are open and are being worked on:

• Subject mismatch rose (12 → 27). The grounded writer uses more approved figures, as instructed, and the number extractor's subject detection (the noun after the figure) misfires on sentences such as "45 countries' worth of placements". Most of the 27 are parser artefacts rather than wrong claims, which is why the finding is advisory, but the count hides the real cases. The fix in progress is a subject resolver that uses the fact's own scope note rather than the adjacent noun.
• Statistic reuse across blogs (33 → 27). Customers with small registers have perhaps a dozen usable figures, so a tenth blog for the same customer cannot avoid reusing them. The rule "at most once per piece" is enforced; "rarely across pieces" cannot be without expanding the register. The practical answer is a customer review page where the customer can add facts the crawler did not find, which also resolves conflict groups properly instead of leaving them held.
• Single-model judging. In the model comparison, each model reviewed and judged its own output. The pairwise judge is a separate prompt and is given the approved facts, which limits but does not remove self-preference. A neutral judge is the obvious fix and is scheduled.

Two engineering notes for anyone reproducing this. First, the fact pipeline has two thresholds, 0.70 to survive consolidation and 0.80 to approve, and they are easy to confuse; the gap is intentional, so that a borderline register entry is kept visible for the customer rather than dropped. Second, the writer prompt inherited a contradiction between "do not use markdown" and a field description that demands markdown. It did not affect results, but it is the kind of thing a ledger makes visible and a prompt-only workflow does not.

12. Transferring the method

Nothing in sections 3–8 is specific to blogs. The method applies to any agent whose output makes claims that can be checked against something: a support agent answering from a policy document, a sales assistant quoting prices, a research agent summarising sources, a report generator working from a dataset. The transferable pattern is:

1. Name the failures that cost you, from real output, in terms a check could detect. "Hallucination" is not a failure class; "a figure with no source" is.
2. Build the evidence boundary before the prompt. Whatever the agent is allowed to assert should exist as a list with provenance, confidence and status, and the agent should see only the approved subset.
3. Check in code where code is reliable, and in a model where it is not. Make the deterministic checks strict and the model's judgement advisory over them, not the other way round.
4. Bound the loop and withhold on failure. One rewrite with explicit, sentence-level feedback; then stop and tell the user why.
5. Keep a ledger and benchmark on real work, blind, before and after. Keep the factual-risk, acceptance and preference measures separate, and publish the misses.

The cost is real: a grounded piece takes around a minute instead of seventeen seconds, and the evidence pipeline is more code than the writer it protects. For output that carries a customer's name, that trade has so far been easy to justify.

Appendix: definitions and data

A.1 Terms

A.2 Primary benchmark (19 assignments, same model)

A.3 Models used

Fact extraction: a small model, one call per page. Consolidation, reviewer and pairwise judge: a mid-sized model. Writer: the same GPT-5 model in both arms of the primary benchmark; GPT-6 in the model comparison of section 10. All calls at low reasoning effort. Provider calls are wrapped with three attempts and exponential back-off.

A.4 Reproducibility

Each generation record stores a hash of the prompt template, the rendered grounding pack, the recalled memory IDs, every round's output and issues, the reviewer's verdict and token usage, and timing. The benchmark's A/B shuffle uses a fixed seed. Given the ledger and the fact register at the time of the run, any result in this paper can be re-derived.

Ferrous Labs builds and improves AI agents for businesses that need their output to be dependable. The client and its customers in this paper are anonymised; figures are from the client's production benchmark of 1–2 October 2026 and have been checked against the generation ledger. Results describe this evaluation and are not a guarantee for other agents or domains.