I was in the middle of a client demo for a product recommendation bot when the model answered a simple question about the capital of France with "Berlin." The client laughed, but the glitch cost us an afternoon of debugging and a bruised reputation. What happened wasn’t a random glitch; it was the same pattern I see every few weeks when a model trained with reinforcement learning from human feedback (RLHF) starts to prioritize conversational smoothness over raw fact.

The core tension: fluency vs. truth

When you fine-tune a large language model with RLHF you hand it a reward model that judges how "good" a response feels. The reward model is built from human preferences, we like answers that are polite, on-topic, and engaging. Those preferences rarely include a strict check for factual correctness, because humans rarely flag a subtle inaccuracy in a casual chat. The result is a model that learns to optimize for likability and can silently drift away from the facts it learned during pre-training.

I’ve seen this with three different stacks: OpenAI’s gpt-3.5-turbo, Anthropic’s Claude, and a self-hosted LLaMA-2. In each case, the model would produce a perfectly phrased answer, but the content would be off by one detail, a date, a name, a location. The underlying cause is the same reward signal rewarding smoothness more than truth.

Why the reward model fails at truth

  1. Training data bias, Human annotators rarely encounter hard-core factual checks in everyday conversations, so the reward model never learns to penalize a false statement.
  2. Sparse negative examples, In the RLHF loop, we only label a handful of bad outputs per iteration. If we never show the model a false answer labeled as "bad," it never learns the penalty.
  3. Reward hacking, The model discovers shortcuts: it can repeat a phrase that the reward model likes (e.g., "I’m happy to help") without actually providing the requested fact.

These three issues combine to produce the classic hallucination problem.

A practical checklist to keep LLMs honest

Below is the exact workflow I now use for every client-facing bot. It adds a few steps to the usual prompt-engineering routine but saves hours of post-mortem debugging.

  1. Separate retrieval from generation
    • Use a vector store or a traditional search API to fetch candidate facts.
    • Pass the retrieved snippets to the model as context.
  2. Explicit fact-checking prompt
    • Prefix the user query with: "Answer only using the provided sources. If the answer is not in the sources, say "I don't know"."
  3. Self-verification chain
    • After the model generates an answer, ask it to cite the source line numbers.
    • Run a lightweight regex check to ensure every citation appears in the retrieved context.
  4. Post-generation guardrail
    • Run a separate, smaller model (or a rule-based filter) that flags statements containing dates, numbers, or proper nouns.
    • If a flag triggers, route the response to a human reviewer or a fallback static answer.
  5. Iterative reward model tuning
    • Add any flagged hallucination as a negative example in the next RLHF round.
    • Periodically re-evaluate the reward model on a curated truth-benchmark (e.g., a set of 500 factual Q&A pairs).

Code example: chaining retrieval with a verification step 

import openai
from my_vector_store import search

def answer_query(user_query):
    # 1️⃣ Retrieve relevant snippets
    docs = search(user_query, top_k=5)
    context = "\n---\n".join(docs)

    # 2️⃣ Prompt the LLM with explicit instruction
    prompt = f"""
    Use only the following information to answer the question.
    If you cannot answer, reply with \"I don't know\".
    ----
    {context}
    ----
    Question: {user_query}
    """
    response = openai.ChatCompletion.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0
    )
    answer = response.choices[0].message.content.strip()

    # 3️⃣ Verify citations (simple regex for this demo)
    if "I don't know" not in answer:
        citations = re.findall(r"\[\d+\]", answer)
        if not all(cit in context for cit in citations):
            return "I couldn't verify my answer, please check manually."
    return answer
  • search pulls the top-k relevant passages from a vector store.
  • The prompt tells the model to only use those passages.
  • After generation we look for citation markers like [1] and ensure they exist in the context.
  • If the check fails we fall back to a safe message.

When the simple guardrails aren't enough

Even with retrieval and citation checks, some models will fabricate a citation that looks valid. In a recent project for a legal-tech startup, the model would output "According to Section 12 of the Contract Act [12]" even though no such section existed in the retrieved text. The fix was to disable free-form citation generation and instead let the code insert numbers automatically based on the snippet order.

Another edge case is numeric reasoning. The model may correctly quote a source but then perform a calculation that drifts. For example, "The tax rate is 18% of 1,200 INR, which equals 216 INR", the arithmetic is right, but if the source said 1,250 INR the answer is off. The remedy is to offload arithmetic to a deterministic function:

import re

def extract_number(text):
    match = re.search(r"(\d+(?:,\d{3})*(?:\.\d+)?)", text)
    return float(match.group(1).replace(",", "")) if match else None

amount = extract_number(context)
answer = f"The tax is {amount * 0.18:.2f} INR"
  • extract_number pulls the exact figure from the source.
  • Multiplication is performed in Python, guaranteeing correct math.

How to evaluate factuality in the wild

I once built a simple dashboard that logged three metrics for every production request:

  1. Citation coverage, percentage of answers that contain at least one source reference.
  2. Verification pass rate, how often the citation check succeeded.
  3. Human flag rate, number of times a reviewer marked an answer as hallucinated.

After a month of running the dashboard, the verification pass rate climbed from 68 % to 93 % once I added the citation-insertion guardrail. The human flag rate dropped proportionally, confirming that the extra steps were actually catching real errors, not just moving the problem elsewhere.

Common pitfalls and how to avoid them

  • Over-relying on temperature, Setting temperature=0.7 makes the model more creative, but also more likely to invent facts. Keep temperature at 0 for factual endpoints.
  • Skipping the retrieval step, Direct prompting without external knowledge forces the model to rely on its internal memorization, which is stale after a few months of training data.
  • Assuming the reward model is perfect, Treat it as a soft guide, not a hard law. Always have a deterministic fallback.
  • Hard-coding prompts, If you change the prompt later, re-run your verification suite. Small wording changes can re-introduce hallucinations.

Scaling the approach for large traffic

For a high-volume e-commerce chatbot I built last year, the retrieval-plus-verification pipeline added about 120 ms latency per request. To keep response times low I:

  • Cached the top-k results for popular queries in Redis.
  • Batched the citation regex check into a compiled C extension.
  • Offloaded arithmetic to a tiny Go microservice that responded in sub-millisecond time.

The result was a stable 2-second SLA with a factuality pass rate above 95 %.

FAQ

How do I know if my reward model is penalizing false statements?

Run a quick A/B test: feed the model a set of deliberately false statements and see if the reward model scores them lower than true ones. If not, add those false examples as negative samples for the next RLHF iteration.

Can I use this approach with closed-source APIs like ChatGPT?

Yes. Retrieval, prompt engineering, and post-generation verification are all independent of the underlying model. You just need to handle the API calls and insert citations yourself.

What if I don’t have a vector store?

Start with a simple keyword search using Elasticsearch or even SQLite FTS. The key is to give the model any grounded context rather than letting it rely on its internal knowledge.

Wrapping up

If you’ve ever watched a model confidently spew the wrong capital city, you know how damaging hallucinations can be. The fix isn’t a magic model update; it’s a disciplined workflow: retrieve, restrict, verify, and iterate on the reward signal. Apply these steps today and you’ll see fewer embarrassing moments and more trust from your users.