How to build an AI chatbot that doesn't hallucinate? (RAG approach)

AI chatbots hallucinate because they generate answers based on general training data, not on your specific content. RAG (Retrieval-Augmented Generation) solves this: with every query, the system first retrieves relevant snippets from your own documents (FAQ, product information, knowledge base) and provides them as context to the LLM. Result: the model cites your sources instead of making things up. Setup requires a vector database (Pinecone, Weaviate, or Postgres with pgvector), embeddings (OpenAI's text-embedding-3 or similar), and a well-structured document store.

Why do AI chatbots actually hallucinate?

An LLM (GPT-4o, Claude, Gemini) is trained on billions of tokens from the internet. It knows a lot but doesn't know what it doesn't know — if you ask "what are your opening hours?" and it doesn't have that information, it generates plausible-sounding text. That is a hallucination.

Three typical triggers for hallucinations:

Questions about specific information not present in the training data (your product information, prices, processes).
Questions about recent events after the training cutoff.
Questions where the answer is difficult to verify but plausible.

What RAG does — in one picture

RAG (Retrieval-Augmented Generation) consists of three steps for each query:

1. Embedding the query. The user query is converted into a numerical vector via an embedding model (OpenAI's text-embedding-3-large, or mxbai-embed-large open-source).
2. Searching in a vector database. This vector is compared with vectors from your documented content (FAQ items, product pages, manuals, blog posts). The top 5 or top 10 most relevant snippets are retrieved.
3. Generating an answer with context. The LLM receives the original query plus these top-relevant snippets as context, with the instruction: "Answer only based on this context. If you don't know? Say so."

Result: the model cites your sources instead of making things up. And you can even add links to the source pages.

The components you need

Document store. Your existing content: FAQ, product pages, blog, knowledge base, manuals. Quality of your documents = quality of your chatbot.
Chunking strategy. Documents are split into chunks of 200-500 tokens (paragraphs or sections). Chunks that are too large = inaccurate retrieval; too small = loss of context.
Embedding model. OpenAI text-embedding-3-large ($0.13/M tokens) is industry-standard. Open-source alternatives: bge-large, mxbai-embed-large.
Vector database. Pinecone (managed, $70+/month), Weaviate (open-source), Postgres with pgvector (free, self-hosted), Supabase (Postgres + pgvector built-in).
Generation LLM. Claude 3.5 Sonnet or GPT-4o for production; Gemini Flash or GPT-4o-mini for budget cases.
Orchestration layer. LangChain or LlamaIndex (Python/TS), or custom-written service in Node.js/Python.

What it costs in practice

For an SME website with 200 FAQ items, blog posts, and product pages (total ~50,000 words) and 1,000 chatbot queries/month:

One-time build: €8,000-€20,000 (architecture, ingestion pipeline, frontend, prompt engineering, testing).
Embedding costs first indexing: €1-€5 (one-time).
Vector DB: Postgres+pgvector: ~€10/month. Pinecone: €70/month.
LLM costs: GPT-4o-mini for 1,000 queries ~€5/month. Claude 3.5 Sonnet for 1,000 queries ~€30/month.
Re-indexing for new content: incremental, a few cents per update.

Total operational: €30-€150/month for an SME bot. Plus 4-8 hours/month maintenance (content updates, prompt tweaks).

How to maintain quality

Only document what you want the bot to know. No "internal notes" or "draft" content in your index.
Update your index with every content change. Automate via cron or webhooks.
Build an "I don't know" flow. If RAG finds no relevant chunks: let the model say so and offer a form or live chat.
Monitor answers. Save query + answer + source chunks; weekly review the worst 10% (based on user feedback or unanswered flag).
Cite your sources. Display the source page below each answer. Builds trust and helps users with follow-up.

Common mistakes

Underestimating document quality. Vague or inconsistent content leads to vague or incorrect answers. Clean up your content first.
Chunking too greedily. Entire pages as one chunk lead to irrelevant retrieval. Split by logical section.
No system prompt with rules. Without explicit instructions ("Answer only based on the context", "Do not hallucinate", "Refer to contact if in doubt"), the model can still draw from its general knowledge.
No evaluation set. Create a set of 50-100 questions with good answers, run them as a regression test with every change.
Forgetting to give the bot a tone. RAG output without a consistent tone sounds clinical. Give the bot a persona in your system prompt.

Frequently asked questions

Does RAG also work in Dutch?

Yes. Embedding models like text-embedding-3-large work multi-lingually with good performance for Dutch. Generation models (GPT-4o, Claude 3.5) have strong Dutch output.

How do I choose between Pinecone, Weaviate, and pgvector?

Pinecone: managed, simple, $70+/month. Weaviate: open-source, powerful, self-host or cloud. pgvector: free with PostgreSQL, perfect for SMEs up to ~1M documents. For SME projects, usually pgvector or Supabase.

Can I train my AI chatbot myself on my data?

Fine-tuning is possible but rarely the right solution. RAG is cheaper, faster to update, and provides better control than fine-tuning for most SME use cases.

How much content do I minimally need for a good RAG bot?

For a usable bot: at least 50-100 well-structured FAQ items or similar (5-10k words). With less, you'll get weak answers or many "I don't know" moments.

Does it also work with PDFs and Word files?

Yes. Tools like LangChain's document loaders parse PDF, Word, Markdown, HTML. Quality of extraction varies — for complex PDFs (tables, formulas): test first.

What if the chatbot still hallucinates?

Add post-processing checks: compare the answer with retrieved chunks (Claude's "citation" function or a custom check). If the answer contains words not in chunks: flag or ask the user to rephrase.

Ready to get started?

Read our approach for an AI assistant for your website, or schedule a brief introductory meeting — we'll take a no-obligation look and provide an honest estimate of scope, costs, and lead time.