RAG Systems in Production: What the Tutorials Don’t Tell You

The demo gap

Every RAG tutorial shows the same thing: embed some documents, store in a vector database, retrieve top-k results, pass to an LLM, get an answer. It works on clean prose. Then you apply it to real enterprise documents — technical manuals, regulatory filings, internal policy docs — and things fall apart in ways that are hard to diagnose.

I’ve built RAG systems over financial reports, telecom technical documentation, and billing policy documents. Here’s what the tutorials skip.

Chunking is the most consequential decision

Most tutorials chunk by character count — split every 500 characters, overlap 100. This is catastrophic for tables (a mid-table split loses all context), numbered lists (items end up in different chunks), and multi-page sections (heading in one chunk, content in another).

What actually works: semantic chunking based on document structure. Parse into sections first, embed at section level. For tables, embed header plus each row separately with table context prepended.

def chunk_document_semantically(doc_text):
    sections = extract_sections(doc_text)  # heading-based parsing
    chunks = []
    for section in sections:
        if section.type == "table":
            header_ctx = f"Table: {section.title}\nColumns: {section.headers}"
            for row in section.rows:
                chunks.append(f"{header_ctx}\nData: {row}")
        elif len(section.content) > 1500:
            # Split long sections at paragraph boundaries only
            for para in section.content.split("\n\n"):
                if para.strip():
                    chunks.append(f"{section.title}\n{para}")
        else:
            chunks.append(f"{section.title}\n{section.content}")
    return chunks

The “I don’t know” problem

The most dangerous RAG failure is confident hallucination — the LLM retrieves something vaguely related, finds no direct answer, and synthesises a plausible-sounding but wrong response. For business documents this isn’t just embarrassing — it can cause real decisions to be made on false information.

SYSTEM_PROMPT = """Answer ONLY using the provided context documents.
If the answer is not in the context, say exactly: 
"I cannot find this information in the available documents."
Never infer or extrapolate. Always cite the source section."""

Hybrid retrieval outperforms pure vector search

Pure semantic search misses exact matches — product codes, section numbers, specific names. The solution is hybrid retrieval: combine dense vector search with sparse BM25 keyword search, merge results using reciprocal rank fusion. In practice this means two retrieval calls and a merge step — the combination consistently outperforms either approach alone.

Evaluation that actually means something

Don’t evaluate on a test set you built yourself — you’ll overfit to your assumptions. Build evaluation from actual user queries. Have a domain expert rate answers 1–3 (wrong / partial / correct). Track this metric over time. A system that correctly answers 85% and refuses 15% is far preferable to one that answers 97% with 12% hallucination rate.