Part 1 — Why Standard RAG Fails in Production

1.1 The fundamental LLM problem

Large language models are extraordinary at generating text that is coherent, fluent, and contextually appropriate. They are deeply unreliable at knowing facts. Their knowledge is frozen at a training cutoff. They have no access to your organisation’s internal documents. When they encounter a question whose answer lies outside their training distribution, they do the most dangerous thing possible: they answer anyway, confidently, in fluent prose. This phenomenon — known as hallucination — is not a bug that will be patched away. It is a structural property of how autoregressive language models work.

For consumer applications, hallucination is annoying. For enterprise applications — legal analysis, medical decision support, financial research, engineering compliance — it is a liability. The system must be grounded in your documents, your data, with claims traceable to your sources.

1.2 The first-generation RAG response

Retrieval Augmented Generation (RAG) was the first serious answer: instead of asking the LLM to generate from parametric memory alone, retrieve relevant context from a knowledge base and give it to the LLM before generating. The LLM becomes a synthesizer and reasoner over supplied evidence rather than a source of facts.

First-generation RAG achieved this through a simple three-step pipeline: chunk documents into fixed-size pieces, embed them with a single embedding model into a dense vector index, and at query time retrieve the top-k nearest chunks by cosine similarity, then pass them to an LLM.

This works. On demo datasets. With clean, digital, single-modal, English-language documents. In controlled evaluations. In a single-tenant setting. Without adversarial queries.

It fails in production, consistently, for reasons that are entirely predictable once you name them.

1.3 What first-generation RAG gets wrong

The documents are not clean digital text. Enterprise knowledge lives in scanned PDFs with misaligned columns, PowerPoint decks with text in image layers, video recordings of design reviews, spreadsheets with merged cells, code repositories, and HTML pages with JavaScript-rendered content. A naive chunker that splits every document at 512 characters destroys structure that took years to create.

A single embedding model is a single point of failure. Dense vector similarity retrieves semantically similar text. It routinely misses documents containing exact technical terms, product codes, legal statute references, or proper nouns that embeddings map to the same region as dozens of unrelated concepts. Conversely, BM25 keyword retrieval has excellent exact-match recall but zero semantic generalisation. Neither alone is sufficient.

Fixed-size chunks strip context. A chunk reading “revenue declined 12% quarter-over-quarter” is meaningless without knowing whose revenue, which quarter, which product line. Extracted from a 300-page annual report and retrieved in isolation, it is actively dangerous — a plausible-sounding figure that may answer a completely different question.

A pipeline has no self-awareness. Once a naive RAG pipeline retrieves and generates, it is done. It cannot tell whether what it retrieved was relevant. It cannot tell whether what it generated is grounded in the retrieved context. It cannot route a multi-hop question to a different strategy. It cannot decide that the corpus simply does not contain the answer and return “I don’t know.” It just generates, every time, from whatever it retrieved.

Scale and multi-tenancy are afterthoughts. First-generation RAG treats the knowledge base as a single, globally-accessible repository. Enterprise systems have tenants with strictly isolated data, users with role-based access to specific document sets, compliance requirements that mandate auditability, and corpora that reach hundreds of millions of chunks.

Multimodal Hybrid Agentic RAG is the structured response to every one of these failures.