Part 3 — Real-World Challenges: The Honest Picture

Before designing patterns, the challenges must be named precisely. Vague awareness of “hallucination” is not enough; the specific failure modes and their sources determine which patterns are worth implementing.

3.1 Ingestion challenges

Format diversity is a fundamental parsing problem. No single parser handles scanned PDFs, PPTX files, audio recordings, Python source code, and HTML simultaneously with equal fidelity. Each format demands a different parsing strategy, and failures at this layer — garbled tables, dropped figures, merged-cell misalignment — are invisible downstream. The retrieval system will confidently retrieve the garbled version, and the LLM will confidently synthesize from it.

Scanned documents are not text documents. A PDF that was printed and scanned contains no machine-readable text layer — only a raster image. Naively applying a text-extraction library to such a file returns an empty string or random noise. OCR is required, and OCR quality varies dramatically by resolution, language, layout complexity, and document condition. Critically, OCR failures are often silent— the engine produces output, but the output may be substantially incorrect.

Context destruction at chunking. Fixed-size chunking severs sentences mid-thought, splits paragraphs from their headings, and divorces figures from their captions. Every severed unit that gets retrieved carries less information than it should and more ambiguity than it should. Retrieval precision suffers before a single embedding is computed.

Semantic orphaning.A retrieved table row that reads “87.4M | 12% YoY | Q3” is useless without its column headers. A retrieved figure that shows a declining curve is useless without its caption. A retrieved sentence that reads “as described above” is useless without its referent.

Embedding model limitations. No embedding model handles every modality equally. Code has a very different distributional character from natural language. Multilingual content needs a multilingual embedder. Dense-only embeddings cannot represent exact tokens reliably.

Scale and cost.Embedding APIs charge per token. Re-embedding unchanged documents wastes money. Recomputing embeddings on every pipeline change — when you have millions of chunks — is not feasible. Idempotency, content-addressable deduplication, and embedding caching are not optimizations; they are prerequisites.

3.2 The eight retrieval failure modes (F1-F8)

These are developed in full detail in the Retrieval Plane article, but named here for completeness:

• F1 — Semantic gap: different vocabulary between question and answer; dense search misses it.
• F2 — Lexical miss: exact technical term, code, proper noun; dense embeddings smear it.
• F3 — Top-k cliff: correct chunk is rank 11; you fetched top-10; bi-encoder ranking is coarse.
• F4 — Multi-hop failure: the answer requires chaining facts across multiple documents; one vector lookup cannot do it.
• F5 — Lost-in-the-middle: correct chunks are retrieved but buried in the prompt; LLMs attend to head and tail, not middle.
• F6 — Distractor poisoning: one plausible-but-wrong chunk is retrieved alongside correct ones; synthesis blends them.
• F7 — No-answer / stale-answer: the corpus does not contain the answer; the pipeline generates anyway.
• F8 — Wrong-tool routing: a multi-hop query goes to flat vector search; a simple factoid goes to GraphRAG (which is empirically worse at simple factoids).

3.3 Generation challenges

Even with perfect retrieval, generation introduces failure modes. The LLM may synthesize across retrieved chunks in ways that blend correct and incorrect claims. It may ignore a relevant chunk that contradicts a confident prior from parametric memory. It may produce a fluent, well-structured answer that is not supported by any retrieved document — the most dangerous kind of hallucination, because it looks authoritative.

Long-context generation amplifies lost-in-the-middle effects. The model attends primarily to the first and last chunks; middle chunks are systematically underweighted. Chunk ordering in the prompt is not cosmetic — it is a material accuracy decision.

3.4 Operational challenges

Multi-tenancy.Different tenants must not see each other’s documents. ACL-at-retrieval — applying access control inside the vector query, not after — is the only safe pattern. Post-hoc filtering is vulnerable: it fetches documents first and filters second, risking both leakage and recall degradation.

Cost control.Every API call costs money. Agentic systems with correction loops multiply LLM call counts: Adaptive RAG approximately 1.5–2x, Self-RAG approximately 2–3x, CRAG approximately 3–5x, multi-hop ReAct approximately 4–10x versus vanilla. These multipliers apply to the underlying API spend. A production system must monitor cost-per-query as a first-class operational metric alongside latency.

Observability.A RAG system that works on 80% of queries and silently fails on 20% is operationally invisible without measurement. The observability architecture must be designed in from the start — not retrofitted — because the traces, span attributes, and metric hooks are structural dependencies of the correction and human-review loops.