Part 3 — Real-World Challenges: The Honest Picture

Before designing patterns, the challenges must be named precisely. Vague awareness of “hallucination” isn’t enough; the specific failure modes and their sources determine which patterns are worth implementing. A useful lens for the whole page: almost every failure here is either silent (it produces plausible output and never errors) or costly (it quietly multiplies your spend). That’s why observability, at the end, isn’t optional.

3.1 Ingestion challenges

You demo the system on your clean Confluence wiki, ship it, and the scanned contracts and slide decks that actually mattered come back as gibberish — and nothing errors out.

Format diversity. No single parser handles scanned PDFs, PPTX, audio, source code, and HTML with equal fidelity. Each format demands a different strategy, and failures here — garbled tables, dropped figures, merged-cell misalignment — are invisible downstream. Retrieval will confidently serve the garbled version, and the LLM will confidently synthesize from it.

You point your PDF library at a scanned contract and get back an empty string — or worse, a confident but wrong number.

Scanned documents are not text documents. A printed-and-scanned PDF has no text layer, only a raster image. OCR is required, and its quality swings wildly with resolution, language, and layout. The dangerous part: OCR failures are usually silent— the engine returns output, and the output is quietly wrong.

You split every document at 512 characters and only later realise you severed each figure from its caption and each number from its heading.

Context destruction at chunking. Fixed-size chunking cuts sentences mid-thought and divorces content from the structure that gave it meaning. Every severed unit carries less information and more ambiguity than it should — retrieval precision suffers before a single embedding is computed.

You retrieve a table row that reads “87.4M | 12% YoY | Q3” and realise it’s useless — you threw away the column headers upstream.

Semantic orphaning.A figure without its caption, a sentence that says “as described above” without its referent — each is a retrieved fragment that can’t stand alone.

You embed code and prose with the same model and wonder why code search is mediocre.

Embedding model limitations.No embedder handles every modality equally: code has a different distributional character from natural language, multilingual content needs a multilingual model, and dense-only embeddings can’t reliably represent exact tokens.

You change one chunking parameter and suddenly face re-embedding twelve million chunks — again.

Scale and cost.Embedding APIs charge per token, and re-embedding unchanged documents burns money for nothing. Idempotency, content-addressable deduplication, and embedding caching aren’t optimizations here; they’re prerequisites:

# Content-addressable caching: never pay to re-embed unchanged text.
import hashlib

def embedding_key(text: str, model: str) -> str:
    digest = hashlib.sha256(text.encode()).hexdigest()
    return f"emb:{model}:{digest}"           # same text + model → same key

def get_or_embed(text: str, model: str, cache, embed_fn):
    key = embedding_key(text, model)
    if (hit := cache.get(key)) is not None:
        return hit                           # cache hit — $0, no API call
    vec = embed_fn(text, model)
    cache.set(key, vec)
    return vec

Because OCR fails silently, it’s worth gating it explicitly rather than trusting whatever comes back:

# Don't trust OCR blindly — quarantine low-confidence pages for review.
def ocr_with_gate(page, min_conf: float = 0.80):
    result = ocr_engine(page)                # returns text + per-word confidence
    mean_conf = sum(w.confidence for w in result.words) / max(len(result.words), 1)
    if mean_conf < min_conf or len(result.words) == 0:
        return QuarantineForReview(page, reason=f"OCR conf {mean_conf:.2f}")
    return result.text

3.2 The eight retrieval failure modes (F1–F8)

These are developed in full in the Retrieval Plane article; here they’re named with the query that exposes each and the fix. If you can write the trigger query for your domain, you can test for the failure.

3.3 Generation challenges

You fix retrieval, feel relieved, and then the model blends a correct chunk with a wrong one into a single fluent paragraph that looks authoritative.

Even with perfect retrieval, generation adds failure modes: the model may blend correct and incorrect claims, ignore a relevant chunk that contradicts a confident parametric prior, or produce a well-structured answer supported by no retrieved document at all — the most dangerous kind, because it reads as authoritative.

And you notice it’s always the chunk in the middle that gets dropped.

Long-context generation amplifies the lost-in-the-middle effect — models attend to the head and tail and underweight the middle. Chunk ordering in the prompt isn’t cosmetic; it’s a material accuracy decision.

3.4 Operational challenges

You onboard your second customer and discover tenant A’s documents can surface in tenant B’s answers.

Multi-tenancy.Different tenants must never see each other’s data. The only safe pattern is ACL-at-retrieval — applying access control inside the vector query, not after it. Post-hoc filtering fetches first and filters second, risking both leakage and recall degradation:

# WRONG — fetch everything, then filter. Leaks and wrecks recall.
hits = index.search(query_vec, k=10)
visible = [h for h in hits if h.tenant_id == user.tenant_id]   # may leave < k results

# RIGHT — constrain the search itself. The index never returns other tenants.
hits = index.search(query_vec, k=10,
                    filter={"tenant_id": user.tenant_id})       # ACL-at-retrieval

You add a self-correction loop, ship it, and your API bill triples.

Cost control. Agentic correction multiplies LLM calls. Those multipliers hit real spend:

Track cost-per-query as a first-class operational metric alongside latency — and apply the expensive loops selectively rather than to every query.

A system that works on 80% of queries and silently fails on the other 20% looks perfectly healthy — until a user shows you the 20%.

Observability. Without measurement, that 20% is invisible. The observability architecture must be designed in from the start, not retrofitted: the traces, span attributes, and metric hooks are structural dependencies of the correction and human-review loops, not add-ons.