Part 6 — The Retrieval Plane: Why Retrieval Fails

Retrieval is where most operational RAG accuracy is actually lost — and where it is most silently lost. A parse failure is visible (an empty table). A retrieval failure is not: the system returns a confident, fluent, fully cited answer — grounded in the wrong chunks.

One shape underlies almost everything here: retrieve wide, rank precise. Five of the nine patterns below are variations on it. And if you ship only three patterns to start, ship R1 (hybrid), R2 (rerank), and R8 (ACL pre-filter) — the first two are the core accuracy move, the third is a security floor that applies to every query. The rest are additive, activated as specific failures show up.

6.1 “My retrieval is bad” isn’t actionable — here are the eight failures that are

Naming these precisely is the prerequisite for designing against them. Each is written below as the symptom you’ll actually observe, with its F-code kept as a tag so it cross-references Parts 1 and 3.

6.2 “So what do I actually build?” — the nine patterns, and which failure each kills

The table is the reference; the three highest-leverage patterns get code below it. Note the coverage: every failure is killed by at least one pattern, but F5 is killed only by R7 — it’s the easiest gap to leave open.

R2 — wide retrieve, then rerank is the single highest-ROI upgrade. Retrieve generously so the right chunk is somewhere in the candidate set (fixes F3), then let a cross-encoder read query and chunk together to reorder precisely (fixes F6):

# R2: retrieve wide for recall, rerank narrow for precision. The core retrieval move.
from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")

def retrieve_and_rerank(query, store, wide_k=150, final_k=8):
    candidates = store.hybrid_search(query, k=wide_k)          # recall: cast wide
    pairs = [(query, c.text) for c in candidates]
    scores = reranker.predict(pairs)                            # precision: read together
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [c for c, _ in ranked[:final_k]]                     # precise top-8

R4 — adaptive routing is where the ~40% cost reduction comes from: don’t run the heavy path on every query. A cheap classifier sends each query to the minimum sufficient strategy (fixes F8):

# R4: route each query to the cheapest strategy that can answer it.
def route(query: str, classify) -> str:
    kind = classify(query)          # small/fast LLM or fine-tuned classifier
    return {
        "factoid":   "no_retrieval",   # model already knows it — skip retrieval
        "simple":    "single_hop",     # one hybrid lookup
        "compound":  "multi_hop",      # R5 iterative
        "relational":"graph",          # R6 GraphRAG — only here, not everywhere
        "numeric":   "sql",            # text-to-SQL over source tables
        "temporal":  "web_search",     # freshness path
    }[kind]

R7 — reorder the context is the only fix for F5, and it’s nearly free: after reranking, place the strongest chunks where the model actually attends — the head and tail — and the weaker ones in the middle:

# R7: defeat lost-in-the-middle. Best chunks go to the edges, not the center.
def reorder_for_attention(chunks):        # chunks arrive sorted best → worst
    head, tail = [], []
    for i, c in enumerate(chunks):
        (head if i % 2 == 0 else tail).append(c)
    return head + tail[::-1]               # strongest at both ends, weakest in middle

For the RRF fusion that powers R1 and R3, see the worked example in Part 2 — it’s the same algorithm, so it isn’t repeated here.