Part 2 — What Is Multimodal Hybrid Agentic RAG?

The name carries four independent concepts, each addressing a distinct failure mode from Part 1. They’re separable — you can adopt them one at a time. The fastest way to understand each is to ask which single query it was invented to answer:

Which word solves your problem?

• · Your corpus is scans, slides, and audio → you need Multimodal.
• · Users search by exact code and by concept → you need Hybrid.
• · Your system fabricates answers instead of saying “I don’t know” → you need Agentic.
• · Answers have no traceable source → you need grounded RAG.

2.1 Multimodal — “What did the CFO say on the Q3 earnings call?”

That answer lives in an audio file, not a text document. A text-only pipeline never finds it. An enterprise knowledge base is a heterogeneous corpus, and each format needs its own faithful parser:

A multimodal system does not convert everything to text and pretend it’s all the same. It dispatches each format to the right parser, preserves structure (tables as grids, not strings), links related content across modalities, and embeds each element appropriately. In practice that’s a router:

# Multimodal ingestion = dispatch each file to a modality-specific parser.
PARSERS = {
    "pdf_scanned":  ocr_parse,        # OCR — Tesseract / Textract / Azure DI
    "pdf_digital":  layout_parse,     # layout-aware — Unstructured / LlamaParse
    "pptx":         slide_parse,      # shapes + speaker notes
    "audio":        transcribe,       # Whisper → timestamped text
    "code":         ast_parse,        # tree-sitter, symbol-aware
    "xlsx":         table_parse,      # preserve the grid, not flatten it
}

def ingest(path: str, modality: str):
    elements = PARSERS[modality](path)          # right tool per format
    return [enrich(e, source=path) for e in elements]  # keep structure + provenance

Most common mistake: running every file through the same PDF text extractor. The scanned contract comes back empty; the slide comes back as garbled shape coordinates.

2.2 Hybrid — “Find the clause about ISO 27001 Clause 9.4.2.”

Hybrid refers specifically to retrieval— combining complementary signals fused at query time.

Dense retrieval(embedding similarity) excels at semantic generalisation. A question about “declining quarterly performance” retrieves “revenue dropped,” “earnings shortfall,” “profit contraction.” It handles paraphrase and concept-level matching — but it will happily miss “Clause 9.4.2,” mapping it into the same region as dozens of unrelated clause references.

Sparse retrieval(BM25 / keyword) excels at exact-term recall. “ISO 27001 Clause 9.4.2” and “part number XR-7741” won’t be paraphrased — the document either contains those tokens or it doesn’t. BM25 finds them; dense retrieval can’t.

Hybrid retrievalruns both channels and fuses the ranked lists with Reciprocal Rank Fusion (RRF) — a parameter-free combination that empirically beats either channel alone on almost every benchmark, often lifting context recall by 15–25%. The formula is just a sum of reciprocal ranks:

# RRF: fuse ranked lists. Each doc scores 1/(k + rank) summed across channels.
def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> list[str]:
    scores: dict[str, float] = {}
    for ranked in rankings:                     # e.g. [dense_ranked, bm25_ranked]
        for rank, doc_id in enumerate(ranked, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

Why it works, concretely. Say doc D ranks 1st in BM25 but 8th in dense, and doc E ranks 3rd in both (k = 60):

D:  1/(60+1) + 1/(60+8)  = 0.01639 + 0.01471 = 0.03110   ← wins
E:  1/(60+3) + 1/(60+3)  = 0.01587 + 0.01587 = 0.03174

A document that any single channel ranks highly floats to the top even if the other channel ranks it poorly — so the exact-match hit and the semantic hit both survive fusion.

After fusion comes reranking: a cross-encoder reads the full query and each candidate together — expensive but far more accurate — and re-orders the top candidates from coarse similarity to fine relevance. This two-stage pattern (retrieve wide, rerank precisely) recovers most of the precision that fast vector search sacrifices.

Most common mistake: normalising and adding raw dense/BM25 scores. Their scales are incomparable; fuse ranks, not scores — that’s the whole point of RRF.

2.3 Agentic — “Does our policy cover this, and if the docs don’t say, admit it.”

A pipeline runs a fixed sequence regardless of the query. An agent observes state, decides the next action, executes, observes the result, and decides again. The critical insight: retrieval quality isn’t knowable before retrieving, and answer quality isn’t knowable before generating. A pipeline can’t self-correct; an agent can. Agentic RAG adds three self-correction gates:

# Agentic RAG = three decision gates around the naive retrieve-then-generate core.
def agentic_answer(query: str) -> str:
    strategy = route(query)             # GATE 1 (pre-retrieval): vector? graph? SQL? web?
    docs = retrieve(query, strategy)

    if not is_relevant(docs, query):    # GATE 2 (post-retrieval): grade the context
        query = rewrite(query)          #   → transform & retry, or fall back
        docs = retrieve(query, strategy)

    answer = generate(query, docs)

    if not is_grounded(answer, docs):   # GATE 3 (post-generation): faithfulness check
        return regenerate_or_abstain(query, docs)  # → fix, or honestly say "I don't know"
    return answer

Each gate costs extra LLM calls, so a calibrated system applies them selectively — not every query needs the full loop, and deciding that is itself an agentic call. The three gates map cleanly onto the three failure points: wrong strategy, irrelevant context, ungrounded output.

Most common mistake: running all three gates on every query. Cost and latency explode; route cheap factoids straight through and reserve the loop for hard or high-stakes queries.

2.4 RAG — “…and cite where you got that.”

The foundational pattern: retrieve context, supply it to the model in the prompt, and generate an answer grounded in that context with citations. The LLM is a reasoning engine over supplied evidence, not a source of facts. Grounding is enforced in the prompt contract:

GROUNDED_PROMPT = """Answer ONLY from the context below.
Cite each claim as [doc_id]. If the context does not contain the
answer, reply exactly: "Not found in the provided sources."

Context:
{context}

Question: {question}"""

The instruction to abstain is what turns a fluent guesser into a trustworthy source — it’s the difference between a demo and a system a compliance team will sign off on.

The combination — multimodal ingestion, hybrid retrieval, agentic orchestration, and grounded generation — is not an incremental improvement over first-generation RAG. It’s a different class of system: aware of its own uncertainty, structured to recover from its own failures, and designed to be more accurate in production than it was on day one.