Your own AI Overview: Retrieve & rerank

The two-stage funnel

Retrieve wide, rerank sharp

One search can’t be both fast over millions of chunks and precise about which one answers the question — so we do two passes.

• Retrieve casts a wide net. Vector search compares the question’s embedding to every chunk’s and returns the top k “roughly similar” ones — cheap, and tuned for recall (don’t miss the answer), not for getting the order right. Default k = 12.
• Rerank decides the answer. A reranker reads the question and each candidate together and scores true relevance, then keeps the top rerank_k (default 5). That smaller, sharper set is the context budget the grounding step gets.
• The order flips. The chunk vector search ranked #9 can be the one the reranker puts #1 — because “similar embeddings” and “actually answers this” are not the same thing.
• It fails soft. If the reranker errors, the pipeline degrades to retrieval order (candidates[:k]) instead of breaking — rerank is a quality boost, not a correctness gate.

That funnel isn’t a diagram of the code — it is the code. The retrieve node embeds each sub-query, searches with a permission filter, and merges the hits, keeping each chunk’s best score:

def run(state, settings, *, embedder: Embedder, store: VectorStore) -> dict:
    identity = state["identity"]
    # The filter is derived only from the signed identity — never the prompt.
    # Fail-closed: no roles => no results. (That RBAC deep-dive is Post 4.)
    flt = build_filter(identity)

    queries = state.get("sub_queries") or [state.get("question", "")]
    queries = [q for q in queries if q and q.strip()]

    best: dict[str, Retrieved] = {}
    for q in queries:
        vec = embedder.embed_query(q)
        for hit in store.search(vec, flt, k=settings.retrieve_k):   # k = 12
            cid = hit.chunk.chunk_id
            existing = best.get(cid)
            # Keep the strongest score seen for a chunk across sub-queries.
            if existing is None or hit.score > existing.score:
                best[cid] = hit

    candidates = sorted(best.values(), key=lambda r: r.score, reverse=True)
    return {"candidates": candidates, ...}

The rerank node is deliberately tiny — reorder by true relevance, trim to the budget, hand the sharper set on to grounding:

def run(state, settings, *, reranker: Reranker) -> dict:
    question = state.get("question", "")
    candidates = state.get("candidates", [])
    # Reorder by true relevance, then trim to the context budget.
    reranked = reranker.rerank(question, candidates, k=settings.rerank_k)  # k = 5
    return {"reranked": reranked, ...}

Where vectors fumble

When meaning isn’t enough

Embeddings are great at “what is this about” and surprisingly bad at “does it contain exactly POL-55012”.

• Dense search matches meaning. It embeds the query and finds chunks with similar meaning — perfect for “why did the premium rise”, weak when the answer hinges on an exact string the embedding blurs.
• Identifiers are exact tokens. POL-55012, claim 88431, a form code, a state abbreviation — a near-miss (POL-55021) is a wrong answer, not a close one. Vectors don’t guarantee the exact token is even present.
• Keyword search nails exact tokens. Classic term matching (BM25) finds the chunk that literally contains POL-55012, even when its surrounding prose isn’t semantically “similar” to the question.
• Hybrid = both, then merge. Run dense and keyword, combine the scores, hand the union to the reranker. You get semantic recall and exact-match precision.

Ask a librarian who only understands topics to find the file numbered 55012 and they’ll hand you five files about the same policy. Add a clerk who reads numbers and the exact file comes first. Production RAG usually wants both.

The reranker is the one place this stage can call an outside service, so it’s built to fail soft — a service error degrades to plain retrieval order rather than breaking the answer. Local mode uses a no-op passthrough, so the graph runs with zero cloud:

class BedrockReranker:
    """Reranker backed by a managed Rerank API, with a graceful fallback."""

    def rerank(self, query, candidates, k):
        candidates = list(candidates)
        if not candidates:
            return []
        try:
            return self._rerank_via_service(query, candidates, k)
        except Exception as exc:            # rerank is best-effort, never fatal
            logger.warning("Rerank failed (%s); falling back to retrieval order.", exc)
            return candidates[:k]           # degrade to dense order, don't break


class NoopReranker:
    """Local / reranker=none passthrough — keep retrieval order, trim to k."""

    def rerank(self, query, candidates, k):
        return list(candidates[:k])

git clone https://github.com/xaviramirezcom/open-ai-overview
cd open-ai-overview
git checkout v3-retrieval

uv sync --extra local
cp .env.example .env         # set the *_PROVIDER / VECTOR_STORE knobs to "local"

own-overview seed            # synthetic P&C insurance data + ingest
own-overview query "Why did the premium on POL-55012 go up?" --role adjuster

Explain it back

Your vector search returns 12 chunks and the one that actually answers the question is sitting at rank #9. You send the top 5 to the model. Two things are now true — name the problem, and name the fix.