Your own AI Overview: chunking

The split

The chunk you keep is the answer you can find

Drag the two knobs and watch the same underwriting note re-chop live. The question only gets answered when its evidence lands inside a single chunk.

• A chunk is the unit of retrieval. Search returns whole chunks, never half of one. Whatever the split leaves stranded, the system can’t hand back.
• Size is a trade-off. Too small and the answer spans several chunks, so no single passage carries the full fact. Too large and each chunk is mostly noise, which dilutes the match and burns the model’s context budget.
• Overlap is the safety net. Chunks share a few characters at their edges, so a fact sitting on a boundary still lands whole in at least one neighbour. It costs storage; it buys recall.
• This is decided once, upstream. Nothing downstream — reranking, a bigger model, a smarter prompt — can rejoin a fact the split already tore apart.

Two ways to cut

Cut by character count, or cut where the meaning changes

The baseline counts characters. The upgrade cuts on the document’s own structure — and that changes what each chunk is about.

• Naive = fixed-size window. Slide an 800-character window across the raw text with a little overlap. Simple, fast, and blind to meaning — a chunk can start mid-sentence and mix the roof detail with the premium reason.
• Structure-aware = cut on the document’s seams. Split on the record’s real boundaries: headings and paragraphs for a prose claim note, fields for a structured policy record. Each chunk becomes one coherent thing.
• Same evidence, cleaner packaging. Structure-aware chunks put “premium change + reason” in one passage and “roof / mitigation” in another, so the reason isn’t diluted by unrelated facts.
• It’s still the honest baseline first. The spine ships naive on purpose — it’s the measurable floor. Structure-aware is the upgrade you reach for when retrieval quality needs it, not a reflex.

Both strategies split the exact same document. The naive window is blind to meaning, so some chunks mix topics and dilute the match. Cutting on the document’s own structure gives you chunks that are each about one thing — easier to retrieve and easier to cite. The spine ships only the naive splitter today; structure-aware is the upgrade this post argues for, not shipped code.

What rides along

A chunk carries its document’s guardrails, not just its words

When you cut a document into chunks, each piece has to keep the labels that say who can see it and where it came from — or you’ve quietly broken security and citations.

• Provenance rides along. Every chunk keeps its source_id (e.g. claim/88431) so any answer built from it can cite the exact record it came from.
• Isolation rides along. Every chunk keeps its scope — the tenant + environment — so one insurer’s data and dev/prod never blur together at search time.
• Permissions ride along. Every chunk keeps its acl_roles, so a passage from an underwriter-only memo stays underwriter-only after it’s been split.
• Lose any of these and the split silently leaks. A chunk with no roles is a chunk the filter can’t gate; a chunk with no source_id is an answer you can’t trace.

That’s why the real splitter copies scope, doc_type, source_id, acl_roles and updated_at onto every chunk verbatim — it isn’t incidental, it’s what lets the retrieval filter run in the query and every answer cite its source. Chunking preserves the guardrails; it doesn’t just cut the text. That thread — permissions enforced at retrieval — is what Post 04 pulls on.

git clone https://github.com/xaviramirezcom/open-ai-overview
cd open-ai-overview
git checkout v2-chunking

# local, zero-cloud mode — no keys, no AWS. Split the sample note and print
# each chunk's char range plus which chunk holds the answer sentence.
uv run python -m own_overview.demo.chunking \
    --doc samples/pol-55012.txt --size 800 --overlap 100

_DEFAULT_SIZE = 800
_DEFAULT_OVERLAP = 100


class NaiveChunker:
    """Fixed-size character window with overlap. Implements the Chunker Protocol."""

    def __init__(self, chunk_size: int = _DEFAULT_SIZE, overlap: int = _DEFAULT_OVERLAP) -> None:
        if chunk_size <= 0:
            raise ValueError("chunk_size must be positive")
        if not 0 <= overlap < chunk_size:
            raise ValueError("overlap must be in [0, chunk_size)")
        self.chunk_size = chunk_size
        self.overlap = overlap

    def split(self, doc: Document) -> list[Chunk]:
        text = doc.text or ""
        if not text.strip():
            return []

        step = self.chunk_size - self.overlap
        chunks: list[Chunk] = []
        start = 0
        idx = 0
        while start < len(text):
            piece = text[start : start + self.chunk_size]
            chunks.append(
                Chunk(
                    chunk_id=f"{doc.doc_id}#{idx}",
                    doc_id=doc.doc_id,
                    # Isolation + ACL metadata copied verbatim so the retrieval
                    # filter and citations keep working on the chunk.
                    scope=doc.scope,
                    doc_type=doc.doc_type,
                    source_id=doc.source_id,
                    text=piece,
                    acl_roles=doc.acl_roles,
                    updated_at=doc.updated_at,
                    metadata=dict(doc.metadata),
                )
            )
            idx += 1
            start += step
        return chunks

@runtime_checkable
class Chunker(Protocol):
    """Splits a Document into retrievable Chunks. The naive splitter ships in
    the spine; structure-aware splitting is its own deep-dive post."""

    def split(self, doc: Document) -> list[Chunk]: ...

Explain it back

A teammate says: “Chunking is just preprocessing — we’ll tune it later once retrieval and the model are good.” In one or two sentences, why is that backwards?