Chunking

Retrieval systems do not fetch whole documents; they fetch passages. Chunking is how a document becomes passages, and the choice has outsized consequences.

Too small and a chunk loses the context that makes it meaningful — a paragraph referring to "this policy" retrieved without knowing which policy is worse than useless. Too large and it dilutes: a chunk covering five subjects matches weakly on all of them and may crowd out a precise answer.

Naive fixed-length splitting cuts through sentences, tables and code blocks indiscriminately. Better approaches respect structure — splitting on headings and paragraphs — overlap adjacent chunks so a boundary does not sever an idea, and prepend document and section titles to each chunk so it carries its own context.

When a retrieval system gives poor answers, chunking is one of the two most likely causes, and it's the less-examined one. Teams tend to reach for a bigger model when the actual problem is that the right passage was chopped in half.

Debug retrieval by looking at what was actually retrieved before touching anything else. If the passages are wrong or fragmentary, no model will save the answer. Respect document structure and add overlap before trying anything cleverer.