Chunk size matters because it changes what can be retrieved as one unit and how much context reaches the model. Studies report smaller chunks working well for concise facts and larger chunks helping when broader context is required, but the best setting depends on document structure, question type, embedding model, top-k and reranking.
Why size changes retrieval
A small segment can match a precise query without unrelated text diluting its embedding. It may also separate a definition from the qualification that makes it correct. A large segment keeps explanation together but consumes more context and can contain competing topics.
The useful unit is not always a token window. A table row, policy section, issue comment or API operation may be the natural boundary.
What research says
A 2025 multi-dataset study reported strong results for 64–128-token chunks on concise factual tasks and benefits from 512–1024 tokens when broader context was needed. Later work examines interactions with top-k, reinforcing that quality and cost move together. These are hypotheses, not universal production settings.
Evidence boundary
Do not copy a paper’s winning number without matching its datasets, retrieval stack and task. Reproduce the comparison on your corpus.
Run a controlled experiment
- Select real questions and expected evidence.
- Build indexes with small, medium and large chunks.
- Hold other settings fixed.
- Compare recall, precision, rank and coverage.
- Measure total context after top-k and expansion.
- Review by document and question type.
Improve more than size
Chunking cannot repair missing metadata, poor parsing or stale content. Add headings, titles and local context before indexing when a passage is ambiguous. Preserve parent identity so a small match can expand into its section.
Then re-run the retrieval evaluation loop; do not judge from two memorable examples.
Built for the next step
Keep retrieval changeable
SPYN3 lets clients query a stable workspace surface while indexing strategy evolves behind it.
Explore the data layerSPYN3 is AI-ready data infrastructure for technical solopreneurs and AI builders. It connects business data into a workspace-isolated knowledge layer and exposes it through a unified REST API and hosted MCP server, with permission-scoped access and source-aware returns. SPYN3 is currently in private beta; current capabilities and product direction are documented on the product facts page.
Sources
- Zhao et al.: Searching for Best Practices in RAG — task-dependent chunk-size findings. Verified 13 Sep 2026
- Li et al.: chunk-size and top-k trade-offs — retrieval quality and cost interactions. Verified 13 Sep 2026
- Anthropic: contextual retrieval — adding chunk-specific context before indexing. Verified 13 Sep 2026