What Is Chunking in RAG?
Chunking splits documents into pieces a RAG system can search. Learn chunk size, overlap, the main strategies and how to test them, with code.
Short answer: chunking in RAG means splitting your documents into smaller pieces (chunks) before you embed and store them, so the system can retrieve just the few pieces that answer a question. Good chunks are small enough to be specific and large enough to make sense on their own. Chunking is one of the biggest factors in how well a RAG system answers.
Why Does RAG Need Chunking?
In retrieval-augmented generation, the model answers using text fetched from your documents. You cannot send a whole 200-page manual with every question: it may not fit in the context window, it is slow, and it buries the useful part among irrelevant text.
There is a second reason. An embedding turns a piece of text into one vector. If that text covers ten topics, the vector is a blurry average of all ten, and search finds it poorly. A chunk about one topic gives a sharp vector.
What Does the Chunking Step Look Like?
Load and clean the document (remove menus, page numbers, repeated headers).
Split it into chunks.
Create an embedding for each chunk and store it in a vector database together with the source name and position.
At question time, find the most similar chunks and pass them to the model.
How Big Should a Chunk Be?
There is no universal number, and anyone who gives one without testing is guessing. The trade-off is:
Small chunks | Large chunks | |
|---|---|---|
Search precision | High: very specific | Lower: mixed topics |
Context for the model | May lack surrounding detail | More complete |
Cost and speed | More chunks to store and search | More text sent to the model per question |
A practical start is a few hundred words (or a few hundred tokens) per chunk, then adjust by testing with real questions.
What Is Chunk Overlap?
Overlap means each chunk repeats the end of the previous one. If a sentence like "Refunds are allowed within 14 days, except for sale items" is cut in half at a chunk boundary, neither half answers the question. Overlap keeps boundary sentences together in at least one chunk. Keep it modest: too much overlap wastes storage and returns near-duplicate results.
What Are the Main Chunking Strategies?
Strategy | How it works | Best for |
|---|---|---|
Fixed-size | Cut every N characters or tokens, with overlap | Plain text, a quick first version |
By paragraph or sentence | Split at natural boundaries, then group up to a size limit | Articles, policies, notes |
By structure | Use headings, sections, list items, code blocks, table rows | Manuals, documentation, FAQs |
Semantic | Start a new chunk when the topic changes (found with embeddings) | Long, loosely structured text |
Start simple. Structure-aware splitting usually beats naive fixed-size cuts, and semantic chunking costs more to run, so use it only if testing shows a gain.
What Does Chunking Look Like in Code?
Two small functions, using only the standard library:
def chunk_text(text, size=200, overlap=40):
"""Fixed-size chunks; each repeats the last `overlap` characters of the one before."""
if overlap >= size:
raise ValueError("overlap must be smaller than size")
chunks, start = [], 0
while start < len(text):
chunks.append(text[start:start + size])
start += size - overlap
return chunks
def chunk_paragraphs(text, max_chars=300):
"""Group whole paragraphs together until a size limit is reached."""
chunks, current = [], ""
for para in text.split("\n\n"):
if current and len(current) + len(para) + 2 > max_chars:
chunks.append(current)
current = para
else:
current = f"{current}\n\n{para}" if current else para
if current:
chunks.append(current)
return chunks
doc = ("Shipping takes three days.\n\nExpress shipping takes one day.\n\n"
"Returns need a receipt.\n\nRefunds are paid within 14 days.")
for chunk in chunk_paragraphs(doc, max_chars=60):
print(repr(chunk))
The second function never cuts a paragraph in half, which keeps each idea whole. (A single paragraph longer than the limit still becomes one large chunk; real tools split it further.)
How Do You Know Your Chunking Works?
Do not judge it by eye. Write 20 to 50 real questions, note which passage should answer each, and check whether your retrieval returns that passage near the top. Change chunk size or strategy, test again, and keep what retrieves best. The post on evaluating AI and LLM systems explains this kind of testing, and the tutorial RAG ingestion and citation flow builds a working pipeline.
Common Mistakes
Cutting mid-sentence or splitting a table from its heading.
Losing context. Store the document title and section heading with each chunk.
Chunks that are too big, so retrieval brings back a lot of noise.
Never testing. The "right" settings depend on your documents and questions.
Key Takeaway
Chunking decides what your RAG system can find. Split along natural boundaries, add a little overlap, keep source details, and tune chunk size by testing real questions.
Ready to start? Build a working RAG pipeline step by step: ingestion, chunking, embeddings, retrieval and evaluation. Begin with the free course AI Engineering with LLMs: Beginner.
Related articles
More in AI Engineering
How to Get an LLM API Key (OpenAI and Gemini) and Understand the Cost
Where to create an OpenAI or Gemini API key, how token-based billing works, how to estimate cost and how to keep your key safe.
4 min readNov 11, 2026What Is Function Calling in LLMs?
Function calling lets an LLM ask your code to run a tool and use the result. Learn the step-by-step flow, a runnable example and the safety rules.
3 min readNov 7, 2026What Is LangChain? When to Use It (and When Not To)
LangChain is a framework for building LLM apps. Learn its core building blocks, a short sketch, and when plain code is the better choice.
4 min readNov 4, 2026How Do You Evaluate an AI or LLM System?
Building an AI system is not enough—you need to measure how accurately, reliably, and safely it performs. Learn the key metrics and methods used to evaluate modern AI and LLM applications.
2 min readSep 6, 2026
What Is AI Hallucination? Why It Happens and How to Reduce It
AI hallucination is when a model gives a confident answer that is wrong or made up. Here is why LLMs hallucinate, a real example, and six practical ways to reduce it.
3 min readSep 5, 2026RAG vs Fine-Tuning: What's the Difference and When Should You Use Each?
RAG gives a model the right knowledge at the moment it answers. Fine-tuning changes how the model behaves. Learn the difference, when to use each, and when to combine them.
3 min readSep 4, 2026
Comments
Sign in to join the discussion.



