Chunking is the process of breaking documents into fragments that a RAG system indexes and retrieves in response to a user’s question. A practical setup is to first convert documents into structured text, then split by headings and paragraphs with a token limit (a starting point is 512–1,024 tokens with 10–25% overlap), add the document and section title to each fragment, and then compare 2–3 variants on a set of real questions. There is no universal size: in Chroma and NVIDIA research, the best option varied from corpus to corpus.
Prepared by AIrassvet October 22, 2026 for IT leaders, analysts, and developers launching an enterprise assistant or knowledge-base search who want it to find the right policy clause, not the neighboring one. This guide is based on research from Chroma, NVIDIA, and Anthropic, as well as Microsoft and LangChain documentation. We did not run our own measurements for this article: the numbers belong to the original source authors.
Contents
- Quick Take
- What This Guide Covers and Excludes
- Why Chunking Determines Answer Quality
- What You Need Before You Start
- Step-by-Step Chunking Setup
- Special Document Types
- Common Mistakes
- Reusable Configuration and Test Protocol Template
- FAQ
- Sources and Method
Quick Take
- Split by structure, not by character count. Start with headings and bullets, then paragraphs, then sentences. That is how recursive splitting works, which LangChain recommends for plain text.
- Start from a proven baseline. Microsoft recommends starting with 512 tokens and 25% overlap; in NVIDIA tests, page-based splitting performed best on average, while factual questions were best served by 256–512-token chunks.
- Add context to every chunk. Document title, section path, date, and access rights. Anthropic reports that a short context note before the chunk reduced failed retrievals by 35%, and by 49% when combined with BM25.
- Test against your own questions. You need a set of several dozen real questions with a known answer location and a metric for whether the needed chunk appears in the top k results.
- Lock down the configuration. Chunk size, overlap, separators, and parser version should be stored with the index so reindexing produces the same result.
What This Guide Covers and Excludes
This guide covers: preparing text for splitting, choosing a strategy, size, and overlap, chunk metadata and context, handling tables and contracts, quality checks, and common mistakes.
This guide does not cover: OCR for scans or choosing a PDF parser (we covered those in the comparison of PDF parsers for RAG), choosing a vector database and embedding model, reranking, or response generation. See the overall architecture in “RAG System for Business”.
Why Chunking Determines Answer Quality
A RAG system does not read documents in full at question time. It first splits them into chunks, turns each chunk into a vector (embedding), and at query time retrieves the most similar chunks. The language model answers only from what was retrieved. If a rule and its exception end up in different chunks, the model sees one without the other.
That creates two opposite risks:
- A chunk that is too large mixes multiple topics. Its vector averages the entire contents, retrieval becomes worse at distinguishing what matters, and the model window gets extra text.
- A chunk that is too small loses context: the line “term — 10 business days” without the heading “Return of Goods of Proper Quality” is useless.
The impact is measurable. In Chroma’s technical report “Evaluating Chunking Strategies for Retrieval” from July 3, 2024, the choice of strategy changed retrieval recall by up to 9%. In a NVIDIA study from June 18, 2025, extreme sizes — 128 and 2,048 tokens — generally underperformed middle-sized chunks. We wrote about how retrieval errors turn into hallucinations in our article on RAG accuracy.
What You Need Before You Start
| What to Prepare | Why It Matters | Who Usually Owns It |
|---|---|---|
| A sample of documents of each type (policies, contracts, instructions, tables, FAQs) | The strategy depends on the document structure | Knowledge base owner |
| 30–50 real user questions with the document and section where the answer is located | This is the benchmark for comparing options | Department subject matter experts |
| A parser that preserves headings, lists, and tables (for example, in Markdown) | Without structure, there is nothing meaningful to split on | Developer |
| The embedding model and its tokenizer, plus the maximum input length | Chunk size is measured in this model’s tokens | Developer |
| A vector store with metadata and filter support | For access rights, dates, and versions | Developer or administrator |
| An evaluation script: search for each question and check whether the needed chunk appears in the results | So you can compare options with data, not by eye | Developer or analyst |
The number of questions is a guideline, not a rule: the set should cover every document type and the wording employees actually use.
Step-by-Step Chunking Setup
Step 1. Split documents by type
A policy with numbered clauses, a contract, a pricing table, and a Q&A knowledge base all require different rules. Make a list of document types and define the natural unit of meaning for each: a policy clause, a contract article, a table row, or a question-answer pair.
Step 2. Convert documents into structured text
Before splitting, the document should become text in which headings are marked as headings, lists as lists, and tables retain their rows and columns. Markdown is a convenient format. Manually check 5–10 documents of each type: if the parser merged columns or lost clause numbering, no chunking setup will fix that.
Step 3. Choose a baseline strategy
For most enterprise texts, start with structural recursive splitting. First, the document is split by headings; then any section that does not fit the limit is split by paragraphs; then by sentences. That is how RecursiveCharacterTextSplitter works in LangChain: by default, it tries the separators "\n\n", "\n", a space, and an empty string to keep paragraphs and sentences together for as long as possible. In the Chroma report, this heuristic approach performed well with the right settings compared with more complex methods.
| Strategy | How it splits | When it works best | Cost |
|---|---|---|---|
| Fixed-size with overlap | Every N tokens | Uniform text without structure | Low |
| Recursive by structure | Headings → paragraphs → sentences | Policies, instructions, articles | Low |
| By page | One page = one chunk | Reports and presentations where a page is a meaningful block | Low |
| Semantic | By topic shift between sentences, using embeddings | Long texts without headings | Medium: requires embedding model calls |
| With an LLM | The model decides where the boundary is | Complex documents, small corpus | High and slow |
In Microsoft's guide this choice also adds economics: every strategy has a development cost and a per-document processing cost, both of which grow with the corpus.
Step 4. Set chunk size and overlap in tokens
Measure size in the tokens of the embedding model you use, not in characters: for Russian text, the character-to-token ratio differs from English and depends on the tokenizer. For more details, see the article on Russian text tokenization. Make sure the chunk plus its contextual heading does not exceed the model's maximum input length, or the end will be truncated.
Starting values from source materials:
| Source | Recommendation or result |
|---|---|
| Microsoft, Azure AI Search | Start with 512 tokens and 25% overlap, meaning 128 tokens; structured data may require less overlap |
| NVIDIA, 2025 | Page-based splitting produced the best average accuracy at 0.648 and the lowest variance; factual questions worked better with 256–512-token chunks, analytical ones with larger chunks; 15% overlap was best among 10%, 15%, and 20% on one dataset |
| Chroma, 2024 | A setting of 800 tokens with 400-token overlap gave slightly lower average recall; recursive splitting into 400-token chunks with no overlap achieved 89.5% recall versus 85.4% for the 800/400 option with five retrieved chunks |
A practical plan: take two or three configurations around 512 tokens (for example, 256, 512, and 1,024 with about 15% overlap) and compare them in step 7. Overlap helps when a fact falls on the boundary between chunks, but too much overlap creates nearly identical chunks that take up space in the results.
Step 5. Add metadata and chunk context
Store each chunk together with these fields: document ID and title, section path ("Returns Policy → 3. Timeframes → 3.2"), effective date or version, department, access level, and a link to the original. Metadata is needed for filters (search only in current versions, only in documents this employee can access) and for citations in the answer.
The second technique is contextual information inside the text itself. In its publication Contextual Retrieval from September 19, 2024, Anthropic describes a method in which a language model writes 50–100 tokens for each chunk about its place in the document, and that text is added before the chunk before building the embedding and the BM25 index. In their experiments, the share of failed searches in the top 20 results dropped from 5.7% to 3.7% with contextual embeddings alone, to 2.9% with contextual BM25 as well, and to 1.9% when reranking was added. The authors estimated the cost of generating the notes at $1.02 per million document tokens under their assumptions and with prompt caching.
If your budget for LLM calls is limited, start with the free option: add the document title and section path to the beginning of the chunk.
Step 6. Set exceptions for special document types
Tables, contracts, FAQs, and code do not handle the general rule well. Specific techniques are described in the section "Special document types" below. Set them up as separate rules in the configuration, not as manual edits to individual files.
Step 7. Build a test set and compare options
For each configuration, build a separate index and run all questions from the reference set against it. The main metric is recall@k: for what share of questions the chunk containing the answer appeared among the first k retrieved results (k equals the number of chunks you pass to the model, for example 5). Also look at:
- how many retrieved chunks are not relevant to the question — this is noise in the model's context window;
- how many chunks and tokens are in the index overall — this is the storage and embedding cost;
- the quality of the final answers on 10–20 questions, evaluated by an expert.
Change one setting at a time and keep the embedding model, the value of k, and the question set unchanged, or there will be nothing to tie the difference to.
Step 8. Review failures manually and freeze the version
For each question where the needed chunk was not found, open the results and determine why: the answer was split across chunks, buried in a large chunk, lost during parsing, or the question was phrased in words that do not appear in the document. Chunking cannot fix the last case — hybrid search and reranking help.
Save the chosen configuration in the repository together with the parser version and the embedding model version. If either changes, re-index the corpus and repeat the test.
Special document types
- Tables. Do not split a table in the middle without repeating the header: repeat the header row in each chunk, or turn each table row into a separate chunk like "Item: ...; Price: ...; Delivery time: ...".
- Contracts and policies. The unit is a numbered clause or article. The number and section title must be in the chunk text, otherwise a question like "what does section 5.3 say" will not find the answer.
- FAQs and Q&A bases. One question-answer pair = one chunk, with no overlap with neighboring pairs.
- Long documents without headings (transcripts, correspondence). Here, semantic splitting or late chunking is appropriate (late chunking): the long text is first passed through a long-context embedding model as a whole, and the chunk vectors are then obtained after that, preserving the context of the entire document in them.
- Scans and presentations. If a page is a complete semantic unit, start by splitting by page, as in NVIDIA tests.
Common mistakes
- Cutting raw text straight out of a PDF. Headers and footers, page numbers, and merged columns end up in fragments and hurt search quality.
- Measuring size in characters for a model that works with tokens. A fragment can end up longer than the model input and be truncated without warning.
- Setting a large overlap “for reliability.” Half-overlap doubles the index size and fills the results with nearly identical fragments.
- Losing headings. A fragment without the document title and section cannot be found by questions containing those words and does not provide a correct citation in the answer.
- Choosing the size based on someone else’s benchmark. In NVIDIA’s study, even similar financial documents worked better with different settings: 1,024 tokens, 512 tokens, or whole pages.
- Test with three questions. The difference between options is visible only on a set that covers all document types.
- Not saving the configuration. Six months later, no one remembers how the index was built, and reindexing changes the quality.
Reusable configuration template and test protocol
Chunking configuration (store next to the index):
chunking_config:
version: "2026-10-22-v1"
parser: "parser name and version"
embedding_model: "embedding model name"
length_unit: "tokens" # embedding model tokens
default_strategy: "recursive_by_structure"
separators: ["# ", "## ", "### ", "\n\n", "\n", ". "]
chunk_size: 512
chunk_overlap: 77 # about 15%
context_header: "{doc_title} > {section_path}"
metadata: [doc_id, doc_title, section_path, version_date, department, access_level, source_url]
rules:
tables: "table row + header row"
contracts: "one contract clause with a number"
faq: "one question-answer pair, no overlap"
slides: "one page"Comparison protocol for options:
Question set: ___ items, document types: ___
Embedding model: ___ k (fragments in the answer): ___
| Option | Size | Overlap | Fragments in index | recall@k | Noise in top-k | Comment |
|--------|------|---------|--------------------|----------|----------------|---------|
| A | | | | | | |
| B | | | | | | |
| C | | | | | | |
Questions without the needed fragment in top-k: ___ (reason for each)
Decision and date: ___ Owner: ___Frequently asked questions
What chunk size should I choose for RAG?
Start with 512 embedding-model tokens and 10–25% overlap — this is the starting point recommended by Microsoft and tested by NVIDIA. Then compare 2–3 sizes, such as 256, 512, and 1,024, using your own questions. For short factual questions, smaller fragments often win; for analytical questions, larger ones do.
Do chunks need overlap?
A small overlap is useful when splitting with fixed windows: it protects facts that fall on a boundary. With structural splitting by sections and for FAQ, it is often unnecessary. In Chroma’s report, recursive splitting at 400 tokens with no overlap showed higher recall than 800 tokens with 400-token overlap.
How is semantic chunking better than regular chunking?
Semantic chunking looks for boundaries where the topic changes, so it is useful for long texts without headings. But it requires embedding-model calls during indexing and does not always win: in Chroma’s report, simple recursive splitting with well-chosen parameters performed well compared with more complex methods. For documents with headings, start with structural splitting.
How can you tell chunking is configured poorly?
Signs include the assistant answering about a neighboring clause, missing exceptions to rules, being unable to name the source, or finding several nearly identical fragments. Check this by taking questions that were answered incorrectly and looking at which fragments were retrieved. If the needed text was split apart or did not appear in the results, the problem is with chunking or retrieval.
Primary sources and methodology
- Smith, Troynikov. Evaluating Chunking Strategies for Retrieval, Chroma Technical Report, July 3, 2024 — a comparison of strategies, recall, and search precision.
- Steve Han. Finding the Best Chunking Strategy for Accurate AI Responses, NVIDIA Technical Blog, June 18, 2025 — sizes from 128 to 2,048 tokens, page-based splitting, overlap.
- Anthropic. Introducing Contextual Retrieval, September 19, 2024 — contextual embeddings and BM25.
- Microsoft Learn. Chunk large documents for vector search solutions and RAG chunking phase — starting parameters and the economics of chunking.
- LangChain. Recursive text splitter — default separators.
- Günther and coauthors. Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models, arXiv:2409.04701.
We checked the sources on October 22, 2026. The figures were obtained by the authors on their own corpora and models, mostly in English; for Russian-language company documents, they should be rechecked using the protocol from step 7.
How AIrassvet helps tune chunking for an enterprise knowledge base
Chunking setup is part of building a RAG system, and the result depends on a company’s specific documents. On the AIrassvet website you can find the areas that cover this work:
- RAG and enterprise knowledge bases: analysis of document types, splitting rules, metadata, and access permissions;
- data preparation: converting policies, contracts, and tables into structured text and building a gold-standard question set;
- testing and launch: comparing configurations on your questions and recording the selected index version.
The first step is to choose one process where employees look for answers in documents and document the current result, data sources, constraints, and acceptance criteria.
Bottom line: tune chunking based on testing, not someone else’s number
Good chunking preserves the document’s semantic units: a policy clause, a contract article, a table row with headers. Start with structural recursive splitting at around 512 tokens with a small overlap, and add the document title and section to each fragment.
Then the test decides: gather 30–50 real questions, compare two or three configurations by retrieval recall, analyze failures, and lock the version. That way you get a setup tailored to your documents, not someone else’s benchmark.
