LangChain Chunk Size: Configure Splitters for RAG

LangChain chunk size for RAG starts with the splitter’s unit: chunk_size=1000 can mean characters or tokens depending on the constructor and length function. This guide covers RecursiveCharacterTextSplitter, token-aware constructors, overlap and source offsets, then shows how to compare configurations at your retriever’s k.

For the framework-independent choice of document boundaries and overlap, start with RAG chunking strategy: picking chunk size and overlap. Use the LangChain settings below to implement that choice.

What chunk_size actually measures

LangChain’s knowledge-base tutorial uses RecursiveCharacterTextSplitter with chunk_size=1000, chunk_overlap=200 and add_start_index=True, and calls it the recommended splitter for generic text (LangChain docs). It tries paragraph breaks, then lines, spaces and single characters, recursing only while a piece is over the limit.

chunk_size is whatever length_function returns, and the default is len, so the unit is characters; pass nothing and the base class emits 4000-character chunks with 200 of overlap (reference). Embedding models count tokens, then truncate rather than fail. OpenAI’s text-embedding-3 models accept 8192 tokens (OpenAI docs); all-MiniLM-L6-v2 truncates past 256 word pieces by default (model card), so most of a 4000-character chunk never enters the index.

So measure in the embedding model’s tokens. RecursiveCharacterTextSplitter.from_tiktoken_encoder swaps in a tiktoken length function; pass encoding_name or model_name, since its default encoding is gpt2 (reference). The token-splitting guide distinguishes CharacterTextSplitter.from_tiktoken_encoder, whose chunks may exceed the token target, from RecursiveCharacterTextSplitter.from_tiktoken_encoder, which recursively splits oversized pieces to enforce it. TokenTextSplitter also enforces a token limit, but can divide the tokens of one Unicode character between chunks. The stages around the splitter are in LangChain RAG pipeline setup.

LangChain splitter parameters to record

The TextSplitter reference defines these settings. Store the chosen values with the index configuration so the retrieval results can be traced to the splitter that produced them.

Parameter	Meaning	Configuration check
chunk_size	Maximum size measured by the length function	Record whether the unit is characters or tokens
chunk_overlap	Target overlap in the same unit	Keep it no larger than chunk_size
length_function	Function used to measure text	Default len counts characters
add_start_index	Adds a source character offset to metadata	Check offsets against the loaded document text

Changing the tokenizer changes the meaning of a token-sized chunk even if the numeric chunk_size stays the same. For sentence-transformer models, the documented SentenceTransformersTokenTextSplitter provides model-specific token counting through model_name and tokens_per_chunk.

Compare chunk_size settings at your retriever’s k

Document hit rate asks whether any chunk from the right source landed in the top k. It hides dilution: a chunk that is mostly boilerplate with one relevant sentence counts as a hit. The Lost in the Middle study found that answer accuracy can fall when relevant information sits in the middle of a long context.

Use token-level recall instead. Annotate each golden question with the exact spans that answer it. Recall is the share of those tokens present in the union of retrieved chunks, each counted once; precision is the share of retrieved tokens that were relevant. Chroma, a vector database vendor, defined it this way and benchmarked splitters with text-embedding-3-large (Chroma Research). RecursiveCharacterTextSplitter at 200 tokens with no overlap scored 88.1 recall and 7.0 precision, 400 tokens scored 89.5 and 3.6, and 800 tokens with 400 overlap fell to 85.4 and 1.5. Doubling the chunk bought about one point of recall for half the precision, and heavy overlap lost recall outright.

Published benchmarks use different corpora, embedding models and retrieval settings. The multi-dataset chunk-size analysis reports that the preferred size varies with the task and embedding model. Treat those results as reasons to evaluate your LangChain configuration, rather than a universal value for chunk_size.

Wiring it up

The illustrative sweep rebuilds the index at three sizes and reports answer-span coverage at the k you serve, plus p95 context tokens per query. Golden spans are character offsets into each loaded document’s page_content, the same coordinates add_start_index writes to metadata["start_index"]; for paged loaders key them on source plus page, since offsets restart per page. The coverage function below counts characters, not token-level recall. To reproduce a token-level metric, map the annotated and retrieved spans through the evaluation tokenizer first. Supply load_documents() and load_golden() for your corpus, and check that every recorded offset points to the expected text before scoring.

import tiktoken
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter

enc = tiktoken.get_encoding("cl100k_base")
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
docs = load_documents()
golden = load_golden("golden.jsonl")


def span_recall(hits, source, spans):
    covered = set()
    for d in hits:
        if d.metadata["source"] == source:
            start = d.metadata["start_index"]
            covered.update(range(start, start + len(d.page_content)))
    wanted = {i for s, e in spans for i in range(s, e)}
    return len(wanted & covered) / len(wanted) if wanted else 0.0


for chunk_tokens in (256, 512, 1024):
    splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
        encoding_name="cl100k_base",
        chunk_size=chunk_tokens,
        chunk_overlap=chunk_tokens // 8,
        add_start_index=True,
    )
    chunks = splitter.split_documents(docs)
    store = InMemoryVectorStore.from_documents(chunks, embeddings)
    recalls, ctx = [], []
    for question, source, spans in golden:
        hits = store.similarity_search(question, k=5)
        recalls.append(span_recall(hits, source, spans))
        ctx.append(sum(len(enc.encode(d.page_content)) for d in hits))
    ctx.sort()
    print(chunk_tokens, len(chunks), round(sum(recalls) / len(recalls), 3), ctx[int(0.95 * (len(ctx) - 1))])

Run it against the production store, not InMemoryVectorStore, before trusting the numbers; HNSW indexes add their own recall loss.

What you’ll see

A healthy sweep has recall rising from the smallest size, then going flat, while p95 context tokens keep climbing. Pick the smallest size on the plateau: precision only worsens to the right of it, and prompt cost scales with chunk size times k.

Bad shapes:

• Recall falls as chunks grow. The embedding model is truncating. Count in its tokens.
• Recall is low and flat at every size. Chunking is not the problem; look at vocabulary mismatch, metadata filters, or the embedding model. Contextual retrieval is the next lever: Anthropic’s vendor benchmark reports that prepending 50 to 100 tokens of document context per chunk cut top-20 retrieval failures by 35 percent, and by 67 percent with BM25 and reranking added (Anthropic).
• Recall is fine but answers are wrong. The generator is losing the span; lower k or move the strongest chunk out of the middle.
• The same passage appears twice in the top k. Overlap is too large and effective k shrinks.

Caveats

• Label leakage. Annotate spans on the raw text before splitting; spans written from chunks produced at one size inherit those boundaries, and that size wins by construction.
• Sampling cost. Every sweep point re-embeds the sampled corpus. Keep every golden source in the sample, and keep the golden set large: Chroma reported standard deviations around 30 points on recall, which swamps a one-point difference.
• A chunk size change is a re-index. Build a new collection, sweep against it, then swap the retriever. Stamp splitter settings into chunk metadata and run the sweep as a CI gate, the way LLM testing separates offline evals from production monitoring.
• Bigger chunks carry more of a poisoned document into the prompt per hit; indirect prompt injection in RAG covers the chunking and sanitisation defences.
• Overlap larger than chunk size is a hard error. The splitter raises ValueError instead of clamping, so a config that scales one setting and not the other fails at index time.