LangChain RAG Pipeline: Setup to First Answer

Retrieval augmented generation is the most common reason people install LangChain, and it is also where most first attempts quietly go wrong. The framework hides the plumbing well enough that a pipeline can be assembled in an afternoon and still return confident nonsense, because almost every decision that governs answer quality is made before the model is ever called.

This is a walk through the five stages of a working pipeline, what each stage decides, and the specific settings worth choosing deliberately instead of accepting the default.

The Five Stages

Every RAG pipeline, in LangChain or anywhere else, is the same shape:

1. Load documents from wherever they live into a common representation.
2. Split them into chunks small enough to embed and to fit in a prompt.
3. Embed each chunk into a vector.
4. Store those vectors in an index that supports similarity search.
5. Retrieve the chunks relevant to a query and assemble them into a prompt.

Stages one through four run offline, usually as a batch job. Stage five runs on every request. The offline half is where quality is determined, and it is the half people rush.

Stage 1: Loading and Metadata

A loader turns a source file into a document object with page content and a metadata dictionary. The content is what gets embedded. The metadata is what makes the answer verifiable.

Carry source identity on every document from the moment it is created: the file path or URL, a title, a section heading if the format has one, and a version or modification date. LangChain propagates metadata through splitting, so whatever is attached to the parent document ends up on each chunk derived from it.

In LangChain a document is just page content plus a metadata dictionary, so the provenance fields are attached at load time:

from langchain_core.documents import Document

docs = [
    Document(
        page_content=text,
        metadata={
            "source": path,          # file path or URL, for the citation
            "title": title,
            "section": heading,      # if the format has one
            "revision": modified_at, # lets stale chunks be invalidated
        },
    )
    for path, title, heading, text, modified_at in corpus
]

The reason this matters is not tidiness. Without per-chunk provenance there is no way to render a citation, no way to filter retrieval to a subset of the corpus, and no way to invalidate stale content when a source document changes. Retrofitting metadata later means reindexing the whole corpus.

Stage 2: Chunking Is the Highest-Leverage Decision

The LangChain documentation groups splitters into length-based, structure-based, and semantic approaches, and the differences between them are not cosmetic.

A fixed-length splitter cuts every N characters or tokens. It is fast, predictable, and indifferent to meaning. It will happily split a definition from the term it defines.

A structure-aware splitter respects the document’s own boundaries: Markdown headings, HTML elements, function definitions in source code. It produces chunks of uneven size, which is a feature. A section is a coherent unit of meaning; 500 characters is not.

A recursive character splitter is the practical middle ground and the usual default. It tries a prioritised list of separators, splitting on paragraph breaks first, then lines, then words, falling through only when a chunk is still too large. It approximates structure awareness without needing a format-specific parser.

Two parameters deserve conscious choice rather than a copied value:

Chunk size. Small chunks embed precisely but lose the surrounding context needed to interpret them. Large chunks embed vaguely, because a single vector has to represent several topics at once, and they consume prompt budget fast. If a corpus is dense reference material, smaller chunks with a retrieval step that pulls more of them usually beats fewer large ones.

Overlap. A fixed overlap between adjacent chunks means a sentence spanning a boundary appears in full in at least one chunk. It costs storage and adds near-duplicate results, so it is a tradeoff, not a free win. LangChain’s own knowledge-base walkthrough uses RecursiveCharacterTextSplitter with chunk_size=1000 and chunk_overlap=200, so twenty percent overlap is the number a reader most often inherits without deciding on it. Treat that pair as a starting point to be tuned against an evaluation set, not as a default that survived a measurement.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    add_start_index=True,  # keeps each chunk's offset in the parent document
)
chunks = splitter.split_documents(docs)

add_start_index=True is the part worth copying: it records where each chunk began in its parent, which is what lets a citation point at a position rather than at a whole file. Metadata from the parent document is carried onto every chunk automatically.

Measure chunk length in tokens, not characters, if the downstream constraint is a context window. Character counts and token counts diverge sharply on code, non-English text, and anything with heavy punctuation.

Stage 3: Embeddings Must Stay Consistent

An embedding model maps text to a vector. The single hard rule is that the same model must embed both the stored documents and the incoming query, because similarity is only meaningful within one model’s vector space. Mixing models produces an index that returns plausible-looking garbage with no error anywhere.

That rule has a consequence people discover late: changing the embedding model means reindexing the entire corpus. There is no incremental migration. Record the model name, version, and dimension count alongside the index itself so that a future change is a deliberate decision rather than an afternoon of confused debugging.

Dimension count is a storage and speed tradeoff. Higher dimensions carry more information and cost more memory and more time per query. Some model families publish shortened variants at reduced dimensions specifically so this can be tuned.

The embedding object is instantiated once and then handed to both the store and the query path, which is what keeps the two in the same vector space:

from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-large")

Batch the embedding calls. Embedding a corpus one document at a time is dominated by network round trips and turns a minutes-long job into an hours-long one.

Stage 4: Choosing a Vector Store

For development, an in-memory store is the correct choice. It has no setup, no service to run, and it disappears when the process exits, which is exactly what a pipeline under active iteration needs.

from langchain_core.vectorstores import InMemoryVectorStore

vector_store = InMemoryVectorStore(embeddings)
ids = vector_store.add_documents(documents=chunks)

Swapping this line for a persistent store is the only change the rest of the pipeline sees, because every store implements the same interface.

For anything persistent, the questions that actually differentiate stores are:

• Does it support metadata filtering alongside vector search? Filtering by source, date, or tenant before the similarity search is what makes multi-tenant and freshness-sensitive applications possible.
• Does it support hybrid search, combining keyword matching with vector similarity?
• What is the update path for a changed document? Some stores make deletion by metadata filter easy; others make it awkward enough that people rebuild from scratch.

Most production stores use an approximate nearest neighbour index rather than exhaustive comparison, which trades a small amount of recall for a large speed gain. That tradeoff is usually correct, but it is a tradeoff, and recall is a tunable parameter rather than a constant.

Stage 5: Retrieval Is Where Quality Shows Up

Pure vector similarity returns text that is topically close to the query. Topically close is not the same as correct. Embeddings are weak at exact tokens: identifiers, error codes, version numbers, product names, and proper nouns are precisely the things a user is most likely to type and precisely what similarity search is worst at matching.

The retrieval step itself is one call, and k is the knob that decides how much of the prompt budget retrieval consumes:

retriever = vector_store.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 5},
)
hits = retriever.invoke("which regions are covered by the 2026 policy?")

Two techniques fix most of this:

Hybrid retrieval runs a keyword search and a vector search, then merges the result lists. The keyword side catches exact strings; the vector side catches paraphrases. LangChain exposes ensemble retrievers for combining rankings from multiple sources.

Reranking takes the merged candidate set, typically twenty to fifty chunks, and scores each one against the query with a cross-encoder that sees query and document together rather than comparing two independent vectors. It is slower per document, which is why it runs on a shortlist rather than the whole corpus, and it substantially improves the ordering of what reaches the prompt.

Ordering matters more than intuition suggests. The “Lost in the Middle” analysis of long-context models found that performance is highest when relevant information sits at the very beginning or very end of the input context and degrades noticeably when the same information is buried in the middle. Retrieving twenty chunks and dumping them in arbitrary order can therefore perform worse than retrieving five well-ranked ones. More context is not automatically better context.

Assembling the Prompt

The generation step is a prompt template with a context slot, and there are three things worth putting in it beyond the retrieved text.

Give each chunk an identifier in the prompt and ask for citations in the output, so answers can be traced back to sources. Instruct the model explicitly to answer only from the provided context and to say when the context does not contain the answer, because the default behaviour is to fill the gap from parametric knowledge. And request structured output when the answer feeds another system, rather than parsing prose afterwards.

Retrieved context dominates the token bill in almost every RAG application. Prompt size scales with the number of chunks multiplied by chunk size, and it is paid on every single request. The token and agent cost sizer is a quick way to see how a change in chunk count or retrieval depth moves the cost per thousand runs before committing to it.

Before Calling It Done

Build a fixed evaluation set of representative questions with expected answers or expected source documents, and rerun it after every change to chunking, embeddings, or retrieval depth. Without it, tuning is guesswork, because an adjustment that fixes one query routinely breaks three others.

Turn on tracing so the assembled prompt is visible. Most RAG failures are retrieval failures, and they are obvious the moment the retrieved chunks are inspected and invisible until then.

Check the failure path. What happens when retrieval returns nothing relevant, when the source corpus does not contain the answer, and when the query is out of scope entirely. A pipeline that never says “not in the documents” is a pipeline that will confidently invent one.

Related Reading

The wider framing of what LangChain abstracts, and when a deterministic chain beats an agent loop, is covered in LangChain building blocks. If the pipeline is going to grow multi-step or stateful behaviour, the framework comparison is worth reading before committing. And when a retrieval-backed agent starts misbehaving, the symptom-by-symptom fixes are in LangChain agent errors.