How to Use LangChain with Ollama: Local Chat, Tools, Structured Output and Embeddings

The operational problem with a local model is never the first call. It is the third call, twenty minutes later, when the weights have been evicted from VRAM and a request that took 800 ms now takes twelve seconds, or the RAG prompt that quietly lost its first two thousand tokens because nobody set the context window. This guide covers how to use LangChain with Ollama end to end: install, chat, streaming, tool calling, structured output and embeddings, plus the two server settings that decide whether the integration behaves in production or only in a notebook.

Install and Pull a Model

Ollama runs as a local HTTP server. By default it binds to 127.0.0.1 on port 11434, per the Ollama FAQ. Install it from ollama.com, then pull a model and confirm it is on disk:

ollama pull llama3.1
ollama list

The Python integration is a separate package, not part of langchain itself. The LangChain ChatOllama docs install it with:

pip install -U langchain-ollama

PyPI lists langchain-ollama 1.1.0, released 2026-04-07, requiring Python 3.10 or newer. It exposes ChatOllama, OllamaLLM (completion-style) and OllamaEmbeddings. Use ChatOllama for anything conversational or agentic.

The Two Settings That Matter: num_ctx and keep_alive

Two server-side defaults cause most “it worked yesterday” reports.

Context window. The FAQ states the default context window is 4096 tokens. The server does not extend it because your prompt is longer; whatever does not fit does not reach the model. A RAG chain stuffing five retrieved chunks plus a system prompt exceeds 4096 easily. Raise it per request with num_ctx, or globally with OLLAMA_CONTEXT_LENGTH=8192 ollama serve. A larger window costs VRAM for the KV cache, so pick the smallest value your longest prompt needs, then check ollama ps to confirm the model still reports 100% GPU rather than a CPU/GPU split.

Model residency. Per the Ollama API reference, keep_alive “controls how long the model will stay loaded into memory following the request (default: 5m)”. After five idle minutes the next request pays a full load. The FAQ documents the accepted values: a duration string such as "10m" or "24h", seconds as an integer, -1 to keep the model loaded indefinitely, or 0 to unload immediately. OLLAMA_KEEP_ALIVE sets the server default; a per-request value overrides it.

Both are constructor arguments on ChatOllama, which is the point of doing this from LangChain rather than a raw HTTP client.

Wiring It Up: ChatOllama

The ChatOllama reference lists model as the only required field. Everything else defaults to None, meaning the server’s own default applies.

from langchain_ollama import ChatOllama

llm = ChatOllama(
    model="llama3.1",
    temperature=0,
    num_ctx=8192,              # context window for this request
    num_predict=1024,          # cap on generated tokens
    keep_alive="30m",          # override the 5m eviction default
    base_url="http://localhost:11434",
    validate_model_on_init=True,  # fail at construction, not on first invoke
)

messages = [
    ("system", "You are a terse assistant. Answer in one sentence."),
    ("human", "What does num_ctx control in Ollama?"),
]
reply = llm.invoke(messages)
print(reply.content)
print(reply.usage_metadata)   # input/output token counts from the server

for chunk in llm.stream("Explain KV cache in two sentences."):
    print(chunk.content, end="", flush=True)

validate_model_on_init=True is worth the extra round trip: without it a typo in the model name surfaces on the first real request, usually from inside a chain three layers deep. base_url only needs setting when Ollama runs on another host. The FAQ’s route for that is OLLAMA_HOST on the server plus a reverse proxy such as Nginx, which is also where authentication belongs, since the server itself does not provide any. For thinking models, the reasoning argument maps to Ollama’s think parameter.

Tool Calling with bind_tools

Ollama added tool calling on 2024-07-25, initially for Llama 3.1, Mistral Nemo, Firefunction v2 and Command-R+. The LangChain docs use gpt-oss:20b; Ollama’s own tool-calling docs use qwen3. Either works with the standard bind_tools interface:

from langchain.tools import tool
from langchain_ollama import ChatOllama

@tool
def get_order_status(order_id: str) -> str:
    """Look up the shipping status for an order id."""
    return f"Order {order_id}: shipped"

llm = ChatOllama(model="gpt-oss:20b", temperature=0).bind_tools([get_order_status])
result = llm.invoke("Where is order 8841?")
print(result.tool_calls)
# [{'name': 'get_order_status', 'args': {'order_id': '8841'}, 'id': '...', 'type': 'tool_call'}]

The model returns a request to call the tool; it executes nothing. You run the function, append a ToolMessage with the result and invoke again, or hand the loop to a LangGraph agent. Two things to know first. The API reference describes tools as used “if supported”: a model whose chat template has no tool grammar will not emit tool_calls, and prompting does not fix that. And when streaming, Ollama’s tool-calling docs note that partial thinking, content and tool_calls fields arrive across chunks and must be accumulated before the next request; LangChain’s AIMessageChunk addition handles this, a hand-rolled loop must.

Running locally does not change the threat model. A local agent that reads untrusted documents and holds a tool that can send data is the exfiltration setup described in Agent Tool-Use Exfiltration, with a different inference endpoint.

Structured Output

Ollama’s format parameter on /api/chat accepts "json" or a full JSON schema, a capability announced on 2024-12-06. LangChain wraps it in with_structured_output; in current langchain-ollama the default method is "json_schema", with "function_calling" and "json_mode" as alternatives, per the source.

from pydantic import BaseModel
from langchain_ollama import ChatOllama

class Country(BaseModel):
    name: str
    capital: str
    languages: list[str]

llm = ChatOllama(model="llama3.1", temperature=0)
structured = llm.with_structured_output(Country)
print(structured.invoke("Tell me about Canada."))
# name='Canada' capital='Ottawa' languages=['English', 'French']

The schema constrains shape, not truth. A str field will be a string; whether it is the right string is still the model’s problem, so keep validation in the Pydantic model where it can reject bad values.

Embeddings for RAG

The OllamaEmbeddings docs show the same pattern for vectors, including a dimensions argument for models that support truncated output sizes:

from langchain_ollama import OllamaEmbeddings

embeddings = OllamaEmbeddings(model="qwen3-embedding:8b", dimensions=1024)
query_vec = embeddings.embed_query("How do I set num_ctx in LangChain?")
doc_vecs = embeddings.embed_documents(["chunk one", "chunk two"])

This drops into any LangChain vector store the way a hosted embedding model does; the walkthrough in LangChain RAG Pipeline: Setup to First Answer applies unchanged. Pull the embedding model separately, and note it shares VRAM and the same keep_alive eviction with the chat model. Two models alternating on one GPU that cannot hold both will thrash.

What You’ll See

Healthy: ollama ps shows 100% GPU, latency is flat across a session, and usage_metadata on each reply reports input tokens that match your prompt length.

Unhealthy: latency spikes to seconds on the first request after a quiet period (eviction); ollama ps shows a CPU/GPU split (VRAM exceeded, often right after raising num_ctx); answers ignore instructions early in a long prompt (context overflow). For the VRAM case the FAQ lists OLLAMA_FLASH_ATTENTION=1 and KV cache quantization via OLLAMA_KV_CACHE_TYPE set to q8_0 or q4_0 instead of the default f16.

Caveats

• Context overflow is silent. Nothing in LangChain warns that the server truncated the prompt. Log usage_metadata and alert when input tokens approach num_ctx.
• Tool support is per model, not per server. Check the model page on ollama.com before assuming bind_tools works.
• Structured output is shape, not correctness. Validate in Pydantic; do not trust a schema-conformant hallucination.
• Network exposure has no auth. OLLAMA_HOST on a non-loopback address without a proxy exposes an unauthenticated inference endpoint.
• You now own the monitoring. A hosted API gives you a status page and a usage dashboard; a local server gives you neither. That shift is laid out in Local Coding Assistants Crossed the Quality Bar: Now Observe Them.

Related across the network

• CVE Roundup: AI/ML Infrastructure Vulnerabilities — Q1 2026 — ai-alert.org
• AI/ML CVE Roundup: May 2026 — What Got Patched — ai-alert.org
• Guardrails AI: Output Validation Without Retraining — aisecreviews.com
• LlamaIndex vs LangChain: Which to Use for RAG — llamaindexhub.com
• Inference Server CVEs: vLLM, Ollama, llama.cpp, Triton — mlcves.com