The answer to how to reduce LangChain token costs is to stop resending tokens you already paid for. Roughly in order of payoff: put static content first and enable provider prompt caching, bound history with SummarizationMiddleware, clear stale tool outputs, send fewer tool schemas, cap model calls, and route cheap steps to a smaller model.
The pattern is familiar. A demo agent costs pennies per conversation, then production brings ten-step tool loops and thirty-turn threads, and spend outgrows request count. Nothing is broken; that is how stateless chat APIs bill.
Why does a LangChain agent use so many tokens?
A LangChain agent resends its entire context on every model call. Each iteration of the tool loop ships the system prompt, every tool’s JSON schema, the full message history and every prior tool result, then pays for all of it again as input. A ten-step run pays for the system prompt ten times.
The arithmetic compounds. If each step appends a tool result of r tokens, call i carries roughly base + i*r input tokens, so total input for an n-step run grows with n². Output and reasoning tokens matter too (LangChain reports reasoning under output_token_details in UsageMetadata), but in tool-heavy agents the growth is on the input side.
The metric that matters: cost per completed task
Tokens per day tracks traffic, not efficiency, and tokens per call hides loops. Track cost per completed task: every model call in a run, priced by billing class, divided by runs that actually finished.
cost_per_task = sum over calls of (
uncached_in * p_in
+ cache_read * p_in * r_read
+ cache_write * p_in * r_write
+ output * p_out
) / completed_tasksAnthropic’s prompt caching docs put r_read at 0.1 for most models and r_write at 1.25 for 5-minute cache writes, or 2 for 1-hour writes. OpenAI’s prompt caching guide lists 0.1x reads and 1.25x writes for GPT-5.6 and later. Dividing by completed tasks charges you for runs that loop into a call limit and give up, which is the point.
Wiring up measurement
LangChain attaches normalized usage to every AIMessage as usage_metadata. The models guide documents get_usage_metadata_callback, which totals it per model across a run:
from langchain_core.callbacks import get_usage_metadata_callback
R_READ, R_WRITE = 0.10, 1.25 # Anthropic 5m cache; confirm per model
def usage_cost(usage: dict, p_in: float, p_out: float) -> float:
"""p_in, p_out: USD per 1M tokens from your provider's price list."""
d = usage.get("input_token_details") or {}
read, write = d.get("cache_read", 0), d.get("cache_creation", 0)
uncached = usage["input_tokens"] - read - write
return (uncached * p_in + read * p_in * R_READ + write * p_in * R_WRITE
+ usage["output_tokens"] * p_out) / 1_000_000
with get_usage_metadata_callback() as cb:
agent.invoke({"messages": [{"role": "user", "content": question}]})
for model_name, usage in cb.usage_metadata.items():
print(model_name, usage_cost(usage, *PRICES[model_name]))In the UsageMetadata reference example, cache_read and cache_creation sit inside the input_tokens total, which is what the subtraction assumes; check one raw response from your integration before trusting the figure. For dashboards, LangSmith cost tracking derives cost from token counts and a workspace model pricing map, with per-run breakdowns in the trace tree and project-level cost panels.
Fix 1: make the prefix cacheable
Prompt caching pays off most in agents because every loop iteration resends an identical prefix. Per Anthropic’s docs, the cache prefix is built in the order tools, system, messages, and changing tool definitions invalidates all three. Below a per-model minimum (1,024 tokens for Claude Sonnet 4.6, 4,096 for Claude Haiku 4.5), requests are processed uncached and “no error is returned.” OpenAI enables caching by default on supported models and reports hits as cached_tokens.
LangChain wraps the Anthropic side in AnthropicPromptCachingMiddleware, which defaults to ttl="5m" and appears in the stack below. Whether it works depends on layout: static content first, dynamic content last, which is also OpenAI’s guidance. A timestamp, request ID or user name interpolated into the system prompt changes the prefix on every request, so you pay the write premium and never collect the read discount.
Fix 2: bound the conversation history
A checkpointer with no context policy resends every turn forever. SummarizationMiddleware fires on a trigger of ("tokens", n), ("messages", n) or ("fraction", x) of the model’s context, folds older messages into a summary, and by default keeps the last 20 messages verbatim, per the built-in middleware reference. Point its model at a small, cheap model. trim_messages is the free alternative that forgets instead of summarizing; the trade-offs are in LangChain memory types explained.
Fix 3: clear stale tool outputs
Tool results such as scraped pages, SQL rows and API payloads are often the largest blocks in an agent transcript, and the model rarely needs the one from six steps ago. ContextEditingMiddleware with ClearToolUsesEdit replaces older tool outputs with a placeholder ("[cleared]" by default) once context crosses a token trigger (default 100,000), keeping the three most recent. The cheaper fix sits upstream: make the tool return 20 rows and a count, not 2,000 rows.
Fix 4: send fewer tool schemas
Every bound tool’s name, description and JSON schema is input on every call. With dozens of tools, LLMToolSelectorMiddleware runs a selector model first to pick at most max_tools, with always_include for tools that must stay; point it at a small model. It collides with caching: a tool list that changes per request invalidates Anthropic’s cache from the top. Use dynamic selection or a cached static tool block, not both.
Fix 5: cap the loop
The most expensive run is the one that never finishes. ModelCallLimitMiddleware takes a run_limit per invocation and a thread_limit across a thread, with exit_behavior of "end" or "error"; ToolCallLimitMiddleware does the same per tool. If runs regularly hit the cap, the cause is usually a parsing or tool-selection bug, covered in LangChain agent errors: loops, tools, parsing.
The combined middleware stack
from langchain.agents import create_agent
from langchain.agents.middleware import (
ClearToolUsesEdit,
ContextEditingMiddleware,
ModelCallLimitMiddleware,
SummarizationMiddleware,
ToolCallLimitMiddleware,
)
from langchain_anthropic import ChatAnthropic
from langchain_anthropic.middleware import AnthropicPromptCachingMiddleware
from langgraph.checkpoint.memory import InMemorySaver
agent = create_agent(
model=ChatAnthropic(model="claude-sonnet-4-6", max_tokens=1024), # caps output
tools=[search_docs, run_sql, fetch_page],
system_prompt=SYSTEM_PROMPT, # static text only: no timestamps, no user IDs
checkpointer=InMemorySaver(), # database-backed saver in production
middleware=[
AnthropicPromptCachingMiddleware(ttl="5m"),
ContextEditingMiddleware(edits=[ClearToolUsesEdit(trigger=50_000, keep=3)]),
SummarizationMiddleware(
model="openai:gpt-5.4-mini",
trigger=("tokens", 80_000),
keep=("messages", 20),
),
ModelCallLimitMiddleware(run_limit=8, thread_limit=60, exit_behavior="end"),
ToolCallLimitMiddleware(tool_name="run_sql", run_limit=3),
],
)Thresholds are illustrative. Tune them against traces, not intuition.
Fix 6: route cheap steps to cheap models
Not every call needs the flagship model. Summarization, tool selection, classification and query rewriting are candidates for a mini model, which is why the docs’ middleware examples pass a smaller model to those steps. The FrugalGPT paper formalized LLM cascades, trying cheap models first and escalating when the answer looks unreliable, and reports matching the best individual model “with up to 98% cost reduction” on its benchmarks. That is an author-reported research result, not an independent benchmark, so validate any cascade on your own eval set.
Fix 7: retrieve fewer tokens in RAG
In a RAG chain, retrieved context is input on every query: k chunks times average chunk length. Cutting k from 10 to 4 removes 60% of that block before any other change, so check recall@k on your golden set before shipping it. Sizing is covered in LangChain chunk size for RAG, and the baseline chain in the LangChain RAG pipeline setup guide. For contexts that cannot shrink, LLMLingua compresses prompts with a small language model and reports “up to 20x compression with little performance loss,” also an author-reported result.
Fix 8: cache answers and batch offline work
set_llm_cache(InMemoryCache()) returns a stored generation when prompt and model string match exactly, per the InMemoryCache reference; SQLiteCache persists to disk. Exact match rarely hits on free-form chat but works for deterministic pipeline steps and eval reruns. RedisSemanticCache from langchain-redis matches by embedding distance, with a default distance_threshold of 0.2. For offline jobs, OpenAI’s Batch API gives a “50% cost discount compared to synchronous APIs” within a 24-hour window. LangChain’s Runnable .batch() is a concurrency helper over normal requests and does not earn that discount by itself.
What you’ll see
Plot input tokens per model call against step index, one line per run. Before: a ramp that steepens with every tool result, plus a cluster of runs pinned at the call cap. After context editing and summarization: a sawtooth that drops each time an edit fires.
Add cache hit ratio, cache_read / input_tokens, per route. A healthy agent route climbs after the first call of each run; a line stuck near zero after enabling caching means the prefix changes every request or sits under the model’s minimum. Cost per task should fall while completion rate holds. If completion drops with it, a summary or a cleared tool output removed something the model needed.
Caveats
- Cache writes cost more than plain input. A prefix not reused within the TTL is pure premium, and 1-hour writes at 2x only pay back under steady traffic.
- Any edit to earlier messages, from summarization or cleared tool outputs, changes the prefix from that point and forces a fresh cache write. Fire edits rarely and in large steps.
- Summaries are lossy rewrites. Numbers, IDs and negations go first.
- A shared response cache is a data-leak path. Scope it per tenant, or one user’s answer reaches another; a loose semantic threshold returns confident answers to different questions.
- Per-model cost labels on Prometheus metrics are fine; per-user or per-thread labels blow up cardinality. Keep those breakdowns in traces.
- Prices and cache multipliers change between model generations. Keep them in config, not code.
FAQ
does langchain cache llm responses by default?
No. LangChain does not cache model responses until you set a global cache with set_llm_cache, using InMemoryCache, SQLiteCache or a semantic cache such as RedisSemanticCache. Provider prompt caching is a separate mechanism: OpenAI applies it automatically on supported models, and for Anthropic models AnthropicPromptCachingMiddleware adds the cache breakpoints.
how much does prompt caching save?
Cached input costs 0.1x the base input price on most Anthropic models and on OpenAI’s GPT-5.6 and later, so each reused prefix token is 90% cheaper after the first call. That first call pays a 1.25x write premium on a 5-minute cache. Savings scale with how many calls reuse the prefix inside the TTL.
is summarization cheaper than trimming langchain messages?
No, trimming is cheaper per call because trim_messages makes no extra model request, but it discards older turns outright. SummarizationMiddleware pays for a summarizer call each time its trigger fires and keeps a compressed history. It is worth it when the tokens it removes from later calls cost more than that summary call.
does streaming reduce langchain token costs?
No. Streaming changes when tokens arrive, not how they are billed, so the same request costs the same either way. It improves time-to-first-token for users. Check that your integration still returns usage metadata on streamed responses, or cost tracking goes blind; see how to stream LangChain responses.
Related across the network
- Semantic Caching for LLMs: Hit Rates and Failure Modes — llmops.report
- LangChain vs LlamaIndex: Production Verdict for 2026 — llamaindexhub.com
