The first time I seriously doubted the framework was not because the model hallucinated. It was because the answer looked plausible, contained a citation, and was still wrong.
The assistant had retrieved a chunk from a deprecated help page because one part of the pipeline applied a metadata filter, another part did not, and the final prompt assembly made the whole thing look coherent. Debugging it meant stepping through wrappers, runnable compositions, and framework-specific assumptions instead of asking the real question: why did retrieval favor the wrong document?
That was the point where I stopped treating LangChain as the core of the RAG system and started treating it as an optional integration layer.
This is not an anti-framework article. LangChain solved a real problem: it gave developers a fast way to compose LLM applications when the ecosystem was young and everyone was still figuring out the basics. But once RAG moved from demo to production, the problems changed. The hard parts stopped being “call the model” and became: permission-aware retrieval stable chunking hybrid search reranking evaluation document ingestion failures embedding migrations traceability when an answer goes wrong
Rebuilding the pipeline without LangChain made some things dramatically better. It also made some things more annoying, more expensive, and more time-consuming than I expected.
If you are deciding whether to keep, adopt, or remove LangChain from a production RAG system: Removing LangChain improved debugging, retrieval control, evaluation, observability, and cost discipline. Removing LangChain made harder document loading, integration maintenance, and the long tail of “small” pipeline decisions. The biggest win was not performance. It was that the pipeline became explicit. The biggest downside was that I became responsible for a lot of boring glue code that frameworks usually hide. My current rule: prototype with high-level tools, but own the retrieval core when the product depends on answer quality.
📋 Table of Contents The abstraction stopped being a shortcut and became a boundary Chunking stopped being “split by 800 characters” Retrieval became a small query planner Hybrid search was the unglamorous fix for exact identifiers Reranking became the highest-leverage quality gate Embedding generation became a data-engineering job Evaluation got easier once the pipeline had seams Observability changed from “the answer looks weird” to “chunk 7f2a was dropped” What got worse: the long tail of boring integration work Where I draw the line now The abstraction stopped being a shortcut and became a boundary
Scenario: A user asks, “What changed in webhook authentication?” The system retrieves something that mentions authentication, but not the correct product version. The final answer sounds confident. The problem is not the model. The problem is that the retrieval request did not carry the right filters, and the abstraction made that hard to see.
Why it matters: In early RAG projects, abstractions help you move quickly. You connect a loader, a splitter, an embedding model, a vector store, and a prompt template. But in production, the interesting failures happen in the spaces between those components.
When those spaces are hidden behind generic chain-like composition, you end up debugging the composition layer instead of the retrieval behavior.
Solution: I rebuilt the pipeline around explicit stages with small interfaces. Not a huge framework. Just enough structure to make each stage testable.
Why this works: The important part is not that this code is “framework-free.” The important part is that the seams are visible.
If retrieval is bad, I look at planquery and retriever.retrieve. If the prompt is bad, I look at buildprompt. If the answer is unfaithful, I inspect evidence.
💡 Practical note: If your LangChain usage already has clear boundaries around retrieval, parsing, and prompt construction, removing the framework may not help much. The problem is not the library itself. It is whether the library hides the decisions you now need to debug. Chunking stopped being “split by 800 characters”
Scenario: A support article contains a table of error codes. The user asks about one specific code. The retriever returns a chunk that includes the correct code, but not the header row explaining what the columns mean. The model guesses. Sometimes it guesses wrong.
Why it matters: A lot of early RAG advice treated chunking as a text-length problem:
That works for simple prose. It falls apart for real documents: tables code blocks numbered steps headings with nested context FAQs legal clauses API reference docs product changelogs
In production, chunking is not a text problem. It is a document-structure problem.