I’m building a chatbot using Langchain that pulls answers from a vector database. The bot works great for questions that have answers in the database, but it won’t respond to anything outside of that context.
I need it to handle two types of questions:
Questions with answers in the vector database
General questions or references to chat history
Here’s my current setup:
my_template = """Please respond to the user's question using the provided context.
When no relevant context exists, use your general knowledge to help.
Context: {context}
Previous conversation: {chat_history}
User: {question}
Bot:"""
my_prompt = PromptTemplate(
input_variables=["context", "chat_history", "question"],
template=my_template
)
# Adding sample conversation to memory
memory.save_context({"input": "Tell me about France's capital"},
{"output": "Paris"})
# Setting up the retrieval system
retrieval_chain = RetrievalQA.from_chain_type(
llm=model,
retriever=my_vectorstore.as_retriever(),
memory=memory,
chain_type_kwargs={'prompt': my_prompt}
)
When I test with:
user_question = "What country did I ask about earlier?"
response = retrieval_chain({"query": user_question})
The bot can’t answer even though this info is in chat history. It only works with vector database content. Any suggestions?
Yeah, RetrievalQA forces everything through vector search first. Even with chat history in your prompt, it still retrieves documents and only uses what the vector store returns.
I hit this same problem building an internal tool last year. What worked was creating a hybrid approach - check the retrieval results first, and if similarity scores are below a threshold (I use 0.7), just bypass the retrieved context completely.
Here’s my pattern:
# Get retrieval results with scores
retriever = my_vectorstore.as_retriever(search_kwargs={'k': 3, 'score_threshold': 0.7})
retrieved_docs = retriever.get_relevant_documents(user_question)
if not retrieved_docs:
# No relevant docs found, use general knowledge/memory only
general_template = """Previous conversation: {chat_history}
User: {question}
Bot:"""
# Use regular LLM call with memory
else:
# Use your existing retrieval chain
Also - your memory setup might not be feeding into the chain properly. Try logging what actually gets passed to the prompt to make sure chat_history contains what you expect.
The router approach others mentioned works too, but this fallback method’s simpler to debug and tune.
Hit this same issue building a customer support system. You’re cramming everything through a retrieval pipeline that only handles vector search - that’s your problem.
I fixed it with smart routing that decides upfront: does this question need database lookup or can I handle it from memory/general knowledge? Built mine in Latenode since you can create decision flows visually instead of wrestling with messy chain logic.
My workflow: LLM classifies the incoming question first. General knowledge or previous conversation? Routes to basic chat completion with memory. Needs specific database info? Goes through retrieval.
Best part - you see the whole decision tree in one view and tweak routing logic without touching code. Can add fallback paths when classification isn’t confident too.
Way cleaner than shoving everything into one chain handling multiple use cases. Each path gets optimized for what it actually does.
Had this exact problem building a knowledge base chatbot. RetrievalQA is your bottleneck here - it always expects retrieved docs to work and treats empty results as a dead end instead of falling back gracefully. Even with chat_history in your prompt, the chain puts retrieved context above everything else. I ditched RetrievalQA completely and built a custom chain with upfront decision logic. Before any retrieval, I use a quick LLM call to classify if the question needs database lookup or can be answered from chat history alone. Questions like “what did I ask earlier” or “tell me more” go straight to chat completion with full memory. For database questions, I kept retrieval but added proper fallbacks. When retrieval confidence is low, it switches to general knowledge mode instead of returning nothing. Way better control over response quality and no more frustrating non-answers on follow-ups.
RetrievalQA just wasn’t built for conversations - it only works with retrieved context and can’t remember past exchanges. I had this same headache last year building a support bot. Switch to ConversationalRetrievalChain instead. It handles both retrieval and chat history at the same time. Here’s what’s different: it takes your chat history, uses it to rewrite questions before searching, then combines both the retrieved docs and conversation context for responses. Another option is building a router that figures out whether questions need database lookup or can be answered from chat history alone. More work to set up, but you get better control over how it classifies questions.