How to include retrieval metadata in LangChain agent response

Other Questions langchain
1

I’m building a LangChain agent and need help with getting additional information from the retrieval process. Currently my agent only returns the answer text, but I want to also get the source documents that were retrieved.

# Setting up my agent tools
my_tools = [
    StructuredTool.from_function(helper_function),
    Tool(
        name="DocumentQA",
        func=(lambda query: search_documents(vector_store, query, org_name, lang, metadata=metadata)),
        description="Use this when you need to find answers from documents. Ask specific questions",
        return_direct=True,
        verbose=True
    )
]

# Creating the agent
my_agent = initialize_agent(
    my_tools,
    llm_model,
    agent=AgentType.OPENAI_FUNCTIONS,
    verbose=True,
    system_message=sys_prompt,
    max_iterations=3,
    agent_kwargs={
        "system_message": sys_prompt
    },
    memory=chat_memory
)

My document search function looks like this:

def search_documents(vector_store: Pinecone, query: str, org_name: str, lang: str = "english", metadata: str = ""):
    search_result = vector_store(inputs={"query": query, "text": query, "org": org_name, "lang": lang})
    metadata = search_result.get("summaries", "")
    return search_result["response"]

The problem is that I only get the text answer back from the agent. I also want to see which documents were found during the similarity search. Is there a way to return both the answer and the source document information together?

2

Try modifying your tool definition to handle the metadata better. I had the same issue and figured out the trick is changing how your tool processes and formats the return data.

Don’t just change the search function return format - wrap it in a formatting function that structures the output so it’s easier to read. This worked for me:

def format_search_response(vector_store, query, org_name, lang, metadata=""):
    result = search_documents(vector_store, query, org_name, lang, metadata)
    formatted_response = f"Answer: {result['response']}\n\nSources: {', '.join([doc.get('id', 'Unknown') for doc in result.get('matches', [])])}"
    return formatted_response

Use this wrapper in your Tool definition instead of the lambda. You’ll get more control over how the metadata shows up in the final agent response while keeping the structured data accessible. The agent works exactly the same but now includes your source information in a readable format.

3

print out ur full search_result first - see what’s actually coming back from the vector store. sometimes it’s not a langchain issue, just the vector db returning docs in a weird format. debug that before u mess with the return structure.

4

Been dealing with this exact scenario for months at work. You’re fighting against how LangChain handles tool outputs when return_direct=True.

Remove return_direct=True from your DocumentQA tool. This lets the agent process the full response instead of immediately returning just the text portion.

Then modify your search function to return a formatted string that includes everything:

def search_documents(vector_store: Pinecone, query: str, org_name: str, lang: str = "english", metadata: str = ""):
    search_result = vector_store(inputs={"query": query, "text": query, "org": org_name, "lang": lang})
    
    answer = search_result["response"]
    sources = search_result.get("matches", [])
    
    source_info = "\n\nRetrieved from:"
    for doc in sources[:3]:  # limit to top 3 sources
        source_info += f"\n- {doc.get('metadata', {}).get('title', 'Document')} (score: {doc.get('score', 'N/A')})"
    
    return answer + source_info

This works because the agent can now see and work with the complete response. I’ve used this pattern in production systems where we needed full traceability of document sources.

The agent will naturally include the source information in its final response since it’s part of the tool output it receives.

5

The problem is your search_documents function only returns the text response. You need it to return structured data with both the answer and source info. I ran into this same issue and fixed it by returning a dictionary instead of just the response string.

Here’s how to modify your search_documents function:

def search_documents(vector_store: Pinecone, query: str, org_name: str, lang: str = "english", metadata: str = ""):
    search_result = vector_store(inputs={"query": query, "text": query, "org": org_name, "lang": lang})
    
    return {
        "answer": search_result["response"],
        "sources": search_result.get("matches", []),
        "metadata": search_result.get("summaries", "")
    }

With return_direct=True, the agent passes this whole dictionary back to you. Now you can grab both the answer text and source documents from the response. This keeps all your retrieval context intact and still works with your current agent setup.

6

I’ve hit this exact metadata issue before. The real problem isn’t tweaking your return format - it’s getting reliable data flow between retrieval and agent response.

Don’t patch together custom formatters or pray your vector store returns consistent structures. Just automate the whole thing. I built something similar when I needed to track document sources, confidence scores, and metadata across different retrieval calls.

Create an automation that handles document search, formats responses with answer and source info, then feeds everything back to your agent consistently. No more debugging weird vector store responses or maintaining custom wrapper functions.

Bonus: the automation logs all retrieval metadata, tracks your most-used sources, and alerts you when certain documents go missing. Way cleaner than hardcoded return formats that break every time your data structure shifts.

You can build this whole flow visually without touching existing LangChain code. Connect your vector store, add formatting logic, pipe it back to your agent.