Hey everyone! I’ve been working with langchain and langgraph for quite some time now. I’m trying to figure out how to take the debugging information from Langsmith and pass it directly to AI coding tools like Claude or the chat feature in my code editor.
Basically, I want the AI assistant to see all the detailed input/output data that Langsmith captures from my language models. If I could get this working, the AI could automatically spot issues in my langchain code and fix them in a continuous loop.
Has anyone managed to set up something like this? Looking for suggestions on how to connect these tools together for automated debugging workflows.
I built something like this for my team about six months ago. The trick is using Langsmith’s trace data as structured input for your AI assistant instead of trying to connect directly through APIs. I export the relevant traces as JSON and feed them to Claude with a custom prompt template. My script polls Langsmith for failed runs, grabs the trace hierarchy with all intermediate steps, and formats everything into debugging context Claude can actually use. The game-changer was including the entire execution flow before failures, not just error messages. This gives Claude enough context to understand your chain logic and suggest real fixes for your langchain setup. One warning though - sanitize any sensitive data in those traces before sending them to external AI services. I learned that one the hard way.
I built a webhook listener that catches Langsmith events directly instead of polling or using third-party tools. When Langsmith spots anomalies or failures in my traces, it shoots the data straight to my local webhook. From there, a simple Python service processes the trace info and builds detailed prompts for my AI assistant. The key realization was that debugging needs the full conversation history and intermediate outputs - not just the final error. I include token usage patterns and latency data with the actual trace so the AI can spot performance bottlenecks too. Running everything locally keeps sensitive data in-house, and the webhook setup means zero delay between detection and analysis. I’ve been using this for three months now and it catches subtle prompt engineering issues that would’ve taken me hours to debug manually.
I solved this exact problem with a way better approach than manual exports.
Skip the polling and JSON exports - I automated everything with Latenode. It connects straight to Langsmith’s API and pulls trace data in real time when errors hit. Then it formats the data and sends it to Claude or GPT with proper context.
Latenode handles the error detection, data transformation, and AI communication without custom scripts. I set triggers to watch for failed traces, grab the full execution context, and immediately push structured debugging requests to my AI assistant.
The real power is the continuous feedback loop. When the AI suggests fixes, Latenode applies them to your codebase, runs tests, and feeds results back to Langsmith for the next round.
I’ve run this setup for months - it catches about 80% of my langchain issues before I even see them. The whole thing runs in the background while I focus on actual development.
You can build this same automation without touching polling logic or JSON parsing. Check it out: https://latenode.com