I’m running into an issue with monitoring LLM interactions through langsmith when using crewai agents. I have langsmith tracing enabled in my project and I’m testing two different approaches to make LLM calls.
The first method uses a crewai agent that wraps the ChatOpenAI model. The second method calls the ChatOpenAI model directly through langchain. Both calls work fine and return the expected responses, but langsmith only shows traces for the direct langchain calls. The crewai agent calls don’t appear in my langsmith dashboard at all.
from crewai import Agent
from langchain_openai import ChatOpenAI
model = ChatOpenAI(
model="gpt-4o",
temperature=0.5,
)
def setup_qa_agent():
# Initialize a simple Q&A agent
qa_agent = Agent(
role='Information Assistant',
goal='Provide clear and helpful answers to user queries',
backstory="""You are an experienced assistant who specializes in giving
straightforward and accurate information. Focus on being helpful.""",
llm=model,
verbose=True
)
return qa_agent
if __name__ == "__main__":
my_agent = setup_qa_agent()
response1 = my_agent.llm.call("Share a funny pun with me")
print(response1)
response2 = model.invoke("Tell me your best pun joke")
print(response2.content)
Has anyone figured out how to get langsmith to properly trace crewai agent interactions?
CrewAI wraps LangChain calls in its own execution layer, which breaks tracing. Use crew.kickoff() instead of accessing .llm directly - it preserves LangSmith context better in my experience.
I faced a similar issue before, and it turns out CrewAI does not maintain the tracing context that LangSmith requires. To resolve this, make sure to set LANGCHAIN_TRACING_V2=“true” before importing CrewAI. Additionally, avoid using my_agent.llm.call(), as this skips CrewAI’s integrated tracing. Instead, use the agent’s execute_task or the appropriate CrewAI methods. Also, double-check your LangSmith project settings, since CrewAI may start separate tracing sessions that relate to different projects. Setting LANGCHAIN_PROJECT explicitly helped me consolidate the traces.
Had the same problem debugging production workflows. CrewAI intercepts the model execution and doesn’t pass tracing metadata through correctly. When you call my_agent.llm.call() directly, you’re bypassing CrewAI completely - it’s just a raw LangChain invoke.
Don’t try forcing tracing at the model level. Set your environment variables first, then import langsmith and call langsmith.configure() before importing crewai. This creates a global tracing context when CrewAI initializes.
Also, CrewAI handles tracing differently for task execution vs direct LLM calls. You’ll probably need to restructure your workflow to use proper CrewAI task patterns instead of accessing the underlying model directly.
This is a common CrewAI issue. The framework creates its own execution context that breaks LangSmith’s tracing hooks.
I hit this exact problem last year building a multi-agent system. What fixed it was wrapping the ChatOpenAI model with explicit LangSmith callbacks before handing it to CrewAI.
Try this:
from langsmith import Client
from langchain.callbacks import LangChainTracer
client = Client()
tracer = LangChainTracer()
model = ChatOpenAI(
model="gpt-4o",
temperature=0.5,
callbacks=[tracer]
)
The key difference? Force the tracer at the model level before CrewAI touches it. This way tracing survives CrewAI’s execution wrapper.
Make sure you’re on the latest versions too. CrewAI’s been improving their LangSmith integration lately.
This walkthrough covers LangSmith observability patterns that work well with agent frameworks like CrewAI.
CrewAI doesn’t pass LangSmith’s tracing context through properly - ran into this building a doc analysis system last month. You’re using my_agent.llm.call() which completely skips CrewAI’s task framework. That’s why only direct model calls show up in traces - you’re basically making raw LangChain calls. Ditch the direct llm access and create actual CrewAI tasks with agent.execute_task(). Tracing works way better when you let CrewAI run the whole execution instead of going around it. Also double-check your LANGCHAIN_ENDPOINT config. CrewAI sometimes starts its own tracing sessions that mess with your main project settings. I had to set the endpoint URL in environment variables to get consistent tracing for both direct calls and agent runs.
The problem is you’re mixing manual LLM calls with CrewAI’s workflow - that’s what’s creating the tracing gaps. Don’t try patching monitoring across different tools. Just automate everything with proper observability from day one.
I’ve dealt with this before. Build automated workflows that track every interaction without worrying about framework compatibility. Create one unified automation layer that handles both agent orchestration and monitoring.
With good automation, you’ll capture all LLM interactions whether they’re going through CrewAI, direct LangChain calls, or whatever framework you’re using. Complete visibility into agent conversations without debugging tracing contexts or wondering which method preserves what.
This has saved me tons of time troubleshooting monitoring gaps in complex agent systems. Stop fighting with different tracing mechanisms and automate it right from the start.
Check out how to build this kind of automated monitoring workflow: https://latenode.com
