LangSmith tracing not working with direct HTTP calls in Modal environment

[Jack81]

Hi everyone! I’m having trouble getting LangSmith to show traces when I make direct HTTP requests to my LLM in a Modal serverless setup. I’m not using any OpenAI SDK or LangChain wrapper, just plain HTTP calls, and I can’t find good docs for this specific scenario.

I’ve configured these environment variables in Modal secrets and they’re accessible:

LANGSMITH_TRACING=true
LANGSMITH_TRACING_V2=true  
LANGSMITH_ENDPOINT="https://api.smith.langchain.com"
LANGSMITH_API_KEY="my_api_key"
LANGSMITH_PROJECT="my_project_name"

My code setup looks like this:

api_key = os.environ.get("LANGSMITH_API_KEY")
endpoint = os.environ.get("LANGSMITH_ENDPOINT")

tracing_client = Client(
    api_key=api_key,
    api_url=endpoint,
)

@traceable(name="CustomLLM.execute", client=tracing_client)
async def execute(self, prompt: str, streaming: bool = True, format_type: dict = None) -> str:
    # LLM call here

I’m making the actual request like this:

async with http_client.stream("POST", f"{self.api_url}/chat/completions", json=request_data, headers=auth_headers) as resp:
    # process response

No traces appear in my LangSmith dashboard. Any ideas what I’m missing?

[deltaDreamer]

The Problem:

You’re experiencing issues with LangSmith trace collection when making direct HTTP requests to your LLM within a Modal serverless function. The @traceable decorator isn’t automatically capturing the HTTP calls, and traces aren’t appearing in your LangSmith dashboard, despite having correctly configured environment variables and initialized the LangSmith client. This is likely due to the ephemeral nature of Modal serverless functions and the way tracing needs to be handled in this environment.

Understanding the “Why” (The Root Cause):

Modal serverless functions have a short lifespan; they spin up and down quickly. This behavior can interfere with trace collection if not handled properly. LangSmith’s @traceable decorator, while convenient, might not be sufficient for automatically capturing traces within the context of such a dynamic environment. Directly using the LangSmith client to manage traces gives you finer control, ensuring your traces are sent before the function terminates.

Step-by-Step Guide:

1. Manually Create and Manage Traces: Instead of relying solely on the @traceable decorator, directly use the LangSmith client to create and manage traces within your execute function. This allows explicit control over when traces are started and ended, ensuring they are properly captured before your Modal function exits. The following code demonstrates the correct approach:

@traceable(name="CustomLLM.execute", client=tracing_client)
async def execute(self, prompt: str, streaming: bool = True, format_type: dict = None) -> str:
    with tracing_client.trace(
        name="llm_http_call",
        inputs={"prompt": prompt, "streaming": streaming}
    ) as run:
        async with http_client.stream("POST", f"{self.api_url}/chat/completions", json=request_data, headers=auth_headers) as resp:
            # process response
            result =  # your processed result

        run.end(outputs={"response": result})
        return result

2. Verify Environment Variables: Double-check that your Modal function correctly mounts and accesses the environment variables defined in your Modal secrets. Add print statements to confirm that the LANGSMITH_API_KEY, LANGSMITH_ENDPOINT, and LANGSMITH_PROJECT variables are accessible within your function:

print(f"LangSmith API Key: {os.environ.get('LANGSMITH_API_KEY')}")
print(f"LangSmith Endpoint: {os.environ.get('LANGSMITH_ENDPOINT')}")
print(f"LangSmith Project: {os.environ.get('LANGSMITH_PROJECT')}")

3. Test with a Simple Trace: Before integrating into your main function, create a simple test trace outside your execute function to verify that the LangSmith client is correctly configured and able to send traces.

4. Force Flush Traces (Optional): In some cases, explicitly closing the LangSmith client using tracing_client.close() at the end of your Modal function might help ensure all traces are sent. This is a safeguard against the function terminating before the traces are fully transmitted.

Common Pitfalls & What to Check Next:

• Modal Function Lifecycle: Remember that Modal functions are ephemeral. Ensure your tracing logic is executed before the function completes, preventing traces from being lost.
• Network Connectivity: Verify network connectivity between your Modal function and the LangSmith API. Temporary network interruptions could cause trace loss.
• Rate Limiting: If you’re sending many traces, you might be hitting LangSmith’s rate limits. Check the LangSmith documentation for information on managing rate limits.
• LANGSMITH_TRACING_V2: Try setting LANGSMITH_TRACING_V2=false as suggested in other answers. Version 2 of the tracing might have compatibility issues with Modal.

Still running into issues? Share your (sanitized) config files, the exact command you ran, and any other relevant details. The community is here to help!

[Liam23]

modal environments are tricky with langsmith. set LANGSMITH_TRACING_V2=false - the v2 flag breaks things sometimes. call tracing_client.create_run() directly instead of using just the decorator. i had the same issue and had to force flush traces with tracing_client.close() at the end of my modal function.

[ryanl]

I hit this exact issue last year setting up tracing for our custom API endpoints.

The @traceable decorator won’t automatically capture HTTP calls. You’ve got to manually create spans for the LLM request.

Here’s what worked:

@traceable(name="CustomLLM.execute", client=tracing_client)
async def execute(self, prompt: str, streaming: bool = True, format_type: dict = None) -> str:
    with tracing_client.trace(
        name="llm_http_call",
        inputs={"prompt": prompt, "streaming": streaming}
    ) as run:
        async with http_client.stream("POST", f"{self.api_url}/chat/completions", json=request_data, headers=auth_headers) as resp:
            # process response
            result = # your processed result
            
        run.end(outputs={"response": result})
        return result

Double check your Modal function’s actually getting those environment variables. I’ve seen Modal secrets not mount properly to the container.

Add a quick print to verify the client initializes correctly:

print(f"LangSmith client initialized: {tracing_client is not None}")
print(f"Project: {os.environ.get('LANGSMITH_PROJECT')}")

If the client looks good but traces still don’t appear, try a simple test trace outside your main function to confirm the connection works.