LangSmith DynamicRunEvaluator throws error about unknown parameter 'evaluator_run_id' in evaluate_run method

Other Questions langsmith
1

Getting TypeError when evaluating dataset results

I’m trying to run an evaluation on my dataset that contains ground truth data. I want to verify the quality of generated SQL queries but keep hitting this error about evaluator_run_id parameter.

The error message says that evaluate_run() method doesn’t accept the evaluator_run_id argument, but the framework seems to be trying to pass it anyway.

from langsmith import wrappers, Client
from pydantic import BaseModel, Field
from openai import OpenAI

langsmith_client = Client()
wrapped_openai = wrappers.wrap_openai(OpenAI())

def my_target_function(input_data: dict) -> dict:  
    request_data = {
        "workflowId": "yyy",
        "parameters": {
            "user_question": input_data['query']
        },
        "action": "executeWorkflow",
        "userId": 'yyy'
    }

    api_response = requests.post(api_url, headers=request_headers, json=request_data)
    print(api_response.json()['output']['answer'])
    return api_response.json()['output']['answer']

eval_results = langsmith_client.evaluate(
  my_target_function,
  data="Test dataset",
  evaluators=[
      precision_evaluator,
  ],
  max_concurrency=2,
)

Full error traceback:

Error running evaluator <DynamicRunEvaluator precision_evaluator> on run 3f599063-df1a-58b8-b223-bde5a05a8fe4: TypeError("DynamicRunEvaluator.evaluate_run() got an unexpected keyword argument evaluator_run_id")
Traceback (most recent call last):
  File c:\Python313\Lib\site-packages\langsmith\evaluation\_runner.py, line 1634, in _run_evaluators
    evaluator_response = evaluator.evaluate_run(  # type: ignore[call-arg]
        run=run,
        example=example,
        evaluator_run_id=evaluator_run_id,
    )
TypeError: DynamicRunEvaluator.evaluate_run() got an unexpected keyword argument evaluator_run_id

Anyone else encountered this issue? How can I fix this parameter mismatch problem?

2

This happens when your LangSmith framework version doesn’t match your evaluator implementation. The framework’s trying to pass evaluator_run_id to your precision_evaluator, but your evaluator class can’t accept it. I ran into this same issue with custom evaluators. Fix it by adding **kwargs to your evaluator’s evaluate_run method signature - something like evaluate_run(self, run, example, **kwargs). This lets the evaluator handle the evaluator_run_id parameter without throwing the TypeError. If you’re using a custom DynamicRunEvaluator subclass, double-check it inherits from the right base class and implements all required methods with the correct signatures.

3

Hit this same bug yesterday! Downgrade langsmith to 0.1.77 if you can’t modify your evaluator code right now. Newer versions expect evaluator_run_id but older evaluators don’t have it in their method signatures. Quick fix until you update your precision_evaluator implementation.

4

Same thing happened to me last month with a SQL evaluation pipeline. LangSmith updated their evaluation runner to pass the evaluator_run_id parameter, but your precision_evaluator can’t handle it.

Check your evaluator class definition. If you’re using a custom DynamicRunEvaluator, your evaluate_run method should look like this:

def evaluate_run(self, run, example, **kwargs):
    # your evaluation logic here
    pass

The **kwargs catches any extra parameters LangSmith sends, including that evaluator_run_id.

Or just add the parameter directly:

def evaluate_run(self, run, example, evaluator_run_id=None):
    # your evaluation logic here
    pass

This video explains how LangSmith tracing works if you want to understand the evaluation flows better:

Fix the method signature and the TypeError goes away. Your SQL query evaluation will work fine after that.

5

I encountered a similar issue recently, and it turned out to be related to the version of the LangSmith package. The error you’re seeing is usually due to an older version of the DynamicRunEvaluator that does not support the evaluator_run_id argument. To resolve this, I recommend upgrading your LangSmith package by running pip install --upgrade langsmith. If the problem persists, consider modifying the evaluate_run method of your precision_evaluator to accept **kwargs, which can handle any additional parameters without causing errors. Also, verify that your precision_evaluator derives from the correct base class, as version mismatches can lead to such issues.

6

I’ve been fighting these LangSmith evaluation issues for months too. That parameter mismatch is such a pain, but I found a way around it.

I ditched LangSmith’s evaluation setup entirely and moved everything to Latenode. Built a workflow that grabs SQL queries, runs them against test data, and checks accuracy automatically.

Setup’s pretty simple - just create nodes for your function calls, add evaluation logic, and wire it all together with error handling. No more signature mismatches or version hell.

For SQL stuff specifically, you can run parallel nodes that hammer through multiple test cases at once. Way more reliable than trying to debug LangSmith’s parameter passing.

Best part is the visual monitoring - you can actually see where things break instead of hunting through stack traces.

Saved me tons of time on dependency nightmares. Worth checking out: https://latenode.com