DynamicRunEvaluator.evaluate_run() receives unsupported parameter 'evaluator_run_id' causing TypeError

Other Questions langsmith
1

I’m having trouble when trying to run an evaluation on my dataset. I want to check if my generated SQL queries are correct by comparing them with the ground truth data.

The problem is that I keep getting a TypeError saying that the evaluate_run method doesn’t accept the evaluator_run_id parameter. I’m not sure how to fix this since I’m not manually passing this parameter anywhere.

Here’s my code setup:

from langsmith import wrappers, Client
from pydantic import BaseModel, Field
from openai import OpenAI

langsmith_client = Client()
openai_wrapper = wrappers.wrap_openai(OpenAI())

def my_function(data_inputs: dict) -> dict:  
    request_data = {
        "processId": "yyy",
        "params": {
            "user_question": data_inputs['query']
        },
        "action": "executeProcess",
        "userId": 'yyy'
    }

    api_response = requests.post(api_url, headers=request_headers, json=request_data)
    print(api_response.json()['response']['output'])
    return api_response.json()['response']['output']

test_results = langsmith_client.evaluate(
  my_function,
  data="Test dataset",
  evaluators=[
      precision_checker,
  ],
  max_concurrency=2,
)

The error I’m getting looks like this:

Error running evaluator <DynamicRunEvaluator precision_checker> on run 3f599063-df1g-58b8-b223-bec50g50edf4: TypeError("DynamicRunEvaluator.evaluate_run() got an unexpected keyword argument evaluator_run_id")
Traceback (most recent call last):
  File c:\Python313\Lib\site-packages\langsmith\evaluation\_runner.py, line 1634, in _run_evaluators
    evaluator_response = evaluator.evaluate_run(  # type: ignore[call-arg]
        run=run,
        example=example,
        evaluator_run_id=evaluator_run_id,
    )
TypeError: DynamicRunEvaluator.evaluate_run() got an unexpected keyword argument evaluator_run_id

Has anyone encountered this issue before? How can I resolve this parameter mismatch?

2

Check if you’re importing precision_checker from an older library or custom code that hasn’t been updated for the latest langsmith API changes. I’ve seen this when people copy evaluator code from outdated tutorials or use evaluators from packages that aren’t maintained anymore. Langsmith recently added the evaluator_run_id parameter for better tracking, but legacy evaluators break when they see it. If you’re using a third-party evaluator, try updating that package first. Otherwise, you’ll need to modify your precision_checker class to inherit from the current DynamicRunEvaluator interface. Make sure your evaluate_run method signature matches what langsmith expects - adding **kwargs is a quick fix to handle unexpected parameters.

3

Quick fix that worked for me - wrap your evaluator in try/except or make a compatibility layer. Just catch the TypeError and call the method without evaluator_run_id. Works fine until you update everything properly. Also check pip show langsmith for your version - mixed versions cause weird parameter conflicts like this.

4

This is a version compatibility issue with langsmith. I ran into the same problem a while back with custom evaluators. The evaluator_run_id parameter was introduced in the newer versions of langsmith, while your precision_checker evaluator presumably still relies on the older interface that does not accommodate this change. A straightforward solution would be to update your evaluator to accept the new parameter. If you’re using a custom DynamicRunEvaluator, you can add evaluator_run_id=None to the method signature of evaluate_run even if you don’t utilize it. Alternatively, downgrading langsmith to align with your evaluator is an option, but typically, updating is the better route. It would be beneficial to verify if there is an updated version of your precision_checker that supports the current langsmith API.

5

Had this exact headache last month building automated SQL validation pipelines. Manual patching of evaluators gets messy fast, especially across multiple environments.

We switched everything to Latenode and it solved the problem. Set up the whole workflow there - data ingestion, API calls to your SQL service, ground truth comparison, and scoring. All automated.

Latenode handles parameter passing and version compatibility automatically. Just connect your data source, set up API endpoints, and define evaluation logic visually. No more wrestling with langsmith version mismatches or modifying evaluator signatures every update.

Automatic retry logic and error handling is huge when running evaluations on large datasets. We went from hours debugging parameters to rock solid pipelines that just work.