DynamicRunEvaluator method throws TypeError with unexpected evaluator_run_id parameter

Other Questions langsmith
1

I’m trying to run an evaluation on my dataset to validate some generated database queries but keep hitting this error. The system complains about an unexpected keyword argument called evaluator_run_id when calling the evaluate_run method.

Here’s my setup code:

from langsmith import wrappers, Client
from pydantic import BaseModel, Field
from openai import OpenAI

client = Client()
openai_wrapper = wrappers.wrap_openai(OpenAI())

def process_query(data: dict) -> dict:  
    request_data = {
        "workflowId": "abc123",
        "parameters": {
            "user_question": data['query']
        },
        "action": "runWorkflow",
        "userId": 'test_user'
    }

    api_response = requests.post(api_url, headers=request_headers, json=request_data)
    print(api_response.json()['output']['answer'])
    return api_response.json()['output']['answer']

test_results = client.evaluate(
  process_query,
  data="Test dataset",
  evaluators=[
      quality_checker,
  ],
  max_concurrency=1,
)

The error message I get is:

Error running evaluator <DynamicRunEvaluator quality_checker> on run 3f599063-df1a-58b8-b223-bef50f60e4d4: TypeError("DynamicRunEvaluator.evaluate_run() got an unexpected keyword argument evaluator_run_id")
Traceback (most recent call last):
  File c:\Python313\Lib\site-packages\langsmith\evaluation\_runner.py, line 1634, in _run_evaluators
    evaluator_response = evaluator.evaluate_run(
        run=run,
        example=example,
        evaluator_run_id=evaluator_run_id,
    )
TypeError: DynamicRunEvaluator.evaluate_run() got an unexpected keyword argument evaluator_run_id

I can’t figure out how to properly pass this evaluator_run_id parameter. Any ideas what might be wrong with my implementation?

2

Had this exact issue last month when we upgraded our evaluation pipeline.

Your quality_checker evaluator is using the old method signature.

If you’ve got a custom evaluator class, just add **kwargs to your evaluate_run method:

 def evaluate_run(self, run, example, **kwargs):
     # your existing logic here

This lets it accept the new evaluator_run_id parameter without breaking.

For function-based evaluators, wrap them with @evaluate decorator from langsmith - it handles the parameter changes automatically.

We went with **kwargs since it’s backward compatible and won’t break when they add more parameters down the road.

3

sounds like maybe a version mismatch? the langsmith library might’ve updated, but ur evaluator isn’t set for that new evaluator_run_id arg yet. maybe try upgrading ur evaluator or downgrading langsmith. hope it helps!

4

Just hit this same issue during our langsmith migration. That signature change totally caught us off guard too. Here’s what worked for us - check your langsmith version first. If you’re on 0.1.77 or later, it expects that evaluator_run_id parameter. Quick fix without changing your evaluator code: pin langsmith to 0.1.76 in your requirements. But if you want to stay current, just modify your quality_checker’s evaluate_run method to accept the new parameter. We used evaluator_run_id=None as the default - keeps it compatible across versions. The parameter’s just for internal tracking anyway, so you don’t actually need to use it in your evaluation logic.

5

This happens when your custom evaluator uses an old version of the DynamicRunEvaluator interface. Langsmith recently added an evaluator_run_id parameter to the evaluate_run method, but your quality_checker is still using the old signature. You’ve got two fixes: If it’s a custom class, update the evaluate_run method to include evaluator_run_id as an optional parameter. If you’re using a function-based evaluator, make sure it’s wrapped with the current langsmith decorators - they’ll handle the new parameter automatically.