LlamaIndex OpenAIAgent always picks first tool instead of choosing appropriate one from multiple Zapier tools

Other Questions Zapier
1

I’m working with llama-index, OpenAI, and Zapier NLA to build an agent that can handle different tasks. The problem is my OpenAIAgent keeps using only the first tool from my tool collection and completely ignores the rest.

Steps I’ve taken:

  • Created a custom ZapierToolSpec class to handle function calls differently
  • Confirmed that my_zapier_spec.to_tool_list() gives back several tools
  • Used the tool collection with OpenAIAgent.from_tools() but it won’t consider other options
import dotenv
import os
from llama_hub.tools.zapier.base import ZapierToolSpec
from llama_index.agent.openai import OpenAIAgent
from llama_index.llms.openai import OpenAI
from llama_index.core.tools.function_tool import FunctionTool

dotenv.load_dotenv()

OPENAI_KEY = os.getenv("OPENAI_API_KEY")
ZAPIER_KEY = os.getenv("ZAPIER_NLA_API_KEY")

model = OpenAI(model="gpt-4o", api_key=OPENAI_KEY)

class MyZapierToolSpec(ZapierToolSpec):
    def to_tool_list(self):
        base_tools = super().to_tool_list()
        updated_tools = []
        
        for item in base_tools:
            base_function = item.fn

            def updated_function(*params, **keyword_params):
                if "keyword_params" in keyword_params and isinstance(keyword_params["keyword_params"], dict):
                    keyword_params = keyword_params["keyword_params"]
                return base_function(**keyword_params)

            updated_tools.append(FunctionTool(fn=updated_function, metadata=item.metadata))

        return updated_tools

my_zapier_spec = MyZapierToolSpec(api_key=ZAPIER_KEY)
tool_collection = my_zapier_spec.to_tool_list()

print(f"Available tools: {tool_collection}")

my_agent = OpenAIAgent.from_tools(tools=tool_collection, verbose=True, llm=model)

user_request = "Schedule a calendar appointment called 'Team Meeting' for next day at 3 PM."

print(my_agent.chat(user_request))

What should happen: The agent picks the right tool based on what I’m asking for.

What actually happens: It always goes with the first tool in the list, no matter if another one fits better.

2

This happens when your tool descriptions or metadata aren’t clear enough for the OpenAI model. I’ve hit the same issue with multiple API integrations. The model picks tools based on function descriptions and metadata, so if they’re too similar, it just grabs the first one. You’re keeping the original metadata in your custom MyZapierToolSpec class, but try beefing it up instead. Check what’s actually in item.metadata for each tool and add more specific descriptions that clearly show what each tool does. Also verify that ZapierToolSpec is creating unique function names - generic names mess up the selection process. One more thing: make sure your Zapier NLA setup actually has different actions configured. Sometimes the API returns multiple “tools” that are just variations of the same action, which would explain why it keeps picking the first one.

3

I had the same frustrating issue. It’s usually how OpenAI picks which tool to use. Your function wrapper modification is probably the culprit - when you restructure parameters with that updated_function, you’re likely stripping out metadata the model needs to tell tools apart. Debug by printing the exact metadata and function signatures for each tool before you create the agent. My Zapier tools had basically identical descriptions even though they did completely different things. I fixed it by manually overriding the metadata descriptions to be super specific - “Creates calendar events in Google Calendar” vs “Sends email notifications via Gmail”. Also check your Zapier NLA actions have distinct operation IDs. They sometimes get generic names that mess with the selection algorithm.

4

First, make sure your gpt-4o model is actually receiving the tool descriptions correctly. The problem might not be your tools - it could be how OpenAI’s processing them. Try adding a system message like “analyze all available tools before selecting one.” This pushes the agent to actually think through its options instead of just grabbing the first thing it sees.