This is a problem I didn’t expect to have. I thought having access to more AI models would be straightforward—pick the best one, use it. But having actually started building workflows with multiple model options, the decision paralysis is real.
I’m working on a data extraction and classification pipeline. For extraction, I’m considering Claude for its accuracy on structured data, but OpenAI’s models are faster. For classification, smaller models like Mistral could work fine and run way cheaper. But what if the specific classification problem actually needs reasoning from a larger model?
I’ve been asking myself questions like: does this task need general reasoning or specialized domain knowledge? How much speed versus accuracy matters here? What’s the cost difference at scale? But honestly, I’m mostly guessing based on vibes and some quick testing.
I feel like there should be a more systematic way to think about this rather than trial-and-error. How are you folks handling model selection in your workflows? Are you going deep with one model that handles most things, or are you mixing and matching based on the task?
This is one of those situations where having the flexibility actually requires being intentional about choices. I approached it systematically at work.
Start with your most common task type. Test three models on that task. Measure accuracy and cost. Pick the winner for that workflow.
Move to your next task type. Repeat.
What I found is that once you’ve set a model for a specific type of task, you rarely need to change it unless requirements shift. You’re not picking one model for everything. You’re building a set of assignments: GPT-4 for anything requiring complex reasoning, Claude for structured data extraction, a smaller model for simple classifications.
The 400+ models are there for optionality, not for you to use all of them. It’s more about having the right tool available when you need it rather than evaluating all options every time.
I treated it like algorithm selection. For each distinct task in your workflow, ask: what’s the minimum capability needed for this task? Then pick the cheapest model that meets that capability.
For extraction with defined output formats, Claude works great and is efficient. For fuzzy classification or open-ended analysis, you probably need GPT-4’s reasoning. For simple transformations or basic categorization, smaller models work fine.
I started by running test batches through different models and comparing cost per successful task. That gave me concrete data instead of opinions. After about a week of testing, the decisions became obvious.
The practical approach is to group similar tasks together and solve the model selection problem once per group. Don’t overthink individual calls. Profile a few different models against a representative sample from each task category, measure accuracy and cost, document the decision, and move forward. You’re looking for sufficiently good models, not perfectly optimal ones every time.
Start simple: GPT-4 for complex reasoning, Claude for structured extraction, small models for basic tasks. Optimize later if costs justify it. Most workflows dont need more than 3-4 models.