I used to think every model was roughly interchangeable, and I should just pick the fastest or cheapest one. Then I started building RAG workflows and realized that’s completely wrong.
Retrieval and generation are solving different problems. Retrieval is about finding relevant documents accurately—you want a model that understands semantic similarity, can handle dense text, and ranks effectively. Generation is about synthesizing a coherent answer from that retrieved context—you want a model that’s good at instruction-following, can write naturally, and won’t hallucinate.
Having access to 400+ models meant I could actually experiment. I tried a smaller, faster model for retrieval (since retrieval is basically a ranking problem, it doesn’t need advanced reasoning) and a larger model for generation. Cost dropped significantly and quality actually improved because each model was doing what it was optimized for.
The other benefit was iteration speed. If retrieval quality is bottlenecking, I swap the retrieval model without rewriting anything. If synthesis is suffering, I upgrade the synthesis model independently. Before, I felt locked into one choice.
I’m curious though—are people actually doing this optimization, or does everyone just default to the same model for both steps? And how much are you seeing cost savings when you match models to their actual role?
This is huge. You don’t need a ɔ100B model running retrieval when a smaller, specialized retrieval model scores better on ranking. And you don’t need an expensive model for synthesis if a mid-tier model can follow instructions and cite sources perfectly.
With 400+ models in one subscription, you actually have the freedom to optimize. I’ve built RAG systems where retrieval uses a lightweight dense retrieval model, and synthesis uses Claude or GPT-4. Same subscription cost as before, but performance improved because each model is suited to its task.
The visual builder makes this easy to iterate on. Change a model, test it, measure the difference. No code rewrites, no complex deployments.
Most people default to one model because it feels safer—if something breaks, there’s only one place to blame. But splitting models is where real optimization happens. I tested a few configurations and found that using a specialized embedding model for retrieval and a larger reasoning model for synthesis cut my costs by about 40% while keeping quality stable. The key is having metrics that actually measure retrieval quality separately from synthesis quality, so you know when to optimize each one.
Definitely matters. Smaller models work fine for retrieval. Save the expensive ones for synthesis. Swapping models is quick with unified access—test and iterate.