I built a document chatbot using Llama2 models with RAG setup. Now I want to measure how well my system performs using evaluation tools like RAGAS. The problem is that RAGAS keeps asking for OpenAI API keys which I don’t want to use.
Is there a way to evaluate my RAG implementation without depending on OpenAI services? I’m looking for alternatives that can work with open source models only.
Has anyone found a workaround for this issue? I would really appreciate any help or suggestions you can share.
Hit this exact problem last month with my RAG pipeline. RAGAS supports way more than just OpenAI now - you can hook it up to local models through HuggingFace Transformers or connect to your Llama2 setup using Ollama or vLLM APIs. Just set up a custom evaluator config and point it to your model instead of OpenAI’s default. I’ve been running Mistral-7B for evaluation and it works pretty well. Same metrics, just slower since local inference can’t match OpenAI’s speed. Don’t go too small on your eval model though - tried some smaller ones first and the quality was trash. The docs are all over the place, but check the GitHub issues for decent examples.