Hey everyone,
I’m working on a project to set up a local RAG system using roughly 2000 JIRA tickets as a data source. The system is intended to help users generate new JIRA tickets by referencing examples of well-structured tickets. Although I’ve already processed ticket details into documents, built a FAISS-based vector store, and implemented a retrieval pipeline, I’m encountering issues with the system fabricating details. I’m relatively new to RAG, so any tips or advice on reducing these inaccuracies would be greatly appreciated!
Thanks a bunch!
As someone who’s implemented a similar system for our dev team, I can relate to the fabrication issue. One thing that worked well for us was incorporating a confidence threshold for retrieved results. Basically, we only used retrieved information if it met a certain similarity score. This helped filter out less relevant matches that the model might try to ‘fill in’ with made-up details.
Another technique we found useful was to use more specific queries during retrieval. Instead of broad searches, we broke down the ticket creation process into distinct steps (e.g., title generation, description formulation, priority setting) and queried the vector store separately for each component. This gave us more precise, relevant results and reduced the likelihood of hallucinations.
Lastly, don’t underestimate the power of prompt engineering. We saw significant improvements by explicitly instructing the model to only use information from the retrieved examples and to indicate when it’s unsure or needs more context. It takes some trial and error, but it’s worth the effort.
hey jackwolf, sounds like ur on the right track! have u tried tweaking ur prompts? sometimes adding explicit instructions to not make up info can help. also, maybe try playing with different similarity metrics in FAISS? cosine similarity might work better for this kinda task. good luck with ur project, dude!
Have you considered implementing a post-processing step to filter out potentially fabricated information? One approach could be to cross-reference generated content against your original dataset, flagging or removing details that don’t appear in the source tickets. Additionally, fine-tuning your language model on your specific JIRA ticket dataset might help it better understand the context and reduce hallucinations. Lastly, experimenting with different chunking strategies for your documents could improve retrieval accuracy and reduce the need for the model to fill in gaps with fabricated information.