Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apply NEF and FFF for RAG #52

Open
MrCsabaToth opened this issue Sep 17, 2024 · 0 comments
Open

Apply NEF and FFF for RAG #52

MrCsabaToth opened this issue Sep 17, 2024 · 0 comments
Labels
enhancement New feature or request RAG Retrieval Augmented Generation related

Comments

@MrCsabaToth
Copy link
Member

Named Entity Filtering (https://blog.cubed.run/eliminating-hallucinations-lesson-1-named-entity-filtering-nef-5f5956d748e0) and Fully Formatted Facts (https://medium.com/@JamesStakelum/the-end-of-ai-hallucinations-a-breakthrough-in-accuracy-for-data-engineers-e67be5cc742a) along Noun-Phrase Dominance supposed to greatly decrease hallucinations and increase RAG performance (in terms of preciseness and correctness).

I've seen even earlier BERT-based and other few models which were able to generate tags for chunks automatically. These tags should be stored as meta-data for the RAG and should be utilized for filtering to avoid problems depicted in the above articles.

We should also tune the resolver prompt we use for conversation piece embedding preparation to include instructions about "avoid any ambiguity" besides resolving / unfolding contextual references.

@MrCsabaToth MrCsabaToth added enhancement New feature or request RAG Retrieval Augmented Generation related labels Sep 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request RAG Retrieval Augmented Generation related
Projects
None yet
Development

No branches or pull requests

1 participant