Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable use of embeddings from Vision Language models #3452

Open
jacopo-chevallard opened this issue Nov 4, 2024 — with Linear · 1 comment
Open

Enable use of embeddings from Vision Language models #3452

jacopo-chevallard opened this issue Nov 4, 2024 — with Linear · 1 comment
Assignees
@jacopo-chevallard
Labels
area: backend Related to backend functionality or under the /backend directory enhancement New feature or request rag: ingestion rag: retrieval

Comments

@jacopo-chevallard Linear
Copy link
Collaborator

jacopo-chevallard commented Nov 4, 2024 •
edited
Loading

Currently, we use text embeddings. This is fine for textual documents, while it present obvious drawbacks for documents containing non-textual content (images, graphs, schemes, …).

An alternative, is to use Visual Language models such as ColPali (see also https://huggingface.co/blog/manu/colpali, https://danielvanstrien.xyz/posts/post-with-code/colpali-qdrant/2024-10-02_using_colpali_with_qdrant.html, https://blog.vespa.ai/retrieval-with-vision-language-models-colpali/, https://blog.vespa.ai/scaling-colpali-to-billions/)

image.png
@jacopo-chevallard jacopo-chevallard added area: backend Related to backend functionality or under the /backend directory rag: ingestion rag: retrieval labels Nov 4, 2024 — with Linear
@jacopo-chevallard jacopo-chevallard self-assigned this Nov 4, 2024
@linear Linear
Copy link

linear bot commented Nov 4, 2024

CORE-280 Enable use of embeddigns Vision Language models

@jacopo-chevallard jacopo-chevallard changed the title Enable use of embeddigns Vision Language models Enable use of embeddings Vision Language models Nov 4, 2024
@jacopo-chevallard jacopo-chevallard changed the title Enable use of embeddings Vision Language models Enable use of embeddings from Vision Language models Nov 4, 2024
@dosubot dosubot bot added the enhancement New feature or request label Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jacopo-chevallard jacopo-chevallard

Labels
area: backend Related to backend functionality or under the /backend directory enhancement New feature or request rag: ingestion rag: retrieval
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jacopo-chevallard