feat(Qdrant): allow payload indexing + on disk vectors

[lambda-science]

This is still a draft:

• I need to modify tests => Index two documents with two differents payload value. Check that user 1 can't access user 2 doc ? Maybe other test ?

Related doc:

• Payload Index https://qdrant.tech/documentation/concepts/indexing/#payload-index
• Multi Tenancy https://qdrant.tech/documentation/guides/multiple-partitions/

This MR aims to allow Qdrant user to create index on payload (metadata in Haystack naming) field.
This allow to filter way more efficiently vectors when searching.
And also it allows to restricti users to their own vectors only.

I'm open to comments, I'm not sure running it at each start-up of the collection is a good way to do it.
But From Qdrant discord:

andrey.vasnetsov —
overall, we recommed creating payload index before you upload data.
AFAIK, if you try to create same index twice, it won't trigger indexing

https://discord.com/channels/907569970500743200/1214958671742902272/1214958671742902272 (if you are on their discord)

Adtionnally this add also the parameter on_disk that was not included in the integration, to store vectors on disk. See https://qdrant.tech/documentation/concepts/storage/#configuring-memmap-storage

[lambda-science]

I guess the PR is ready.
For adding test specific for the function I'm adding I'm not sure how to do, as other function were not that much tested also ? hm

Should we ping Qdrant people ? Because I read somewhere that they are the one maintaining this ? https://qdrant.tech/documentation/frameworks/haystack/

[davidsbatista]

Hi @lambda-science

It was someone from Qdrant who originally wrote the code, so let's ask them to review it.

[davidsbatista]

@Anush008 could you review this PR?

[Anush008]

Great! Thank you🙏