[Feat] Add support of logical merge in Cagra

[rhdong]

• [FEA] Add support of logical merge in Cagra #712

[rhdong]

Hi @cjnolet @achirkin , I had to add a new search for composite_index, the root cause is the cuve::neighbors::index is not designed to be an abstract virtual class, which makes many pathways impossible, like inheriting from index, which can avoid declaring a standalone composible_index and a new search API. I was trying to refactor it, but I found it was used widely in different algo codes than give up under a tight timeline; maybe we can make it in the long term.

[cjnolet]

I think it's okay that we don't require a strict class hierarchy initially, so long as we have proper overloads so that from a user's perspective, they can call auto index = cuvs::neighbors::cagra::merge(...) and get back a proper object from which they can later call cuvs::neighbors::cagra::search(..., index,..)

[achirkin]

I may have missed some prior discussion, but the cagra-specific logical merge seems a little bit artificial for me.
If I understand this correctly, you essentially implement a multi-index search. Then, why don't we go one step further and make it independent of a particular index implementation? Moreover, it looks like much of this functionality is implemented already in multi-gpu index. Maybe we can generalize that one a bit to decouple "multi-index" from "multi-gpu" aspect? Optionally, one can also combine different index types and erase the upstream index type like we do in dynamic batching index.

[cjnolet]

I may have missed some prior discussion, but the cagra-specific logical merge seems a little bit artificial for me.

@achirkin you and I haven't discussed this yet but this feature is critical for certain database architectures like Lucene, which are variants of the log-structured-merge pattern but build indexes immediately after segments (flat files containing vectors) are created, rather than merging the segments together before building the indexes. This causes Lucene to have to first merge many tiny CAGRA indexes together, but eventually it'll end up merging very large indexes together. It is this latter case thart we care about doing a logical merge. It's more efficient to create a single cagra index with many (potentially hundreds) of tiny cagra indexes but when they reach a certain size, it's actually more efficient to logically merge them; meaning we essentially wrap them as if they were shards and broadcast the query to all the shards during search.

I agree this is similar in theory to the multi-gpu sharding and perhaps there is some code to be reused there. The next step for the merge() API is to be able to implement a SMART merge strategy, whereby we can more efficiently merge two cagra graphs together without having to rebuild from scratch, and that's ultimately the strategy we have discussed offline in more detail. The problem is that Lucene can make use of this today, so it gives us time to work towards the SMART option.

Then, why don't we go one step further and make it independent of a particular index implementation?

I do agree with this- it doesn't have to be (and ideally wouldn't be) specifically for CAGRA, though that just happens to be the index we care about today because it unblocks Lucene.

[achirkin]

I totally agree and don't doubt the usefulness of logical merge.
I've just pointed out that I think we can accomplish a more general solution (supporting any index type) with actually less code by re-using/adjusting what we already have in multi-index/dynamic batching.

[rhdong]

I may have missed some prior discussion, but the cagra-specific logical merge seems a little bit artificial for me. If I understand this correctly, you essentially implement a multi-index search. Then, why don't we go one step further and make it independent of a particular index implementation? Moreover, it looks like much of this functionality is implemented already in multi-gpu index. Maybe we can generalize that one a bit to decouple "multi-index" from "multi-gpu" aspect? Optionally, one can also combine different index types and erase the upstream index type like we do in dynamic batching index.

Hi @achirkin , sorry for the late response to your comments here. I studied the code of SNMG, which is a super useful feature. When I tried to reuse some code in it, I found the caller has to work with locker, nccl, which is unnecessary in the logical merge of CAGRA. I'd like to keep the simple implementation if you agree. Many thanks! cc: @cjnolet

[rhdong]

Hi @cjnolet @achirkin, could you please merge it if there are no more comments? Many thanks!