enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) #39180

cqy123456 · 2025-01-12T16:46:32Z

issue: #27678
related: #39753
some raw data status will change:
Intermin index has raw data:

sparse vector	growing segment	sealed segment
BM25	no	no
IP	yes	no

dense vector	growing segment	sealed segment
ivf flat cc	yes	yes
scann_dvr	no	no

mergify · 2025-01-12T16:47:11Z

@cqy123456 Please associate the related pr of master to the body of your Pull Request. (eg. “pr: #”)

mergify · 2025-01-12T16:47:47Z

@cqy123456 Please associate the related issue to the body of your Pull Request. (eg. “issue: #”)

mergify · 2025-01-12T17:34:54Z

@cqy123456 go-sdk check failed, comment rerun go-sdk can trigger the job again.

codecov · 2025-01-12T18:32:01Z

Codecov Report

Attention: Patch coverage is 67.62402% with 124 lines in your changes missing coverage. Please review.

Project coverage is 81.02%. Comparing base (0f4d3ef) to head (7e89f9d).
Report is 2 commits behind head on 2.5.

Files with missing lines	Patch %	Lines
...rnal/core/src/segcore/ChunkedSegmentSealedImpl.cpp	4.08%	47 Missing ⚠️
internal/core/src/segcore/SegmentSealedImpl.cpp	60.65%	24 Missing ⚠️
internal/core/src/segcore/segcore_init_c.cpp	5.00%	19 Missing ⚠️
internal/core/src/segcore/FieldIndexing.cpp	77.33%	17 Missing ⚠️
internal/core/src/storage/storage_c.cpp	0.00%	6 Missing ⚠️
internal/core/src/index/VectorMemIndex.cpp	72.72%	3 Missing ⚠️
internal/querynodev2/server.go	25.00%	2 Missing and 1 partial ⚠️
pkg/util/paramtable/component_param.go	90.32%	2 Missing and 1 partial ⚠️
internal/core/src/segcore/FieldIndexing.h	85.71%	2 Missing ⚠️

❌ Your patch status has failed because the patch coverage (67.62%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##              2.5   #39180      +/-   ##
==========================================
- Coverage   81.06%   81.02%   -0.04%     
==========================================
  Files        1415     1415              
  Lines      198960   199201     +241     
==========================================
+ Hits       161279   161397     +118     
- Misses      32027    32136     +109     
- Partials     5654     5668      +14

Components	Coverage Δ
Client	`78.26% <ø> (ø)`
Core	`69.37% <62.17%> (-0.10%)`	⬇️
Go	`83.02% <91.54%> (-0.02%)`	⬇️

Files with missing lines	Coverage Δ
internal/core/src/index/VectorMemIndex.h	`100.00% <ø> (ø)`
internal/core/src/mmap/ChunkVector.h	`87.09% <ø> (+3.22%)`	⬆️
internal/core/src/query/SearchOnGrowing.cpp	`93.02% <100.00%> (ø)`
internal/core/src/segcore/ConcurrentVector.h	`96.68% <100.00%> (+0.85%)`	⬆️
internal/core/src/segcore/IndexConfigGenerator.cpp	`82.53% <100.00%> (+0.87%)`	⬆️
internal/core/src/segcore/InsertRecord.h	`89.55% <100.00%> (+0.63%)`	⬆️
internal/core/src/segcore/SegcoreConfig.h	`100.00% <100.00%> (ø)`
internal/core/src/segcore/SegmentGrowingImpl.cpp	`82.66% <100.00%> (-0.14%)`	⬇️
internal/core/src/segcore/SegmentGrowingImpl.h	`68.29% <100.00%> (-0.76%)`	⬇️
internal/core/src/segcore/vector_index_c.cpp	`22.22% <ø> (ø)`
... and 13 more

... and 26 files with indirect coverage changes

mergify · 2025-01-13T05:55:35Z

@cqy123456 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2025-01-13T05:56:51Z

@cqy123456 go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2025-01-13T05:56:52Z

@cqy123456 cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2025-01-13T06:16:19Z

@cqy123456 go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2025-01-13T06:16:19Z

@cqy123456 cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2025-01-13T06:17:19Z

@cqy123456 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

cqy123456 · 2025-01-13T07:57:24Z

configs/milvus.yaml

      nprobe: 16 # nprobe to search small index, based on your accuracy requirement, must smaller than nlist
+      subDim: 4 # interim index sub dim, recommend to (subDim % vector dim == 0)
+      refineRatio: 3.5 # interim index parameters, should set to be >= 1.0
+      withRawData: true # Whether to keep raw data inside the intermin index


replace withRawData by denseVectorIndexType

foxspy · 2025-01-13T07:57:02Z

configs/milvus.yaml

      nprobe: 16 # nprobe to search small index, based on your accuracy requirement, must smaller than nlist
+      subDim: 4 # interim index sub dim, recommend to (subDim % vector dim == 0)
+      refineRatio: 3.5 # interim index parameters, should set to be >= 1.0
+      withRawData: true # Whether to keep raw data inside the intermin index


This should be an index feature, not a configuration item.

updated with a index type as config

foxspy · 2025-01-13T08:21:51Z

internal/core/src/segcore/SegmentGrowingImpl.cpp

@@ -102,7 +103,8 @@ SegmentGrowingImpl::Insert(int64_t reserved_offset,
        AssertInfo(field_id_to_offset.count(field_id),
                   fmt::format("can't find field {}", field_id.get()));
        auto data_offset = field_id_to_offset[field_id];
-        if (!indexing_record_.SyncDataWithIndex(field_id)) {
+        if (!indexing_record_.SyncDataWithIndex(field_id) ||
+            !indexing_record_.HasRawData(field_id)) {


This may seem difficult to understand; please change it to insert_raw_to_index = indexing_record_.SyncDataWithIndex(field_id) && indexing_record_.HasRawData(field_id); if (!insert_raw_to_index) xxx , which means that only the index holds the original data and the index data has been matched with the inserted data, and no more data will be written to the chunk.

replace indexing_record_.SyncDataWithIndex(field_id) && indexing_record_.HasRawData(field_id) with indexing_record_.HasRawData(field_id), because indexing_record_.HasRawData(field_id) include indexing_record_.SyncDataWithIndex(field_id).

mergify · 2025-01-13T09:14:49Z

@cqy123456 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

cqy123456 · 2025-01-13T09:18:56Z

/run-cpu-e2e

foxspy · 2025-02-11T02:36:30Z

rerun go-sdk

foxspy · 2025-02-11T02:59:29Z

/run-cpu-e2e

mergify · 2025-02-11T05:22:20Z

@cqy123456 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

foxspy · 2025-02-11T06:14:30Z

/run-cpu-e2e

internal/core/src/segcore/ConcurrentVector.h

mergify · 2025-02-11T08:01:55Z

@cqy123456 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

czs007 · 2025-02-11T08:17:51Z

internal/core/src/segcore/ChunkedSegmentSealedImpl.cpp

@@ -1760,11 +1757,9 @@ ChunkedSegmentSealedImpl::bulk_subscript(FieldId field_id,
            return get_raw_data(field_id, field_meta, seg_offsets, count);
        }
        return get_vector(field_id, seg_offsets, count);
+    } else {
+        return fill_with_empty(field_id, count);


why not panic?

czs007 · 2025-02-11T08:23:18Z

internal/core/src/segcore/ChunkedSegmentSealedImpl.cpp

@@ -2091,19 +2129,26 @@ ChunkedSegmentSealedImpl::generate_interim_index(const FieldId field_id) {

        if (enable_binlog_index()) {
            std::unique_lock lck(mutex_);
+            if (vec_index->HasRawData()) {
+                // some knowhere view data index not has raw data, still keep it


comment in wrong place.

internal/core/thirdparty/knowhere/CMakeLists.txt

foxspy

/lgtm

mergify · 2025-02-12T07:29:11Z

@cqy123456 go-sdk check failed, comment rerun go-sdk can trigger the job again.

…e fp16/bf16 Signed-off-by: cqy123456 <[email protected]>

foxspy

/lgtm

czs007 · 2025-02-13T01:56:03Z

/approve
/lgtm

sre-ci-robot · 2025-02-13T01:56:17Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cqy123456, czs007, foxspy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [czs007]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sre-ci-robot requested review from aoiasd and bigsheeper January 12, 2025 16:46

cqy123456 changed the title ~~enhance: intermin index support index without raw data and metric type fp16/bf16~~ enhance: [2.5]intermin index support index without raw data and metric type fp16/bf16 Jan 12, 2025

sre-ci-robot added area/compilation size/XL Denotes a PR that changes 500-999 lines. labels Jan 12, 2025

mergify bot added the dco-passed DCO check passed. label Jan 12, 2025

mergify bot added do-not-merge/missing-related-pr kind/enhancement Issues or changes related to enhancement labels Jan 12, 2025

mergify bot added the do-not-merge/missing-related-issue label Jan 12, 2025

cqy123456 force-pushed the new-intermin-index branch from 189ec69 to 476fb9c Compare January 13, 2025 05:50

cqy123456 changed the title ~~enhance: [2.5]intermin index support index without raw data and metric type fp16/bf16~~ enhance: [2.5]intermin index support different index type and more metric type(fp16/bf16) Jan 13, 2025

cqy123456 force-pushed the new-intermin-index branch from 476fb9c to eea69ac Compare January 13, 2025 06:10

cqy123456 commented Jan 13, 2025

View reviewed changes

cqy123456 force-pushed the new-intermin-index branch from eea69ac to 3d2a8d5 Compare January 13, 2025 08:02

cqy123456 changed the title ~~enhance: [2.5]intermin index support different index type and more metric type(fp16/bf16)~~ enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) Jan 13, 2025

foxspy reviewed Jan 13, 2025

View reviewed changes

cqy123456 force-pushed the new-intermin-index branch from 3d2a8d5 to 451f7bc Compare January 15, 2025 18:32

sre-ci-robot removed the size/XL Denotes a PR that changes 500-999 lines. label Jan 15, 2025

yanliang567 added this to the 2.5.5 milestone Feb 11, 2025

czs007 reviewed Feb 11, 2025

View reviewed changes

internal/core/src/segcore/ConcurrentVector.h Show resolved Hide resolved

czs007 reviewed Feb 11, 2025

View reviewed changes

internal/core/thirdparty/knowhere/CMakeLists.txt Outdated Show resolved Hide resolved

mergify bot removed do-not-merge/missing-related-issue do-not-merge/missing-related-pr labels Feb 11, 2025

cqy123456 force-pushed the new-intermin-index branch 2 times, most recently from 9b0ac56 to de7d75a Compare February 12, 2025 06:28

foxspy approved these changes Feb 12, 2025

View reviewed changes

sre-ci-robot assigned foxspy Feb 12, 2025

sre-ci-robot added the lgtm label Feb 12, 2025

enhance: intermin index support index without raw data and metric typ…

7e89f9d

…e fp16/bf16 Signed-off-by: cqy123456 <[email protected]>

cqy123456 force-pushed the new-intermin-index branch from de7d75a to 7e89f9d Compare February 12, 2025 08:42

sre-ci-robot removed the lgtm label Feb 12, 2025

foxspy approved these changes Feb 12, 2025

View reviewed changes

sre-ci-robot added the lgtm label Feb 12, 2025

mergify bot added the ci-passed label Feb 12, 2025

sre-ci-robot assigned czs007 Feb 13, 2025

sre-ci-robot added the approved label Feb 13, 2025

sre-ci-robot merged commit d121ac3 into milvus-io:2.5 Feb 13, 2025
17 of 20 checks passed

cqy123456 mentioned this pull request Feb 13, 2025

[Feature]: Need to support interim index for bf16,fp16 and binary vector #31158

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) #39180

enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) #39180

cqy123456 commented Jan 12, 2025 •

edited

Loading

mergify bot commented Jan 12, 2025

mergify bot commented Jan 12, 2025

mergify bot commented Jan 12, 2025

codecov bot commented Jan 12, 2025 •

edited

Loading

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

cqy123456 Jan 13, 2025

cqy123456 Feb 11, 2025

foxspy Jan 13, 2025

cqy123456 Feb 11, 2025

foxspy Jan 13, 2025

cqy123456 Feb 11, 2025

mergify bot commented Jan 13, 2025

cqy123456 commented Jan 13, 2025

foxspy commented Feb 11, 2025

foxspy commented Feb 11, 2025

mergify bot commented Feb 11, 2025

foxspy commented Feb 11, 2025

mergify bot commented Feb 11, 2025

czs007 Feb 11, 2025

cqy123456 Feb 11, 2025

czs007 Feb 11, 2025

cqy123456 Feb 12, 2025

foxspy left a comment

mergify bot commented Feb 12, 2025

foxspy left a comment

czs007 commented Feb 13, 2025

sre-ci-robot commented Feb 13, 2025

enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) #39180

enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) #39180

Conversation

cqy123456 commented Jan 12, 2025 • edited Loading

mergify bot commented Jan 12, 2025

mergify bot commented Jan 12, 2025

mergify bot commented Jan 12, 2025

codecov bot commented Jan 12, 2025 • edited Loading

Codecov Report

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mergify bot commented Jan 13, 2025

cqy123456 commented Jan 13, 2025

foxspy commented Feb 11, 2025

foxspy commented Feb 11, 2025

mergify bot commented Feb 11, 2025

foxspy commented Feb 11, 2025

mergify bot commented Feb 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

foxspy left a comment

Choose a reason for hiding this comment

mergify bot commented Feb 12, 2025

foxspy left a comment

Choose a reason for hiding this comment

czs007 commented Feb 13, 2025

sre-ci-robot commented Feb 13, 2025

cqy123456 commented Jan 12, 2025 •

edited

Loading

codecov bot commented Jan 12, 2025 •

edited

Loading