ROADMAP 2024

[writinwaters]

v0.6.0 planning

Core

• Unify the memory management of index and data
• Use memory usage to decide when flush the memory index
• Switch for near-real-time and real-time index
• Performance improvement of 'long text'
• Supports DiskANN index. [Feature Request]: DiskAnn unify #1953
• Supports system level data backup and recovery
• Cluster fail over

Tools

• GUI [Feature Request]: Added support for GUI #1841

v0.5.0

Core:

• Supports Product quantization. [Feature Request]: Support product quantization for IVF index #2077
• Supports Scalar quantization (u8). [Feature Request]: Support scalar quantization for IVF index #2085
• Supports Binary vector with Hamming distance.[Feature Request]: Supports Binary vector with Hamming distance. #2069
• Supports Cluster management, log replication
• Supports result caching and paginate function [Feature Request]: Search result caching and paginate funtion. #1903
• Supports regular expression on varchar field. [Feature Request]: Support regular expression in filter and project list. #1986
• Support providing comment when create database / index / table Add comment for database creation #2050 Add comment on table creation #2052
• Support boolean similarity for fulltext search [Feature Request]: Support keyword analyzer and boolean similarity for fulltext search #2139
• Support threshold for BM25, L2/IP/Cosine metrics for full-text/dense vector/sparse vector/tensor matching [Feature Request]: Support threshold for BM25, L2/IP/Cosine metrics for full-text/dense vector/sparse vector/tensor matching #2115
• Supports Aarch64.[Feature Request]: Building for ARM64 CPUs #1084

Integration

• Integrates with RAGFlow [Feature Request]: Integrates Infinity with RAGFlow #1405

Tools

• GUI [Feature Request]: Added support for GUI #1841

v0.4.0

Core:

• Enable IVF Index. [Feature Request]: Add support for IVF index #1917
• Supports filter expression to match_dense/match_sparse/match_text/fusion respectively. [Feature Request]: add filter to match functions #1803
• Supports using full text as filter. [Feature Request]: add filter to match functions #1803
• Supports MinShouldMatch as a full text filter. [Feature Request]: Support MinShouldMatch as a full text filter #1862
• Supports date and time type. Add a type supporting dates. #1804 fix parser/type/date_type_std.cpp : check ymd.ok() after adding years to it #1824
• Supports IN operator in filter expression. [Feature Request]: Supports IN operator in filter expression. #1839
• Supports lock/unlock table to prevent manipulating. [Feature Request]: Lock table. #1813
• Add/Remove column by locking table.
• Supports Korean for full text search. [Feature Request]: Add Korean morphological analyzer to support Korean for full text search #1228
• Supports highlighter on full text search [Feature Request]: Support highlighter on full text search #1861
• Suports to add column comment when creating table. [Feature] Column comment #2038
• Supports analyzer from RAGFlow [Feature Request]: Implement analyzer import from RAGFlow #1973

Integration

• Supports S3 storage. [Feature Request]: Supports S3 storage #1809
• Refactor file IO part to integrates S3, NFS, and local filesystem.

API

• Create an specific embedded Infinity python module. [Feature Request]: Create an specific embedded Infinity python module #1786
• Support orderby / sort function. Add sort/orderby for infinity-sdk and infinity-embedded-sdk #1944

Tools

• GUI, list databases / tables / show variables / show configs [Feature Request]: Added support for GUI #1841

v0.3.0

Core:

• Virtual file system. [Feature Request]: Virtual file system #1184
• Memory optimization of sparse index building. Fix optimize index #1436
• Support unordered sparse embedding index when importing data. Fix: sort sparse expr in import and match statement #1419
• Unify SIMD operations together. Support simd functions dispatch at runtime #1473
• Supports bf16 embedding data type. Support float16 and bfloat16 embedding data type #1579
• Supports f16 embedding data type. Support float16 and bfloat16 embedding data type #1579
• Supports int8 embedding data type. Support int8 embedding and uint8 embedding #1527
• Supports multiple vectors on one document. [Feature Request]: Support multiple vectors on one document. #1679
• Smart full-text query syntax. Add "operator" option for fulltext query #1622
• Use full-checkpoint, export and import parquet file to support data backup and restore.
• Supports export parquet files. [Feature Request]: Supports importing data from parquet file. #1330
• Supports import parquet files. [Feature Request]: Supports importing data from parquet file. #1330

v0.2.0

• Supports sparse vector index. [Feature Request]: Support sparse vector #1174
• Supports tensor data type. [Feature Request]: Supports tensor data type #1179
• Supports cosine similarity. [Feature Request]: Supports cosine similarity #1176
• Supports configurable reciprocal rank fusion operator. [Feature Request]: Supports configurable reciprocal rank fusion operator #1177
• Multiple recall supports more than two ways. [Feature Request]: Multiple recall supports more than two ways. #1178
• HTTP API: Supports GET/SET variables. [Feature Request]: HTTP API: Supports GET/SET variables. #1180
• Embedded infinity. [Feature Request]: Embedded infinity #1181
• Export data into CSV and JSONL file type. [Feature Request]: Export data into CSV and JSONL file type #1175
• Unify the background computation task running into task executor. [Feature Request]: Unify the background computation task running into task executor. #1182
• Integrates later interaction models, such as Colbert. [Feature Request]: Run ColBERT reranker as a example for tensor type data #1279
• Support building secondary index on string columns. [Feature Request]: Support string for secondary index #1235
• Support Japanese for full text search. [Feature Request]: Integrate Japanese morphological analyzer into Infinity #1137
• Support traditional Chinese for full text search. [Feature Request]: Support traditional Chinese #1376
• Support near by query for full text search. [Feature Request]: Support near by search for full text retrieval #1346

v0.1.0

• Building HNSW index in parallel. [Feature Request]: Add parallel construction of knn index. #341
• Supports aggregate operation. [Feature Request]: Supports aggregate operation. #357
• Supports order by (sort) operation. [Feature Request]: Support Order By #339
• Supports limit operation. [Feature Request]: Support Limit #362
• Supports order by + limit as top operation. [Feature Request]: Support order by + limit as top operation #408
• Secondary index on structured data type. [Feature Request]: Secondary index #360
• New full text search. [Feature Request]: New full text index #358
• Minmax of column data. Minmax of column data. #448
• Bloomfilter of structured data column. [Feature Request]: Add bloomfilter to segment/block column. #467
• Refactor ColumnVector: Reduce serialization times as much as possible. [Feature Request]: Refactor ColumnVector #449
• Supports new data type: date. [Feature Request]: Support DATE data type #371
• Supports new data type: bool. [Feature Request]: Support BOOL data type #394
• Refactor meta data: Provides a clear interface to access meta data, instead of traversing meta data tree. [Refactor]: Refactor catalog #368
• Refactor error handling: Provides normalized error code and error message. [Feature Request]: Unified error message and error code #439
• Segment GC and segment compaction. [Feature Request]: Segment compaction #466
• Refactor WAL with physical log, instead of logical log. [Feature Request]: WAL physical log #431
• Asynchronous index building: Data become query-able once imported / inserted.
• Storage clean up: Deprecated index/segment/catalog ... files need to be clean up to save the disk space. [Feature Request]: Clean up #635
• Incremental checkpoint. [Feature Request]: Incremental checkpoint #438
• New python API to show database system value. [Feature Request]: Show Variable #495
• New python API to explain the query plan. [Feature Request]: Add Explain api for infinity_sdk #496
• HTTP API [Feature Request]: HTTP API support #779

Backlog

Core

• Native supports MacOS(m1) and Windows
• Supports authentication with default roles.
• Use KV store as the meta data store.

Integration

• Supports NFS.
• Integrates with Langchain
• Integrates with llamdindex
• Embedding function.

Tools

• Infinity database backup and restore tools. [Feature Request]: Infinity database backup and restore function. #1183
• Monitoring tools.
• Data migration tool.

[yuzhichang]

CI improvement: post logs of infinity when CI failure, use Ubuntu 20.04 as base of dev image.
Fuzz test of infinity.

[cjkbjhb]

Secordary index on structured data type.
--->
Secondary index on structured data types.

Here is a mis-spelling error.

[JinHai-CN]

• Secondary

Fixed and thank you.

[yuzhichang]

compatibility testing

image	tag	refer
centos	7 8	https://hub.docker.com/_/centos/
ubuntu	20.04 22.04 24.04	https://hub.docker.com/_/ubuntu https://releases.ubuntu.com/
debian	8 9 10 11 12	https://hub.docker.com/_/debian https://www.debian.org/releases/
opensuse/leap	15.0 15.1 15.2 15.3 15.4 15.5	https://hub.docker.com/r/opensuse/leap
openeuler/openeuler	20.03   22.03	https://hub.docker.com/r/openeuler/openeuler
openanolis/anolisos	8.6 23	https://hub.docker.com/r/openanolis/anolisos
openkylin/openkylin	1.0	https://hub.docker.com/r/openkylin/openkylin

[Kelvinyu1117]

I would like to contribute to this project, which issue would be a good start?

[JinHai-CN]

@Kelvinyu1117
We do have a couple of issues that might work for contributors new to this project.

1. Add minmax information to blocks/segments in the current datastore. This information is primarily used for data filtering. (Minmax of column data. #448)
2. Implement a bloomfilter for the blocks/segments to enhance point queries. ([Feature Request]: Add bloomfilter to segment/block column. #467)
3. Currently, query results are stored in memory in a columnar format. However, the client expects the results in Apache Arrow format. At the moment, the format conversion is executed on the Python client, but this worsens the performance, so we plan to convert the results to Apache Arrow format on the server side before sending them to the client.
4. There are several optimizer rules to implement, such as constant folding and simplification of arithmetic expressions, which are not yet on the roadmap. Feel free to work on them if interested.
5. We have additional complicated tasks not listed here. For instance, the current executor operates with one thread per CPU. We're considering using coroutine to enhance efficiency, but we don't have a solid solution yet. If you have experience in this area, you are very welcome to propose your solution.
6. We understand you're interested in contributing C++ code. However, if that's not the case, there's also unimplemented Python code, such as test cases and the Python SDK API.

[abdullah-alnahas]

Your work is exceptional! I would like to propose that, considering the current landscape, incorporating binary quantization and ColBERT-like ranking would be crucial for any vector database.
Apologies for commenting on the road map issue instead of creating a separate feature request.

[JinHai-CN]

Your work is exceptional! I would like to propose that, considering the current landscape, incorporating binary quantization and ColBERT-like ranking would be crucial for any vector database. Apologies for commenting on the road map issue instead of creating a separate feature request.

Nice, we will put this request into v0.2.0 release.

[niebayes]

@JinHai-CN Hi, I have experience in developing a database using Arrow. Is the issue that converting query results to Arrow format still active? I'd like to take it.

[JinHai-CN]

@niebayes #1198, issue is created and we can discuss the requirement in that issue.