Add documentation for Vector Collections in Jet [HZAI-77] (hazelcast#…

…1125)

TODO:
- [ ] beta and enterprise markers
- [ ] link to [Vector collection data
structure](hazelcast#1124) docs
- [ ] use consistent wording with V[ector collection data structure
docs](hazelcast#1124)

---------

Co-authored-by: rebekah-lawrence <[email protected]>
Co-authored-by: Oliver Howell <[email protected]>

docs/modules/ROOT/nav.adoc

@@ -189,6 +189,7 @@ include::wan:partial$nav.adoc[]
 ** xref:integrate:jcache-connector.adoc[]
 ** xref:integrate:list-connector.adoc[]
 ** xref:integrate:map-connector.adoc[]
+** xref:integrate:vector-collection-connector.adoc[]
 ** xref:integrate:reliable-topic-connector.adoc[]
 * Databases
 ** xref:integrate:database-connectors.adoc[Overview]

docs/modules/integrate/pages/connectors.adoc

@@ -235,6 +235,18 @@ The Jet API supports more connectors than SQL.
 |streaming
 |none
 
+|xref:integrate:legacy-file-connector.adoc#fvecs-and-ivecs[VectorSources.fvecs]
+[.enterprise]*{enterprise-product-name}*
+|hazelcast-enterprise
+|batch
+|N/A
+
+|xref:integrate:legacy-file-connector.adoc#fvecs-and-ivecs[VectorSources.ivecs]
+[.enterprise]*{enterprise-product-name}*
+|hazelcast-enterprise
+|batch
+|N/A
+
 |xref:integrate:test-connectors.adoc[TestSources.items]
 |hazelcast-jet
 |batch
@@ -346,4 +358,11 @@ The Jet API supports more connectors than SQL.
 |hazelcast-jet
 |streaming
 |at-least-once
+
+|xref:integrate:vector-collection-connector.adoc[VectorSinks.vectorCollection]
+[.enterprise]*{enterprise-product-name}*
+|hazelcast-enterprise
+|batch, streaming
+|at-least-once
+
 |===

docs/modules/integrate/pages/file-connector.adoc

@@ -631,6 +631,37 @@ This file format is not supported by SQL.
 --
 ====
 
+
+=== Fvecs and ivecs
+[.enterprise]*{enterprise-product-name}*
+
+Fvecs and ivecs files are binary files containing, respectively, float and integer vectors with IDs.
+
+[tabs]
+====
+Jet::
++
+--
+```java
+BatchSource<Map.Entry<Integer, VectorValues>> vectors = FileSources.files("/data")
+  .glob("embeddings.fvecs")
+  .format(VectorSources.fvecsFormat())
+  .build();
+
+BatchSource<Map.Entry<Integer, int[]>> groundTruth = FileSources.files("/data")
+  .glob("groundtruth.ivecs")
+  .format(VectorSources.ivecsFormat())
+  .build();
+```
+--
+SQL::
++
+--
+This file format is not supported by SQL.
+--
+====
+
+
 == Authentication
 
 To allow Hazelcast to access files in remote systems, you must pass authentication credentials to the file connector.

docs/modules/integrate/pages/legacy-file-connector.adoc

@@ -349,4 +349,24 @@ S3 sink uses the multi-part upload feature of S3 SDK. The sink buffers
 the items to parts and uploads them after buffer reaches to the
 threshold. The multi-part upload is completed when the job completes and
 makes the objects available on the S3. Since a streaming jobs never
-complete, S3 sink is not currently applicable to streaming jobs.
+complete, S3 sink is not currently applicable to streaming jobs.
+
+== Fvecs and ivecs
+[.enterprise]*{enterprise-product-name}*
+
+Fvecs and ivecs files are binary files containing, respectively, float and integer vectors with IDs. The connectors are similar to local file connectors, but work with fvecs and ivecs files.
+Fvecs and ivecs files can only be sources, and can therefore only be read; fvecs and ivecs files cannot be used as sinks.
+
+```java
+Pipeline p = Pipeline.create();
+p.readFrom(VectorSources.fvecs("/home/data", "query.fvecs"))
+  .writeTo(Sinks.map("testvectors"));
+```
+
+```java
+Pipeline p = Pipeline.create();
+p.readFrom(VectorSources.ivecs("/home/data", "groundtruth.ivecs"))
+  .writeTo(Sinks.map("groundtruth"));
+```
+
+Legacy file connectors provide only basic options, if you need more features, like reading fvecs and ivecs files from cloud storage, use xref:file-connector.adoc#fvecs-and-ivecs[unified file connector].

docs/modules/integrate/pages/vector-collection-connector.adoc

@@ -0,0 +1,58 @@
+= Vector Collection Connector
+:description: Vector collection stores vectors with their related metadata. This allows entries to be found efficiently based on vector distance.
+:page-enterprise: true
+:page-beta: true
+
+{description}
+
+For further information on vector collections, see xref:data-structures:vector-collections.adoc[].
+
+== Installing the Connector
+
+This connector is included in the full and slim {enterprise-product-name} distributions of Hazelcast.
+
+== Permissions
+If xref:security:enabling-jaas.adoc[security] is enabled, you can set up permissions to restrict clients' access to these data structures.
+
+To search in vector collection, you must add the `create` and `read` permissions for those collections. If you use the vector collection sink to write to vector collections, you must add the `create` and `put` permissions for those collections.
+
+For further information on adding these permissions, see xref:security:native-client-security.adoc[].
+
+
+== Vector Collection as a Sink
+
+To write an entry to a vector collection, to index it for searching, create a key and `VectorDocument`, which consists of
+additional metadata and vectors (embeddings). Embeddings can be generated earlier in the pipeline or loaded or obtained from external source.
+
+
+```java
+Pipeline p = Pipeline.create();
+p.readFrom(Sources.<String, String>map("idToDocumentText"))
+  // generate embeddings
+  .mapUsingService(getAllMiniLmL6V2EmbeddingModelServiceFactory(),
+            (service, e) -> tuple3(e.getKey(), e.getValue(), VectorValues.of(service.embed(e.getValue()).content().vector())))
+  // write to vector collection
+  .writeTo(VectorSinks.vectorCollection("indexedDocuments", Tuple3::f0, Tuple3::f1, Tuple3::f2));
+```
+
+
+== Searching in Vector Collection
+
+You can search vector collections in Jet pipelines using `VectorTransforms.mapUsingVectorSearch` transformation.
+
+```java
+Pipeline p = Pipeline.create();
+p.readFrom(TestSources.items("text to search for"))
+  // generate embedding for the object for which we are finding similarities
+  .mapUsingService(getAllMiniLmL6V2EmbeddingModelServiceFactory(),
+            (service, query) -> tuple2(query, VectorValues.of(service.embed(query).content().vector())))
+  // find similar objects
+  .apply(VectorTransforms.mapUsingVectorSearch("indexedDocuments",
+            SearchOptions.builder().limit(10).includeValue().build(),
+            // query vector
+            Tuple3::f1,
+            // process the search results
+            (input, result) -> tuple2(input, result)))
+  // use the results
+  .writeTo(Sinks.logger());
+```