Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data connection #1417

Merged
merged 3 commits into from
Dec 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion docs/modules/ROOT/nav.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,11 @@ include::clusters:partial$nav.adoc[]
** xref:query:overview.adoc[]
include::mapstore:partial$nav.adoc[]
include::pipelines:partial$nav.adoc[]
* Data Connections
** xref:data-connections:data-connections-configuration.adoc[Configure Data Connections]
** xref:data-connections:data-connection-service.adoc[]
** xref:data-connections:build-map-loader-data-connection.adoc[Build MapLoader]
** xref:data-connections:build-pipeline-service-data-connection.adoc[Build pipeline service]
* SQL
** xref:sql:sql-overview.adoc[Overview]
** SQL Over Maps
Expand Down Expand Up @@ -157,7 +162,6 @@ include::cp-subsystem:partial$nav.adoc[]
* xref:storage:high-density-memory.adoc[]
include::tiered-storage:partial$nav.adoc[]
* xref:cluster-performance:thread-per-core-tpc.adoc[]
include::data-connections:partial$nav.adoc[]
include::wan:partial$nav.adoc[]
* xref:extending-hazelcast:extending-hazelcast.adoc[]
** xref:extending-hazelcast:operationparker.adoc[]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
= Map Loader using Data Connection
= Build MapLoader using Data Connection

:description: In this tutorial you build a custom map loader that uses a configured data connection to load data not present in an IMap.
:description: This tutorial shows how to build a custom map loader that uses a configured data connection to load data not present in an IMap.

{description}

NOTE: This tutorial builds a custom implementation of MapLoader. For the most common use cases we also provide an out-of-the-box implementation xref:mapstore:configuring-a-generic-maploader.adoc[GenericMapLoader].
NOTE: This tutorial builds a custom implementation of MapLoader. For the most common use cases an out-of-the-box implementation is also provided via xref:mapstore:configuring-a-generic-maploader.adoc[GenericMapLoader].

== Before you begin

Expand All @@ -23,7 +23,7 @@ To complete this tutorial, you need the following:

|===

=== Step 1. Create and Populate the Database
=== Step 1. Create and populate database

This tutorial uses Docker to run the Postgres database.

Expand Down Expand Up @@ -59,7 +59,7 @@ INSERT INTO my_table VALUES (8, 'eight');
INSERT INTO my_table VALUES (9, 'nine');
----

== Step 2. Create a New Java Project
== Step 2. Create new java project

Create a blank Java project named pipeline-service-data-connection-example and copy the Gradle or Maven file into it:

Expand Down Expand Up @@ -106,7 +106,7 @@ Create a blank Java project named pipeline-service-data-connection-example and c
</project>
----

== Step 3. MapLoader
== Step 3. Implement MapLoader

The following map loader implements the `com.hazelcast.map.MapLoader` and `com.hazelcast.map.MapLoaderLifecycleSupport`
interfaces.
Expand All @@ -121,7 +121,7 @@ public class SimpleMapLoader implements MapLoader<Integer, String>, MapLoaderLif
}
----

To implement the `MapLoaderLifecycleSupport` interface we need the following methods:
To implement the `MapLoaderLifecycleSupport` interface you need the following methods:

[source,java]
----
Expand Down Expand Up @@ -207,7 +207,7 @@ To implement the `MapLoader` interface we need the following methods:
}
----

== Step 4. MapLoader Example App
== Step 4. Create example MapLoader app

Configure the data connection:

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
= Pipeline Service using Data Connection
= Build pipeline service using Data Connection

:description: This tutorial builds a service that transforms a stream of items. The service uses a data connection to retrieve a connection to a relational database, uses a table in the database to enrich a stream of numbers with a textual representation of the last digit.
:description: This tutorial builds a service that transforms a stream of items. The service uses a data connection to retrieve a connection to a relational database, and uses a table in the database to enrich a stream of numbers with a textual representation of the last digit.

{description}

Expand All @@ -21,7 +21,7 @@ To complete this tutorial, you need the following:

|===

== Step 1. Create and Populate the Database
== Step 1. Create and populate database

This tutorial uses Docker to run the Postgres database.

Expand Down Expand Up @@ -57,9 +57,9 @@ INSERT INTO my_table VALUES (8, 'eight');
INSERT INTO my_table VALUES (9, 'nine');
----

== Step 2. Create a New Java Project
== Step 2. Create new java project

Create a blank Java project named pipeline-service-data-connection-example and copy the Gradle or Maven file into it:
Create a blank Java project named `pipeline-service-data-connection-example`` and copy the Gradle or Maven file into it:

[source,xml]
----
Expand Down Expand Up @@ -104,7 +104,7 @@ Create a blank Java project named pipeline-service-data-connection-example and c
</project>
----

== Step 3. Create the Pipeline and the Job
== Step 3. Create pipeline and job

Create a class `EnrichUsingDataConnection` with `main` method that will be the main job class. The `main` method will do the following steps:

Expand Down
31 changes: 15 additions & 16 deletions docs/modules/data-connections/pages/data-connection-service.adoc
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
= Using Data Connections in custom components
= Use Data Connection service
:description: Using the Data Connection Service gives access to the configured xref:data-connections-configuration.adoc[data connections] in custom components.

{description}

== Typical Usage
== Typical usage

The typical steps to use a data connection are as follows:

1. Obtain the data connection from the data connection service.
2. Retrieve the underlying resource from the `DataConnection` instance. This step varies based on the specific implementation of `DataConnection` (e.g., `JdbcDataConnection` provides `getConnection()` which returns a `java.sql.Connection`; `HazelcastDataConnection` provides `getClient()` which returns a `HazelcastInstance`).
2. Retrieve the underlying resource from the `DataConnection` instance. This step varies based on the specific implementation of `DataConnection` (e.g. `JdbcDataConnection` provides `getConnection()` which returns a `java.sql.Connection`; `HazelcastDataConnection` provides `getClient()` which returns a `HazelcastInstance`).
3. Use the resource to perform the required operations.
4. Dispose of the resource (e.g., by calling `Connection#close` or `HazelcastInstance#destroy`).
5. Release the `DataConnection` instance (by calling `DataConnection#release()`).

Steps 2, 3, and 4 should be completed as quickly as possible to maximize the efficiency of connection pooling.
You should complete steps 2, 3, and 4 as quickly as possible to maximize the efficiency of connection pooling.

[source,java]
----
Expand All @@ -31,7 +31,7 @@ try (Connection connection = jdbcDataConnection.getConnection()) { <2>
jdbcDataConnection.release(); <4>
----

== Retrieve Data Connection Service
== Retrieve Data Connection service

Before working with data connections you need to retrieve an instance of the `DataConnectionService`. Use
https://docs.hazelcast.org/docs/{full-version}/javadoc/com/hazelcast/core/HazelcastInstance.html#getDataConnectionService()[`HazelcastInstance#getDataConnectionService()`]
Expand All @@ -43,32 +43,31 @@ to the `HazelcastInstance`.
In the pipeline API you can use
https://docs.hazelcast.org/docs/{full-version}/javadoc/com/hazelcast/jet/core/ProcessorMetaSupplier.Context.html#dataConnectionService()[ProcessorMetaSupplier.Context#dataConnectionService()].

NOTE: The Data Connection Service is only available on the member side. Calling `getDataConnectionService()` on client will result in `UnsupportedOperationException`.
NOTE: The Data Connection Service is only available on the member side. Calling `getDataConnectionService()` on a client results in `UnsupportedOperationException`.

== Retrieve Configured DataConnection
== Retrieve configured DataConnection

Use the `DataConnectionService` to get an instance of previously configured data connection https://docs.hazelcast.org/docs/{full-version}/javadoc/com/hazelcast/dataconnection/DataConnectionService.html#getAndRetainDataConnection(java.lang.String,java.lang.Class)[DataConnectionService#getAndRetainDataConnection(String, Class)]. For details how to configure a data connection, please refer
to the xref:data-connections-configuration.adoc[Configuring Data Connections] page.

== Data Connection Scope
== Data Connection scope

The data connection configuration is per-member. For example, when a data connection is created
with maximum pool size of 10 and the cluster has 3 members, there will be up to 30 connections
with maximum pool size of 10 and the cluster has 3 members, up to 30 connections will be
created.

== Data Connection Sharing
== Data Connection sharing

Data connection is shared by default. It means that when the data connection is requested in multiple places, the same
underlying resource (e.g. Jdbc pool, remote client) is used.
Data connection is shared by default. This means that when the data connection is requested in multiple places, the same
underlying resource (e.g. JDBC pool, remote client) is used.
If you want to share the data connection configuration, but use a different instance of the underlying resource,
set the `DataConnectionConfig#setShared` to false.

== Configuration Considerations
== Configuration considerations

If the data connection is defined in the Hazelcast configuration, it remains immutable for the entire lifespan of the Hazelcast member. In this case, whether you retrieve the DataConnection instance once or each time before accessing the underlying resource, the result will be the same.

However, if the data connection is created dynamically via SQL, it can be replaced using `CREATE OR REPLACE DATA CONNECTION`
(see xref:sql.adoc).
However, if the data connection is created dynamically via SQL, it can be replaced using `CREATE OR REPLACE DATA CONNECTION`. For more information, see xref:sql:get-started-sql.adoc[].
In such cases, the DataConnection instance will stay valid until you release it, allowing you to retrieve the underlying resource as needed. This approach can be useful for adapting to changes in data connection configuration.

For example, if you are running a batch job and want to use the same data connection throughout, request the connection at the start of the job. For a streaming job that may need updated configurations, retrieve both the data connection and the underlying resource just before use (e.g., when processing each item in the pipeline).
For example, if you are running a batch job and want to use the same data connection throughout, request the connection at the start of the job. For a streaming job that may need updated configurations, retrieve both the data connection and the underlying resource just before use (e.g. when processing each item in the pipeline).
2 changes: 0 additions & 2 deletions docs/modules/data-connections/partials/nav.adoc

This file was deleted.

Loading