Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add UCN support in Jet documentation [HZG-194] #1383

Merged
merged 7 commits into from
Dec 18, 2024
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/modules/clusters/pages/user-code-namespaces.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Your resources can be run on a Hazelcast cluster to support your business logic.
* MapStore
* Split brain events
* `ExecutorService` runnable
* Jet jobs

To use {ucn}, you must enable it as described in the xref:clusters:ucn-enable.adoc[] topic.

Expand Down
2 changes: 1 addition & 1 deletion docs/modules/clusters/partials/ucn-migrate-tip.adoc
Original file line number Diff line number Diff line change
@@ -1 +1 @@
CAUTION: {ucd} has been deprecated and will be removed in the next major version. To continue deploying your user code after this time, {open-source-product-name} users can either upgrade to {enterprise-product-name}, or add their resources to the Hazelcast member class paths. Hazelcast recommends that {enterprise-product-name} users migrate their user code to use {ucn} for all purposes other than Jet stream processing. For further information on migrating from {ucd} to {ucn}, see xref:clusters:ucn-migrate-ucd.adoc[].
CAUTION: {ucd} has been deprecated and will be removed in the next major version. To continue deploying your user code after this time, {open-source-product-name} users can either upgrade to {enterprise-product-name}, or add their resources to the Hazelcast member class paths. Hazelcast recommends that {enterprise-product-name} users migrate their user code to use {ucn}. For further information on migrating from {ucd} to {ucn}, see xref:clusters:ucn-migrate-ucd.adoc[].
k-jamroz marked this conversation as resolved.
Show resolved Hide resolved
2 changes: 2 additions & 0 deletions docs/modules/pipelines/pages/cdc-join.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -622,6 +622,8 @@ You should see the following jars:

. Enable user code deployment:
+
include::clusters:partial$ucn-migrate-tip.adoc[]
+
Due to the type of sink we are using in our pipeline we need to make
some extra changes in order for the Hazelcast cluster to be aware of the custom classes we have defined.
+
Expand Down
41 changes: 40 additions & 1 deletion docs/modules/pipelines/pages/configuring-jobs.adoc
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= Configuring Jobs
:description: Before submitting a job to a cluster, you can configure its name, whether to record metrics, and how it behaves when cluster members fail.
:description: Before submitting a job to a cluster, you can configure its name, whether to record metrics, how it behaves when cluster members fail, and more.

{description}

Expand Down Expand Up @@ -51,6 +51,11 @@ a|Depends on the source or sink.
|`string`
|''

|userCodeNamespace
|The User Code Namespace to associate these jobs with for access to custom resources, or `null` if not associated with one. xref:clusters:user-code-namespaces.adoc[{ucn}] must be enabled and the provided namespace configured.
|`string`
|null

|===

In addition to the above, there are also options to configure the behavior of retrieving job metrics; see xref:pipelines:job-monitoring.adoc#accessing-job-metrics[Accessing Job Metrics].
Expand Down Expand Up @@ -169,3 +174,37 @@ cluster change depending on these two settings.
|suspend job
|===

== Configuring a User Code Namespace
JamesHazelcast marked this conversation as resolved.
Show resolved Hide resolved

Normal Jet jobs can be configured to use a xref:clusters:user-code-namespaces.adoc[User Code Namespace] (UCN), which provides them
access to custom resources associated with that UCN. {ucn} must be enabled and a namespace configured with the desired resources
created before it can be used in jobs. Any data structures used in these jobs should also have their UCN configured to ensure
operations for that data structure also have access to the necessary custom resources. Light jobs do not support UCN.
JamesHazelcast marked this conversation as resolved.
Show resolved Hide resolved

{ucn} are managed separately to the Jet system, so these resources persist between individual jobs. This support allows for jobs to
be submitted and updated while still being able to use the same custom resources within pipelines.

WARNING: {ucn} can be dynamically updated at runtime, but it is recommended that namespaces associated with Jet jobs are not changed
while those jobs are running. Doing so may cause job execution inconsistency between member nodes if jobs restart during the namespace update. If all related jobs are suspended, then the associated namespace can be safely updated before resuming jobs.

The following is an example of how to associate a namespace with a job. The namespace must already be configured
in the Hazelcast member configuration.

[tabs]
====
Java::
+
--
[source,java]
----
JobConfig jobConfig = new JobConfig();
jobConfig.setUserCodeNamespace("my_namespace");
----
--

SQL::
+
--
NOTE: Definition of a UCN is not supported via SQL.
--
====
6 changes: 6 additions & 0 deletions docs/modules/pipelines/pages/job-update.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,12 @@ NOTE: The accumulator type must stay the same.

* Renaming a stateless stage.

* Updating an associated User Code Namespace
+
WARNING: All jobs using a specific namespace should be suspended while updating the namespace's resources, otherwise jobs may become
inconsistent due to differing namespace versions in use. Take care to ensure that the updated resources are compatible with any
existing data used by the jobs.

=== What you Cannot Do

Making any of these changes will make your pipeline incompatible with the saved state:
Expand Down
2 changes: 1 addition & 1 deletion docs/modules/sql/pages/create-job.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ The `CREATE JOB` statement accepts the following parameters.
|<<create-a-job, Create a Job>>

|option_name, option_value
|xref:pipelines:configuring-jobs.adoc[].
|See xref:pipelines:configuring-jobs.adoc[] for available parameters. Note that `userCodeNamespace` is not supported.
|<<create-a-job, Create a Job>>

|===
Expand Down