13 Dec 21:26

vincentvilo-aws

v2.6.1

c69ccb6

Release 2.6.1 of the Amazon Kinesis Client Library for Java

#1365 Add config to enable point in time recovery for the DynamoDB lease table created by KCL
#1325 Add logic to sync from lease table on first app bootstrap
#1372 Fix debug and trace level logging to log all worker loops
#1319 Fix a race condition between ShardConsumer shutdown and initialization
#1354 Fix bug where the in-memory lease is updated even when DDB fails to update
#1363 Fix some of the warnings emitted by maven during build
#1358 Remove ShutdownNotificationAware and update javadocs
#1329 Remove unnecessary lambda
#1340 Generate wrappers from proto files instead of shipping them directly
#1332 Enforce consistent code style with palantir-java-format
#1294 Upgrade org.apache.maven.plugins:maven-checkstyle-plugin from 3.3.0 to 3.3.1
#1296 Upgrade org.apache.maven.plugins:maven-surefire-plugin from 3.1.2 to 3.2.5
#1312 Upgrade commons-io:commons-io from 2.15.1 to 2.16.1
#1331 Upgrade maven-compiler-plugin from 3.11.0 to 3.13.0
#1335 Upgrade org.slf4j:slf4j-api from 2.0.7 to 2.0.13
#1336 Upgrade io.reactivex.rxjava3:rxjava from 3.1.6 to 3.1.8
#1337 Upgrade com.google.protobuf:protobuf-java from 3.21.12 to 4.27.0
#1341 Upgrade awssdk.version from 2.25.11 to 2.25.64
#1342 Upgrade org.apache.maven.plugins:maven-javadoc-plugin from 3.6.3 to 3.7.0
#1343 Upgrade org.apache.maven.plugins:maven-failsafe-plugin from 3.1.2 to 3.2.5
#1383 Upgrade com.google.protobuf:protobuf-java from 4.27.0 to 4.27.5

Assets 2

15 Nov 04:54

chenylee-aws

v3.0.1

01d0e5c

v3.0.1 Latest

Latest

What's Changed

Update backward compatibility check script by @akidambisrinivasan in #1393
Update version to 3.0.1-SNAPSHOT by @akidambisrinivasan in #1394
Address KCLv3 issues from github by @akidambisrinivasan in #1398
Fix checkpointOwner copy issue for multistream lease by @chenylee-aws in #1401
Preparation for v3.0.1 by @chenylee-aws in #1402

Full Changelog: v3.0.0...v3.0.1

Contributors

chenylee-aws and akidambisrinivasan

Assets 2

06 Nov 17:38

lucienlu-aws

v3.0.0

fbfd74d

Release 3.0.0 of the Amazon Kinesis Client Library for Java

Changelog

For 1.x release notes, please see v1.x/CHANGELOG.md

For 2.x release notes, please see v2.x/CHANGELOG.md

Release 3.0.0 (November 06, 2024)

New lease assignment / load balancing algorithm
- KCL 3.x introduces a new lease assignment and load balancing algorithm. It assigns leases among workers based on worker utilization metrics and throughput on each lease, replacing the previous lease count-based lease assignment algorithm.
- When KCL detects higher variance in CPU utilization among workers, it proactively reassigns leases from over-utilized workers to under-utilized workers for even load balancing. This ensures even CPU utilization across workers and removes the need to over-provision the stream processing compute hosts.
Optimized DynamoDB RCU usage
- KCL 3.x optimizes DynamoDB read capacity unit (RCU) usage on the lease table by implementing a global secondary index with leaseOwner as the partition key. This index mirrors the leaseKey attribute from the base lease table, allowing workers to efficiently discover their assigned leases by querying the index instead of scanning the entire table.
- This approach significantly reduces read operations compared to earlier KCL versions, where workers performed full table scans, resulting in higher RCU consumption.
Graceful lease handoff
- KCL 3.x introduces a feature called "graceful lease handoff" to minimize data reprocessing during lease reassignments. Graceful lease handoff allows the current worker to complete checkpointing of processed records before transferring the lease to another worker. For graceful lease handoff, you should implement checkpointing logic within the existing shutdownRequested() method.
- This feature is enabled by default in KCL 3.x, but you can turn off this feature by adjusting the configuration property isGracefulLeaseHandoffEnabled.
- While this approach significantly reduces the probability of data reprocessing during lease transfers, it doesn't completely eliminate the possibility. To maintain data integrity and consistency, it's crucial to design your downstream consumer applications to be idempotent. This ensures that the application can handle potential duplicate record processing without adverse effects.
New DynamoDB metadata management artifacts
- KCL 3.x introduces two new DynamoDB tables for improved lease management:
  - Worker metrics table: Records CPU utilization metrics from each worker. KCL uses these metrics for optimal lease assignments, balancing resource utilization across workers. If CPU utilization metric is not available, KCL assigns leases to balance the total sum of shard throughput per worker instead.
  - Coordinator state table: Stores internal state information for workers. Used to coordinate in-place migration from KCL 2.x to KCL 3.x and leader election among workers.
- Follow this documentation to add required IAM permissions for your KCL application.
Other improvements and changes
- Dependency on the AWS SDK for Java 1.x has been fully removed.
  - The Glue Schema Registry integration functionality no longer depends on AWS SDK for Java 1.x. Previously, it required this as a transient dependency.
  - Multilangdaemon has been upgraded to use AWS SDK for Java 2.x. It no longer depends on AWS SDK for Java 1.x.
- idleTimeBetweenReadsInMillis (PollingConfig) now has a minimum default value of 200.
  - This polling configuration property determines the publishers wait time between GetRecords calls in both success and failure cases. Previously, setting this value below 200 caused unnecessary throttling. This is because Amazon Kinesis Data Streams supports up to five read transactions per second per shard for shared-throughput consumers.
- Shard lifecycle management is improved to deal with edge cases around shard splits and merges to ensure records continue being processed as expected.
Migration
- The programming interfaces of KCL 3.x remain identical with KCL 2.x for an easier migration, with the exception of those applications that do not use the recommended approach of using the Config Builder. These applications will have to refer to the troubleshooting guide. For detailed migration instructions, please refer to the Migrate consumers from KCL 2.x to KCL 3.x page in the Amazon Kinesis Data Streams developer guide.
Configuration properties
- New configuration properties introduced in KCL 3.x are listed in this doc.
- Deprecated configuration properties in KCL 3.x are listed in this doc. You need to keep the deprecated configuration properties during the migration from any previous KCL version to KCL 3.x.
Metrics
- New CloudWatch metrics introduced in KCL 3.x are explained in the Monitor the Kinesis Client Library with Amazon CloudWatch in the Amazon Kinesis Data Streams developer guide. The following operations are newly added in KCL 3.x:
  - LeaseAssignmentManager
  - WorkerMetricStatsReporter
  - LeaseDiscovery

Assets 2

16 Aug 20:31

lucienlu-aws

v1.15.2

19818cc

Release 1.15.2 of the Amazon Kinesis Client Library for Java

Release (1.15.2 - Aug 14, 2024)

#1371 Fix a bug in debug and trace logging levels for worker
#1224 Modify RecordProcessorCheckpointer#advancePosition Metrics usage to ensure proper closure
#1345 Generate wrappers from proto files instead of shipping them directly
#1346 Upgrade com.google.protobuf:protobuf-java from 3.23.4 to 4.27.1
#1338 Upgrade org.apache.logging.log4j:log4j-api from 2.20.0 to 2.23.1
#1327 Upgrade com.google.guava:guava from 33.0.0-jre to 33.2.0-jre
#1283 Upgrade com.fasterxml.jackson.core:jackson-core from 2.15.2 to 2.17.0
#1284 Upgrade aws-java-sdk.version from 1.12.647 to 1.12.681
#1288 Upgrade commons-logging:commons-logging from 1.2 to 1.3.1
#1289 Upgrade org.projectlombok:lombok from 1.18.22 to 1.18.32
#1248 Upgrade org.apache.maven.plugins:maven-surefire-plugin from 2.22.2 to 3.2.5
#1234 Upgrade org.apache.maven.plugins:maven-javadoc-plugin from 3.4.1 to 3.6.3
#1137 Upgrade maven-failsafe-plugin from 2.22.2 to 3.1.2
#1134 Upgrade jackson-core from 2.15.0 to 2.15.2
#1119 Upgrade maven-source-plugin from 3.2.1 to 3.3.0
#1165 Upgrade protobuf-java from 3.19.6 to 3.23.4

Assets 2

02 May 00:48

lucienlu-aws

v2.6.0

89a90e3

Release 2.6.0 of Amazon Kinesis Client Library for Java

Release 2.6.0 (2024-05-01)

#1317 Add enablePriorityLeaseAssignment config
#1320 Update lease taker to get unassigned leases
#1318 Internally construct and use stream ARNs for all streams in multi-stream mode
#1291 Update RetrievalFactory implementations to utilize the StreamIdentifier field of StreamConfig
#1308 Move shutdownComplete call to ShardConsumer
#1313 Add additional integration tests for multi-stream and cross account access
#1273 Optimize currentStreamConfigMap by cleaning up lingering streams
#1302 Fix gracefulShutdown behavior in Scheduler

Assets 2

27 Mar 22:19

brendan-p-lynch

v2.5.8

bf5ab60

Release 2.5.8 of Amazon Kinesis Client Library for Java

Release 2.5.8 (2024-03-27)

#1278 Upgrade awssdk.version from 2.25.3 to 2.25.11
#1279 Upgrade org.apache.maven.plugins:maven-gpg-plugin from 3.1.0 to 3.2.1
#1280 Upgrade org.apache.commons:commons-lang3 from 3.12.0 to 3.14.0
#1282 Upgrade org.apache.maven.plugins:maven-javadoc-plugin from 3.5.0 to 3.6.3
#1277 Reuse 'ShardSyncTaskManager' instance for existing stream to avoid duplicate enqueue of 'ShardSyncTask'

Assets 2

19 Mar 18:09

brendan-p-lynch

v2.5.7

f205673

Release 2.5.7 of Amazon Kinesis Client Library for Java

Changelog

For 1.x release notes, please see v1.x/CHANGELOG.md

Release 2.5.7 (2024-03-19)

#1275 Update PollingConfig maxRecords method to return PollingConfig
#1236 Upgrade commons-io:commons-io from 2.11.0 to 2.15.1
#1189 Upgrade org.apache.maven.plugins:maven-resources-plugin from 3.3.0 to 3.3.1
#1139 Upgrade maven-surefire-plugin from 2.22.2 to 3.1.2
#1138 Upgrade maven-failsafe-plugin from 2.22.2 to 3.1.2
#1125 Upgrade maven-gpg-plugin from 3.0.1 to 3.1.0