fs2-kafka v2.5.0-M3

This milestone release exposes some additional consumer functionality and fixes a long-standing bug/shortcoming in the Vulcan integration module, inherited from the Confluent Kafka Avro serializer, whereby auto-registration of Union and some other schemas would not work as expected.

What's Changed

Enhancements

• Additional consumer functionality exposed by @dmedser in #919
• Fix auto-registration of Avro union (etc) types by @bplommer in #932

Internals and updates

• Move derived consumer methods to traits by @bplommer in #916
• Update testcontainers version, set image name and version explicitly by @bplommer in #923
• Use permit for fetch, rework permits by @bplommer in #918
• Use a shared test container for each suite by @bplommer in #925
• Update testcontainers-scala-kafka, ... to 0.40.4 by @scala-steward in #928
• Update sbt-mdoc to 2.3.2 by @scala-steward in #929
• Update cats-effect-laws, ... to 3.3.9 by @scala-steward in #930
• Update sbt-gha, add JDK 17 to pipeline by @bplommer in #927

New Contributors

• @dmedser made their first contribution in #919

Full Changelog: 2.5.0-M1...v2.5.0-M3

fs2-kafka v3.0.0-M6

This milestone makes the following improvements to the API, in addition to the changes in v2.5.0-M1. As always, feedback is welcome!

What's Changed

• Introduce type-level distinction between key and value serializers/deserializers by @bplommer in #898
• Simplify ProducerRecords types by removing passthrough by @bplommer in #901
  • The passthrough functionality was orthogonal to the producer functionality - if needed it should instead be replicated in user code.

Full Changelog: v3.0.0-M5...v3.0.0-M6

fs2-kafka v2.5.0-M2

This release is identical to v2.5.0-M1, which failed to publish.

This release adds some new features and significantly simplifies the internals of the consumer. Although we don't expect these changes to cause any problems, we're releasing this as a milestone to allow for some extra battle testing before the next stable release.

Note: there is a small change to the behaviour of consumers when using partitionedStream or partitionsMapStream. Previously, failure to deserialize a record would cause the entire consumer to fail; now, it will only cause a single partition stream to fail.

What's Changed

New features

• Add produce method to KafkaProducerConnection by @bplommer in #900
  • KafkaProducerConnection allocates a producer without being tied to particular serializers, allowing creation of multiple KafkaProducer instances that share a single set of connections to the broker. With this change, it is possible to produce records directly from KafkaProducerConnection by providing serializers when calling produce.
• Allow creating settings with placeholder serdes, changing serdes in settings by @bplommer in #911
  • This allows creation of ProducerSettings and ConsumerSettings to be decoupled from provision of serializers and deserializers, which is particularly useful when creating multiple producers or consumers with different types.

Docs improvements

• withParallelism default 100 -> 10000 in docs by @jbwheatley in #907
• Use IOApp.Simple in docs by @bplommer in #908

Simplification of internals

• Move deserialization from KafkaConsumerActor to KafkaConsumer by @bplommer in #902
• Use semaphore-like permits for some actor operations by @bplommer in #906

Dependency updates

• Update cats-effect-laws, ... to 3.3.7 by @scala-steward in #910
• Update logback-classic to 1.2.11 by @scala-steward in #909

New Contributors

• @jbwheatley made their first contribution in #907

Full Changelog: v2.4.0...2.5.0-M1

fs2-kafka v2.5.0-M1

This release adds some new features and significantly simplifies the internals of the consumer. Although we don't expect these changes to cause any problems, we're releasing this as a milestone to allow for some extra battle testing before the next stable release.

Note: there is a small change to the behaviour of consumers when using partitionedStream or partitionsMapStream. Previously, failure to deserialize a record would cause the entire consumer to fail; now, it will only cause a single partition stream to fail.

What's Changed

New features

• Add produce method to KafkaProducerConnection by @bplommer in #900
  • KafkaProducerConnection allocates a producer without being tied to particular serializers, allowing creation of multiple KafkaProducer instances that share a single set of connections to the broker. With this change, it is possible to produce records directly from KafkaProducerConnection by providing serializers when calling produce.
• Allow creating settings with placeholder serdes, changing serdes in settings by @bplommer in #911
  • This allows creation of ProducerSettings and ConsumerSettings to be decoupled from provision of serializers and deserializers, which is particularly useful when creating multiple producers or consumers with different types.

Docs improvements

• withParallelism default 100 -> 10000 in docs by @jbwheatley in #907
• Use IOApp.Simple in docs by @bplommer in #908

Simplification of internals

• Move deserialization from KafkaConsumerActor to KafkaConsumer by @bplommer in #902
• Use semaphore-like permits for some actor operations by @bplommer in #906

Dependency updates

• Update cats-effect-laws, ... to 3.3.7 by @scala-steward in #910
• Update logback-classic to 1.2.11 by @scala-steward in #909

New Contributors

• @jbwheatley made their first contribution in #907

Full Changelog: v2.4.0...2.5.0-M1

fs2-kafka v3.0.0-M5

This is the latest milestone release for the fs2-kafka v3.x series, which uses Kafka 3. It updates the kafka-clients library to 3.1 and includes the same changes as v2.4.0.

fs2-kafka v2.4.0

This release adds several new features to the fs2-kafka 2.x line, which uses cats-effect 3 and Kafka 2. It also fixes a concurrency-related bug in TransactionalKafkaProducer.

What's Changed

Enhancements

• Add support for using CooperativeStickyAssignor with KafkaConsumer by @biochimia in #844
• Allow transaction without upstream offsets by @janstenpickle in #883
• Add attempt method to RecordDeserializer by @bplommer in #891

Bugfixes

• Threadsafe transactions by @janstenpickle in #892

Doc improvements

• Add a basic version matrix to the readme by @bplommer in #841
• Slightly improve docs for stream methods by @LMnet in #872
• Publish certs docs by @bplommer in #881
• Update transactional producer docs by @janstenpickle in #890

Dependency updates

• Update cats-effect, cats-effect-laws, ... to 3.3.4 by @scala-steward in #835
• Update scala-library to 2.13.8 by @scala-steward in #838
• Update testcontainers-scala-kafka, ... to 0.40.0 in series/1.x by @scala-steward in #846
• Update fs2-core to 3.2.5 by @scala-steward in #868
• Update sbt-buildinfo to 0.11.0 by @scala-steward in #865
• Update sbt-header to 5.6.5 by @scala-steward in #862
• Update sbt to 1.6.2 by @scala-steward in #854
• Update scala3-library to 3.1.1 by @scala-steward in #857
• Update cats-effect-laws, ... to 3.3.6 by @scala-steward in #882
• Update vulcan version to 1.8.0 by @bplommer in #896

Housekeeping

• Update copyright notices by @bplommer in #833

New Contributors

• @biochimia made their first contribution in #844

Full Changelog: v2.3.0...v2.4.0

v1.10.0

This is a maintenance release for the 1.x series, which is based on cats-effect 2 and is no longer receiving new features. It fixes a concurrency-related bug in TransactionalKafkaProducer (see #888) and updates various dependencies.

What's Changed

• Update scala-library to 2.13.8 in series/1.x by @scala-steward in #839
• Update testcontainers-scala-kafka, ... to 0.40.0 in series/1.x by @scala-steward in #846
• Update sbt-buildinfo to 0.11.0 in series/1.x by @scala-steward in #866
• Update sbt-header to 5.6.5 in series/1.x by @scala-steward in #863
• Update sbt to 1.6.2 in series/1.x by @scala-steward in #855
• Improve scala-steward config for 1.x series by @LMnet in #871
• Publish certs docs by @bplommer in #881
• Update testcontainers-scala-kafka, ... to 0.40.1 in series/1.x by @scala-steward in #860
• Update logback-classic to 1.2.11 in series/1.x by @scala-steward in #884
• Update sbt-mima-plugin to 1.0.1 in series/1.x by @scala-steward in #711
• Threadsafe transactions by @janstenpickle in #888
• Update vulcan version to 1.8.0 in series/1.x by @bplommer in #895
• Don't publish docs when releasing 1.x by @bplommer in #893

Full Changelog: v1.9.0...v1.10.0

fs2-kafka v3.0.0-M4

This is the first milestone release of fs2-kafka following the release of 2.0.0. (We have previously released v3.0.0-M1 through v3.0.0-M3 but these became the 2.x series following a change in our roadmap.)

This is the first release of fs2-kafka to use version 3.0.0 of the Apache Kafka client library, which is not binary-compatible with previous versions. It is available for Scala 2.13 and Scala 3.1; Scala 2.12 has been dropped for the 3.x series but will continue to be supported in the 1.x and 2.x series.

fs2-kafka v2.3.0

This release brings new functionality and dependency updates to the fs2-kafka 2.x series, which is based on cats-effect 2. These include an update to kafka-clients that resolves a security vulnerability (see #821)

What's Changed

Functionality

• Add deleteConsumerGroups method by @keirlawson in #730
• Add avro settings register schema method by @agustafson in #687
• Add producer convenience methods by @keirlawson in #632

Updates

• Update sbt-mima-plugin to 1.0.1 by @scala-steward in #698
• Update kind-projector to 0.13.2 by @scala-steward in #682
• Update kafka-avro-serializer to 6.2.1 by @scala-steward in #695
• Update munit to 0.7.29 by @scala-steward in #679
• Update kafka-clients to 2.8.1 by @scala-steward in #690
• Update sbt-mdoc to 2.2.23 by @scala-steward in #675
• Update scala-library to 2.12.15 by @scala-steward in #688
• Update sbt-ci-release to 1.5.10 by @scala-steward in #700
• Update fs2-core to 3.1.6 by @scala-steward in #704
• Update logback-classic to 1.2.10 by @scala-steward in #804
• Update sbt to 1.6.1 by @scala-steward in #814
• Update scala-library to 2.13.7 by @scala-steward in #746
• Update sbt-unidoc to 0.5.0 by @scala-steward in #736
• Update sbt-scalafmt to 2.4.6 by @scala-steward in #807
• Update kafka-avro-serializer to 6.2.2 by @scala-steward in #781
• Update testcontainers-scala-kafka, ... to 0.39.12 by @scala-steward in #759
• Update scala3-library to 3.1.0 by @scala-steward in #705
• Update cats-effect, cats-effect-laws, ... to 3.3.3 by @scala-steward in #819
• Update fs2-core to 3.2.4 by @scala-steward in #810
• Update sbt-mdoc to 2.2.24 by @scala-steward in #727

Other

• Update code to use modern cats import by @Daenyth in #765

New Contributors

• @keirlawson made their first contribution in #632

Full Changelog: v2.2.0...v2.3.0

fs2-kafka v1.9.0

This is a maintenance release for the fs2-kafka 1.x series, which is based on cats-effect 2. It includes various dependency updates, including an update to kafka-clients that resolves a security vulnerability (see #821), and adds docs for the munit module. There are no other user-facing changes.

An update to the fs2-kafka 2.x series, which is based on cats-effect 3, will be released soon.

What's Changed

Docs

• Add docs for munit module by @keirlawson in #672

Dependency updates

• Update munit to 0.7.29 in series/1.x by @scala-steward in #716
• Update sbt-ci-release to 1.5.9 in series/1.x by @scala-steward in #710
• Update sbt-mdoc to 2.2.24 in series/1.x by @scala-steward in #728
• Update logback-classic to 1.2.10 in series/1.x by @scala-steward in #805
• Update sbt to 1.6.1 in series/1.x by @scala-steward in #815
• Update kind-projector to 0.13.2 in series/1.x by @scala-steward in #720
• Update kafka-avro-serializer to 6.2.2 in series/1.x by @scala-steward in #782
• Update sbt-scalafmt to 2.4.6 in series/1.x by @scala-steward in #808
• Update testcontainers-scala-kafka, ... to 0.39.12 in series/1.x by @scala-steward in #760
• Update scala-library to 2.12.15 in series/1.x by @scala-steward in #714
• Update kafka-clients to 2.8.1 in series/1.x by @scala-steward in #713
• Update sbt-unidoc to 0.5.0 in series/1.x by @scala-steward in #737
• Update cats-effect, cats-effect-laws to 2.5.4 in series/1.x by @scala-steward in #719
• Update scala-library to 2.13.7 in series/1.x by @scala-steward in #748
• Update fs2-core to 2.5.10 in series/1.x by @scala-steward in #707

Housekeeping

• Update copyright notices by @bplommer in #833

Full Changelog: v1.8.0...v1.9.0