Skip to content

Latest commit

 

History

History
1187 lines (991 loc) · 53.1 KB

CHANGELOG.md

File metadata and controls

1187 lines (991 loc) · 53.1 KB
Raw

Changelog

Version 1.31.1 (2022-02-01)

  • #2126 - @bai - Populate missing kafka versions
  • #2124 - @bai - Add Kafka 3.1.0 to CI matrix, migrate to bitnami kafka image
  • #2123 - @bai - Update klauspost/compress to 0.14
  • #2122 - @dnwe - fix(test): make it simpler to re-use toxiproxy
  • #2119 - @bai - Add Kafka 3.1.0 version number
  • #2005 - @raulnegreiros - feat: add methods to pause/resume consumer's consumption
  • #2051 - @seveas - Expose the TLS connection state of a broker connection
  • #2117 - @wuhuizuo - feat: add method MockApiVersionsResponse.SetApiKeys
  • #2110 - @dnwe - fix: ensure heartbeats only stop after cleanup
  • #2113 - @mosceo - Fix typo

Version 1.31.0 (2022-01-18)

What's Changed

🎉 New Features / Improvements

  • feat: expose IncrementalAlterConfigs API in admin.go by @fengyinqiao in IBM#2088
  • feat: allow AsyncProducer to have MaxOpenRequests inflight produce requests per broker by @xujianhai666 in IBM#1686
  • Support request pipelining in AsyncProducer by @slaunay in IBM#2094

🐛 Fixes

  • fix(test): add fluent interface for mocks where missing by @grongor in IBM#2080
  • fix(test): test for ConsumePartition with OffsetOldest by @grongor in IBM#2081
  • fix: set HWMO during creation of partitionConsumer (fix incorrect HWMO before first fetch) by @grongor in IBM#2082
  • fix: ignore non-nil but empty error strings in Describe/Alter client quotas responses by @agriffaut in IBM#2096
  • fix: skip over KIP-482 tagged fields by @dnwe in IBM#2107
  • fix: clear preferredReadReplica if broker shutdown by @dnwe in IBM#2108
  • fix(test): correct wrong offsets in mock Consumer by @grongor in IBM#2078
  • fix: correct bugs in DescribeGroupsResponse by @dnwe in IBM#2111

🔧 Maintenance

  • chore: bump runtime and test dependencies by @dnwe in IBM#2100

📝 Documentation

  • docs: refresh README.md for Kafka 3.0.0 by @dnwe in IBM#2099

➕ Other Changes

New Contributors

  • @grongor made their first contribution in IBM#2080
  • @fengyinqiao made their first contribution in IBM#2088
  • @xujianhai666 made their first contribution in IBM#1686
  • @mosceo made their first contribution in IBM#2084

Full Changelog: https://github.com/Shopify/sarama/compare/v1.30.1...v1.31.0

Version 1.30.1 (2021-12-04)

What's Changed

🎉 New Features / Improvements

  • feat(zstd): pass level param through to compress/zstd encoder by @lizthegrey in IBM#2045

🐛 Fixes

  • fix: set min-go-version to 1.16 by @troyanov in IBM#2048
  • logger: fix debug logs' formatting directives by @utrack in IBM#2054
  • fix: stuck on the batch with zero records length by @pachmu in IBM#2057
  • fix: only update preferredReadReplica if valid by @dnwe in IBM#2076

🔧 Maintenance

  • chore: add release notes configuration by @dnwe in IBM#2046
  • chore: confluent platform version bump by @lizthegrey in IBM#2070

Notes

  • ℹ️ from Sarama 1.30.x onward the minimum version of Go toolchain required is 1.16.x

New Contributors

  • @troyanov made their first contribution in IBM#2048
  • @lizthegrey made their first contribution in IBM#2045
  • @utrack made their first contribution in IBM#2054
  • @pachmu made their first contribution in IBM#2057

Full Changelog: https://github.com/Shopify/sarama/compare/v1.30.0...v1.30.1

Version 1.30.0 (2021-09-29)

⚠️ This release has been superseded by v1.30.1 and should not be used.

regression: enabling rackawareness causes severe throughput drops (#2071) — fixed in v1.30.1 via #2076

ℹ️ Note: from Sarama 1.30.0 the minimum version of Go toolchain required is 1.16.x

New Features / Improvements

  • #1983 - @zifengyu - allow configure AllowAutoTopicCreation argument in metadata refresh
  • #2000 - @matzew - Using xdg-go module for SCRAM
  • #2003 - @gdm85 - feat: add counter metrics for consumer group join/sync and their failures
  • #1992 - @zhaomoran - feat: support SaslHandshakeRequest v0 for SCRAM
  • #2006 - @faillefer - Add support for DeleteOffsets operation
  • #1909 - @agriffaut - KIP-546 Client quota APIs
  • #1633 - @aldelucca1 - feat: allow balance strategies to provide initial state
  • #1275 - @dnwe - log: add a DebugLogger that proxies to Logger
  • #2018 - @dnwe - feat: use DebugLogger reference for goldenpath log
  • #2019 - @dnwe - feat: add logging & a metric for producer throttle
  • #2023 - @dnwe - feat: add Controller() to ClusterAdmin interface
  • #2025 - @dnwe - feat: support ApiVersionsRequest V3 protocol
  • #2028 - @dnwe - feat: send ApiVersionsRequest on broker open
  • #2034 - @bai - Add support for kafka 3.0.0

Fixes

  • #1990 - @doxsch - fix: correctly pass ValidateOnly through to CreatePartitionsRequest
  • #1988 - @LubergAlexander - fix: correct WithCustomFallbackPartitioner implementation
  • #2001 - @HurSungYun - docs: inform AsyncProducer Close pitfalls
  • #1973 - @qiangmzsx - fix: metrics still taking up too much memory when metrics.UseNilMetrics=true
  • #2007 - @bai - Add support for Go 1.17
  • #2009 - @dnwe - fix: enable nilerr linter and fix iferr checks
  • #2010 - @dnwe - chore: enable exportloopref and misspell linters
  • #2013 - @faillefer - fix(test): disable encoded response/request check when map contains multiple elements
  • #2015 - @bai - Change default branch to main
  • #1718 - @crivera-fastly - fix: correct the error handling in client.InitProducerID()
  • #1984 - @null-sleep - fix(test): bump confluentPlatformVersion from 6.1.1 to 6.2.0
  • #2016 - @dnwe - chore: replace deprecated Go calls
  • #2017 - @dnwe - chore: delete legacy vagrant script
  • #2020 - @dnwe - fix(test): remove testLogger from TrackLeader test
  • #2024 - @dnwe - chore: bump toxiproxy container to v2.1.5
  • #2033 - @bai - Update dependencies
  • #2031 - @gdm85 - docs: do not mention buffered messages in sync producer Close method
  • #2035 - @dnwe - chore: populate the missing kafka versions
  • #2038 - @dnwe - feat: add a fuzzing workflow to github actions

New Contributors

  • @zifengyu made their first contribution in IBM#1983
  • @doxsch made their first contribution in IBM#1990
  • @LubergAlexander made their first contribution in IBM#1988
  • @HurSungYun made their first contribution in IBM#2001
  • @gdm85 made their first contribution in IBM#2003
  • @qiangmzsx made their first contribution in IBM#1973
  • @zhaomoran made their first contribution in IBM#1992
  • @faillefer made their first contribution in IBM#2006
  • @crivera-fastly made their first contribution in IBM#1718
  • @null-sleep made their first contribution in IBM#1984

Full Changelog: https://github.com/Shopify/sarama/compare/v1.29.1...v1.30.0

Version 1.29.1 (2021-06-24)

New Features / Improvements

  • #1966 - @ajanikow - KIP-339: Add Incremental Config updates API
  • #1964 - @ajanikow - Add DelegationToken ResourceType

Fixes

  • #1962 - @hanxiaolin - fix(consumer): call interceptors when MaxProcessingTime expire
  • #1971 - @KerryJava - fix kafka-producer-performance throughput panic
  • #1968 - @dnwe - chore: bump golang.org/x versions
  • #1956 - @joewreschnig - Allow checking the entire ProducerMessage in the mock producers
  • #1963 - @dnwe - fix: ensure backoff timer is re-used
  • #1949 - @dnwe - fix: explicitly use uint64 for payload length

Version 1.29.0 (2021-05-07)

New Features / Improvements

  • #1917 - @arkady-emelyanov - KIP-554: Add Broker-side SCRAM Config API
  • #1869 - @wyndhblb - zstd: encode+decode performance improvements
  • #1541 - @izolight - add String, (Un)MarshalText for acl types.
  • #1921 - @bai - Add support for Kafka 2.8.0

Fixes

  • #1936 - @dnwe - fix(consumer): follow preferred broker
  • #1933 - @ozzieba - Use gofork for encoding/asn1 to fix ASN errors during Kerberos authentication
  • #1929 - @celrenheit - Handle isolation level in Offset(Request|Response) and require stable offset in FetchOffset(Request|Response)
  • #1926 - @dnwe - fix: correct initial CodeQL findings
  • #1925 - @bai - Test out CodeQL
  • #1923 - @bestgopher - Remove redundant switch-case, fix doc typos
  • #1922 - @bai - Update go dependencies
  • #1898 - @mmaslankaprv - Parsing only known control batches value
  • #1887 - @withshubh - Fix: issues affecting code quality

Version 1.28.0 (2021-02-15)

Note that with this release we change RoundRobinBalancer strategy to match Java client behavior. See #1788 for details.

  • #1870 - @kvch - Update Kerberos library to latest major
  • #1876 - @bai - Update docs, reference pkg.go.dev
  • #1846 - @wclaeys - Do not ignore Consumer.Offsets.AutoCommit.Enable config on Close
  • #1747 - @XSAM - fix: mock sync producer does not handle the offset while sending messages
  • #1863 - @bai - Add support for Kafka 2.7.0 + update lz4 and klauspost/compress dependencies
  • #1788 - @kzinglzy - feat[balance_strategy]: announcing a new round robin balance strategy
  • #1862 - @bai - Fix CI setenv permissions issues
  • #1832 - @ilyakaznacheev - Update Godoc link to pkg.go.dev
  • #1822 - @danp - KIP-392: Allow consumers to fetch from closest replica

Version 1.27.2 (2020-10-21)

Improvements

#1750 - @krantideep95 Adds missing mock responses for mocking consumer group

Fixes

#1817 - reverts #1785 - Add private method to Client interface to prevent implementation

Version 1.27.1 (2020-10-07)

Improvements

#1775 - @d1egoaz - Adds a Producer Interceptor example #1781 - @justin-chen - Refresh brokers given list of seed brokers #1784 - @justin-chen - Add randomize seed broker method #1790 - @d1egoaz - remove example binary #1798 - @bai - Test against Go 1.15 #1785 - @justin-chen - Add private method to Client interface to prevent implementation #1802 - @uvw - Support Go 1.13 error unwrapping

Fixes

#1791 - @stanislavkozlovski - bump default version to 1.0.0

Version 1.27.0 (2020-08-11)

Improvements

#1466 - @rubenvp8510 - Expose kerberos fast negotiation configuration #1695 - @KJTsanaktsidis - Use docker-compose to run the functional tests #1699 - @wclaeys - Consumer group support for manually comitting offsets #1714 - @bai - Bump Go to version 1.14.3, golangci-lint to 1.27.0 #1726 - @d1egoaz - Include zstd on the functional tests #1730 - @d1egoaz - KIP-42 Add producer and consumer interceptors #1738 - @varun06 - fixed variable names that are named same as some std lib package names #1741 - @varun06 - updated zstd dependency to latest v1.10.10 #1743 - @varun06 - Fixed declaration dependencies and other lint issues in code base #1763 - @alrs - remove deprecated tls options from test #1769 - @bai - Add support for Kafka 2.6.0

Fixes

#1697 - @kvch - Use gofork for encoding/asn1 to fix ASN errors during Kerberos authentication #1744 - @alrs - Fix isBalanced Function Signature

Version 1.26.4 (2020-05-19)

Fixes

  • #1701 - @d1egoaz - Set server name only for the current broker
  • #1694 - @dnwe - testfix: set KAFKA_HEAP_OPTS for zk and kafka

Version 1.26.3 (2020-05-07)

Fixes

  • #1692 - @d1egoaz - Set tls ServerName to fix issue: either ServerName or InsecureSkipVerify must be specified in the tls.Config

Version 1.26.2 (2020-05-06)

⚠️ Known Issues

This release has been marked as not ready for production and may be unstable, please use v1.26.4.

Improvements

  • #1560 - @iyacontrol - add sync pool for gzip 1-9
  • #1605 - @dnwe - feat: protocol support for V11 fetch w/ rackID
  • #1617 - @sladkoff / @dwi-di / @random-dwi - Add support for alter/list partition reassignements APIs
  • #1632 - @bai - Add support for Go 1.14
  • #1640 - @random-dwi - Feature/fix list partition reassignments
  • #1646 - @mimaison - Add DescribeLogDirs to admin client
  • #1667 - @bai - Add support for kafka 2.5.0

Fixes

  • #1594 - @sladkoff - Sets ConfigEntry.Default flag in addition to the ConfigEntry.Source for Kafka versions > V1_1_0_0
  • #1601 - @alrs - fix: remove use of testing.T.FailNow() inside goroutine
  • #1602 - @d1egoaz - adds a note about consumer groups Consume method
  • #1607 - @darklore - Fix memory leak when Broker.Open and Broker.Close called repeatedly
  • #1613 - @wblakecaldwell - Updated "retrying" log message when BackoffFunc implemented
  • #1614 - @alrs - produce_response.go: Remove Unused Functions
  • #1619 - @alrs - tools/kafka-producer-performance: prune unused flag variables
  • #1639 - @agriffaut - Handle errors with no message but error code
  • #1643 - @kzinglzy - fix config.net.keepalive
  • #1644 - @KJTsanaktsidis - Fix brokers continually allocating new Session IDs
  • #1645 - @Stephan14 - Remove broker(s) which no longer exist in metadata
  • #1650 - @lavoiesl - Return the response error in heartbeatLoop
  • #1661 - @KJTsanaktsidis - Fix "broker received out of order sequence" when brokers die
  • #1666 - @KevinJCross - Bugfix: Allow TLS connections to work over socks proxy.

Version 1.26.1 (2020-02-04)

Improvements:

  • Add requests-in-flight metric (1539)
  • Fix misleading example for cluster admin (1595)
  • Replace Travis with GitHub Actions, linters housekeeping (1573)
  • Allow BalanceStrategy to provide custom assignment data (1592)

Bug Fixes:

  • Adds back Consumer.Offsets.CommitInterval to fix API (1590)
  • Fix error message s/CommitInterval/AutoCommit.Interval (1589)

Version 1.26.0 (2020-01-24)

New Features:

  • Enable zstd compression (1574, 1582)
  • Support headers in tools kafka-console-producer (1549)

Improvements:

  • Add SASL AuthIdentity to SASL frames (authzid) (1585).

Bug Fixes:

  • Sending messages with ZStd compression enabled fails in multiple ways (1252).
  • Use the broker for any admin on BrokerConfig (1571).
  • Set DescribeConfigRequest Version field (1576).
  • ConsumerGroup flooding logs with client/metadata update req (1578).
  • MetadataRequest version in DescribeCluster (1580).
  • Fix deadlock in consumer group handleError (1581)
  • Fill in the Fetch{Request,Response} protocol (1582).
  • Retry topic request on ControllerNotAvailable (1586).

Version 1.25.0 (2020-01-13)

New Features:

  • Support TLS protocol in kafka-producer-performance (1538).
  • Add support for kafka 2.4.0 (1552).

Improvements:

  • Allow the Consumer to disable auto-commit offsets (1164).
  • Produce records with consistent timestamps (1455).

Bug Fixes:

  • Fix incorrect SetTopicMetadata name mentions (1534).
  • Fix client.tryRefreshMetadata Println (1535).
  • Fix panic on calling updateMetadata on closed client (1531).
  • Fix possible faulty metrics in TestFuncProducing (1545).

Version 1.24.1 (2019-10-31)

New Features:

  • Add DescribeLogDirs Request/Response pair (1520).

Bug Fixes:

  • Fix ClusterAdmin returning invalid controller ID on DescribeCluster (1518).
  • Fix issue with consumergroup not rebalancing when new partition is added (1525).
  • Ensure consistent use of read/write deadlines (1529).

Version 1.24.0 (2019-10-09)

New Features:

  • Add sticky partition assignor (1416).
  • Switch from cgo zstd package to pure Go implementation (1477).

Improvements:

  • Allow creating ClusterAdmin from client (1415).
  • Set KafkaVersion in ListAcls method (1452).
  • Set request version in CreateACL ClusterAdmin method (1458).
  • Set request version in DeleteACL ClusterAdmin method (1461).
  • Handle missed error codes on TopicMetaDataRequest and GroupCoordinatorRequest (1464).
  • Remove direct usage of gofork (1465).
  • Add support for Go 1.13 (1478).
  • Improve behavior of NewMockListAclsResponse (1481).

Bug Fixes:

  • Fix race condition in consumergroup example (1434).
  • Fix brokerProducer goroutine leak (1442).
  • Use released version of lz4 library (1469).
  • Set correct version in MockDeleteTopicsResponse (1484).
  • Fix CLI help message typo (1494).

Known Issues:

  • Please don't use Zstd, as it doesn't work right now. See IBM#1252

Version 1.23.1 (2019-07-22)

Bug Fixes:

  • Fix fetch delete bug record (1425).
  • Handle SASL/OAUTHBEARER token rejection (1428).

Version 1.23.0 (2019-07-02)

New Features:

  • Add support for Kafka 2.3.0 (1418).
  • Add support for ListConsumerGroupOffsets v2 (1374).
  • Add support for DeleteConsumerGroup (1417).
  • Add support for SASLVersion configuration (1410).
  • Add kerberos support (1366).

Improvements:

  • Improve sasl_scram_client example (1406).
  • Fix shutdown and race-condition in consumer-group example (1404).
  • Add support for error codes 77—81 (1397).
  • Pool internal objects allocated per message (1385).
  • Reduce packet decoder allocations (1373).
  • Support timeout when fetching metadata (1359).

Bug Fixes:

  • Fix fetch size integer overflow (1376).
  • Handle and log throttled FetchResponses (1383).
  • Refactor misspelled word Resouce to Resource (1368).

Version 1.22.1 (2019-04-29)

Improvements:

  • Use zstd 1.3.8 (1350).
  • Add support for SaslHandshakeRequest v1 (1354).

Bug Fixes:

  • Fix V5 MetadataRequest nullable topics array (1353).
  • Use a different SCRAM client for each broker connection (1349).
  • Fix AllowAutoTopicCreation for MetadataRequest greater than v3 (1344).

Version 1.22.0 (2019-04-09)

New Features:

  • Add Offline Replicas Operation to Client (1318).
  • Allow using proxy when connecting to broker (1326).
  • Implement ReadCommitted (1307).
  • Add support for Kafka 2.2.0 (1331).
  • Add SASL SCRAM-SHA-512 and SCRAM-SHA-256 mechanismes (1331).

Improvements:

  • Unregister all broker metrics on broker stop (1232).
  • Add SCRAM authentication example (1303).
  • Add consumergroup examples (1304).
  • Expose consumer batch size metric (1296).
  • Add TLS options to console producer and consumer (1300).
  • Reduce client close bookkeeping (1297).
  • Satisfy error interface in create responses (1154).
  • Please lint gods (1346).

Bug Fixes:

  • Fix multi consumer group instance crash (1338).
  • Update lz4 to latest version (1347).
  • Retry ErrNotCoordinatorForConsumer in new consumergroup session (1231).
  • Fix cleanup error handler (1332).
  • Fix rate condition in PartitionConsumer (1156).

Version 1.21.0 (2019-02-24)

New Features:

  • Add CreateAclRequest, DescribeAclRequest, DeleteAclRequest (1236).
  • Add DescribeTopic, DescribeConsumerGroup, ListConsumerGroups, ListConsumerGroupOffsets admin requests (1178).
  • Implement SASL/OAUTHBEARER (1240).

Improvements:

  • Add Go mod support (1282).
  • Add error codes 73—76 (1239).
  • Add retry backoff function (1160).
  • Maintain metadata in the producer even when retries are disabled (1189).
  • Include ReplicaAssignment in ListTopics (1274).
  • Add producer performance tool (1222).
  • Add support LogAppend timestamps (1258).

Bug Fixes:

  • Fix potential deadlock when a heartbeat request fails (1286).
  • Fix consuming compacted topic (1227).
  • Set correct Kafka version for DescribeConfigsRequest v1 (1277).
  • Update kafka test version (1273).

Version 1.20.1 (2019-01-10)

New Features:

  • Add optional replica id in offset request (1100).

Improvements:

  • Implement DescribeConfigs Request + Response v1 & v2 (1230).
  • Reuse compression objects (1185).
  • Switch from png to svg for GoDoc link in README (1243).
  • Fix typo in deprecation notice for FetchResponseBlock.Records (1242).
  • Fix typos in consumer metadata response file (1244).

Bug Fixes:

  • Revert to individual msg retries for non-idempotent (1203).
  • Respect MaxMessageBytes limit for uncompressed messages (1141).

Version 1.20.0 (2018-12-10)

New Features:

  • Add support for zstd compression (#1170).
  • Add support for Idempotent Producer (#1152).
  • Add support support for Kafka 2.1.0 (#1229).
  • Add support support for OffsetCommit request/response pairs versions v1 to v5 (#1201).
  • Add support support for OffsetFetch request/response pair up to version v5 (#1198).

Improvements:

  • Export broker's Rack setting (#1173).
  • Always use latest patch version of Go on CI (#1202).
  • Add error codes 61 to 72 (#1195).

Bug Fixes:

  • Fix build without cgo (#1182).
  • Fix go vet suggestion in consumer group file (#1209).
  • Fix typos in code and comments (#1228).

Version 1.19.0 (2018-09-27)

New Features:

  • Implement a higher-level consumer group (#1099).

Improvements:

  • Add support for Go 1.11 (#1176).

Bug Fixes:

  • Fix encoding of MetadataResponse with version 2 and higher (#1174).
  • Fix race condition in mock async producer (#1174).

Version 1.18.0 (2018-09-07)

New Features:

  • Make Partitioner.RequiresConsistency vary per-message (#1112).
  • Add customizable partitioner (#1118).
  • Add ClusterAdmin support for CreateTopic, DeleteTopic, CreatePartitions, DeleteRecords, DescribeConfig, AlterConfig, CreateACL, ListAcls, DeleteACL (#1055).

Improvements:

  • Add support for Kafka 2.0.0 (#1149).
  • Allow setting LocalAddr when dialing an address to support multi-homed hosts (#1123).
  • Simpler offset management (#1127).

Bug Fixes:

  • Fix mutation of ProducerMessage.MetaData when producing to Kafka (#1110).
  • Fix consumer block when response did not contain all the expected topic/partition blocks (#1086).
  • Fix consumer block when response contains only constrol messages (#1115).
  • Add timeout config for ClusterAdmin requests (#1142).
  • Add version check when producing message with headers (#1117).
  • Fix MetadataRequest for empty list of topics (#1132).
  • Fix producer topic metadata on-demand fetch when topic error happens in metadata response (#1125).

Version 1.17.0 (2018-05-30)

New Features:

  • Add support for gzip compression levels (#1044).
  • Add support for Metadata request/response pairs versions v1 to v5 (#1047, #1069).
  • Add versioning to JoinGroup request/response pairs (#1098)
  • Add support for CreatePartitions, DeleteGroups, DeleteRecords request/response pairs (#1065, #1096, #1027).
  • Add Controller() method to Client interface (#1063).

Improvements:

  • ConsumerMetadataReq/Resp has been migrated to FindCoordinatorReq/Resp (#1010).
  • Expose missing protocol parts: msgSet and recordBatch (#1049).
  • Add support for v1 DeleteTopics Request (#1052).
  • Add support for Go 1.10 (#1064).
  • Claim support for Kafka 1.1.0 (#1073).

Bug Fixes:

  • Fix FindCoordinatorResponse.encode to allow nil Coordinator (#1050, #1051).
  • Clear all metadata when we have the latest topic info (#1033).
  • Make PartitionConsumer.Close idempotent (#1092).

Version 1.16.0 (2018-02-12)

New Features:

  • Add support for the Create/Delete Topics request/response pairs (#1007, #1008).
  • Add support for the Describe/Create/Delete ACL request/response pairs (#1009).
  • Add support for the five transaction-related request/response pairs (#1016).

Improvements:

  • Permit setting version on mock producer responses (#999).
  • Add NewMockBrokerListener helper for testing TLS connections (#1019).
  • Changed the default value for Consumer.Fetch.Default from 32KiB to 1MiB which results in much higher throughput in most cases (#1024).
  • Reuse the time.Ticker across fetch requests in the PartitionConsumer to reduce CPU and memory usage when processing many partitions (#1028).
  • Assign relative offsets to messages in the producer to save the brokers a recompression pass (#1002, #1015).

Bug Fixes:

  • Fix producing uncompressed batches with the new protocol format (#1032).
  • Fix consuming compacted topics with the new protocol format (#1005).
  • Fix consuming topics with a mix of protocol formats (#1021).
  • Fix consuming when the broker includes multiple batches in a single response (#1022).
  • Fix detection of PartialTrailingMessage when the partial message was truncated before the magic value indicating its version (#1030).
  • Fix expectation-checking in the mock of SyncProducer.SendMessages (#1035).

Version 1.15.0 (2017-12-08)

New Features:

  • Claim official support for Kafka 1.0, though it did already work (#984).
  • Helper methods for Kafka version numbers to/from strings (#989).
  • Implement CreatePartitions request/response (#985).

Improvements:

  • Add error codes 45-60 (#986).

Bug Fixes:

  • Fix slow consuming for certain Kafka 0.11/1.0 configurations (#982).
  • Correctly determine when a FetchResponse contains the new message format (#990).
  • Fix producing with multiple headers (#996).
  • Fix handling of truncated record batches (#998).
  • Fix leaking metrics when closing brokers (#991).

Version 1.14.0 (2017-11-13)

New Features:

  • Add support for the new Kafka 0.11 record-batch format, including the wire protocol and the necessary behavioural changes in the producer and consumer. Transactions and idempotency are not yet supported, but producing and consuming should work with all the existing bells and whistles (batching, compression, etc) as well as the new custom headers. Thanks to Vlad Hanciuta of Arista Networks for this work. Part of (#901).

Bug Fixes:

  • Fix encoding of ProduceResponse versions in test (#970).
  • Return partial replicas list when we have it (#975).

Version 1.13.0 (2017-10-04)

New Features:

  • Support for FetchRequest version 3 (#905).
  • Permit setting version on mock FetchResponses (#939).
  • Add a configuration option to support storing only minimal metadata for extremely large clusters (#937).
  • Add PartitionOffsetManager.ResetOffset for backtracking tracked offsets (#932).

Improvements:

  • Provide the block-level timestamp when consuming compressed messages (#885).
  • Client.Replicas and Client.InSyncReplicas now respect the order returned by the broker, which can be meaningful (#930).
  • Use a Ticker to reduce consumer timer overhead at the cost of higher variance in the actual timeout (#933).

Bug Fixes:

  • Gracefully handle messages with negative timestamps (#907).
  • Raise a proper error when encountering an unknown message version (#940).

Version 1.12.0 (2017-05-08)

New Features:

  • Added support for the ApiVersions request and response pair, and Kafka version 0.10.2 (#867). Note that you still need to specify the Kafka version in the Sarama configuration for the time being.
  • Added a Brokers method to the Client which returns the complete set of active brokers (#813).
  • Added an InSyncReplicas method to the Client which returns the set of all in-sync broker IDs for the given partition, now that the Kafka versions for which this was misleading are no longer in our supported set (#872).
  • Added a NewCustomHashPartitioner method which allows constructing a hash partitioner with a custom hash method in case the default (FNV-1a) is not suitable (#837, #841).

Improvements:

  • Recognize more Kafka error codes (#859).

Bug Fixes:

  • Fix an issue where decoding a malformed FetchRequest would not return the correct error (#818).
  • Respect ordering of group protocols in JoinGroupRequests. This fix is transparent if you're using the AddGroupProtocol or AddGroupProtocolMetadata helpers; otherwise you will need to switch from the GroupProtocols field (now deprecated) to use OrderedGroupProtocols (#812).
  • Fix an alignment-related issue with atomics on 32-bit architectures (#859).

Version 1.11.0 (2016-12-20)

Important: As of Sarama 1.11 it is necessary to set the config value of Producer.Return.Successes to true in order to use the SyncProducer. Previous versions would silently override this value when instantiating a SyncProducer which led to unexpected values and data races.

New Features:

  • Metrics! Thanks to Sébastien Launay for all his work on this feature (#701, #746, #766).
  • Add support for LZ4 compression (#786).
  • Add support for ListOffsetRequest v1 and Kafka 0.10.1 (#775).
  • Added a HighWaterMarks method to the Consumer which aggregates the HighWaterMarkOffset values of its child topic/partitions (#769).

Bug Fixes:

  • Fixed producing when using timestamps, compression and Kafka 0.10 (#759).
  • Added missing decoder methods to DescribeGroups response (#756).
  • Fix producer shutdown when Return.Errors is disabled (#787).
  • Don't mutate configuration in SyncProducer (#790).
  • Fix crash on SASL initialization failure (#795).

Version 1.10.1 (2016-08-30)

Bug Fixes:

  • Fix the documentation for HashPartitioner which was incorrect (#717).
  • Permit client creation even when it is limited by ACLs (#722).
  • Several fixes to the consumer timer optimization code, regressions introduced in v1.10.0. Go's timers are finicky (#730, #733, #734).
  • Handle consuming compressed relative offsets with Kafka 0.10 (#735).

Version 1.10.0 (2016-08-02)

Important: As of Sarama 1.10 it is necessary to tell Sarama the version of Kafka you are running against (via the config.Version value) in order to use features that may not be compatible with old Kafka versions. If you don't specify this value it will default to 0.8.2 (the minimum supported), and trying to use more recent features (like the offset manager) will fail with an error.

Also: The offset-manager's behaviour has been changed to match the upstream java consumer (see #705 and #713). If you use the offset-manager, please ensure that you are committing one greater than the last consumed message offset or else you may end up consuming duplicate messages.

New Features:

  • Support for Kafka 0.10 (#672, #678, #681, and others).
  • Support for configuring the target Kafka version (#676).
  • Batch producing support in the SyncProducer (#677).
  • Extend producer mock to allow setting expectations on message contents (#667).

Improvements:

  • Support nil compressed messages for deleting in compacted topics (#634).
  • Pre-allocate decoding errors, greatly reducing heap usage and GC time against misbehaving brokers (#690).
  • Re-use consumer expiry timers, removing one allocation per consumed message (#707).

Bug Fixes:

  • Actually default the client ID to "sarama" like we say we do (#664).
  • Fix a rare issue where Client.Leader could return the wrong error (#685).
  • Fix a possible tight loop in the consumer (#693).
  • Match upstream's offset-tracking behaviour (#705).
  • Report UnknownTopicOrPartition errors from the offset manager (#706).
  • Fix possible negative partition value from the HashPartitioner (#709).

Version 1.9.0 (2016-05-16)

New Features:

  • Add support for custom offset manager retention durations (#602).
  • Publish low-level mocks to enable testing of third-party producer/consumer implementations (#570).
  • Declare support for Golang 1.6 (#611).
  • Support for SASL plain-text auth (#648).

Improvements:

  • Simplified broker locking scheme slightly (#604).
  • Documentation cleanup (#605, #621, #654).

Bug Fixes:

  • Fix race condition shutting down the OffsetManager (#658).

Version 1.8.0 (2016-02-01)

New Features:

  • Full support for Kafka 0.9:
    • All protocol messages and fields (#586, #588, #590).
    • Verified that TLS support works (#581).
    • Fixed the OffsetManager compatibility (#585).

Improvements:

  • Optimize for fewer system calls when reading from the network (#584).
  • Automatically retry InvalidMessage errors to match upstream behaviour (#589).

Version 1.7.0 (2015-12-11)

New Features:

  • Preliminary support for Kafka 0.9 (#572). This comes with several caveats:
    • Protocol-layer support is mostly in place (#577), however Kafka 0.9 renamed some messages and fields, which we did not in order to preserve API compatibility.
    • The producer and consumer work against 0.9, but the offset manager does not (#573).
    • TLS support may or may not work (#581).

Improvements:

  • Don't wait for request timeouts on dead brokers, greatly speeding recovery when the TCP connection is left hanging (#548).
  • Refactored part of the producer. The new version provides a much more elegant solution to #449. It is also slightly more efficient, and much more precise in calculating batch sizes when compression is used (#549, #550, #551).

Bug Fixes:

  • Fix race condition in consumer test mock (#553).

Version 1.6.1 (2015-09-25)

Bug Fixes:

  • Fix panic that could occur if a user-supplied message value failed to encode (#449).

Version 1.6.0 (2015-09-04)

New Features:

  • Implementation of a consumer offset manager using the APIs introduced in Kafka 0.8.2. The API is designed mainly for integration into a future high-level consumer, not for direct use, although it is possible to use it directly. (#461).

Improvements:

  • CRC32 calculation is much faster on machines with SSE4.2 instructions, removing a major hotspot from most profiles (#255).

Bug Fixes:

  • Make protocol decoding more robust against some malformed packets generated by go-fuzz (#523, #525) or found in other ways (#528).
  • Fix a potential race condition panic in the consumer on shutdown (#529).

Version 1.5.0 (2015-08-17)

New Features:

  • TLS-encrypted network connections are now supported. This feature is subject to change when Kafka releases built-in TLS support, but for now this is enough to work with TLS-terminating proxies (#154).

Improvements:

  • The consumer will not block if a single partition is not drained by the user; all other partitions will continue to consume normally (#485).
  • Formatting of error strings has been much improved (#495).
  • Internal refactoring of the producer for code cleanliness and to enable future work (#300).

Bug Fixes:

  • Fix a potential deadlock in the consumer on shutdown (#475).

Version 1.4.3 (2015-07-21)

Bug Fixes:

  • Don't include the partitioner in the producer's "fetch partitions" circuit-breaker (#466).
  • Don't retry messages until the broker is closed when abandoning a broker in the producer (#468).
  • Update the import path for snappy-go, it has moved again and the API has changed slightly (#486).

Version 1.4.2 (2015-05-27)

Bug Fixes:

  • Update the import path for snappy-go, it has moved from google code to github (#456).

Version 1.4.1 (2015-05-25)

Improvements:

  • Optimizations when decoding snappy messages, thanks to John Potocny (#446).

Bug Fixes:

  • Fix hypothetical race conditions on producer shutdown (#450, #451).

Version 1.4.0 (2015-05-01)

New Features:

  • The consumer now implements Topics() and Partitions() methods to enable users to dynamically choose what topics/partitions to consume without instantiating a full client (#431).
  • The partition-consumer now exposes the high water mark offset value returned by the broker via the HighWaterMarkOffset() method (#339).
  • Added a kafka-console-consumer tool capable of handling multiple partitions, and deprecated the now-obsolete kafka-console-partitionConsumer (#439, #442).

Improvements:

  • The producer's logging during retry scenarios is more consistent, more useful, and slightly less verbose (#429).
  • The client now shuffles its initial list of seed brokers in order to prevent thundering herd on the first broker in the list (#441).

Bug Fixes:

  • The producer now correctly manages its state if retries occur when it is shutting down, fixing several instances of confusing behaviour and at least one potential deadlock (#419).
  • The consumer now handles messages for different partitions asynchronously, making it much more resilient to specific user code ordering (#325).

Version 1.3.0 (2015-04-16)

New Features:

  • The client now tracks consumer group coordinators using ConsumerMetadataRequests similar to how it tracks partition leadership using regular MetadataRequests (#411). This adds two methods to the client API:
    • Coordinator(consumerGroup string) (*Broker, error)
    • RefreshCoordinator(consumerGroup string) error

Improvements:

  • ConsumerMetadataResponses now automatically create a Broker object out of the ID/address/port combination for the Coordinator; accessing the fields individually has been deprecated (#413).
  • Much improved handling of OffsetOutOfRange errors in the consumer. Consumers will fail to start if the provided offset is out of range (#418) and they will automatically shut down if the offset falls out of range (#424).
  • Small performance improvement in encoding and decoding protocol messages (#427).

Bug Fixes:

  • Fix a rare race condition in the client's background metadata refresher if it happens to be activated while the client is being closed (#422).

Version 1.2.0 (2015-04-07)

Improvements:

  • The producer's behaviour when Flush.Frequency is set is now more intuitive (#389).
  • The producer is now somewhat more memory-efficient during and after retrying messages due to an improved queue implementation (#396).
  • The consumer produces much more useful logging output when leadership changes (#385).
  • The client's GetOffset method will now automatically refresh metadata and retry once in the event of stale information or similar (#394).
  • Broker connections now have support for using TCP keepalives (#407).

Bug Fixes:

  • The OffsetCommitRequest message now correctly implements all three possible API versions (#390, #400).

Version 1.1.0 (2015-03-20)

Improvements:

  • Wrap the producer's partitioner call in a circuit-breaker so that repeatedly broken topics don't choke throughput (#373).

Bug Fixes:

  • Fix the producer's internal reference counting in certain unusual scenarios (#367).
  • Fix the consumer's internal reference counting in certain unusual scenarios (#369).
  • Fix a condition where the producer's internal control messages could have gotten stuck (#368).
  • Fix an issue where invalid partition lists would be cached when asking for metadata for a non-existant topic (#372).

Version 1.0.0 (2015-03-17)

Version 1.0.0 is the first tagged version, and is almost a complete rewrite. The primary differences with previous untagged versions are:

  • The producer has been rewritten; there is now a SyncProducer with a blocking API, and an AsyncProducer that is non-blocking.
  • The consumer has been rewritten to only open one connection per broker instead of one connection per partition.
  • The main types of Sarama are now interfaces to make depedency injection easy; mock implementations for Consumer, SyncProducer and AsyncProducer are provided in the github.com/Shopify/sarama/mocks package.
  • For most uses cases, it is no longer necessary to open a Client; this will be done for you.
  • All the configuration values have been unified in the Config struct.
  • Much improved test suite.