Add Kafka heartbeat interval and session timeout to component reference #4074

denisbchrsk · 2024-03-06T10:14:54Z

Thank you for helping make the Dapr documentation better!

Please follow this checklist before submitting:

Commits are signed with Developer Certificate of Origin (DCO - learn more)
Read the contribution guide
Commands include options for Linux, MacOS, and Windows within codetabs
New file and folder names are globally unique
Page references use shortcodes instead of markdown or URL links
Images use HTML style and have alternative text
Places where multiple code/command options are given have codetabs

In addition, please fill out the following to help reviewers understand this pull request:

Description

Added documentation on the addition of session timeout and heartbeat interval in Kafka's component references (binding and pubsub).

Relevant to the PR that allows these parameters to be configurable in dapr/components-contrib#3375.

Issue reference

…rence Signed-off-by: denisbchrsk <[email protected]>

…erence Signed-off-by: denisbchrsk <[email protected]>

msfussell · 2024-03-06T17:28:19Z

daprdocs/content/en/reference/components-reference/supported-bindings/kafka.md

@@ -78,6 +82,8 @@ spec:
 | `saslMechanism` | N | Input/Output | The SASL authentication mechanism you'd like to use. Only required if `authtype` is set to `"password"`. If not provided, defaults to `PLAINTEXT`, which could cause a break for some services, like Amazon Managed Service for Kafka. | `"SHA-512", "SHA-256", "PLAINTEXT"` |
 | `initialOffset`   | N | Input | The initial offset to use if no offset was previously committed. Should be "newest" or "oldest". Defaults to "newest". | `"oldest"` |
 | `maxMessageBytes` | N | Input/Output | The maximum size in bytes allowed for a single Kafka message. Defaults to 1024. | `"2048"` |
+| `heartbeatInterval` | N | The interval between heartbeats to the consumer coordinator. Defaults to 3s. | `5s` |


Any guidance that you would consider adding here on when to change this setting?

Added some guidance per confluent's documentation (https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#heartbeat-interval-ms), do you think that this comment is now also relevant to sessionTimeout?

Signed-off-by: denisbchrsk <[email protected]>

…eat-params Signed-off-by: denisbchrsk <[email protected]>

Signed-off-by: denisbchrsk <[email protected]>

denisbchrsk · 2024-03-07T09:15:10Z

Fixed, also changed base branch to v1.14 after the release of v1.13.
Also, this should probably be reviewed after dapr/components-contrib#3375 is approved and merged.

denisbchrsk · 2024-03-21T21:08:15Z

dapr/components-contrib#3375 has been merged, @msfussell can you take a look at this again?

msfussell · 2024-04-12T01:12:20Z

daprdocs/content/en/reference/components-reference/supported-bindings/kafka.md

@@ -78,6 +82,8 @@ spec:
 | `saslMechanism` | N | Input/Output | The SASL authentication mechanism you'd like to use. Only required if `authtype` is set to `"password"`. If not provided, defaults to `PLAINTEXT`, which could cause a break for some services, like Amazon Managed Service for Kafka. | `"SHA-512", "SHA-256", "PLAINTEXT"` |
 | `initialOffset`   | N | Input | The initial offset to use if no offset was previously committed. Should be "newest" or "oldest". Defaults to "newest". | `"oldest"` |
 | `maxMessageBytes` | N | Input/Output | The maximum size in bytes allowed for a single Kafka message. Defaults to 1024. | `"2048"` |
+| `heartbeatInterval` | N | Input | The interval between heartbeats to the consumer coordinator. The value should be set to at most a 1/3 of `sessionTimeout`'s value. Defaults to 3s. | `5s` |
+| `sessionTimeout`    | N | Input | The maximum time between heartbeats before the consumer is considered inactive and will timeout. Defaults to 10s. | `30s` |


Do you really mean "maximum time between heartbeats" here? Shouldn't this say "Maximum session time before all heartbeats with the heartbeatInterval stop" or similar? I would take some of the working from the Kafka documentation https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#session-timeout-ms

After considering this for a bit, I do agree that it's not exactly clear, so I changed to the following (taking part of Kafka's documentation):
"The timeout used to detect client failures when using Kafka’s group management facility. If the broker fails to receive any heartbeats from the consumer before the expiration of this session timeout, then the consumer will be removed and initiate a rebalance."

I could also take most of the explanation on the sessionTimeout from confluent's documentation entirely instead of trying to change it.

…ation Signed-off-by: denisbchrsk <[email protected]>

Signed-off-by: denisbchrsk <[email protected]>

denisbchrsk · 2024-04-16T20:00:55Z

It seems that in this PR #4099 which was merged recently, added the documentation of sessionTimeout and heartbeatInterval as is, directly from the component's metadata.yaml (without the suggested changes in this PR), so I'm assuming for this PR that I'll continue to try to improve the explanation on those fields.

hhunter-ms

quick review

hhunter-ms · 2024-06-18T19:50:27Z

daprdocs/content/en/reference/components-reference/supported-bindings/kafka.md

@@ -93,8 +97,8 @@ spec:
 | `clientConnectionTopicMetadataRefreshInterval` | N | Input/Output | The interval for the client connection's topic metadata to be refreshed with the broker as a Go duration. Defaults to `9m`. | `"4m"` |
 | `clientConnectionKeepAliveInterval` | N | Input/Output | The maximum time for the client connection to be kept alive with the broker, as a Go duration, before closing the connection. A zero value (default) means keeping alive indefinitely. | `"4m"` |
 | `consumerFetchDefault` | N | Input/Output | The default number of message bytes to fetch from the broker in each request. Default is `"1048576"` bytes. | `"2097152"` |
-| `heartbeatInterval` | N | Input/Output | The interval between heartbeats to the consumer coordinator for the client connection. Default is `"3s"`. | `"5s"` |
-| `sessionTimeout` | N | Input/Output | The maximum time between heartbeats before the consumer is considered inactive and will timeout. Default is `"10s"`. | `"20s"` |
+| `heartbeatInterval` | N | Input | The interval between heartbeats to the consumer coordinator. The value should be set to at most a 1/3 of `sessionTimeout`'s value. Defaults to `"3s"`. | `"5s"` |


hhunter-ms · 2024-06-18T19:51:05Z

daprdocs/content/en/reference/components-reference/supported-bindings/kafka.md

-| `heartbeatInterval` | N | Input/Output | The interval between heartbeats to the consumer coordinator for the client connection. Default is `"3s"`. | `"5s"` |
-| `sessionTimeout` | N | Input/Output | The maximum time between heartbeats before the consumer is considered inactive and will timeout. Default is `"10s"`. | `"20s"` |
+| `heartbeatInterval` | N | Input | The interval between heartbeats to the consumer coordinator. The value should be set to at most a 1/3 of `sessionTimeout`'s value. Defaults to `"3s"`. | `"5s"` |
+| `sessionTimeout` | N | Input | The timeout used to detect client failures when using Kafka’s group management facility. If the broker fails to receive any heartbeats from the consumer before the expiration of this session timeout, then the consumer will be removed and initiate a rebalance. Defaults to `"10s"`. | `"20s"` |


Suggested change

| `sessionTimeout` | N | Input | The timeout used to detect client failures when using Kafka’s group management facility. If the broker fails to receive any heartbeats from the consumer before the expiration of this session timeout, then the consumer will be removed and initiate a rebalance. Defaults to `"10s"`. | `"20s"` |

| `sessionTimeout` | N | Input | The timeout used to detect client failures when using Kafka’s group management facility. If the broker fails to receive any heartbeats from the consumer before the expiration of this session timeout, then the consumer is removed and initiates a rebalance. Defaults to `"10s"`. | `"20s"` |

hhunter-ms · 2024-06-18T19:51:50Z

daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md

@@ -106,8 +110,8 @@ spec:
 | clientConnectionTopicMetadataRefreshInterval | N | The interval for the client connection's topic metadata to be refreshed with the broker as a Go duration. Defaults to `9m`. | `"4m"` |
 | clientConnectionKeepAliveInterval | N | The maximum time for the client connection to be kept alive with the broker, as a Go duration, before closing the connection. A zero value (default) means keeping alive indefinitely. | `"4m"` |
 | consumerFetchDefault | N | The default number of message bytes to fetch from the broker in each request. Default is `"1048576"` bytes. | `"2097152"` |
-| heartbeatInterval | N | The interval between heartbeats to the consumer coordinator for the client connection. Default is `"3s"`. | `"5s"` |
-| sessionTimeout | N | The maximum time between heartbeats before the consumer is considered inactive and will timeout. Default is `"10s"`. | `"20s"` |
+| heartbeatInterval | N | The interval between heartbeats to the consumer coordinator. The value should be set to at most a 1/3 of `sessionTimeout`'s value. Defaults to "3s". | `"5s"` |


Suggested change

| heartbeatInterval | N | The interval between heartbeats to the consumer coordinator. The value should be set to at most a 1/3 of `sessionTimeout`'s value. Defaults to "3s". | `"5s"` |

| heartbeatInterval | N | The interval between heartbeats to the consumer coordinator. At most, the value should be set to a 1/3 of the `sessionTimeout` value. Defaults to "3s". | `"5s"` |

hhunter-ms · 2024-06-18T19:52:29Z

daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md

-| heartbeatInterval | N | The interval between heartbeats to the consumer coordinator for the client connection. Default is `"3s"`. | `"5s"` |
-| sessionTimeout | N | The maximum time between heartbeats before the consumer is considered inactive and will timeout. Default is `"10s"`. | `"20s"` |
+| heartbeatInterval | N | The interval between heartbeats to the consumer coordinator. The value should be set to at most a 1/3 of `sessionTimeout`'s value. Defaults to "3s". | `"5s"` |
+| sessionTimeout | N | The timeout used to detect client failures when using Kafka’s group management facility. If the broker fails to receive any heartbeats from the consumer before the expiration of this session timeout, then the consumer will be removed and initiate a rebalance. Defaults to "10s". | `"20s"` |


Suggested change

| sessionTimeout | N | The timeout used to detect client failures when using Kafka’s group management facility. If the broker fails to receive any heartbeats from the consumer before the expiration of this session timeout, then the consumer will be removed and initiate a rebalance. Defaults to "10s". | `"20s"` |

| sessionTimeout | N | The timeout used to detect client failures when using Kafka’s group management facility. If the broker fails to receive any heartbeats from the consumer before the expiration of this session timeout, then the consumer is removed and initiates a rebalance. Defaults to "10s". | `"20s"` |

msfussell · 2024-06-19T18:48:50Z

@denisbchrsk - Can you please take a look at this review. Would like to get this merged.

msfussell · 2024-06-22T05:21:43Z

@denisbchrsk - Another ping here. Thanks

Signed-off-by: denisbchrsk <[email protected]>

denisbchrsk · 2024-06-22T06:57:58Z

@msfussell Apologies for the delay, I've added the changes requested by the review above.

marcduiker · 2024-08-15T12:10:06Z

@holopin-bot @denisbchrsk Thank you!

holopin-bot · 2024-08-15T12:10:10Z

Congratulations @denisbchrsk, the maintainer of this repository has issued you a badge! Here it is: https://holopin.io/claim/clzv8nfsr158360clcqrbbi1fy

This badge can only be claimed by you, so make sure that your GitHub account is linked to your Holopin account. You can manage those preferences here: https://holopin.io/account.
Or if you're new to Holopin, you can simply sign up with GitHub, which will do the trick!

denisbchrsk added 2 commits March 6, 2024 09:41

docs: add session timeout and heartbeat interval to kafka pubsub refe…

830aeb2

…rence Signed-off-by: denisbchrsk <[email protected]>

docs: add session timeout and heartbeat interval to kafka binding ref…

5430bde

…erence Signed-off-by: denisbchrsk <[email protected]>

denisbchrsk requested review from a team as code owners March 6, 2024 10:14

denisbchrsk mentioned this pull request Mar 6, 2024

Kafka: Add support to configure heartbeat interval and session timeout to kafka's consumer dapr/components-contrib#3375

Merged

3 tasks

msfussell reviewed Mar 6, 2024

View reviewed changes

denisbchrsk added 3 commits March 7, 2024 08:37

docs: add recommendation to how to set heartbeat interval

3ba6c9b

Signed-off-by: denisbchrsk <[email protected]>

Merge remote-tracking branch 'origin/v1.14' into feature/kafka-heartb…

6416a20

…eat-params Signed-off-by: denisbchrsk <[email protected]>

docs: small change to heartbeat interval doc

071e2af

Signed-off-by: denisbchrsk <[email protected]>

denisbchrsk changed the base branch from v1.13 to v1.14 March 7, 2024 09:04

msfussell added the waiting-on-code-pr The code PR needs to be merged before the docs are updated label Mar 8, 2024

msfussell assigned denisbchrsk Mar 8, 2024

msfussell reviewed Apr 12, 2024

View reviewed changes

msfussell and others added 3 commits April 11, 2024 20:15

Merge branch 'v1.14' into feature/kafka-heartbeat-params

e27b9be

docs: update sessionTimeout and take new merged changes into consider…

17e3670

…ation Signed-off-by: denisbchrsk <[email protected]>

docs: remove redundant doc lines

40dcaad

Signed-off-by: denisbchrsk <[email protected]>

msfussell removed the waiting-on-code-pr The code PR needs to be merged before the docs are updated label Jun 17, 2024

msfussell added this to the 1.14 milestone Jun 17, 2024

msfussell added the area/components label Jun 17, 2024

hhunter-ms requested changes Jun 18, 2024

View reviewed changes

Merge branch 'v1.14' into feature/kafka-heartbeat-params

b648b2b

docs: fix heartbeatInterval and sessionTimeout

84ad437

Signed-off-by: denisbchrsk <[email protected]>

Merge branch 'v1.14' into feature/kafka-heartbeat-params

7c17a24

hhunter-ms approved these changes Jun 25, 2024

View reviewed changes

msfussell merged commit a079545 into dapr:v1.14 Jun 25, 2024
6 checks passed

jjcollinge mentioned this pull request Aug 5, 2024

Add subscribe.v1alpha1 to the API allow list docs #4297

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Kafka heartbeat interval and session timeout to component reference #4074

Add Kafka heartbeat interval and session timeout to component reference #4074

denisbchrsk commented Mar 6, 2024 •

edited

Loading

msfussell Mar 6, 2024

denisbchrsk Mar 7, 2024

denisbchrsk commented Mar 7, 2024

denisbchrsk commented Mar 21, 2024

msfussell Apr 12, 2024 •

edited

Loading

denisbchrsk Apr 16, 2024

denisbchrsk commented Apr 16, 2024

hhunter-ms left a comment

hhunter-ms Jun 18, 2024

hhunter-ms Jun 18, 2024

hhunter-ms Jun 18, 2024

hhunter-ms Jun 18, 2024

msfussell commented Jun 19, 2024

msfussell commented Jun 22, 2024

denisbchrsk commented Jun 22, 2024

marcduiker commented Aug 15, 2024

holopin-bot bot commented Aug 15, 2024

	\| `heartbeatInterval` \| N \| Input \| The interval between heartbeats to the consumer coordinator. The value should be set to at most a 1/3 of `sessionTimeout`'s value. Defaults to `"3s"`. \| `"5s"` \|
	\| `heartbeatInterval` \| N \| Input \| The interval between heartbeats to the consumer coordinator. At most, the value should be set to a 1/3 of the `sessionTimeout` value. Defaults to `"3s"`. \| `"5s"` \|

	\| `sessionTimeout` \| N \| Input \| The timeout used to detect client failures when using Kafka’s group management facility. If the broker fails to receive any heartbeats from the consumer before the expiration of this session timeout, then the consumer will be removed and initiate a rebalance. Defaults to `"10s"`. \| `"20s"` \|
	\| `sessionTimeout` \| N \| Input \| The timeout used to detect client failures when using Kafka’s group management facility. If the broker fails to receive any heartbeats from the consumer before the expiration of this session timeout, then the consumer is removed and initiates a rebalance. Defaults to `"10s"`. \| `"20s"` \|

	\| heartbeatInterval \| N \| The interval between heartbeats to the consumer coordinator. The value should be set to at most a 1/3 of `sessionTimeout`'s value. Defaults to "3s". \| `"5s"` \|
	\| heartbeatInterval \| N \| The interval between heartbeats to the consumer coordinator. At most, the value should be set to a 1/3 of the `sessionTimeout` value. Defaults to "3s". \| `"5s"` \|

	\| sessionTimeout \| N \| The timeout used to detect client failures when using Kafka’s group management facility. If the broker fails to receive any heartbeats from the consumer before the expiration of this session timeout, then the consumer will be removed and initiate a rebalance. Defaults to "10s". \| `"20s"` \|
	\| sessionTimeout \| N \| The timeout used to detect client failures when using Kafka’s group management facility. If the broker fails to receive any heartbeats from the consumer before the expiration of this session timeout, then the consumer is removed and initiates a rebalance. Defaults to "10s". \| `"20s"` \|

Add Kafka heartbeat interval and session timeout to component reference #4074

Add Kafka heartbeat interval and session timeout to component reference #4074

Conversation

denisbchrsk commented Mar 6, 2024 • edited Loading

Description

Issue reference

msfussell Mar 6, 2024

Choose a reason for hiding this comment

denisbchrsk Mar 7, 2024

Choose a reason for hiding this comment

denisbchrsk commented Mar 7, 2024

denisbchrsk commented Mar 21, 2024

msfussell Apr 12, 2024 • edited Loading

Choose a reason for hiding this comment

denisbchrsk Apr 16, 2024

Choose a reason for hiding this comment

denisbchrsk commented Apr 16, 2024

hhunter-ms left a comment

Choose a reason for hiding this comment

hhunter-ms Jun 18, 2024

Choose a reason for hiding this comment

hhunter-ms Jun 18, 2024

Choose a reason for hiding this comment

hhunter-ms Jun 18, 2024

Choose a reason for hiding this comment

hhunter-ms Jun 18, 2024

Choose a reason for hiding this comment

msfussell commented Jun 19, 2024

msfussell commented Jun 22, 2024

denisbchrsk commented Jun 22, 2024

marcduiker commented Aug 15, 2024

holopin-bot bot commented Aug 15, 2024

denisbchrsk commented Mar 6, 2024 •

edited

Loading

msfussell Apr 12, 2024 •

edited

Loading