[receiver/kafkareceiver] add topic_regex configuration option #36909
Conversation
|
|
wojtekzyla
force-pushed
the
feat/kafkareceiver-regex-topics
branch
from
c862f18 to
46dfdd9
Compare
receiver/kafkareceiver/config.go
Outdated
| @@ -96,5 +99,8 @@ var _ component.Config = (*Config)(nil) | |||
|
|
|||
| // Validate checks the receiver configuration is valid | |||
| func (cfg *Config) Validate() error { | |||
| if len(cfg.Topic) > 0 && len(cfg.TopicRegex) > 0 { | |||
There was a problem hiding this comment.
I wonder if it is more appropriate to use the logic of priority: If TopicRegex is not empty, then ignore Topic
There was a problem hiding this comment.
This might be right I changed that. I removed this validation rule. Remaining code works in such a way that when TopicRegex is configured we start looking for new topics which match the regex and if they are detected we overwrite list of topics that we subscribe to: link
wojtekzyla
force-pushed
the
feat/kafkareceiver-regex-topics
branch
2 times, most recently
from
4d7678d to
ec5d0d9
Compare
update changelog when both TopicRegex and Topic are configured, ignore Topic Update config_test.go bring back empty Validate() in config.go test literal topic settings and update documentation
wojtekzyla
force-pushed
the
feat/kafkareceiver-regex-topics
branch
from
ec5d0d9 to
0eb64bd
Compare
There was a problem hiding this comment.
For this kind of change an issue is required so there can be a discussion on it.
I have concerns on future maintainability of the receiver including a feature like this and potential expectations on how it should work.
My preference is a feature like this should be done within the receiver creator using an Observer and not to complicate the Kafka configuration further.
Description
By adding
topic_regexoption in configuration we can allow kafka topics subscription based on name pattern instead of literal name string. Existingtopicoption still remains, so users can choose which solution they prefer.Testing
Unit tests and manual testing
Documentation
Added new record in README.md