Partialy and temporary disable vector configs

[alexeynl]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

For parser development purpose i have a second Vector instance with copy of the main instance config. During development stage I make a copy of main instance config to dev instance. I need easy way to disable some part of configuration of dev instanstance:

1. To skip validation of that parts that will not changed during development.
2. To avoid read and ingest data from/to prod systems.

Attempted Solutions

1. Commenting some part of configuration on dev Vector instances.
   Drawbacks:

• I need revert back changes when move config back to main instance. This causes a lot work and mistakes.
• If config uses env variables Vector fails even if it is commented

3. Commenting sources on dev Vector instances.
   Drawbacks:

• This causes the validation failure for transforms used it.

Proposal

1. Add ability to exclude some directories from directories structure passed to --config-dir. Add option --exclude-dir with list of dirs that will be excluded from loading.

For example i a have the following directory structure of my vector config:

_main
folder1
--parser1
----subparser1
--parser2
----subparser1
----subparser2
folder2
--parser1
----subparser2

I want to enable all configs during startup except
whole folder1/parser2
only folder1/parser2/subparser1.

In this cases i will pass the following command line options:

--config-dir='_main,folder1/**,folder2/**' --exclude-dir='folder1/parser2/**'

--config-dir='_main,folder1/**,folder2/**' --exclude-dir=folder1/parser2/subparser1

2.1. For sources and sinks add option to disable it such as disable: true:

sources:
  type: kafka
  ...
  disable: true

If source is disabled Vector must not "up" port or try to read from topic (in case of kafka source) but includes this source during config validation. In case of disabled sinks Vector has not sent data to ingested system.

2.2. Add command line option to apply disable: true during startup and validation.
2.3. Add warning to Vector logs for sources and sinks those were actualy disabled.
2.4. Optionaly to prevent Vector pass validation with configuration contains disabled: true add command line option --allow-disabled=true/false with default to true. I could apply this on main Vector instance. This will force developer to enable all sources and sinks or remove it from config if it is no longer needed when config moved back from dev to main instance of Vector.

References

No response

Version

No response

[jszwedko]

Hi @alexeynl !

Thanks for opening this issue!

For (2) we actually already have an issue tracking this feature request: #18664

For (1) I think once you get into wanting sophisticated selection of which config files to load that you'd be better off using something like find to list all the files and pass those to Vector rather than adding that complexity to Vector's configuration loading itself. What do you think?