Override connection config by environment variable

[farshidtz]

Hi,

It is possible to replace fields specified in configuration files for example in in kuiper.yaml (KUIPER__ prefix), mqtt_source.yaml (MQTT_SOURCE__ prefix), or even etc/edgex.yaml (EDGEX__ prefix). But that doesn't seem to be possible for connection/connection.yaml.

Here is what I've tried with no success: CONNECTION__EDGEX__REDISMSGBUS__OPTIONAL__USERNAME=my-user
to set edgex.optional.Username as:

edgex:
  redisMsgBus:
    ...
    optional:
      Username: my-user

Any suggestions?

Thanks!

[ngjaying]

This is supposed to work. Like setting the port: CONNECTION__EDGEX__REDISMSGBUS__PORT: 6379.
It may be a bug due to the Username is case sensitive here. Does it work for you to set port other than the properties inside optional?

[superrxan]

could you please check the logs by docker log edgex-kuiper

Here is my output:

time="2022-11-02 02:56:11" level=info msg="open sink node 1 instances" file="node/sink_node.go:120" rule=rule1
time="2022-11-02 02:56:11" level=info msg="Set config 'edgex.default.port' to '6379' by environment variable" file="conf/load.go:130"
time="2022-11-02 02:56:11" level=info msg="Set config 'edgex.default.topic' to 'rules-events' by environment variable" file="conf/load.go:130"
time="2022-11-02 02:56:11" level=info msg="Set config 'connection.edgex.redismsgbus.protocol' to 'redis' by environment variable" file="conf/load.go:130"
time="2022-11-02 02:56:11" level=info msg="Set config 'connection.edgex.redismsgbus.server' to 'edgex-redis' by environment variable" file="conf/load.go:130"
time="2022-11-02 02:56:11" level=info msg="Set config 'connection.edgex.redismsgbus.optional.username' to 'redis' by environment variable" file="conf/load.go:130"
time="2022-11-02 02:56:11" level=info msg="Set config 'connection.edgex.redismsgbus.type' to 'redis' by environment variable" file="conf/load.go:130"
time="2022-11-02 02:56:11" level=info msg="Set config 'connection.edgex.redismsgbus.port' to '6379' by environment variable" file="conf/load.go:130"
time="2022-11-02 02:56:11" level=info msg="Set config 'connection.edgex.redismsgbus.optional.password' to '*' by environment variable" file="conf/load.go:130"
time="2022-11-02 02:56:11" level=info msg="Set config 'edgex.default.protocol' to 'redis' by environment variable" file="conf/load.go:130"
time="2022-11-02 02:56:11" level=info msg="Set config 'edgex.default.server' to 'edgex-redis' by environment variable" file="conf/load.go:130"
time="2022-11-02 02:56:11" level=info msg="Set config 'edgex.default.type' to 'redis' by environment variable" file="conf/load.go:130"
time="2022-11-02 02:56:11" level=info msg="Use configuration for edgex messagebus {{edgex-redis 6379 redis} {edgex-redis 6379 redis} redis map[Password:* Username:redis]}" file="edgex/edgex.go:54"
time="2022-11-02 02:56:11" level=info msg="Init client wrapper for client type edgex and connection selector edgex.redismsgbus" file="clients/client_registy.go:97"
time="2022-11-02 02:56:11" level=info msg="open source node with props map[format:json key: messageType:request optional:map[KeepAlive:50] port:6379 protocol:redis server:edgex-redis topic:edgex/events/# type:redis], concurrency: 1, bufferLength: 102400" file="node/source_node.go:116" rule=rule1
time="2022-11-02 02:56:11" level=info msg="Use configuration for edgex messagebus {{edgex-redis 6379 redis} {edgex-redis 6379 redis} redis map[KeepAlive:50]}" file="edgex/edgex.go:54"
time="2022-11-02 02:56:11" level=info msg="Init client wrapper for client type edgex" file="clients/client_registy.go:106"
time="2022-11-02 02:56:11" level=info msg="Start source demo instance 0 successfully" file="node/source_node.go:149" rule=rule1
time="2022-11-02 02:56:11" level=info msg="new subscription for topic edgex/events/#, reqId is $$source_pool_edgex.demo_source_pool_edgex.demo_0" file="edgex/edgex_wrapper.go:184" source_pool=edgex.demo

if you saw the output like this, then it set the values correctly

time="2022-11-02 02:56:11" level=info msg="Set config 'connection.edgex.redismsgbus.optional.username' to 'redis' by environment variable" file="conf/load.go:130"
time="2022-11-02 02:56:11" level=info msg="Set config 'connection.edgex.redismsgbus.type' to 'redis' by environment variable" file="conf/load.go:130"
time="2022-11-02 02:56:11" level=info msg="Set config 'connection.edgex.redismsgbus.port' to '6379' by environment variable" file="conf/load.go:130"
time="2022-11-02 02:56:11" level=info msg="Set config 'connection.edgex.redismsgbus.optional.password' to '*' by environment variable" file="conf/load.go:130"

The reason I can think of it does not take effect is you need reference the connnection info in your stream
for example, in your envoronment you need set , please check this for more info
https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sources/builtin/edgex.md#connectionselector

EDGEX__DEFAULT__CONNECTIONSELECTOR=edgex.redisMsgBus

or if you use edgex sink, you also need set the connectionSelector, please check this https://github.com/lf-edge/ekuiper/blob/master/docs/en_US/rules/sinks/builtin/edgex.md#connection-reuse-publish-example

[MonicaisHer]

Hey, I tried to override connection.yaml by setting connection.edgex.redismsgbus.port=6666. However, from the logs, I only saw the message Set config 'connection.edgex.redismsgbus.port' to '6666' by environment variable, and no further related logs appeared, indicating that the override did not get applied at all.

Later, I discovered that if I set a rule at the same time, the overrides of the connection would be applied. Then, I could see the expected logs, such as Use configuration for edgex messagebus {{localhost 6666 redis} {localhost 6666 redis} redis map[Password:* Username:default]}}.

For your information, in my case, the connectionSelector was already set to edgex.redisMsgBus initially. Although the overrides take effect immediately on other config files such as kuiper.yaml and edgex.yaml, they only take effect on connection.yaml depending on the initiation of a rule.

Here are the steps that I have set for the example above:

1. Setup

snap install ./edgex-ekuiper_1.7.3+snap+git9.fa58b1a-dirty_amd64.snap --dangerous
snap install edgexfoundry --edge
snap connect edgexfoundry:edgex-secretstore-token edgex-ekuiper:edgex-secretstore-token

2. Override connection

snap set edgex-ekuiper config.connection.edgex.redismsgbus.port=6666
snap start edgex-ekuiper

logs:

Apr 03 18:01:19 ubuntu edgex-ekuiper.kuiperd[81212]: time="2023-04-03 18:01:19" level=info msg="Set config 'connection.edgex.redismsgbus.port' to '6666' by environment variable" file="conf/load.go:130"

3. Initial a stream and rule

edgex-ekuiper.kuiper create stream demo '() WITH (TYPE="edgex")'

edgex-ekuiper.kuiper create rule rule1 '
{
  "sql": "SELECT * FROM demo",
  "actions": [{
    "memory": {
      "topic": "home1"
    }
  }]
}'

logs:

Apr 03 18:02:52 ubuntu edgex-ekuiper.kuiperd[81212]: time="2023-04-03 18:02:52" level=info msg="open source node with props map[connectionSelector:edgex.redisMsgBus format:json key: messageType:request optional:map[Password:*** Username:default] port:6379 protocol:redis server:localhost topic:edgex/events/# type:redis], concurrency: 1, bufferLength: 102400" file="node/source_node.go:116" rule=rule1
Apr 03 18:02:52 ubuntu edgex-ekuiper.kuiperd[81212]: time="2023-04-03 18:02:52" level=info msg="Set config 'connection.edgex.redismsgbus.port' to '6666' by environment variable" file="conf/load.go:130"
Apr 03 18:02:52 ubuntu edgex-ekuiper.kuiperd[81212]: time="2023-04-03 18:02:52" level=info msg="Use configuration for edgex messagebus {{localhost 6666 redis} {localhost 6666 redis} redis map[Password:* Username:default]}" file="edgex/edgex.go:54"
Apr 03 18:02:52 ubuntu edgex-ekuiper.kuiperd[81212]: time="2023-04-03 18:02:52" level=info msg="Init client wrapper for client type edgex and connection selector edgex.redismsgbus" file="clients/client_registy.go:97"