Automatically set the microgrid (and other) API host/port

[leandro-lucarella-frequenz]

What's needed?

We need the SDK to be able to connect to the microgrid API without SDK users passing them explicitly.

Proposed solution

Load the config from a file. The config file could come from the UI or not depending on if we want to allow this or not. If we don't allow it, we need to have a mechanism to make sure this config file is included when deploying apps.

The user might be able to override this host/port, but it is still undecided. @thomas-nicolai-frequenz mention we should consider the security implications of this before enabling it.

We could also use env variables for this.

Use cases

No response

Alternatives and workarounds

No response

Additional context

This could supersede #40.

[daniel-zullo-frequenz]

Probably we should align this issue with #191 which is still under discussion

[thomas-nicolai-frequenz]

We need the SDK to be able to connect to the microgrid API without SDK users passing them explicitly.

Whats on my mind here is that user are able to simple configure an app to run against different API endpoints. On a local test system the user might want to point to the cloud sandbox whereas after deployment it points to localhost on the edge controller. However, by default the host and port should be set in the SDK so that there is no config required. The config would only be used as an explicit override imho.

[leandro-lucarella-frequenz]

I think both play well together. The config at the end will come from a config file, which might be written by the cloud sync (production) or by the developer manually (testing).

I'm still not sure if (UI) users should be able to change the host/port of the microgrid API via the UI, if they don't, then we don't even have to go through the config system.

[thomas-nicolai-frequenz]

I'm still not sure if (UI) users should be able to change the host/port of the microgrid API via the UI

hell no. SECURITY ALERT! :-)

[leandro-lucarella-frequenz]

The more i look at this the more i think this should come from env variables. If there is no UI config there is no strong requirement for this to be in s config file, then env variables are the simplest option and if it becomes annoying we can add support to load them from an .env file.

Then when can inject the values for each microgrid when building or running the docker image and we are set.

[mathias-baumann-frequenz]

Sounds like a good first step at least

[thomas-nicolai-frequenz]

then env variables are the simplest option and if it becomes annoying we can add support to load them from an .env file.

Is that convenient for a user that wants to use the SDK locally on his computer without a container against the sandbox and/or if he uses the container locally? I'm not disagreeing just trying to understand how this could potentially work in a local environment / sandbox test environment. What about that if being given in the env variables it would override the default in the SDK or the other way around?

[leandro-lucarella-frequenz]

then env variables are the simplest option and if it becomes annoying we can add support to load them from an .env file.

Is that convenient for a user that wants to use the SDK locally on his computer without a container against the sandbox and/or if he uses the container locally?

Yes, very much. And pretty standard.There are plenty of tools to even set these automatically, from cli tools (like direnv, where you can configure a directory to automatically load certain env variables when you cd to it) to support for loading dotenv files directly in probably every possible language (js, py, ruby, c, rust, etc.).

I'm not disagreeing just trying to understand how this could potentially work in a local environment / sandbox test environment. What about that if being given in the env variables it would override the default in the SDK or the other way around?

The environment variable always wins. I'm not even sure if the SDK should provide any default. What would be a sensible default? localhost and some port we chose? Maybe, but I would be happy even if the SDK refuses to start if it's started without either the app passing the host, port in microgrid.initialize() or if the env variables are not present.

[leandro-lucarella-frequenz]

So, what I suggest is to keep the current initialize() but making the host and port optional. The user at the end wants to run some App, not the SDK, so the user could even decide to hardcode the host and port when calling initialize() for testing, but when this is deployed, we can still inject the correct host and port as defined by the controller (or microgrid configuration when we have more than one controller and the microgrid services runs only in one of these controllers) when deployed for real.

So the SDK initialize() should do something like:

def initialize(host = None, port = None, ...):
    if host is None:
        host = os.getenv("MICROGRID_HOST")
        if host is None:
            raise InitializationError("Missing MICROGRID_HOST env variable")
    if port is None:
        port = os.getenv("MICROGRID_PORT")
        if port is None:
            raise InitializationError("Missing MICROGRID_PORT env variable")
        port = int(port)  # Also raise if it's not an int
    ...

Alternatively, if we really want to have a SDK provided default:

DEFAULT_MICROGRID_HOST = 'localhost'
DEFAULT_MICROGRID_PORT = '9999'

def initialize(host = None, port = None, ...):
    if host is None:
        host = os.getenv("MICROGRID_HOST", DEFAULT_MICROGRID_HOST)
    if port is None:
        port = os.getenv("MICROGRID_PORT", DEFAULT_MICROGRID_PORT)
        port = int(port)
    ...

[thomas-nicolai-frequenz]

localhost and some port we chose

exactly, yes.

[thomas-nicolai-frequenz]

Maybe, but I would be happy even if the SDK refuses to start if it's started without either the app passing the host, port in microgrid.initialize() or if the env variables are not present.

yeah, also a way to do it.

[leandro-lucarella-frequenz]

IMHO if we are going to provide or open source some real or mock microgrid service that the user can run locally, then it makes sense to provide default that matches the default when running this service in localhost. Otherwise not sure if it makes sense to provide a default, as the user will have to configure it explicitly to some external sandbox anyway and when deploying in controllers it should also always be set explicitly.

[thomas-nicolai-frequenz]

IMHO if we are going to provide or open source some real or mock microgrid service that the user can run locally

No, thats exactly not the idea. We want to provide a sandbox environment in the cloud where one can run simple or complex simulations with historic data.

[leandro-lucarella-frequenz]

Then users will have to configure the host and port of the sandbox anyway (and probably credentials too), so a default doesn't look very useful.

[thomas-nicolai-frequenz]

Keep in mind we will have to configure many hosts and ports and I'm not sure its a good idea needing to provide them all as env variables. Any thoughts on that?

[daniel-zullo-frequenz]

Then we could use a single env variable to store multiple hosts and ports (e.g. "hostA:portA,hostB:portB,hostC:portC") in that case. Though we should still set the default host and port as separate env variables and also include them in the list.

[leandro-lucarella-frequenz]

Nah, I think if we use env vars, it should be one per API. Again, env vars can be abstracted away in a config file too, namely .env, which is basically a simple key-value format:

MICROGRID_API_HOST=...
MICROGRID_API_PORT=...
REPORTING_API_HOST=...
REPORTING_API_PORT=...

For me it would be also fine to go with a .toml file to configure the SDK. It is just a convention to use .env files set default values for env variables.

For me the advantage of env vars over a plain config file is the possibility to easily override an API when running an App:

MICROGRID_API_HOST=localhost ./my-app

Vs.

./my-app --microgrid-api-host=localhost

This requires:

1. Parsing command-line arguments in every app
2. Forwarding those options to sdk.initialize() (so the interface of sdk.initialize() will need to provide setting all of this (this might be a good idea anyway, but in this case is the only option)

Vs. the env var approach which only requires the SDK to use os.env['MICROGRID_API_HOST'] to get the host.

Optionally, apps (or the SDK, but in this case I think it might make more sense to have it in the apps) can just use from dotenv import load_dotenv; load_dotenv() to parse and load a .env file. Or the SDK could parse a toml file.

Also when deploying, all the kubernetes stuff is usually based on env vars, it is very easy to provide env vars, while adding config files to images requires a tiny bit of more work.

Also, a config file is not mutually exclusive with this, the SDK could still parse a config file if one exist, and env vars will override that.

[daniel-zullo-frequenz]

ok, I agree env variables are suitable in any environment if we only have simple key-value pairs like in this case. However the toml files will be a better choice if at some point we need to represent more complex configurations like nested data structures or different data types.

[daniel-zullo-frequenz]

Also, a config file is not mutually exclusive with this, the SDK could still parse a config file if one exist, and env vars will override that.

This will cover the current use-case and future ones

[thomas-nicolai-frequenz]

I like it.

[leandro-lucarella-frequenz]

ok, I agree env variables are suitable in any environment if we only have simple key-value pairs like in this case. However the toml files will be a better choice if at some point we need to represent more complex configurations like nested data structures or different data types.

True. I think we have 2 types of configuration possible in the SDK. Most dynamic configuration should come from the UI, and should be handled by the ConfigManager actor (for example, the global resampling period could be configured this way, or timeouts for when talking to external APIs). In there even when at the end we'll probably still have key-value pairs (as this is all the UI supports for now), by having naming conventions in the keys we could map it to a more structured config (that is the plan right now).

But the API config is not something the user needs to know it even exists when using the UI, the UI needs to "inject" these values depending on in which controller an app is being installed on. And this will probably need to be done in Dockerfiles, and if it's not, it is still a purely deployment issue, this is why using common devops practices (like using env vars and dotfiles) could be beneficial, as it is a common idiom for those use cases.

The user only need to care about this when doing local development, so it is very useful to have an easy way to override it and point to some sandbox instead. This won't happen with other config variables.

[llucax]

This will come from environment variables.