Proxy

[LeitLei]

Is there a way to force LME (elastic, kibana,...) to use the system proxy for internet connection instead of direct access?

Regards
Patrick

[llwaterhouse]

You can probably force LME to use the system proxy by manually configuring it in the docker containers.

[LeitLei]

I have to edit the docker-compose-stack-live.yml and add at every "environment" block my proxy settings, yes.
Also there are some tests included in your docker-compose-stack-live.yml like "curl -k -s -I https://127.0.0.1:5601". This tests would not work unless I add the --noproxy parameter.

But as I understand, the docker-compose-stack-live.yml would be updateded with every LME update and probably this will delete my changes. Or is this not correct?

[cbaxley]

In your bash environment (.bashrc) you can use this pattern to skip proxies for certain addresses.
export http_proxy="http://proxy.example.com:8080"
export https_proxy="http://proxy.example.com:8080"
export no_proxy="localhost,127.0.0.1,192.168.0.100,192.168.0.101"

[LeitLei]

Thanks for your reply.
So just adding this to my bashrc (instead to docker-compose-stack-live.yml) would fix it?
Or do you just mean the no_proxy setting?

[cbaxley]

Yea. You can add it to your ~/.bashrc and a whole lot of processes in Linux will respect it. Curl for sure does.
For the container, you can set the environment variables in the docker-compose file and the processes in the container may respect it.

[LeitLei]

you can set the environment variables in the docker-compose file and the processes in the container may respect it
And this is exactly my problem.

Linux using my proxy. But not the container. And as I description above, it will work when I editing docker-compose-stack-live.yml (add the proxy, add --noproxy to curl).
But as I unterstand LME, the docker-compose-stack-live.yml will be rewritten every update. So I have to edit the docker-compose-stack-live.yml after every update. And this is not really userfriendly.

[aarz-snl]

you can globally edit your docker settings :

https://docs.docker.com/network/proxy/

This will apply the proxy settings to all containers running on that linux machine

[aarz-snl]

also we do provide instructions here:

https://github.com/cisagov/LME/blob/fcb7199ac04bc3930211c86761cd7d6d70406b04/docs/markdown/chapter3/chapter3.md#312-web-proxy-settings

This uses the http-proxy.conf file instead
More info on doing it like this: https://docs.docker.com/config/daemon/systemd/

Either way should work... systemd is a little more 'corporate environment -- mandate proxy usage' type of way. I'd recommend that as per our instructions.

just keep in mind this will impact all containers. So, if you happen to have something else running on this server that isn't LME -- and maybe you dont want these settings applied you will have to adjust specifically for those containers. Much easier if the server is just LME though

As long as you dont do a full uninstall of docker during upgrades (which shouldn't be the case) then these settings will always remain.

[LeitLei]

Thanks for your help @aarz-snl.
I know this section in the manual and I set these settings long time ago. But this doesn't work. I can see many access attempts to epr.elastic.co that do not use the proxy. So there is something, that doesn't accept/read the proxy settings.

[llwaterhouse]

Capturing discussion from lme-devchat:
if they edit the config.json is completely separate from the docker compose file -- is basically what i was getting at there.
12:08
it will be completely separate from our install / uninstall /upgrades of lme... its docker specific

We could also put in entries into the docker-compose file that set those variables from the env. If they have them set on the env it will set them in the containers. If they are not set, they will just be empty vars.

I was trying to think of a way to make general customizations to the docker compose file, not just the specific instance of wanting to set up proxies.

for 2.0 i would think the less 'extra directions' we have to give the better. Some folks I work with on issues would struggle with creating a config.json and implementing / understanding what theyre doing.

I think they can both easily be automated. We could just generate the config.json based on env variables. The benefit of doing it directly in the docker compose would be if they had other containers that weren't LME that they wanted different settings for it wouldn't take over the entire thing

[LeitLei]

That sounds good. Thanks @llwaterhouse