docker-compose.yml doesn't contain important directives and definitions

[kkovaletp]

Hi Team,

I've recently deployed the product (4.6.0) using the docker-compose approach and had to make noticeable changes in the compose file, as it doesn't provide the best possible setup out of the box. Of course, we could negotiate in the comments about my next proposals, but I think that they are valuable enough to be mentioned here:

• Before proposing changes to the compose file, I'd like to propose a change to your docker image release strategy: now users have to subscribe to the repo releases notification and manually upgrade to the new product version. Is there some real technical reason for that? It would be much more convenient to have a stable tag for the latest LTS version, a release tag for the latest released one, and a latest for the latest published one in your docker hub repo - this will allow those users, who want to always stay on some of the mentioned branches, follow corresponding tag in their setup configs and reduce the amount of human factor and manual work.
• restart: unless-stopped would be much better than always, as it works the same, except for the case when the service was stopped intentionally and shouldn't be restarted again.
• stop_grace_period: 30s or some better value lets application components gracefully finish the execution before the container is stopped
• the next section ensures better compatibility on the system with AppArmor or SELinux enabled, which should be the main scenario, I guess.

security_opt:
       - seccomp:unconfined
       - apparmor:unconfined

• The devices: section might be useful to map some physical resources of the host (CPU, GPU, etc) and speed up the service. Of course, the application components need to utilize such hardware and work with the /dev/ directly - only in this case it would be useful
• there is no healthcheck defined in the compose file, not in the Dockerfile for images, so users couldn't use built-in functionality to monitor the health of the containers, and, more importantly, the dependency from the dashboard to indexer unable to ensure the application is ready, so, instead it checks only that the container is up and running.
• to continue the healthcheck topic, the condition: service_healthy under the wazuh.indexer in the depends_on: section of the dashboard service ensures the app is ready by parsing the healthcheck results of the dependency container
• the next section of the compose file allows the solution to utilize the IPv6 capabilities in case it is working on the host:

networks:
   wazuh_net:
     enable_ipv6: true
     ipam:
       config:
         - subnet: 172.20.0.0/16
         - subnet: 2001:0DB8::/64

Of course, the docker engine has to be configured to allow IPv6 communication by adding the next lines into the /etc/docker/daemon.json:

{
  "experimental": true,
  "ip6tables": true
}

Also, each service in the compose needs to have the same network connected by adding the next lines to the service definition:

networks:
       - wazuh_net

• the volumes section could be enhanced by specifying the path on the host, where the data is stored. It is much better than the default docker system location of all volumes because it is not hidden from the user and the user can manage where to store that files and how to backup them. The example:

   wazuh_api_configuration:
     driver: local
     driver_opts:
       type: 'none'
       o: 'bind'
       device: './volumes/wazuh_api_configuration'

Of course, all mentioned paths have to exist before the 1st start of the solution.

• To improve simplicity and reduce the amount of manually executed commands, a Makefile can be created with scripts for all typical use-cases, like volume paths creation, backup, and more
• The next 2 volume mappings are useful to apply server's time zone and local time to the running containers, so that all logs and other data on server has the same time and users don't need to remember the difference:

   volumes:
       - "/etc/localtime:/etc/localtime:ro" ## use timezone from host
       - "/etc/timezone:/etc/timezone:ro"   ## use timezone from host

I could create a PR with such changes, but I'd like to know your point of view 1st to not do extra work, so, waiting for your replies)

[muhdidhamm]

do you have any sample of your full compose file with best practice setup? from that we can discuss either it suitable to use in production or not.

[kkovaletp]

Sure, here is my compose, based on the proposals and assumptions, listed above:

version: '3.9'

services:
   wazuh.manager:
     image: wazuh/wazuh-manager:4.6.0
     hostname: wazuh.manager
     restart: unless-stopped
     stop_grace_period: 30s
     ulimits:
       memlock:
         soft: -1
         hard: -1
       nofile:
         soft: 655360
         hard: 655360
     networks:
       - wazuh_net
     ports:
       - "1514:1514"
       - "1515:1515"
       - "514:514/udp"
       - "55000:55000"
     security_opt:
       - seccomp:unconfined
       - apparmor:unconfined
     environment:
       - INDEXER_URL=https://wazuh.indexer:9200
       - INDEXER_USERNAME=admin
       - INDEXER_PASSWORD=<password>
       - FILEBEAT_SSL_VERIFICATION_MODE=full
       - SSL_CERTIFICATE_AUTHORITIES=/etc/ssl/root-ca.pem
       - SSL_CERTIFICATE=/etc/ssl/filebeat.pem
       - SSL_KEY=/etc/ssl/filebeat.key
       - API_USERNAME=wazuh-wui
       - API_PASSWORD=<password>
     devices:
       - "/dev/dri:/dev/dri"
     volumes:
       - "/etc/localtime:/etc/localtime:ro" ## use timezone from host
       - "/etc/timezone:/etc/timezone:ro"   ## use timezone from host
       - wazuh_api_configuration:/var/ossec/api/configuration
       - wazuh_etc:/var/ossec/etc
       - wazuh_logs:/var/ossec/logs
       - wazuh_queue:/var/ossec/queue
       - wazuh_var_multigroups:/var/ossec/var/multigroups
       - wazuh_integrations:/var/ossec/integrations
       - wazuh_active_response:/var/ossec/active-response/bin
       - wazuh_agentless:/var/ossec/agentless
       - wazuh_wodles:/var/ossec/wodles
       - filebeat_etc:/etc/filebeat
       - filebeat_var:/var/lib/filebeat
       - ./config/wazuh_indexer_ssl_certs/root-ca-manager.pem:/etc/ssl/root-ca.pem
       - ./config/wazuh_indexer_ssl_certs/wazuh.manager.pem:/etc/ssl/filebeat.pem
       - ./config/wazuh_indexer_ssl_certs/wazuh.manager-key.pem:/etc/ssl/filebeat.key
       - ./config/wazuh_cluster/wazuh_manager.conf:/wazuh-config-mount/etc/ossec.conf

   wazuh.indexer:
     image: wazuh/wazuh-indexer:4.6.0
     hostname: wazuh.indexer
     restart: unless-stopped
     stop_grace_period: 30s
     networks:
       - wazuh_net
     ports:
       - "9200:9200"
     security_opt:
       - seccomp:unconfined
       - apparmor:unconfined
     environment:
       - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
     ulimits:
       memlock:
         soft: -1
         hard: -1
       nofile:
         soft: 65536
         hard: 65536
     devices:
       - "/dev/dri:/dev/dri"
     volumes:
       - "/etc/localtime:/etc/localtime:ro" ## use timezone from host
       - "/etc/timezone:/etc/timezone:ro"   ## use timezone from host
       - wazuh-indexer-data:/var/lib/wazuh-indexer
       - ./config/wazuh_indexer_ssl_certs/root-ca.pem:/usr/share/wazuh-indexer/certs/root-ca.pem
       - ./config/wazuh_indexer_ssl_certs/wazuh.indexer-key.pem:/usr/share/wazuh-indexer/certs/wazuh.indexer.key
       - ./config/wazuh_indexer_ssl_certs/wazuh.indexer.pem:/usr/share/wazuh-indexer/certs/wazuh.indexer.pem
       - ./config/wazuh_indexer_ssl_certs/admin.pem:/usr/share/wazuh-indexer/certs/admin.pem
       - ./config/wazuh_indexer_ssl_certs/admin-key.pem:/usr/share/wazuh-indexer/certs/admin-key.pem
       - ./config/wazuh_indexer/wazuh.indexer.yml:/usr/share/wazuh-indexer/opensearch.yml
       - ./config/wazuh_indexer/internal_users.yml:/usr/share/wazuh-indexer/opensearch-security/internal_users.yml

   wazuh.dashboard:
     image: wazuh/wazuh-dashboard:4.6.0
     hostname: wazuh.dashboard
     restart: unless-stopped
     stop_grace_period: 10s
     networks:
       - wazuh_net
     ports:
       - "8443:5601"
     security_opt:
       - seccomp:unconfined
       - apparmor:unconfined
     environment:
       - INDEXER_USERNAME=admin
       - INDEXER_PASSWORD=<password>
       - WAZUH_API_URL=https://wazuh.manager
       - DASHBOARD_USERNAME=kibanaserver
       - DASHBOARD_PASSWORD=<password>
       - API_USERNAME=wazuh-wui
       - API_PASSWORD=<password>
     devices:
       - "/dev/dri:/dev/dri"
     volumes:
       - "/etc/localtime:/etc/localtime:ro" ## use timezone from host
       - "/etc/timezone:/etc/timezone:ro"   ## use timezone from host
       - ./config/wazuh_indexer_ssl_certs/wazuh.dashboard.pem:/usr/share/wazuh-dashboard/certs/wazuh-dashboard.pem
       - ./config/wazuh_indexer_ssl_certs/wazuh.dashboard-key.pem:/usr/share/wazuh-dashboard/certs/wazuh-dashboard-key.pem
       - ./config/wazuh_indexer_ssl_certs/root-ca.pem:/usr/share/wazuh-dashboard/certs/root-ca.pem
       - ./config/wazuh_dashboard/opensearch_dashboards.yml:/usr/share/wazuh-dashboard/config/opensearch_dashboards.yml
       - ./config/wazuh_dashboard/wazuh.yml:/usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml
       - wazuh-dashboard-config:/usr/share/wazuh-dashboard/data/wazuh/config
       - wazuh-dashboard-custom:/usr/share/wazuh-dashboard/plugins/wazuh/public/assets/custom
     depends_on:
       - wazuh.indexer
        # condition: service_healthy
     links:
       - wazuh.indexer:wazuh.indexer
       - wazuh.manager:wazuh.manager

networks:
   wazuh_net:
     enable_ipv6: true
     ipam:
       config:
         - subnet: 172.20.0.0/16
         - subnet: 2001:0DB8::/64

volumes:
   wazuh_api_configuration:
     driver: local
     driver_opts:
       type: 'none'
       o: 'bind'
       device: './volumes/wazuh_api_configuration'
   wazuh_etc:
     driver: local
     driver_opts:
       type: 'none'
       o: 'bind'
       device: './volumes/wazuh_etc'
   wazuh_logs:
     driver: local
     driver_opts:
       type: 'none'
       o: 'bind'
       device: './volumes/wazuh_logs'
   wazuh_queue:
     driver: local
     driver_opts:
       type: 'none'
       o: 'bind'
       device: './volumes/wazuh_queue'
   wazuh_var_multigroups:
     driver: local
     driver_opts:
       type: 'none'
       o: 'bind'
       device: './volumes/wazuh_var_multigroups'
   wazuh_integrations:
     driver: local
     driver_opts:
       type: 'none'
       o: 'bind'
       device: './volumes/wazuh_integrations'
   wazuh_active_response:
     driver: local
     driver_opts:
       type: 'none'
       o: 'bind'
       device: './volumes/wazuh_active_response'
   wazuh_agentless:
     driver: local
     driver_opts:
       type: 'none'
       o: 'bind'
       device: './volumes/wazuh_agentless'
   wazuh_wodles:
     driver: local
     driver_opts:
       type: 'none'
       o: 'bind'
       device: './volumes/wazuh_wodles'
   filebeat_etc:
     driver: local
     driver_opts:
       type: 'none'
       o: 'bind'
       device: './volumes/filebeat_etc'
   filebeat_var:
     driver: local
     driver_opts:
       type: 'none'
       o: 'bind'
       device: './volumes/filebeat_var'
   wazuh-indexer-data:
     driver: local
     driver_opts:
       type: 'none'
       o: 'bind'
       device: './volumes/wazuh-indexer-data'
   wazuh-dashboard-config:
     driver: local
     driver_opts:
       type: 'none'
       o: 'bind'
       device: './volumes/wazuh-dashboard-config'
   wazuh-dashboard-custom:
     driver: local
     driver_opts:
       type: 'none'
       o: 'bind'
       device: './volumes/wazuh-dashboard-custom'

[kkovaletp]

This YAML needs the next improvements (which require contribution of product experts):

• the stop_grace_period has to be set to some better values according to the app components shutdown time
• The devices section needs to be enhanced with devices mapping according to device utilization capabilities of app components per service
• healthcheck has to be defined here per service or in corresponding Dockerfile per image, then the condition line can be uncommented in the dependency section of the dashboard service
• IP subnets could be modified to better match the solution. Unfortunately, it is impossible to not specify them at all, so users have to review and modify them to match their docker network setup

[kkovaletp]

I've just added 1 more useful proposal to the original issue and to the compose file:

• The next 2 volume mappings are useful to apply server's time zone and local time to the running containers, so that all logs and other data on server has the same time and users don't need to remember the difference:

   volumes:
       - "/etc/localtime:/etc/localtime:ro" ## use timezone from host
       - "/etc/timezone:/etc/timezone:ro"   ## use timezone from host

[BAStos525]

Any updates regarding healthchecks?

[schneich]

I was desperately looking for an example like yours. Unfortunately, I did find it just now, after having spend 3 nights on creating my own custom yml. It is very similar to yours, with some light differences. (my docker compose.yml)

• I have added a top level name, so that volumes and networks get named after that.
• I run my Docker containers in my home environment in a Docker macvlan. (-> containers get published with a mac address directly into my Unifi home network vlan and they are visible, as if they are servers on their own. Dashboard and Manager get their own IP, as they need to be reachable from within my network. Downside: port mapping becomes irrelevant.)
• I have added a second Docker internal network. All traffic between the containers is routed over this and does not bother my home network. This traffic is isolated from all other containers in my environment.
• I have added a container_name to be able to give them other names
• For the Timezone, I added the following - but I have no yet checked, if it works at all. :-)

    environment:
      - TZ=Europe/Zurich

[ilsaloving]

Has anyone at Wazuh actually tried following the docker deployment instructions & docker compose files? The provided instructions cannot possibly produce a production-worthy environment, assuming one can make it work at all.

To give another example:
The docker-compose.yaml file (either single or multi) does not contain any links to internal-users.yml, so if you go through the password change instructions, you're going to have a bad time.

Anyone new to Wazuh is going to have a really hard time getting this set up. Heck, even with familiarity of Wazuh, getting this going is going to be challenging, as the docker instructions are very different (and missing a lot of details) compared to the standard installation instructions.

[tuomotalvitie]

I'm having the same bad time as #1528 and probably also looking at fresh install. What I am wondering here is if the proposals here would have helped, and will I avoid the problems in the future updates by applying any of it. I'm slightly leaning towards no, as the update from 4.7 to 4.8 was somewhat painless, and I do have the internal_users mount.

While trying to figure this out, I may (or may not have) noticed an issue with the dashboard mounts (including the above improved definitions). Wazuh.yml in config is defined before the mount of the directory it is in for the dashboard. Based on the commit logs they were originally #968 (and are currently in master) the other way round (but not in any of the tags).

      - ./config/wazuh_dashboard/wazuh.yml:/usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml
       - wazuh-dashboard-config:/usr/share/wazuh-dashboard/data/wazuh/config

However, changes to the mounted file did seem to propagate into the container, so I guess this is not an actual problem, and/or my understanding of the mount order is flawed. I'm unsure if this is defined behaviour, or if I just didn't find the part of docker documentation referencing this case. My understanding is that as the wazuh-dashboard-config is created empty, which then copies the /usr/share/wazuh-dashboard/data/wazuh/config contents (based on Dockerfile only wazuh.yml) into the volume. Which would then subsequently, next time, shadow (because it is not empty) the wazuh.yml (and possible changes). At least without the explicit mount(?)

Master branch probably addresses this, but decided to write my musings down just in case this currently relies on some undefined behaviour (and perhaps I'll be pointed to some part of docs I missed if not)

From https://docs.docker.com/engine/storage/volumes/

If you mount a non-empty volume into a directory in the container in which files or directories exist, the pre-existing files are obscured by the mount.

If you mount an empty volume into a directory in the container in which files or directories exist, these files or directories are propagated (copied) into the volume by default.