Added support for listening for an MQTT message to do an “on demand” sync. #517

davidabacon · 2025-02-03T23:44:58Z

OBJECTIVE

My changes are minor - just added a subroutine to listen for MQTT message (only if MQT_BROKER_URL is defined). Upon receipt of the message, a sync is fired.

Hopefully this could be of use to the wider community.

jjb14778 · 2025-02-20T19:37:54Z

well i simple did this to make a trigged sync between multiple shared piholes as a service to make it work myself easy. This is based on my R&D work on it:

to start, KEEP it SIMPLE!!!

perhaps build it in orbital-sync with a config setting that enables this build in trigger service that runs the triggersync service when using in a stack with a mgmt primary pihole (using internally shared log volume with a pihole container as shown in my dns-mgmt orbital pihole docker-compose.yml example below)

i have a mgmt pihole where i manage the pihole configs localy (not docker) . (The secondary on primary pihole on docker also works, you simple need to redirect the primary pihole docker fltlog location to a volume in your compose file and use the mapped local ftllog location in the service script).

This can be build in
what is needed further development than the stuff below:
config variables for execute commands within the trigger-service that triggers the action from a script on a needed orbital-sync mgmt config docker volume for config or sripts location. As i run my dns-mgmt host local i can simple trigger the sync container start, to trigger it from a container needs more work.

i read the FTL logs and compare them so when it's changed after a update or change, it triggers the start of the orbital-sync containers i host there too with the run once config option.

Changes to make it usable yourself from in the container from my localy build R&D example:

The obvious stuff, log file locations, docker container names, or custom start command for example to use ssh and docker start to run orbital triggered sync on a other docker host for example. Or fetch the porbital-sync container output and print it to the servicee log as well for example.

How to maybe integrate it in de orbital-sync container

Create a service config file (debian/ubuntu):

sudo -i mkdir /etc/pupdate
sudo -i nano /etc/pupdate/pihole-update.service
sudo -i chown -R 0644 /etc/pupdate/pihole-update.service
sudo -i ln -s /etc/pupdate/pihole-update.service /etc/systemd/system/pihole-update.service

pihole-update.service filecontent to save

[Unit]
Description="Pi-Hole Update Trigger"
After=multi-user.target
After=network-online.target
Wants=network-online.target

[Service]
Type=idle
Restart=always
ExecStart=bash /etc/pupdate/check-pihole.sh

#log location for the Pi-Hole Update Trigger service itself
# needs updating to orbital container logoutput location
StandardOutput=file:/var/log/PiholeUpdateTrigger.log # needs updating to orbital container logoutput
StandardError=file:/var/log/PiholeUpdateTrigger-error.log
Restart=always

[Install]
WantedBy=multi-user.target

Create the Pi-Hole Update Trigger script used in the service.

Create a folder /etc/pupdate and the check-pihole.sh service-script

sudo -i nano /etc/pupdate/check-pihole.sh 
sudo -i chmod -R 0764 /etc/pupdate/check-pihole.sh

/etc/pupdate/check-pihole.sh filecontent to save

while true
do
      FILE="/var/log/pihole/FTL.log"
      tail "$FILE" -n1 > /tmp/update.new
      STRING=$(cat /tmp/update.chk)
      SUB=$(cat /tmp/update.new)
    
	while [[ "$STRING" == "$SUB" ]]
             do
      		tail "$FILE" -n1 > /tmp/update.new
      		STRING=""
      		SUB=""
      		STRING=$(cat /tmp/update.chk)
      		SUB=$(cat /tmp/update.new)
      		#echo "No Pi-hole Update Needed"
      		#echo "first entry:" $STRING
      		#echo " last entry:" $SUB
               #time for the lofile loop check to restart in seconds.
      		sleep 3;
	     done

	        echo "starting orbital sync"
	        #resetting the sync loop after triggering or not.
                tail "$FILE" -n1 > /tmp/update.chk
                tail "$FILE" -n1 > /tmp/update.new
                #executing the trigger commands below to run docker container <name> of your orbital-sync runonce container
                echo "triggered update of dns-pri.services.local"
                docker start dns-mgmt-sync-pri-1 -i # use the  SECONDARY_HOSTS_1_TRIGGERCMD for example.

                #change to MQTT message to do an “on demand” sync execute to an orbital-sync container running as MQTT.
                #perhaps with additional needed container variables to make it work and configurable.
                #use this against a triggerable orbital-sync container running MQTT triiger listening mode.
                
               # add more for extra sync containers, to sync them sequentially one after another.
               #time for the next pihole sync starts after triggering in seconds.
      		#sleep 3;
                #echo "triggered update of pihole-pri.services.local"
                #docker start dns-mgmt-sync-sec-1 -i
done
echo "End" #close the sservice script endless loop.

Register the Pi-Hole trigger service in the container to make it work! (debian/ubuntu).

Enable service

sudo -i systemctl enable pihole-update.service
# triggered service start with the propossed orbital-sync sync trigger container var PRIMARY_HOST_TRIGGER_SVC.
# make the var unusable combined with the run_once var for example.
#sudo -i systemctl start pihole-update.service

Propossed pihole orbital-sync mgmt stack example

docker-compose.yml

version: "3"

volumes:
  log-pihole: { driver: local }
  etc-pihole: { driver: local }
  etc-dnsmasq.d: { driver: local }

#networks: # ipv6
#  default: # ipv6
#    driver: host # ipv6
#    driver_opts: # ipv6
#      com.docker.network.enable_ipv6: "true" # ipv6

services:
  dns-mgmt:
    container_name: dns-mgmt
    hostname: dns-mgmt.services.local
    image: pihole/pihole:latest
    ports:
      #- "[::]:53:53/tcp" # ipv6
      #- "[::]:53:53/udp" # ipv6
      - "xxx.xxx.xxx.xxx:53:53/tcp"
      - "xxx.xxx.xxx.xxx:53:53/udp"
      - "xxx.xxx.xxx.xxx:67:67/udp" #Pi-hole as your DHCP server      
      - "xxx.xxx.xxx.xxx:80:80/tcp"
    environment:
      SKIPGRAVITYONBOOT: '1'
      QUERY_LOGGING: 'true'
      CUSTOM_CACHE_SIZE: 5000
      QUERY_LOGGING: 'false'
      PIHOLE_DNS_: '1.1.1.1;1.0.0.1;8.8.8.8;8.8.4.4'
      #PIHOLE_DNS_: '1.1.1.1;1.0.0.1;8.8.8.8;8.8.4.4;2001:4860:4860:0:0:0:0:8888;2606:4700:4700::1111' # ipv6
      DNS_BOGUS_PRIV: 'true'
      DNS_FQDN_REQUIRED: 'true'
      DNSMASQ_LISTENING: 'all'
      CORS_HOSTS: 'services.local'
      REV_SERVER: 'true'
      REV_SERVER_DOMAIN: 'services.local'
      REV_SERVER_TARGET: 'xxx.xxx.xxx.xxx' #internal service.local dnsdomain forward server ip.
      REV_SERVER_CIDR: 'xxx.xxx.xxx.xxx/xx' #CIDR subnet and range 10.0.0.0/8 for example.
      PIHOLE_DOMAIN: 'services.local'
      DNSSEC: 'false'
      IPv6: 'false'
      #IPv6: 'true' # ipv6
      TZ: 'Europe/Amsterdam'
      WEBPASSWORD: ''
    volumes:
      - 'log-pihole:/var/log/pihole' #volume location where pi-hole trigger service can work with the FLTlog.
      - 'etc-pihole:/etc/pihole'
      - 'etc-dnsmasq.d:/etc/dnsmasq.d'
    cap_add:
      - NET_ADMIN # Required if you are using Pi-hole as your DHCP server, else not needed
    restart: always

#new trigger docker feature usage example, with this stuff build in activated with the example below.
#  sync-mgmt:
#    image: mattwebbio/orbital-sync:latest
#    depends_on:
#      - dns-mgmt
#    extra_hosts:
#      - "dns-mgmt.services.local:xxx.xxx.xxx.xxx" #container dnshostname
#      - "dns-pri.services.local:xxx.xxx.xxx.xxx" #container dnshostname
#      #- "dns-sec.services.local:xxx.xxx.xxx.xxx" #container dnshostname
#    environment:
#      RUN_ONCE: 'false'
#      TZ: 'Europe/Amsterdam'
#      PRIMARY_HOST_TRIGGER_SVC: 'true'
#      PRIMARY_HOST_CONTAINER_NAME: dns-mgmt
#      SECONDARY_HOSTS_1_MQTT_TRIGGERCMD: ''
#      #SECONDARY_HOSTS_2_MQTT_TRIGGERCMD: ''
#      INTERVAL_TRIGGERNEXT_TIMEOUT: 30 #seconds between sync targets so you minimalise downtime on dns resolution.
#    volumes:
#      - 'log-pihole:/var/log/pihole' #volume location where pi-hole trigger service can work with the FLTlog on the shared stack volume.      
 
  sync-pri:
    image: mattwebbio/orbital-sync:latest
    depends_on:
      - dns-mgmt
    extra_hosts:
      - "dns-mgmt.services.local:xxx.xxx.xxx.xxx" #container dnshostname
      - "dns-pri.services.local:xxx.xxx.xxx.xxx" #container dnshostname
    environment:
      RUN_ONCE: 'true'
      TZ: 'Europe/Amsterdam'
      PRIMARY_HOST_BASE_URL: 'http://dns-mgmt' #use stack internal pihole service name resolvable without fqdn.
      PRIMARY_HOST_PASSWORD: ''
      SECONDARY_HOSTS_1_BASE_URL: 'http://dns-pri.services.local'
      SECONDARY_HOSTS_1_PASSWORD: ''
      INTERVAL_MINUTES: 1
      SYNC_V5_WHITELIST: 'true'
      SYNC_V5_REGEX_WHITELIST: 'true'
      SYNC_V5_BLACKLIST: 'true'
      SYNC_V5_REGEX_LIST: 'true'
      SYNC_V5_AD_LIST: 'true'
      SYNC_V5_CLIENT: 'true'
      SYNC_V5_GROUP: 'true'
      SYNC_V5_AUDIT_LOG: 'true'
      SYNC_V5_STATIC_DHCP_LEASES: 'true'
      SYNC_V5_LOCAL_DNS_RECORDS: 'true'
      SYNC_V5_LOCAL_CNAME_RECORDS: 'true'
      SYNC_V5_FLUSH_TABLES: 'true'
      
#  sync-sec: #sequentialy so eliminate pihole shared service availability.
#    image: mattwebbio/orbital-sync:latest
#    depends_on:
#      - dns-mgmt
#    extra_hosts:
#      - "dns-mgmt.services.local:xxx.xxx.xxx.xxx" #container internal dnshostname
#      - "dns-sec.services.local:xxx.xxx.xxx.xxx" #container internal dnshostname
#    environment:
#      RUN_ONCE: 'true'    
#      TZ: 'Europe/Amsterdam'
#      PRIMARY_HOST_BASE_URL: 'http://dns-mgmt' #use stack internal pihole service name resolvable without fqdn.
#      PRIMARY_HOST_PASSWORD: ''
#      SECONDARY_HOSTS_1_BASE_URL: 'http://dns-sec.services.local'
#      SECONDARY_HOSTS_1_PASSWORD: ''
#      INTERVAL_MINUTES: 1
#      SYNC_V5_WHITELIST: 'true'
#      SYNC_V5_REGEX_WHITELIST: 'true'
#      SYNC_V5_BLACKLIST: 'true'
#      SYNC_V5_REGEX_LIST: 'true'
#      SYNC_V5_AD_LIST: 'true'
#      SYNC_V5_CLIENT: 'true'
#      SYNC_V5_GROUP: 'true'
#      SYNC_V5_AUDIT_LOG: 'true'
#      SYNC_V5_STATIC_DHCP_LEASES: 'true'
#      SYNC_V5_LOCAL_DNS_RECORDS: 'true'
#      SYNC_V5_LOCAL_CNAME_RECORDS: 'true'
#      SYNC_V5_FLUSH_TABLES: 'true'

docker-compose.yml dns-pri stack example.

version: "3"
volumes:
  etc-pihole: { driver: local }
  etc-dnsmasq.d: { driver: local }

#networks: # ipv6
#  default: # ipv6
#    driver: host # ipv6
#    driver_opts: # ipv6
#      com.docker.network.enable_ipv6: "true" # ipv6

services:
  pihole:
    container_name: dns-pri
    hostname: dns-pri.services.local
    image: pihole/pihole:latest
    ports:
      #- "[::]:53:53/tcp" # ipv6
      #- "[::]:53:53/udp" # ipv6
      - "xxx.xxx.xxx.xxx:53:53/tcp"
      - "xxx.xxx.xxx.xxx:53:53/udp"
      - "xxx.xxx.xxx.xxx:67:67/udp" #Pi-hole as your DHCP server      
      - "xxx.xxx.xxx.xxx:80:80/tcp"
    environment:
      SKIPGRAVITYONBOOT: '1'
      QUERY_LOGGING: 'true'
      CUSTOM_CACHE_SIZE: 5000
      QUERY_LOGGING: 'false'
      PIHOLE_DNS_: '1.1.1.1;1.0.0.1;8.8.8.8;8.8.4.4'
      #PIHOLE_DNS_: '1.1.1.1;1.0.0.1;8.8.8.8;8.8.4.4;2001:4860:4860:0:0:0:0:8888;2606:4700:4700::1111' # ipv6
      DNS_BOGUS_PRIV: 'true'
      DNS_FQDN_REQUIRED: 'true'
      DNSMASQ_LISTENING: 'all'
      CORS_HOSTS: 'services.local'
      REV_SERVER: 'true'
      REV_SERVER_DOMAIN: 'services.local'
      REV_SERVER_TARGET: 'xxx.xxx.xxx.xxx' #internal service.local dnsdomain forward server ip.
      REV_SERVER_CIDR: 'xxx.xxx.xxx.xxx/xx' #CIDR subnet and range 10.0.0.0/8 for example.
      PIHOLE_DOMAIN: 'services.local'
      DNSSEC: 'false'
      IPv6: 'false'
      #IPv6: 'true' # ipv6
      TZ: 'Europe/Amsterdam'
      WEBPASSWORD: ''
    volumes:
      - 'etc-pihole:/etc/pihole'
      - 'etc-dnsmasq.d:/etc/dnsmasq.d'
    cap_add:
      - NET_ADMIN # Required if you are using Pi-hole as your DHCP server, else not needed
    restart: always

Cheerz 🍺🦝🦝🦝

do whatever you want with it. i use my own stuff until you build it better ;)
after you fix: #535 (comment)
i have minimum time to spend on this, just liked to share my simple out of the box working build idea.

davidabacon and others added 6 commits February 3, 2025 03:45

Added in code to listen for mqtt messages to do an on-demand sync

faeb8ea

Update README.md

99db0f8

Update README.md

e02ba86

Update CHANGELOG.md

257cf75

Update CHANGELOG.md

f021515

Update README.md

358c9fd

jjb14778 mentioned this pull request Feb 20, 2025

Feature Request: Disable ssl certificate check and more #534

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for listening for an MQTT message to do an “on demand” sync. #517

Added support for listening for an MQTT message to do an “on demand” sync. #517

davidabacon commented Feb 3, 2025

jjb14778 commented Feb 20, 2025 •

edited

Loading

Added support for listening for an MQTT message to do an “on demand” sync. #517

Are you sure you want to change the base?

Added support for listening for an MQTT message to do an “on demand” sync. #517

Conversation

davidabacon commented Feb 3, 2025

OBJECTIVE

jjb14778 commented Feb 20, 2025 • edited Loading

well i simple did this to make a trigged sync between multiple shared piholes as a service to make it work myself easy. This is based on my R&D work on it:

to start, KEEP it SIMPLE!!!

How to maybe integrate it in de orbital-sync container

Create a service config file (debian/ubuntu):

Create the Pi-Hole Update Trigger script used in the service.

Create a folder /etc/pupdate and the check-pihole.sh service-script

Register the Pi-Hole trigger service in the container to make it work! (debian/ubuntu).

Propossed pihole orbital-sync mgmt stack example

docker-compose.yml

docker-compose.yml dns-pri stack example.

Cheerz 🍺🦝🦝🦝

jjb14778 commented Feb 20, 2025 •

edited

Loading