NOISSUE - Simplify docker deployment #275

rodneyosodo · 2024-01-19T14:39:10Z

What type of PR is this?

This is an optimization because it improves docker-compose deployment

What does this do?

It simplifies docker-compose deployment to use one configuration where nats is used as the MQTT broker, Message Broker and Events Store.

Which issue(s) does this PR fix/relate to?

No issue

Have you included tests for your changes?

Yes, I have tested it manually

Did you document any new/modified feature?

Yes, I have added documentation about the deployment

Notes

arvindh123 · 2024-01-22T01:08:51Z

docker/README.md

+    image: rabbitmq:3.9.20-management-alpine
+    environment:


We can add container_name for this example.

arvindh123 · 2024-01-22T01:35:50Z

@rodneyosodo
In .env file
we have at present for ES we have :

MG_ES_TYPE=${MG_MESSAGE_BROKER_TYPE}
MG_ES_URL=${MG_NATS_URL}

Instead having ${MG_MESSAGE_BROKER_TYPE} value for env varabileMG_ES_TYPE , Can we have MG_ES_TYPE value as nats/redis,like given below

MG_ES_TYPE=nats
MG_ES_URL=${MG_NATS_URL}

rodneyosodo · 2024-01-23T09:38:13Z

For now, the Event Store depends on the Message broker apart from when it is Redis. I have changed the env file to reference MG_MESSAGE_BROKER_* and documented when it is redis

ianmuchyri · 2024-01-23T09:49:30Z

docker/README.md

+
+Events store: This is the same as MESSAGE_BROKER. This can either be 'nats' or 'rabbitmq' or 'redis'. This is used by Magistrala services to store events for distributed processing. If redis is used as an events store, then rabbitmq or nats is used as a message broker.
+
+Since nats is used as both MQTT_BROKER and MESSAGE_BROKER, it is not possible to run nats as an MQTT_BROKER and nats as a MESSAGE_BROKER at the same time, this is the current depolyment strategy for Magistrala in `docker/docker-compose.yml`.


I don't quite understand this statement. It's used as both MQTT and message broker but, it's not possible to run it as a MQTT and message broker simultaneously. Isn't that contradicting?

ianmuchyri · 2024-01-23T11:04:38Z

docker/README.md

+Magistrala supports configurable MQTT broker and Message broker, which also acts as events store. Magistrala uses two types of brokers:
+
+1. MQTT_BROKER: Handles MQTT communication between MQTT adapters and message broker. This can either be 'vernemq' or 'nats'.
+2. MESSAGE_BROKER: Manages communication between adapters and Magistrala writer services. This can either be 'nats' or 'rabbitmq' or 'redis'. This is used to store messages for distributed processing.


Suggested change

2. MESSAGE_BROKER: Manages communication between adapters and Magistrala writer services. This can either be 'nats' or 'rabbitmq' or 'redis'. This is used to store messages for distributed processing.

2. MESSAGE_BROKER: Manages communication between adapters and Magistrala writer services. This can either be 'nats' or 'rabbitmq'. This is used to store messages for distributed processing.

In the next paragraph you say redis cannot be used as message broker

ianmuchyri · 2024-01-23T11:11:20Z

docker/README.md

+
+For Message brokers other than nats, you would need to build the docker images with rabbtitmq as the build tag and change the `docker/.env`. For example, to use rabbitmq as a message broker:
+
+```env


no need for this env code, as it is in the one below

https://github.com/DavidAnson/markdownlint/blob/v0.31.1/doc/md040.md

This line is a terminal command, passing with varabiles, So it can be bash, sh or anyother

docker/README.md

Signed-off-by: rodneyosodo <[email protected]>

This commit fixes the container name and restart policy for the RabbitMQ service in the docker-compose file. The container name is changed to "magistrala-rabbitmq" and the restart policy is set to "on-failure". This ensures that the RabbitMQ container has a unique and identifiable name and will automatically restart in case of failure. Signed-off-by: Rodney Osodo <[email protected]>

The system now supports the use of different message brokers and events stores by modifying the docker environment variables and updating the `.env` file. This allows for easy integration of alternative options such as RabbitMQ as the message broker. Signed-off-by: Rodney Osodo <[email protected]>

Signed-off-by: Rodney Osodo <[email protected]>

Updates the NATS and RabbitMQ URLs in the .env file to reflect the latest configurations. The README.md file is also updated to include information about the Magistrala services using an events store and lists the available options. The docker-compose.yml file is modified to include the current deployment strategy for Magistrala. It now includes configurations for the MQTT broker, message broker, and events store. The provided YAML snippet contains RabbitMQ configuration for a Docker Compose setup. Additionally, a configuration file for the Docker container "vernemq" and a network called "magistrala-base-net" are mentioned in the summary. Signed-off-by: Rodney Osodo <[email protected]>

Signed-off-by: Rodney Osodo <[email protected]>

dborovcanin · 2024-01-29T10:00:05Z

docker/README.md

+1. MQTT_BROKER: Handles MQTT communication between MQTT adapters and message broker. This can either be 'VerneMQ' or 'NATS'.
+2. MESSAGE_BROKER: Manages communication between adapters and Magistrala writer services. This can either be 'NATS' or 'RabbitMQ. This is used to store messages for distributed processing.
+
+Events store: This is the same as MESSAGE_BROKER. This can either be 'NATS' or 'RabbitMQ' or 'Redis'. This is used by Magistrala services to store events for distributed processing. If Redis is used as an events store, then RabbitMQ or NATS is used as a message broker since Redis cannot be used as a message broker.


Replace with something like: By default, we are using the same broker for messages and events. You can also remove it since it's already mentioned above.

dborovcanin · 2024-01-29T10:02:34Z

docker/README.md

+Magistrala supports configurable MQTT broker and Message broker, which also acts as an events store. Magistrala uses two types of brokers:
+
+1. MQTT_BROKER: Handles MQTT communication between MQTT adapters and message broker. This can either be 'VerneMQ' or 'NATS'.
+2. MESSAGE_BROKER: Manages communication between adapters and Magistrala writer services. This can either be 'NATS' or 'RabbitMQ. This is used to store messages for distributed processing.


It's not only writers but any internal consumer. Manages message exchange between Mainflux core, optional, and external components.

dborovcanin · 2024-01-29T10:07:01Z

docker/README.md

+MG_ES_URL=${MG_REDIS_URL}
+```
+
+For MQTT broker other than NATS, you would need to change the `docker/.env`. For example, to use VerneMQ as a MQTT broker:


Make VerneMQ a default MQTT broker.

Signed-off-by: Rodney Osodo <[email protected]>

rodneyosodo force-pushed the compose branch from d8e92bc to 4a73434 Compare January 19, 2024 14:40

arvindh123 reviewed Jan 22, 2024

View reviewed changes

rodneyosodo force-pushed the compose branch from 4a73434 to 116c5ed Compare January 23, 2024 09:38

ianmuchyri reviewed Jan 23, 2024

View reviewed changes

rodneyosodo force-pushed the compose branch from ea59978 to 7fef88e Compare January 23, 2024 12:31

dborovcanin force-pushed the compose branch from 7fef88e to 0def31b Compare January 24, 2024 14:56

rodneyosodo force-pushed the compose branch 3 times, most recently from fe1ca47 to 8822c18 Compare January 25, 2024 15:11

rodneyosodo and others added 6 commits January 29, 2024 11:11

Simplify docker deployment

474d41c

Signed-off-by: rodneyosodo <[email protected]>

Fix typo in MESSAGE_BROKER description

cb01fc6

Signed-off-by: Rodney Osodo <[email protected]>

change code block to bash

fee8008

Signed-off-by: Rodney Osodo <[email protected]>

rodneyosodo force-pushed the compose branch from 8822c18 to fee8008 Compare January 29, 2024 08:12

dborovcanin suggested changes Jan 29, 2024

View reviewed changes

change docs to default vernemq as MQTT broker

5a84c10

Signed-off-by: Rodney Osodo <[email protected]>

dborovcanin approved these changes Jan 29, 2024

View reviewed changes

fix message broker and events store context

7c31eaa

Signed-off-by: Rodney Osodo <[email protected]>

dborovcanin merged commit 562c578 into absmach:main Jan 29, 2024
4 checks passed

rodneyosodo deleted the compose branch January 30, 2024 09:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NOISSUE - Simplify docker deployment #275

NOISSUE - Simplify docker deployment #275

rodneyosodo commented Jan 19, 2024

arvindh123 Jan 22, 2024

arvindh123 commented Jan 22, 2024

rodneyosodo commented Jan 23, 2024

ianmuchyri Jan 23, 2024

ianmuchyri Jan 23, 2024

ianmuchyri Jan 23, 2024 •

edited

Loading

rodneyosodo Jan 23, 2024

arvindh123 Jan 26, 2024 •

edited

Loading

dborovcanin Jan 29, 2024

dborovcanin Jan 29, 2024

dborovcanin Jan 29, 2024


		Events store: This is the same as MESSAGE_BROKER. This can either be 'nats' or 'rabbitmq' or 'redis'. This is used by Magistrala services to store events for distributed processing. If redis is used as an events store, then rabbitmq or nats is used as a message broker.

		Since nats is used as both MQTT_BROKER and MESSAGE_BROKER, it is not possible to run nats as an MQTT_BROKER and nats as a MESSAGE_BROKER at the same time, this is the current depolyment strategy for Magistrala in `docker/docker-compose.yml`.

	2. MESSAGE_BROKER: Manages communication between adapters and Magistrala writer services. This can either be 'nats' or 'rabbitmq' or 'redis'. This is used to store messages for distributed processing.
	2. MESSAGE_BROKER: Manages communication between adapters and Magistrala writer services. This can either be 'nats' or 'rabbitmq'. This is used to store messages for distributed processing.


		For Message brokers other than nats, you would need to build the docker images with rabbtitmq as the build tag and change the `docker/.env`. For example, to use rabbitmq as a message broker:

		```env

NOISSUE - Simplify docker deployment #275

NOISSUE - Simplify docker deployment #275

Conversation

rodneyosodo commented Jan 19, 2024

What type of PR is this?

What does this do?

Which issue(s) does this PR fix/relate to?

Have you included tests for your changes?

Did you document any new/modified feature?

Notes

arvindh123 Jan 22, 2024

Choose a reason for hiding this comment

arvindh123 commented Jan 22, 2024

rodneyosodo commented Jan 23, 2024

ianmuchyri Jan 23, 2024

Choose a reason for hiding this comment

ianmuchyri Jan 23, 2024

Choose a reason for hiding this comment

ianmuchyri Jan 23, 2024 • edited Loading

Choose a reason for hiding this comment

rodneyosodo Jan 23, 2024

Choose a reason for hiding this comment

arvindh123 Jan 26, 2024 • edited Loading

Choose a reason for hiding this comment

dborovcanin Jan 29, 2024

Choose a reason for hiding this comment

dborovcanin Jan 29, 2024

Choose a reason for hiding this comment

dborovcanin Jan 29, 2024

Choose a reason for hiding this comment

ianmuchyri Jan 23, 2024 •

edited

Loading

arvindh123 Jan 26, 2024 •

edited

Loading