Merge pull request #17 from CDOT-CV/mongo-script-permission-fix

Mongo Connector Fixes

README.md

@@ -151,9 +151,16 @@ This needs to be put after the createSink function definition.
 
 ## Quick Run
 1. Create a copy of `sample.env` and rename it to `.env`.
-2. Update the variable `DOCKER_HOST_IP` to the local IP address of the system running docker.
+2. Update the variable `DOCKER_HOST_IP` to the local IP address of the system running docker and set an admin user password with the `MONGO_DB_PASS` variable.
    1. If connecting to a separately deployed mongo cluster make sure to specify the `MONGO_IP` and `MONGO_PORT`.
 3. Navigate back to the root directory and run the following command: `docker compose -f docker-compose-mongo.yml up -d`
 4. Using either a local kafka install or [kcat](https://github.com/edenhill/kcat) to produce a sample message to one of the sink topics. Optionally, you can separately run the [ODE](https://github.com/usdot-jpo-ode/jpo-ode) and process messages directly from it's output.
-5. Using [MongoDB Compass](https://www.mongodb.com/products/compass) or another DB visualizer connect to the MongoDB database using this connection string: `mongodb://localhost:27017`
-6. Now we are done! If everything is working properly you should see an ODE database with a collection for each kafka sink topic that contains messages.
+5. Using [MongoDB Compass](https://www.mongodb.com/products/compass) or another DB visualizer connect to the MongoDB database using this connection string: `mongodb://[admin_user]:[admin_password]@localhost:27017/`
+6. Now we are done! If everything is working properly you should see an ODE database with a collection for each kafka sink topic that contains messages.
+
+## Debugging
+If the Kafka connect image crashes with the following error:
+``` bash
+bash: /scripts/connect_wait.sh: /bin/bash^M: bad interpreter: No such file or directory
+```
+Please verify that the line endings in the ([connect_start.sh](./mongo-connector/connect_start.sh)) and ([connect_wait.sh](./mongo-connector/connect_wait.sh)) are set to LF instead of CRLF.

docker-compose-mongo.yml

@@ -14,11 +14,10 @@ services:
       KAFKA_ADVERTISED_HOST_NAME: ${DOCKER_HOST_IP}
       KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
       KAFKA_CREATE_TOPICS: "topic.OdeSpatJson:1:1,topic.OdeBsmJson:1:1,topic.OdeTimJson:1:1,topic.OdePsmJson:1:1,topic.OdeRawEncodedPsmJson,topic.OdeRawEncodedBSMJson:1:1,topic.OdeRawEncodedSPATJson:1:1,topic.OdeRawEncodedTIMJson:1:1,topic.OdeRawEncodedMAPJson:1:1,topic.OdeMapJson:1:1,topic.OdeRawEncodedPSMJson:1:1,topic.OdePsmJson:1:1"
+      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true" # Allows kakfa connect to create its own topics
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock
 
-  # WARNING: This is a development instance. Do not use this in production.
-  # There is no Authentication or Encryption to this MongoDB instance.
   mongodb_container:
     image: mongo:6
     environment:
@@ -55,14 +54,14 @@ services:
       CONNECT_REST_ADVERTISED_HOST_NAME: connect
       CONNECT_REST_PORT: 8083
       CONNECT_GROUP_ID: compose-connect-group
-      CONNECT_CONFIG_STORAGE_TOPIC: kafka-connect-configs
+      CONNECT_CONFIG_STORAGE_TOPIC: topic.kafka-connect-configs
       CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
       CONNECT_CONFIG_STORAGE_CLEANUP_POLICY: compact
       CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
-      CONNECT_OFFSET_STORAGE_TOPIC: kafka-connect-offsets
+      CONNECT_OFFSET_STORAGE_TOPIC: topic.kafka-connect-offsets
       CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
       CONNECT_OFFSET_STORAGE_CLEANUP_POLICY: compact
-      CONNECT_STATUS_STORAGE_TOPIC: kafka-connect-status
+      CONNECT_STATUS_STORAGE_TOPIC: topic.kafka-connect-status
       CONNECT_STATUS_STORAGE_CLEANUP_POLICY: compact
       CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
       CONNECT_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"