Merge pull request #238 from LD4P/update-readme

README.md

@@ -4,24 +4,32 @@
 
 # Sinopia Indexing Pipeline
 
-This is the repository for the Sinopia Indexing Pipeline. The pipeline is a Node application that listens for messages (sent to a queue via STOMP), and for each message (a W3C Activity Streams message):
+This is the repository for the Sinopia Indexing Pipeline. The pipeline is a Node application that
+responds to MongoDB [Change Streams](https://docs.mongodb.com/manual/changeStreams/) and
+indexes parts of the document into ElasticSearch:
 
-* Parses a subject URI out of the message. This URI corresponds to a Trellis resource that has been changed (add, edited, deleted).
-* Dereferences the URI, asking for a JSON-LD representation.
+* Extracts the JSON-LD representation from the `data` property in the MongoDB document
 * Indexes the JSON-LD, or a derivative thereof, in ElasticSearch.
 
-The pipeline also includes a `bin/reindex` command that will wipe all ElasticSearch indices and reindex Trellis by crawling the tree of resources contained within Trellis.
+The pipeline also includes a `bin/reindex` command that will wipe all ElasticSearch indices and reindex by crawling MongoDB.
 
-**Note** that if the Trellis platform is running in a container, the reindexer will also need to run in a container, else it will fail to resolve Trellis's internal hostname. In that event, run `docker-compose run reindexer` instead.
 
 ## Testing
 
-Using `docker-compose`, you can spin up containers for Trellis, ActiveMQ, ElasticSearch, Postgres, and the pipeline::
+Using `docker-compose`, you can spin up containers for MongoDB, ElasticSearch, and
+the pipeline:
 
 ```shell
 $ docker-compose up pipeline # add -d to run in background
 ```
 
+You should also run the MongoDB setup to create the collections and load the
+supporting files for resource templates:
+
+```shell
+$ docker-compose run mongo-setup
+```
+
 To shut it down and clean up, run:
 
 ```shell
@@ -41,38 +49,17 @@ $ npm run lint
 $ npm test
 ```
 
-To run the integration tests, they must be invoked independent of the unit tests:
-
-```shell
-$ npm run integration
-```
-
-**NOTE**: The `pipeline` `docker-compose` service must be running for the integration tests to pass.
-
 ### Continuous Integration
 
-We are using CircleCI to run continuous integration. CircleCI invokes the integration tests using a container, which works around inter-container networking constraints in the CI environment. If you prefer to run integration tests in a manner that more closely matches what runs in CI, you can do that via:
-
-```shell
-$ docker-compose run integration
-```
-
-### Create a Trellis resource
-
-To create a Trellis container and test integration between the pipeline components, you may do so using a curl incantation like follows:
-
-```shell
-$ curl -i -X POST -H 'Content-Type: application/ld+json' -H 'Link: <http://www.w3.org/ns/ldp#BasicContainer>; rel="type"' -H "Slug: repository" -d '{ "@context": { "dcterms": "http://purl.org/dc/terms/" }, "@id": "", "dcterms:title": "Repository container" }' http://localhost:8080
-```
-
-See [Sinopia Server notes](https://github.com/LD4P/sinopia_server/wiki/Draft-Notes-for-Sinopia-Server-API-Spec) for more Trellis `curl` incantations.
+We are using CircleCI to run continuous integration. 
 
 ## Development
 
-For development purposes, you may wish to spin up all the components other than the pipeline if you'll be iterating:
+For development purposes, you may wish to spin up all the components other than the
+pipeline:
 
 ```shell
-$ docker-compose up -d platform search searchui
+$ docker-compose up -d mongo search searchui
 ```
 
 And then spin up the pipeline using:

docker-compose.yml

@@ -1,58 +1,31 @@
 version: "3"
 services:
-  broker:
-    image: rmohr/activemq
-    ports:
-      - 61613:61613
-      - 8161:8161
-  platform:
-    image: ld4p/trellis-ext-db:latest
-    environment:
-      TRELLIS_BASE_URL: http://platform:8080
-      TRELLIS_LOGGING_LEVEL: INFO
-      TRELLIS_CONSOLE_LOGGING_THRESHOLD: INFO
-      COGNITO_USER_POOL_ID: ${COGNITO_USER_POOL_ID:-us-west-2_CGd9Wq136}
-      AWS_REGION: ${AWS_REGION:-us-west-2}
-    ports:
-      - 8080:8080
-      - 8081:8081
-    depends_on:
-      - database
-      - broker
-      - migration
-  platformdata:
-    image: ld4p/sinopia_acl:latest
-    environment:
-      INSIDE_CONTAINER: 'true'
-      TRELLIS_BASE_URL: http://platform:8080
-      COGNITO_USER_POOL_ID: ${COGNITO_USER_POOL_ID:-us-west-2_CGd9Wq136}
-      AWS_REGION: ${AWS_REGION:-us-west-2}
-      COGNITO_ADMIN_PASSWORD: "${COGNITO_ADMIN_PASSWORD}" # add to .env file (DO NOT CHECK IN)
-      AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID}" # add to .env file (DO NOT CHECK IN)
-      AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY}" # add to .env file (DO NOT CHECK IN)
-    command: dockerize -wait tcp://platform:8080 -timeout 3m bin/migrate
-    depends_on:
-      - platform
   pipeline:
     build:
       context: .
     environment:
       INDEX_URL: http://search:9200
       BROKER_HOST: broker
     depends_on:
-      - broker
       - search
-      - platform
-  integration:
-    build:
-      context: .
+      - mongo
+  mongo:
+    image: mongo:4.4
+    ports:
+      - 27017:27017
+      - 28017:28017
     environment:
-      INDEX_URL: http://search:9200
-      BROKER_HOST: broker
-      INSIDE_CONTAINER: 'true'
-    command: dockerize -wait tcp://broker:61613 -wait tcp://search:9200 -wait tcp://platform:8080 -timeout 3m npm run integration
+      MONGO_INITDB_ROOT_USERNAME: sinopia
+      MONGO_INITDB_ROOT_PASSWORD: sekret
+    entrypoint: [ "/usr/bin/mongod", "--bind_ip_all", "--replSet", "rs0" ]
+  mongo-setup:
+    image: ld4p/sinopia_dev_setup:latest
+    # Replace image with build to build from local code. Do not commit docker-compose.yml.
+    # build:
+    #   context: ../sinopia_api
+    #   dockerfile: Dockerfile-setup
     depends_on:
-      - pipeline
+      - mongo
   reindexer:
     build:
       context: .
@@ -64,15 +37,6 @@ services:
     command: bin/reindex
     depends_on:
       - pipeline
-  database:
-    image: postgres:11
-    environment:
-      POSTGRES_USER: sinopia
-      POSTGRES_DB: sinopia
-      POSTGRES_PASSWORD: S1n0p1a
-      PGDATA: /var/lib/postgresql/data/pgdata/mydata
-    ports:
-      - 5432:5432
   search:
     image: ld4p/sinopia_elasticsearch:latest
     entrypoint:
@@ -94,8 +58,3 @@ services:
       - 1358:1358
     depends_on:
       - search
-  migration:
-    image: ld4p/trellis-ext-db:latest
-    command: ["/opt/trellis/bin/trellis-db", "db", "migrate", "/opt/trellis/etc/config.yml"]
-    depends_on:
-      - database