Skip to content

Getting started

Jump to bottom
ao508 edited this page Jun 7, 2022 · 7 revisions

Note: this page is a work in progress but is meant to get someone started with running a local instance of the whole SMILE network of services.

Versions:

  • docker: 20.10.2
  • java: 1.8.0_261
  • mvn: 3.6.1

Contents:

Simplified schematic of SMILE components

FIGURE_SIMPLIFIED-01

Overview of SMILE repositories

Docker images are available for the following SMILE repositories:

We use docker-compose to launch all of these services in addition to neo4j and nats but these can also be run with a simple docker run command.

Snippet of docker-compose file:

  cmo-metadb:
    container_name: cmo-metadb
    image: cmometadb/cmo-metadb:1.3.4.RELEASE
    environment:
      - METADB_CONFIG_HOME=${METADB_CONFIG_HOME}
      - METADB_DATA_DIR=${METADB_DATA_DIR}
      - SPRING_CONFIG_LOCATION=${METADB_CONFIG_HOME}/resources/cmo-metadb/application.properties
    command: -jar /smile-server/smile_server.jar
    volumes:
      - type: bind
        source: ${METADB_DATA_DIR}/logs/cmo-metadb
        target: ${METADB_DATA_DIR}/logs/cmo-metadb
      - type: bind
        source: ${METADB_CONFIG_HOME}/resources/cmo-metadb
        target: ${METADB_CONFIG_HOME}/resources/cmo-metadb
    links:
      - nats-jetstream
      - neo4j
    ports:
      - 3000:3000
    depends_on:
      - nats-jetstream
      - nats-stream-manager
      - neo4j

Where $METADB_CONFIG_HOME and $METADB_DATA_DIR are env variables for centralized locations for the configuration files for the individual SMILE components and data directories.

NATS Jetstream configuration

Resources:

NATS Jetstream config snippet

  nats-jetstream:
    container_name: nats-jetstream
    hostname: jetstream
    image: nats:2.7.1
    environment:
      - METADB_CONFIG_HOME=${METADB_CONFIG_HOME}
      - NATS_DATA_HOME=${NATS_DATA_HOME}
      - JS_KEY=${JS_KEY}
      - JS_CERT_FILE=${METADB_CONFIG_HOME}/nats/nats-cert.pem
      - JS_KEY_FILE=${METADB_CONFIG_HOME}/nats/nats-key.pem
      - JS_CA_FILE=${METADB_CONFIG_HOME}/nats/rootCA.crt
      - JS_EXTRA_ARGS=${JS_EXTRA_ARGS}
    entrypoint: /nats-server
    command: -js -c ${METADB_CONFIG_HOME}/nats/config/jetstream.conf ${JS_EXTRA_ARGS} --http_port 8222
    ports:
    - 4222:4222
    - 8222:8222
    volumes:
      - type: bind
        source: ${METADB_CONFIG_HOME}/nats
        target: /usr/local/share/ca-certificates
      - type: bind
        source: ${METADB_CONFIG_HOME}/nats
        target: ${METADB_CONFIG_HOME}/nats
      - type: bind
        source: ${NATS_DATA_HOME}
        target: ${NATS_DATA_HOME}

Where $METADB_CONFIG_HOME and $NATS_DATA_HOME are env variables for centralized locations for the NATS configuration files and data directory.

The $JS_EXTRA_ARGS variable, if set, will allow you to run NATS with additional settings such as -DV which would run NATS in verbose debug mode.

SMILE uses TLS but this won't be necessary for local development purposes since you will not require connecting to the MSK network.

Snippet of the jetstream.conf file (client authorization section):

# Client authorization
authorization = {
  ADMIN = {
    subscribe = ">"
    publish = ">"
  }
. . . 

  CMO_LABEL_GENERATOR = {
    subscribe = ["label_generator.MDB_STREAM", "_INBOX.>", "$JS.>", "MDB_STREAM.label-generator.cmo-sample", "MDB_STREAM.label-generator.cmo-sample-update", "MDB_STREAM.label-generator.promoted-sample", "MDB_REQ_REPLY.>"]
    publish = ["label_generator.MDB_STREAM", "_INBOX.>", "$JS.>", "MDB_STREAM.server.consistency-checker.igo-new-request", "MDB_STREAM.server.igo-sample-update", "MDB_STREAM.server.igo-promoted-request", "MDB_REQ_REPLY.>"]
  }

. . . 

  users = [
    {user: admin, password: "[bcryptpass]", permissions: $ADMIN}
    {user: label_generator, password: "[bcryptpass]", permissions: $CMO_LABEL_GENERATOR}

}

NATS Streams and Consumers Setup

NATS stream set up (TLS optional for local dev)

nats str add MDB_STREAM --subjects "MDB_STREAM.>" --ack --max-msgs=-1 --max-bytes=-1 --max-age=-1 --storage file --retention interest --max-msg-size=1073741824 --discard=old --dupe-window="2m" --user=${NATS_ADMIN_USERNAME} --password="${NATS_ADMIN_PASSWORD}" --server=${NATS_URL} --tlskey=${NATS_TLSKEY} --tlscert=${NATS_TLSCERT}

NATS consumers set up (TLS optional for local dev)

    nats con add ${stream_name} ${consumer_client_name} \
        --target="${consumer_client_name}.MDB_STREAM" \
        --filter="${filter_subject}" \
        --ack=${DEF_ACK} \
        --deliver=${DEF_DELIVER} \
        --max-deliver=${DEF_MAX_DELIVER} \
        --sample=${DEF_SAMPLE} \
        --replay=${DEF_REPLAY} \
        --wait=${DEF_WAIT}  \
        --user=${NATS_ADMIN_USERNAME} \
        --password="${NATS_ADMIN_PASSWORD}"  \
        --server=${NATS_URL} \
        --tlskey=${NATS_TLSKEY} \
        --tlscert=${NATS_TLSCERT}

Example values for variables:

  • ${stream_name} = MDB_STREAM
  • ${consumer_client_name} = label_generator
  • ${filter_subject} = MDB_STREAM.label-generator.*

Default NATS consumer variables we're using:

DEF_FILTER=""
DEF_ACK="all"
DEF_DELIVER="new"
DEF_MAX_DELIVER=10
DEF_SAMPLE=100
DEF_REPLAY="instant"
DEF_WAIT="300s"

Neo4j configuration

Resources:

Neo4j config (in our docker-compose):

  neo4j:
    container_name: neo4j
    image: neo4j:4.1.3
    restart: unless-stopped
    ports:
      - 7474:7474
      - 7687:7687
    environment:
      - NEO4J_AUTH=${NEO4J_USERNAME}/${NEO4J_PASSWORD}
      - NEO4J_dbms_memory_pagecache_size=1G
      - NEO4J_dbms.memory.heap.initial_size=1G
      - NEO4J_dbms_memory_heap_max__size=1G
      - NEO4J_DATA_HOME=${NEO4J_DATA_HOME}
    volumes:
      - type: bind
        source: ${NEO4J_DATA_HOME}/conf
        target: /conf
      - type: bind
        source: ${NEO4J_DATA_HOME}/data
        target: /data
      - type: bind
        source: ${NEO4J_DATA_HOME}/import
        target: /import
      - type: bind
        source: ${NEO4J_DATA_HOME}/logs
        target: /logs
      - type: bind
        source: ${NEO4J_DATA_HOME}/plugins
        target: /plugins

Quick links

Clone this wiki locally