Skip to content

Latest commit

 

History

History
165 lines (126 loc) · 5.7 KB

installation.md

File metadata and controls

165 lines (126 loc) · 5.7 KB
Raw

Installation Setup and Testing

The following instructions represent the "hard" way to install and test the PPM. A docker image can be built to make this easier (see X). The directions that follow were developed for a clean installation of Ubuntu.

1. Install Git

$ sudo apt install git

2. Install Oracle’s Java

$ sudo add-apt-repository -y ppa:webupd8team/java
$ sudo apt update
$ sudo apt install oracle-java8-installer -y
$ sudo java -version

3. Install CMake to build the PPM

$ sudo apt install cmake

4. Install Docker

  • When following the website instructions, setup the Docker repos and follow the Linux post-install instructions.
  • The CE version seems to work fine.
  • Docker installation instructions
  • ORNL specific, but may apply to others with organizational security
    • Correct for internal Google DNS blocking
    • As root ($ sudo su), create a daemon.json file in the /etc/docker directory that contains the following information:
          {
              "debug": true,
              "default-runtime": "runc",
              "dns": ["160.91.126.23","160.91.126.28”],
              "icc": true,
              "insecure-registries": [],
              "ip": "0.0.0.0",
              "log-driver": "json-file",
              "log-level": "info",
              "max-concurrent-downloads": 3,
              "max-concurrent-uploads": 5,
              "oom-score-adjust": -500
          }
  • NOTE: The DNS IP addresses are ORNL specific.

5. Restart the docker daemon to consume the new configuration file.

$ service docker stop
$ service docker start

6. Check the configuration using the command below to confirm the updates above are taken if needed:

$ docker info

7. Install Docker Compose

  • Comprehensive instructions can be found on this website
  • Follow steps 1 and 2.

10. Create a base directory from which to install all the necessary components to test the PPM.

$ export BASE_PPM_DIR=~/some/dir/you/want/to/put/this/stuff

11. Install kafka-docker so kafka and zookeeper can run in a separate container.

  • These services will need to be running to test the PPM.
$ cd $BASE_PPM_DIR
$ git clone https://github.com/wurstmeister/kafka-docker.git
$ ifconfig                                              // to get the host ip address (device ens33 maybe)
$ export DOCKER_HOST_IP=<HOST IP>
$ cd kafka-docker
$ vim docker-compose.yml	                        // Set karka: ports: to 9092:9092
$ docker-compose up --no-recreate -d                    // to startup kafka and zookeeper containers and not recreate
$ docker-compose ps                                     // to check that they are running.

12. When you want to stop kafka and zookeeper

$ cd $BASE_PPM_DIR/kafka-docker
$ docker-compose down

13. Download and install the Kafka binary.

  • The Kafka binary provides a producer and consumer tool that can act as surrogates for the ODE (among other items).
  • Kafka Binary
  • Kafka Quickstart is a very useful reference.
  • Move and unpack the Kafka code as follows:
$ cd $BASE_PPM_DIR
$ wget http://apache.claz.org/kafka/0.10.2.1/kafka_2.12-0.10.2.1.tgz   // mirror and kafka version may change; check website.
$ tar -xzf kafka_2.12-0.10.2.1.tgz			               // the kafka version may be different.
$ mv kafka_2.12-0.10.2.1 kafka

14. Download and install librdkafka, the C++ Kafka library we use to build the PPM.

  • Upon completion of the instructions below, the header files for librdkafka should be located in /usr/local/include/librdkafka and the libraries (static and dynamic) should be located in /usr/local/lib. If you put them in another location the PPM may not build.
$ cd $BASE_PPM_DIR
$ git clone https://github.com/edenhill/librdkafka.git
$ cd librdkafka
$ ./configure
$ make
$ sudo make install

15. Download, build, and install the Privacy Protection Module (PPM)

$ cd $BASE_PPM_DIR
$ git clone https://github.com/usdot-jpo-ode/jpo-cvdp.git
$ cd jpo-cvdp
$ mkdir build && cd build
$ cmake ..
$ make

16. Integrating with the ODE.

Using the Docker container.

This will run the PPM module in separate container. First set the required environmental variables. You need to tell the PPM container where the Kafka Docker container is running with the DOCKER_HOST_IP variable. Also tell the PPM container where to find the map file and PPM Configuration file by setting the DOCKER_SHARED_VOLUME:

$ export DOCKER_HOST_IP=your.docker.host.ip
$ export DOCKER_SHARED_VOLUME=/your/shared/directory

Note that the map file and configuration file must be located in the DOCKER_SHARED_VOLUME root directory and named config.properties and road_file.csv respectively.

Add the following service to the end of your docker-compose.yml file:

  ppm:
    build: /path/to/jpo-cvdp/repo
    environment:
      DOCKER_HOST_IP: ${DOCKER_HOST_IP}
    volumes:
      - ${DOCKER_SHARED_VOLUME}:/ppm_data

Start the ODE containers as normal. Note that the topics for raw BSMs must be created ahead of time.

Additional information

  • The PPM uses RapidJSON, but it is a header-only library included in the repository.