AWS DataBricks - DQ Pipeline

Getting started

Required configuration

DataBricks Runtime: 5.4 (includes Apache Spark 2.4.3, Scala 2.11)
Python Version: 3

Required libraries

OWL jar file Example: owl_core_trunk_jar_with_dependencies.jar
JDBC driver: org.springframework:spring-jdbc:4.3.16.RELEASE
Database specific JDBC connector drivers Example#1: mysql:mysql-connector-java:8.0.17 Example#2: org.postgresql:postgresql:jar:42.2.8