README.md

description
In this section you can learn how to work with Rules in Notebooks written in Scala.

Owl Rules - DQ Pipeline

Instantiation

Code	Description
val rule = new Rule()	Instantiating a Rule object
rule.setDataset(<DATASET>)	Adding the name of the dataset
rule.setRuleNm(<RULE_NAME>)	Adding the name of the given rule
rule.setRuleValue(<RULE_EXPRESSION>)	Setting the simple RULE_EXPRESSION
rule.setRuleType(<RULE_TYPE>)	Setting the rule type
rule.setPerc(<RULE_PERCENTAGE>)	Setting the percentage
rule.setPoints(<RULE_POINT>)	Setting how many points will be deducted from total for each percentage
rule.setIsActive(<RULE_IS_ACTIVE>)	Making rule active/inactive Possible values: ACTIVE: 1 / true INACTIVE: 0 / false
rule.setUserNm(<RULE_OWNER_USERNAME>)	Adding the owner

Rule types

{% page-ref page="sql-based-rules/" %}

{% page-ref page="global-rules.md" %}

{% page-ref page="data-type-based-rules.md" %}

Requirements

Required imports

import com.owl.core.Owl
import com.owl.core.util.OwlUtils
import com.owl.common.options.OwlOptions
import com.owl.common.{Props, Utils}

import com.owl.common.domain2.Rule

import org.apache.spark.sql.functions._

SparkSession initialization

DataBricks and other notebook execution frameworks are working with managed/shared spark session, therefor we recommend to use this code snippet in your notebook to initialize the current spark session properly.

//----- Init Spark ----- //
def sparkInit(): Unit = {
  sparkSession = SparkSession.builder
    .master("local")
    .appName("test")
    .getOrCreate()
}

{% hint style="danger" %} Don't call spark.stop at any of your notebooks, otherwise the execution engine will exit immediately! {% endhint %}

{% hint style="warning" %} Make sure OwlContext is already created before using any method from OwlUtils! {% endhint %}