UNION SQL Support

[chloeminkyung]

query1 UNION query2

According to how postgresql supports UNION:

UNION effectively appends the result of query2 to the result of query1 (although there is no guarantee that this is the order in which the rows are actually returned). Furthermore, it eliminates duplicate rows from its result, in the same way as DISTINCT, unless UNION ALL is used.

Dozer's UNION support functionality will follow the above requirement.

pub enum SetOperatorType {
    Union,
    UnionAll,
    Except,
    Intersect,
}

impl SetOperatorType {
    pub fn execute(
        &self,
        left_schema: &Schema,
        right_schema: &Schema,
        record: &Record,
    ) -> Result<Field, PipelineError> {
        match self {
            SetOperatorType::Union => execute_union(left_schema, right_schema, record),
            _ => Err(PipelineError::UnsupportedSqlError(UnsupportedSqlError::GenericError(self.to_string())))
        }
    }
}

Introducing above SetOperatorType as a placeholder for the rest of the set operators support as well.

Implementation Approach

query1 UNION query2

• only support <select-query> UNION <select-query>
• follows similar structure with other operators
• difference is currently supported operators are field-to-field operators, set operators are expression-to-expression

Example - Single Field With Same Name

Let's look at how to use the SQL UNION operator that returns one field. In this simple example, the field in both SELECT statements will have the same name and data type.

For example:

suppliers and orders table, it would appear once in your result set. The UNION operator removes duplicates. If you do not wish to remove duplicates, should try using the UNION ALL operator. Now, let's explore this example further will some data. If you had the suppliers table populated with the following records:

supplier_id	supplier_name
1000	Microsoft
2000	Oracle
3000	Apple
4000	Samsung

order_id	order_date	supplier_id
1	2015-08-01	2000
2	2015-08-01	6000
3	2015-08-02	7000
4	2015-08-03	8000

And you executed the following UNION statement:

SELECT supplier_id
FROM suppliers
UNION
SELECT supplier_id
FROM orders
ORDER BY supplier_id;

You would get the following results:

supplier_id
1000
2000
3000
4000
6000
7000
8000

Example - Different Field Names

It is not necessary that the corresponding columns in each SELECT statement have the same name, but they do need to be the same corresponding data types.

When you don't have the same column names between the SELECT statements, it gets a bit tricky, especially when you want to order the results of the query using the ORDER BY clause.

Let's look at how to use the UNION operator with different column names and order the query results.

For example:

SELECT supplier_id, supplier_name
FROM suppliers
WHERE supplier_id > 2000
UNION
SELECT company_id, company_name
FROM companies
WHERE company_id > 1000
ORDER BY 1;

In this SQL UNION example, since the column names are different between the two SELECT statements, it is more advantageous to reference the columns in the ORDER BY clause by their position in the result set. In this example, we've sorted the results by supplier_id / company_id in ascending order, as denoted by the ORDER BY 1. The supplier_id / company_id fields are in position #1 in the result set.

Now, let's explore this example further with data.

If you had the suppliers table populated with the following records:

supplier_id	supplier_name
1000	Microsoft
2000	Oracle
3000	Apple
4000	Samsung

And the companies table populated with the following records:

company_id	company_name
1000	Microsoft
3000	Apple
7000	Sony
8000	IBM

And you executed the following UNION statement:

SELECT supplier_id, supplier_name
FROM suppliers
WHERE supplier_id > 2000
UNION
SELECT company_id, company_name
FROM companies
WHERE company_id > 1000
ORDER BY 1;

You would get the following results:

supplier_id	supplier_name
3000	Apple
4000	Samsung
7000	Sony
8000	IBM

First, notice that the record with supplier_id of 3000 only appears once in the result set because the UNION query removed duplicate entries.

Second, notice that the column headings in the result set are called supplier_id and supplier_name. This is because these were the column names used in the first SELECT statement in the UNION.

If you had wanted to, you could have aliased the columns as follows:

SELECT supplier_id AS ID_Value, supplier_name AS Name_Value
FROM suppliers
WHERE supplier_id > 2000
UNION
SELECT company_id AS ID_Value, company_name AS Name_Value
FROM companies
WHERE company_id > 1000
ORDER BY 1;

Now the column headings in the result will be aliased as ID_Value for the first column and Name_Value for the second column.

ID_Value	Name_Value
3000	Apple
4000	Samsung
7000	Sony
8000	IBM

[snork-alt]

I think it would be good to document how the union will work for inserts, deletes and updates and what actions are taken for each of the CDC message

[snork-alt]

I think it's fair to require that colum names should be the same (through aliases)