Skip to content

Commit

Permalink
Updated Google Cloud Storage, AWS S3 and Azure Container connections …
Browse files Browse the repository at this point in the history
…descriptions in the documentation. Minor fixes in other connectors.
  • Loading branch information
Andrzej Nagalski committed Jul 12, 2024
1 parent 673bbf2 commit 162feee
Show file tree
Hide file tree
Showing 20 changed files with 466 additions and 495 deletions.
12 changes: 6 additions & 6 deletions docs/data-sources/athena.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ To add Athena data source connection to DQOps you need the following:
- Writing objects in the S3 bucket
- Updating of the Lake Formation database

## Add Athena connection using the user interface
## Add an Athena connection using the user interface

### **Navigate to the connection settings**

Expand Down Expand Up @@ -69,7 +69,7 @@ For example:

![Adding connection JDBC settings](https://dqops.com/docs/images/working-with-dqo/adding-connections/connection-settings-JDBC-properties2.png){ loading=lazy; width="1200px" }

To remove the property click on the trash icon at the end of the input field.
To remove the property, click the trash icon at the end of the input field.

After filling in the connection settings, click the **Test Connection** button to test the connection.

Expand Down Expand Up @@ -98,7 +98,7 @@ or modify the schedule for newly imported tables.
![Importing tables - advisor](https://dqops.com/docs/images/working-with-dqo/adding-connections/importing-tables-advisor.png){ loading=lazy; width="1200px" }


## Add Athena connection using DQOps Shell
## Add an Athena connection using DQOps Shell

To add a connection run the following command in DQOps Shell.

Expand Down Expand Up @@ -160,7 +160,7 @@ After adding connection run `table import -c=connection1` to select schemas and

DQOps will ask you to select the schema from which the tables will be imported.

You can also add the schema and table name as a parameter to import tables in just a single step.
You can also add the schema and table name as parameters to import tables in just a single step.

```
dqo> table import --connection={connection name}
Expand All @@ -170,7 +170,7 @@ dqo> table import --connection={connection name}

DQOps supports the use of the asterisk character * as a wildcard when selecting schemas and tables, which can substitute
any number of characters. For example, use pub* to find all schema a name with a name starting with "pub". The *
character can be used at the beginning, in the middle or at the end of the name.
character can be used at the beginning, middle, or end of the name.

## Connections configuration files

Expand Down Expand Up @@ -256,6 +256,6 @@ To set the credential file in DQOps, follow these steps:

## Next steps

- We have provided a variety of use cases that use openly available datasets from [Google Cloud](https://cloud.google.com/datasets) to help you in using DQOps effectively. You can find the [full list of use cases here](../examples/index.md).
- We have provided a variety of use cases that use openly available datasets from [Google Cloud](https://cloud.google.com/datasets) to help you in using DQOps effectively. You can find the [complete list of use cases here](../examples/index.md).
- DQOps allows you to keep track of the issues that arise during data quality monitoring and send alert notifications directly to Slack. Learn more about [incidents](../working-with-dqo/managing-data-quality-incidents-with-dqops.md) and [notifications](../integrations/webhooks/index.md).
- The data in the table often comes from different data sources and vendors or is loaded by different data pipelines. Learn how [data grouping in DQOps](../working-with-dqo/set-up-data-grouping-for-data-quality-checks.md) can help you calculate separate data quality KPI scores for different groups of rows.
281 changes: 133 additions & 148 deletions docs/data-sources/aws.md

Large diffs are not rendered by default.

317 changes: 155 additions & 162 deletions docs/data-sources/azure.md

Large diffs are not rendered by default.

10 changes: 5 additions & 5 deletions docs/data-sources/bigquery.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ To add BigQuery data source connection to DQOps you need the following:
- A service account key in JSON format for JSON key authentication. For details refer to [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete)
- A working [Google Cloud CLI](https://cloud.google.com/sdk/docs/install) if you want to use [Google Application Credentials](#using-google-application-credentials-authentication) authentication

## Add BigQuery connection using the user interface
## Add a BigQuery connection using the user interface

### **Navigate to the connection settings**

Expand Down Expand Up @@ -78,7 +78,7 @@ the schedule for newly imported tables.
![Importing tables - advisor](https://dqops.com/docs/images/working-with-dqo/adding-connections/importing-tables-advisor.png)


## Add BigQuery connection using DQOps Shell
## Add a BigQuery connection using DQOps Shell

To add a connection run the following command in DQOps Shell.

Expand Down Expand Up @@ -131,7 +131,7 @@ After adding connection run `table import -c=connection1` to select schemas and

DQOps will ask you to select the schema from which the tables will be imported.

You can also add the schema and table name as a parameter to import tables in just a single step.
You can also add the schema and table name as parameters to import tables in just a single step.

```
dqo> table import --connection={connection name}
Expand All @@ -141,7 +141,7 @@ dqo> table import --connection={connection name}

DQOps supports the use of the asterisk character * as a wildcard when selecting schemas and tables, which can substitute
any number of characters. For example, use pub* to find all schema a name with a name starting with "pub". The *
character can be used at the beginning, in the middle or at the end of the name.
character can be used at the beginning, middle, or end of the name.


## Connections configuration files
Expand Down Expand Up @@ -235,6 +235,6 @@ To set the credential file in DQOps, follow these steps:

## Next steps

- We have provided a variety of use cases that use openly available datasets from [Google Cloud](https://cloud.google.com/datasets) to help you in using DQOps effectively. You can find the [full list of use cases here](../examples/index.md).
- We have provided a variety of use cases that use openly available datasets from [Google Cloud](https://cloud.google.com/datasets) to help you in using DQOps effectively. You can find the [complete list of use cases here](../examples/index.md).
- DQOps allows you to keep track of the issues that arise during data quality monitoring and send alert notifications directly to Slack. Learn more about [incidents](../working-with-dqo/managing-data-quality-incidents-with-dqops.md) and [notifications](../integrations/webhooks/index.md).
- The data in the table often comes from different data sources and vendors or is loaded by different data pipelines. Learn how [data grouping in DQOps](../working-with-dqo/set-up-data-grouping-for-data-quality-checks.md) can help you calculate separate data quality KPI scores for different groups of rows.
14 changes: 7 additions & 7 deletions docs/data-sources/csv.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Additional configuration is required **only when using remote storage** (AWS S3,
When using remote cloud storage, make sure your account has access to the remote directory containing CSV files.
The permissions granted should allow you to list the files and directories, as well as read the contents of the files.

## Add connection to CSV files using the user interface
## Add a connection to CSV files using the user interface

### **Navigate to the connection settings**

Expand Down Expand Up @@ -147,7 +147,7 @@ For example:

![Adding connection JDBC settings](https://dqops.com/docs/images/working-with-dqo/adding-connections/connection-settings-JDBC-properties2.png){ loading=lazy; width="1200px" }

To remove the property click on the trash icon at the end of the input field.
To remove the property, click the trash icon at the end of the input field.

After filling in the connection settings, click the **Test Connection** button to test the connection.

Expand Down Expand Up @@ -190,7 +190,7 @@ Enter the table name and the path absolute to the file. Save the new table confi

!!! tip "Use of the relative path"

If the schema specifies the folder path, use only the file name with extension instead of an absolute path.
If the schema specifies the folder path, use only the file name with an extension instead of an absolute path.

!!! tip "Path in table name"

Expand All @@ -209,7 +209,7 @@ If the job completes successfully, the created table will be imported and ready

![Register table](https://dqops.com/docs/images/working-with-dqo/adding-connections/duckdb/register-single-table-4.png){ loading=lazy; width="1200px" }

## Add CSV connection using DQOps Shell
## Add a CSV connection using DQOps Shell

To add a connection run the following command in DQOps Shell.

Expand Down Expand Up @@ -273,7 +273,7 @@ After adding connection run `table import -c=connection1` to select schemas and

DQOps will ask you to select the schema from which the tables will be imported.

You can also add the schema and table name as a parameter to import tables in just a single step.
You can also add the schema and table name as parameters to import tables in just a single step.

```
dqo> table import --connection={connection name}
Expand All @@ -283,7 +283,7 @@ dqo> table import --connection={connection name}

DQOps supports the use of the asterisk character * as a wildcard when selecting schemas and tables, which can substitute
any number of characters. For example, use pub* to find all schema a name with a name starting with "pub". The *
character can be used at the beginning, in the middle or at the end of the name.
character can be used at the beginning, middle, or end of the name.


## Connections configuration files
Expand Down Expand Up @@ -368,6 +368,6 @@ To set the credential file for AWS in DQOps, follow steps:
## Next steps

- Learn about more advanced importing when [working with files](../working-with-dqo/working-with-files.md)
- We have provided a variety of use cases that use openly available datasets from [Google Cloud](https://cloud.google.com/datasets) to help you in using DQOps effectively. You can find the [full list of use cases here](../examples/index.md).
- We have provided a variety of use cases that use openly available datasets from [Google Cloud](https://cloud.google.com/datasets) to help you in using DQOps effectively. You can find the [complete list of use cases here](../examples/index.md).
- DQOps allows you to keep track of the issues that arise during data quality monitoring and send alert notifications directly to Slack. Learn more about [incidents](../working-with-dqo/managing-data-quality-incidents-with-dqops.md) and [notifications](../integrations/webhooks/index.md).
- The data in the table often comes from different data sources and vendors or is loaded by different data pipelines. Learn how [data grouping in DQOps](../working-with-dqo/set-up-data-grouping-for-data-quality-checks.md) can help you calculate separate data quality KPI scores for different groups of rows.
12 changes: 6 additions & 6 deletions docs/data-sources/databricks.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ To add Databricks data source connection to DQOps you need a Databricks SQL Ware
It is also recommended to use an access token to connect an instance, so a permission to generate access token or a
possession of a previously generated token is necessary.

## Add Databricks connection using the user interface
## Add a Databricks connection using the user interface

### **Navigate to the connection settings**

Expand Down Expand Up @@ -62,7 +62,7 @@ For example:

![Adding connection JDBC settings](https://dqops.com/docs/images/working-with-dqo/adding-connections/connection-settings-JDBC-properties2.png){ loading=lazy; width="1200px" }

To remove the property click on the trash icon at the end of the input field.
To remove the property, click the trash icon at the end of the input field.

After filling in the connection settings, click the **Test Connection** button to test the connection.

Expand Down Expand Up @@ -91,7 +91,7 @@ or modify the schedule for newly imported tables.
![Importing tables - advisor](https://dqops.com/docs/images/working-with-dqo/adding-connections/importing-tables-advisor.png){ loading=lazy; width="1200px" }


## Add Databricks connection using DQOps Shell
## Add a Databricks connection using DQOps Shell

To add a connection run the following command in DQOps Shell.

Expand Down Expand Up @@ -142,7 +142,7 @@ After adding connection run `table import -c=connection1` to select schemas and

DQOps will ask you to select the schema from which the tables will be imported.

You can also add the schema and table name as a parameter to import tables in just a single step.
You can also add the schema and table name as parameters to import tables in just a single step.

```
dqo> table import --connection={connection name}
Expand All @@ -152,7 +152,7 @@ dqo> table import --connection={connection name}

DQOps supports the use of the asterisk character * as a wildcard when selecting schemas and tables, which can substitute
any number of characters. For example, use pub* to find all schema a name with a name starting with "pub". The *
character can be used at the beginning, in the middle or at the end of the name.
character can be used at the beginning, middle, or end of the name.

## Connections configuration files

Expand Down Expand Up @@ -325,6 +325,6 @@ The Catalog should be filled with **hive_metastore** to access the catalog with
## Next steps
- We have provided a variety of use cases that use openly available datasets from [Google Cloud](https://cloud.google.com/datasets) to help you in using DQOps effectively. You can find the [full list of use cases here](../examples/index.md).
- We have provided a variety of use cases that use openly available datasets from [Google Cloud](https://cloud.google.com/datasets) to help you in using DQOps effectively. You can find the [complete list of use cases here](../examples/index.md).
- DQOps allows you to keep track of the issues that arise during data quality monitoring and send alert notifications directly to Slack. Learn more about [incidents](../working-with-dqo/managing-data-quality-incidents-with-dqops.md) and [notifications](../integrations/webhooks/index.md).
- The data in the table often comes from different data sources and vendors or is loaded by different data pipelines. Learn how [data grouping in DQOps](../working-with-dqo/set-up-data-grouping-for-data-quality-checks.md) can help you calculate separate data quality KPI scores for different groups of rows.
2 changes: 1 addition & 1 deletion docs/data-sources/duckdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Additional configuration is required **only when using remote storage** (AWS S3,
When using remote cloud storage, make sure your account has access to the remote directory containing CSV, JSON, or Parquet files.
The permissions granted should allow you to list the files and directories, as well as read the contents of the files.

## Add connection to the files using the user interface
## Add a connection to the files using the user interface

### **Navigate to the connection settings**

Expand Down
Loading

0 comments on commit 162feee

Please sign in to comment.