Example DAG for GX operators #1
Conversation
There was a problem hiding this comment.
@joshua-stauffer This is looking great to me. Thanks a lot for your efforts on this one.
I have some minor questions/suggestions in-line, and it's really looking great to me.
| dag_id="gx_provider_example_dag", | ||
| ) as dag: | ||
| # define a consistent set of expectations we'll use throughout the pipeline | ||
| expectation_suite = ExpectationSuite( |
There was a problem hiding this comment.
Since this is not an Airflow task can we move this ExpectationSuite outside of the dag definition?
There was a problem hiding this comment.
sure, i'll move it out of the DAG, thanks
| from airflow import DAG | ||
| from airflow.models.baseoperator import chain | ||
| from airflow.operators.empty import EmptyOperator | ||
| from great_expectations_provider import ( |
There was a problem hiding this comment.
can we have these imports from great_expectation_provider.operators here and then avoid exposing them in the __init__.py py here
That would make it easier from maintainability perspective and newer dev efforts adding/removing/deprecating operators so that we don't have to keep on changing them in the __init__.py provider module.
There was a problem hiding this comment.
sure, i'll update that, thanks 👍
| ) | ||
| from airflow.decorators import task | ||
| from airflow.models.param import Param | ||
| from great_expectations import ExpectationSuite, ValidationDefinition, Checkpoint |
There was a problem hiding this comment.
Where are these ExpectationSuite, ValidationDefinition and Checkpoint interfaces defined? Or are they yet to be defined?
There was a problem hiding this comment.
these are great expectations objects, available in the top level GX namespace
This PR adds an example dag which uses the three GX operators.