Skip to content

Commit

Permalink
Provider moving/microsoft azure (apache#46254)
Browse files Browse the repository at this point in the history
* Update provider.yaml

* renaming files

* moving microsoft.azure provider files to new folder structure

* moving microsoft.azure provider files to new folder structure

* Fixing breeze unit test params

* Static checks fixes

* Remove ms azure ignore from pytest args

* Add pattern match to leave out ms azure providers

* Fix pytest args for test types

* Removed dead code

* Added pytest fixtures to conftest.py

* Added powerbi pytest fixture to conftest.py

* Fixes for failing tests

* Fixes for failing tests

* Moved conftest.py

* Fix for failing compat and provider tests

* Move conftest.py to providers/microsoft/azure/tests/ and associated imports changes

* Path changes to read resource files correctly

* Removed microsoft from providers/tests

* fixup! Removed microsoft from providers/tests

---------

Co-authored-by: Jarek Potiuk <[email protected]>
  • Loading branch information
kunaljubce and potiuk authored Feb 8, 2025
1 parent 42db67b commit fd606b1
Show file tree
Hide file tree
Showing 208 changed files with 1,157 additions and 262 deletions.
5 changes: 1 addition & 4 deletions .github/boring-cyborg.yml
Original file line number Diff line number Diff line change
Expand Up @@ -175,10 +175,7 @@ labelPRBasedOnFilePath:
- providers/jenkins/**

provider:microsoft-azure:
- providers/src/airflow/providers/microsoft/azure/**/*
- providers/tests/microsoft/azure/**/*
- docs/apache-airflow-providers-microsoft-azure/**/*
- providers/tests/system/microsoft/azure/**/*
- providers/microsoft/azure/**

provider:microsoft-mssql:
- providers/microsoft/mssql/**
Expand Down
4 changes: 2 additions & 2 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -640,7 +640,7 @@ repos:
^providers/google/src/airflow/providers/google/cloud/operators/cloud_build.py$|
^providers/google/src/airflow/providers/google/cloud/operators/dataproc.py$|
^providers/google/src/airflow/providers/google/cloud/operators/mlengine.py$|
^providers/src/airflow/providers/microsoft/azure/hooks/cosmos.py$|
^providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/cosmos.py$|
^providers/microsoft/winrm/src/airflow/providers/microsoft/winrm/hooks/winrm.py$|
^airflow/www/fab_security/manager.py$|
^docs/.*commits.rst$|
Expand All @@ -663,9 +663,9 @@ repos:
^providers/apache/hdfs/docs/connections.rst$|
^docs/apache-airflow-providers-apache-kafka/connections/kafka.rst$|
^providers/apache/spark/docs/decorators/pyspark.rst$|
^providers/microsoft/azure/docs/connections/azure_cosmos.rst$|
^providers/fab/docs/auth-manager/webserver-authentication.rst$|
^providers/google/docs/operators/cloud/kubernetes_engine.rst$|
^docs/apache-airflow-providers-microsoft-azure/connections/azure_cosmos.rst$|
^providers/cncf/kubernetes/docs/operators.rst$|
^docs/conf.py$|
^docs/exts/removemarktransform.py$|
Expand Down
6 changes: 4 additions & 2 deletions dev/breeze/tests/test_pytest_args_for_test_types.py
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,10 @@ def _all_new_providers() -> list[str]:
(
GroupOfTests.PROVIDERS,
"Providers",
[*[f"providers/{provider}/tests" for provider in _all_new_providers()], "providers/tests"],
[
*[f"providers/{provider}/tests" for provider in _all_new_providers()],
"providers/tests",
],
),
(
GroupOfTests.PROVIDERS,
Expand Down Expand Up @@ -112,7 +115,6 @@ def _all_new_providers() -> list[str]:
],
"providers/tests",
"--ignore=providers/tests/amazon",
"--ignore=providers/tests/microsoft/azure",
],
),
(
Expand Down
2 changes: 1 addition & 1 deletion dev/breeze/tests/test_selective_checks.py
Original file line number Diff line number Diff line change
Expand Up @@ -1890,7 +1890,7 @@ def test_files_provided_trigger_full_build_for_any_event_type(github_event):
id="pyproject.toml changed but no dependency change",
),
pytest.param(
("providers/src/airflow/providers/microsoft/azure/provider.yaml",),
("providers/microsoft/azure/src/airflow/providers/microsoft/azure/provider.yaml",),
{
"upgrade-to-newer-dependencies": "false",
},
Expand Down
1 change: 1 addition & 0 deletions docs/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ apache-airflow-providers-github
apache-airflow-providers-google
apache-airflow-providers-grpc
apache-airflow-providers-http
apache-airflow-providers-microsoft-azure
apache-airflow-providers-jdbc
apache-airflow-providers-influxdb
apache-airflow-providers-microsoft-mssql
Expand Down
25 changes: 0 additions & 25 deletions docs/apache-airflow-providers-microsoft-azure/changelog.rst

This file was deleted.

109 changes: 109 additions & 0 deletions providers/microsoft/azure/README.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@

.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
.. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
.. IF YOU WANT TO MODIFY TEMPLATE FOR THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
`PROVIDER_README_TEMPLATE.rst.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
Package ``apache-airflow-providers-microsoft-azure``

Release: ``12.0.0``


`Microsoft Azure <https://azure.microsoft.com/>`__


Provider package
----------------

This is a provider package for ``microsoft.azure`` provider. All classes for this provider package
are in ``airflow.providers.microsoft.azure`` python package.

You can find package information and changelog for the provider
in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/12.0.0/>`_.

Installation
------------

You can install this package on top of an existing Airflow 2 installation (see ``Requirements`` below
for the minimum Airflow version supported) via
``pip install apache-airflow-providers-microsoft-azure``

The package supports the following python versions: 3.9,3.10,3.11,3.12

Requirements
------------

====================================== ===================
PIP package Version required
====================================== ===================
``apache-airflow`` ``>=2.9.0``
``adlfs`` ``>=2023.10.0``
``azure-batch`` ``>=8.0.0``
``azure-cosmos`` ``>=4.6.0``
``azure-mgmt-cosmosdb`` ``>=3.0.0``
``azure-datalake-store`` ``>=0.0.45``
``azure-identity`` ``>=1.3.1``
``azure-keyvault-secrets`` ``>=4.1.0``
``azure-mgmt-datalake-store`` ``>=0.5.0``
``azure-mgmt-resource`` ``>=2.2.0``
``azure-storage-blob`` ``>=12.14.0``
``azure-mgmt-storage`` ``>=16.0.0``
``azure-storage-file-share`` ``>=12.7.0``
``azure-servicebus`` ``>=7.12.1``
``azure-synapse-spark`` ``>=0.2.0``
``azure-synapse-artifacts`` ``>=0.17.0``
``adal`` ``>=1.2.7``
``azure-storage-file-datalake`` ``>=12.9.1``
``azure-kusto-data`` ``>=4.1.0,!=4.6.0``
``azure-mgmt-datafactory`` ``>=2.0.0``
``azure-mgmt-containerregistry`` ``>=8.0.0``
``azure-mgmt-containerinstance`` ``>=10.1.0``
``msgraph-core`` ``>=1.0.0,!=1.1.8``
``microsoft-kiota-http`` ``>=1.3.0,!=1.3.4``
``microsoft-kiota-serialization-json`` ``==1.0.0``
``microsoft-kiota-serialization-text`` ``==1.0.0``
``microsoft-kiota-abstractions`` ``<1.4.0``
====================================== ===================

Cross provider package dependencies
-----------------------------------

Those are dependencies that might be needed in order to use all the features of the package.
You need to install the specified provider packages in order to use them.

You can install such cross-provider dependencies when installing from PyPI. For example:

.. code-block:: bash
pip install apache-airflow-providers-microsoft-azure[amazon]
================================================================================================================== =================
Dependent package Extra
================================================================================================================== =================
`apache-airflow-providers-amazon <https://airflow.apache.org/docs/apache-airflow-providers-amazon>`_ ``amazon``
`apache-airflow-providers-common-compat <https://airflow.apache.org/docs/apache-airflow-providers-common-compat>`_ ``common.compat``
`apache-airflow-providers-oracle <https://airflow.apache.org/docs/apache-airflow-providers-oracle>`_ ``oracle``
`apache-airflow-providers-sftp <https://airflow.apache.org/docs/apache-airflow-providers-sftp>`_ ``sftp``
================================================================================================================== =================

The changelog for the provider package can be found in the
`changelog <https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/12.0.0/changelog.html>`_.
Original file line number Diff line number Diff line change
Expand Up @@ -616,7 +616,7 @@ Bug Fixes
~~~~~~~~~

* ``Fix updating account url for WasbHook (#33457)``
* ``Fix Azure Batch Hook instantation (#33731)``
* ``Fix Azure Batch Hook instantiation (#33731)``
* ``Truncate Wasb storage account name if it's more than 24 characters (#33851)``
* ``Remove duplicated message commit in Azure MessageHook (#33776)``
* ``fix(providers/azure): remove json.dumps when querying AzureCosmosDBHook (#33653)``
Expand Down
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ This functionality can be disabled for an asynchronous wait -- typically with th

Below is an example of using this operator to execute an Azure Data Factory pipeline.

.. exampleinclude:: /../../providers/tests/system/microsoft/azure/example_adf_run_pipeline.py
.. exampleinclude:: /../../providers/microsoft/azure/tests/system/microsoft/azure/example_adf_run_pipeline.py
:language: python
:dedent: 0
:start-after: [START howto_operator_adf_run_pipeline]
Expand All @@ -40,23 +40,23 @@ Below is an example of using this operator to execute an Azure Data Factory pipe
Below is an example of using this operator to execute an Azure Data Factory pipeline with a deferrable flag
so that polling for the status of the pipeline run occurs on the Airflow Triggerer.

.. exampleinclude:: /../../providers/tests/system/microsoft/azure/example_adf_run_pipeline.py
.. exampleinclude:: /../../providers/microsoft/azure/tests/system/microsoft/azure/example_adf_run_pipeline.py
:language: python
:dedent: 4
:start-after: [START howto_operator_adf_run_pipeline_with_deferrable_flag]
:end-before: [END howto_operator_adf_run_pipeline_with_deferrable_flag]

Here is a different example of using this operator to execute a pipeline but coupled with the :class:`~airflow.providers.microsoft.azure.sensors.data_factory.AzureDataFactoryPipelineRunStatusSensor` to perform an asynchronous wait.

.. exampleinclude:: /../../providers/tests/system/microsoft/azure/example_adf_run_pipeline.py
.. exampleinclude:: /../../providers/microsoft/azure/tests/system/microsoft/azure/example_adf_run_pipeline.py
:language: python
:dedent: 0
:start-after: [START howto_operator_adf_run_pipeline_async]
:end-before: [END howto_operator_adf_run_pipeline_async]

Also you can use deferrable mode in :class:`~airflow.providers.microsoft.azure.sensors.data_factory.AzureDataFactoryPipelineRunStatusSensor` if you would like to free up the worker slots while the sensor is running.

.. exampleinclude:: /../../providers/tests/system/microsoft/azure/example_adf_run_pipeline.py
.. exampleinclude:: /../../providers/microsoft/azure/tests/system/microsoft/azure/example_adf_run_pipeline.py
:language: python
:dedent: 0
:start-after: [START howto_operator_adf_run_pipeline_async]
Expand All @@ -72,7 +72,7 @@ status of a data factory pipeline run asynchronously. This sensor will free up t
polling for job status happens on the Airflow triggerer, leading to efficient utilization
of resources within Airflow.

.. exampleinclude:: /../../providers/tests/system/microsoft/azure/example_adf_run_pipeline.py
.. exampleinclude:: /../../providers/microsoft/azure/tests/system/microsoft/azure/example_adf_run_pipeline.py
:language: python
:dedent: 4
:start-after: [START howto_operator_adf_run_pipeline_async]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ upload data to Azure DataLake Storage

Below is an example of using this operator to upload data to ADL.

.. exampleinclude:: /../../providers/tests/system/microsoft/azure/example_adls_create.py
.. exampleinclude:: /../../providers/microsoft/azure/tests/system/microsoft/azure/example_adls_create.py
:language: python
:dedent: 0
:start-after: [START howto_operator_adls_create]
Expand All @@ -52,7 +52,7 @@ file(s) from Azure DataLake Storage

Below is an example of using this operator to delete a file from ADL.

.. exampleinclude:: /../../providers/tests/system/microsoft/azure/example_adls_delete.py
.. exampleinclude:: /../../providers/microsoft/azure/tests/system/microsoft/azure/example_adls_delete.py
:language: python
:dedent: 0
:start-after: [START howto_operator_adls_delete]
Expand All @@ -69,7 +69,7 @@ file(s) from Azure DataLake Storage

Below is an example of using this operator to list files from ADL.

.. exampleinclude:: /../../providers/tests/system/microsoft/azure/example_adls_list.py
.. exampleinclude:: /../../providers/microsoft/azure/tests/system/microsoft/azure/example_adls_list.py
:language: python
:dedent: 0
:start-after: [START howto_operator_adls_list]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ To create Azure service bus queue with specific Parameter you can use

Below is an example of using this operator to execute an Azure Service Bus Create Queue.

.. exampleinclude:: /../../providers/tests/system/microsoft/azure/example_azure_service_bus.py
.. exampleinclude:: /../../providers/microsoft/azure/tests/system/microsoft/azure/example_azure_service_bus.py
:language: python
:dedent: 4
:start-after: [START howto_operator_create_service_bus_queue]
Expand All @@ -58,7 +58,7 @@ To Send message or list of message or batch Message to the Azure Service Bus Que

Below is an example of using this operator to execute an Azure Service Bus Send Message to Queue.

.. exampleinclude:: /../../providers/tests/system/microsoft/azure/example_azure_service_bus.py
.. exampleinclude:: /../../providers/microsoft/azure/tests/system/microsoft/azure/example_azure_service_bus.py
:language: python
:dedent: 4
:start-after: [START howto_operator_send_message_to_service_bus_queue]
Expand All @@ -75,7 +75,7 @@ To Receive Message or list of message or Batch message in a Queue you can use

Below is an example of using this operator to execute an Azure Service Bus Create Queue.

.. exampleinclude:: /../../providers/tests/system/microsoft/azure/example_azure_service_bus.py
.. exampleinclude:: /../../providers/microsoft/azure/tests/system/microsoft/azure/example_azure_service_bus.py
:language: python
:dedent: 4
:start-after: [START howto_operator_receive_message_service_bus_queue]
Expand All @@ -92,7 +92,7 @@ To Delete the Azure service bus queue you can use

Below is an example of using this operator to execute an Azure Service Bus Delete Queue.

.. exampleinclude:: /../../providers/tests/system/microsoft/azure/example_azure_service_bus.py
.. exampleinclude:: /../../providers/microsoft/azure/tests/system/microsoft/azure/example_azure_service_bus.py
:language: python
:dedent: 4
:start-after: [START howto_operator_delete_service_bus_queue]
Expand All @@ -113,7 +113,7 @@ To create Azure service bus topic with specific Parameter you can use

Below is an example of using this operator to execute an Azure Service Bus Create Topic.

.. exampleinclude:: /../../providers/tests/system/microsoft/azure/example_azure_service_bus.py
.. exampleinclude:: /../../providers/microsoft/azure/tests/system/microsoft/azure/example_azure_service_bus.py
:language: python
:dedent: 4
:start-after: [START howto_operator_create_service_bus_topic]
Expand All @@ -129,7 +129,7 @@ To Delete the Azure service bus topic you can use

Below is an example of using this operator to execute an Azure Service Bus Delete topic.

.. exampleinclude:: /../../providers/tests/system/microsoft/azure/example_azure_service_bus.py
.. exampleinclude:: /../../providers/microsoft/azure/tests/system/microsoft/azure/example_azure_service_bus.py
:language: python
:dedent: 4
:start-after: [START howto_operator_delete_service_bus_topic]
Expand All @@ -150,7 +150,7 @@ To create Azure service bus topic Subscription with specific Parameter you can u

Below is an example of using this operator to execute an Azure Service Bus Create Subscription.

.. exampleinclude:: /../../providers/tests/system/microsoft/azure/example_azure_service_bus.py
.. exampleinclude:: /../../providers/microsoft/azure/tests/system/microsoft/azure/example_azure_service_bus.py
:language: python
:dedent: 4
:start-after: [START howto_operator_create_service_bus_subscription]
Expand All @@ -166,7 +166,7 @@ To Update the Azure service bus topic Subscription which is already created, wit

Below is an example of using this operator to execute an Azure Service Bus Update Subscription.

.. exampleinclude:: /../../providers/tests/system/microsoft/azure/example_azure_service_bus.py
.. exampleinclude:: /../../providers/microsoft/azure/tests/system/microsoft/azure/example_azure_service_bus.py
:language: python
:dedent: 4
:start-after: [START howto_operator_update_service_bus_subscription]
Expand All @@ -182,7 +182,7 @@ To Receive a Batch messages from a Service Bus Subscription under specific Topic

Below is an example of using this operator to execute an Azure Service Bus Receive Subscription Message.

.. exampleinclude:: /../../providers/tests/system/microsoft/azure/example_azure_service_bus.py
.. exampleinclude:: /../../providers/microsoft/azure/tests/system/microsoft/azure/example_azure_service_bus.py
:language: python
:dedent: 4
:start-after: [START howto_operator_receive_message_service_bus_subscription]
Expand All @@ -198,7 +198,7 @@ To Delete the Azure service bus topic Subscription you can use

Below is an example of using this operator to execute an Azure Service Bus Delete Subscription under topic.

.. exampleinclude:: /../../providers/tests/system/microsoft/azure/example_azure_service_bus.py
.. exampleinclude:: /../../providers/microsoft/azure/tests/system/microsoft/azure/example_azure_service_bus.py
:language: python
:dedent: 4
:start-after: [START howto_operator_delete_service_bus_subscription]
Expand Down
Loading

0 comments on commit fd606b1

Please sign in to comment.