diff --git a/README.md b/README.md index 4c54e976..f1fc8ab1 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,16 @@ # Nautilus Connectors Kit -**NCK is a Command-Line Interface (CLI), allowing you to easily request, stream and store raw reports, from the API source to the destination of your choice.** +**NCK is an E(T)L tool specialized in API data ingestion. It is accessible through a Command-Line Interface. The application allows you to easily extract, stream and load data (with minimum transformations), from the API source to the destination of your choice.** -The official documentation is available [here](https://artefactory.github.io/nautilus-connectors-kit/). +As of now, the most common output format of data loaded by the application is .njson (i.e. a file of n lines, where each line is a json-like dictionary). + +Official documentation is available [here](https://artefactory.github.io/nautilus-connectors-kit/). + +--- ## Philosophy -The application is composed of **3 main components** (*implemented as Python classes*). When combined, these components act as data connectors, allowing you to stream data from a source to the destination of your choice: +The application is composed of **3 main components** (*implemented as Python classes*). When combined, these components act as an E(T)L pipeline, allowing you to stream data from a source to the destination of your choice: - [Readers](nck/readers) are reading data from an API source, and transform it into a stream object. - [Streams](nck/streams) (*transparent to the end-user*) are local objects used by writers to process individual records collected from the source. @@ -14,77 +18,48 @@ The application is composed of **3 main components** (*implemented as Python cla ## Available connectors -As of now, the application is offering: +As of now, the application is offering the following Readers & Writers: ### Readers -**Analytics** - -- Adobe Analytics 1.4 -- Adobe Analytics 2.0 -- Google Analytics - -**Advertising** - -- **DSP** - +- **Analytics** + - Adobe Analytics 1.4 + - Adobe Analytics 2.0 + - Google Analytics +- **Advertising - Adserver** + - Google Campaign Manager +- **Advertising - DSP** - Google Display & Video 360 - The Trade Desk - -- **Adserver** - - - Google Campaign Manager - -- **Search** - +- **Advertising - Search** - Google Ads - Google Search Ads 360 - Google Search Console - Yandex Campaign - Yandex Statistics - -- **Social** - +- **Advertising - Social** - Facebook Marketing - MyTarget - Radarly - Twitter Ads - -**CRM** - -- SalesForce - -**Databases** - -- MySQL - -**Files (.csv, .njson)** - -- Amazon S3 -- Google Cloud Storage -- Google Sheets - -**DevTools** - -- Confluence - +- **CRM** + - SalesForce +- **Databases** + - MySQL +- **DevTools** + - Confluence +- **Files (.csv, .njson)** + - Amazon S3 + - Google Cloud Storage + - Google Sheets ### Writers -**Files (.njson)** - -- Amazon S3 -- Google Cloud Storage -- Local file - -**Data Warehouse** - -- Google BigQuery - -**Debugging** - -- Console - -*A data connector could be, for instance, the combination of a Google Analytics reader + a Google Cloud Storage writer, collecting data from the Google Analytics API, and storing output stream records into a Google Cloud Storage bucket.* - -For more information on how to use NCK, check out the [official documentation](https://artefactory.github.io/nautilus-connectors-kit/). +- **Data Warehouses** + - Google BigQuery +- **Debugging** + - Console +- **Files (.njson)** + - Amazon S3 + - Google Cloud Storage + - Local file diff --git a/docs/source/getting_started.rst b/docs/source/getting_started.rst index cdf1d3ae..e5706cf3 100644 --- a/docs/source/getting_started.rst +++ b/docs/source/getting_started.rst @@ -181,21 +181,39 @@ How to develop a new reader To create a new reader, you should: -1. Create the following modules: ``nck/readers/_reader.py``` and ``nck/helpers/_helper.py`` +1. Create a ``nck/readers//`` directory, having the following structure: -The ``nck/readers/_reader.py`` module should implement 2 components: +.. code-block:: shell + + - nck/ + -- readers/ + --- / + ---- cli.py + ---- reader.py + ---- helper.py # Optional + ---- config.py # Optional + +``cli.py`` -- A click-decorated reader function +This module should implement a click-decorated reader function: - - The reader function should be decorated with: a ``@click.command()`` decorator, several ``@click.option()`` decorators (*one for each input that should be provided by end-users*) and a ``@processor()`` decorator (*preventing secrets to appear in logs*). For further information on how to implement these decorators, please refer to `click documentation `__. - - The reader function should return a reader class (*more details below*). A source name prefix should be added to the name of each class attribute, using the ``extract_args()`` function. + - The reader function should be decorated with: a ``@click.command()`` decorator, several ``@click.option()`` decorators (*one for each input provided by end-users*) and a ``@processor()`` decorator (*preventing secrets to appear in logs*). For further information on how to implement these decorators, please refer to `click documentation `__. + - The reader function should return a reader class (*more details below*). The source prefix of each option will be removed when passed to the writer class, using the ``extract_args()`` function. -- A reader class +``reader.py`` + +This module should implement a reader class: - Class attributes should be the previously defined click options. - - The class should have a ``read()`` method, yielding a stream object. This stream object can be chosen from `available stream classes `__, and has 2 attributes: a stream name and a source generator function named ``result_generator()``, and yielding individual source records. + - The class should have a ``read()`` method, yielding a stream object. This stream object can be chosen from `available stream classes `__, and has 2 attributes: a stream name and a source generator function named ``result_generator()``, yielding individual source records. + +``helper.py`` (Optional) -The ``nck/helpers/_helper.py`` module should implement helper methods and configuration variables (*warning: we are planning to move configuration variables to a separate module for reasons of clarity*). +This module gathers all helper functions used in the ``reader.py`` module. + +``config.py`` (Optional) + +This module gathers all configuration variables. 2. In parallell, create unit tests for your methods under the ``tests/`` directory @@ -204,8 +222,8 @@ The ``nck/helpers/_helper.py`` module should implement helper metho 4. Complete the documentation: - Add your reader to the list of existing readers in the :ref:`overview:Available Connectors` section. + - Add your reader to the list of existing readers in the repo's ``./README.md``. - Create dedicated documentation for your reader CLI command on the :ref:`readers:Readers` page. It should include the followings sections: *Source API - How to obtain credentials - Quickstart - Command name - Command options* - - Add your reader to the reader list in the README, at the root of the GitHub project --------------------------- How to develop a new stream @@ -228,24 +246,46 @@ How to develop a new writer To develop a new writer, you should: -1. Create the following module: ``nck/writers/_writer.py`` +1. Create a ``nck/writers//`` directory, having the following structure: + +.. code-block:: shell + + - nck/ + -- writers/ + --- / + ---- cli.py + ---- writer.py + ---- helper.py # Optional + ---- config.py # Optional + +``cli.py`` -This module should implement 2 components: +This module should implement a click-decorated writer function: -- A click-decorated writer function + - The writer function should be decorated with: a ``@click.command()`` decorator, several ``@click.option()`` decorators (*one for each input provided by end-users*) and a ``@processor()`` decorator (*preventing secrets to appear in logs*). For further information on how to implement these decorators, please refer to `click documentation `__. + - The writer function should return a writer class (*more details below*). The destination prefix of each option will be removed when passed to the writer class, using the ``extract_args()`` function. - - The writer function should be decorated with: a ``@click.command()`` decorator, several ``@click.option()`` decorators (*one for each input that should be provided by end-users*) and a ``@processor()`` decorator (*preventing secrets to appear in logs*). For further information on how to implement these decorators, please refer to `click documentation `__. - - The writer function should return a writer class (*more details below*). A destination name prefix should be added to the name of each class attribute, using the `extract_args` function. +``writer.py`` -- A writer class +This module should implement a writer class: - Class attributes should be the previously defined click options. - The class should have a ``write()`` method, writing the stream object to the destination. -2. Add your click-decorated writer function to the ``nck/writers/__init__.py`` file +``helper.py`` (Optional) + +This module gathers all helper functions used in the ``writer.py`` module. + +``config.py`` (Optional) + +This module gathers all configuration variables. -3. Complete the documentation: +2. In parallell, create unit tests for your methods under the ``tests/`` directory + +3. Add your click-decorated writer function to the ``nck/writers/__init__.py`` file + +4. Complete the documentation: - Add your writer to the list of existing writers in the :ref:`overview:Available Connectors` section. + - Add your reader to the list of existing readers in the repo's ``./README.md``. - Create dedicated documentation for your writer CLI command on the :ref:`writers:Writers` page. It should include the followings sections: *Quickstart - Command name - Command options* - - Add your writer to the writer list in the README, at the root of the GitHub project diff --git a/docs/source/overview.rst b/docs/source/overview.rst index a4eb2702..3bbc0164 100644 --- a/docs/source/overview.rst +++ b/docs/source/overview.rst @@ -2,13 +2,15 @@ Overview ######## -**NCK is a Command-Line Interface (CLI), allowing you to easily request, stream and store raw reports, from the API source to the destination of your choice.** As of now, the most common output format of data extracted by the application is .njson (i.e. a file of n lines, where each line is a json-like dictionary). +**NCK is an E(T)L tool specialized in API data ingestion. It is accessible through a Command-Line Interface. The application allows you to easily extract, stream and load data (with minimum transformations), from the API source to the destination of your choice.** + +As of now, the most common output format of data loaded by the application is .njson (i.e. a file of n lines, where each line is a json-like dictionary). ========== Philosophy ========== -The application is composed of **3 main components** (*implemented as Python classes*). When combined, these components act as data connectors, allowing you to stream data from a source to the destination of your choice: +The application is composed of **3 main components** (*implemented as Python classes*). When combined, these components act as an E(T)L pipeline, allowing you to stream data from a source to the destination of your choice: - :ref:`readers:Readers` are reading data from an API source, and transform it into a stream object. - :ref:`streams:Streams` (*transparent to the end-user*) are local objects used by writers to process individual records collected from the source. @@ -18,80 +20,52 @@ The application is composed of **3 main components** (*implemented as Python cla Available connectors ==================== -As of now, the application is offering: +As of now, the application is offering the following Readers & Writers: ******* Readers -******* - -**Analytics** - -- Adobe Analytics 1.4 -- Adobe Analytics 2.0 -- Google Analytics - -**Advertising** - -- **DSP** +******* +- **Analytics** + - Adobe Analytics 1.4 + - Adobe Analytics 2.0 + - Google Analytics +- **Advertising - Adserver** + - Google Campaign Manager +- **Advertising - DSP** - Google Display & Video 360 - The Trade Desk - -- **Adserver** - - - Google Campaign Manager - -- **Search** - +- **Advertising - Search** - Google Ads - Google Search Ads 360 - Google Search Console - Yandex Campaign - Yandex Statistics - -- **Social** - +- **Advertising - Social** - Facebook Marketing - MyTarget - Radarly - Twitter Ads - -**CRM** - -- SalesForce - -**Databases** - -- MySQL - -**Files (.csv, .njson)** - -- Amazon S3 -- Google Cloud Storage -- Google Sheets - -**DevTools** - -- Confluence - +- **CRM** + - SalesForce +- **Databases** + - MySQL +- **DevTools** + - Confluence +- **Files (.csv, .njson)** + - Amazon S3 + - Google Cloud Storage + - Google Sheets ******* Writers ******* -**Files (.njson)** - -- Amazon S3 -- Google Cloud Storage -- Local file - -**Data Warehouse** - -- Google BigQuery - -**Debugging** - -- Console - - -*A data connector could be, for instance, the combination of a Google Analytics reader + a Google Cloud Storage writer, collecting data from the Google Analytics API, and storing output stream records into a Google Cloud Storage bucket.* +- **Data Warehouses** + - Google BigQuery +- **Debugging** + - Console +- **Files (.njson)** + - Amazon S3 + - Google Cloud Storage + - Local file diff --git a/docs/source/readers.rst b/docs/source/readers.rst index 5996f530..a6ead065 100644 --- a/docs/source/readers.rst +++ b/docs/source/readers.rst @@ -463,7 +463,7 @@ See the `documentation here . +The following command retrieves insights about the Ads of ``my_first_campaign`` and ``my_second_campaign`` in the Google Ads Account . .. code-block:: shell @@ -662,6 +662,7 @@ Options Definition ``--dcm-filter`` association, used to narrow the scope of the report. For instance "dfa:advertiserId XXXXX" will narrow report scope to the performance of Advertiser ID XXXXX. Possible filter types can be found `here `__. ``--dcm-start-date`` Start date of the period to request (format: YYYY-MM-DD) ``--dcm-end-date`` End date of the period to request (format: YYYY-MM-DD) +``--dcm-date-range`` Date range. By default, not available in DCM, so choose among NCK default values: YESTERDAY, LAST_7_DAYS, PREVIOUS_WEEK, PREVIOUS_MONTH, LAST_90_DAYS ============================== ======================================================================================================================================================================================================================================================================================================================================= =========================================== @@ -830,6 +831,7 @@ Options Definition ``--search-console-site-url`` Site URL whose performance you want to request ``--search-console-start-date`` Start date of the period to request (format: YYYY-MM-DD) ``--search-console-end-date`` End date of the period to request (format: YYYY-MM-DD) +``--search-console-date-range`` Date range. By default, not available in Search Console, so choose among NCK default values: YESTERDAY, LAST_7_DAYS, PREVIOUS_WEEK, PREVIOUS_MONTH, LAST_90_DAYS ``--search-console-date-column`` If set to True, a date column will be included in the report ``--search-console-row-limit`` Row number by report page ================================== ============================================================================================================================================================================================================ @@ -879,9 +881,9 @@ Command name Command options --------------- -============================== ===================================================================================================================================================== +============================== ======================================================================================================================================================= Options Definition -============================== ===================================================================================================================================================== +============================== ======================================================================================================================================================= ``--sa360-client-id`` OAuth2 ID ``--sa360-client-secret`` OAuth2 secret ``--sa360-access-token`` (Optional) Access token @@ -894,7 +896,8 @@ Options Definition ``--sa360-saved-column`` (Optional) Saved columns to report. Documentation can be found `here `__. ``--sa360-start-date`` Start date of the period to request (format: YYYY-MM-DD) ``--sa360-end-date`` End date of the period to request (format: YYYY-MM-DD) -============================== ===================================================================================================================================================== +``--sa360-date-range`` Date range. By default, not available in SA360, so choose among NCK default values: YESTERDAY, LAST_7_DAYS, PREVIOUS_WEEK, PREVIOUS_MONTH, LAST_90_DAYS +============================== ======================================================================================================================================================= See documentation `here `__ for a better understanding of the parameters. @@ -1005,16 +1008,17 @@ Command name Command options --------------- -============================== =============================================================== +============================== ========================================================================================================================================================== Options Definition -============================== =============================================================== +============================== ========================================================================================================================================================== ``--mytarget-client-id`` Client ID you generated ``--mytarget-client-secret`` Client secret you generated. ``--mytarget-refresh-token`` Secret token you retrieved during the process of getting tokens ``--mytarget-request-type`` Type of report you want to retrieve: performance or budgets. ``--mytarget-start-date`` Start date of the period to request (format: YYYY-MM-DD) ``--mytarget-end-date`` End date of the period to request (format: YYYY-MM-DD) -============================== =============================================================== +``--mytarget-date-range`` Date range. By default, not available in MyTarget, so choose among NCK default values: YESTERDAY, LAST_7_DAYS, PREVIOUS_WEEK, PREVIOUS_MONTH, LAST_90_DAYS +============================== ========================================================================================================================================================== ============ MySQL Reader @@ -1222,6 +1226,7 @@ Options Definition ``--ttd-report-schedule-name`` Name of the Report Schedule to create ``--ttd-start-date`` Start date of the period to request (format: YYYY-MM-DD) ``--ttd-end-date`` End date of the period to request (format: YYYY-MM-DD) +``--ttd-date-range`` Date range. By default, not available in The Trade Desk, so choose among NCK default values: YESTERDAY, LAST_7_DAYS, PREVIOUS_WEEK, PREVIOUS_MONTH, LAST_90_DAYS ============================== =========================================================================================================================================================================================== If you need any further information, the documentation of The Trade Desk API can be found `here `__. @@ -1304,6 +1309,7 @@ Options Definition ``--twitter-country`` Specific to ANALYTICS reports. Required if segmentation_type is set to CITIES, POSTAL_CODES, or REGION. Possible values can be identified through the GET targeting_criteria/platforms endpoint. ``--twitter-start-date`` Start date of the period to request (format: YYYY-MM-DD). ``--twitter-end-date`` End date of the period to request (format: YYYY-MM-DD). +``--twitter-date-range`` Date range. By default, not available in Twitter, so choose among NCK default values: YESTERDAY, LAST_7_DAYS, PREVIOUS_WEEK, PREVIOUS_MONTH, LAST_90_DAYS ``--twitter-add-request-date-to-report`` If set to True (default: False), the date on which the request is made will appear on each report record. ========================================== ================================================================================================================================================================================================================================= diff --git a/docs/source/writers.rst b/docs/source/writers.rst index a1434ab6..d1ef46fa 100644 --- a/docs/source/writers.rst +++ b/docs/source/writers.rst @@ -137,7 +137,7 @@ The following command would allow you to write a file ``google_analytics_report_ .. code-block:: shell - write_local --local-directory ~/Desktop/ --file-name google_analytics_report_2020-01-01.njson + write_local --local-directory ~/Desktop/ --local-file-name google_analytics_report_2020-01-01.njson ------------ Command name @@ -152,8 +152,8 @@ Command options ============================== =============================================================== Options Definition ============================== =============================================================== -``--local-directory (-d)`` Local directory in which the destination file should be stored -``--file-name (-n)`` Destination file name +``--local-directory (-d)`` Directory in which the file should be stored +``--local-file-name (-n)`` File name ============================== =============================================================== ============== diff --git a/nck/commands/__init__.py b/nck/clients/__init__.py similarity index 100% rename from nck/commands/__init__.py rename to nck/clients/__init__.py diff --git a/nck/helpers/__init__.py b/nck/clients/adobe_analytics/__init__.py similarity index 100% rename from nck/helpers/__init__.py rename to nck/clients/adobe_analytics/__init__.py diff --git a/nck/clients/adobe_client.py b/nck/clients/adobe_analytics/client.py similarity index 97% rename from nck/clients/adobe_client.py rename to nck/clients/adobe_analytics/client.py index 38b9da7a..0ceb70d9 100644 --- a/nck/clients/adobe_client.py +++ b/nck/clients/adobe_analytics/client.py @@ -16,17 +16,18 @@ # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from nck.config import logger from datetime import datetime, timedelta -import requests + import jwt -from tenacity import retry, wait_exponential, stop_after_delay +import requests +from nck.config import logger +from tenacity import retry, stop_after_delay, wait_exponential IMS_HOST = "ims-na1.adobelogin.com" IMS_EXCHANGE = "https://ims-na1.adobelogin.com/ims/exchange/jwt" -class AdobeClient: +class AdobeAnalyticsClient: """ Create an Adobe Client for JWT Authentification. Doc: https://github.com/AdobeDocs/adobeio-auth/blob/stage/JWT/JWT.md diff --git a/tests/helpers/__init__.py b/nck/clients/api/__init__.py similarity index 100% rename from tests/helpers/__init__.py rename to nck/clients/api/__init__.py diff --git a/nck/clients/api_client.py b/nck/clients/api/client.py similarity index 97% rename from nck/clients/api_client.py rename to nck/clients/api/client.py index f772cf7d..e9613570 100644 --- a/nck/clients/api_client.py +++ b/nck/clients/api/client.py @@ -15,7 +15,8 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from typing import Dict, Any + +from typing import Any, Dict from requests_toolbelt import sessions diff --git a/nck/helpers/api_client_helper.py b/nck/clients/api/helper.py similarity index 99% rename from nck/helpers/api_client_helper.py rename to nck/clients/api/helper.py index 9733d43d..d54173fe 100644 --- a/nck/helpers/api_client_helper.py +++ b/nck/clients/api/helper.py @@ -15,7 +15,9 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + from typing import Dict + from nck.config import logger POSSIBLE_STRING_FORMATS = ["PascalCase"] diff --git a/nck/clients/google/__init__.py b/nck/clients/google/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/clients/google/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/helpers/google_base.py b/nck/clients/google/client.py similarity index 80% rename from nck/helpers/google_base.py rename to nck/clients/google/client.py index 241eb287..60679675 100644 --- a/nck/helpers/google_base.py +++ b/nck/clients/google/client.py @@ -15,38 +15,38 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -""" - This was adapted from airflow google Base Hook. - A base hook for Google cloud-related hooks. Google cloud has a shared REST - API client that is built in the same way no matter which service you use. - This class helps construct and authorize the credentials needed to then - call googleapiclient.discovery.build() to actually discover and build a client - for a Google cloud service. +# This was adapted from Airflow Google Base Hook. - Three ways of authentication are supported: - Default credentials: Only the 'Project Id' is required. You'll need to - have set up default credentials, such as by the - ``GOOGLE_APPLICATION_DEFAULT`` environment variable or from the metadata - server on Google Compute Engine. - JSON key file: Specify 'Project Id', 'Keyfile Path' and 'Scope'. - Legacy P12 key files are not supported. - JSON data provided the parameters -""" +# A base hook for Google cloud-related hooks. Google cloud has a shared REST +# API client that is built in the same way no matter which service you use. +# This class helps construct and authorize the credentials needed to then +# call googleapiclient.discovery.build() to actually discover and build a client +# for a Google cloud service. + +# Three ways of authentication are supported: +# - Default credentials: only the 'Project Id' is required. You'll need to +# have set up default credentials, such as by the +# - GOOGLE_APPLICATION_DEFAULT environment variable or from the metadata +# server on Google Compute Engine. +# - JSON key file: specify 'Project Id', 'Keyfile Path' and 'Scope'. +# - Legacy P12 key files are not supported. +# - JSON data provided the parameters -from nck.config import logger import json import os +from typing import Dict, Optional, Sequence + +from nck.config import logger + import google.auth import google.oauth2.service_account -from typing import Dict, Optional, Sequence - _DEFAULT_SCOPES = ("https://www.googleapis.com/auth/cloud-platform",) # type: Sequence[str] -class GoogleBaseClass: +class GoogleClient: scopes = _DEFAULT_SCOPES def _get_credentials_and_project_id(self) -> google.auth.credentials.Credentials: diff --git a/nck/clients/google_dcm/__init__.py b/nck/clients/google_dcm/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/clients/google_dcm/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/clients/dcm_client.py b/nck/clients/google_dcm/client.py similarity index 99% rename from nck/clients/dcm_client.py rename to nck/clients/google_dcm/client.py index 82b60772..093a4bea 100644 --- a/nck/clients/dcm_client.py +++ b/nck/clients/google_dcm/client.py @@ -15,6 +15,7 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + from nck.config import logger import httplib2 import requests @@ -26,7 +27,7 @@ DOWNLOAD_FORMAT = "CSV" -class DCMClient: +class GoogleDCMClient: API_NAME = "dfareporting" API_VERSION = "v3.3" diff --git a/nck/clients/google_sa360/__init__.py b/nck/clients/google_sa360/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/clients/google_sa360/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/clients/sa360_client.py b/nck/clients/google_sa360/client.py similarity index 92% rename from nck/clients/sa360_client.py rename to nck/clients/google_sa360/client.py index 5c71ac11..d140fdb3 100644 --- a/nck/clients/sa360_client.py +++ b/nck/clients/google_sa360/client.py @@ -15,6 +15,7 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + from nck.config import logger import httplib2 import requests @@ -26,7 +27,7 @@ DOWNLOAD_FORMAT = "CSV" -class SA360Client: +class GoogleSA360Client: API_NAME = "doubleclicksearch" API_VERSION = "v2" @@ -58,12 +59,12 @@ def get_all_advertisers_of_agency(self, agency_id): @staticmethod def generate_report_body(agency_id, advertiser_id, report_type, columns, start_date, end_date, saved_columns): - all_columns = SA360Client.generate_columns(columns, saved_columns) + all_columns = GoogleSA360Client.generate_columns(columns, saved_columns) body = { "reportScope": {"agencyId": agency_id, "advertiserId": advertiser_id}, "reportType": report_type, "columns": all_columns, - "timeRange": SA360Client.get_date_range(start_date, end_date), + "timeRange": GoogleSA360Client.get_date_range(start_date, end_date), "downloadFormat": "csv", "maxRowsPerFile": 4000000, "statisticsCurrency": "usd", @@ -80,9 +81,9 @@ def request_report_id(self, body): def assert_report_file_ready(self, report_id): """Poll the API with the reportId until the report is ready, up to 100 times. - Args: - report_id: The ID SA360 has assigned to a report. - """ + Args: + report_id: The ID SA360 has assigned to a report. + """ request = self._service.reports().get(reportId=report_id) report_data = request.execute() if report_data["isReportReady"]: @@ -105,10 +106,10 @@ def download_report_files(self, json_data, report_id): def download_fragment(self, report_id, fragment): """Generate and convert to df a report fragment. - Args: - report_id: The ID SA360 has assigned to a report. - fragment: The 0-based index of the file fragment from the files array. - """ + Args: + report_id: The ID SA360 has assigned to a report. + fragment: The 0-based index of the file fragment from the files array. + """ request = self._service.reports().getFile(reportId=report_id, reportFragment=fragment) headers = request.headers headers.update({"Authorization": self.auth}) diff --git a/nck/clients/salesforce/__init__.py b/nck/clients/salesforce/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/clients/salesforce/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/clients/salesforce/client.py b/nck/clients/salesforce/client.py new file mode 100644 index 00000000..540a497a --- /dev/null +++ b/nck/clients/salesforce/client.py @@ -0,0 +1,116 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import urllib + +import requests +from nck.config import logger +from nck.readers.salesforce.config import ( + SALESFORCE_DESCRIBE_ENDPOINT, + SALESFORCE_LOGIN_ENDPOINT, + SALESFORCE_LOGIN_REDIRECT, + SALESFORCE_QUERY_ENDPOINT, +) + + +class SalesforceClient: + def __init__(self, user, password, consumer_key, consumer_secret): + self._user = user + self._password = password + self._consumer_key = consumer_key + self._consumer_secret = consumer_secret + + self._headers = None + self._access_token = None + self._instance_url = None + + @property + def headers(self): + return { + "Content-type": "application/json", + "Accept-Encoding": "gzip", + "Authorization": f"Bearer {self.access_token}", + } + + @property + def access_token(self): + if not self._access_token: + self._load_access_info() + + return self._access_token + + @property + def instance_url(self): + if not self._instance_url: + self._load_access_info() + + return self._instance_url + + def _load_access_info(self): + logger.info("Retrieving Salesforce access token") + + res = requests.post(SALESFORCE_LOGIN_ENDPOINT, params=self._get_login_params()) + + res.raise_for_status() + + self._access_token = res.json().get("access_token") + self._instance_url = res.json().get("instance_url") + + return self._access_token, self._instance_url + + def _get_login_params(self): + return { + "grant_type": "password", + "client_id": self._consumer_key, + "client_secret": self._consumer_secret, + "username": self._user, + "password": self._password, + "redirect_uri": SALESFORCE_LOGIN_REDIRECT, + } + + def _request_data(self, path, params=None): + + endpoint = urllib.parse.urljoin(self.instance_url, path) + response = requests.get(endpoint, headers=self.headers, params=params, timeout=30) + + response.raise_for_status() + + return response.json() + + def describe(self, object_type): + path = SALESFORCE_DESCRIBE_ENDPOINT.format(obj=object_type) + return self._request_data(path) + + def query(self, query): + + logger.info(f"Running Salesforce query: {query}") + + response = self._request_data(SALESFORCE_QUERY_ENDPOINT, {"q": query}) + + generating = True + + while generating: + + for rec in response["records"]: + yield rec + + if "nextRecordsUrl" in response: + logger.info("Fetching next page of Salesforce results") + response = self._request_data(response["nextRecordsUrl"]) + else: + generating = False diff --git a/nck/helpers/mytarget_helper.py b/nck/helpers/mytarget_helper.py deleted file mode 100644 index 20ee47b3..00000000 --- a/nck/helpers/mytarget_helper.py +++ /dev/null @@ -1,53 +0,0 @@ - -REQUEST_TYPES = ["performance", "budget"] - -REQUEST_CONFIG = { - 'refresh_agency_token': { - 'url': 'https://target.my.com/api/v2/oauth2/token.json', - 'headers_type': "content_type", - "offset": False, - '_campaign_id': False, - 'dates_required': False, - 'ids': False - }, - 'get_campaign_ids_names': { - 'url': 'https://target.my.com/api/v2/campaigns.json?fields=id,name', - 'headers_type': "authorization", - "offset": True, - '_campaign_id': False, - 'dates_required': False, - 'ids': False - }, - 'get_banner_ids_names': { - 'url': 'https://target.my.com/api/v2/banners.json?fields=id,name,campaign_id', - 'headers_type': "authorization", - 'offset': True, - '_campaign_id': False, - 'dates_required': False, - 'ids': False - }, - 'get_banner_stats': { - 'url': 'https://target.my.com/api/v2/statistics/banners/day.json', - 'headers_type': 'authorization', - 'offset': False, - '_campaign_id': False, - 'dates_required': True, - 'ids': False - }, - 'get_campaign_budgets': { - 'url': "https://target.my.com/api/v2/campaigns.json?fields=id,name,budget_limit,budget_limit_day", - 'headers_type': "authorization", - "offset": True, - '_campaign_id': False, - 'dates_required': False, - 'ids': False - }, - 'get_campaign_dates': { - 'url': 'https://target.my.com/api/v2/campaigns.json?fields=id,name,date_start,date_end,status', - 'headers_type': "authorization", - "offset": True, - '_campaign_id': False, - 'dates_required': False, - 'ids': False - } -} diff --git a/nck/readers/__init__.py b/nck/readers/__init__.py index d7eb1cbe..8f2b0be4 100644 --- a/nck/readers/__init__.py +++ b/nck/readers/__init__.py @@ -15,56 +15,57 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from nck.readers.dv360_reader import dv360 + from nck.readers.reader import Reader +from nck.readers.adobe_analytics_1_4.cli import adobe_analytics_1_4 +from nck.readers.adobe_analytics_2_0.cli import adobe_analytics_2_0 +from nck.readers.amazon_s3.cli import amazon_s3 +from nck.readers.confluence.cli import confluence +from nck.readers.facebook.cli import facebook +from nck.readers.google_ads.cli import google_ads +from nck.readers.google_analytics.cli import google_analytics +from nck.readers.google_cloud_storage.cli import google_cloud_storage +from nck.readers.google_dbm.cli import google_dbm +from nck.readers.google_dcm.cli import google_dcm +from nck.readers.google_dv360.cli import google_dv360 +from nck.readers.google_sa360.cli import google_sa360 +from nck.readers.google_search_console.cli import google_search_console +from nck.readers.google_sheets.cli import google_sheets +from nck.readers.google_sheets_old.cli import google_sheets_old +from nck.readers.mysql.cli import mysql +from nck.readers.mytarget.cli import mytarget +from nck.readers.radarly.cli import radarly +from nck.readers.salesforce.cli import salesforce +from nck.readers.the_trade_desk.cli import the_trade_desk +from nck.readers.twitter.cli import twitter +from nck.readers.yandex_campaign.cli import yandex_campaigns +from nck.readers.yandex_statistics.cli import yandex_statistics -from nck.readers.mysql_reader import mysql -from nck.readers.gcs_reader import gcs -from nck.readers.googleads_reader import google_ads -from nck.readers.s3_reader import s3 -from nck.readers.sa360_reader import sa360_reader -from nck.readers.gsheets_reader import gsheets -from nck.readers.salesforce_reader import salesforce -from nck.readers.facebook_reader import facebook -from nck.readers.ttd_reader import the_trade_desk -from nck.readers.twitter_reader import twitter -from nck.readers.dbm_reader import dbm -from nck.readers.dcm_reader import dcm -from nck.readers.ga_reader import ga -from nck.readers.search_console_reader import search_console -from nck.readers.adobe_reader import adobe -from nck.readers.adobe_reader_2_0 import adobe_2_0 -from nck.readers.radarly_reader import radarly -from nck.readers.yandex_campaign_reader import yandex_campaigns -from nck.readers.yandex_statistics_reader import yandex_statistics -from nck.readers.gs_reader import google_sheets -from nck.readers.confluence_reader import confluence -from nck.readers.mytarget_reader import mytarget readers = [ + adobe_analytics_1_4, + adobe_analytics_2_0, + amazon_s3, + confluence, + facebook, + google_ads, + google_analytics, + google_cloud_storage, + google_dbm, + google_dcm, + google_dv360, + google_sa360, + google_search_console, + google_sheets, + google_sheets_old, mysql, + mytarget, + radarly, salesforce, - gsheets, - gcs, - google_ads, - s3, - sa360_reader, - facebook, the_trade_desk, twitter, - dv360, - dbm, - dcm, - ga, - search_console, - adobe, - adobe_2_0, - radarly, yandex_campaigns, yandex_statistics, - google_sheets, - confluence, - mytarget, ] diff --git a/nck/readers/adobe_analytics_1_4/__init__.py b/nck/readers/adobe_analytics_1_4/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/adobe_analytics_1_4/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/adobe_analytics_1_4/cli.py b/nck/readers/adobe_analytics_1_4/cli.py new file mode 100644 index 00000000..4e68f0b7 --- /dev/null +++ b/nck/readers/adobe_analytics_1_4/cli.py @@ -0,0 +1,82 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.adobe_analytics_1_4.reader import AdobeAnalytics14Reader +from nck.utils.args import extract_args +from nck.utils.processor import processor + + +def format_key_if_needed(ctx, param, value): + """ + In some cases, newlines are escaped when passed as a click.option(). + This callback corrects this unexpected behaviour. + """ + return value.replace("\\n", "\n") + + +@click.command(name="read_adobe") +@click.option( + "--adobe-client-id", + required=True, + help="Client ID, that you can find in your integration section on Adobe Developper Console.", +) +@click.option( + "--adobe-client-secret", + required=True, + help="Client Secret, that you can find in your integration section on Adobe Developper Console.", +) +@click.option( + "--adobe-tech-account-id", + required=True, + help="Technical Account ID, that you can find in your integration section on Adobe Developper Console.", +) +@click.option( + "--adobe-org-id", + required=True, + help="Organization ID, that you can find in your integration section on Adobe Developper Console.", +) +@click.option( + "--adobe-private-key", + required=True, + callback=format_key_if_needed, + help="Content of the private.key file, that you had to provide to create the integration. " + "Make sure to enter the parameter in quotes, include headers, and indicate newlines as '\\n'.", +) +@click.option( + "--adobe-global-company-id", + required=True, + help="Global Company ID, to be requested to Discovery API. " + "Doc: https://www.adobe.io/apis/experiencecloud/analytics/docs.html#!AdobeDocs/analytics-2.0-apis/master/discovery.md)", +) +@click.option("--adobe-list-report-suite", type=click.BOOL, default=False) +@click.option("--adobe-report-suite-id") +@click.option("--adobe-report-element-id", multiple=True) +@click.option("--adobe-report-metric-id", multiple=True) +@click.option("--adobe-date-granularity", default=None) +@click.option( + "--adobe-day-range", + type=click.Choice(["PREVIOUS_DAY", "LAST_30_DAYS", "LAST_7_DAYS", "LAST_90_DAYS"]), + default=None, +) +@click.option("--adobe-start-date", type=click.DateTime()) +@click.option("--adobe-end-date", default=None, type=click.DateTime()) +@processor("adobe_password", "adobe_username") +def adobe_analytics_1_4(**kwargs): + # Should handle valid combinations dimensions/metrics in the API + return AdobeAnalytics14Reader(**extract_args("adobe_", kwargs)) diff --git a/nck/readers/adobe_analytics_1_4/config.py b/nck/readers/adobe_analytics_1_4/config.py new file mode 100644 index 00000000..06620200 --- /dev/null +++ b/nck/readers/adobe_analytics_1_4/config.py @@ -0,0 +1,21 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +ADOBE_API_ENDPOINT = "https://api.omniture.com/admin/1.4/rest/" +LIMIT_NVIEWS_PER_REQ = 5 +MAX_WAIT_REPORT_DELAY = 4096 diff --git a/nck/helpers/adobe_helper.py b/nck/readers/adobe_analytics_1_4/helper.py similarity index 91% rename from nck/helpers/adobe_helper.py rename to nck/readers/adobe_analytics_1_4/helper.py index 9f70fdc6..fa5822b6 100644 --- a/nck/helpers/adobe_helper.py +++ b/nck/readers/adobe_analytics_1_4/helper.py @@ -15,16 +15,17 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +# Credit goes to Mr Martin Winkel for the base code provided : +# github : https://github.com/SaturnFromTitan/adobe_analytics + import datetime + import more_itertools from nck.config import logger from nck.utils.text import reformat_naming_for_bq -# Credit goes to Mr Martin Winkel for the base code provided : -# github : https://github.com/SaturnFromTitan/adobe_analytics - - def _parse_header(report): dimensions = [_classification_or_name(dimension) for dimension in report["elements"]] metrics = [metric["name"] for metric in report["metrics"]] @@ -103,7 +104,12 @@ def _dimension_value_is_nan(chunk): def _to_datetime(chunk): - time_stamp = datetime.datetime(year=chunk["year"], month=chunk["month"], day=chunk["day"], hour=chunk.get("hour", 0),) + time_stamp = datetime.datetime( + year=chunk["year"], + month=chunk["month"], + day=chunk["day"], + hour=chunk.get("hour", 0), + ) return time_stamp.strftime("%Y-%m-%d %H:00:00") @@ -120,15 +126,3 @@ def parse(raw_response): yield {headers[i]: row[i] for i in range(len(headers))} else: yield {header: None for header in headers} - - -class ReportDescriptionError(Exception): - def __init__(self, message): - super().__init__(message) - logger.error(message) - - -class ReportNotReadyError(Exception): - def __init__(self, message): - super().__init__(message) - logger.error(message) diff --git a/nck/readers/adobe_reader.py b/nck/readers/adobe_analytics_1_4/reader.py similarity index 73% rename from nck/readers/adobe_reader.py rename to nck/readers/adobe_analytics_1_4/reader.py index 4a438392..a175c885 100644 --- a/nck/readers/adobe_reader.py +++ b/nck/readers/adobe_analytics_1_4/reader.py @@ -16,95 +16,28 @@ # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +# Credit goes to Mr Martin Winkel for the base code provided : +# github : https://github.com/SaturnFromTitan/adobe_analytics + import datetime import json from itertools import chain from time import sleep -import click import requests from click import ClickException -from nck.clients.adobe_client import AdobeClient -from nck.commands.command import processor +from nck.clients.adobe_analytics.client import AdobeAnalyticsClient from nck.config import logger -from nck.helpers.adobe_helper import ReportDescriptionError, ReportNotReadyError, parse +from nck.readers.adobe_analytics_1_4.config import ADOBE_API_ENDPOINT, MAX_WAIT_REPORT_DELAY +from nck.readers.adobe_analytics_1_4.helper import parse from nck.readers.reader import Reader from nck.streams.json_stream import JSONStream -from nck.utils.args import extract_args from nck.utils.date_handler import check_date_range_definition_conformity +from nck.utils.exceptions import ReportDescriptionError, ReportNotReadyError from nck.utils.retry import retry -# Credit goes to Mr Martin Winkel for the base code provided : -# github : https://github.com/SaturnFromTitan/adobe_analytics - -LIMIT_NVIEWS_PER_REQ = 5 - -ADOBE_API_ENDPOINT = "https://api.omniture.com/admin/1.4/rest/" - -MAX_WAIT_REPORT_DELAY = 4096 - - -def format_key_if_needed(ctx, param, value): - """ - In some cases, newlines are escaped when passed as a click.option(). - This callback corrects this unexpected behaviour. - """ - return value.replace("\\n", "\n") - - -@click.command(name="read_adobe") -@click.option( - "--adobe-client-id", - required=True, - help="Client ID, that you can find in your integration section on Adobe Developper Console.", -) -@click.option( - "--adobe-client-secret", - required=True, - help="Client Secret, that you can find in your integration section on Adobe Developper Console.", -) -@click.option( - "--adobe-tech-account-id", - required=True, - help="Technical Account ID, that you can find in your integration section on Adobe Developper Console.", -) -@click.option( - "--adobe-org-id", - required=True, - help="Organization ID, that you can find in your integration section on Adobe Developper Console.", -) -@click.option( - "--adobe-private-key", - required=True, - callback=format_key_if_needed, - help="Content of the private.key file, that you had to provide to create the integration. " - "Make sure to enter the parameter in quotes, include headers, and indicate newlines as '\\n'.", -) -@click.option( - "--adobe-global-company-id", - required=True, - help="Global Company ID, to be requested to Discovery API. " - "Doc: https://www.adobe.io/apis/experiencecloud/analytics/docs.html#!AdobeDocs/analytics-2.0-apis/master/discovery.md)", -) -@click.option("--adobe-list-report-suite", type=click.BOOL, default=False) -@click.option("--adobe-report-suite-id") -@click.option("--adobe-report-element-id", multiple=True) -@click.option("--adobe-report-metric-id", multiple=True) -@click.option("--adobe-date-granularity", default=None) -@click.option( - "--adobe-day-range", - type=click.Choice(["PREVIOUS_DAY", "LAST_30_DAYS", "LAST_7_DAYS", "LAST_90_DAYS"]), - default=None, -) -@click.option("--adobe-start-date", type=click.DateTime()) -@click.option("--adobe-end-date", default=None, type=click.DateTime()) -@processor("adobe_password", "adobe_username") -def adobe(**kwargs): - # Should handle valid combinations dimensions/metrics in the API - return AdobeReader(**extract_args("adobe_", kwargs)) - -class AdobeReader(Reader): +class AdobeAnalytics14Reader(Reader): def __init__( self, client_id, @@ -115,7 +48,7 @@ def __init__( global_company_id, **kwargs, ): - self.adobe_client = AdobeClient(client_id, client_secret, tech_account_id, org_id, private_key) + self.adobe_client = AdobeAnalyticsClient(client_id, client_secret, tech_account_id, org_id, private_key) self.global_company_id = global_company_id self.kwargs = kwargs diff --git a/nck/readers/adobe_analytics_2_0/__init__.py b/nck/readers/adobe_analytics_2_0/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/adobe_analytics_2_0/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/adobe_analytics_2_0/cli.py b/nck/readers/adobe_analytics_2_0/cli.py new file mode 100644 index 00000000..6f1e2787 --- /dev/null +++ b/nck/readers/adobe_analytics_2_0/cli.py @@ -0,0 +1,112 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.adobe_analytics_2_0.reader import AdobeAnalytics20Reader +from nck.utils.args import extract_args +from nck.utils.date_handler import DEFAULT_DATE_RANGE_FUNCTIONS +from nck.utils.processor import processor + + +def format_key_if_needed(ctx, param, value): + """ + In some cases, newlines are escaped when passed as a click.option(). + This callback corrects this unexpected behaviour. + """ + return value.replace("\\n", "\n") + + +@click.command(name="read_adobe_2_0") +@click.option( + "--adobe-2-0-client-id", + required=True, + help="Client ID, that you can find in your integration section on Adobe Developper Console.", +) +@click.option( + "--adobe-2-0-client-secret", + required=True, + help="Client Secret, that you can find in your integration section on Adobe Developper Console.", +) +@click.option( + "--adobe-2-0-tech-account-id", + required=True, + help="Technical Account ID, that you can find in your integration section on Adobe Developper Console.", +) +@click.option( + "--adobe-2-0-org-id", + required=True, + help="Organization ID, that you can find in your integration section on Adobe Developper Console.", +) +@click.option( + "--adobe-2-0-private-key", + required=True, + callback=format_key_if_needed, + help="Content of the private.key file, that you had to provide to create the integration. " + "Make sure to enter the parameter in quotes, include headers, and indicate newlines as '\\n'.", +) +@click.option( + "--adobe-2-0-global-company-id", + required=True, + help="Global Company ID, to be requested to Discovery API. " + "Doc: https://www.adobe.io/apis/experiencecloud/analytics/docs.html#!AdobeDocs/analytics-2.0-apis/master/discovery.md)", +) +@click.option( + "--adobe-2-0-report-suite-id", + required=True, + help="ID of the requested Adobe Report Suite", +) +@click.option( + "--adobe-2-0-dimension", + required=True, + multiple=True, + help="To get dimension names, enable the Debugger feature in Adobe Analytics Workspace: " + "it will allow you to visualize the back-end JSON requests made by Adobe Analytics UI to Reporting API 2.0. " + "Doc: https://github.com/AdobeDocs/analytics-2.0-apis/blob/master/reporting-tricks.md", +) +@click.option( + "--adobe-2-0-metric", + required=True, + multiple=True, + help="To get metric names, enable the Debugger feature in Adobe Analytics Workspace: " + "it will allow you to visualize the back-end JSON requests made by Adobe Analytics UI to Reporting API 2.0. " + "Doc: https://github.com/AdobeDocs/analytics-2.0-apis/blob/master/reporting-tricks.md", +) +@click.option( + "--adobe-2-0-start-date", + type=click.DateTime(), + help="Start date of the report", +) +@click.option( + "--adobe-2-0-end-date", + type=click.DateTime(), + help="End date of the report", +) +@click.option( + "--adobe-2-0-date-range", + type=click.Choice(DEFAULT_DATE_RANGE_FUNCTIONS.keys()), + help=f"One of the available NCK default date ranges: {DEFAULT_DATE_RANGE_FUNCTIONS.keys()}", +) +@processor( + "adobe_2_0_client_id", + "adobe_2_0_client_secret", + "adobe_2_0_tech_account_id", + "adobe_2_0_org_id", + "adobe_2_0_private_key", +) +def adobe_analytics_2_0(**kwargs): + return AdobeAnalytics20Reader(**extract_args("adobe_2_0_", kwargs)) diff --git a/nck/readers/adobe_analytics_2_0/config.py b/nck/readers/adobe_analytics_2_0/config.py new file mode 100644 index 00000000..a58f7c8d --- /dev/null +++ b/nck/readers/adobe_analytics_2_0/config.py @@ -0,0 +1,21 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +DATEFORMAT = "%Y-%m-%dT%H:%M:%S" +API_WINDOW_DURATION = 6 +API_REQUESTS_OVER_WINDOW_LIMIT = 12 diff --git a/nck/helpers/adobe_helper_2_0.py b/nck/readers/adobe_analytics_2_0/helper.py similarity index 95% rename from nck/helpers/adobe_helper_2_0.py rename to nck/readers/adobe_analytics_2_0/helper.py index 8c607f4c..1f0c6a88 100644 --- a/nck/helpers/adobe_helper_2_0.py +++ b/nck/readers/adobe_analytics_2_0/helper.py @@ -16,16 +16,9 @@ # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from nck.config import logger from datetime import datetime -class APIRateLimitError(Exception): - def __init__(self, message): - super().__init__(message) - logger.error(message) - - def add_metric_container_to_report_description(rep_desc, dimensions, metrics, breakdown_item_ids): """ Filling the metricContainer section of a report description: diff --git a/nck/readers/adobe_reader_2_0.py b/nck/readers/adobe_analytics_2_0/reader.py similarity index 71% rename from nck/readers/adobe_reader_2_0.py rename to nck/readers/adobe_analytics_2_0/reader.py index 57ff6163..a8010c27 100644 --- a/nck/readers/adobe_reader_2_0.py +++ b/nck/readers/adobe_analytics_2_0/reader.py @@ -17,17 +17,15 @@ # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. import json -from nck.config import logger import time from datetime import timedelta from itertools import chain -import click import requests -from nck.clients.adobe_client import AdobeClient -from nck.commands.command import processor -from nck.helpers.adobe_helper_2_0 import ( - APIRateLimitError, +from nck.clients.adobe_analytics.client import AdobeAnalyticsClient +from nck.config import logger +from nck.readers.adobe_analytics_2_0.config import API_REQUESTS_OVER_WINDOW_LIMIT, API_WINDOW_DURATION, DATEFORMAT +from nck.readers.adobe_analytics_2_0.helper import ( add_metric_container_to_report_description, get_item_ids_from_nodes, get_node_values_from_response, @@ -35,108 +33,12 @@ ) from nck.readers.reader import Reader from nck.streams.json_stream import JSONStream -from nck.utils.args import extract_args -from nck.utils.date_handler import ( - DEFAULT_DATE_RANGE_FUNCTIONS, - build_date_range, -) +from nck.utils.date_handler import build_date_range +from nck.utils.exceptions import APIRateLimitError from nck.utils.retry import retry -DATEFORMAT = "%Y-%m-%dT%H:%M:%S" -API_WINDOW_DURATION = 6 -API_REQUESTS_OVER_WINDOW_LIMIT = 12 - - -def format_key_if_needed(ctx, param, value): - """ - In some cases, newlines are escaped when passed as a click.option(). - This callback corrects this unexpected behaviour. - """ - return value.replace("\\n", "\n") - - -@click.command(name="read_adobe_2_0") -@click.option( - "--adobe-2-0-client-id", - required=True, - help="Client ID, that you can find in your integration section on Adobe Developper Console.", -) -@click.option( - "--adobe-2-0-client-secret", - required=True, - help="Client Secret, that you can find in your integration section on Adobe Developper Console.", -) -@click.option( - "--adobe-2-0-tech-account-id", - required=True, - help="Technical Account ID, that you can find in your integration section on Adobe Developper Console.", -) -@click.option( - "--adobe-2-0-org-id", - required=True, - help="Organization ID, that you can find in your integration section on Adobe Developper Console.", -) -@click.option( - "--adobe-2-0-private-key", - required=True, - callback=format_key_if_needed, - help="Content of the private.key file, that you had to provide to create the integration. " - "Make sure to enter the parameter in quotes, include headers, and indicate newlines as '\\n'.", -) -@click.option( - "--adobe-2-0-global-company-id", - required=True, - help="Global Company ID, to be requested to Discovery API. " - "Doc: https://www.adobe.io/apis/experiencecloud/analytics/docs.html#!AdobeDocs/analytics-2.0-apis/master/discovery.md)", -) -@click.option( - "--adobe-2-0-report-suite-id", - required=True, - help="ID of the requested Adobe Report Suite", -) -@click.option( - "--adobe-2-0-dimension", - required=True, - multiple=True, - help="To get dimension names, enable the Debugger feature in Adobe Analytics Workspace: " - "it will allow you to visualize the back-end JSON requests made by Adobe Analytics UI to Reporting API 2.0. " - "Doc: https://github.com/AdobeDocs/analytics-2.0-apis/blob/master/reporting-tricks.md", -) -@click.option( - "--adobe-2-0-metric", - required=True, - multiple=True, - help="To get metric names, enable the Debugger feature in Adobe Analytics Workspace: " - "it will allow you to visualize the back-end JSON requests made by Adobe Analytics UI to Reporting API 2.0. " - "Doc: https://github.com/AdobeDocs/analytics-2.0-apis/blob/master/reporting-tricks.md", -) -@click.option( - "--adobe-2-0-start-date", - type=click.DateTime(), - help="Start date of the report", -) -@click.option( - "--adobe-2-0-end-date", - type=click.DateTime(), - help="End date of the report", -) -@click.option( - "--adobe-2-0-date-range", - type=click.Choice(DEFAULT_DATE_RANGE_FUNCTIONS.keys()), - help=f"One of the available NCK default date ranges: {DEFAULT_DATE_RANGE_FUNCTIONS.keys()}", -) -@processor( - "adobe_2_0_client_id", - "adobe_2_0_client_secret", - "adobe_2_0_tech_account_id", - "adobe_2_0_org_id", - "adobe_2_0_private_key", -) -def adobe_2_0(**kwargs): - return AdobeReader_2_0(**extract_args("adobe_2_0_", kwargs)) - -class AdobeReader_2_0(Reader): +class AdobeAnalytics20Reader(Reader): def __init__( self, client_id, @@ -152,7 +54,7 @@ def __init__( end_date, date_range, ): - self.adobe_client = AdobeClient(client_id, client_secret, tech_account_id, org_id, private_key) + self.adobe_client = AdobeAnalyticsClient(client_id, client_secret, tech_account_id, org_id, private_key) self.global_company_id = global_company_id self.report_suite_id = report_suite_id self.dimensions = list(dimension) @@ -182,10 +84,7 @@ def build_report_description(self, metrics, breakdown_item_ids=[]): } rep_desc = add_metric_container_to_report_description( - rep_desc=rep_desc, - dimensions=self.dimensions, - metrics=metrics, - breakdown_item_ids=breakdown_item_ids, + rep_desc=rep_desc, dimensions=self.dimensions, metrics=metrics, breakdown_item_ids=breakdown_item_ids, ) return rep_desc diff --git a/nck/readers/amazon_s3/__init__.py b/nck/readers/amazon_s3/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/amazon_s3/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/amazon_s3/cli.py b/nck/readers/amazon_s3/cli.py new file mode 100644 index 00000000..de19e41e --- /dev/null +++ b/nck/readers/amazon_s3/cli.py @@ -0,0 +1,34 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.amazon_s3.reader import AmazonS3Reader +from nck.utils.args import extract_args +from nck.utils.processor import processor + + +@click.command(name="read_s3") +@click.option("--s3-bucket", required=True) +@click.option("--s3-prefix", required=True, multiple=True) +@click.option("--s3-format", required=True, type=click.Choice(["csv", "gz", "njson"])) +@click.option("--s3-dest-key-split", default=-1, type=int) +@click.option("--s3-csv-delimiter", default=",") +@click.option("--s3-csv-fieldnames", default=None) +@processor() +def amazon_s3(**kwargs): + return AmazonS3Reader(**extract_args("s3_", kwargs)) diff --git a/nck/readers/s3_reader.py b/nck/readers/amazon_s3/reader.py similarity index 73% rename from nck/readers/s3_reader.py rename to nck/readers/amazon_s3/reader.py index a33524b8..70c311ed 100644 --- a/nck/readers/s3_reader.py +++ b/nck/readers/amazon_s3/reader.py @@ -15,27 +15,12 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -import click import boto3 -from nck.commands.command import processor -from nck.readers.objectstorage_reader import ObjectStorageReader -from nck.utils.args import extract_args +from nck.readers.object_storage.reader import ObjectStorageReader -@click.command(name="read_s3") -@click.option("--s3-bucket", required=True) -@click.option("--s3-prefix", required=True, multiple=True) -@click.option("--s3-format", required=True, type=click.Choice(["csv", "gz", "njson"])) -@click.option("--s3-dest-key-split", default=-1, type=int) -@click.option("--s3-csv-delimiter", default=",") -@click.option("--s3-csv-fieldnames", default=None) -@processor() -def s3(**kwargs): - return S3Reader(**extract_args("s3_", kwargs)) - - -class S3Reader(ObjectStorageReader): +class AmazonS3Reader(ObjectStorageReader): def __init__(self, bucket, prefix, format, dest_key_split=-1, **kwargs): super().__init__(bucket, prefix, format, dest_key_split, platform="S3", **kwargs) diff --git a/nck/readers/confluence/__init__.py b/nck/readers/confluence/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/confluence/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/confluence/cli.py b/nck/readers/confluence/cli.py new file mode 100644 index 00000000..0b63877a --- /dev/null +++ b/nck/readers/confluence/cli.py @@ -0,0 +1,48 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.confluence.reader import ConfluenceReader +from nck.utils.args import extract_args +from nck.utils.processor import processor + + +@click.command(name="read_confluence") +@click.option("--confluence-user-login", required=True, help="User login associated with your Atlassian account") +@click.option("--confluence-api-token", required=True, help="API token associated with your Atlassian account") +@click.option( + "--confluence-atlassian-domain", + required=True, + help="Atlassian domain under which the content to request is located", +) +@click.option( + "--confluence-content-type", + type=click.Choice(["page", "blogpost"]), + default="page", + help="Type of content on which the report should be filtered", +) +@click.option("--confluence-spacekey", multiple=True, help="Space keys on which the report should be filtered") +@click.option( + "--confluence-field", + required=True, + multiple=True, + help="Fields that should be included in the report (path.to.field.value or custom_field)", +) +@processor("confluence_user_login", "confluence_api_token") +def confluence(**kwargs): + return ConfluenceReader(**extract_args("confluence_", kwargs)) diff --git a/nck/readers/confluence/config.py b/nck/readers/confluence/config.py new file mode 100644 index 00000000..785b1fe6 --- /dev/null +++ b/nck/readers/confluence/config.py @@ -0,0 +1,20 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +RECORDS_PER_PAGE = 100 +CONTENT_ENDPOINT = "wiki/rest/api/content" diff --git a/nck/helpers/confluence_helper.py b/nck/readers/confluence/helper.py similarity index 89% rename from nck/helpers/confluence_helper.py rename to nck/readers/confluence/helper.py index e3584708..310d2384 100644 --- a/nck/helpers/confluence_helper.py +++ b/nck/readers/confluence/helper.py @@ -1,8 +1,27 @@ -from typing import Optional, List, Dict +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import re +from typing import Dict, List, Optional + from bs4 import BeautifulSoup from bs4.element import Tag from unidecode import unidecode -import re def parse_response(raw_response, fields): diff --git a/nck/readers/confluence_reader.py b/nck/readers/confluence/reader.py similarity index 74% rename from nck/readers/confluence_reader.py rename to nck/readers/confluence/reader.py index cb3bff56..a3defe3c 100644 --- a/nck/readers/confluence_reader.py +++ b/nck/readers/confluence/reader.py @@ -17,49 +17,18 @@ # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. import base64 -import requests from itertools import chain -import click -from click import ClickException -from nck.utils.args import extract_args -from nck.commands.command import processor +import requests +from click import ClickException +from nck.readers.confluence.config import CONTENT_ENDPOINT, RECORDS_PER_PAGE +from nck.readers.confluence.helper import CUSTOM_FIELDS, parse_response from nck.readers.reader import Reader -from nck.helpers.confluence_helper import parse_response, CUSTOM_FIELDS from nck.streams.json_stream import JSONStream -RECORDS_PER_PAGE = 100 -CONTENT_ENDPOINT = "wiki/rest/api/content" - - -@click.command(name="read_confluence") -@click.option("--confluence-user-login", required=True, help="User login associated with your Atlassian account") -@click.option("--confluence-api-token", required=True, help="API token associated with your Atlassian account") -@click.option( - "--confluence-atlassian-domain", - required=True, - help="Atlassian domain under which the content to request is located", -) -@click.option( - "--confluence-content-type", - type=click.Choice(["page", "blogpost"]), - default="page", - help="Type of content on which the report should be filtered", -) -@click.option("--confluence-spacekey", multiple=True, help="Space keys on which the report should be filtered") -@click.option( - "--confluence-field", - required=True, - multiple=True, - help="Fields that should be included in the report (path.to.field.value or custom_field)", -) -@processor("confluence_user_login", "confluence_api_token") -def confluence(**kwargs): - return ConfluenceReader(**extract_args("confluence_", kwargs)) - class ConfluenceReader(Reader): - def __init__(self, user_login, api_token, atlassian_domain, content_type, spacekey, field, normalize_stream): + def __init__(self, user_login, api_token, atlassian_domain, content_type, spacekey, field): self.user_login = user_login self.api_token = api_token self._build_headers() @@ -67,7 +36,6 @@ def __init__(self, user_login, api_token, atlassian_domain, content_type, spacek self.content_type = content_type self.spacekeys = list(spacekey) self.fields = list(field) - self.normalize_stream = normalize_stream self._validate_spacekeys() diff --git a/nck/readers/facebook/__init__.py b/nck/readers/facebook/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/facebook/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/facebook/cli.py b/nck/readers/facebook/cli.py new file mode 100644 index 00000000..0fe4913a --- /dev/null +++ b/nck/readers/facebook/cli.py @@ -0,0 +1,73 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from click import ClickException +from nck.readers.facebook.config import ACTION_BREAKDOWNS, BREAKDOWNS, DATE_PRESETS, FACEBOOK_OBJECTS +from nck.readers.facebook.reader import FacebookReader +from nck.utils.args import extract_args +from nck.utils.processor import processor + + +def check_object_id(ctx, param, values): + try: + [int(value) for value in values] + return values + except ValueError: + raise ClickException("Wrong format. Ad object IDs should only contains digits.") + + +@click.command(name="read_facebook") +@click.option("--facebook-app-id", default="", help="Not mandatory for AdsInsights reporting if access-token provided") +@click.option("--facebook-app-secret", default="", help="Not mandatory for AdsInsights reporting if access-token provided") +@click.option("--facebook-access-token", required=True) +@click.option("--facebook-object-id", required=True, multiple=True, callback=check_object_id) +@click.option("--facebook-object-type", type=click.Choice(FACEBOOK_OBJECTS), default="account") +@click.option("--facebook-level", type=click.Choice(FACEBOOK_OBJECTS), default="ad", help="Granularity of result") +@click.option( + "--facebook-ad-insights", + type=click.BOOL, + default=True, + help="https://developers.facebook.com/docs/marketing-api/insights", +) +@click.option( + "--facebook-breakdown", + multiple=True, + type=click.Choice(BREAKDOWNS), + help="https://developers.facebook.com/docs/marketing-api/insights/breakdowns/", +) +@click.option( + "--facebook-action-breakdown", + multiple=True, + type=click.Choice(ACTION_BREAKDOWNS), + help="https://developers.facebook.com/docs/marketing-api/insights/breakdowns#actionsbreakdown", +) +@click.option("--facebook-field", multiple=True, help="API fields, following Artefact format") +@click.option("--facebook-time-increment") +@click.option("--facebook-start-date", type=click.DateTime()) +@click.option("--facebook-end-date", type=click.DateTime()) +@click.option("--facebook-date-preset", type=click.Choice(DATE_PRESETS)) +@click.option( + "--facebook-add-date-to-report", + type=click.BOOL, + default=False, + help="If set to true, the date of the request will appear in the report", +) +@processor("facebook_app_secret", "facebook_access_token") +def facebook(**kwargs): + return FacebookReader(**extract_args("facebook_", kwargs)) diff --git a/nck/readers/facebook/config.py b/nck/readers/facebook/config.py new file mode 100644 index 00000000..f13eb7c7 --- /dev/null +++ b/nck/readers/facebook/config.py @@ -0,0 +1,57 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +from facebook_business.adobjects.ad import Ad +from facebook_business.adobjects.adaccount import AdAccount +from facebook_business.adobjects.adcreative import AdCreative +from facebook_business.adobjects.adset import AdSet +from facebook_business.adobjects.adsinsights import AdsInsights +from facebook_business.adobjects.adspixel import AdsPixel +from facebook_business.adobjects.campaign import Campaign + +DATEFORMAT = "%Y-%m-%d" +BATCH_SIZE_LIMIT = 50 + +FACEBOOK_OBJECTS = ["pixel", "creative", "ad", "adset", "campaign", "account"] +DATE_PRESETS = [v for k, v in AdsInsights.DatePreset.__dict__.items() if not k.startswith("__")] +BREAKDOWNS = [v for k, v in AdsInsights.Breakdowns.__dict__.items() if not k.startswith("__")] +ACTION_BREAKDOWNS = [v for k, v in AdsInsights.ActionBreakdowns.__dict__.items() if not k.startswith("__")] + +OBJECT_CREATION_MAPPING = { + "account": AdAccount, + "campaign": Campaign, + "adset": AdSet, + "ad": Ad, + "creative": AdCreative, + "pixel": AdsPixel, +} + +EDGE_MAPPING = { + "account": ["campaign", "adset", "ad", "creative", "pixel"], + "campaign": ["adset", "ad"], + "adset": ["ad", "creative"], + "ad": ["creative"], +} + +EDGE_QUERY_MAPPING = { + "campaign": lambda obj: obj.get_campaigns(), + "adset": lambda obj: obj.get_ad_sets(), + "ad": lambda obj: obj.get_ads(), + "creative": lambda obj: obj.get_ad_creatives(), + "pixel": lambda obj: obj.get_ads_pixels(), +} diff --git a/nck/helpers/facebook_helper.py b/nck/readers/facebook/helper.py similarity index 94% rename from nck/helpers/facebook_helper.py rename to nck/readers/facebook/helper.py index a2d99f16..7c4253d1 100644 --- a/nck/helpers/facebook_helper.py +++ b/nck/readers/facebook/helper.py @@ -16,19 +16,10 @@ # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from nck.config import logger import json from time import sleep -from facebook_business.adobjects.adsinsights import AdsInsights - -FACEBOOK_OBJECTS = ["pixel", "creative", "ad", "adset", "campaign", "account"] - -DATE_PRESETS = [v for k, v in AdsInsights.DatePreset.__dict__.items() if not k.startswith("__")] - -BREAKDOWNS = [v for k, v in AdsInsights.Breakdowns.__dict__.items() if not k.startswith("__")] - -ACTION_BREAKDOWNS = [v for k, v in AdsInsights.ActionBreakdowns.__dict__.items() if not k.startswith("__")] +from nck.config import logger def get_action_breakdown_filters(field_path): diff --git a/nck/readers/facebook_reader.py b/nck/readers/facebook/reader.py similarity index 79% rename from nck/readers/facebook_reader.py rename to nck/readers/facebook/reader.py index 3d82c560..245acd45 100644 --- a/nck/readers/facebook_reader.py +++ b/nck/readers/facebook/reader.py @@ -16,117 +16,30 @@ # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. - import logging import re from datetime import datetime from math import ceil -import click from click import ClickException -from facebook_business.adobjects.ad import Ad -from facebook_business.adobjects.adaccount import AdAccount -from facebook_business.adobjects.adcreative import AdCreative from facebook_business.adobjects.adreportrun import AdReportRun -from facebook_business.adobjects.adset import AdSet -from facebook_business.adobjects.adspixel import AdsPixel -from facebook_business.adobjects.campaign import Campaign from facebook_business.api import FacebookAdsApi -from nck.commands.command import processor from nck.config import logger -from nck.helpers.facebook_helper import ( - ACTION_BREAKDOWNS, +from nck.readers.facebook.config import ( + BATCH_SIZE_LIMIT, BREAKDOWNS, - DATE_PRESETS, + DATEFORMAT, + EDGE_MAPPING, + EDGE_QUERY_MAPPING, FACEBOOK_OBJECTS, - generate_batches, - get_action_breakdown_filters, - get_field_values, - monitor_usage, + OBJECT_CREATION_MAPPING, ) +from nck.readers.facebook.helper import generate_batches, get_action_breakdown_filters, get_field_values, monitor_usage from nck.readers.reader import Reader from nck.streams.json_stream import JSONStream -from nck.utils.args import extract_args from nck.utils.date_handler import check_date_range_definition_conformity from tenacity import retry, stop_after_attempt, stop_after_delay, wait_exponential, wait_none -DATEFORMAT = "%Y-%m-%d" - -OBJECT_CREATION_MAPPING = { - "account": AdAccount, - "campaign": Campaign, - "adset": AdSet, - "ad": Ad, - "creative": AdCreative, - "pixel": AdsPixel, -} - -EDGE_MAPPING = { - "account": ["campaign", "adset", "ad", "creative", "pixel"], - "campaign": ["adset", "ad"], - "adset": ["ad", "creative"], - "ad": ["creative"], -} - -EDGE_QUERY_MAPPING = { - "campaign": lambda obj: obj.get_campaigns(), - "adset": lambda obj: obj.get_ad_sets(), - "ad": lambda obj: obj.get_ads(), - "creative": lambda obj: obj.get_ad_creatives(), - "pixel": lambda obj: obj.get_ads_pixels(), -} - -BATCH_SIZE_LIMIT = 50 - - -def check_object_id(ctx, param, values): - try: - [int(value) for value in values] - return values - except ValueError: - raise ClickException("Wrong format. Ad object IDs should only contains digits.") - - -@click.command(name="read_facebook") -@click.option("--facebook-app-id", default="", help="Not mandatory for AdsInsights reporting if access-token provided") -@click.option("--facebook-app-secret", default="", help="Not mandatory for AdsInsights reporting if access-token provided") -@click.option("--facebook-access-token", required=True) -@click.option("--facebook-object-id", required=True, multiple=True, callback=check_object_id) -@click.option("--facebook-object-type", type=click.Choice(FACEBOOK_OBJECTS), default="account") -@click.option("--facebook-level", type=click.Choice(FACEBOOK_OBJECTS), default="ad", help="Granularity of result") -@click.option( - "--facebook-ad-insights", - type=click.BOOL, - default=True, - help="https://developers.facebook.com/docs/marketing-api/insights", -) -@click.option( - "--facebook-breakdown", - multiple=True, - type=click.Choice(BREAKDOWNS), - help="https://developers.facebook.com/docs/marketing-api/insights/breakdowns/", -) -@click.option( - "--facebook-action-breakdown", - multiple=True, - type=click.Choice(ACTION_BREAKDOWNS), - help="https://developers.facebook.com/docs/marketing-api/insights/breakdowns#actionsbreakdown", -) -@click.option("--facebook-field", multiple=True, help="API fields, following Artefact format") -@click.option("--facebook-time-increment") -@click.option("--facebook-start-date", type=click.DateTime()) -@click.option("--facebook-end-date", type=click.DateTime()) -@click.option("--facebook-date-preset", type=click.Choice(DATE_PRESETS)) -@click.option( - "--facebook-add-date-to-report", - type=click.BOOL, - default=False, - help="If set to true, the date of the request will appear in the report", -) -@processor("facebook_app_secret", "facebook_access_token") -def facebook(**kwargs): - return FacebookReader(**extract_args("facebook_", kwargs)) - class FacebookReader(Reader): def __init__( diff --git a/nck/readers/google_ads/__init__.py b/nck/readers/google_ads/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/google_ads/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/google_ads/cli.py b/nck/readers/google_ads/cli.py new file mode 100644 index 00000000..c497459b --- /dev/null +++ b/nck/readers/google_ads/cli.py @@ -0,0 +1,92 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.google_ads.config import DATE_RANGE_TYPE_POSSIBLE_VALUES, REPORT_TYPE_POSSIBLE_VALUES +from nck.readers.google_ads.reader import GoogleAdsReader +from nck.utils.args import extract_args +from nck.utils.processor import processor + + +@click.command(name="read_googleads") +@click.option("--googleads-developer-token", required=True) +@click.option("--googleads-client-id", required=True) +@click.option("--googleads-client-secret", required=True) +@click.option("--googleads-refresh-token", required=True) +@click.option( + "--googleads-manager-id", + help="Google Ads Manager Account. " "Optional: can be used to get the reports from all accounts in hierarchy", +) +@click.option( + "--googleads-client-customer-id", + "googleads_client_customer_ids", + multiple=True, + help="Google Ads Client Account(s) to be called, thanks to their IDs.\n " + "This field is ignored if manager_id is specified (replaced by the accounts linked to the MCC)", +) +@click.option("--googleads-report-name", default="CustomReport", help="Name given to your Report") +@click.option( + "--googleads-report-type", + type=click.Choice(REPORT_TYPE_POSSIBLE_VALUES), + default=REPORT_TYPE_POSSIBLE_VALUES[0], + help="Desired Report Type to fetch\n" "https://developers.google.com/adwords/api/docs/appendix/reports#available-reports", +) +@click.option( + "--googleads-date-range-type", + type=click.Choice(DATE_RANGE_TYPE_POSSIBLE_VALUES), + help="Desired Date Range Type to fetch\n" "https://developers.google.com/adwords/api/docs/guides/reporting#date_ranges", +) +@click.option("--googleads-start-date", type=click.DateTime()) +@click.option("--googleads-end-date", type=click.DateTime()) +@click.option( + "--googleads-field", + "googleads_fields", + multiple=True, + help="Google Ads API fields for the request\n" + "https://developers.google.com/adwords/api/docs/appendix/reports#available-reports", +) +@click.option( + "--googleads-report-filter", + default="{}", + help="A filter can be applied on a chosen field, " + "in the form of a String containing a Dictionary \"{'field','operator','values'}\"\n" + "https://developers.google.com/adwords/api/docs/guides/reporting#create_a_report_definition", +) +@click.option( + "--googleads-include-zero-impressions", + default=True, + type=click.BOOL, + help="A boolean indicating whether the report should show rows with zero impressions", +) +@click.option( + "--googleads-filter-on-video-campaigns", + default=False, + type=click.BOOL, + help="A boolean indicating whether the report should return only Video campaigns\n" + "Only available if CampaignId is requested as a report field", +) +@click.option( + "--googleads-include-client-customer-id", + default=False, + type=click.BOOL, + help="A boolean indicating whether the Account ID should be included as a field in the output stream\n" + "(because AccountId is not available as a report field in the API)", +) +@processor("googleads_developer_token", "googleads_app_secret", "googleads_refresh_token") +def google_ads(**kwargs): + return GoogleAdsReader(**extract_args("googleads_", kwargs)) diff --git a/nck/helpers/googleads_helper.py b/nck/readers/google_ads/config.py similarity index 99% rename from nck/helpers/googleads_helper.py rename to nck/readers/google_ads/config.py index 4961a67c..3a6113b2 100644 --- a/nck/helpers/googleads_helper.py +++ b/nck/readers/google_ads/config.py @@ -15,6 +15,10 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +DATEFORMAT = "%Y%m%d" +ENCODING = "utf-8" + # https://developers.google.com/adwords/api/docs/appendix/reports#available-reports REPORT_TYPE_POSSIBLE_VALUES = [ "KEYWORDS_PERFORMANCE_REPORT", @@ -83,6 +87,3 @@ "LAST_WEEK_SUN_SAT", "CUSTOM_DATE", ] - -# Encoding for Stream Reader -ENCODING = "utf-8" diff --git a/nck/readers/googleads_reader.py b/nck/readers/google_ads/reader.py similarity index 78% rename from nck/readers/googleads_reader.py rename to nck/readers/google_ads/reader.py index b2ddf34a..c9286e00 100644 --- a/nck/readers/googleads_reader.py +++ b/nck/readers/google_ads/reader.py @@ -15,96 +15,24 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + import ast import codecs import csv import re from io import StringIO -import click from click import ClickException -from googleads import adwords -from googleads.errors import AdWordsReportBadRequestError -from googleads.oauth2 import GoogleRefreshTokenClient -from nck.commands.command import processor from nck.config import logger -from nck.helpers.googleads_helper import DATE_RANGE_TYPE_POSSIBLE_VALUES, ENCODING, REPORT_TYPE_POSSIBLE_VALUES +from nck.readers.google_ads.config import DATEFORMAT, ENCODING from nck.readers.reader import Reader from nck.streams.json_stream import JSONStream -from nck.utils.args import extract_args from nck.utils.exceptions import InconsistentDateDefinitionException, NoDateDefinitionException from nck.utils.retry import retry -DATEFORMAT = "%Y%m%d" - - -@click.command(name="read_googleads") -@click.option("--googleads-developer-token", required=True) -@click.option("--googleads-client-id", required=True) -@click.option("--googleads-client-secret", required=True) -@click.option("--googleads-refresh-token", required=True) -@click.option( - "--googleads-manager-id", - help="Google Ads Manager Account. " "Optional: can be used to get the reports from all accounts in hierarchy", -) -@click.option( - "--googleads-client-customer-id", - "googleads_client_customer_ids", - multiple=True, - help="Google Ads Client Account(s) to be called, thanks to their IDs.\n " - "This field is ignored if manager_id is specified (replaced by the accounts linked to the MCC)", -) -@click.option("--googleads-report-name", default="CustomReport", help="Name given to your Report") -@click.option( - "--googleads-report-type", - type=click.Choice(REPORT_TYPE_POSSIBLE_VALUES), - default=REPORT_TYPE_POSSIBLE_VALUES[0], - help="Desired Report Type to fetch\n" "https://developers.google.com/adwords/api/docs/appendix/reports#available-reports", -) -@click.option( - "--googleads-date-range-type", - type=click.Choice(DATE_RANGE_TYPE_POSSIBLE_VALUES), - help="Desired Date Range Type to fetch\n" "https://developers.google.com/adwords/api/docs/guides/reporting#date_ranges", -) -@click.option("--googleads-start-date", type=click.DateTime()) -@click.option("--googleads-end-date", type=click.DateTime()) -@click.option( - "--googleads-field", - "googleads_fields", - multiple=True, - help="Google Ads API fields for the request\n" - "https://developers.google.com/adwords/api/docs/appendix/reports#available-reports", -) -@click.option( - "--googleads-report-filter", - default="{}", - help="A filter can be applied on a chosen field, " - "in the form of a String containing a Dictionary \"{'field','operator','values'}\"\n" - "https://developers.google.com/adwords/api/docs/guides/reporting#create_a_report_definition", -) -@click.option( - "--googleads-include-zero-impressions", - default=True, - type=click.BOOL, - help="A boolean indicating whether the report should show rows with zero impressions", -) -@click.option( - "--googleads-filter-on-video-campaigns", - default=False, - type=click.BOOL, - help="A boolean indicating whether the report should return only Video campaigns\n" - "Only available if CampaignId is requested as a report field", -) -@click.option( - "--googleads-include-client-customer-id", - default=False, - type=click.BOOL, - help="A boolean indicating whether the Account ID should be included as a field in the output stream\n" - "(because AccountId is not available as a report field in the API)", -) -@processor("googleads_developer_token", "googleads_app_secret", "googleads_refresh_token") -def google_ads(**kwargs): - return GoogleAdsReader(**extract_args("googleads_", kwargs)) +from googleads import adwords +from googleads.errors import AdWordsReportBadRequestError +from googleads.oauth2 import GoogleRefreshTokenClient class GoogleAdsReader(Reader): diff --git a/nck/readers/google_analytics/__init__.py b/nck/readers/google_analytics/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/google_analytics/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/google_analytics/cli.py b/nck/readers/google_analytics/cli.py new file mode 100644 index 00000000..ec5bff38 --- /dev/null +++ b/nck/readers/google_analytics/cli.py @@ -0,0 +1,46 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.google_analytics.reader import GoogleAnalyticsReader +from nck.utils.args import extract_args +from nck.utils.processor import processor + + +@click.command(name="read_ga") +@click.option("--ga-access-token", default=None) +@click.option("--ga-refresh-token", required=True) +@click.option("--ga-client-id", required=True) +@click.option("--ga-client-secret", required=True) +@click.option("--ga-view-id", default="", multiple=True) +@click.option("--ga-account-id", default=[], multiple=True) +@click.option("--ga-dimension", multiple=True) +@click.option("--ga-metric", multiple=True) +@click.option("--ga-segment-id", multiple=True) +@click.option("--ga-start-date", type=click.DateTime(), default=None) +@click.option("--ga-end-date", type=click.DateTime(), default=None) +@click.option("--ga-date-range", nargs=2, type=click.DateTime(), default=None) +@click.option( + "--ga-day-range", type=click.Choice(["PREVIOUS_DAY", "LAST_30_DAYS", "LAST_7_DAYS", "LAST_90_DAYS"]), default=None +) +@click.option("--ga-sampling-level", type=click.Choice(["SMALL", "DEFAULT", "LARGE"]), default="LARGE") +@click.option("--ga-add-view", is_flag=True) +@processor("ga_access_token", "ga_refresh_token", "ga_client_secret") +def google_analytics(**kwargs): + # Should handle valid combinations dimensions/metrics in the API + return GoogleAnalyticsReader(**extract_args("ga_", kwargs)) diff --git a/tests/writers/test_gcs_writer.py b/nck/readers/google_analytics/config.py similarity index 74% rename from tests/writers/test_gcs_writer.py rename to nck/readers/google_analytics/config.py index 5a250098..d78c4eb2 100644 --- a/tests/writers/test_gcs_writer.py +++ b/nck/readers/google_analytics/config.py @@ -15,12 +15,9 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -import unittest -from nck.writers.gcs_writer import GCSWriter +GOOGLE_TOKEN_URI = "https://accounts.google.com/o/oauth2/token" +DISCOVERY_URI = "https://analyticsreporting.googleapis.com/$discovery/rest" +DATEFORMAT = "%Y-%m-%d" -class TestGCSWriter(unittest.TestCase): - def test_extract_extension(self): - filename = "test.py" - print(GCSWriter._extract_extension(filename)) - assert GCSWriter._extract_extension(filename) == ("test", ".py") +PREFIX = "^ga:" diff --git a/nck/readers/ga_reader.py b/nck/readers/google_analytics/reader.py similarity index 82% rename from nck/readers/ga_reader.py rename to nck/readers/google_analytics/reader.py index a3b5a736..c2eac9e1 100644 --- a/nck/readers/ga_reader.py +++ b/nck/readers/google_analytics/reader.py @@ -18,49 +18,19 @@ from datetime import datetime, timedelta -import click import httplib2 from click import ClickException from googleapiclient import discovery -from nck.commands.command import processor from nck.config import logger +from nck.readers.google_analytics.config import DATEFORMAT, DISCOVERY_URI, GOOGLE_TOKEN_URI, PREFIX from nck.readers.reader import Reader from nck.streams.json_stream import JSONStream -from nck.utils.args import extract_args from nck.utils.retry import retry from nck.utils.text import strip_prefix from oauth2client import GOOGLE_REVOKE_URI, client -DISCOVERY_URI = "https://analyticsreporting.googleapis.com/$discovery/rest" -DATEFORMAT = "%Y-%m-%d" -PREFIX = "^ga:" - - -@click.command(name="read_ga") -@click.option("--ga-access-token", default=None) -@click.option("--ga-refresh-token", required=True) -@click.option("--ga-client-id", required=True) -@click.option("--ga-client-secret", required=True) -@click.option("--ga-view-id", default="", multiple=True) -@click.option("--ga-account-id", default=[], multiple=True) -@click.option("--ga-dimension", multiple=True) -@click.option("--ga-metric", multiple=True) -@click.option("--ga-segment-id", multiple=True) -@click.option("--ga-start-date", type=click.DateTime(), default=None) -@click.option("--ga-end-date", type=click.DateTime(), default=None) -@click.option("--ga-date-range", nargs=2, type=click.DateTime(), default=None) -@click.option( - "--ga-day-range", type=click.Choice(["PREVIOUS_DAY", "LAST_30_DAYS", "LAST_7_DAYS", "LAST_90_DAYS"]), default=None -) -@click.option("--ga-sampling-level", type=click.Choice(["SMALL", "DEFAULT", "LARGE"]), default="LARGE") -@click.option("--ga-add-view", is_flag=True) -@processor("ga_access_token", "ga_refresh_token", "ga_client_secret") -def ga(**kwargs): - # Should handle valid combinations dimensions/metrics in the API - return GaReader(**extract_args("ga_", kwargs)) - - -class GaReader(Reader): + +class GoogleAnalyticsReader(Reader): def __init__(self, access_token, refresh_token, client_id, client_secret, **kwargs): credentials = client.GoogleCredentials( access_token=access_token, @@ -68,7 +38,7 @@ def __init__(self, access_token, refresh_token, client_id, client_secret, **kwar client_secret=client_secret, refresh_token=refresh_token, token_expiry=None, - token_uri="https://accounts.google.com/o/oauth2/token", + token_uri=GOOGLE_TOKEN_URI, user_agent=None, revoke_uri=GOOGLE_REVOKE_URI, ) @@ -200,7 +170,7 @@ def format_and_yield(self, view_id, report): formatted_response[metric] = metric_value if "date" in formatted_response: - formatted_response["date"] = GaReader.format_date(formatted_response["date"]) + formatted_response["date"] = GoogleAnalyticsReader.format_date(formatted_response["date"]) yield formatted_response diff --git a/nck/readers/google_cloud_storage/__init__.py b/nck/readers/google_cloud_storage/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/google_cloud_storage/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/google_cloud_storage/cli.py b/nck/readers/google_cloud_storage/cli.py new file mode 100644 index 00000000..4760a508 --- /dev/null +++ b/nck/readers/google_cloud_storage/cli.py @@ -0,0 +1,34 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.google_cloud_storage.reader import GoogleCloudStorageReader +from nck.utils.args import extract_args +from nck.utils.processor import processor + + +@click.command(name="read_gcs") +@click.option("--gcs-bucket", required=True) +@click.option("--gcs-prefix", required=True, multiple=True) +@click.option("--gcs-format", required=True, type=click.Choice(["csv", "gz", "njson"])) +@click.option("--gcs-dest-key-split", default=-1, type=int) +@click.option("--gcs-csv-delimiter", default=",") +@click.option("--gcs-csv-fieldnames", default=None) +@processor() +def google_cloud_storage(**kwargs): + return GoogleCloudStorageReader(**extract_args("gcs_", kwargs)) diff --git a/nck/readers/gcs_reader.py b/nck/readers/google_cloud_storage/reader.py similarity index 70% rename from nck/readers/gcs_reader.py rename to nck/readers/google_cloud_storage/reader.py index 6a948507..f1d7a8f0 100644 --- a/nck/readers/gcs_reader.py +++ b/nck/readers/google_cloud_storage/reader.py @@ -15,29 +15,15 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -import click -from google.cloud import storage -from nck.commands.command import processor -from nck.readers.objectstorage_reader import ObjectStorageReader -from nck.utils.args import extract_args -from nck.helpers.google_base import GoogleBaseClass import urllib - -@click.command(name="read_gcs") -@click.option("--gcs-bucket", required=True) -@click.option("--gcs-prefix", required=True, multiple=True) -@click.option("--gcs-format", required=True, type=click.Choice(["csv", "gz", "njson"])) -@click.option("--gcs-dest-key-split", default=-1, type=int) -@click.option("--gcs-csv-delimiter", default=",") -@click.option("--gcs-csv-fieldnames", default=None) -@processor() -def gcs(**kwargs): - return GCSReader(**extract_args("gcs_", kwargs)) +from google.cloud import storage +from nck.clients.google.client import GoogleClient +from nck.readers.object_storage.reader import ObjectStorageReader -class GCSReader(ObjectStorageReader, GoogleBaseClass): +class GoogleCloudStorageReader(ObjectStorageReader, GoogleClient): def __init__(self, bucket, prefix, format, dest_key_split=-1, **kwargs): super().__init__(bucket, prefix, format, dest_key_split, platform="GCS", **kwargs) diff --git a/nck/readers/google_dbm/__init__.py b/nck/readers/google_dbm/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/google_dbm/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/google_dbm/cli.py b/nck/readers/google_dbm/cli.py new file mode 100644 index 00000000..45e3da36 --- /dev/null +++ b/nck/readers/google_dbm/cli.py @@ -0,0 +1,64 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.google_dbm.config import POSSIBLE_REQUEST_TYPES +from nck.readers.google_dbm.reader import GoogleDBMReader +from nck.utils.args import extract_args +from nck.utils.processor import processor + + +@click.command(name="read_dbm") +@click.option("--dbm-access-token", default=None) +@click.option("--dbm-refresh-token", required=True) +@click.option("--dbm-client-id", required=True) +@click.option("--dbm-client-secret", required=True) +@click.option("--dbm-query-metric", multiple=True) +@click.option("--dbm-query-dimension", multiple=True) +@click.option("--dbm-request-type", type=click.Choice(POSSIBLE_REQUEST_TYPES), required=True) +@click.option("--dbm-query-id") +@click.option("--dbm-query-title") +@click.option("--dbm-query-frequency", default="ONE_TIME") +@click.option("--dbm-query-param-type", default="TYPE_TRUEVIEW") +@click.option("--dbm-start-date", type=click.DateTime()) +@click.option("--dbm-end-date", type=click.DateTime()) +@click.option( + "--dbm-add-date-to-report", + type=click.BOOL, + default=False, + help=( + "Sometimes the date range on which metrics are computed is missing from the report. " + "If this option is set to True, this range will be added." + ), +) +@click.option("--dbm-filter", type=click.Tuple([str, str]), multiple=True) +@click.option("--dbm-file-type", multiple=True) +@click.option( + "--dbm-date-format", + default="%Y-%m-%d", + help="And optional date format for the output stream. " + "Follow the syntax of https://docs.python.org/3.8/library/datetime.html#strftime-strptime-behavior", +) +@click.option( + "--dbm-day-range", + type=click.Choice(["PREVIOUS_DAY", "LAST_30_DAYS", "LAST_90_DAYS", "LAST_7_DAYS", "PREVIOUS_MONTH", "PREVIOUS_WEEK"]), +) +@processor("dbm_access_token", "dbm_refresh_token", "dbm_client_secret") +def google_dbm(**kwargs): + # Should add validation argument in function of request_type + return GoogleDBMReader(**extract_args("dbm_", kwargs)) diff --git a/nck/helpers/dbm_helper.py b/nck/readers/google_dbm/config.py similarity index 93% rename from nck/helpers/dbm_helper.py rename to nck/readers/google_dbm/config.py index 8d321cd2..55e5a411 100644 --- a/nck/helpers/dbm_helper.py +++ b/nck/readers/google_dbm/config.py @@ -15,6 +15,9 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +GOOGLE_TOKEN_URI = "https://accounts.google.com/o/oauth2/token" + POSSIBLE_REQUEST_TYPES = [ "existing_query", "custom_query", diff --git a/nck/readers/dbm_reader.py b/nck/readers/google_dbm/reader.py similarity index 80% rename from nck/readers/dbm_reader.py rename to nck/readers/google_dbm/reader.py index 856642dd..3cee718e 100644 --- a/nck/readers/dbm_reader.py +++ b/nck/readers/google_dbm/reader.py @@ -18,67 +18,21 @@ import datetime -import click import httplib2 import requests from click import ClickException from googleapiclient import discovery -from nck.commands.command import processor from nck.config import logger -from nck.helpers.dbm_helper import POSSIBLE_REQUEST_TYPES +from nck.readers.google_dbm.config import GOOGLE_TOKEN_URI from nck.readers.reader import Reader from nck.streams.format_date_stream import FormatDateStream -from nck.utils.args import extract_args from nck.utils.date_handler import check_date_range_definition_conformity, get_date_start_and_date_stop_from_date_range from nck.utils.text import get_report_generator_from_flat_file, skip_last from oauth2client import GOOGLE_REVOKE_URI, client from tenacity import retry, stop_after_delay, wait_exponential -DISCOVERY_URI = "https://analyticsreporting.googleapis.com/$discovery/rest" - - -@click.command(name="read_dbm") -@click.option("--dbm-access-token", default=None) -@click.option("--dbm-refresh-token", required=True) -@click.option("--dbm-client-id", required=True) -@click.option("--dbm-client-secret", required=True) -@click.option("--dbm-query-metric", multiple=True) -@click.option("--dbm-query-dimension", multiple=True) -@click.option("--dbm-request-type", type=click.Choice(POSSIBLE_REQUEST_TYPES), required=True) -@click.option("--dbm-query-id") -@click.option("--dbm-query-title") -@click.option("--dbm-query-frequency", default="ONE_TIME") -@click.option("--dbm-query-param-type", default="TYPE_TRUEVIEW") -@click.option("--dbm-start-date", type=click.DateTime()) -@click.option("--dbm-end-date", type=click.DateTime()) -@click.option( - "--dbm-add-date-to-report", - type=click.BOOL, - default=False, - help=( - "Sometimes the date range on which metrics are computed is missing from the report. " - "If this option is set to True, this range will be added." - ), -) -@click.option("--dbm-filter", type=click.Tuple([str, str]), multiple=True) -@click.option("--dbm-file-type", multiple=True) -@click.option( - "--dbm-date-format", - default="%Y-%m-%d", - help="And optional date format for the output stream. " - "Follow the syntax of https://docs.python.org/3.8/library/datetime.html#strftime-strptime-behavior", -) -@click.option( - "--dbm-day-range", - type=click.Choice(["PREVIOUS_DAY", "LAST_30_DAYS", "LAST_90_DAYS", "LAST_7_DAYS", "PREVIOUS_MONTH", "PREVIOUS_WEEK"]), -) -@processor("dbm_access_token", "dbm_refresh_token", "dbm_client_secret") -def dbm(**kwargs): - # Should add validation argument in function of request_type - return DbmReader(**extract_args("dbm_", kwargs)) - - -class DbmReader(Reader): + +class GoogleDBMReader(Reader): API_NAME = "doubleclickbidmanager" API_VERSION = "v1.1" @@ -89,7 +43,7 @@ def __init__(self, access_token, refresh_token, client_secret, client_id, **kwar client_secret=client_secret, refresh_token=refresh_token, token_expiry=None, - token_uri="https://accounts.google.com/o/oauth2/token", + token_uri=GOOGLE_TOKEN_URI, user_agent=None, revoke_uri=GOOGLE_REVOKE_URI, ) diff --git a/nck/readers/google_dcm/__init__.py b/nck/readers/google_dcm/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/google_dcm/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/google_dcm/cli.py b/nck/readers/google_dcm/cli.py new file mode 100644 index 00000000..296e2999 --- /dev/null +++ b/nck/readers/google_dcm/cli.py @@ -0,0 +1,64 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.google_dcm.config import REPORT_TYPES +from nck.readers.google_dcm.reader import GoogleDCMReader +from nck.utils.args import extract_args +from nck.utils.date_handler import DEFAULT_DATE_RANGE_FUNCTIONS +from nck.utils.processor import processor + + +@click.command(name="read_dcm") +@click.option("--dcm-access-token", default=None) +@click.option("--dcm-client-id", required=True) +@click.option("--dcm-client-secret", required=True) +@click.option("--dcm-refresh-token", required=True) +@click.option("--dcm-profile-id", "dcm_profile_ids", required=True, multiple=True) +@click.option("--dcm-report-name", default="DCM Report") +@click.option("--dcm-report-type", type=click.Choice(REPORT_TYPES), default=REPORT_TYPES[0]) +@click.option( + "--dcm-metric", + "dcm_metrics", + multiple=True, + help="https://developers.google.com/doubleclick-advertisers/v3.3/dimensions/#standard-metrics", +) +@click.option( + "--dcm-dimension", + "dcm_dimensions", + multiple=True, + help="https://developers.google.com/doubleclick-advertisers/v3.3/dimensions/#standard-dimensions", +) +@click.option("--dcm-start-date", type=click.DateTime(), help="Start date of the report") +@click.option("--dcm-end-date", type=click.DateTime(), help="End date of the report") +@click.option( + "--dcm-filter", + "dcm_filters", + type=click.Tuple([str, str]), + multiple=True, + help="A filter is a tuple following this pattern: (dimensionName, dimensionValue). " + "https://developers.google.com/doubleclick-advertisers/v3.3/dimensions/#standard-filters", +) +@click.option( + "--dcm-date-range", + type=click.Choice(DEFAULT_DATE_RANGE_FUNCTIONS.keys()), + help=f"One of the available NCK default date ranges: {DEFAULT_DATE_RANGE_FUNCTIONS.keys()}", +) +@processor("dcm_access_token", "dcm_refresh_token", "dcm_client_secret") +def google_dcm(**kwargs): + return GoogleDCMReader(**extract_args("dcm_", kwargs)) diff --git a/nck/helpers/dcm_helper.py b/nck/readers/google_dcm/config.py similarity index 97% rename from nck/helpers/dcm_helper.py rename to nck/readers/google_dcm/config.py index 05f64a7f..91afeec8 100644 --- a/nck/helpers/dcm_helper.py +++ b/nck/readers/google_dcm/config.py @@ -15,6 +15,10 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +ENCODING = "utf-8" +PREFIX = "^dfa:" + CRITERIA_MAPPING = { "STANDARD": "criteria", "REACH": "reachCriteria", diff --git a/nck/readers/dcm_reader.py b/nck/readers/google_dcm/reader.py similarity index 60% rename from nck/readers/dcm_reader.py rename to nck/readers/google_dcm/reader.py index 4736a27b..999ad4fa 100644 --- a/nck/readers/dcm_reader.py +++ b/nck/readers/google_dcm/reader.py @@ -15,66 +15,19 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -import csv -import click +import csv from io import StringIO -from nck.commands.command import processor +from nck.clients.google_dcm.client import GoogleDCMClient +from nck.readers.google_dcm.config import ENCODING, PREFIX from nck.readers.reader import Reader -from nck.utils.args import extract_args from nck.streams.json_stream import JSONStream -from nck.clients.dcm_client import DCMClient -from nck.helpers.dcm_helper import REPORT_TYPES -from nck.utils.date_handler import DEFAULT_DATE_RANGE_FUNCTIONS, build_date_range +from nck.utils.date_handler import build_date_range from nck.utils.text import strip_prefix -DATEFORMAT = "%Y-%m-%d" -ENCODING = "utf-8" -PREFIX = "^dfa:" - - -@click.command(name="read_dcm") -@click.option("--dcm-access-token", default=None) -@click.option("--dcm-client-id", required=True) -@click.option("--dcm-client-secret", required=True) -@click.option("--dcm-refresh-token", required=True) -@click.option("--dcm-profile-id", "dcm_profile_ids", required=True, multiple=True) -@click.option("--dcm-report-name", default="DCM Report") -@click.option("--dcm-report-type", type=click.Choice(REPORT_TYPES), default=REPORT_TYPES[0]) -@click.option( - "--dcm-metric", - "dcm_metrics", - multiple=True, - help="https://developers.google.com/doubleclick-advertisers/v3.3/dimensions/#standard-metrics", -) -@click.option( - "--dcm-dimension", - "dcm_dimensions", - multiple=True, - help="https://developers.google.com/doubleclick-advertisers/v3.3/dimensions/#standard-dimensions", -) -@click.option("--dcm-start-date", type=click.DateTime(), help="Start date of the report") -@click.option("--dcm-end-date", type=click.DateTime(), help="End date of the report") -@click.option( - "--dcm-filter", - "dcm_filters", - type=click.Tuple([str, str]), - multiple=True, - help="A filter is a tuple following this pattern: (dimensionName, dimensionValue). " - "https://developers.google.com/doubleclick-advertisers/v3.3/dimensions/#standard-filters", -) -@click.option( - "--dcm-date-range", - type=click.Choice(DEFAULT_DATE_RANGE_FUNCTIONS.keys()), - help=f"One of the available NCK default date ranges: {DEFAULT_DATE_RANGE_FUNCTIONS.keys()}", -) -@processor("dcm_access_token", "dcm_refresh_token", "dcm_client_secret") -def dcm(**kwargs): - return DcmReader(**extract_args("dcm_", kwargs)) - -class DcmReader(Reader): +class GoogleDCMReader(Reader): def __init__( self, access_token, @@ -91,7 +44,7 @@ def __init__( filters, date_range, ): - self.dcm_client = DCMClient(access_token, client_id, client_secret, refresh_token) + self.dcm_client = GoogleDCMClient(access_token, client_id, client_secret, refresh_token) self.profile_ids = list(profile_ids) self.report_name = report_name self.report_type = report_type diff --git a/nck/readers/google_dv360/__init__.py b/nck/readers/google_dv360/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/google_dv360/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/google_dv360/cli.py b/nck/readers/google_dv360/cli.py new file mode 100644 index 00000000..76b9e2dd --- /dev/null +++ b/nck/readers/google_dv360/cli.py @@ -0,0 +1,37 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.google_dv360.config import FILE_TYPES, FILTER_TYPES, REQUEST_TYPES +from nck.readers.google_dv360.reader import GoogleDV360Reader +from nck.utils.args import extract_args +from nck.utils.processor import processor + + +@click.command(name="read_dv360") +@click.option("--dv360-access-token", default=None, required=True) +@click.option("--dv360-refresh-token", required=True) +@click.option("--dv360-client-id", required=True) +@click.option("--dv360-client-secret", required=True) +@click.option("--dv360-advertiser-id", required=True) +@click.option("--dv360-request-type", type=click.Choice(REQUEST_TYPES), required=True) +@click.option("--dv360-file-type", type=click.Choice(FILE_TYPES), multiple=True) +@click.option("--dv360-filter-type", type=click.Choice(FILTER_TYPES)) +@processor("dv360_access_token", "dv360_refresh_token", "dv360_client_secret") +def google_dv360(**kwargs): + return GoogleDV360Reader(**extract_args("dv360_", kwargs)) diff --git a/nck/helpers/dv360_helper.py b/nck/readers/google_dv360/config.py similarity index 99% rename from nck/helpers/dv360_helper.py rename to nck/readers/google_dv360/config.py index 3fffab41..e60302b0 100644 --- a/nck/helpers/dv360_helper.py +++ b/nck/readers/google_dv360/config.py @@ -15,6 +15,7 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + FILE_NAMES = { "FILE_TYPE_INSERTION_ORDER": "InsertionOrders", "FILE_TYPE_CAMPAIGN": "Campaigns", diff --git a/nck/readers/dv360_reader.py b/nck/readers/google_dv360/reader.py similarity index 87% rename from nck/readers/dv360_reader.py rename to nck/readers/google_dv360/reader.py index f31ef3f6..95173b4a 100644 --- a/nck/readers/dv360_reader.py +++ b/nck/readers/google_dv360/reader.py @@ -15,45 +15,27 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from nck.streams.json_stream import JSONStream -import click -from nck.config import logger -import io -import httplib2 +import io from itertools import chain from typing import List +import httplib2 from googleapiclient import discovery from googleapiclient.http import MediaIoBaseDownload -from oauth2client import client, GOOGLE_REVOKE_URI -from tenacity import retry, wait_exponential, stop_after_delay - -from nck.helpers.dv360_helper import FILE_NAMES, FILE_TYPES, FILTER_TYPES, REQUEST_TYPES -from nck.utils.exceptions import RetryTimeoutError, SdfOperationError -from nck.commands.command import processor +from nck.config import logger +from nck.readers.google_dv360.config import FILE_NAMES from nck.readers.reader import Reader -from nck.utils.file_reader import sdf_to_njson_generator, unzip -from nck.utils.args import extract_args from nck.streams.format_date_stream import FormatDateStream +from nck.streams.json_stream import JSONStream +from nck.utils.exceptions import RetryTimeoutError, SdfOperationError +from nck.utils.file_reader import sdf_to_njson_generator, unzip from nck.utils.stdout_to_log import http_log, http_log_for_init +from oauth2client import GOOGLE_REVOKE_URI, client +from tenacity import retry, stop_after_delay, wait_exponential -@click.command(name="read_dv360") -@click.option("--dv360-access-token", default=None, required=True) -@click.option("--dv360-refresh-token", required=True) -@click.option("--dv360-client-id", required=True) -@click.option("--dv360-client-secret", required=True) -@click.option("--dv360-advertiser-id", required=True) -@click.option("--dv360-request-type", type=click.Choice(REQUEST_TYPES), required=True) -@click.option("--dv360-file-type", type=click.Choice(FILE_TYPES), multiple=True) -@click.option("--dv360-filter-type", type=click.Choice(FILTER_TYPES)) -@processor("dbm_access_token", "dbm_refresh_token", "dbm_client_secret") -def dv360(**kwargs): - return DV360Reader(**extract_args("dv360_", kwargs)) - - -class DV360Reader(Reader): +class GoogleDV360Reader(Reader): API_NAME = "displayvideo" API_VERSION = "v1" diff --git a/nck/readers/google_sa360/__init__.py b/nck/readers/google_sa360/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/google_sa360/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/google_sa360/cli.py b/nck/readers/google_sa360/cli.py new file mode 100644 index 00000000..4c248bca --- /dev/null +++ b/nck/readers/google_sa360/cli.py @@ -0,0 +1,59 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.google_sa360.config import REPORT_TYPES +from nck.readers.google_sa360.reader import GoogleSA360Reader +from nck.utils.args import extract_args +from nck.utils.date_handler import DEFAULT_DATE_RANGE_FUNCTIONS +from nck.utils.processor import processor + + +@click.command(name="read_sa360") +@click.option("--sa360-access-token", default=None) +@click.option("--sa360-client-id", required=True) +@click.option("--sa360-client-secret", required=True) +@click.option("--sa360-refresh-token", required=True) +@click.option("--sa360-agency-id", required=True) +@click.option( + "--sa360-advertiser-id", + "sa360_advertiser_ids", + multiple=True, + help="If empty, all advertisers from agency will be requested", +) +@click.option("--sa360-report-name", default="SA360 Report") +@click.option("--sa360-report-type", type=click.Choice(REPORT_TYPES), default=REPORT_TYPES[0]) +@click.option( + "--sa360-column", "sa360_columns", multiple=True, help="https://developers.google.com/search-ads/v2/report-types", +) +@click.option( + "--sa360-saved-column", + "sa360_saved_columns", + multiple=True, + help="https://developers.google.com/search-ads/v2/how-tos/reporting/saved-columns", +) +@click.option("--sa360-start-date", type=click.DateTime(), help="Start date of the report") +@click.option("--sa360-end-date", type=click.DateTime(), help="End date of the report") +@click.option( + "--sa360-date-range", + type=click.Choice(DEFAULT_DATE_RANGE_FUNCTIONS.keys()), + help=f"One of the available NCK default date ranges: {DEFAULT_DATE_RANGE_FUNCTIONS.keys()}", +) +@processor("sa360_access_token", "sa360_refresh_token", "sa360_client_secret") +def google_sa360(**kwargs): + return GoogleSA360Reader(**extract_args("sa360_", kwargs)) diff --git a/nck/helpers/sa360_helper.py b/nck/readers/google_sa360/config.py similarity index 99% rename from nck/helpers/sa360_helper.py rename to nck/readers/google_sa360/config.py index 7375996c..005b2acf 100644 --- a/nck/helpers/sa360_helper.py +++ b/nck/readers/google_sa360/config.py @@ -15,6 +15,7 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + REPORT_TYPES = [ "advertiser", "account", diff --git a/nck/readers/sa360_reader.py b/nck/readers/google_sa360/reader.py similarity index 58% rename from nck/readers/sa360_reader.py rename to nck/readers/google_sa360/reader.py index 42876cbd..0621a1bd 100644 --- a/nck/readers/sa360_reader.py +++ b/nck/readers/google_sa360/reader.py @@ -15,57 +15,15 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -import click -from nck.commands.command import processor +from nck.clients.google_sa360.client import GoogleSA360Client from nck.readers.reader import Reader from nck.streams.json_stream import JSONStream -from nck.clients.sa360_client import SA360Client -from nck.helpers.sa360_helper import REPORT_TYPES -from nck.utils.args import extract_args -from nck.utils.date_handler import DEFAULT_DATE_RANGE_FUNCTIONS, build_date_range +from nck.utils.date_handler import build_date_range from nck.utils.text import get_report_generator_from_flat_file -DATEFORMAT = "%Y-%m-%d" -ENCODING = "utf-8" - -@click.command(name="read_sa360") -@click.option("--sa360-access-token", default=None) -@click.option("--sa360-client-id", required=True) -@click.option("--sa360-client-secret", required=True) -@click.option("--sa360-refresh-token", required=True) -@click.option("--sa360-agency-id", required=True) -@click.option( - "--sa360-advertiser-id", - "sa360_advertiser_ids", - multiple=True, - help="If empty, all advertisers from agency will be requested", -) -@click.option("--sa360-report-name", default="SA360 Report") -@click.option("--sa360-report-type", type=click.Choice(REPORT_TYPES), default=REPORT_TYPES[0]) -@click.option( - "--sa360-column", "sa360_columns", multiple=True, help="https://developers.google.com/search-ads/v2/report-types", -) -@click.option( - "--sa360-saved-column", - "sa360_saved_columns", - multiple=True, - help="https://developers.google.com/search-ads/v2/how-tos/reporting/saved-columns", -) -@click.option("--sa360-start-date", type=click.DateTime(), help="Start date of the report") -@click.option("--sa360-end-date", type=click.DateTime(), help="End date of the report") -@click.option( - "--sa360-date-range", - type=click.Choice(DEFAULT_DATE_RANGE_FUNCTIONS.keys()), - help=f"One of the available NCK default date ranges: {DEFAULT_DATE_RANGE_FUNCTIONS.keys()}", -) -@processor("sa360_access_token", "sa360_refresh_token", "sa360_client_secret") -def sa360_reader(**kwargs): - return SA360Reader(**extract_args("sa360_", kwargs)) - - -class SA360Reader(Reader): +class GoogleSA360Reader(Reader): def __init__( self, access_token, @@ -82,7 +40,7 @@ def __init__( end_date, date_range, ): - self.sa360_client = SA360Client(access_token, client_id, client_secret, refresh_token) + self.sa360_client = GoogleSA360Client(access_token, client_id, client_secret, refresh_token) self.agency_id = agency_id self.advertiser_ids = list(advertiser_ids) self.report_name = report_name diff --git a/nck/readers/google_search_console/__init__.py b/nck/readers/google_search_console/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/google_search_console/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/google_search_console/cli.py b/nck/readers/google_search_console/cli.py new file mode 100644 index 00000000..dde2f711 --- /dev/null +++ b/nck/readers/google_search_console/cli.py @@ -0,0 +1,44 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.google_search_console.reader import GoogleSearchConsoleReader +from nck.utils.args import extract_args +from nck.utils.date_handler import DEFAULT_DATE_RANGE_FUNCTIONS +from nck.utils.processor import processor + + +@click.command(name="read_search_console") +@click.option("--search-console-client-id", required=True) +@click.option("--search-console-client-secret", required=True) +@click.option("--search-console-access-token", default="") +@click.option("--search-console-refresh-token", required=True) +@click.option("--search-console-dimensions", required=True, multiple=True) +@click.option("--search-console-site-url", required=True) +@click.option("--search-console-start-date", type=click.DateTime(), default=None) +@click.option("--search-console-end-date", type=click.DateTime(), default=None) +@click.option("--search-console-date-column", type=click.BOOL, default=False) +@click.option("--search-console-row-limit", type=click.INT, default=25000) +@click.option( + "--search-console-date-range", + type=click.Choice(DEFAULT_DATE_RANGE_FUNCTIONS.keys()), + help=f"One of the available NCK default date ranges: {DEFAULT_DATE_RANGE_FUNCTIONS.keys()}", +) +@processor() +def google_search_console(**params): + return GoogleSearchConsoleReader(**extract_args("search_console_", params)) diff --git a/nck/readers/google_search_console/config.py b/nck/readers/google_search_console/config.py new file mode 100644 index 00000000..d7abd201 --- /dev/null +++ b/nck/readers/google_search_console/config.py @@ -0,0 +1,20 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +DATEFORMAT = "%Y-%m-%d" +GOOGLE_TOKEN_URI = "https://accounts.google.com/o/oauth2/token" diff --git a/nck/readers/search_console_reader.py b/nck/readers/google_search_console/reader.py similarity index 77% rename from nck/readers/search_console_reader.py rename to nck/readers/google_search_console/reader.py index a965ae25..bb9bddee 100644 --- a/nck/readers/search_console_reader.py +++ b/nck/readers/google_search_console/reader.py @@ -15,50 +15,25 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from oauth2client.client import GoogleCredentials -from oauth2client import GOOGLE_REVOKE_URI -from googleapiclient.discovery import build -import httplib2 + from datetime import datetime, timedelta -import click +import httplib2 +from googleapiclient.discovery import build from nck.config import logger - -from nck.commands.command import processor +from nck.readers.google_search_console.config import DATEFORMAT, GOOGLE_TOKEN_URI from nck.readers.reader import Reader from nck.streams.json_stream import JSONStream -from nck.utils.args import extract_args -from nck.utils.date_handler import DEFAULT_DATE_RANGE_FUNCTIONS, build_date_range +from nck.utils.date_handler import build_date_range from nck.utils.retry import retry +from oauth2client import GOOGLE_REVOKE_URI +from oauth2client.client import GoogleCredentials - -@click.command(name="read_search_console") -@click.option("--search-console-client-id", required=True) -@click.option("--search-console-client-secret", required=True) -@click.option("--search-console-access-token", default="") -@click.option("--search-console-refresh-token", required=True) -@click.option("--search-console-dimensions", required=True, multiple=True) -@click.option("--search-console-site-url", required=True) -@click.option("--search-console-start-date", type=click.DateTime(), default=None) -@click.option("--search-console-end-date", type=click.DateTime(), default=None) -@click.option("--search-console-date-column", type=click.BOOL, default=False) -@click.option("--search-console-row-limit", type=click.INT, default=25000) -@click.option( - "--search-console-date-range", - type=click.Choice(DEFAULT_DATE_RANGE_FUNCTIONS.keys()), - help=f"One of the available NCK default date ranges: {DEFAULT_DATE_RANGE_FUNCTIONS.keys()}", -) -@processor() -def search_console(**params): - return SearchConsoleReader(**extract_args("search_console_", params)) - - -DATEFORMAT = "%Y-%m-%d" -# most recent data available is often 2 days ago. +# Most recent data available is often 2 days ago MAX_END_DATE = datetime.today() - timedelta(days=2) -class SearchConsoleReader(Reader): +class GoogleSearchConsoleReader(Reader): def __init__( self, client_id, @@ -95,7 +70,7 @@ def initialize_analyticsreporting(self): client_secret=self.client_secret, refresh_token=self.refresh_token, token_expiry=None, - token_uri="https://accounts.google.com/o/oauth2/token", + token_uri=GOOGLE_TOKEN_URI, user_agent=None, revoke_uri=GOOGLE_REVOKE_URI, ) diff --git a/nck/readers/google_sheets/__init__.py b/nck/readers/google_sheets/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/google_sheets/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/google_sheets/cli.py b/nck/readers/google_sheets/cli.py new file mode 100644 index 00000000..d5ee33e6 --- /dev/null +++ b/nck/readers/google_sheets/cli.py @@ -0,0 +1,73 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.google_sheets.reader import GoogleSheetsReader +from nck.utils.args import extract_args +from nck.utils.processor import processor + + +@click.command(name="read_gs") +@click.option( + "--gs-project-id", + required=True, + help="Project ID that is given by Google services once you have \ + created your project in the google cloud console. You can retrieve it in the JSON credential file", +) +@click.option( + "--gs-private-key-id", + required=True, + help="Private key ID given by Google services once you have added credentials \ + to the project. You can retrieve it in the JSON credential file", +) +@click.option( + "--gs-private-key", + required=True, + help="The private key given by Google services once you have added credentials \ + to the project. \ + You can retrieve it first in the JSON credential file", +) +@click.option( + "--gs-client-email", + required=True, + help="Client e-mail given by Google services once you have added credentials \ + to the project. You can retrieve it in the JSON credential file", +) +@click.option( + "--gs-client-id", + required=True, + help="Client ID given by Google services once you have added credentials \ + to the project. You can retrieve it in the JSON credential file", +) +@click.option( + "--gs-client-cert", + required=True, + help="Client certificate given by Google services once you have added credentials \ + to the project. You can retrieve it in the JSON credential file", +) +@click.option("--gs-sheet-key", required=True, help="Google spreadsheet key that is availbale in the url") +@click.option( + "--gs-page-number", + default=0, + type=click.INT, + help="The page number you want to access.\ + The number pages starts at 0", +) +@processor("gs_private_key_id", "gs_private_key", "gs_client_id", "gs_client_cert") +def google_sheets(**kwargs): + return GoogleSheetsReader(**extract_args("gs_", kwargs)) diff --git a/nck/readers/gs_reader.py b/nck/readers/google_sheets/reader.py similarity index 57% rename from nck/readers/gs_reader.py rename to nck/readers/google_sheets/reader.py index 9b553814..470d076a 100644 --- a/nck/readers/gs_reader.py +++ b/nck/readers/google_sheets/reader.py @@ -15,69 +15,15 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -import click + import gspread from google.auth.transport.requests import AuthorizedSession from google.oauth2 import service_account - -from nck.commands.command import processor from nck.readers.reader import Reader -from nck.utils.args import extract_args from nck.streams.json_stream import JSONStream -@click.command(name="read_gs") -@click.option( - "--gs-project-id", - required=True, - help="Project ID that is given by Google services once you have \ - created your project in the google cloud console. You can retrieve it in the JSON credential file", -) -@click.option( - "--gs-private-key-id", - required=True, - help="Private key ID given by Google services once you have added credentials \ - to the project. You can retrieve it in the JSON credential file", -) -@click.option( - "--gs-private-key", - required=True, - help="The private key given by Google services once you have added credentials \ - to the project. \ - You can retrieve it first in the JSON credential file", -) -@click.option( - "--gs-client-email", - required=True, - help="Client e-mail given by Google services once you have added credentials \ - to the project. You can retrieve it in the JSON credential file", -) -@click.option( - "--gs-client-id", - required=True, - help="Client ID given by Google services once you have added credentials \ - to the project. You can retrieve it in the JSON credential file", -) -@click.option( - "--gs-client-cert", - required=True, - help="Client certificate given by Google services once you have added credentials \ - to the project. You can retrieve it in the JSON credential file", -) -@click.option("--gs-sheet-key", required=True, help="Google spreadsheet key that is availbale in the url") -@click.option( - "--gs-page-number", - default=0, - type=click.INT, - help="The page number you want to access.\ - The number pages starts at 0", -) -@processor("gs_private_key_id", "gs_private_key", "gs_client_id", "gs_client_cert") -def google_sheets(**kwargs): - return GSheetsReader(**extract_args("gs_", kwargs)) - - -class GSheetsReader(Reader): +class GoogleSheetsReader(Reader): _scopes = [ "https://www.googleapis.com/auth/spreadsheets.readonly", "https://www.googleapis.com/auth/spreadsheets", @@ -98,9 +44,7 @@ def __init__( ): self._sheet_key = sheet_key self._page_number = page_number - credentials = self.__init_credentials( - project_id, private_key_id, private_key, client_email, client_id, client_cert - ) + credentials = self.__init_credentials(project_id, private_key_id, private_key, client_email, client_id, client_cert) scoped_credentials = credentials.with_scopes(self._scopes) self._gc = gspread.Client(auth=scoped_credentials) self._gc.session = AuthorizedSession(scoped_credentials) diff --git a/nck/readers/google_sheets_old/__init__.py b/nck/readers/google_sheets_old/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/google_sheets_old/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/google_sheets_old/cli.py b/nck/readers/google_sheets_old/cli.py new file mode 100644 index 00000000..5f3bef74 --- /dev/null +++ b/nck/readers/google_sheets_old/cli.py @@ -0,0 +1,30 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.google_sheets_old.reader import GoogleSheetsReaderOld +from nck.utils.args import extract_args +from nck.utils.processor import processor + + +@click.command(name="read_gsheets") +@click.option("--gsheets-url", required=True) +@click.option("--gsheets-worksheet-name", required=True, multiple=True) +@processor() +def google_sheets_old(**kwargs): + return GoogleSheetsReaderOld(**extract_args("gsheets_", kwargs)) diff --git a/nck/readers/gsheets_reader.py b/nck/readers/google_sheets_old/reader.py similarity index 82% rename from nck/readers/gsheets_reader.py rename to nck/readers/google_sheets_old/reader.py index e12d4a89..6656f726 100644 --- a/nck/readers/gsheets_reader.py +++ b/nck/readers/google_sheets_old/reader.py @@ -15,25 +15,14 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -import click -import gspread -from oauth2client.client import GoogleCredentials -from nck.commands.command import processor +import gspread from nck.readers.reader import Reader -from nck.utils.args import extract_args from nck.streams.json_stream import JSONStream +from oauth2client.client import GoogleCredentials -@click.command(name="read_gsheets") -@click.option("--gsheets-url", required=True) -@click.option("--gsheets-worksheet-name", required=True, multiple=True) -@processor() -def gsheets(**kwargs): - return GSheetsReader(**extract_args("gsheets_", kwargs)) - - -class GSheetsReader(Reader): +class GoogleSheetsReaderOld(Reader): _scopes = [ "https://spreadsheets.google.com/feeds", diff --git a/nck/readers/mysql/__init__.py b/nck/readers/mysql/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/mysql/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/mysql/cli.py b/nck/readers/mysql/cli.py new file mode 100644 index 00000000..5739c7af --- /dev/null +++ b/nck/readers/mysql/cli.py @@ -0,0 +1,72 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.mysql.reader import MySQLReader +from nck.utils.args import extract_args, has_arg, hasnt_arg +from nck.utils.processor import processor + + +@click.command(name="read_mysql") +@click.option("--mysql-user", required=True) +@click.option("--mysql-password", required=True) +@click.option("--mysql-host", required=True) +@click.option("--mysql-port", default=3306) +@click.option("--mysql-database", required=True) +@click.option("--mysql-watermark-column") +@click.option("--mysql-watermark-init") +@click.option("--mysql-query") +@click.option("--mysql-query-name") +@click.option("--mysql-table") +@click.option("--mysql-redis-state-service-name") +@click.option("--mysql-redis-state-service-host") +@click.option("--mysql-redis-state-service-port", default=6379) +@processor("mysql_password") +def mysql(**kwargs): + query_key = "mysql_query" + query_name_key = "mysql_query_name" + table_key = "mysql_table" + watermark_column_key = "mysql_watermark_column" + watermark_init_key = "mysql_watermark_init" + redis_state_service_keys = [ + "mysql_redis_state_service_name", + "mysql_redis_state_service_host", + "mysql_redis_state_service_port", + ] + + if hasnt_arg(query_key, kwargs) and hasnt_arg(table_key, kwargs): + raise click.BadParameter("Must specify either a table or a query for MySQL reader") + + if has_arg(query_key, kwargs) and has_arg(table_key, kwargs): + raise click.BadParameter("Cannot specify both a query and a table") + + if has_arg(query_key, kwargs) and hasnt_arg(query_name_key, kwargs): + raise click.BadParameter("Must specify a query name when running a MySQL query") + + redis_state_service_enabled = all([has_arg(key, kwargs) for key in redis_state_service_keys]) + + if has_arg(watermark_column_key, kwargs) and not redis_state_service_enabled: + raise click.BadParameter("You must configure state management to use MySQL watermarks") + + if hasnt_arg(watermark_column_key, kwargs) and redis_state_service_enabled: + raise click.BadParameter("You must specify a MySQL watermark when using state management") + + if hasnt_arg(watermark_init_key, kwargs) and redis_state_service_enabled: + raise click.BadParameter("You must specify an initial MySQL watermark value when using state management") + + return MySQLReader(**extract_args("mysql_", kwargs)) diff --git a/nck/utils/sql.py b/nck/readers/mysql/helper.py similarity index 74% rename from nck/utils/sql.py rename to nck/readers/mysql/helper.py index 4730a48b..388a3b33 100644 --- a/nck/utils/sql.py +++ b/nck/readers/mysql/helper.py @@ -15,52 +15,49 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + import sqlalchemy _engine_meta = {} -def get_meta(engine, schema): - global _engine_meta +def build_table_query(engine, schema, table, watermark_column, watermark_value): + if watermark_column and watermark_value: + return _build_table_query_with_watermark(engine, schema, table, watermark_column, watermark_value) + else: + return _build_table_query_without_watermark(engine, schema, table) - if engine not in _engine_meta: - _engine_meta[engine] = sqlalchemy.MetaData(engine, schema=schema) - return _engine_meta[engine] +def build_custom_query(engine, query, watermark_column, watermark_value): + statement = sqlalchemy.text(query) + if watermark_column: + params = {watermark_column: watermark_value} + statement = statement.bindparams(**params) -def get_table(engine, schema, table): - meta = get_meta(engine, schema) - table = sqlalchemy.Table(table, meta, autoload=True, autoload_with=engine) + return statement - return table +def _build_table_query_without_watermark(engine, schema, table): + return _get_table(engine, schema, table).select() -def build_table_query(engine, schema, table, watermark_column, watermark_value): - if watermark_column and watermark_value: - return build_table_query_with_watermark( - engine, schema, table, watermark_column, watermark_value - ) - else: - return build_table_query_without_watermark(engine, schema, table) +def _build_table_query_with_watermark(engine, schema, table, watermark_column, watermark_value): + t = _get_table(engine, schema, table) + return t.select().where(t.columns[watermark_column] > watermark_value) -def build_table_query_without_watermark(engine, schema, table): - return get_table(engine, schema, table).select() +def _get_table(engine, schema, table): + meta = _get_meta(engine, schema) + table = sqlalchemy.Table(table, meta, autoload=True, autoload_with=engine) -def build_table_query_with_watermark( - engine, schema, table, watermark_column, watermark_value -): - t = get_table(engine, schema, table) - return t.select().where(t.columns[watermark_column] > watermark_value) + return table -def build_custom_query(engine, query, watermark_column, watermark_value): - statement = sqlalchemy.text(query) +def _get_meta(engine, schema): + global _engine_meta - if watermark_column: - params = {watermark_column: watermark_value} - statement = statement.bindparams(**params) + if engine not in _engine_meta: + _engine_meta[engine] = sqlalchemy.MetaData(engine, schema=schema) - return statement + return _engine_meta[engine] diff --git a/nck/readers/mysql_reader.py b/nck/readers/mysql/reader.py similarity index 59% rename from nck/readers/mysql_reader.py rename to nck/readers/mysql/reader.py index c3686350..4668dd68 100644 --- a/nck/readers/mysql_reader.py +++ b/nck/readers/mysql/reader.py @@ -15,67 +15,14 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from nck.config import logger -import click import sqlalchemy +from nck.config import logger from nck.readers.reader import Reader -from nck.commands.command import processor from nck.streams.json_stream import JSONStream -from nck.utils.args import extract_args, has_arg, hasnt_arg from nck.utils.redis import RedisStateService from nck.utils.retry import retry -from nck.utils.sql import build_custom_query, build_table_query - - -@click.command(name="read_mysql") -@click.option("--mysql-user", required=True) -@click.option("--mysql-password", required=True) -@click.option("--mysql-host", required=True) -@click.option("--mysql-port", default=3306) -@click.option("--mysql-database", required=True) -@click.option("--mysql-watermark-column") -@click.option("--mysql-watermark-init") -@click.option("--mysql-query") -@click.option("--mysql-query-name") -@click.option("--mysql-table") -@click.option("--mysql-redis-state-service-name") -@click.option("--mysql-redis-state-service-host") -@click.option("--mysql-redis-state-service-port", default=6379) -@processor("mysql_password") -def mysql(**kwargs): - query_key = "mysql_query" - query_name_key = "mysql_query_name" - table_key = "mysql_table" - watermark_column_key = "mysql_watermark_column" - watermark_init_key = "mysql_watermark_init" - redis_state_service_keys = [ - "mysql_redis_state_service_name", - "mysql_redis_state_service_host", - "mysql_redis_state_service_port", - ] - - if hasnt_arg(query_key, kwargs) and hasnt_arg(table_key, kwargs): - raise click.BadParameter("Must specify either a table or a query for MySQL reader") - - if has_arg(query_key, kwargs) and has_arg(table_key, kwargs): - raise click.BadParameter("Cannot specify both a query and a table") - - if has_arg(query_key, kwargs) and hasnt_arg(query_name_key, kwargs): - raise click.BadParameter("Must specify a query name when running a MySQL query") - - redis_state_service_enabled = all([has_arg(key, kwargs) for key in redis_state_service_keys]) - - if has_arg(watermark_column_key, kwargs) and not redis_state_service_enabled: - raise click.BadParameter("You must configure state management to use MySQL watermarks") - - if hasnt_arg(watermark_column_key, kwargs) and redis_state_service_enabled: - raise click.BadParameter("You must specify a MySQL watermark when using state management") - - if hasnt_arg(watermark_init_key, kwargs) and redis_state_service_enabled: - raise click.BadParameter("You must specify an initial MySQL watermark value when using state management") - - return MySQLReader(**extract_args("mysql_", kwargs)) +from nck.readers.mysql.helper import build_custom_query, build_table_query class MySQLReader(Reader): diff --git a/nck/readers/mytarget/__init__.py b/nck/readers/mytarget/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/mytarget/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/mytarget/cli.py b/nck/readers/mytarget/cli.py new file mode 100644 index 00000000..d61fc1eb --- /dev/null +++ b/nck/readers/mytarget/cli.py @@ -0,0 +1,41 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.mytarget.config import REQUEST_TYPES +from nck.readers.mytarget.reader import MyTargetReader +from nck.utils.args import extract_args +from nck.utils.date_handler import DEFAULT_DATE_RANGE_FUNCTIONS +from nck.utils.processor import processor + + +@click.command(name="read_mytarget") +@click.option("--mytarget-client-id", required=True) +@click.option("--mytarget-client-secret", required=True) +@click.option("--mytarget-refresh-token", required=True) +@click.option("--mytarget-request-type", type=click.Choice(REQUEST_TYPES), required=True) +@click.option( + "--mytarget-date-range", + type=click.Choice(DEFAULT_DATE_RANGE_FUNCTIONS.keys()), + help=f"One of the available NCK default date ranges: {DEFAULT_DATE_RANGE_FUNCTIONS.keys()}", +) +@click.option("--mytarget-start-date", type=click.DateTime()) +@click.option("--mytarget-end-date", type=click.DateTime()) +@processor("mytarget-client-id", "mytarget-client-secret") +def mytarget(**kwargs): + return MyTargetReader(**extract_args("mytarget_", kwargs)) diff --git a/nck/readers/mytarget/config.py b/nck/readers/mytarget/config.py new file mode 100644 index 00000000..cbcda967 --- /dev/null +++ b/nck/readers/mytarget/config.py @@ -0,0 +1,72 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +LIMIT_REQUEST_MYTARGET = 20 + +REQUEST_TYPES = ["performance", "budget"] + +REQUEST_CONFIG = { + "refresh_agency_token": { + "url": "https://target.my.com/api/v2/oauth2/token.json", + "headers_type": "content_type", + "offset": False, + "_campaign_id": False, + "dates_required": False, + "ids": False, + }, + "get_campaign_ids_names": { + "url": "https://target.my.com/api/v2/campaigns.json?fields=id,name", + "headers_type": "authorization", + "offset": True, + "_campaign_id": False, + "dates_required": False, + "ids": False, + }, + "get_banner_ids_names": { + "url": "https://target.my.com/api/v2/banners.json?fields=id,name,campaign_id", + "headers_type": "authorization", + "offset": True, + "_campaign_id": False, + "dates_required": False, + "ids": False, + }, + "get_banner_stats": { + "url": "https://target.my.com/api/v2/statistics/banners/day.json", + "headers_type": "authorization", + "offset": False, + "_campaign_id": False, + "dates_required": True, + "ids": False, + }, + "get_campaign_budgets": { + "url": "https://target.my.com/api/v2/campaigns.json?fields=id,name,budget_limit,budget_limit_day", + "headers_type": "authorization", + "offset": True, + "_campaign_id": False, + "dates_required": False, + "ids": False, + }, + "get_campaign_dates": { + "url": "https://target.my.com/api/v2/campaigns.json?fields=id,name,date_start,date_end,status", + "headers_type": "authorization", + "offset": True, + "_campaign_id": False, + "dates_required": False, + "ids": False, + }, +} diff --git a/nck/readers/mytarget_reader.py b/nck/readers/mytarget/reader.py similarity index 90% rename from nck/readers/mytarget_reader.py rename to nck/readers/mytarget/reader.py index 0f55d3c8..a4936ffb 100644 --- a/nck/readers/mytarget_reader.py +++ b/nck/readers/mytarget/reader.py @@ -15,39 +15,18 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + import itertools -from datetime import date, datetime +from datetime import datetime from typing import Any, Dict, List, Tuple -import click + import requests -from tenacity import retry, wait_exponential, stop_after_delay -from nck.commands.command import processor -from nck.helpers.mytarget_helper import REQUEST_CONFIG, REQUEST_TYPES +from nck.readers.mytarget.config import LIMIT_REQUEST_MYTARGET, REQUEST_CONFIG from nck.readers.reader import Reader from nck.streams.json_stream import JSONStream from nck.utils.exceptions import MissingItemsInResponse -from nck.utils.args import extract_args -from nck.utils.date_handler import DEFAULT_DATE_RANGE_FUNCTIONS, build_date_range - - -@click.command(name="read_mytarget") -@click.option("--mytarget-client-id", required=True) -@click.option("--mytarget-client-secret", required=True) -@click.option("--mytarget-refresh-token", required=True) -@click.option("--mytarget-request-type", type=click.Choice(REQUEST_TYPES), required=True) -@click.option( - "--mytarget-date-range", - type=click.Choice(DEFAULT_DATE_RANGE_FUNCTIONS.keys()), - help=f"One of the available NCK default date ranges: {DEFAULT_DATE_RANGE_FUNCTIONS.keys()}", -) -@click.option("--mytarget-start-date", type=click.DateTime()) -@click.option("--mytarget-end-date", type=click.DateTime()) -@processor("mytarget-client-id", "mytarget-client-secret") -def mytarget(**kwargs): - return MyTargetReader(**extract_args("mytarget_", kwargs)) - - -LIMIT_REQUEST_MYTARGET = 20 +from tenacity import retry, stop_after_delay, wait_exponential +from nck.utils.date_handler import build_date_range class MyTargetReader(Reader): @@ -82,10 +61,10 @@ def __check_date_input_validity(self) -> bool: ) def __check_date_not_in_future(self, end_date: datetime) -> bool: - if end_date <= date.today(): + if end_date <= datetime.today(): return True else: - raise ValueError(f"The end date {end_date} is posterior to current date {date.today()}") + raise ValueError(f"The end date {end_date} is posterior to current date {datetime.today()}") def __check_end_posterior_to_start(self, start_date: datetime, end_date: datetime) -> bool: if start_date > end_date: diff --git a/nck/readers/object_storage/__init__.py b/nck/readers/object_storage/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/object_storage/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/objectstorage_reader.py b/nck/readers/object_storage/reader.py similarity index 99% rename from nck/readers/objectstorage_reader.py rename to nck/readers/object_storage/reader.py index 63391250..7a107ad9 100644 --- a/nck/readers/objectstorage_reader.py +++ b/nck/readers/object_storage/reader.py @@ -15,6 +15,7 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + import tempfile from nck import config diff --git a/nck/readers/radarly/__init__.py b/nck/readers/radarly/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/radarly/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/radarly/cli.py b/nck/readers/radarly/cli.py new file mode 100644 index 00000000..a24dac7b --- /dev/null +++ b/nck/readers/radarly/cli.py @@ -0,0 +1,80 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.radarly.reader import RadarlyReader +from nck.utils.args import extract_args +from nck.utils.processor import processor + + +@click.command(name="read_radarly") +@click.option("--radarly-pid", required=True, type=click.INT, help="Radarly Project ID") +@click.option("--radarly-client-id", required=True, type=click.STRING) +@click.option("--radarly-client-secret", required=True, type=click.STRING) +@click.option( + "--radarly-focus-id", + required=True, + multiple=True, + type=click.INT, + help="Focus IDs (from Radarly queries)", +) +@click.option( + "--radarly-start-date", + required=True, + type=click.DateTime(formats=["%Y-%m-%d", "%Y-%m-%dT%H:%M:%SZ", "%Y-%m-%d %H:%M:%S"]), +) +@click.option( + "--radarly-end-date", + required=True, + type=click.DateTime(formats=["%Y-%m-%d", "%Y-%m-%dT%H:%M:%SZ", "%Y-%m-%d %H:%M:%S"]), +) +@click.option( + "--radarly-api-request-limit", + default=250, + type=click.INT, + help="Max number of posts per API request", +) +@click.option( + "--radarly-api-date-period-limit", + default=int(1e4), + type=click.INT, + help="Max number of posts in a single API search query", +) +@click.option( + "--radarly-api-quarterly-posts-limit", + default=int(45e3), + type=click.INT, + help="Max number of posts requested in the window (usually 15 min) (see Radarly documentation)", +) +@click.option( + "--radarly-api-window", + default=300, + type=click.INT, + help="Duration of the window (usually 300 seconds)", +) +@click.option( + "--radarly-throttle", + default=True, + type=click.BOOL, + help="""If set to True, forces the connector to abide by official Radarly API limitations + (using the api-quarterly-posts-limit parameter)""", +) +@click.option("--radarly-throttling-threshold-coefficient", default=0.95, type=click.FLOAT) +@processor("radarly_client_id", "radarly_client_secret") +def radarly(**kwargs): + return RadarlyReader(**extract_args("radarly_", kwargs)) diff --git a/nck/readers/radarly_reader.py b/nck/readers/radarly/reader.py similarity index 81% rename from nck/readers/radarly_reader.py rename to nck/readers/radarly/reader.py index 9fbc0693..d3973ba5 100644 --- a/nck/readers/radarly_reader.py +++ b/nck/readers/radarly/reader.py @@ -15,28 +15,23 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -import click -from nck.config import logger import sys -import traceback import time -import numpy as np -from datetime import datetime, timedelta - -from typing import List, Dict, Tuple -from typing import NamedTuple +import traceback from collections import OrderedDict +from datetime import datetime, timedelta +from typing import Dict, List, NamedTuple, Tuple +import numpy as np +from nck.config import logger from nck.readers import Reader -from nck.commands.command import processor from nck.streams.json_stream import JSONStream from nck.utils.retry import retry -from nck.utils.args import extract_args from radarly import RadarlyApi -from radarly.project import Project from radarly.parameters import SearchPublicationParameter as Payload +from radarly.project import Project class DateRangeSplit(NamedTuple): @@ -44,64 +39,6 @@ class DateRangeSplit(NamedTuple): is_compliant: bool -@click.command(name="read_radarly") -@click.option("--radarly-pid", required=True, type=click.INT, help="Radarly Project ID") -@click.option("--radarly-client-id", required=True, type=click.STRING) -@click.option("--radarly-client-secret", required=True, type=click.STRING) -@click.option( - "--radarly-focus-id", - required=True, - multiple=True, - type=click.INT, - help="Focus IDs (from Radarly queries)", -) -@click.option( - "--radarly-start-date", - required=True, - type=click.DateTime(formats=["%Y-%m-%d", "%Y-%m-%dT%H:%M:%SZ", "%Y-%m-%d %H:%M:%S"]), -) -@click.option( - "--radarly-end-date", - required=True, - type=click.DateTime(formats=["%Y-%m-%d", "%Y-%m-%dT%H:%M:%SZ", "%Y-%m-%d %H:%M:%S"]), -) -@click.option( - "--radarly-api-request-limit", - default=250, - type=click.INT, - help="Max number of posts per API request", -) -@click.option( - "--radarly-api-date-period-limit", - default=int(1e4), - type=click.INT, - help="Max number of posts in a single API search query", -) -@click.option( - "--radarly-api-quarterly-posts-limit", - default=int(45e3), - type=click.INT, - help="Max number of posts requested in the window (usually 15 min) (see Radarly documentation)", -) -@click.option( - "--radarly-api-window", - default=300, - type=click.INT, - help="Duration of the window (usually 300 seconds)", -) -@click.option( - "--radarly-throttle", - default=True, - type=click.BOOL, - help="""If set to True, forces the connector to abide by official Radarly API limitations - (using the api-quarterly-posts-limit parameter)""", -) -@click.option("--radarly-throttling-threshold-coefficient", default=0.95, type=click.FLOAT) -@processor("radarly_client_id", "radarly_client_secret") -def radarly(**kwargs): - return RadarlyReader(**extract_args("radarly_", kwargs)) - - class RadarlyReader(Reader): def __init__( self, diff --git a/nck/readers/salesforce/__init__.py b/nck/readers/salesforce/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/salesforce/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/salesforce/cli.py b/nck/readers/salesforce/cli.py new file mode 100644 index 00000000..5544a566 --- /dev/null +++ b/nck/readers/salesforce/cli.py @@ -0,0 +1,71 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.salesforce.reader import SalesforceReader +from nck.utils.args import extract_args, has_arg, hasnt_arg +from nck.utils.processor import processor + + +@click.command(name="read_salesforce") +@click.option("--salesforce-consumer-key", required=True) +@click.option("--salesforce-consumer-secret", required=True) +@click.option("--salesforce-user", required=True) +@click.option("--salesforce-password", required=True) +@click.option("--salesforce-object-type") +@click.option("--salesforce-query") +@click.option("--salesforce-query-name") +@click.option("--salesforce-watermark-column") +@click.option("--salesforce-watermark-init") +@click.option("--salesforce-redis-state-service-name") +@click.option("--salesforce-redis-state-service-host") +@click.option("--salesforce-redis-state-service-port", default=6379) +@processor("salesforce_consumer_key", "salesforce_consumer_secret", "salesforce_password") +def salesforce(**kwargs): + query_key = "salesforce_query" + query_name_key = "salesforce_query_name" + object_type_key = "salesforce_object_type" + watermark_column_key = "salesforce_watermark_column" + watermark_init_key = "salesforce_watermark_init" + redis_state_service_keys = [ + "salesforce_redis_state_service_name", + "salesforce_redis_state_service_host", + "salesforce_redis_state_service_port", + ] + + if hasnt_arg(query_key, kwargs) and hasnt_arg(object_type_key, kwargs): + raise click.BadParameter("Must specify either an object type or a query for Salesforce") + + if has_arg(query_key, kwargs) and has_arg(object_type_key, kwargs): + raise click.BadParameter("Cannot specify both a query and an object type for Salesforce") + + if has_arg(query_key, kwargs) and hasnt_arg(query_name_key, kwargs): + raise click.BadParameter("Must specify a query name when running a Salesforce query") + + redis_state_service_enabled = all([has_arg(key, kwargs) for key in redis_state_service_keys]) + + if has_arg(watermark_column_key, kwargs) and not redis_state_service_enabled: + raise click.BadParameter("You must configure state management to use Salesforce watermarks") + + if hasnt_arg(watermark_column_key, kwargs) and redis_state_service_enabled: + raise click.BadParameter("You must specify a Salesforce watermark when using state management") + + if hasnt_arg(watermark_init_key, kwargs) and redis_state_service_enabled: + raise click.BadParameter("You must specify an initial Salesforce watermark value when using state management") + + return SalesforceReader(**extract_args("salesforce_", kwargs)) diff --git a/nck/readers/salesforce/config.py b/nck/readers/salesforce/config.py new file mode 100644 index 00000000..72d73262 --- /dev/null +++ b/nck/readers/salesforce/config.py @@ -0,0 +1,23 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +SALESFORCE_LOGIN_ENDPOINT = "https://login.salesforce.com/services/oauth2/token" +SALESFORCE_LOGIN_REDIRECT = "https://login.salesforce.com/services/oauth2/success" +SALESFORCE_SERVICE_ENDPOINT = "https://eu16.force.com" +SALESFORCE_QUERY_ENDPOINT = "/services/data/v42.0/query/" +SALESFORCE_DESCRIBE_ENDPOINT = "/services/data/v42.0/sobjects/{obj}/describe" diff --git a/nck/readers/salesforce/reader.py b/nck/readers/salesforce/reader.py new file mode 100644 index 00000000..2d1aa659 --- /dev/null +++ b/nck/readers/salesforce/reader.py @@ -0,0 +1,125 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import collections + +from nck.readers.reader import Reader +from nck.clients.salesforce.client import SalesforceClient +from nck.streams.json_stream import JSONStream +from nck.utils.redis import RedisStateService +from nck.utils.retry import retry + + +class SalesforceReader(Reader): + def __init__( + self, + consumer_key, + consumer_secret, + user, + password, + query, + query_name, + object_type, + watermark_column, + watermark_init, + redis_state_service_name, + redis_state_service_host, + redis_state_service_port, + ): + self._name = query_name or object_type + self._client = SalesforceClient(user, password, consumer_key, consumer_secret) + self._watermark_column = watermark_column + self._watermark_init = watermark_init + self._object_type = object_type + self._query = query + self._redis_state_service = RedisStateService( + redis_state_service_name, redis_state_service_host, redis_state_service_port + ) + + def build_object_type_query(self, object_type, watermark_column): + description = self._client.describe(object_type) + fields = [f["name"] for f in description["fields"]] + + field_projection = ", ".join(fields) + query = "SELECT {fields} FROM {object_type}".format(fields=field_projection, object_type=object_type) + + if watermark_column: + query = "{base} WHERE {watermark_column} > {{{watermark_column}}}".format( + base=query, watermark_column=watermark_column + ) + + return query + + @retry + def read(self): + def result_generator(): + + watermark_value = None + + if self._watermark_column: + watermark_value = self._redis_state_service.get(self._name) or self._watermark_init + + if self._object_type: + self._query = self.build_object_type_query(self._object_type, self._watermark_column) + + if self._watermark_column: + self._query = self._query.format(**{self._watermark_column: watermark_value}) + + records = self._client.query(self._query) + + for rec in records: + row = self._clean_record(rec) + yield row + + if self._watermark_column: + self._redis_state_service.set(self._name, row[self._watermark_column]) + + yield JSONStream(self._name, result_generator()) + + @classmethod + def _clean_record(cls, record): + """ + Salesforces records contains metadata which we don't need during ingestion + """ + return cls._flatten(cls._delete_metadata_from_record(record)) + + @classmethod + def _delete_metadata_from_record(cls, record): + + if isinstance(record, dict): + strip_keys = ["attributes", "totalSize", "done"] + return {k: cls._delete_metadata_from_record(v) for k, v in record.items() if k not in strip_keys} + elif isinstance(record, list): + return [cls._delete_metadata_from_record(i) for i in record] + else: + return record + + @classmethod + def _flatten(cls, json_dict, parent_key="", sep="_"): + """ + Reduce number of dict levels + Note: useful to bigquery autodetect schema + """ + items = [] + for k, v in json_dict.items(): + new_key = parent_key + sep + k if parent_key else k + if isinstance(v, collections.MutableMapping): + items.extend(cls._flatten(v, new_key, sep=sep).items()) + else: + items.append((new_key, v)) + return dict(items) diff --git a/nck/readers/salesforce_reader.py b/nck/readers/salesforce_reader.py deleted file mode 100644 index 824734a3..00000000 --- a/nck/readers/salesforce_reader.py +++ /dev/null @@ -1,272 +0,0 @@ -# GNU Lesser General Public License v3.0 only -# Copyright (C) 2020 Artefact -# licence-information@artefact.com -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 3 of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public License -# along with this program; if not, write to the Free Software Foundation, -# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -import collections -from nck.config import logger -import urllib - -import click -import requests -from nck.commands.command import processor -from nck.readers.reader import Reader -from nck.streams.json_stream import JSONStream -from nck.utils.args import extract_args, has_arg, hasnt_arg -from nck.utils.redis import RedisStateService -from nck.utils.retry import retry - -SALESFORCE_LOGIN_ENDPOINT = "https://login.salesforce.com/services/oauth2/token" -SALESFORCE_LOGIN_REDIRECT = "https://login.salesforce.com/services/oauth2/success" -SALESFORCE_SERVICE_ENDPOINT = "https://eu16.force.com" -SALESFORCE_QUERY_ENDPOINT = "/services/data/v42.0/query/" -SALESFORCE_DESCRIBE_ENDPOINT = "/services/data/v42.0/sobjects/{obj}/describe" - - -@click.command(name="read_salesforce") -@click.option("--salesforce-consumer-key", required=True) -@click.option("--salesforce-consumer-secret", required=True) -@click.option("--salesforce-user", required=True) -@click.option("--salesforce-password", required=True) -@click.option("--salesforce-object-type") -@click.option("--salesforce-query") -@click.option("--salesforce-query-name") -@click.option("--salesforce-watermark-column") -@click.option("--salesforce-watermark-init") -@click.option("--salesforce-redis-state-service-name") -@click.option("--salesforce-redis-state-service-host") -@click.option("--salesforce-redis-state-service-port", default=6379) -@processor("salesforce_consumer_key", "salesforce_consumer_secret", "salesforce_password") -def salesforce(**kwargs): - query_key = "salesforce_query" - query_name_key = "salesforce_query_name" - object_type_key = "salesforce_object_type" - watermark_column_key = "salesforce_watermark_column" - watermark_init_key = "salesforce_watermark_init" - redis_state_service_keys = [ - "salesforce_redis_state_service_name", - "salesforce_redis_state_service_host", - "salesforce_redis_state_service_port", - ] - - if hasnt_arg(query_key, kwargs) and hasnt_arg(object_type_key, kwargs): - raise click.BadParameter("Must specify either an object type or a query for Salesforce") - - if has_arg(query_key, kwargs) and has_arg(object_type_key, kwargs): - raise click.BadParameter("Cannot specify both a query and an object type for Salesforce") - - if has_arg(query_key, kwargs) and hasnt_arg(query_name_key, kwargs): - raise click.BadParameter("Must specify a query name when running a Salesforce query") - - redis_state_service_enabled = all([has_arg(key, kwargs) for key in redis_state_service_keys]) - - if has_arg(watermark_column_key, kwargs) and not redis_state_service_enabled: - raise click.BadParameter("You must configure state management to use Salesforce watermarks") - - if hasnt_arg(watermark_column_key, kwargs) and redis_state_service_enabled: - raise click.BadParameter("You must specify a Salesforce watermark when using state management") - - if hasnt_arg(watermark_init_key, kwargs) and redis_state_service_enabled: - raise click.BadParameter("You must specify an initial Salesforce watermark value when using state management") - - return SalesforceReader(**extract_args("salesforce_", kwargs)) - - -class SalesforceClient: - def __init__(self, user, password, consumer_key, consumer_secret): - self._user = user - self._password = password - self._consumer_key = consumer_key - self._consumer_secret = consumer_secret - - self._headers = None - self._access_token = None - self._instance_url = None - - @property - def headers(self): - return { - "Content-type": "application/json", - "Accept-Encoding": "gzip", - "Authorization": f"Bearer {self.access_token}", - } - - @property - def access_token(self): - if not self._access_token: - self._load_access_info() - - return self._access_token - - @property - def instance_url(self): - if not self._instance_url: - self._load_access_info() - - return self._instance_url - - def _load_access_info(self): - logger.info("Retrieving Salesforce access token") - - res = requests.post(SALESFORCE_LOGIN_ENDPOINT, params=self._get_login_params()) - - res.raise_for_status() - - self._access_token = res.json().get("access_token") - self._instance_url = res.json().get("instance_url") - - return self._access_token, self._instance_url - - def _get_login_params(self): - return { - "grant_type": "password", - "client_id": self._consumer_key, - "client_secret": self._consumer_secret, - "username": self._user, - "password": self._password, - "redirect_uri": SALESFORCE_LOGIN_REDIRECT, - } - - def _request_data(self, path, params=None): - - endpoint = urllib.parse.urljoin(self.instance_url, path) - response = requests.get(endpoint, headers=self.headers, params=params, timeout=30) - - response.raise_for_status() - - return response.json() - - def describe(self, object_type): - path = SALESFORCE_DESCRIBE_ENDPOINT.format(obj=object_type) - return self._request_data(path) - - def query(self, query): - - logger.info(f"Running Salesforce query: {query}") - - response = self._request_data(SALESFORCE_QUERY_ENDPOINT, {"q": query}) - - generating = True - - while generating: - - for rec in response["records"]: - yield rec - - if "nextRecordsUrl" in response: - logger.info("Fetching next page of Salesforce results") - response = self._request_data(response["nextRecordsUrl"]) - else: - generating = False - - -class SalesforceReader(Reader): - def __init__( - self, - consumer_key, - consumer_secret, - user, - password, - query, - query_name, - object_type, - watermark_column, - watermark_init, - redis_state_service_name, - redis_state_service_host, - redis_state_service_port, - ): - self._name = query_name or object_type - self._client = SalesforceClient(user, password, consumer_key, consumer_secret) - self._watermark_column = watermark_column - self._watermark_init = watermark_init - self._object_type = object_type - self._query = query - self._redis_state_service = RedisStateService( - redis_state_service_name, redis_state_service_host, redis_state_service_port - ) - - def build_object_type_query(self, object_type, watermark_column): - description = self._client.describe(object_type) - fields = [f["name"] for f in description["fields"]] - - field_projection = ", ".join(fields) - query = "SELECT {fields} FROM {object_type}".format(fields=field_projection, object_type=object_type) - - if watermark_column: - query = "{base} WHERE {watermark_column} > {{{watermark_column}}}".format( - base=query, watermark_column=watermark_column - ) - - return query - - @retry - def read(self): - def result_generator(): - - watermark_value = None - - if self._watermark_column: - watermark_value = self._redis_state_service.get(self._name) or self._watermark_init - - if self._object_type: - self._query = self.build_object_type_query(self._object_type, self._watermark_column) - - if self._watermark_column: - self._query = self._query.format(**{self._watermark_column: watermark_value}) - - records = self._client.query(self._query) - - for rec in records: - row = self._clean_record(rec) - yield row - - if self._watermark_column: - self._redis_state_service.set(self._name, row[self._watermark_column]) - - yield JSONStream(self._name, result_generator()) - - @classmethod - def _clean_record(cls, record): - """ - Salesforces records contains metadata which we don't need during ingestion - """ - return cls._flatten(cls._delete_metadata_from_record(record)) - - @classmethod - def _delete_metadata_from_record(cls, record): - - if isinstance(record, dict): - strip_keys = ["attributes", "totalSize", "done"] - return {k: cls._delete_metadata_from_record(v) for k, v in record.items() if k not in strip_keys} - elif isinstance(record, list): - return [cls._delete_metadata_from_record(i) for i in record] - else: - return record - - @classmethod - def _flatten(cls, json_dict, parent_key="", sep="_"): - """ - Reduce number of dict levels - Note: useful to bigquery autodetect schema - """ - items = [] - for k, v in json_dict.items(): - new_key = parent_key + sep + k if parent_key else k - if isinstance(v, collections.MutableMapping): - items.extend(cls._flatten(v, new_key, sep=sep).items()) - else: - items.append((new_key, v)) - return dict(items) diff --git a/nck/readers/the_trade_desk/__init__.py b/nck/readers/the_trade_desk/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/the_trade_desk/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/the_trade_desk/cli.py b/nck/readers/the_trade_desk/cli.py new file mode 100644 index 00000000..c43fd5c1 --- /dev/null +++ b/nck/readers/the_trade_desk/cli.py @@ -0,0 +1,54 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.the_trade_desk.reader import TheTradeDeskReader +from nck.utils.args import extract_args +from nck.utils.date_handler import DEFAULT_DATE_RANGE_FUNCTIONS +from nck.utils.processor import processor + + +@click.command(name="read_ttd") +@click.option("--ttd-login", required=True, help="Login of your API account") +@click.option("--ttd-password", required=True, help="Password of your API account") +@click.option( + "--ttd-advertiser-id", required=True, multiple=True, help="Advertiser Ids for which report data should be fetched", +) +@click.option( + "--ttd-report-template-name", + required=True, + help="Exact name of the Report Template to request. Existing Report Templates " + "can be found within the MyReports section of The Trade Desk UI.", +) +@click.option( + "--ttd-report-schedule-name", required=True, help="Name of the Report Schedule to create.", +) +@click.option( + "--ttd-start-date", type=click.DateTime(), help="Start date of the period to request (format: YYYY-MM-DD)", +) +@click.option( + "--ttd-end-date", type=click.DateTime(), help="End date of the period to request (format: YYYY-MM-DD)", +) +@click.option( + "--ttd-date-range", + type=click.Choice(DEFAULT_DATE_RANGE_FUNCTIONS.keys()), + help=f"One of the available NCK default date ranges: {DEFAULT_DATE_RANGE_FUNCTIONS.keys()}", +) +@processor("ttd_login", "ttd_password") +def the_trade_desk(**kwargs): + return TheTradeDeskReader(**extract_args("ttd_", kwargs)) diff --git a/nck/helpers/ttd_helper.py b/nck/readers/the_trade_desk/config.py similarity index 66% rename from nck/helpers/ttd_helper.py rename to nck/readers/the_trade_desk/config.py index d92124fc..2d0ea068 100644 --- a/nck/helpers/ttd_helper.py +++ b/nck/readers/the_trade_desk/config.py @@ -1,3 +1,9 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public # License as published by the Free Software Foundation; either # version 3 of the License, or (at your option) any later version. # @@ -10,15 +16,15 @@ # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from nck.config import logger -from datetime import datetime - API_HOST = "https://api.thetradedesk.com/v3" API_ENDPOINTS = { "get_report_template_id": ("POST", "myreports/reporttemplateheader/query"), "create_report_schedule": ("POST", "myreports/reportschedule"), - "get_report_execution_details": ("POST", "myreports/reportexecution/query/advertisers",), + "get_report_execution_details": ( + "POST", + "myreports/reportexecution/query/advertisers", + ), "delete_report_schedule": ("DELETE", "/myreports/reportschedule"), } @@ -39,23 +45,3 @@ API_DATEFORMAT = "%Y-%m-%dT%H:%M:%S" BQ_DATEFORMAT = "%Y-%m-%d" - - -class ReportTemplateNotFoundError(Exception): - def __init__(self, message): - super().__init__(message) - logger.error(message) - - -class ReportScheduleNotReadyError(Exception): - def __init__(self, message): - super().__init__(message) - logger.error(message) - - -def format_date(date_string): - """ - Input: "2020-01-01T00:00:00" - Output: "2020-01-01" - """ - return datetime.strptime(date_string, API_DATEFORMAT).strftime(BQ_DATEFORMAT) diff --git a/nck/readers/the_trade_desk/helper.py b/nck/readers/the_trade_desk/helper.py new file mode 100644 index 00000000..0da2e302 --- /dev/null +++ b/nck/readers/the_trade_desk/helper.py @@ -0,0 +1,23 @@ +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +from datetime import datetime + +from nck.readers.the_trade_desk.config import API_DATEFORMAT, BQ_DATEFORMAT + + +def format_date(date_string): + """ + Input: "2020-01-01T00:00:00" + Output: "2020-01-01" + """ + return datetime.strptime(date_string, API_DATEFORMAT).strftime(BQ_DATEFORMAT) diff --git a/nck/readers/ttd_reader.py b/nck/readers/the_trade_desk/reader.py similarity index 79% rename from nck/readers/ttd_reader.py rename to nck/readers/the_trade_desk/reader.py index d59292b6..9afc9b76 100644 --- a/nck/readers/ttd_reader.py +++ b/nck/readers/the_trade_desk/reader.py @@ -1,3 +1,9 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public # License as published by the Free Software Foundation; either # version 3 of the License, or (at your option) any later version. # @@ -10,58 +16,19 @@ # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from nck.config import logger -import click -import requests + from datetime import timedelta -from tenacity import retry, wait_exponential, stop_after_delay -from nck.utils.args import extract_args -from nck.commands.command import processor +import requests +from nck.config import logger from nck.readers.reader import Reader +from nck.readers.the_trade_desk.config import API_ENDPOINTS, API_HOST, DEFAULT_PAGING_ARGS, DEFAULT_REPORT_SCHEDULE_ARGS +from nck.readers.the_trade_desk.helper import format_date from nck.streams.json_stream import JSONStream -from nck.helpers.ttd_helper import ( - API_HOST, - API_ENDPOINTS, - DEFAULT_REPORT_SCHEDULE_ARGS, - DEFAULT_PAGING_ARGS, - ReportTemplateNotFoundError, - ReportScheduleNotReadyError, - format_date, -) -from nck.utils.date_handler import DEFAULT_DATE_RANGE_FUNCTIONS, build_date_range +from nck.utils.date_handler import build_date_range +from nck.utils.exceptions import ReportScheduleNotReadyError, ReportTemplateNotFoundError from nck.utils.text import get_report_generator_from_flat_file - - -@click.command(name="read_ttd") -@click.option("--ttd-login", required=True, help="Login of your API account") -@click.option("--ttd-password", required=True, help="Password of your API account") -@click.option( - "--ttd-advertiser-id", required=True, multiple=True, help="Advertiser Ids for which report data should be fetched", -) -@click.option( - "--ttd-report-template-name", - required=True, - help="Exact name of the Report Template to request. Existing Report Templates " - "can be found within the MyReports section of The Trade Desk UI.", -) -@click.option( - "--ttd-report-schedule-name", required=True, help="Name of the Report Schedule to create.", -) -@click.option( - "--ttd-start-date", type=click.DateTime(), help="Start date of the period to request (format: YYYY-MM-DD)", -) -@click.option( - "--ttd-end-date", type=click.DateTime(), help="End date of the period to request (format: YYYY-MM-DD)", -) -@click.option( - "--ttd-date-range", - type=click.Choice(DEFAULT_DATE_RANGE_FUNCTIONS.keys()), - help=f"One of the available NCK default date ranges: {DEFAULT_DATE_RANGE_FUNCTIONS.keys()}", -) -@processor("ttd_login", "ttd_password") -def the_trade_desk(**kwargs): - return TheTradeDeskReader(**extract_args("ttd_", kwargs)) +from tenacity import retry, stop_after_delay, wait_exponential class TheTradeDeskReader(Reader): diff --git a/nck/readers/twitter/__init__.py b/nck/readers/twitter/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/twitter/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/twitter/cli.py b/nck/readers/twitter/cli.py new file mode 100644 index 00000000..2704494d --- /dev/null +++ b/nck/readers/twitter/cli.py @@ -0,0 +1,132 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.twitter.config import ( + ENTITY_ATTRIBUTES, + GRANULARITIES, + METRIC_GROUPS, + PLACEMENTS, + REPORT_TYPES, + SEGMENTATION_TYPES, +) +from nck.readers.twitter.reader import TwitterReader +from nck.utils.args import extract_args +from nck.utils.date_handler import DEFAULT_DATE_RANGE_FUNCTIONS +from nck.utils.processor import processor + + +@click.command(name="read_twitter") +@click.option( + "--twitter-consumer-key", + required=True, + help="API key, available in the 'Keys and tokens' section of your Twitter Developper App.", +) +@click.option( + "--twitter-consumer-secret", + required=True, + help="API secret key, available in the 'Keys and tokens' section of your Twitter Developper App.", +) +@click.option( + "--twitter-access-token", + required=True, + help="Access token, available in the 'Keys and tokens' section of your Twitter Developper App.", +) +@click.option( + "--twitter-access-token-secret", + required=True, + help="Access token secret, available in the 'Keys and tokens' section of your Twitter Developper App.", +) +@click.option( + "--twitter-account-id", required=True, help="Specifies the Twitter Account ID for which the data should be returned.", +) +@click.option( + "--twitter-report-type", + required=True, + type=click.Choice(REPORT_TYPES), + help="Specifies the type of report to collect: " + "ANALYTICS (performance report, any kind of metrics), " + "REACH (performance report, focus on reach and frequency metrics), " + "ENTITY (entity configuration report)", +) +@click.option( + "--twitter-entity", + required=True, + type=click.Choice(list(ENTITY_ATTRIBUTES.keys())), + help="Specifies the entity type to retrieve data for.", +) +@click.option( + "--twitter-entity-attribute", + multiple=True, + help="Specific to 'ENTITY' reports. " "Specifies the entity attribute (a.k.a. dimension) that should be returned.", +) +@click.option( + "--twitter-granularity", + type=click.Choice(GRANULARITIES), + default="TOTAL", + help="Specific to 'ANALYTICS' reports. Specifies how granular the retrieved data should be.", +) +@click.option( + "--twitter-metric-group", + multiple=True, + type=click.Choice(METRIC_GROUPS), + help="Specific to 'ANALYTICS' reports. Specifies the list of metrics (as a group) that should be returned: " + "https://developer.twitter.com/en/docs/ads/analytics/overview/metrics-and-segmentation", +) +@click.option( + "--twitter-placement", + type=click.Choice(PLACEMENTS), + default="ALL_ON_TWITTER", + help="Specific to 'ANALYTICS' reports. Scopes the retrieved data to a particular placement.", +) +@click.option( + "--twitter-segmentation-type", + type=click.Choice(SEGMENTATION_TYPES), + help="Specific to 'ANALYTICS' reports. Specifies how the retrieved data should be segmented: " + "https://developer.twitter.com/en/docs/ads/analytics/overview/metrics-and-segmentation", +) +@click.option( + "--twitter-platform", + help="Specific to 'ANALYTICS' reports. Required if segmentation_type is set to 'DEVICES' or 'PLATFORM_VERSION'. " + "To get possible values: GET targeting_criteria/locations", +) +@click.option( + "--twitter-country", + help="Specific to 'ANALYTICS' reports. Required if segmentation_type is set to 'CITIES', 'POSTAL_CODES', or 'REGION'. " + "To get possible values: GET targeting_criteria/platforms", +) +@click.option("--twitter-start-date", type=click.DateTime(), help="Specifies report start date.") +@click.option( + "--twitter-end-date", type=click.DateTime(), help="Specifies report end date (inclusive).", +) +@click.option( + "--twitter-add-request-date-to-report", + type=click.BOOL, + default=False, + help="If set to 'True', the date on which the request is made will appear on each report record.", +) +@click.option( + "--twitter-date-range", + type=click.Choice(DEFAULT_DATE_RANGE_FUNCTIONS.keys()), + help=f"One of the available NCK default date ranges: {DEFAULT_DATE_RANGE_FUNCTIONS.keys()}", +) +@processor( + "twitter_consumer_key", "twitter_consumer_secret", "twitter_access_token", "twitter_access_token_secret", +) +def twitter(**kwargs): + return TwitterReader(**extract_args("twitter_", kwargs)) diff --git a/nck/helpers/twitter_helper.py b/nck/readers/twitter/config.py similarity index 71% rename from nck/helpers/twitter_helper.py rename to nck/readers/twitter/config.py index 042752a6..c2c868b7 100644 --- a/nck/helpers/twitter_helper.py +++ b/nck/readers/twitter/config.py @@ -1,3 +1,9 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public # License as published by the Free Software Foundation; either # version 3 of the License, or (at your option) any later version. # @@ -10,9 +16,14 @@ # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from twitter_ads.campaign import FundingInstrument, Campaign, LineItem -from twitter_ads.creative import MediaCreative, PromotedTweet, CardsFetch +from twitter_ads.campaign import Campaign, FundingInstrument, LineItem +from twitter_ads.creative import CardsFetch, MediaCreative, PromotedTweet +API_DATEFORMAT = "%Y-%m-%dT%H:%M:%SZ" +REP_DATEFORMAT = "%Y-%m-%d" +MAX_WAITING_SEC = 3600 +MAX_ENTITY_IDS_PER_JOB = 20 +MAX_CONCURRENT_JOBS = 100 REPORT_TYPES = ["ANALYTICS", "REACH", "ENTITY"] @@ -25,10 +36,7 @@ } ENTITY_ATTRIBUTES = { - **{ - entity: list(ENTITY_OBJECTS[entity].__dict__["PROPERTIES"].keys()) - for entity in ENTITY_OBJECTS - }, + **{entity: list(ENTITY_OBJECTS[entity].__dict__["PROPERTIES"].keys()) for entity in ENTITY_OBJECTS}, "CARD": list(CardsFetch.__dict__["PROPERTIES"].keys()), } diff --git a/nck/readers/twitter_reader.py b/nck/readers/twitter/reader.py similarity index 78% rename from nck/readers/twitter_reader.py rename to nck/readers/twitter/reader.py index db86e363..334e297d 100644 --- a/nck/readers/twitter_reader.py +++ b/nck/readers/twitter/reader.py @@ -1,3 +1,9 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public # License as published by the Free Software Foundation; either # version 3 of the License, or (at your option) any later version. # @@ -10,144 +16,32 @@ # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from nck.config import logger -import click -from click import ClickException -from itertools import chain from datetime import datetime, timedelta -from tenacity import retry, wait_exponential, stop_after_delay +from itertools import chain -from nck.utils.args import extract_args -from nck.commands.command import processor +from click import ClickException +from nck.config import logger from nck.readers.reader import Reader -from nck.streams.json_stream import JSONStream -from nck.helpers.twitter_helper import ( - REPORT_TYPES, - ENTITY_OBJECTS, +from nck.readers.twitter.config import ( + API_DATEFORMAT, + # MAX_WAITING_SEC, ENTITY_ATTRIBUTES, - GRANULARITIES, - METRIC_GROUPS, - PLACEMENTS, - SEGMENTATION_TYPES, + ENTITY_OBJECTS, + MAX_CONCURRENT_JOBS, + MAX_ENTITY_IDS_PER_JOB, + REP_DATEFORMAT, ) - -from twitter_ads.client import Client -from twitter_ads.utils import split_list +from nck.streams.json_stream import JSONStream +from nck.utils.date_handler import build_date_range +from tenacity import retry, stop_after_delay, wait_exponential from twitter_ads import API_VERSION -from twitter_ads.http import Request -from twitter_ads.cursor import Cursor +from twitter_ads.client import Client # from twitter_ads.creative import TweetPreview from twitter_ads.creative import CardsFetch - -from nck.utils.date_handler import DEFAULT_DATE_RANGE_FUNCTIONS, build_date_range - -API_DATEFORMAT = "%Y-%m-%dT%H:%M:%SZ" -REP_DATEFORMAT = "%Y-%m-%d" -MAX_WAITING_SEC = 3600 -MAX_ENTITY_IDS_PER_JOB = 20 -MAX_CONCURRENT_JOBS = 100 - - -@click.command(name="read_twitter") -@click.option( - "--twitter-consumer-key", - required=True, - help="API key, available in the 'Keys and tokens' section of your Twitter Developper App.", -) -@click.option( - "--twitter-consumer-secret", - required=True, - help="API secret key, available in the 'Keys and tokens' section of your Twitter Developper App.", -) -@click.option( - "--twitter-access-token", - required=True, - help="Access token, available in the 'Keys and tokens' section of your Twitter Developper App.", -) -@click.option( - "--twitter-access-token-secret", - required=True, - help="Access token secret, available in the 'Keys and tokens' section of your Twitter Developper App.", -) -@click.option( - "--twitter-account-id", required=True, help="Specifies the Twitter Account ID for which the data should be returned.", -) -@click.option( - "--twitter-report-type", - required=True, - type=click.Choice(REPORT_TYPES), - help="Specifies the type of report to collect: " - "ANALYTICS (performance report, any kind of metrics), " - "REACH (performance report, focus on reach and frequency metrics), " - "ENTITY (entity configuration report)", -) -@click.option( - "--twitter-entity", - required=True, - type=click.Choice(list(ENTITY_ATTRIBUTES.keys())), - help="Specifies the entity type to retrieve data for.", -) -@click.option( - "--twitter-entity-attribute", - multiple=True, - help="Specific to 'ENTITY' reports. " "Specifies the entity attribute (a.k.a. dimension) that should be returned.", -) -@click.option( - "--twitter-granularity", - type=click.Choice(GRANULARITIES), - default="TOTAL", - help="Specific to 'ANALYTICS' reports. Specifies how granular the retrieved data should be.", -) -@click.option( - "--twitter-metric-group", - multiple=True, - type=click.Choice(METRIC_GROUPS), - help="Specific to 'ANALYTICS' reports. Specifies the list of metrics (as a group) that should be returned: " - "https://developer.twitter.com/en/docs/ads/analytics/overview/metrics-and-segmentation", -) -@click.option( - "--twitter-placement", - type=click.Choice(PLACEMENTS), - default="ALL_ON_TWITTER", - help="Specific to 'ANALYTICS' reports. Scopes the retrieved data to a particular placement.", -) -@click.option( - "--twitter-segmentation-type", - type=click.Choice(SEGMENTATION_TYPES), - help="Specific to 'ANALYTICS' reports. Specifies how the retrieved data should be segmented: " - "https://developer.twitter.com/en/docs/ads/analytics/overview/metrics-and-segmentation", -) -@click.option( - "--twitter-platform", - help="Specific to 'ANALYTICS' reports. Required if segmentation_type is set to 'DEVICES' or 'PLATFORM_VERSION'. " - "To get possible values: GET targeting_criteria/locations", -) -@click.option( - "--twitter-country", - help="Specific to 'ANALYTICS' reports. Required if segmentation_type is set to 'CITIES', 'POSTAL_CODES', or 'REGION'. " - "To get possible values: GET targeting_criteria/platforms", -) -@click.option("--twitter-start-date", type=click.DateTime(), help="Specifies report start date.") -@click.option( - "--twitter-end-date", type=click.DateTime(), help="Specifies report end date (inclusive).", -) -@click.option( - "--twitter-add-request-date-to-report", - type=click.BOOL, - default=False, - help="If set to 'True', the date on which the request is made will appear on each report record.", -) -@click.option( - "--twitter-date-range", - type=click.Choice(DEFAULT_DATE_RANGE_FUNCTIONS.keys()), - help=f"One of the available NCK default date ranges: {DEFAULT_DATE_RANGE_FUNCTIONS.keys()}", -) -@processor( - "twitter_consumer_key", "twitter_consumer_secret", "twitter_access_token", "twitter_access_token_secret", -) -def twitter(**kwargs): - return TwitterReader(**extract_args("twitter_", kwargs)) +from twitter_ads.cursor import Cursor +from twitter_ads.http import Request +from twitter_ads.utils import split_list class TwitterReader(Reader): diff --git a/nck/readers/yandex_campaign/__init__.py b/nck/readers/yandex_campaign/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/yandex_campaign/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/yandex_campaign/cli.py b/nck/readers/yandex_campaign/cli.py new file mode 100644 index 00000000..45f7f32d --- /dev/null +++ b/nck/readers/yandex_campaign/cli.py @@ -0,0 +1,53 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.readers.yandex_campaign.config import CAMPAIGN_FIELDS, CAMPAIGN_PAYMENT_STATUSES, CAMPAIGN_STATES, CAMPAIGN_STATUSES +from nck.readers.yandex_campaign.reader import YandexCampaignReader +from nck.utils.args import extract_args +from nck.utils.processor import processor + + +@click.command(name="read_yandex_campaigns") +@click.option("--yandex-campaigns-token", "yandex_token", required=True) +@click.option("--yandex-campaigns-campaign-id", "yandex_campaign_ids", multiple=True) +@click.option("--yandex-campaigns-campaign-state", "yandex_campaign_states", multiple=True, type=click.Choice(CAMPAIGN_STATES)) +@click.option( + "--yandex-campaigns-campaign-status", "yandex_campaign_statuses", multiple=True, type=click.Choice(CAMPAIGN_STATUSES) +) +@click.option( + "--yandex-campaigns-campaign-payment-status", + "yandex_campaign_payment_statuses", + multiple=True, + type=click.Choice(CAMPAIGN_PAYMENT_STATUSES), +) +@click.option( + "--yandex-campaigns-field-name", + "yandex_fields", + multiple=True, + type=click.Choice(CAMPAIGN_FIELDS), + required=True, + help=( + "Fields to output in the report (columns)." + "For the full list of fields and their meanings, " + "see https://tech.yandex.com/direct/doc/reports/fields-list-docpage/" + ), +) +@processor("yandex_token") +def yandex_campaigns(**kwargs): + return YandexCampaignReader(**extract_args("yandex_", kwargs)) diff --git a/nck/readers/yandex_campaign/config.py b/nck/readers/yandex_campaign/config.py new file mode 100644 index 00000000..9a594d04 --- /dev/null +++ b/nck/readers/yandex_campaign/config.py @@ -0,0 +1,50 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +YANDEX_DIRECT_API_BASE_URL = "https://api.direct.yandex.com/json/v5/" + +CAMPAIGN_FIELDS = [ + "BlockedIps", + "ExcludedSites", + "Currency", + "DailyBudget", + "Notification", + "EndDate", + "Funds", + "ClientInfo", + "Id", + "Name", + "NegativeKeywords", + "RepresentedBy", + "StartDate", + "Statistics", + "State", + "Status", + "StatusPayment", + "StatusClarification", + "SourceId", + "TimeTargeting", + "TimeZone", + "Type", +] + +CAMPAIGN_STATES = ["ARCHIVED", "CONVERTED", "ENDED", "OFF", "ON", "SUSPENDED"] + +CAMPAIGN_STATUSES = ["ACCEPTED", "DRAFT", "MODERATION", "REJECTED"] + +CAMPAIGN_PAYMENT_STATUSES = ["ALLOWED", "DISALLOWED"] diff --git a/nck/readers/yandex_campaign_reader.py b/nck/readers/yandex_campaign/reader.py similarity index 54% rename from nck/readers/yandex_campaign_reader.py rename to nck/readers/yandex_campaign/reader.py index 105f5887..193cc5de 100644 --- a/nck/readers/yandex_campaign_reader.py +++ b/nck/readers/yandex_campaign/reader.py @@ -15,73 +15,16 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -import click -import nck.helpers.api_client_helper as api_client_helper -from nck.clients.api_client import ApiClient -from nck.commands.command import processor -from nck.helpers.yandex_helper import (CAMPAIGN_FIELDS, CAMPAIGN_STATES, - CAMPAIGN_STATUSES, CAMPAIGN_PAYMENT_STATUSES) +import nck.clients.api.helper as api_client_helper +from nck.clients.api.client import ApiClient from nck.readers.reader import Reader +from nck.readers.yandex_campaign.config import YANDEX_DIRECT_API_BASE_URL from nck.streams.json_stream import JSONStream -from nck.utils.args import extract_args - - -@click.command(name="read_yandex_campaigns") -@click.option("--yandex-campaigns-token", - "yandex_token", - required=True) -@click.option( - "--yandex-campaigns-campaign-id", - "yandex_campaign_ids", - multiple=True -) -@click.option( - "--yandex-campaigns-campaign-state", - "yandex_campaign_states", - multiple=True, - type=click.Choice(CAMPAIGN_STATES) -) -@click.option( - "--yandex-campaigns-campaign-status", - "yandex_campaign_statuses", - multiple=True, - type=click.Choice(CAMPAIGN_STATUSES) -) -@click.option( - "--yandex-campaigns-campaign-payment-status", - "yandex_campaign_payment_statuses", - multiple=True, - type=click.Choice(CAMPAIGN_PAYMENT_STATUSES) -) -@click.option( - "--yandex-campaigns-field-name", - "yandex_fields", - multiple=True, - type=click.Choice(CAMPAIGN_FIELDS), - required=True, - help=( - "Fields to output in the report (columns)." - "For the full list of fields and their meanings, " - "see https://tech.yandex.com/direct/doc/reports/fields-list-docpage/" - ) -) -@processor("yandex_token") -def yandex_campaigns(**kwargs): - return YandexCampaignReader(**extract_args("yandex_", kwargs)) - - -YANDEX_DIRECT_API_BASE_URL = "https://api.direct.yandex.com/json/v5/" class YandexCampaignReader(Reader): - - def __init__( - self, - token, - fields, - **kwargs - ): + def __init__(self, token, fields, **kwargs): self.token = token self.fields = list(fields) self.campaign_ids = list(kwargs["campaign_ids"]) @@ -108,13 +51,9 @@ def _build_request_body(self): if len(self.campaign_payment_statuses) != 0: selection_criteria["StatusesPayment"] = self.campaign_payment_statuses body["params"] = api_client_helper.get_dict_with_keys_converted_to_new_string_format( - field_names=self.fields, - selection_criteria=selection_criteria + field_names=self.fields, selection_criteria=selection_criteria ) return body def read(self): - yield JSONStream( - "results_CAMPAIGN_OBJECT_REPORT_", - self.result_generator() - ) + yield JSONStream("results_CAMPAIGN_OBJECT_REPORT_", self.result_generator()) diff --git a/nck/readers/yandex_statistics/__init__.py b/nck/readers/yandex_statistics/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/readers/yandex_statistics/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/readers/yandex_statistics/cli.py b/nck/readers/yandex_statistics/cli.py new file mode 100644 index 00000000..b75a0649 --- /dev/null +++ b/nck/readers/yandex_statistics/cli.py @@ -0,0 +1,77 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import datetime +import random + +import click +from nck.readers.yandex_statistics.config import DATE_RANGE_TYPES, LANGUAGES, OPERATORS, REPORT_TYPES, STATS_FIELDS +from nck.readers.yandex_statistics.reader import YandexStatisticsReader +from nck.utils.args import extract_args +from nck.utils.processor import processor + + +class StrList(click.ParamType): + def convert(self, value, param, ctx): + return value.split(",") + + +STR_LIST_TYPE = StrList() + + +@click.command(name="read_yandex_statistics") +@click.option("--yandex-statistics-token", "yandex_token", required=True) +@click.option("--yandex-statistics-report-language", "yandex_report_language", type=click.Choice(LANGUAGES), default="en") +@click.option( + "--yandex-statistics-filter", + "yandex_filters", + multiple=True, + type=click.Tuple([click.Choice(STATS_FIELDS), click.Choice(OPERATORS), STR_LIST_TYPE]), +) +@click.option("--yandex-statistics-max-rows", "yandex_max_rows", type=int) +@click.option( + "--yandex-statistics-field-name", + "yandex_fields", + multiple=True, + type=click.Choice(STATS_FIELDS), + required=True, + help=( + "Fields to output in the report (columns)." + "For the full list of fields and their meanings, " + "see https://tech.yandex.com/direct/doc/reports/fields-list-docpage/" + ), +) +@click.option( + "--yandex-statistics-report-name", + "yandex_report_name", + default=f"stats_report_{datetime.date.today()}_{random.randrange(10000)}", +) +@click.option("--yandex-statistics-report-type", "yandex_report_type", type=click.Choice(REPORT_TYPES), required=True) +@click.option("--yandex-statistics-date-range", "yandex_date_range", type=click.Choice(DATE_RANGE_TYPES), required=True) +@click.option( + "--yandex-statistics-include-vat", + "yandex_include_vat", + type=click.BOOL, + required=True, + help="Whether to include VAT in the monetary amounts in the report.", +) +@click.option("--yandex-statistics-date-start", "yandex_date_start", type=click.DateTime()) +@click.option("--yandex-statistics-date-stop", "yandex_date_stop", type=click.DateTime()) +@processor("yandex_token") +def yandex_statistics(**kwargs): + return YandexStatisticsReader(**extract_args("yandex_", kwargs)) diff --git a/nck/helpers/yandex_helper.py b/nck/readers/yandex_statistics/config.py similarity index 67% rename from nck/helpers/yandex_helper.py rename to nck/readers/yandex_statistics/config.py index 9990ecba..b2b64603 100644 --- a/nck/helpers/yandex_helper.py +++ b/nck/readers/yandex_statistics/config.py @@ -1,3 +1,23 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +YANDEX_DIRECT_API_BASE_URL = "https://api.direct.yandex.com/json/v5/" + LANGUAGES = ["en", "ru", "uk"] REPORT_TYPES = [ @@ -8,7 +28,7 @@ "CRITERIA_PERFORMANCE_REPORT", "CUSTOM_REPORT", "REACH_AND_FREQUENCY_PERFORMANCE_REPORT", - "SEARCH_QUERY_PERFORMANCE_REPORT" + "SEARCH_QUERY_PERFORMANCE_REPORT", ] STATS_FIELDS = [ @@ -75,32 +95,7 @@ "Week", "WeightedCtr", "WeightedImpressions", - "Year" -] - -CAMPAIGN_FIELDS = [ - "BlockedIps", - "ExcludedSites", - "Currency", - "DailyBudget", - "Notification", - "EndDate", - "Funds", - "ClientInfo", - "Id", - "Name", - "NegativeKeywords", - "RepresentedBy", - "StartDate", - "Statistics", - "State", - "Status", - "StatusPayment", - "StatusClarification", - "SourceId", - "TimeTargeting", - "TimeZone", - "Type" + "Year", ] DATE_RANGE_TYPES = [ @@ -122,7 +117,7 @@ "LAST_14_DAYS", "LAST_30_DAYS", "LAST_90_DAYS", - "LAST_365_DAYS" + "LAST_365_DAYS", ] OPERATORS = [ @@ -135,26 +130,5 @@ "STARTS_WITH_IGNORE_CASE", "DOES_NOT_START_WITH_IGNORE_CASE", "STARTS_WITH_ANY_IGNORE_CASE", - "DOES_NOT_START_WITH_ALL_IGNORE_CASE" -] - -CAMPAIGN_STATES = [ - "ARCHIVED", - "CONVERTED", - "ENDED", - "OFF", - "ON", - "SUSPENDED" -] - -CAMPAIGN_STATUSES = [ - "ACCEPTED", - "DRAFT", - "MODERATION", - "REJECTED" -] - -CAMPAIGN_PAYMENT_STATUSES = [ - "ALLOWED", - "DISALLOWED" + "DOES_NOT_START_WITH_ALL_IGNORE_CASE", ] diff --git a/nck/readers/yandex_statistics_reader.py b/nck/readers/yandex_statistics/reader.py similarity index 62% rename from nck/readers/yandex_statistics_reader.py rename to nck/readers/yandex_statistics/reader.py index 6fbdbc6f..f3e424b6 100644 --- a/nck/readers/yandex_statistics_reader.py +++ b/nck/readers/yandex_statistics/reader.py @@ -15,95 +15,31 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -import datetime -from nck.config import logger -import random + import time from http import HTTPStatus from typing import Dict, Tuple -import click -import nck.helpers.api_client_helper as api_client_helper -from nck.clients.api_client import ApiClient -from nck.commands.command import processor -from nck.helpers.yandex_helper import DATE_RANGE_TYPES, LANGUAGES, OPERATORS, REPORT_TYPES, STATS_FIELDS +import nck.clients.api.helper as api_client_helper +from click import ClickException +from nck.clients.api.client import ApiClient +from nck.config import logger from nck.readers.reader import Reader +from nck.readers.yandex_statistics.config import YANDEX_DIRECT_API_BASE_URL from nck.streams.json_stream import JSONStream -from nck.utils.args import extract_args from nck.utils.text import get_report_generator_from_flat_file -class StrList(click.ParamType): - def convert(self, value, param, ctx): - return value.split(",") - - -STR_LIST_TYPE = StrList() - - -@click.command(name="read_yandex_statistics") -@click.option("--yandex-statistics-token", - "yandex_token", - required=True) -@click.option("--yandex-statistics-report-language", - "yandex_report_language", - type=click.Choice(LANGUAGES), default="en") -@click.option( - "--yandex-statistics-filter", - "yandex_filters", - multiple=True, - type=click.Tuple([click.Choice(STATS_FIELDS), click.Choice(OPERATORS), STR_LIST_TYPE]), -) -@click.option("--yandex-statistics-max-rows", - "yandex_max_rows", - type=int) -@click.option( - "--yandex-statistics-field-name", - "yandex_fields", - multiple=True, - type=click.Choice(STATS_FIELDS), - required=True, - help=( - "Fields to output in the report (columns)." - "For the full list of fields and their meanings, " - "see https://tech.yandex.com/direct/doc/reports/fields-list-docpage/" - ), -) -@click.option( - "--yandex-statistics-report-name", - "yandex_report_name", - default=f"stats_report_{datetime.date.today()}_{random.randrange(10000)}", -) -@click.option("--yandex-statistics-report-type", - "yandex_report_type", - type=click.Choice(REPORT_TYPES), required=True) -@click.option("--yandex-statistics-date-range", - "yandex_date_range", - type=click.Choice(DATE_RANGE_TYPES), required=True) -@click.option( - "--yandex-statistics-include-vat", - "yandex_include_vat", - type=click.BOOL, - required=True, - help="Whether to include VAT in the monetary amounts in the report.", -) -@click.option("--yandex-statistics-date-start", - "yandex_date_start", - type=click.DateTime()) -@click.option("--yandex-statistics-date-stop", - "yandex_date_stop", - type=click.DateTime()) -@processor("yandex_token") -def yandex_statistics(**kwargs): - return YandexStatisticsReader(**extract_args("yandex_", kwargs)) - - -YANDEX_DIRECT_API_BASE_URL = "https://api.direct.yandex.com/json/v5/" - - class YandexStatisticsReader(Reader): def __init__( - self, token, fields: Tuple[str], report_type: str, report_name: str, date_range: str, include_vat: bool, **kwargs, + self, + token, + fields: Tuple[str], + report_type: str, + report_name: str, + date_range: str, + include_vat: bool, + **kwargs, ): self.token = token self.fields = list(fields) @@ -134,7 +70,11 @@ def result_generator(self): skip_n_first=1, ) - return get_report_generator_from_flat_file(response.iter_lines(), delimiter="\t", skip_n_first=1,) + return get_report_generator_from_flat_file( + response.iter_lines(), + delimiter="\t", + skip_n_first=1, + ) elif response.status_code == HTTPStatus.BAD_REQUEST: logger.error("Invalid request.") @@ -155,7 +95,9 @@ def _build_request_body(self) -> Dict: if len(self.kwargs["filters"]) > 0: selection_criteria["Filter"] = [ api_client_helper.get_dict_with_keys_converted_to_new_string_format( - field=filter_element[0], operator=filter_element[1], values=filter_element[2], + field=filter_element[0], + operator=filter_element[1], + values=filter_element[2], ) for filter_element in self.kwargs["filters"] ] @@ -188,18 +130,18 @@ def _add_custom_dates_if_set(self) -> Dict: elif ( self.kwargs["date_start"] is not None and self.kwargs["date_stop"] is not None and self.date_range != "CUSTOM_DATE" ): - raise click.ClickException("Wrong date range. If start and stop dates are set, should be CUSTOM_DATE.") + raise ClickException("Wrong date range. If start and stop dates are set, should be CUSTOM_DATE.") elif ( self.kwargs["date_start"] is not None or self.kwargs["date_stop"] is not None ) and self.date_range != "CUSTOM_DATE": - raise click.ClickException( + raise ClickException( ( "Wrong combination of date parameters. " "Only use date start and date stop with date range set to CUSTOM_DATE." ) ) elif (self.kwargs["date_start"] is None or self.kwargs["date_stop"] is None) and self.date_range == "CUSTOM_DATE": - raise click.ClickException("Missing at least one date. Have you set start and stop dates?") + raise ClickException("Missing at least one date. Have you set start and stop dates?") return selection_criteria def read(self): diff --git a/nck/utils/date_handler.py b/nck/utils/date_handler.py index a1513adb..7dcfab0e 100644 --- a/nck/utils/date_handler.py +++ b/nck/utils/date_handler.py @@ -1,3 +1,21 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + import calendar from datetime import date, timedelta, datetime from typing import Tuple diff --git a/nck/utils/exceptions.py b/nck/utils/exceptions.py index 76239073..55752c40 100644 --- a/nck/utils/exceptions.py +++ b/nck/utils/exceptions.py @@ -57,3 +57,33 @@ class MissingItemsInResponse(Exception): """Raised when the body of the response is missing items""" pass + + +class APIRateLimitError(Exception): + """Raised when the API rate limit is reached""" + + pass + + +class ReportDescriptionError(Exception): + """Raised when report description is not valid""" + + pass + + +class ReportNotReadyError(Exception): + """Raised when report is not ready yet""" + + pass + + +class ReportTemplateNotFoundError(Exception): + """Raised when The Trade Desk report template was not found""" + + pass + + +class ReportScheduleNotReadyError(Exception): + """Raised when The Trade Desk report schedule is not ready yet""" + + pass diff --git a/nck/utils/file_reader.py b/nck/utils/file_reader.py index eaf3e8d1..f333f148 100644 --- a/nck/utils/file_reader.py +++ b/nck/utils/file_reader.py @@ -15,11 +15,12 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -import csv + import codecs +import csv import gzip -import zipfile import json +import zipfile csv.field_size_limit(1000000) diff --git a/nck/commands/command.py b/nck/utils/processor.py similarity index 99% rename from nck/commands/command.py rename to nck/utils/processor.py index c4896b50..e44597ce 100644 --- a/nck/commands/command.py +++ b/nck/utils/processor.py @@ -15,7 +15,9 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + from functools import update_wrapper + from nck.config import logger diff --git a/nck/utils/redis.py b/nck/utils/redis.py index c5a388cd..80415e23 100644 --- a/nck/utils/redis.py +++ b/nck/utils/redis.py @@ -15,9 +15,11 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from nck.config import logger + import pickle +from nck.config import logger + import redis diff --git a/nck/utils/retry.py b/nck/utils/retry.py index f99111ec..79e2e874 100644 --- a/nck/utils/retry.py +++ b/nck/utils/retry.py @@ -15,15 +15,13 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from tenacity import ( - retry as _retry, - wait_exponential, - before_sleep_log, - before_log, - stop_after_attempt, -) + import logging + from nck.config import logger +from tenacity import before_log, before_sleep_log +from tenacity import retry as _retry +from tenacity import stop_after_attempt, wait_exponential def retry(fn): diff --git a/nck/utils/stdout_to_log.py b/nck/utils/stdout_to_log.py index ee83604c..b88a3435 100644 --- a/nck/utils/stdout_to_log.py +++ b/nck/utils/stdout_to_log.py @@ -1,5 +1,24 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + import logging import sys + import httplib2 diff --git a/nck/utils/text.py b/nck/utils/text.py index cb2f6c55..a1f50306 100644 --- a/nck/utils/text.py +++ b/nck/utils/text.py @@ -15,13 +15,15 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from nck.config import logger -import re + import csv -from io import StringIO +import re from collections import deque +from io import StringIO from itertools import islice +from nck.config import logger + def get_report_generator_from_flat_file( line_iterator, diff --git a/nck/writers/__init__.py b/nck/writers/__init__.py index 62e57a83..d437e0ca 100644 --- a/nck/writers/__init__.py +++ b/nck/writers/__init__.py @@ -15,15 +15,15 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from nck.writers.writer import Writer -from nck.writers.gcs_writer import gcs -from nck.writers.console_writer import console -from nck.writers.local_writer import local -from nck.writers.bigquery_writer import bq -from nck.writers.s3_writer import s3 +from nck.writers.writer import Writer +from nck.writers.amazon_s3.cli import amazon_s3 +from nck.writers.console.cli import console +from nck.writers.google_bigquery.cli import google_bigquery +from nck.writers.google_cloud_storage.cli import google_cloud_storage +from nck.writers.local.cli import local -writers = [gcs, console, local, bq, s3] +writers = [amazon_s3, console, google_bigquery, google_cloud_storage, local] __all__ = ["writers", "Writer"] diff --git a/nck/writers/amazon_s3/__init__.py b/nck/writers/amazon_s3/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/writers/amazon_s3/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/writers/amazon_s3/cli.py b/nck/writers/amazon_s3/cli.py new file mode 100644 index 00000000..73e5b837 --- /dev/null +++ b/nck/writers/amazon_s3/cli.py @@ -0,0 +1,34 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.utils.args import extract_args +from nck.utils.processor import processor +from nck.writers.amazon_s3.writer import AmazonS3Writer + + +@click.command(name="write_s3") +@click.option("--s3-bucket-name", help="S3 Bucket name", required=True) +@click.option("--s3-bucket-region", required=True) +@click.option("--s3-access-key-id", required=True) +@click.option("--s3-access-key-secret", required=True) +@click.option("--s3-prefix", help="s3 Prefix", default=None) +@click.option("--s3-filename", help="Override the default name of the file (don't add the extension)") +@processor("s3_access_key_id", "s3_access_key_secret") +def amazon_s3(**kwargs): + return AmazonS3Writer(**extract_args("s3_", kwargs)) diff --git a/nck/writers/s3_writer.py b/nck/writers/amazon_s3/writer.py similarity index 69% rename from nck/writers/s3_writer.py rename to nck/writers/amazon_s3/writer.py index 4bc69134..88ae877c 100644 --- a/nck/writers/s3_writer.py +++ b/nck/writers/amazon_s3/writer.py @@ -15,26 +15,12 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -import click -import boto3 -from nck.writers.objectstorage_writer import ObjectStorageWriter -from nck.commands.command import processor -from nck.utils.args import extract_args - -@click.command(name="write_s3") -@click.option("--s3-bucket-name", help="S3 Bucket name", required=True) -@click.option("--s3-bucket-region", required=True) -@click.option("--s3-access-key-id", required=True) -@click.option("--s3-access-key-secret", required=True) -@click.option("--s3-prefix", help="s3 Prefix", default=None) -@click.option("--s3-filename", help="Override the default name of the file (don't add the extension)") -@processor("s3_access_key_id", "s3_access_key_secret") -def s3(**kwargs): - return S3Writer(**extract_args("s3_", kwargs)) +import boto3 +from nck.writers.object_storage.writer import ObjectStorageWriter -class S3Writer(ObjectStorageWriter): +class AmazonS3Writer(ObjectStorageWriter): def __init__(self, bucket_name, bucket_region, access_key_id, access_key_secret, prefix=None, filename=None, **kwargs): self.boto_config = { "region_name": bucket_region, diff --git a/nck/writers/console/__init__.py b/nck/writers/console/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/writers/console/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/writers/console/cli.py b/nck/writers/console/cli.py new file mode 100644 index 00000000..38fd3bf4 --- /dev/null +++ b/nck/writers/console/cli.py @@ -0,0 +1,28 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.utils.args import extract_args +from nck.utils.processor import processor +from nck.writers.console.writer import ConsoleWriter + + +@click.command(name="write_console") +@processor() +def console(**kwargs): + return ConsoleWriter(**extract_args("console_", kwargs)) diff --git a/nck/writers/console_writer.py b/nck/writers/console/writer.py similarity index 80% rename from nck/writers/console_writer.py rename to nck/writers/console/writer.py index cede0720..9a240a91 100644 --- a/nck/writers/console_writer.py +++ b/nck/writers/console/writer.py @@ -15,18 +15,10 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -import click + import sys from nck.writers.writer import Writer -from nck.commands.command import processor -from nck.utils.args import extract_args - - -@click.command(name="write_console") -@processor() -def console(**kwargs): - return ConsoleWriter(**extract_args("console_", kwargs)) class ConsoleWriter(Writer): @@ -35,7 +27,7 @@ def __init__(self): def write(self, stream): """ - Write file to console, mainly used for debugging + Write file to console, mainly used for debugging """ # this is how to read from a file as stream file = stream.as_file() diff --git a/nck/writers/google_bigquery/__init__.py b/nck/writers/google_bigquery/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/writers/google_bigquery/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/writers/google_bigquery/cli.py b/nck/writers/google_bigquery/cli.py new file mode 100644 index 00000000..cfb7f65e --- /dev/null +++ b/nck/writers/google_bigquery/cli.py @@ -0,0 +1,39 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.utils.args import extract_args +from nck.utils.processor import processor +from nck.writers.google_bigquery.writer import GoogleBigQueryWriter + + +@click.command(name="write_bq") +@click.option("--bq-dataset", required=True) +@click.option("--bq-table", required=True) +@click.option("--bq-bucket", required=True) +@click.option("--bq-partition-column") +@click.option( + "--bq-write-disposition", + default="truncate", + type=click.Choice(["truncate", "append"]), +) +@click.option("--bq-location", default="EU", type=click.Choice(["EU", "US"])) +@click.option("--bq-keep-files", is_flag=True, default=False) +@processor() +def google_bigquery(**kwargs): + return GoogleBigQueryWriter(**extract_args("bq_", kwargs)) diff --git a/nck/writers/bigquery_writer.py b/nck/writers/google_bigquery/writer.py similarity index 77% rename from nck/writers/bigquery_writer.py rename to nck/writers/google_bigquery/writer.py index 2fbf51c9..c4d1c648 100644 --- a/nck/writers/bigquery_writer.py +++ b/nck/writers/google_bigquery/writer.py @@ -15,39 +15,29 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -import click + from google.cloud import bigquery from nck import config -from nck.commands.command import processor from nck.config import logger -from nck.helpers.google_base import GoogleBaseClass +from nck.clients.google.client import GoogleClient from nck.streams.normalized_json_stream import NormalizedJSONStream -from nck.utils.args import extract_args from nck.utils.retry import retry -from nck.writers.gcs_writer import GCSWriter +from nck.writers.google_cloud_storage.writer import GoogleCloudStorageWriter from nck.writers.writer import Writer -@click.command(name="write_bq") -@click.option("--bq-dataset", required=True) -@click.option("--bq-table", required=True) -@click.option("--bq-bucket", required=True) -@click.option("--bq-partition-column") -@click.option( - "--bq-write-disposition", default="truncate", type=click.Choice(["truncate", "append"]), -) -@click.option("--bq-location", default="EU", type=click.Choice(["EU", "US"])) -@click.option("--bq-keep-files", is_flag=True, default=False) -@processor() -def bq(**kwargs): - return BigQueryWriter(**extract_args("bq_", kwargs)) - - -class BigQueryWriter(Writer, GoogleBaseClass): +class GoogleBigQueryWriter(Writer, GoogleClient): _client = None def __init__( - self, dataset, table, bucket, partition_column, write_disposition, location, keep_files, + self, + dataset, + table, + bucket, + partition_column, + write_disposition, + location, + keep_files, ): self._project_id = config.PROJECT_ID @@ -65,7 +55,7 @@ def write(self, stream): normalized_stream = NormalizedJSONStream.create_from_stream(stream) - gcs_writer = GCSWriter(self._bucket, self._project_id) + gcs_writer = GoogleCloudStorageWriter(self._bucket, self._project_id) gcs_uri, blob = gcs_writer.write(normalized_stream) table_ref = self._get_table_ref() diff --git a/nck/writers/google_cloud_storage/__init__.py b/nck/writers/google_cloud_storage/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/writers/google_cloud_storage/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/writers/google_cloud_storage/cli.py b/nck/writers/google_cloud_storage/cli.py new file mode 100644 index 00000000..b4ec8191 --- /dev/null +++ b/nck/writers/google_cloud_storage/cli.py @@ -0,0 +1,34 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.utils.args import extract_args +from nck.utils.processor import processor +from nck.writers.google_cloud_storage.writer import GoogleCloudStorageWriter + + +@click.command(name="write_gcs") +@click.option("--gcs-bucket", help="GCS Bucket", required=True) +@click.option("--gcs-prefix", help="GCS path to write the file.") +@click.option("--gcs-project-id", help="GCS Project Id") +@click.option( + "--gcs-filename", help="Override the default name of the file (don't add the extension)", +) +@processor() +def google_cloud_storage(**kwargs): + return GoogleCloudStorageWriter(**extract_args("gcs_", kwargs)) diff --git a/nck/writers/gcs_writer.py b/nck/writers/google_cloud_storage/writer.py similarity index 73% rename from nck/writers/gcs_writer.py rename to nck/writers/google_cloud_storage/writer.py index 00d3b82d..a2903974 100644 --- a/nck/writers/gcs_writer.py +++ b/nck/writers/google_cloud_storage/writer.py @@ -19,26 +19,11 @@ import click from google.cloud import storage from nck import config -from nck.commands.command import processor -from nck.helpers.google_base import GoogleBaseClass -from nck.utils.args import extract_args -from nck.writers.objectstorage_writer import ObjectStorageWriter +from nck.clients.google.client import GoogleClient +from nck.writers.object_storage.writer import ObjectStorageWriter -@click.command(name="write_gcs") -@click.option("--gcs-bucket", help="GCS Bucket", required=True) -@click.option("--gcs-prefix", help="GCS path to write the file.") -@click.option("--gcs-project-id", help="GCS Project Id") -@click.option( - "--gcs-filename", - help="Override the default name of the file (don't add the extension)", -) -@processor() -def gcs(**kwargs): - return GCSWriter(**extract_args("gcs_", kwargs)) - - -class GCSWriter(ObjectStorageWriter, GoogleBaseClass): +class GoogleCloudStorageWriter(ObjectStorageWriter, GoogleClient): def __init__(self, bucket, project_id, prefix=None, filename=None, **kwargs): self._project_id = self.get_project_id(project_id) super().__init__(bucket, prefix, filename, platform="GCS", **kwargs) @@ -66,7 +51,6 @@ def get_project_id(project_id): return config.PROJECT_ID except Exception: raise click.exceptions.MissingParameter( - "Please provide a project id in ENV var or params.", - param_type="--gcs-project-id", + "Please provide a project id in ENV var or params.", param_type="--gcs-project-id", ) return project_id diff --git a/nck/writers/local/__init__.py b/nck/writers/local/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/writers/local/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/writers/local/cli.py b/nck/writers/local/cli.py new file mode 100644 index 00000000..2dfd5f2d --- /dev/null +++ b/nck/writers/local/cli.py @@ -0,0 +1,30 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import click +from nck.utils.args import extract_args +from nck.utils.processor import processor +from nck.writers.local.writer import LocalWriter + + +@click.command(name="write_local") +@click.option("--local-directory", "-d", required=True, help="Destination directory") +@click.option("--local-file-name", "-n", help="Destination file name") +@processor() +def local(**kwargs): + return LocalWriter(**extract_args("local_", kwargs)) diff --git a/nck/writers/local_writer.py b/nck/writers/local/writer.py similarity index 72% rename from nck/writers/local_writer.py rename to nck/writers/local/writer.py index 5765d69f..6ec1c2d5 100644 --- a/nck/writers/local_writer.py +++ b/nck/writers/local/writer.py @@ -15,33 +15,24 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -import click -from nck.config import logger + import os +from nck.config import logger from nck.writers.writer import Writer -from nck.commands.command import processor - - -@click.command(name="write_local") -@click.option("--local-directory", "-d", required=True, help="Destination directory") -@click.option("--file-name", "-n", help="Destination file name") -@processor() -def local(**kwargs): - return LocalWriter(**kwargs) class LocalWriter(Writer): - def __init__(self, local_directory, file_name): - self._local_directory = local_directory + def __init__(self, directory, file_name): + self._directory = directory self._file_name = file_name def write(self, stream): """ - Write file to disk at location given as parameter. + Write file to disk at location given as parameter. """ file_name = self._file_name or stream.name - path = os.path.join(self._local_directory, file_name) + path = os.path.join(self._directory, file_name) logger.info(f"Writing stream {file_name} to {path}") file = stream.as_file() diff --git a/nck/writers/object_storage/__init__.py b/nck/writers/object_storage/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/nck/writers/object_storage/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/nck/writers/objectstorage_writer.py b/nck/writers/object_storage/writer.py similarity index 100% rename from nck/writers/objectstorage_writer.py rename to nck/writers/object_storage/writer.py diff --git a/tests/clients/__init__.py b/tests/clients/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/clients/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/clients/api/__init__.py b/tests/clients/api/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/clients/api/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/helpers/test_api_client_helper.py b/tests/clients/api/test_helper.py similarity index 94% rename from tests/helpers/test_api_client_helper.py rename to tests/clients/api/test_helper.py index bc325b8a..ffb03467 100644 --- a/tests/helpers/test_api_client_helper.py +++ b/tests/clients/api/test_helper.py @@ -17,10 +17,9 @@ # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. import unittest +from nck.clients.api.helper import get_dict_with_keys_converted_to_new_string_format, to_pascal_key from parameterized import parameterized -from nck.helpers.api_client_helper import get_dict_with_keys_converted_to_new_string_format, to_pascal_key - class ApiClientHelperTest(unittest.TestCase): def test_string_conversion_to_camel_case(self): diff --git a/tests/clients/google/__init__.py b/tests/clients/google/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/clients/google/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/clients/google/test_client.py b/tests/clients/google/test_client.py new file mode 100644 index 00000000..62dbfd49 --- /dev/null +++ b/tests/clients/google/test_client.py @@ -0,0 +1,72 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +import json +import os +import unittest +from unittest import mock + +import nck.clients.google.client + +MODULE_NAME = "nck.clients.google.client" + + +class GoogleClientTest(unittest.TestCase): + def setUp(self): + self.instance = nck.clients.google.client.GoogleClient() + + @mock.patch(MODULE_NAME + ".google.auth.default", return_value=("CREDENTIALS", "PROJECT_ID")) + def test_get_credentials_and_project_id_with_default_auth(self, mock_auth_default): + result = self.instance._get_credentials_and_project_id() + mock_auth_default.assert_called_once_with(scopes=self.instance.scopes) + self.assertEqual(("CREDENTIALS", "PROJECT_ID"), result) + + @mock.patch( + MODULE_NAME + ".google.oauth2.service_account.Credentials" ".from_service_account_file", + **{"return_value.project_id": "PROJECT_ID"} + ) + @mock.patch.dict(os.environ, {"GCP_KEY_PATH": "KEY_PATH.json"}) + def test_get_credentials_and_project_id_with_service_account_file(self, mock_from_service_account_file): + result = self.instance._get_credentials_and_project_id() + mock_from_service_account_file.assert_called_once_with("KEY_PATH.json", scopes=self.instance.scopes) + self.assertEqual((mock_from_service_account_file.return_value, "PROJECT_ID"), result) + + @mock.patch(MODULE_NAME + ".google.oauth2.service_account.Credentials" ".from_service_account_file") + @mock.patch.dict(os.environ, {"GCP_KEY_PATH": "KEY_PATH.p12"}) + def test_get_credentials_and_project_id_with_service_account_file_and_p12_key(self, mock_from_service_account_file): + with self.assertRaises(Exception): + self.instance._get_credentials_and_project_id() + + @mock.patch(MODULE_NAME + ".google.oauth2.service_account.Credentials" ".from_service_account_file") + @mock.patch.dict(os.environ, {"GCP_KEY_PATH": "KEY_PATH.unknown"}) + def test_get_credentials_and_project_id_with_service_account_file_and_unknown_key(self, mock_from_service_account_file): + with self.assertRaises(Exception): + self.instance._get_credentials_and_project_id() + + @mock.patch( + MODULE_NAME + ".google.oauth2.service_account.Credentials" ".from_service_account_info", + **{"return_value.project_id": "PROJECT_ID"} + ) + @mock.patch.dict(os.environ, {"GCP_KEY_JSON": json.dumps({"private_key": "PRIVATE_KEY"})}) + def test_get_credentials_and_project_id_with_service_account_info(self, mock_from_service_account_file): + result = self.instance._get_credentials_and_project_id() + mock_from_service_account_file.assert_called_once_with({"private_key": "PRIVATE_KEY"}, scopes=self.instance.scopes) + self.assertEqual((mock_from_service_account_file.return_value, "PROJECT_ID"), result) + + def test_default_scopes(self): + self.assertEqual(self.instance.scopes, ("https://www.googleapis.com/auth/cloud-platform",)) diff --git a/tests/clients/google_dcm/__init__.py b/tests/clients/google_dcm/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/clients/google_dcm/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/clients/test_dcm_client.py b/tests/clients/google_dcm/test_client.py similarity index 82% rename from tests/clients/test_dcm_client.py rename to tests/clients/google_dcm/test_client.py index 0b3629ad..2b2f00c6 100644 --- a/tests/clients/test_dcm_client.py +++ b/tests/clients/google_dcm/test_client.py @@ -15,10 +15,11 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from unittest import TestCase, mock + from datetime import datetime +from unittest import TestCase, mock -from nck.clients.dcm_client import DCMClient +from nck.clients.google_dcm.client import GoogleDCMClient class MockService: @@ -36,20 +37,20 @@ def mock_service(*args, **kwargs): return MockService() -class DCMClientTest(TestCase): +class GoogleDCMClientTest(TestCase): def mock_dcm_client(self, **kwargs): for param, value in kwargs.items(): setattr(self, param, value) kwargs = {"_service": mock_service()} - @mock.patch.object(DCMClient, "__init__", mock_dcm_client) + @mock.patch.object(GoogleDCMClient, "__init__", mock_dcm_client) def test_add_report_criteria(self): report = {"name": "report"} start = datetime(year=2020, month=1, day=1) end = datetime(year=2020, month=2, day=1) elements = ["a", "b"] - DCMClient(**self.kwargs).add_report_criteria(report, start, end, elements, elements) + GoogleDCMClient(**self.kwargs).add_report_criteria(report, start, end, elements, elements) expected = { "name": "report", "criteria": { @@ -60,13 +61,13 @@ def test_add_report_criteria(self): } assert report == expected - @mock.patch.object(DCMClient, "__init__", mock_dcm_client) + @mock.patch.object(GoogleDCMClient, "__init__", mock_dcm_client) @mock.patch.object(MockService, "execute", lambda *args: {"items": [{"value": "ok"}, {"value": "nok"}]}) - @mock.patch("tests.clients.test_dcm_client.MockService") + @mock.patch("tests.clients.google_dcm.test_client.MockService") def test_add_dimension_filters(self, mock_filter): report = {"criteria": {"dateRange": {"endDate": "", "startDate": ""}}} profile_id = "" filters = [("filter", "ok")] - DCMClient(**self.kwargs).add_dimension_filters(report, profile_id, filters) + GoogleDCMClient(**self.kwargs).add_dimension_filters(report, profile_id, filters) expected = {"criteria": {"dateRange": {"endDate": "", "startDate": ""}, "dimensionFilters": [{"value": "ok"}]}} assert report == expected diff --git a/tests/clients/google_sa360/__init__.py b/tests/clients/google_sa360/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/clients/google_sa360/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/clients/test_sa360_client.py b/tests/clients/google_sa360/test_client.py similarity index 85% rename from tests/clients/test_sa360_client.py rename to tests/clients/google_sa360/test_client.py index 87f606f3..2b96b66a 100644 --- a/tests/clients/test_sa360_client.py +++ b/tests/clients/google_sa360/test_client.py @@ -16,12 +16,12 @@ # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. from unittest import TestCase -from nck.clients.sa360_client import SA360Client +from nck.clients.google_sa360.client import GoogleSA360Client -class SA360ClientTest(TestCase): +class GoogleSA360ClientTest(TestCase): def test_generate_all_columns(self): standard = ["clicks", "impressions"] saved = ["savedColumn"] expected = [{"columnName": "clicks"}, {"columnName": "impressions"}, {"savedColumnName": "savedColumn"}] - self.assertEqual(SA360Client.generate_columns(standard, saved), expected) + self.assertEqual(GoogleSA360Client.generate_columns(standard, saved), expected) diff --git a/tests/clients/test_api_client.py b/tests/clients/test_api_client.py deleted file mode 100644 index c737a88c..00000000 --- a/tests/clients/test_api_client.py +++ /dev/null @@ -1,64 +0,0 @@ -# GNU Lesser General Public License v3.0 only -# Copyright (C) 2020 Artefact -# licence-information@artefact.com -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 3 of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public License -# along with this program; if not, write to the Free Software Foundation, -# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from unittest import TestCase - -from nck.clients.api_client import ApiClient - - -class ApiClientTest(TestCase): - - def test_get_formatted_request_body(self): - selection_criteria = { - "Filter": [ - { - "Field": "CampaignId", - "Operator": "IN", - "Values": ["123", "456"] - } - ] - } - page = { - "Limit": 10 - } - field_names = ["AdGroupId", "Year", "CampaignName"] - report_name = "test" - report_type = "CAMPAIGN_PERFORMANCE_REPORT" - date_range_type = "ALL_TIME" - include_vat = "NO" - - expected_output = { - "SelectionCriteria": selection_criteria, - "Page": page, - "FieldNames": field_names, - "ReportName": report_name, - "ReportType": report_type, - "DateRangeType": date_range_type, - "IncludeVAT": include_vat - } - self.assertDictEqual( - ApiClient.get_formatted_request_body( - selection_criteria=selection_criteria, - page=page, - field_names=field_names, - report_name=report_name, - report_type=report_type, - date_range_type=date_range_type, - include_v_a_t=include_vat - ), - expected_output - ) diff --git a/tests/helpers/google_base_tests.py b/tests/helpers/google_base_tests.py deleted file mode 100644 index fee08e05..00000000 --- a/tests/helpers/google_base_tests.py +++ /dev/null @@ -1,94 +0,0 @@ -# GNU Lesser General Public License v3.0 only -# Copyright (C) 2020 Artefact -# licence-information@artefact.com -# -# This program is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 3 of the License, or (at your option) any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public License -# along with this program; if not, write to the Free Software Foundation, -# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -import unittest -import nck.helpers.google_base -from unittest import mock -import json -import os - -MODULE_NAME = 'nck.helpers.google_base' - - -class TestGoogleCloudBaseClass(unittest.TestCase): - def setUp(self): - self.instance = nck.helpers.google_base.GoogleBaseClass() - - @mock.patch(MODULE_NAME + '.google.auth.default', - return_value=("CREDENTIALS", "PROJECT_ID")) - def test_get_credentials_and_project_id_with_default_auth(self, mock_auth_default): - result = self.instance._get_credentials_and_project_id() - mock_auth_default.assert_called_once_with(scopes=self.instance.scopes) - self.assertEqual(('CREDENTIALS', 'PROJECT_ID'), result) - - @mock.patch( - MODULE_NAME + '.google.oauth2.service_account.Credentials' - '.from_service_account_file', - **{'return_value.project_id': "PROJECT_ID"} - ) - @mock.patch.dict(os.environ, {'GCP_KEY_PATH': 'KEY_PATH.json'}) - def test_get_credentials_and_project_id_with_service_account_file(self, - mock_from_service_account_file): - result = self.instance._get_credentials_and_project_id() - mock_from_service_account_file.assert_called_once_with('KEY_PATH.json', - scopes=self.instance.scopes) - self.assertEqual((mock_from_service_account_file.return_value, 'PROJECT_ID'), - result) - - @mock.patch( - MODULE_NAME + '.google.oauth2.service_account.Credentials' - '.from_service_account_file') - @mock.patch.dict(os.environ, {'GCP_KEY_PATH': 'KEY_PATH.p12'}) - def test_get_credentials_and_project_id_with_service_account_file_and_p12_key( - self, - mock_from_service_account_file - ): - with self.assertRaises(Exception): - self.instance._get_credentials_and_project_id() - - @mock.patch( - MODULE_NAME + '.google.oauth2.service_account.Credentials' - '.from_service_account_file') - @mock.patch.dict(os.environ, {'GCP_KEY_PATH': 'KEY_PATH.unknown'}) - def test_get_credentials_and_project_id_with_service_account_file_and_unknown_key( - self, - mock_from_service_account_file - ): - with self.assertRaises(Exception): - self.instance._get_credentials_and_project_id() - - @mock.patch( - MODULE_NAME + '.google.oauth2.service_account.Credentials' - '.from_service_account_info', - **{'return_value.project_id': "PROJECT_ID"} - ) - @mock.patch.dict(os.environ, {'GCP_KEY_JSON': json.dumps({ - 'private_key': "PRIVATE_KEY" - })}) - def test_get_credentials_and_project_id_with_service_account_info(self, - mock_from_service_account_file): - result = self.instance._get_credentials_and_project_id() - mock_from_service_account_file.assert_called_once_with({ - 'private_key': "PRIVATE_KEY" - }, - scopes=self.instance.scopes) - self.assertEqual((mock_from_service_account_file.return_value, 'PROJECT_ID'), - result) - - def test_default_scopes(self): - self.assertEqual(self.instance.scopes, - ('https://www.googleapis.com/auth/cloud-platform',)) diff --git a/tests/readers/adobe_analytics_1_4/__init__.py b/tests/readers/adobe_analytics_1_4/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/readers/adobe_analytics_1_4/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/readers/test_adobe_reader.py b/tests/readers/adobe_analytics_1_4/test_reader.py similarity index 73% rename from tests/readers/test_adobe_reader.py rename to tests/readers/adobe_analytics_1_4/test_reader.py index 66480100..9bc558b0 100644 --- a/tests/readers/test_adobe_reader.py +++ b/tests/readers/adobe_analytics_1_4/test_reader.py @@ -15,12 +15,14 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + import datetime -from nck.readers.adobe_reader import AdobeReader from unittest import TestCase, mock +from nck.readers.adobe_analytics_1_4.reader import AdobeAnalytics14Reader + -class AdobeReaderTest(TestCase): +class AdobeAnalytics14ReaderTest(TestCase): DATEFORMAT = "%Y-%m-%d" @@ -52,16 +54,11 @@ class AdobeReaderTest(TestCase): "end_date": datetime.datetime(2020, 1, 3), } - @mock.patch("nck.clients.adobe_client.AdobeClient.__init__", return_value=None) - @mock.patch( - "nck.readers.adobe_reader.AdobeReader.query_report", - return_value={"reportID": "XXXXX"}, - ) + @mock.patch("nck.clients.adobe_analytics.client.AdobeAnalyticsClient.__init__", return_value=None) @mock.patch( - "nck.readers.adobe_reader.AdobeReader.download_report", return_value=None + "nck.readers.adobe_analytics_1_4.reader.AdobeAnalytics14Reader.query_report", return_value={"reportID": "XXXXX"}, ) - def test_read_empty_data( - self, mock_adobe_client, mock_query_report, mock_download_report - ): - reader = AdobeReader(**self.kwargs) + @mock.patch("nck.readers.adobe_analytics_1_4.reader.AdobeAnalytics14Reader.download_report", return_value=None) + def test_read_empty_data(self, mock_adobe_client, mock_query_report, mock_download_report): + reader = AdobeAnalytics14Reader(**self.kwargs) self.assertFalse(len(list(reader.read())) > 1) diff --git a/tests/readers/adobe_analytics_2_0/__init__.py b/tests/readers/adobe_analytics_2_0/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/readers/adobe_analytics_2_0/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/readers/test_adobe_reader_2_0.py b/tests/readers/adobe_analytics_2_0/test_reader.py similarity index 70% rename from tests/readers/test_adobe_reader_2_0.py rename to tests/readers/adobe_analytics_2_0/test_reader.py index 55f68bd6..817b4a1f 100644 --- a/tests/readers/test_adobe_reader_2_0.py +++ b/tests/readers/adobe_analytics_2_0/test_reader.py @@ -16,13 +16,13 @@ # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from nck.readers.adobe_reader_2_0 import AdobeReader_2_0 +import datetime from unittest import TestCase, mock -import datetime +from nck.readers.adobe_analytics_2_0.reader import AdobeAnalytics20Reader -class AdobeReaderTest_2_0(TestCase): +class AdobeAnalytics20ReaderTest(TestCase): kwargs = { "client_id": "", @@ -39,95 +39,59 @@ class AdobeReaderTest_2_0(TestCase): "date_range": None, } - @mock.patch("nck.clients.adobe_client.AdobeClient.__init__", return_value=None) + @mock.patch("nck.clients.adobe_analytics.client.AdobeAnalyticsClient.__init__", return_value=None) def test_format_date_range(self, mock_adobe_client): - output = AdobeReader_2_0(**self.kwargs).format_date_range() + output = AdobeAnalytics20Reader(**self.kwargs).format_date_range() expected = "2020-01-01T00:00:00/2020-01-03T00:00:00" self.assertEqual(output, expected) - @mock.patch("nck.clients.adobe_client.AdobeClient.__init__", return_value=None) + @mock.patch("nck.clients.adobe_analytics.client.AdobeAnalyticsClient.__init__", return_value=None) def test_build_report_description_one_dimension(self, mock_adobe_client): temp_kwargs = self.kwargs.copy() temp_kwargs.update({"dimension": ["daterangeday"]}) metrics = ["visits", "bounces"] - output = AdobeReader_2_0(**temp_kwargs).build_report_description(metrics) + output = AdobeAnalytics20Reader(**temp_kwargs).build_report_description(metrics) expected = { "rsid": "XXXXXXXXX", - "globalFilters": [ - { - "type": "dateRange", - "dateRange": "2020-01-01T00:00:00/2020-01-03T00:00:00", - } - ], + "globalFilters": [{"type": "dateRange", "dateRange": "2020-01-01T00:00:00/2020-01-03T00:00:00"}], "metricContainer": { "metricFilters": [], - "metrics": [ - {"id": "metrics/visits", "filters": []}, - {"id": "metrics/bounces", "filters": []}, - ], + "metrics": [{"id": "metrics/visits", "filters": []}, {"id": "metrics/bounces", "filters": []}], }, "dimension": "variables/daterangeday", "settings": {"countRepeatInstances": "true", "limit": "5000"}, } self.assertEqual(output, expected) - @mock.patch("nck.clients.adobe_client.AdobeClient.__init__", return_value=None) + @mock.patch("nck.clients.adobe_analytics.client.AdobeAnalyticsClient.__init__", return_value=None) def test_build_report_description_multiple_dimensions(self, mock_adobe_client): temp_kwargs = self.kwargs.copy() temp_kwargs.update({"dimension": ["daterangeday", "campaign", "pagename"]}) metrics = ["visits", "bounces"] breakdown_item_ids = ["000000000", "111111111"] - output = AdobeReader_2_0(**temp_kwargs).build_report_description(metrics, breakdown_item_ids) + output = AdobeAnalytics20Reader(**temp_kwargs).build_report_description(metrics, breakdown_item_ids) expected = { "rsid": "XXXXXXXXX", - "globalFilters": [ - { - "type": "dateRange", - "dateRange": "2020-01-01T00:00:00/2020-01-03T00:00:00", - } - ], + "globalFilters": [{"type": "dateRange", "dateRange": "2020-01-01T00:00:00/2020-01-03T00:00:00"}], "metricContainer": { "metricFilters": [ - { - "id": 0, - "type": "breakdown", - "dimension": "variables/daterangeday", - "itemId": "000000000", - }, - { - "id": 1, - "type": "breakdown", - "dimension": "variables/campaign", - "itemId": "111111111", - }, - { - "id": 2, - "type": "breakdown", - "dimension": "variables/daterangeday", - "itemId": "000000000", - }, - { - "id": 3, - "type": "breakdown", - "dimension": "variables/campaign", - "itemId": "111111111", - }, - ], - "metrics": [ - {"id": "metrics/visits", "filters": [0, 1]}, - {"id": "metrics/bounces", "filters": [2, 3]}, + {"id": 0, "type": "breakdown", "dimension": "variables/daterangeday", "itemId": "000000000"}, + {"id": 1, "type": "breakdown", "dimension": "variables/campaign", "itemId": "111111111"}, + {"id": 2, "type": "breakdown", "dimension": "variables/daterangeday", "itemId": "000000000"}, + {"id": 3, "type": "breakdown", "dimension": "variables/campaign", "itemId": "111111111"}, ], + "metrics": [{"id": "metrics/visits", "filters": [0, 1]}, {"id": "metrics/bounces", "filters": [2, 3]}], }, "dimension": "variables/pagename", "settings": {"countRepeatInstances": "true", "limit": "5000"}, } self.assertEqual(output, expected) - @mock.patch("nck.clients.adobe_client.AdobeClient.__init__", return_value=None) + @mock.patch("nck.clients.adobe_analytics.client.AdobeAnalyticsClient.__init__", return_value=None) @mock.patch( - "nck.readers.adobe_reader_2_0.AdobeReader_2_0.get_report_page", + "nck.readers.adobe_analytics_2_0.reader.AdobeAnalytics20Reader.get_report_page", side_effect=[ { "totalPages": 2, @@ -154,15 +118,11 @@ def test_build_report_description_multiple_dimensions(self, mock_adobe_client): def test_get_parsed_report(self, mock_adobe_client, mock_get_report_page): temp_kwargs = self.kwargs.copy() temp_kwargs.update( - { - "dimension": ["daterangeday"], - "start_date": datetime.date(2020, 1, 1), - "end_date": datetime.date(2020, 1, 4), - } + {"dimension": ["daterangeday"], "start_date": datetime.date(2020, 1, 1), "end_date": datetime.date(2020, 1, 4)} ) metrics = ["visits", "bounces"] - output = AdobeReader_2_0(**temp_kwargs).get_parsed_report({"dimension": "variables/daterangeday"}, metrics) + output = AdobeAnalytics20Reader(**temp_kwargs).get_parsed_report({"dimension": "variables/daterangeday"}, metrics) expected = [ {"daterangeday": "2020-01-01", "visits": 11, "bounces": 21}, {"daterangeday": "2020-01-02", "visits": 12, "bounces": 22}, @@ -172,13 +132,10 @@ def test_get_parsed_report(self, mock_adobe_client, mock_get_report_page): for output_record, expected_record in zip(output, expected): self.assertEqual(output_record, expected_record) - @mock.patch("nck.clients.adobe_client.AdobeClient.__init__", return_value=None) + @mock.patch("nck.clients.adobe_analytics.client.AdobeAnalyticsClient.__init__", return_value=None) @mock.patch( - "nck.readers.adobe_reader_2_0.AdobeReader_2_0.get_node_values", - return_value={ - "lasttouchchannel_1": "Paid Search", - "lasttouchchannel_2": "Natural_Search", - }, + "nck.readers.adobe_analytics_2_0.reader.AdobeAnalytics20Reader.get_node_values", + return_value={"lasttouchchannel_1": "Paid Search", "lasttouchchannel_2": "Natural_Search"}, ) def test_add_child_nodes_to_graph(self, mock_adobe_client, mock_get_node_values): graph = { @@ -189,7 +146,7 @@ def test_add_child_nodes_to_graph(self, mock_adobe_client, mock_get_node_values) node = "daterangeday_1200201" path_to_node = ["daterangeday_1200201"] - output = AdobeReader_2_0(**self.kwargs).add_child_nodes_to_graph(graph, node, path_to_node) + output = AdobeAnalytics20Reader(**self.kwargs).add_child_nodes_to_graph(graph, node, path_to_node) expected = { "root": ["daterangeday_1200201", "daterangeday_1200202"], "daterangeday_1200201": ["lasttouchchannel_1", "lasttouchchannel_2"], @@ -199,9 +156,9 @@ def test_add_child_nodes_to_graph(self, mock_adobe_client, mock_get_node_values) } self.assertEqual(output, expected) - @mock.patch("nck.clients.adobe_client.AdobeClient.__init__", return_value=None) + @mock.patch("nck.clients.adobe_analytics.client.AdobeAnalyticsClient.__init__", return_value=None) @mock.patch( - "nck.readers.adobe_reader_2_0.AdobeReader_2_0.get_parsed_report", + "nck.readers.adobe_analytics_2_0.reader.AdobeAnalytics20Reader.get_parsed_report", return_value=[ {"daterangeday": "2020-01-01", "visits": 11, "bounces": 21}, {"daterangeday": "2020-01-02", "visits": 12, "bounces": 22}, @@ -211,7 +168,7 @@ def test_read_one_dimension_reports(self, mock_adobe_client, mock_get_parsed_rep temp_kwargs = self.kwargs.copy() temp_kwargs.update({"dimension": ["daterangeday"], "metric": ["visits", "bounces"]}) - output = next(AdobeReader_2_0(**temp_kwargs).read()) + output = next(AdobeAnalytics20Reader(**temp_kwargs).read()) expected = [ {"daterangeday": "2020-01-01", "visits": 11, "bounces": 21}, {"daterangeday": "2020-01-02", "visits": 12, "bounces": 22}, @@ -219,9 +176,9 @@ def test_read_one_dimension_reports(self, mock_adobe_client, mock_get_parsed_rep for output_record, expected_output in zip(output.readlines(), iter(expected)): self.assertEqual(output_record, expected_output) - @mock.patch("nck.clients.adobe_client.AdobeClient.__init__", return_value=None) + @mock.patch("nck.clients.adobe_analytics.client.AdobeAnalyticsClient.__init__", return_value=None) @mock.patch( - "nck.readers.adobe_reader_2_0.AdobeReader_2_0.add_child_nodes_to_graph", + "nck.readers.adobe_analytics_2_0.reader.AdobeAnalytics20Reader.add_child_nodes_to_graph", side_effect=[ { "root": ["daterangeday_1200201", "daterangeday_1200202"], @@ -244,7 +201,7 @@ def test_read_one_dimension_reports(self, mock_adobe_client, mock_get_parsed_rep ], ) @mock.patch( - "nck.readers.adobe_reader_2_0.AdobeReader_2_0.get_parsed_report", + "nck.readers.adobe_analytics_2_0.reader.AdobeAnalytics20Reader.get_parsed_report", side_effect=[ [ { @@ -282,13 +239,8 @@ def test_read_one_dimension_reports(self, mock_adobe_client, mock_get_parsed_rep ) def test_read_multiple_dimension_reports(self, mock_adobe_client, mock_add_child_nodes_to_graph, mock_get_parsed_report): temp_kwargs = self.kwargs.copy() - temp_kwargs.update( - { - "dimension": ["daterangeday", "lastouchchannel", "campaign"], - "metric": ["visits", "bounces"], - } - ) - reader = AdobeReader_2_0(**temp_kwargs) + temp_kwargs.update({"dimension": ["daterangeday", "lastouchchannel", "campaign"], "metric": ["visits", "bounces"]}) + reader = AdobeAnalytics20Reader(**temp_kwargs) reader.node_values = { "daterangeday_1200201": "Jan 1, 2020", "daterangeday_1200202": "Jan 2, 2020", diff --git a/tests/readers/confluence/__init__.py b/tests/readers/confluence/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/readers/confluence/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/helpers/test_confluence_helper.py b/tests/readers/confluence/test_helper.py similarity index 84% rename from tests/helpers/test_confluence_helper.py rename to tests/readers/confluence/test_helper.py index b71b7eab..1dbdbf1a 100644 --- a/tests/helpers/test_confluence_helper.py +++ b/tests/readers/confluence/test_helper.py @@ -1,6 +1,25 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + from unittest import TestCase -from parameterized import parameterized + from bs4 import BeautifulSoup +from parameterized import parameterized PARAGRAPH_OF_200_CHARACTERS = ( "Lorem ipsum sit amet cursus sit amet dictum sit amet justo donec" @@ -69,7 +88,7 @@ class CustomFieldsTest(TestCase): def test__get_tiny_link(self): - from nck.helpers.confluence_helper import _get_tiny_link + from nck.readers.confluence.helper import _get_tiny_link field_value = {"self": "https://your-domain.com/wiki/rest/api/content/00001", "tinyui": "/x/aBcD"} expected = "https://your-domain.com/wiki/x/aBcD" @@ -77,17 +96,17 @@ def test__get_tiny_link(self): @parameterized.expand([([{"name": "nck"}, {"name": "api"}], "name", "nck|api"), ([], "name", "")]) def test__get_key_values_from_list_of_dct(self, field_value, key, expected): - from nck.helpers.confluence_helper import _get_key_values_from_list_of_dct + from nck.readers.confluence.helper import _get_key_values_from_list_of_dct self.assertEqual(_get_key_values_from_list_of_dct(field_value, key), expected) def test__get_client_properties(self): - from nck.helpers.confluence_helper import _get_client_properties + from nck.readers.confluence.helper import _get_client_properties self.assertDictEqual(_get_client_properties(HTML_BODY), EXPECTED_CLIENT_PROPERTIES) def test__get_client_completion(self): - from nck.helpers.confluence_helper import _get_client_completion + from nck.readers.confluence.helper import _get_client_completion self.assertDictEqual(_get_client_completion(HTML_BODY), EXPECTED_CLIENT_COMPLETION) @@ -110,7 +129,7 @@ def test__get_client_completion(self): ] ) def test__get_section_by_title(self, html_body, expected): - from nck.helpers.confluence_helper import _get_section_by_title + from nck.readers.confluence.helper import _get_section_by_title searched_title = "APPROACH" html_soup = BeautifulSoup(html_body, "lxml") @@ -120,7 +139,7 @@ def test__get_section_by_title(self, html_body, expected): class DictToCleanTest(TestCase): def test__clean(self): - from nck.helpers.confluence_helper import DictToClean + from nck.readers.confluence.helper import DictToClean dct = {"CLIENT COMPANY": "Michelin", "SCOPE": "France", "TEAM SIZE": 10} expected_keys = ["CLIENT COMPANY", "AMOUNT SOLD", "SCOPE"] @@ -137,7 +156,7 @@ class ParseResponseTest(TestCase): [("title", ["title"]), ("space.name", ["space", "name"]), ("label_names", ["metadata", "labels", "results"])] ) def test__get_field_path(self, field, expected): - from nck.helpers.confluence_helper import _get_field_path + from nck.readers.confluence.helper import _get_field_path self.assertListEqual(_get_field_path(field), expected) @@ -150,7 +169,7 @@ def test__get_field_path(self, field, expected): ] ) def test__get_field_value(self, field_path, expected): - from nck.helpers.confluence_helper import _get_field_value + from nck.readers.confluence.helper import _get_field_value self.assertEqual(_get_field_value(CONTENT_DCT, field_path), expected) @@ -162,12 +181,12 @@ def test__get_field_value(self, field_path, expected): ] ) def test__format_field_as_dct(self, field, field_value, expected): - from nck.helpers.confluence_helper import _format_field_as_dct + from nck.readers.confluence.helper import _format_field_as_dct self.assertDictEqual(_format_field_as_dct(field, field_value), expected) def test__parse_response(self): - from nck.helpers.confluence_helper import parse_response + from nck.readers.confluence.helper import parse_response raw_response = { "results": [ @@ -206,6 +225,6 @@ def test__parse_response(self): @parameterized.expand([("\u2705 Title with \ud83d\udd36 emoji \ud83d\udd34", "Title with emoji"), (0, 0)]) def test__decode(self, raw_value, expected): - from nck.helpers.confluence_helper import _decode + from nck.readers.confluence.helper import _decode self.assertEqual(_decode(raw_value), expected) diff --git a/tests/readers/test_confluence_reader.py b/tests/readers/confluence/test_reader.py similarity index 94% rename from tests/readers/test_confluence_reader.py rename to tests/readers/confluence/test_reader.py index e0d7685b..a075df42 100644 --- a/tests/readers/test_confluence_reader.py +++ b/tests/readers/confluence/test_reader.py @@ -17,9 +17,9 @@ # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. from unittest import TestCase, mock -from click import ClickException -from nck.readers.confluence_reader import ConfluenceReader +from click import ClickException +from nck.readers.confluence.reader import ConfluenceReader KEY1_RAW_RESPONSE_PAGE0 = { "results": [ @@ -70,11 +70,10 @@ class ConfluenceReaderTest(TestCase): "content_type": "page", "spacekey": [], "field": ["title", "space.name", "label_names"], - "normalize_stream": False, } @mock.patch( - "nck.readers.confluence_reader.CUSTOM_FIELDS", + "nck.readers.confluence.reader.CUSTOM_FIELDS", { "custom_field_A": {"specific_to_spacekeys": ["KEY1"]}, "custom_field_B": {"specific_to_spacekeys": ["KEY1", "KEY2"]}, @@ -100,9 +99,7 @@ def test__build_params(self): expected = {"type": "page", "expand": "title,space.name,metadata.labels.results"} self.assertDictEqual(output, expected) - @mock.patch.object( - ConfluenceReader, "_get_raw_response", side_effect=[KEY1_RAW_RESPONSE_PAGE0, KEY1_RAW_RESPONSE_PAGE1] - ) + @mock.patch.object(ConfluenceReader, "_get_raw_response", side_effect=[KEY1_RAW_RESPONSE_PAGE0, KEY1_RAW_RESPONSE_PAGE1]) def test__get_report_generator(self, mock_get_raw_response): temp_kwargs = self.kwargs.copy() temp_kwargs.update({"spacekey": ["KEY1"]}) diff --git a/tests/readers/facebook/__init__.py b/tests/readers/facebook/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/readers/facebook/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/readers/test_facebook_reader.py b/tests/readers/facebook/test_reader.py similarity index 97% rename from tests/readers/test_facebook_reader.py rename to tests/readers/facebook/test_reader.py index ccfa606a..1cd9b813 100644 --- a/tests/readers/test_facebook_reader.py +++ b/tests/readers/facebook/test_reader.py @@ -22,7 +22,7 @@ from facebook_business.adobjects.ad import Ad from facebook_business.adobjects.adsinsights import AdsInsights from facebook_business.api import FacebookAdsApi -from nck.readers.facebook_reader import FacebookReader +from nck.readers.facebook.reader import FacebookReader from parameterized import parameterized @@ -136,7 +136,7 @@ def test_get_field_paths(self): ] self.assertEqual(FacebookReader(**temp_kwargs)._field_paths, expected) - @mock.patch("nck.readers.facebook_reader.FacebookReader.query_ad_insights") + @mock.patch("nck.readers.facebook.reader.FacebookReader.query_ad_insights") @mock.patch.object(FacebookReader, "get_params", lambda *args: {}) @mock.patch.object(FacebookAdsApi, "init", lambda *args: None) def test_read_with_ad_insights_query(self, mock_query_ad_insights): @@ -157,7 +157,7 @@ def test_read_with_ad_insights_query(self, mock_query_ad_insights): for record, report in zip(data.readlines(), iter(expected)): self.assertEqual(record, report) - @mock.patch("nck.readers.facebook_reader.FacebookReader.query_ad_management") + @mock.patch("nck.readers.facebook.reader.FacebookReader.query_ad_management") @mock.patch.object(FacebookReader, "get_params", lambda *args: {}) @mock.patch.object(FacebookAdsApi, "init", lambda *args: None) def test_read_with_ad_management_query(self, mock_query_ad_management): @@ -287,7 +287,7 @@ def test_format_and_yield(self, name, parameters, record, expected): ] ) def test_obj_follows_action_breakdown_pattern(self, name, obj, expected): - from nck.helpers.facebook_helper import obj_follows_action_breakdown_pattern + from nck.readers.facebook.helper import obj_follows_action_breakdown_pattern output = obj_follows_action_breakdown_pattern(obj) self.assertEqual(output, expected) @@ -299,7 +299,7 @@ def test_obj_follows_action_breakdown_pattern(self, name, obj, expected): ] ) def test_obj_is_list_of_single_values(self, name, obj, expected): - from nck.helpers.facebook_helper import obj_is_list_of_single_values + from nck.readers.facebook.helper import obj_is_list_of_single_values output = obj_is_list_of_single_values(obj) self.assertEqual(output, expected) diff --git a/tests/readers/google_ads/__init__.py b/tests/readers/google_ads/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/readers/google_ads/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/readers/test_googleads_reader.py b/tests/readers/google_ads/test_reader.py similarity index 94% rename from tests/readers/test_googleads_reader.py rename to tests/readers/google_ads/test_reader.py index 8d6b6e4d..d37bd7b7 100644 --- a/tests/readers/test_googleads_reader.py +++ b/tests/readers/google_ads/test_reader.py @@ -15,11 +15,13 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + import datetime from unittest import TestCase, mock from click import ClickException -from nck.readers.googleads_reader import DATEFORMAT, GoogleAdsReader +from nck.readers.google_ads.reader import GoogleAdsReader +from nck.readers.google_ads.config import DATEFORMAT from nck.utils.exceptions import InconsistentDateDefinitionException, NoDateDefinitionException from parameterized import parameterized @@ -98,8 +100,8 @@ def test_invalid_report_filter(self): temp_kwargs.update({"report_filter": not_a_dict}) GoogleAdsReader(**temp_kwargs).add_report_filter(report_definition) - @mock.patch("nck.readers.googleads_reader.GoogleAdsReader.fetch_report_from_gads_client_customer_obj") - @mock.patch("nck.readers.googleads_reader.codecs.getreader", side_effect=mock_query) + @mock.patch("nck.readers.google_ads.reader.GoogleAdsReader.fetch_report_from_gads_client_customer_obj") + @mock.patch("nck.readers.google_ads.reader.codecs.getreader", side_effect=mock_query) @mock.patch.object(GoogleAdsReader, "__init__", mock_googleads_reader) def test_read_data(self, mock_report, mock_query): reader = GoogleAdsReader(**self.kwargs) @@ -113,8 +115,8 @@ def test_read_data(self, mock_report, mock_query): for record, output in zip(data.readlines(), iter(expected)): assert record == output - @mock.patch("nck.readers.googleads_reader.GoogleAdsReader.fetch_report_from_gads_client_customer_obj") - @mock.patch("nck.readers.googleads_reader.codecs.getreader", side_effect=mock_query) + @mock.patch("nck.readers.google_ads.reader.GoogleAdsReader.fetch_report_from_gads_client_customer_obj") + @mock.patch("nck.readers.google_ads.reader.codecs.getreader", side_effect=mock_query) @mock.patch.object(GoogleAdsReader, "__init__", mock_googleads_reader) def test_read_data_and_include_account_id(self, mock_report, mock_query): temp_kwargs = self.kwargs.copy() @@ -131,8 +133,8 @@ def test_read_data_and_include_account_id(self, mock_report, mock_query): for record, output in zip(data.readlines(), iter(expected)): assert record == output - @mock.patch("nck.readers.googleads_reader.GoogleAdsReader.fetch_report_from_gads_client_customer_obj") - @mock.patch("nck.readers.googleads_reader.codecs.getreader", side_effect=mock_video_query) + @mock.patch("nck.readers.google_ads.reader.GoogleAdsReader.fetch_report_from_gads_client_customer_obj") + @mock.patch("nck.readers.google_ads.reader.codecs.getreader", side_effect=mock_video_query) @mock.patch.object(GoogleAdsReader, "__init__", mock_googleads_reader) def test_list_video_campaign_ids(self, mock_report, mock_query): temp_kwargs = self.kwargs.copy() diff --git a/tests/readers/google_analytics/__init__.py b/tests/readers/google_analytics/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/readers/google_analytics/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/readers/test_ga_reader.py b/tests/readers/google_analytics/test_reader.py similarity index 85% rename from tests/readers/test_ga_reader.py rename to tests/readers/google_analytics/test_reader.py index e1e8876c..c249b909 100644 --- a/tests/readers/test_ga_reader.py +++ b/tests/readers/google_analytics/test_reader.py @@ -15,14 +15,15 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + from datetime import datetime from unittest import TestCase, mock -from click import ClickException -from nck.readers.ga_reader import GaReader +from click import ClickException +from nck.readers.google_analytics.reader import GoogleAnalyticsReader -class GaReaderTest(TestCase): +class GoogleAnalyticsReaderTest(TestCase): DATEFORMAT = "%Y-%m-%d" def mock_ga_reader(self, **kwargs): @@ -32,19 +33,19 @@ def mock_ga_reader(self, **kwargs): def test_format_date(self): test_date = "20190101" wrong_date = "01/01/2019" - assert GaReader.format_date(test_date) == "2019-01-01" - self.assertRaises(ValueError, GaReader.format_date, wrong_date) + assert GoogleAnalyticsReader.format_date(test_date) == "2019-01-01" + self.assertRaises(ValueError, GoogleAnalyticsReader.format_date, wrong_date) def test_get_days_delta(self): inputs = ["PREVIOUS_DAY", "LAST_7_DAYS", "LAST_30_DAYS", "LAST_90_DAYS"] expected = [1, 7, 30, 90] - output = [GaReader.get_days_delta(input) for input in inputs] + output = [GoogleAnalyticsReader.get_days_delta(input) for input in inputs] assert output == expected fail = "PVRIOUES_DAY" - self.assertRaises(ClickException, GaReader.get_days_delta, fail) + self.assertRaises(ClickException, GoogleAnalyticsReader.get_days_delta, fail) - @mock.patch("nck.readers.ga_reader.GaReader._run_query") - @mock.patch.object(GaReader, "__init__", mock_ga_reader) + @mock.patch("nck.readers.google_analytics.reader.GoogleAnalyticsReader._run_query") + @mock.patch.object(GoogleAnalyticsReader, "__init__", mock_ga_reader) def test_read(self, mock_query): kwargs = { @@ -55,9 +56,9 @@ def test_read(self, mock_query): "end_date": datetime(2019, 1, 1), "add_view": False, } - reader = GaReader(**kwargs) + reader = GoogleAnalyticsReader(**kwargs) kwargs["add_view"] = True - reader_with_view_id = GaReader(**kwargs) + reader_with_view_id = GoogleAnalyticsReader(**kwargs) format_data_return_value = [ { diff --git a/tests/readers/google_dbm/__init__.py b/tests/readers/google_dbm/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/readers/google_dbm/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/readers/test_dbm_reader.py b/tests/readers/google_dbm/test_reader.py similarity index 88% rename from tests/readers/test_dbm_reader.py rename to tests/readers/google_dbm/test_reader.py index eb676bc9..8e9ccdc5 100644 --- a/tests/readers/test_dbm_reader.py +++ b/tests/readers/google_dbm/test_reader.py @@ -15,22 +15,23 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + import datetime import unittest from unittest import mock -from nck.readers.dbm_reader import DbmReader +from nck.readers.google_dbm.reader import GoogleDBMReader -class TestDbmReader(unittest.TestCase): +class TestGoogleDBMReader(unittest.TestCase): def mock_dbm_reader(self, **kwargs): for param, value in kwargs.items(): setattr(self, param, value) - @mock.patch.object(DbmReader, "__init__", mock_dbm_reader) + @mock.patch.object(GoogleDBMReader, "__init__", mock_dbm_reader) def test_get_query_body(self): kwargs = {} - reader = DbmReader(**kwargs) + reader = GoogleDBMReader(**kwargs) reader.kwargs = {"filter": [("FILTER_ADVERTISER", 1)]} expected_query_body = { @@ -47,10 +48,10 @@ def test_get_query_body(self): self.assertDictEqual(reader.get_query_body(), expected_query_body) - @mock.patch.object(DbmReader, "__init__", mock_dbm_reader) + @mock.patch.object(GoogleDBMReader, "__init__", mock_dbm_reader) def test_get_query_body_ms_conversion(self): kwargs = {} - reader = DbmReader(**kwargs) + reader = GoogleDBMReader(**kwargs) reader.kwargs = { "filter": [("FILTER_ADVERTISER", 1)], "start_date": datetime.datetime(2020, 1, 15, tzinfo=datetime.timezone.utc), diff --git a/tests/readers/google_dcm/__init__.py b/tests/readers/google_dcm/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/readers/google_dcm/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/readers/test_dcm_reader.py b/tests/readers/google_dcm/test_reader.py similarity index 84% rename from tests/readers/test_dcm_reader.py rename to tests/readers/google_dcm/test_reader.py index c92ca65e..72056688 100644 --- a/tests/readers/test_dcm_reader.py +++ b/tests/readers/google_dcm/test_reader.py @@ -15,29 +15,30 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + from unittest import TestCase, mock -from nck.config import logger -from nck.readers.dcm_reader import DcmReader +from nck.config import logger +from nck.readers.google_dcm.reader import GoogleDCMReader -class DCMReaderTest(TestCase): +class GoogleDCMReaderTest(TestCase): def mock_dcm_reader(self, **kwargs): for param, value in kwargs.items(): setattr(self, param, value) kwargs = {"metrics": ["impressions", "clicks"], "dimensions": ["date"]} - @mock.patch.object(DcmReader, "__init__", mock_dcm_reader) + @mock.patch.object(GoogleDCMReader, "__init__", mock_dcm_reader) def test_empty_data(self): - reader = DcmReader(**self.kwargs) + reader = GoogleDCMReader(**self.kwargs) input_report = (row for row in [b"No", b"Consistent", b"Data"]) if len(list(reader.format_response(input_report))) > 0: assert False, "Data is not empty" - @mock.patch.object(DcmReader, "__init__", mock_dcm_reader) + @mock.patch.object(GoogleDCMReader, "__init__", mock_dcm_reader) def test_format_data(self): - reader = DcmReader(**self.kwargs) + reader = GoogleDCMReader(**self.kwargs) input_report = (row for row in [b"x", b"x", b"Report Fields", b"headers", b"1,2,3", b"4,5,6", b"Grand Total"]) expected = [{"date": "1", "impressions": "2", "clicks": "3"}, {"date": "4", "impressions": "5", "clicks": "6"}] input_list = list(reader.format_response(input_report)) diff --git a/tests/readers/google_dv360/__init__.py b/tests/readers/google_dv360/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/readers/google_dv360/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/readers/google_dv360/test_reader.py b/tests/readers/google_dv360/test_reader.py new file mode 100644 index 00000000..20e045d8 --- /dev/null +++ b/tests/readers/google_dv360/test_reader.py @@ -0,0 +1,47 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + +from nck.readers.google_dv360.reader import GoogleDV360Reader +from unittest import TestCase, mock + + +class TestGoogleDV360Reader(TestCase): + def mock_dv360_reader(self, **kwargs): + for param, value in kwargs.items(): + setattr(self, param, value) + + @mock.patch.object(GoogleDV360Reader, "__init__", mock_dv360_reader) + def test_get_sdf_body(self): + kwargs = {} + reader = GoogleDV360Reader(**kwargs) + reader.kwargs = { + "file_type": ["FILE_TYPE_INSERTION_ORDER", "FILE_TYPE_CAMPAIGN"], + "filter_type": "FILTER_TYPE_ADVERTISER_ID", + "advertiser_id": "4242424", + } + + expected_query_body = { + "parentEntityFilter": { + "fileType": ["FILE_TYPE_INSERTION_ORDER", "FILE_TYPE_CAMPAIGN"], + "filterType": "FILTER_TYPE_ADVERTISER_ID", + }, + "version": "SDF_VERSION_5_2", + "advertiserId": "4242424", + } + + self.assertDictEqual(reader._GoogleDV360Reader__get_sdf_body(), expected_query_body) diff --git a/tests/readers/google_search_console/__init__.py b/tests/readers/google_search_console/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/readers/google_search_console/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/readers/test_search_console_reader.py b/tests/readers/google_search_console/test_reader.py similarity index 90% rename from tests/readers/test_search_console_reader.py rename to tests/readers/google_search_console/test_reader.py index 749f252e..541db08e 100644 --- a/tests/readers/test_search_console_reader.py +++ b/tests/readers/google_search_console/test_reader.py @@ -15,15 +15,17 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + from datetime import datetime -from nck.readers.search_console_reader import SearchConsoleReader from unittest import TestCase, mock +from nck.readers.google_search_console.reader import GoogleSearchConsoleReader + -class SearchConsoleReaderTest(TestCase): +class GoogleSearchConsoleReaderTest(TestCase): DATEFORMAT = "%Y-%m-%d" - @mock.patch("nck.readers.search_console_reader.SearchConsoleReader._run_query") + @mock.patch("nck.readers.google_search_console.reader.GoogleSearchConsoleReader._run_query") def test_read(self, mock_query): kwargs = { "client_id": "", @@ -38,7 +40,7 @@ def test_read(self, mock_query): "row_limit": "", "date_range": None, } - reader = SearchConsoleReader(**kwargs) + reader = GoogleSearchConsoleReader(**kwargs) def test_read_empty_data(mock_query): mock_query.return_value = [{"responseAgregationType": "byPage"}] @@ -63,7 +65,7 @@ def test_format_data(mock_query): def test_format_data_with_date_column(mock_query): kwargs["date_column"] = True - reader = SearchConsoleReader(**kwargs) + reader = GoogleSearchConsoleReader(**kwargs) mock_query.return_value = [ {"rows": [{"keys": ["MOBILE"], "clicks": 1, "impressions": 2}], "responseAgregationType": "byPage"}, {"rows": [{"keys": ["DESKTOP"], "clicks": 3, "impressions": 4}], "responseAgregationType": "byPage"}, diff --git a/tests/readers/object_storage/__init__.py b/tests/readers/object_storage/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/readers/object_storage/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/readers/test_objectstorage_reader.py b/tests/readers/object_storage/test_reader.py similarity index 73% rename from tests/readers/test_objectstorage_reader.py rename to tests/readers/object_storage/test_reader.py index c2d52e28..b0b1ba27 100644 --- a/tests/readers/test_objectstorage_reader.py +++ b/tests/readers/object_storage/test_reader.py @@ -1,11 +1,28 @@ -import io +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + import csv +import io import json - -from parameterized import parameterized -from nck.readers.objectstorage_reader import ObjectStorageReader from unittest import TestCase, mock +from nck.readers.object_storage.reader import ObjectStorageReader +from parameterized import parameterized mock_csv_names = ["a.csv", "a.njson", "b.csv", "b.njson"] mock_csv_files = [ @@ -59,8 +76,8 @@ def mock_get_key(self, _object, **kwargs): return _object[0] -@mock.patch("nck.readers.objectstorage_reader.ObjectStorageReader.create_client") -@mock.patch("nck.readers.objectstorage_reader.ObjectStorageReader.create_bucket") +@mock.patch("nck.readers.object_storage.reader.ObjectStorageReader.create_client") +@mock.patch("nck.readers.object_storage.reader.ObjectStorageReader.create_bucket") @mock.patch.object(ObjectStorageReader, "download_object_to_file", write_to_file) @mock.patch.object(ObjectStorageReader, "to_object", mock_to_object) @mock.patch.object(ObjectStorageReader, "get_timestamp", mock_get_timestamp) diff --git a/tests/readers/radarly/__init__.py b/tests/readers/radarly/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/readers/radarly/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/readers/test_radarly_reader.py b/tests/readers/radarly/test_reader.py similarity index 93% rename from tests/readers/test_radarly_reader.py rename to tests/readers/radarly/test_reader.py index cc95093a..e2d4d96d 100644 --- a/tests/readers/test_radarly_reader.py +++ b/tests/readers/radarly/test_reader.py @@ -15,15 +15,16 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from nck.readers.radarly_reader import RadarlyReader -from unittest import TestCase, mock -from unittest.mock import MagicMock -from nck.config import logger +import json from datetime import datetime, timedelta from typing import Tuple +from unittest import TestCase, mock +from unittest.mock import MagicMock + import numpy as np -import json +from nck.config import logger +from nck.readers.radarly.reader import RadarlyReader def create_mock_payload(start_date: datetime, end_date: datetime) -> Tuple[datetime, datetime, int]: @@ -45,9 +46,9 @@ def create_mock_publications_iterator(param: Tuple[datetime, datetime, int]) -> class RadarlyReaderTest(TestCase): - @mock.patch("nck.readers.radarly_reader.RadarlyApi") - @mock.patch("nck.readers.radarly_reader.Project") - @mock.patch("nck.readers.radarly_reader.RadarlyReader.get_payload") + @mock.patch("nck.readers.radarly.reader.RadarlyApi") + @mock.patch("nck.readers.radarly.reader.Project") + @mock.patch("nck.readers.radarly.reader.RadarlyReader.get_payload") def test_read(self, mock_get_payload, mock_Project, mock_RadarlyApi): mock_RadarlyApi.init.side_effect = lambda client_id, client_secret: logger.info( "Mock RadarlyApi successfully initiated" diff --git a/tests/readers/test_dv360_reader.py b/tests/readers/test_dv360_reader.py deleted file mode 100644 index 3bb30c38..00000000 --- a/tests/readers/test_dv360_reader.py +++ /dev/null @@ -1,29 +0,0 @@ -from nck.readers.dv360_reader import DV360Reader -from unittest import TestCase, mock - - -class TestDV360Reader(TestCase): - def mock_dv360_reader(self, **kwargs): - for param, value in kwargs.items(): - setattr(self, param, value) - - @mock.patch.object(DV360Reader, "__init__", mock_dv360_reader) - def test_get_sdf_body(self): - kwargs = {} - reader = DV360Reader(**kwargs) - reader.kwargs = { - "file_type": ["FILE_TYPE_INSERTION_ORDER", "FILE_TYPE_CAMPAIGN"], - "filter_type": "FILTER_TYPE_ADVERTISER_ID", - "advertiser_id": "4242424", - } - - expected_query_body = { - "parentEntityFilter": { - "fileType": ["FILE_TYPE_INSERTION_ORDER", "FILE_TYPE_CAMPAIGN"], - "filterType": "FILTER_TYPE_ADVERTISER_ID", - }, - "version": "SDF_VERSION_5_2", - "advertiserId": "4242424", - } - - self.assertDictEqual(reader._DV360Reader__get_sdf_body(), expected_query_body) diff --git a/tests/readers/the_trade_desk/__init__.py b/tests/readers/the_trade_desk/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/readers/the_trade_desk/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/readers/test_ttd.py b/tests/readers/the_trade_desk/test_reader.py similarity index 83% rename from tests/readers/test_ttd.py rename to tests/readers/the_trade_desk/test_reader.py index f6acc3d8..517b7527 100644 --- a/tests/readers/test_ttd.py +++ b/tests/readers/the_trade_desk/test_reader.py @@ -16,11 +16,10 @@ # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. -from unittest import TestCase, mock -from nck.readers.ttd_reader import TheTradeDeskReader - from datetime import datetime +from unittest import TestCase, mock +from nck.readers.the_trade_desk.reader import TheTradeDeskReader from nck.utils.exceptions import DateDefinitionException @@ -37,7 +36,7 @@ class TheTradeDeskReaderTest(TestCase): "date_range": None, } - @mock.patch("nck.readers.ttd_reader.TheTradeDeskReader._build_headers", return_value={}) + @mock.patch("nck.readers.the_trade_desk.reader.TheTradeDeskReader._build_headers", return_value={}) def test_validate_dates(self, mock_build_headers): temp_kwargs = self.kwargs.copy() params = {"start_date": datetime(2020, 1, 3), "end_date": datetime(2020, 1, 1)} @@ -45,9 +44,9 @@ def test_validate_dates(self, mock_build_headers): with self.assertRaises(DateDefinitionException): TheTradeDeskReader(**temp_kwargs) - @mock.patch("nck.readers.ttd_reader.TheTradeDeskReader._build_headers", return_value={}) + @mock.patch("nck.readers.the_trade_desk.reader.TheTradeDeskReader._build_headers", return_value={}) @mock.patch( - "nck.readers.ttd_reader.TheTradeDeskReader._make_api_call", + "nck.readers.the_trade_desk.reader.TheTradeDeskReader._make_api_call", return_value={ "Result": [ { @@ -66,9 +65,9 @@ def test_get_report_template_id_if_exactly_1_match(self, mock_build_headers, moc reader._get_report_template_id() self.assertEqual(reader.report_template_id, 1234) - @mock.patch("nck.readers.ttd_reader.TheTradeDeskReader._build_headers", return_value={}) + @mock.patch("nck.readers.the_trade_desk.reader.TheTradeDeskReader._build_headers", return_value={}) @mock.patch( - "nck.readers.ttd_reader.TheTradeDeskReader._make_api_call", + "nck.readers.the_trade_desk.reader.TheTradeDeskReader._make_api_call", return_value={ "Result": [ { @@ -93,17 +92,17 @@ def test_get_report_template_id_if_more_than_1_match(self, mock_build_headers, m with self.assertRaises(Exception): TheTradeDeskReader(**self.kwargs)._get_report_template_id() - @mock.patch("nck.readers.ttd_reader.TheTradeDeskReader._build_headers", return_value={}) + @mock.patch("nck.readers.the_trade_desk.reader.TheTradeDeskReader._build_headers", return_value={}) @mock.patch( - "nck.readers.ttd_reader.TheTradeDeskReader._make_api_call", return_value={"Result": [], "ResultCount": 0}, + "nck.readers.the_trade_desk.reader.TheTradeDeskReader._make_api_call", return_value={"Result": [], "ResultCount": 0}, ) def test_get_report_template_id_if_no_match(self, mock_build_headers, mock_api_call): with self.assertRaises(Exception): TheTradeDeskReader(**self.kwargs)._get_report_template_id() - @mock.patch("nck.readers.ttd_reader.TheTradeDeskReader._build_headers", return_value={}) + @mock.patch("nck.readers.the_trade_desk.reader.TheTradeDeskReader._build_headers", return_value={}) @mock.patch( - "nck.readers.ttd_reader.TheTradeDeskReader._make_api_call", + "nck.readers.the_trade_desk.reader.TheTradeDeskReader._make_api_call", return_value={"ReportScheduleId": 5678, "ReportScheduleName": "adgroup_performance_schedule"}, ) def test_create_report_schedule(self, mock_build_headers, mock_api_call): @@ -112,10 +111,10 @@ def test_create_report_schedule(self, mock_build_headers, mock_api_call): reader._create_report_schedule() self.assertEqual(reader.report_schedule_id, 5678) - @mock.patch("nck.readers.ttd_reader.TheTradeDeskReader._build_headers", return_value={}) + @mock.patch("nck.readers.the_trade_desk.reader.TheTradeDeskReader._build_headers", return_value={}) @mock.patch("tenacity.BaseRetrying.wait", side_effect=lambda *args, **kwargs: 0) @mock.patch( - "nck.readers.ttd_reader.TheTradeDeskReader._make_api_call", + "nck.readers.the_trade_desk.reader.TheTradeDeskReader._make_api_call", side_effect=[ { "Result": [ @@ -149,13 +148,13 @@ def test_wait_for_download_url(self, mock_build_headers, mock_retry, mock_api_ca reader._wait_for_download_url() self.assertEqual(reader.download_url, "https://download.url") - @mock.patch("nck.readers.ttd_reader.TheTradeDeskReader._build_headers", return_value={}) + @mock.patch("nck.readers.the_trade_desk.reader.TheTradeDeskReader._build_headers", return_value={}) @mock.patch("tenacity.BaseRetrying.wait", side_effect=lambda *args, **kwargs: 0) @mock.patch.object(TheTradeDeskReader, "_get_report_template_id", lambda *args: None) @mock.patch.object(TheTradeDeskReader, "_create_report_schedule", lambda *args: None) @mock.patch.object(TheTradeDeskReader, "_wait_for_download_url", lambda *args: None) @mock.patch( - "nck.readers.ttd_reader.TheTradeDeskReader._download_report", + "nck.readers.the_trade_desk.reader.TheTradeDeskReader._download_report", return_value=iter( [ {"Date": "2020-01-01T00:00:00", "Advertiser ID": "XXXXX", "Impressions": 10}, diff --git a/tests/readers/twitter/__init__.py b/tests/readers/twitter/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/readers/twitter/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/readers/test_twitter_reader.py b/tests/readers/twitter/test_reader.py similarity index 99% rename from tests/readers/test_twitter_reader.py rename to tests/readers/twitter/test_reader.py index e2504454..9b5861a8 100644 --- a/tests/readers/test_twitter_reader.py +++ b/tests/readers/twitter/test_reader.py @@ -16,15 +16,14 @@ # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. +from datetime import datetime from unittest import TestCase, mock + from click import ClickException from freezegun import freeze_time -from datetime import datetime - -from twitter_ads.client import Client - -from nck.readers.twitter_reader import TwitterReader +from nck.readers.twitter.reader import TwitterReader from nck.utils.exceptions import DateDefinitionException +from twitter_ads.client import Client class TwitterReaderTest(TestCase): diff --git a/tests/readers/yandex_campaign/__init__.py b/tests/readers/yandex_campaign/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/readers/yandex_campaign/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/readers/test_yandex_campaign_reader.py b/tests/readers/yandex_campaign/test_reader.py similarity index 52% rename from tests/readers/test_yandex_campaign_reader.py rename to tests/readers/yandex_campaign/test_reader.py index 48b4b0d2..b89ec6ff 100644 --- a/tests/readers/test_yandex_campaign_reader.py +++ b/tests/readers/yandex_campaign/test_reader.py @@ -15,63 +15,50 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + import unittest +from nck.readers.yandex_campaign.reader import YandexCampaignReader from parameterized import parameterized -from nck.readers.yandex_campaign_reader import YandexCampaignReader - class TestYandexCampaignReader(unittest.TestCase): - - @parameterized.expand([ - ( - { - "campaign_states": (), - "campaign_ids": (), - "campaign_statuses": (), - "campaign_payment_statuses": () - }, - { - "method": "get", - "params": { - "SelectionCriteria": { + @parameterized.expand( + [ + ( + {"campaign_states": (), "campaign_ids": (), "campaign_statuses": (), "campaign_payment_statuses": ()}, + { + "method": "get", + "params": { + "SelectionCriteria": {}, + "FieldNames": ["Id", "Name", "TimeZone", "DailyBudget", "Currency", "EndDate", "StartDate"], }, - "FieldNames": ["Id", "Name", "TimeZone", "DailyBudget", "Currency", "EndDate", "StartDate"] - } - } - ), - ( - { - "campaign_states": ("ON",), - "campaign_ids": (), - "campaign_statuses": ("ACCEPTED",), - "campaign_payment_statuses": ("ALLOWED",) - }, - { - "method": "get", - "params": { - "SelectionCriteria": { - "States": ["ON"], - "Statuses": ["ACCEPTED"], - "StatusesPayment": ["ALLOWED"] + }, + ), + ( + { + "campaign_states": ("ON",), + "campaign_ids": (), + "campaign_statuses": ("ACCEPTED",), + "campaign_payment_statuses": ("ALLOWED",), + }, + { + "method": "get", + "params": { + "SelectionCriteria": {"States": ["ON"], "Statuses": ["ACCEPTED"], "StatusesPayment": ["ALLOWED"]}, + "FieldNames": ["Id", "Name", "TimeZone", "DailyBudget", "Currency", "EndDate", "StartDate"], }, - "FieldNames": ["Id", "Name", "TimeZone", "DailyBudget", "Currency", "EndDate", "StartDate"] - } - } - ) - ]) - def test_get_query_body( - self, - kwargs, - expected_query_body - ): + }, + ), + ] + ) + def test_get_query_body(self, kwargs, expected_query_body): reader = YandexCampaignReader( "123", ("Id", "Name", "TimeZone", "DailyBudget", "Currency", "EndDate", "StartDate"), campaign_ids=kwargs["campaign_ids"], campaign_states=kwargs["campaign_states"], campaign_statuses=kwargs["campaign_statuses"], - campaign_payment_statuses=kwargs["campaign_payment_statuses"] + campaign_payment_statuses=kwargs["campaign_payment_statuses"], ) self.assertDictEqual(reader._build_request_body(), expected_query_body) diff --git a/tests/readers/yandex_statistics/__init__.py b/tests/readers/yandex_statistics/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/readers/yandex_statistics/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/readers/test_yandex_statistics_reader.py b/tests/readers/yandex_statistics/test_reader.py similarity index 99% rename from tests/readers/test_yandex_statistics_reader.py rename to tests/readers/yandex_statistics/test_reader.py index fcecf729..70096cf5 100644 --- a/tests/readers/test_yandex_statistics_reader.py +++ b/tests/readers/yandex_statistics/test_reader.py @@ -15,11 +15,12 @@ # You should have received a copy of the GNU Lesser General Public License # along with this program; if not, write to the Free Software Foundation, # Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. + import datetime import unittest import click -from nck.readers.yandex_statistics_reader import YandexStatisticsReader +from nck.readers.yandex_statistics.reader import YandexStatisticsReader from parameterized import parameterized diff --git a/tests/writers/__init__.py b/tests/writers/__init__.py new file mode 100644 index 00000000..d46139b7 --- /dev/null +++ b/tests/writers/__init__.py @@ -0,0 +1,17 @@ +# GNU Lesser General Public License v3.0 only +# Copyright (C) 2020 Artefact +# licence-information@artefact.com +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 3 of the License, or (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public License +# along with this program; if not, write to the Free Software Foundation, +# Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. diff --git a/tests/writers/amazon_s3/__init__.py b/tests/writers/amazon_s3/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/tests/writers/test_s3_writer.py b/tests/writers/amazon_s3/test_writer.py similarity index 76% rename from tests/writers/test_s3_writer.py rename to tests/writers/amazon_s3/test_writer.py index 0a6647b6..bb5a793e 100644 --- a/tests/writers/test_s3_writer.py +++ b/tests/writers/amazon_s3/test_writer.py @@ -1,12 +1,11 @@ -import boto3 import json +from unittest import TestCase +import boto3 from moto import mock_s3 -from unittest import TestCase -from parameterized import parameterized -from nck.writers.s3_writer import S3Writer from nck.streams.json_stream import JSONStream - +from nck.writers.amazon_s3.writer import AmazonS3Writer +from parameterized import parameterized list_dict = [{"a": "4", "b": "5", "c": "6"}, {"a": "7", "b": "8", "c": "9"}] @@ -21,7 +20,7 @@ def mock_stream(list_dict, name): @mock_s3 -class S3WriterTest(TestCase): +class AmazonS3WriterTest(TestCase): @classmethod @mock_s3 def setUpClass(cls): @@ -30,21 +29,18 @@ def setUpClass(cls): def test_bucket_doesnt_exist(self): with self.assertRaisesRegex(Exception, "non-existing-bucket bucket does not exist. available buckets are \['test'\]"): - S3Writer("non-existing-bucket", "us-east-1", "", "") + AmazonS3Writer("non-existing-bucket", "us-east-1", "", "") @parameterized.expand( - [ - (None, "stream_name.format", "stream_name.format"), - ("file_name", "stream_name.format", "file_name.format"), - ] + [(None, "stream_name.format", "stream_name.format"), ("file_name", "stream_name.format", "file_name.format")] ) def test_valid_filename(self, file_name, stream_name, expected): - writer = S3Writer("test", "us-east-1", "", "", prefix=None, filename=file_name) + writer = AmazonS3Writer("test", "us-east-1", "", "", prefix=None, filename=file_name) writer._set_valid_file_name(stream_name) self.assertEqual(expected, writer._file_name) def test_Write(self): - writer = S3Writer("test", "us-east-1", "", "") + writer = AmazonS3Writer("test", "us-east-1", "", "") writer.write(mock_stream(list_dict, "test")) client = boto3.resource("s3", region_name="us-east-1")