Merge pull request GoogleCloudPlatform#69 from morgante/gsuite-report…

…-sync

G Suite Admin Report Sync

README.md

@@ -23,8 +23,9 @@ The tools folder contains ready-made utilities which can simpilfy Google Cloud P
 * [CloudConnect](tools/cloudconnect) - A package that automates the setup of dual VPN tunnels between AWS and GCP.
 * [DNS Sync](tools/dns-sync) - Sync a Cloud DNS zone with GCE resources. Instances and load balancers are added to the cloud DNS zone as they start from compute_engine_activity log events sent from a pub/sub push subscription. Can sync multiple projects to a single Cloud DNS zone.
 * [GCE Quota Sync](tools/gce-quota-sync) - A tool that fetches resource quota usage from the GCE API and synchronizes it to Stackdriver as a custom metric, where it can be used to define automated alerts.
-* [GCS Bucket Mover](tools/gcs-bucket-mover) - A tool to move user's bucket, including objects, metadata, and ACL, from one project to another. 
+* [GCS Bucket Mover](tools/gcs-bucket-mover) - A tool to move user's bucket, including objects, metadata, and ACL, from one project to another.
 * [GKE Billing Export](tools/gke-billing-export) - Google Kubernetes Engine fine grained billing export.
+* [GSuite Exporter](tools/gsuite-exporter/) - A Python package that automates syncing Admin SDK APIs activity reports to a GCP destination. The module takes entries from the chosen Admin SDK API, converts them into the appropriate format for the destination, and exports them to a destination (e.g: Stackdriver Logging).
 * [LabelMaker](tools/labelmaker) - A tool that reads key:value pairs from a json file and labels the running instance and all attached drives accordingly.
 * [Maven Archetype Dataflow](tools/maven-archetype-dataflow) - A maven archetype which bootstraps a Dataflow project with common plugins pre-configured to help maintain high code quality.
 * [Site Verification Group Sync](tools/site-verification-group-sync) - A tool to provision "verified owner" permissions (to create GCS buckets with custom dns) based on membership of a Google Group.

tools/gsuite-exporter/.gitignore

@@ -0,0 +1,125 @@
+# Created by https://www.gitignore.io/api/python
+
+### Python ###
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+
+### Python Patch ###
+.venv/
+
+### Python.VirtualEnv Stack ###
+# Virtualenv
+# http://iamzed.com/2009/05/07/a-primer-on-virtualenv/
+[Bb]in
+[Ii]nclude
+[Ll]ib
+[Ll]ib64
+[Ll]ocal
+[Ss]cripts
+pyvenv.cfg
+pip-selfcheck.json
+
+
+# End of https://www.gitignore.io/api/python

tools/gsuite-exporter/README.md

@@ -0,0 +1,80 @@
+# GSuite Exporter
+
+This package handles exporting data from the GSuite Admin APIs to a destination.
+
+The following GSuite Admin APIs are currently supported:
+
+- `reports_v1` - [Reports API](https://developers.google.com/admin-sdk/reports/v1/get-start/getting-started)
+    - `admin` - [Admin activity reports](https://developers.google.com/admin-sdk/reports/v1/guides/manage-audit-admin.html)
+    - `drive` - [Google Drive activity reports](https://developers.google.com/admin-sdk/reports/v1/guides/manage-audit-drive.html)
+    - `login` - [Login activity reports](https://developers.google.com/admin-sdk/reports/v1/guides/manage-audit-login.html)
+    - `mobile` - [Mobile activity reports](https://developers.google.com/admin-sdk/reports/v1/guides/manage-audit-mobile.html)
+    - `token` - [Authorization Token activity reports](https://developers.google.com/admin-sdk/reports/v1/guides/manage-audit-tokens.html)
+
+The following destinations are currently supported:
+- [Stackdriver Logging](https://cloud.google.com/logging/docs/)
+
+## Requirements
+* A GSuite Admin account
+* A service account with:
+  * [GSuite domain-wide delegation](https://developers.google.com/admin-sdk/reports/v1/guides/delegation) enabled.
+  * The IAM role `roles/iam.tokenCreator` set on the organization.
+
+### Collectors
+To collect data from the Admin SDK APIs, you need to grant extra permissions to your service account:
+* Go to your [Admin Console](https://admin.google.com) and login with your GSuite administrator account
+* Navigate to `Security > Advanced Settings > Manage API client access`
+
+#### Reports API
+* Grant the following scopes (comma-separated) to your service account's `client_id`:
+  - https://www.googleapis.com/auth/admin.reports.audit.readonly
+  - https://www.googleapis.com/auth/iam
+
+### Exporters
+
+#### Stackdriver Logging
+To use the Stackdriver Logging exporter, you need to grant extra IAM roles to your service account:
+* `roles/logging.viewer` on the destination project
+* `roles/logging.logWriter` on the destination project
+
+## Installation
+```sh
+pip install gsuite-exporter
+```
+
+## Usage
+
+The **GSuite Exporter** can be used either using the command-line interface or as a Python library.
+
+### Using the CLI
+An example sync from the Admin Reports API to Stackdriver Logging for the 'login' application looks like:
+```sh
+gsuite-exporter
+  --credentials-path='/path/to/service/account/credentials.json'
+  --admin-user='<your_gsuite_admin>@<your_domain>'
+  --api='report_v1'
+  --applications='login drive token'
+  --project-id='<logging_project>'
+  --exporter='stackdriver_exporter.StackdriverExporter'
+```
+
+The `credentials_path` variable is optional and you can use [Application Default Credentials](https://cloud.google.com/docs/authentication/production#providing_credentials_to_your_application) instead.
+
+### Using as a library
+
+An example sync from the Admin Reports API to Stackdriver Logging looks like:
+
+```python
+from gsuite_exporter.cli import sync_all
+
+sync_all(
+    credentials_path=/path/to/service/account/credentials.json,
+    admin_user='<user>@<domain>',
+    api='reports_v1',
+    applications=['login', 'drive', 'token'],
+    project_id='<project-id>',
+    exporter_class='stackdriver_exporter.StackdriverExporter'
+)
+```
+
+More examples are available using the library functions under the [examples/](./examples/) directory.

tools/gsuite-exporter/examples/__init__.py

@@ -0,0 +1,14 @@
+# Copyright 2018 Google Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#            http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+

tools/gsuite-exporter/examples/sync_reports.py

@@ -0,0 +1,26 @@
+# Copyright 2018 Google Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#            http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+from gsuite_exporter.cli import sync_all
+
+if __name__ == '__main__':
+    sync_all(
+        admin_user='<gsuite_admin>@<domain>',
+        api='reports_v1',
+        applications=['login', 'admin', 'drive', 'mobile', 'token'],
+        project_id='<logging_project_id>',
+        exporter_cls='stackdriver_exporter.StackdriverExporter',
+        credentials_path=os.environ['GOOGLE_APPLICATION_CREDENTIALS']
+    )

tools/gsuite-exporter/gsuite_exporter/__init__.py

@@ -0,0 +1,14 @@
+# Copyright 2018 Google Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#            http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+

tools/gsuite-exporter/gsuite_exporter/auth.py

@@ -0,0 +1,86 @@
+# Copyright 2018 Google Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#            http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import google.auth
+from google.auth import iam
+from google.auth.credentials import with_scopes_if_required
+from google.auth._default import _load_credentials_from_file
+from google.auth.transport import requests
+from google.oauth2 import service_account
+from googleapiclient import discovery
+
+logger = logging.getLogger(__name__)
+
+_TOKEN_URI = 'https://accounts.google.com/o/oauth2/token'
+_TOKEN_SCOPE = frozenset(['https://www.googleapis.com/auth/iam'])
+
+def build_service(api, version, credentials_path=None, user_email=None, scopes=None):
+    """Build and returns a service object.
+
+    Allows delegation of GSuite permissions to the service account when the `user_email` argument is passed.
+
+    Args:
+      api (str): The Admin SDK API to use.
+      version (str): The Admin SDK API version to use.
+      credentials_path (str, optional): The path to the service account credentials.
+      user_email (str): The email of the user. Needs permissions to access the Admin APIs.
+      scopes (list, optional): A list of scopes to authenticate the request with.
+
+    Returns:
+      Google Service object.
+    """
+    if credentials_path is not None:
+        logger.info("Getting credentials from file '%s' ...", credentials_path)
+        credentials, _ = _load_credentials_from_file(credentials_path)
+    else:
+        logger.info("Getting default application credentials ...")
+        credentials, _ = google.auth.default()
+
+    if user_email is not None:  # make delegated credentials
+        credentials = _make_delegated_credentials(
+                credentials,
+                user_email,
+                scopes)
+
+    return discovery.build(api, version, credentials=credentials)
+
+def _make_delegated_credentials(credentials, user_email, scopes):
+    """Make delegated credentials.
+
+    Allows a service account to impersonate the user passed in `user_email`,
+    using a restricted set of scopes.
+
+    Args:
+        credentials (service_account.Credentials): The service account credentials.
+        user_email (str): The email for the user to impersonate.
+        scopes (list): A list of scopes.
+
+    Returns:
+        service_account.Credentials: The delegated credentials
+    """
+    request = requests.Request()
+    credentials = with_scopes_if_required(credentials, _TOKEN_SCOPE)
+    credentials.refresh(request)
+    email = credentials.service_account_email
+    signer = iam.Signer(
+        request,
+        credentials,
+        email)
+    return service_account.Credentials(
+        signer,
+        email,
+        _TOKEN_URI,
+        scopes=scopes,
+        subject=user_email)