Cloud automation self-learning project : Cron-send-receipt-spotify

A Python job executed by Cloud Run and scheduled by Cloud Scheduler at the end of every month) which retrieves from Gmail the latest receipt of Spotify Family and notifies all family members within a Facebook Messenger group.

Prerequisites

• Services:
  • Gmail + Facebook account
  • GCP account
• Authentification:
  • Enabled GCP APIs:
    • Cloud Run Admin API
    • Cloud Scheduler API
    • Secret Manager API
    • Compute Engine API
    • Gmail API
    • Cloud Logging API
    • Eventarc API
    • Cloud Resource Manager API
    • Identity and Access Management (IAM) API
    • Cloud Billing API
  • OAuth 2.0 Client IDs generated
• Tools:
  • Poetry >= 1.1.10
  • Python >= 3.7.2
  • Docker latest
  • Terraform 1.0.11

Installation

1. Clone the repository

With Terminal:

  git clone [email protected]:doanhat/cron-send-receipt-spotify.git
  cd cron-send-receipt-spotify/

With Pycharm:

Open Pycharm -> GET FROM VCS -> Copy and paste the git url in the field URL

2. Install dependencies:

Open Terminal (system or Pycharm), within the repository folder:

  poetry install # install virtual environment and python packages
  poetry export -f requirements.txt --output requirements.txt --without-hashes # generate requirements.txt

⚠️ In Pycharm, remember to config interpreter to choose the virtual environment generated by Poetry

⚠️ If some dependecy problems occur, delete the poetry.lock file and retry the above command

3. Set up the cron job !!

1. Set up authentification:

   1. Google API credentials for Gmail readonly access:
      • Create a new project in Google Cloud Console, for example send-facture-spotify
      • In APIs and Services, enable Gmail API
      • In APIs and Services/OAuth Consent Screen, enter application name and add your email to Test users
      • In APIs and Services/Credentials, create a OAuth 2.0 Client ID and download the JSON file, save it to Google Secret Manager as gmail_credentials
   2. Set up GCP service account:
      • https://cloud.google.com/docs/authentication/getting-started
      • Grant roles for the service account: Cloud Run Admin, Cloud Scheduler Admin, Service Account user , Secret Manager Accessor and Version Adder
   3. Set up remote credentials, perform these additional steps:
      1. In Google Cloud Console/APIs and Services/Credentials, add a Service Account for the project if not yet created
      2. In Google Cloud Console/Security/Secret Manager, add this secret :
         1. name: facebook_login, value:

            {
               "FB_USER_EMAIL_ADDRESS":<fb_email>,
               "FB_USER_PASSWORD":<fb_password>
            }
            

         2. At this moment, two secrets have been created, which are facebook_login and gmail_credentials, after the first run of Cloud Run, there will be also facebook_session and gmail_token generated, which will be reused for the next run

2. Build image Docker:

• In Terminal (system or Pycharm), within the repository folder:

  docker build -t cron-send-receipt-spotify:<version> -f docker/gcp_cloud_run/Dockerfile .
  docker tag cron-send-receipt-spotify:<version> <gcp_region>/<gcp_project_name>/cron-send-receipt-spotify
  docker push <gcp_region>/<gcp_project_name>/cron-send-receipt-spotify 

3. Deploy GCP resources with terraform:

• Define environment variables in terraform/main.tf, in locals:

  locals {
     service_account = "service-send-facture-spotify@send-facture-spotify.iam.gserviceaccount.com"
     thread_id = "<messenger_group_id>" # messenger group id
     project_id = "send-facture-spotify" 
     cre_secret_id = "gmail_credentials"
     tok_secret_id = "gmail_token"
     ses_secret_id = "facebook_session"
     log_secret_id = "facebook_login"
     gcp_region = "europe-west1"
     imap_url = "imap.gmail.com"
     ...
  }

  ℹ️ <messenger_group_id> example: 6947468915279299 is the id of https://www.facebook.com/messages/t/6947468915279299

• Verify the job schedule :

   resource "google_cloud_scheduler_job" "scheduler-send-receipt-spotify" {
      name             = "monthly-report-spotify-receipt"
      description      = "It will send monthly receipt to messenger group"
      schedule         = "59 10 28-31 * *"
      time_zone        = "CET"
      http_target {
         http_method = "GET"
         uri         = "${google_cloud_run_service.run-send-receipt-spotify.status[0].url}/?sender=${local.email_sender}&subject=${local.email_subject}&key_words=${local.email_key_words}"
         oidc_token {
            service_account_email = local.service_account
        }
     }
  }

     which means: At 10:59 on every day-of-month from 28 through 31, run the script

  ℹ️ Check out crontab.guru for more cron schedule expressions

• In Terminal (system or Pycharm), within the repository folder:

  • ⚠️ Replace PROJECT_NUMBER with current gcp project number

  make terraform-init
  make terraform-plan
  make terraform-apply

5. Voilà, all set !!

6. Improvement :

There's always enough rooms for improvement, we can take advantage of this project to learn :

• Using Airflow to schedule more complicated tasks
• Using CI/CD for more reliable deployment

Author

• @doanhat

Links

License

Reference:

Read Emails from Gmail using Gmail API

Gmail API - Quick Started

Fbchat documentation