example/fixerio-gsheet-twice.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dynamic_singer as dsinger"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## source\n",
    "\n",
    "For source, I am going to use fixer-io, https://github.com/huseinzol05/dynamic-singer/tree/master/tap-fixerio\n",
    "\n",
    "Make sure we already installed the module in our local,\n",
    "\n",
    "```bash\n",
    "git clone https://github.com/huseinzol05/dynamic-singer.git\n",
    "cd dynamic-singer\n",
    "pip install -e tap-fixerio\n",
    "```\n",
    "\n",
    "And if we check in https://github.com/huseinzol05/dynamic-singer/tree/master/tap-fixerio Readme, we need to build a json config,\n",
    "\n",
    "```json\n",
    "{\n",
    "    \"base\": \"USD\",\n",
    "    \"start_date\": \"<YYYY-MM-DD>\",\n",
    "    \"access_key\": \"<your_access_key>\",\n",
    "    \"symbols\": [\"USD\",\"EUR\",\"GBP\",\"JPY\"]\n",
    "}\n",
    "```\n",
    "\n",
    "So we need to create one. Just get our `access_token` at https://fixer.io/quickstart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "config = {\n",
    "    'base': 'USD',\n",
    "    'start_date': '2020-05-05',\n",
    "    'access_key': '6b3f8a3cd7843530014751f7e6a8066d',\n",
    "    'symbols': ['GBP','JPY']\n",
    "}\n",
    "\n",
    "with open('fixer-config.json', 'w') as fopen:\n",
    "    json.dump(config, fopen)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## start\n",
    "\n",
    "So to dynamic-singer, we need to initiate `Source` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "source = dsinger.Source('tap-fixerio --config fixer-config.json')\n",
    "# source.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are not yet able to run it, because we do not add any data sources to sink. So, let's add one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## target\n",
    "\n",
    "For target, I am going to use gsheet, https://github.com/singer-io/target-gsheet, In this example, I am not going to share my gcp secret.\n",
    "\n",
    "This is the spreadsheet, https://docs.google.com/spreadsheets/d/1fH7C2KCi3P1Uef5wNv8-f_oJlYGYat9d5e5zKxkMoOk/edit?usp=sharing\n",
    "\n",
    "Second spreadsheet, https://docs.google.com/spreadsheets/d/1lN3o4UiT1SRV7Wqxg_JWHtKiaoNLJXGtm8kdoJ7NHO8/edit?usp=sharing\n",
    "\n",
    "Make sure we already installed the module in our local,\n",
    "\n",
    "```bash\n",
    "pip3 install target-gsheet\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = {\n",
    "    'spreadsheet_id': '1fH7C2KCi3P1Uef5wNv8-f_oJlYGYat9d5e5zKxkMoOk'\n",
    "}\n",
    "with open('gsheet-config.json', 'w') as fopen:\n",
    "    json.dump(config, fopen)\n",
    "    \n",
    "config = {\n",
    "    'spreadsheet_id': '1lN3o4UiT1SRV7Wqxg_JWHtKiaoNLJXGtm8kdoJ7NHO8'\n",
    "}\n",
    "with open('gsheet-config1.json', 'w') as fopen:\n",
    "    json.dump(config, fopen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "source.add('target-gsheet --config gsheet-config.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "source.add('target-gsheet --config gsheet-config1.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['target-gsheet --config gsheet-config.json',\n",
       " 'target-gsheet --config gsheet-config1.json']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "source.get_targets()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO {\"type\": \"SCHEMA\", \"stream\": \"exchange_rate\", \"schema\": {\"type\": \"object\", \"properties\": {\"date\": {\"type\": \"string\", \"format\": \"date-time\"}}, \"additionalProperties\": true}, \"key_properties\": [\"date\"]}\n",
      "INFO \n",
      "INFO \n",
      "INFO {\"type\": \"RECORD\", \"stream\": \"exchange_rate\", \"record\": {\"GBP\": \"0.871002\", \"JPY\": \"115.375629\", \"EUR\": \"1.0\", \"date\": \"2020-05-05T00:00:00Z\"}}\n",
      "INFO \n",
      "INFO \n",
      "INFO {\"type\": \"RECORD\", \"stream\": \"exchange_rate\", \"record\": {\"GBP\": \"0.871898\", \"JPY\": \"115.277119\", \"EUR\": \"1.0\", \"date\": \"2020-05-06T00:00:00Z\"}}\n",
      "INFO \n",
      "INFO \n",
      "INFO {\"type\": \"STATE\", \"value\": {\"start_date\": \"2020-05-06\"}}\n",
      "INFO \n",
      "INFO \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO Replicating exchange rate data from fixer.io starting from 2020-05-05\n",
      "INFO Tap exiting normally\n",
      "INFO Sending version information to stitchdata.com. To disable sending anonymous usage data, set the config parameter \"disable_collection\" to true\n",
      "INFO URL being requested: GET https://sheets.googleapis.com/v4/spreadsheets/1fH7C2KCi3P1Uef5wNv8-f_oJlYGYat9d5e5zKxkMoOk?alt=json\n",
      "INFO URL being requested: POST https://sheets.googleapis.com/v4/spreadsheets/1fH7C2KCi3P1Uef5wNv8-f_oJlYGYat9d5e5zKxkMoOk:batchUpdate?alt=json\n",
      "INFO URL being requested: GET https://sheets.googleapis.com/v4/spreadsheets/1fH7C2KCi3P1Uef5wNv8-f_oJlYGYat9d5e5zKxkMoOk?alt=json\n",
      "INFO URL being requested: POST https://sheets.googleapis.com/v4/spreadsheets/1fH7C2KCi3P1Uef5wNv8-f_oJlYGYat9d5e5zKxkMoOk/values/exchange_rate%21A1%3AZZZ:append?valueInputOption=USER_ENTERED&alt=json\n",
      "INFO URL being requested: POST https://sheets.googleapis.com/v4/spreadsheets/1fH7C2KCi3P1Uef5wNv8-f_oJlYGYat9d5e5zKxkMoOk/values/exchange_rate%21A1%3AZZZ:append?valueInputOption=USER_ENTERED&alt=json\n",
      "INFO URL being requested: POST https://sheets.googleapis.com/v4/spreadsheets/1fH7C2KCi3P1Uef5wNv8-f_oJlYGYat9d5e5zKxkMoOk/values/exchange_rate%21A1%3AZZZ:append?valueInputOption=USER_ENTERED&alt=json\n",
      "INFO Sending version information to stitchdata.com. To disable sending anonymous usage data, set the config parameter \"disable_collection\" to true\n",
      "INFO URL being requested: GET https://sheets.googleapis.com/v4/spreadsheets/1lN3o4UiT1SRV7Wqxg_JWHtKiaoNLJXGtm8kdoJ7NHO8?alt=json\n",
      "INFO URL being requested: POST https://sheets.googleapis.com/v4/spreadsheets/1lN3o4UiT1SRV7Wqxg_JWHtKiaoNLJXGtm8kdoJ7NHO8:batchUpdate?alt=json\n",
      "INFO URL being requested: GET https://sheets.googleapis.com/v4/spreadsheets/1lN3o4UiT1SRV7Wqxg_JWHtKiaoNLJXGtm8kdoJ7NHO8?alt=json\n",
      "INFO URL being requested: POST https://sheets.googleapis.com/v4/spreadsheets/1lN3o4UiT1SRV7Wqxg_JWHtKiaoNLJXGtm8kdoJ7NHO8/values/exchange_rate%21A1%3AZZZ:append?valueInputOption=USER_ENTERED&alt=json\n",
      "INFO URL being requested: POST https://sheets.googleapis.com/v4/spreadsheets/1lN3o4UiT1SRV7Wqxg_JWHtKiaoNLJXGtm8kdoJ7NHO8/values/exchange_rate%21A1%3AZZZ:append?valueInputOption=USER_ENTERED&alt=json\n",
      "INFO URL being requested: POST https://sheets.googleapis.com/v4/spreadsheets/1lN3o4UiT1SRV7Wqxg_JWHtKiaoNLJXGtm8kdoJ7NHO8/values/exchange_rate%21A1%3AZZZ:append?valueInputOption=USER_ENTERED&alt=json\n"
     ]
    }
   ],
   "source": [
    "source.start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}