Twitter_collection_of_violence.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4PorU7qOb8u_"
   },
   "source": [
    "## STEP 1: PYTHON PACKAGES INSTALLATION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q6XuSF_ob8vD"
   },
   "source": [
    "Install the following python packages that will help you to collect data from twiter.com "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-03T14:32:56.801170Z",
     "start_time": "2021-05-03T14:32:26.228917Z"
    },
    "id": "adv09Rkqb8vG"
   },
   "outputs": [],
   "source": [
    "#!pip install tweepy "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-03T14:33:01.468090Z",
     "start_time": "2021-05-03T14:32:56.803163Z"
    },
    "id": "1EzDYGg8b8vJ"
   },
   "outputs": [],
   "source": [
    "# !pip install unidecode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aOyPPhNBb8vL"
   },
   "source": [
    "## STEP 2: IMPORT IMPORTANT PACKAGES "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-04T18:12:13.904195Z",
     "start_time": "2021-05-04T18:12:04.514042Z"
    },
    "id": "NNzIyruUb8vO"
   },
   "outputs": [],
   "source": [
    "#import dependencies\n",
    "import tweepy\n",
    "from tweepy import OAuthHandler\n",
    "from tweepy.streaming import StreamListener\n",
    "import json\n",
    "from unidecode import unidecode\n",
    "import time\n",
    "import datetime\n",
    "from tqdm import tqdm \n",
    "import pandas as pd  \n",
    "import numpy as np "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WQ7rD4HDb8vS"
   },
   "source": [
    "## STEP 3:AUTHENTICATING TO Twitter’s API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VAy8TZR9b8vW"
   },
   "source": [
    "(a) First, apply for a developer account to access the API. The Standard APIs are sufficient for this tutorial. They’re free, but have some limitations that we’ll learn to work around in this tutorial.\n",
    "\n",
    "Click here to apply: [apply for developer account to acces the API](https://developer.twitter.com/en/apply-for-access)\n",
    "\n",
    "(b) Once your developer account is setup, create an app that will make use of the API by clicking on your username in the top right corner to open the drop down menu, and clicking “Apps” as shown below. Then select “Create an app” and fill out the form. \n",
    "\n",
    "(c) Now that you have created a developer account and an app, you should have a set of keys to connect to the Twitter API. Specifically, you’ll have an\n",
    "- API key\n",
    "- API secret key\n",
    "- Access token\n",
    "- Access token secret\n",
    "\n",
    "These could be inserted directly into your code to connect to the Twitter API, as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-04T18:18:13.572270Z",
     "start_time": "2021-05-04T18:18:13.563845Z"
    },
    "id": "l_tGb2Rqb8vZ"
   },
   "outputs": [],
   "source": [
    "consumer_key = ''\n",
    "consumer_secret = ''\n",
    "access_token = ''\n",
    "access_token_secret = ''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "x-YqKa8nb8va"
   },
   "source": [
    "## STEP 4:  CONNECT TO TWITTER API USING THE SECRETS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-04T18:18:14.913899Z",
     "start_time": "2021-05-04T18:18:14.892829Z"
    },
    "id": "Ao2JlutAb8vb"
   },
   "outputs": [],
   "source": [
    "auth = tweepy.OAuthHandler(consumer_key, consumer_secret)\n",
    "auth.set_access_token(access_token, access_token_secret)\n",
    "\n",
    "api = tweepy.API(auth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yP5bme2Mb8vc"
   },
   "source": [
    "## STEP 5: DEFINE A FUNCTION THAT WILL TAKE OUR SEARCH QUERY"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-04T18:18:16.137584Z",
     "start_time": "2021-05-04T18:18:16.118507Z"
    },
    "code_folding": [],
    "id": "8v5n7E90b8ve"
   },
   "outputs": [],
   "source": [
    "def tweetSearch(query, limit):\n",
    "    \"\"\"\n",
    "    This function will search a query provided in the twitter and,\n",
    "    retun a list of all tweets that have a query. \n",
    "    \"\"\"\n",
    "\n",
    "    # Create a blank variable\n",
    "    tweets = []\n",
    "\n",
    "    # Iterate through Twitter using Tweepy to find our query with our defined limit\n",
    "    for page in tweepy.Cursor(\n",
    "        api.search, q=query, count=limit, tweet_mode=\"extended\"\n",
    "    ).pages(limit):\n",
    "        for tweet in page:\n",
    "            tweets.append(tweet)\n",
    "\n",
    "    # return tweets\n",
    "    return tweets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XICSxwlZb8vf"
   },
   "source": [
    "## STEP 6: CREATE A FUNCTION TO SAVE TWEETS INTO A DATAFRAME"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-04T18:18:20.595196Z",
     "start_time": "2021-05-04T18:18:20.584124Z"
    },
    "id": "yDRT_rkLb8vg"
   },
   "outputs": [],
   "source": [
    "def tweets_to_data_frame(tweets):\n",
    "    \"\"\"\n",
    "    This function will receive tweets and collect specific data from it such as place, tweet's text,likes \n",
    "    retweets and save them into a pandas data frame.\n",
    "    \n",
    "    This function will return a pandas data frame that contains data from twitter.\n",
    "    \"\"\"\n",
    "    df = pd.DataFrame(data=[tweet.full_text.encode('utf-8') for tweet in tweets], columns=[\"Tweets\"])\n",
    "\n",
    "    df[\"id\"] = np.array([tweet.id for tweet in tweets])\n",
    "    df[\"lens\"] = np.array([len(tweet.full_text) for tweet in tweets])\n",
    "    df[\"date\"] = np.array([tweet.created_at for tweet in tweets])\n",
    "    df[\"place\"] = np.array([tweet.place for tweet in tweets])\n",
    "    df[\"coordinateS\"] = np.array([tweet.coordinates for tweet in tweets])\n",
    "    df[\"lang\"] = np.array([tweet.lang for tweet in tweets])\n",
    "    df[\"source\"] = np.array([tweet.source for tweet in tweets])\n",
    "    df[\"likes\"] = np.array([tweet.favorite_count for tweet in tweets])\n",
    "    df[\"retweets\"] = np.array([tweet.retweet_count for tweet in tweets])\n",
    "\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TVvMHBRnb8vi"
   },
   "source": [
    "## STEP 7: ADD TWITTER HASHTAGS RELATED TO GENDER BASED VIOLENCE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-04T18:18:22.174643Z",
     "start_time": "2021-05-04T18:18:22.154524Z"
    },
    "id": "r0tVDXcBb8vj"
   },
   "outputs": [],
   "source": [
    "# add hashtags in the following list\n",
    "hashtags = [\n",
    "'#GBV',\n",
    "'#sexism',\n",
    "'#rape'    \n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UysjRrU-b8vj"
   },
   "source": [
    "## STEP 8: RUN BOTH FUNCTIONS TO COLLECT DATA FROM TWITTER RELATED TO THE HASHTAGS LISTED ABOVE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-04T18:19:58.127497Z",
     "start_time": "2021-05-04T18:19:57.247013Z"
    },
    "id": "omenu95db8vk"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 3/3 [05:30<00:00, 110.23s/it]\n"
     ]
    }
   ],
   "source": [
    "total_tweets = 0\n",
    "\n",
    "\"\"\"\n",
    "The following for loop will collect a tweets that have the hashtags\n",
    " mentioned in the list and save the tweets into csv file\n",
    "\"\"\"\n",
    "\n",
    "for n in tqdm(hashtags):\n",
    "    # first we fetch all tweets that have specific hashtag\n",
    "    hash_tweets = tweetSearch(query=n,limit=7000)\n",
    "    total_tweets += int(len(hash_tweets))\n",
    "    \n",
    "    # second we convert our tweets into datarame\n",
    "    df = tweets_to_data_frame(hash_tweets)\n",
    "    \n",
    "    #third we save the dataframe into csv file\n",
    "    df.to_csv(\"{}_tweets.csv\".format(n))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-04T18:16:33.102245Z",
     "start_time": "2021-05-04T18:16:32.502Z"
    },
    "id": "Rw7W2zsqb8vm"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total_tweets: 9158\n"
     ]
    }
   ],
   "source": [
    "# show total number of tweets collected\n",
    "print(\"total_tweets: {}\".format(total_tweets))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Tweets</th>\n",
       "      <th>id</th>\n",
       "      <th>lens</th>\n",
       "      <th>date</th>\n",
       "      <th>place</th>\n",
       "      <th>coordinateS</th>\n",
       "      <th>lang</th>\n",
       "      <th>source</th>\n",
       "      <th>likes</th>\n",
       "      <th>retweets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>b'RT @KaceyKells: KELLCEY \"This story not only...</td>\n",
       "      <td>1389684729218093058</td>\n",
       "      <td>140</td>\n",
       "      <td>2021-05-04 20:53:53</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>0</td>\n",
       "      <td>10284</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>b'Had Twitter for like a month and still dk ho...</td>\n",
       "      <td>1389684562792484868</td>\n",
       "      <td>67</td>\n",
       "      <td>2021-05-04 20:53:13</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>b'RT @PixelProject: #AFRICA: Abuse of women in...</td>\n",
       "      <td>1389684068271468551</td>\n",
       "      <td>132</td>\n",
       "      <td>2021-05-04 20:51:15</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for Android</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>b'RT @CDCInjury: 97% of males who survived #ra...</td>\n",
       "      <td>1389683952424833027</td>\n",
       "      <td>140</td>\n",
       "      <td>2021-05-04 20:50:47</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>b'RT @CDCInjury: 97% of males who survived #ra...</td>\n",
       "      <td>1389683668688515072</td>\n",
       "      <td>140</td>\n",
       "      <td>2021-05-04 20:49:40</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4952</th>\n",
       "      <td>4952</td>\n",
       "      <td>b'RT @eliasamare: 6) The #SexualViolence, #rap...</td>\n",
       "      <td>1386847674482102272</td>\n",
       "      <td>144</td>\n",
       "      <td>2021-04-27 01:00:26</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>0</td>\n",
       "      <td>220</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4953</th>\n",
       "      <td>4953</td>\n",
       "      <td>b\"RT @eliasamare: 8) Again, here's Muluberhan ...</td>\n",
       "      <td>1386847625199058946</td>\n",
       "      <td>140</td>\n",
       "      <td>2021-04-27 01:00:14</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>0</td>\n",
       "      <td>232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4954</th>\n",
       "      <td>4954</td>\n",
       "      <td>b'RT @FeministLibya: In a football match today...</td>\n",
       "      <td>1386847010288975875</td>\n",
       "      <td>139</td>\n",
       "      <td>2021-04-27 00:57:48</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4955</th>\n",
       "      <td>4955</td>\n",
       "      <td>b'.\\xe2\\x80\\x98A Tigrayan womb should never gi...</td>\n",
       "      <td>1386846185474854912</td>\n",
       "      <td>275</td>\n",
       "      <td>2021-04-27 00:54:31</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4956</th>\n",
       "      <td>4956</td>\n",
       "      <td>b'Suit alleges LSU conspired to keep women qui...</td>\n",
       "      <td>1386846160271204352</td>\n",
       "      <td>217</td>\n",
       "      <td>2021-04-27 00:54:25</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4957 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Unnamed: 0                                             Tweets  \\\n",
       "0              0  b'RT @KaceyKells: KELLCEY \"This story not only...   \n",
       "1              1  b'Had Twitter for like a month and still dk ho...   \n",
       "2              2  b'RT @PixelProject: #AFRICA: Abuse of women in...   \n",
       "3              3  b'RT @CDCInjury: 97% of males who survived #ra...   \n",
       "4              4  b'RT @CDCInjury: 97% of males who survived #ra...   \n",
       "...          ...                                                ...   \n",
       "4952        4952  b'RT @eliasamare: 6) The #SexualViolence, #rap...   \n",
       "4953        4953  b\"RT @eliasamare: 8) Again, here's Muluberhan ...   \n",
       "4954        4954  b'RT @FeministLibya: In a football match today...   \n",
       "4955        4955  b'.\\xe2\\x80\\x98A Tigrayan womb should never gi...   \n",
       "4956        4956  b'Suit alleges LSU conspired to keep women qui...   \n",
       "\n",
       "                       id  lens                 date place coordinateS lang  \\\n",
       "0     1389684729218093058   140  2021-05-04 20:53:53   NaN         NaN   en   \n",
       "1     1389684562792484868    67  2021-05-04 20:53:13   NaN         NaN   en   \n",
       "2     1389684068271468551   132  2021-05-04 20:51:15   NaN         NaN   en   \n",
       "3     1389683952424833027   140  2021-05-04 20:50:47   NaN         NaN   en   \n",
       "4     1389683668688515072   140  2021-05-04 20:49:40   NaN         NaN   en   \n",
       "...                   ...   ...                  ...   ...         ...  ...   \n",
       "4952  1386847674482102272   144  2021-04-27 01:00:26   NaN         NaN   en   \n",
       "4953  1386847625199058946   140  2021-04-27 01:00:14   NaN         NaN   en   \n",
       "4954  1386847010288975875   139  2021-04-27 00:57:48   NaN         NaN   en   \n",
       "4955  1386846185474854912   275  2021-04-27 00:54:31   NaN         NaN   en   \n",
       "4956  1386846160271204352   217  2021-04-27 00:54:25   NaN         NaN   en   \n",
       "\n",
       "                   source  likes  retweets  \n",
       "0         Twitter Web App      0     10284  \n",
       "1      Twitter for iPhone      0         0  \n",
       "2     Twitter for Android      0         2  \n",
       "3      Twitter for iPhone      0         2  \n",
       "4         Twitter Web App      0         2  \n",
       "...                   ...    ...       ...  \n",
       "4952   Twitter for iPhone      0       220  \n",
       "4953   Twitter for iPhone      0       232  \n",
       "4954      Twitter Web App      0         4  \n",
       "4955      Twitter Web App      0         0  \n",
       "4956      Twitter Web App      0         0  \n",
       "\n",
       "[4957 rows x 11 columns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_rape = pd.read_csv('#rape_tweets.csv')\n",
    "data_rape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Unnamed: 0        0\n",
       "Tweets            0\n",
       "id                0\n",
       "lens              0\n",
       "date              0\n",
       "place          4928\n",
       "coordinateS    4955\n",
       "lang              0\n",
       "source            0\n",
       "likes             0\n",
       "retweets          0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_rape.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Tweets</th>\n",
       "      <th>id</th>\n",
       "      <th>lens</th>\n",
       "      <th>date</th>\n",
       "      <th>place</th>\n",
       "      <th>coordinateS</th>\n",
       "      <th>lang</th>\n",
       "      <th>source</th>\n",
       "      <th>likes</th>\n",
       "      <th>retweets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>b'Death to #religion\\nDeath to #homophobia\\nDe...</td>\n",
       "      <td>1389681824906063877</td>\n",
       "      <td>169</td>\n",
       "      <td>2021-05-04 20:42:20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for Android</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>b'RT @DingaBelle: @_snowbunting @Flomoll \\xf0\\...</td>\n",
       "      <td>1389680927031369734</td>\n",
       "      <td>140</td>\n",
       "      <td>2021-05-04 20:38:46</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for Android</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>b'RT @DingaBelle: @_snowbunting @Flomoll \\xf0\\...</td>\n",
       "      <td>1389676992648318980</td>\n",
       "      <td>140</td>\n",
       "      <td>2021-05-04 20:23:08</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>b\"Finally watched #SexyLamp after years of try...</td>\n",
       "      <td>1389676944782934017</td>\n",
       "      <td>229</td>\n",
       "      <td>2021-05-04 20:22:57</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for Android</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>b'\\xf0\\x9f\\x98\\xb1\\xf0\\x9f\\x98\\xb1\\xf0\\x9f\\x98...</td>\n",
       "      <td>1389675769522184199</td>\n",
       "      <td>302</td>\n",
       "      <td>2021-05-04 20:18:17</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for Android</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1140</th>\n",
       "      <td>1140</td>\n",
       "      <td>b'Have you ever given thought to the various f...</td>\n",
       "      <td>1386855800992198668</td>\n",
       "      <td>280</td>\n",
       "      <td>2021-04-27 01:32:44</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1141</th>\n",
       "      <td>1141</td>\n",
       "      <td>b'RT @SkaKeller: Very good that @vonderleyen p...</td>\n",
       "      <td>1386855523027492865</td>\n",
       "      <td>140</td>\n",
       "      <td>2021-04-27 01:31:37</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for Android</td>\n",
       "      <td>0</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1142</th>\n",
       "      <td>1142</td>\n",
       "      <td>b'How #Sexism \\n\\nIs #Coded Into the #Tech Ind...</td>\n",
       "      <td>1386854623391125507</td>\n",
       "      <td>289</td>\n",
       "      <td>2021-04-27 01:28:03</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Buffer</td>\n",
       "      <td>24</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1143</th>\n",
       "      <td>1143</td>\n",
       "      <td>b'RT @ESWNtweets: \"While there were significan...</td>\n",
       "      <td>1386853519538331655</td>\n",
       "      <td>140</td>\n",
       "      <td>2021-04-27 01:23:40</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for Android</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1144</th>\n",
       "      <td>1144</td>\n",
       "      <td>b\"This embarrassing faux pas was definitely dr...</td>\n",
       "      <td>1386852761988444164</td>\n",
       "      <td>281</td>\n",
       "      <td>2021-04-27 01:20:39</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1145 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Unnamed: 0                                             Tweets  \\\n",
       "0              0  b'Death to #religion\\nDeath to #homophobia\\nDe...   \n",
       "1              1  b'RT @DingaBelle: @_snowbunting @Flomoll \\xf0\\...   \n",
       "2              2  b'RT @DingaBelle: @_snowbunting @Flomoll \\xf0\\...   \n",
       "3              3  b\"Finally watched #SexyLamp after years of try...   \n",
       "4              4  b'\\xf0\\x9f\\x98\\xb1\\xf0\\x9f\\x98\\xb1\\xf0\\x9f\\x98...   \n",
       "...          ...                                                ...   \n",
       "1140        1140  b'Have you ever given thought to the various f...   \n",
       "1141        1141  b'RT @SkaKeller: Very good that @vonderleyen p...   \n",
       "1142        1142  b'How #Sexism \\n\\nIs #Coded Into the #Tech Ind...   \n",
       "1143        1143  b'RT @ESWNtweets: \"While there were significan...   \n",
       "1144        1144  b\"This embarrassing faux pas was definitely dr...   \n",
       "\n",
       "                       id  lens                 date place  coordinateS lang  \\\n",
       "0     1389681824906063877   169  2021-05-04 20:42:20   NaN          NaN   en   \n",
       "1     1389680927031369734   140  2021-05-04 20:38:46   NaN          NaN   en   \n",
       "2     1389676992648318980   140  2021-05-04 20:23:08   NaN          NaN   en   \n",
       "3     1389676944782934017   229  2021-05-04 20:22:57   NaN          NaN   en   \n",
       "4     1389675769522184199   302  2021-05-04 20:18:17   NaN          NaN   en   \n",
       "...                   ...   ...                  ...   ...          ...  ...   \n",
       "1140  1386855800992198668   280  2021-04-27 01:32:44   NaN          NaN   en   \n",
       "1141  1386855523027492865   140  2021-04-27 01:31:37   NaN          NaN   en   \n",
       "1142  1386854623391125507   289  2021-04-27 01:28:03   NaN          NaN   en   \n",
       "1143  1386853519538331655   140  2021-04-27 01:23:40   NaN          NaN   en   \n",
       "1144  1386852761988444164   281  2021-04-27 01:20:39   NaN          NaN   en   \n",
       "\n",
       "                   source  likes  retweets  \n",
       "0     Twitter for Android      0         0  \n",
       "1     Twitter for Android      0         3  \n",
       "2      Twitter for iPhone      0         3  \n",
       "3     Twitter for Android      1         0  \n",
       "4     Twitter for Android      1         0  \n",
       "...                   ...    ...       ...  \n",
       "1140      Twitter Web App      0         0  \n",
       "1141  Twitter for Android      0        22  \n",
       "1142               Buffer     24        16  \n",
       "1143  Twitter for Android      0         8  \n",
       "1144      Twitter Web App      0         0  \n",
       "\n",
       "[1145 rows x 11 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_sexism = pd.read_csv('#sexism_tweets.csv')\n",
    "data_sexism"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Unnamed: 0        0\n",
       "Tweets            0\n",
       "id                0\n",
       "lens              0\n",
       "date              0\n",
       "place          1134\n",
       "coordinateS    1145\n",
       "lang              0\n",
       "source            0\n",
       "likes             0\n",
       "retweets          0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_sexism.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_GBV = pd.read_csv('#GBV_tweets.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Tweets</th>\n",
       "      <th>id</th>\n",
       "      <th>lens</th>\n",
       "      <th>date</th>\n",
       "      <th>place</th>\n",
       "      <th>coordinateS</th>\n",
       "      <th>lang</th>\n",
       "      <th>source</th>\n",
       "      <th>likes</th>\n",
       "      <th>retweets</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>b'Can\\xe2\\x80\\x99t wait for you kids to hear t...</td>\n",
       "      <td>1389683945143427076</td>\n",
       "      <td>136</td>\n",
       "      <td>2021-05-04 20:50:46</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>b'RT @CedawPT: Women journalists can be target...</td>\n",
       "      <td>1389683664150319113</td>\n",
       "      <td>148</td>\n",
       "      <td>2021-05-04 20:49:39</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for Android</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>b'RT @mobkimarkus7: Save the Girl Child, suppo...</td>\n",
       "      <td>1389681203838656516</td>\n",
       "      <td>140</td>\n",
       "      <td>2021-05-04 20:39:52</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for Android</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>b'RT @CedawPT: Women journalists can be target...</td>\n",
       "      <td>1389681007369011209</td>\n",
       "      <td>148</td>\n",
       "      <td>2021-05-04 20:39:05</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for Android</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>b'Important study about how how construction w...</td>\n",
       "      <td>1389680772123086849</td>\n",
       "      <td>114</td>\n",
       "      <td>2021-05-04 20:38:09</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for Android</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3051</th>\n",
       "      <td>3051</td>\n",
       "      <td>b'RT @GBV_CHILE: Seccion #MTCero #GBV y @Carab...</td>\n",
       "      <td>1386885052034912260</td>\n",
       "      <td>140</td>\n",
       "      <td>2021-04-27 03:28:58</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>es</td>\n",
       "      <td>Twitter for Android</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3052</th>\n",
       "      <td>3052</td>\n",
       "      <td>b'Seccion #MTCero #GBV y @Carabdechile Com. #E...</td>\n",
       "      <td>1386882452837257218</td>\n",
       "      <td>297</td>\n",
       "      <td>2021-04-27 03:18:38</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>es</td>\n",
       "      <td>Twitter for Android</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3053</th>\n",
       "      <td>3053</td>\n",
       "      <td>b'RT @JackieRange: \"If you go into our factori...</td>\n",
       "      <td>1386871901834731521</td>\n",
       "      <td>140</td>\n",
       "      <td>2021-04-27 02:36:42</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3054</th>\n",
       "      <td>3054</td>\n",
       "      <td>b'RT @LesMichif: Only THREE days left to regis...</td>\n",
       "      <td>1386868002197426177</td>\n",
       "      <td>139</td>\n",
       "      <td>2021-04-27 02:21:13</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3055</th>\n",
       "      <td>3055</td>\n",
       "      <td>b'RT @MzeeBwanika: Man beats wife and breaks h...</td>\n",
       "      <td>1386867184031391744</td>\n",
       "      <td>139</td>\n",
       "      <td>2021-04-27 02:17:58</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3056 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Unnamed: 0                                             Tweets  \\\n",
       "0              0  b'Can\\xe2\\x80\\x99t wait for you kids to hear t...   \n",
       "1              1  b'RT @CedawPT: Women journalists can be target...   \n",
       "2              2  b'RT @mobkimarkus7: Save the Girl Child, suppo...   \n",
       "3              3  b'RT @CedawPT: Women journalists can be target...   \n",
       "4              4  b'Important study about how how construction w...   \n",
       "...          ...                                                ...   \n",
       "3051        3051  b'RT @GBV_CHILE: Seccion #MTCero #GBV y @Carab...   \n",
       "3052        3052  b'Seccion #MTCero #GBV y @Carabdechile Com. #E...   \n",
       "3053        3053  b'RT @JackieRange: \"If you go into our factori...   \n",
       "3054        3054  b'RT @LesMichif: Only THREE days left to regis...   \n",
       "3055        3055  b'RT @MzeeBwanika: Man beats wife and breaks h...   \n",
       "\n",
       "                       id  lens                 date place  coordinateS lang  \\\n",
       "0     1389683945143427076   136  2021-05-04 20:50:46   NaN          NaN   en   \n",
       "1     1389683664150319113   148  2021-05-04 20:49:39   NaN          NaN   en   \n",
       "2     1389681203838656516   140  2021-05-04 20:39:52   NaN          NaN   en   \n",
       "3     1389681007369011209   148  2021-05-04 20:39:05   NaN          NaN   en   \n",
       "4     1389680772123086849   114  2021-05-04 20:38:09   NaN          NaN   en   \n",
       "...                   ...   ...                  ...   ...          ...  ...   \n",
       "3051  1386885052034912260   140  2021-04-27 03:28:58   NaN          NaN   es   \n",
       "3052  1386882452837257218   297  2021-04-27 03:18:38   NaN          NaN   es   \n",
       "3053  1386871901834731521   140  2021-04-27 02:36:42   NaN          NaN   en   \n",
       "3054  1386868002197426177   139  2021-04-27 02:21:13   NaN          NaN   en   \n",
       "3055  1386867184031391744   139  2021-04-27 02:17:58   NaN          NaN   en   \n",
       "\n",
       "                   source  likes  retweets  \n",
       "0      Twitter for iPhone      0         0  \n",
       "1     Twitter for Android      0        18  \n",
       "2     Twitter for Android      0         5  \n",
       "3     Twitter for Android      0        18  \n",
       "4     Twitter for Android      0         0  \n",
       "...                   ...    ...       ...  \n",
       "3051  Twitter for Android      0         2  \n",
       "3052  Twitter for Android      2         2  \n",
       "3053   Twitter for iPhone      0         2  \n",
       "3054   Twitter for iPhone      0         5  \n",
       "3055   Twitter for iPhone      0         2  \n",
       "\n",
       "[3056 rows x 11 columns]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_GBV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Unnamed: 0        0\n",
       "Tweets            0\n",
       "id                0\n",
       "lens              0\n",
       "date              0\n",
       "place          3036\n",
       "coordinateS    3056\n",
       "lang              0\n",
       "source            0\n",
       "likes             0\n",
       "retweets          0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_GBV.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "en     2921\n",
       "es       57\n",
       "und      38\n",
       "de       29\n",
       "ne        3\n",
       "nl        2\n",
       "ar        1\n",
       "in        1\n",
       "it        1\n",
       "da        1\n",
       "el        1\n",
       "tl        1\n",
       "Name: lang, dtype: int64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_GBV['lang'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       b'Can\\xe2\\x80\\x99t wait for you kids to hear t...\n",
       "1       b'RT @CedawPT: Women journalists can be target...\n",
       "2       b'RT @mobkimarkus7: Save the Girl Child, suppo...\n",
       "3       b'RT @CedawPT: Women journalists can be target...\n",
       "4       b'Important study about how how construction w...\n",
       "                              ...                        \n",
       "3051    b'RT @GBV_CHILE: Seccion #MTCero #GBV y @Carab...\n",
       "3052    b'Seccion #MTCero #GBV y @Carabdechile Com. #E...\n",
       "3053    b'RT @JackieRange: \"If you go into our factori...\n",
       "3054    b'RT @LesMichif: Only THREE days left to regis...\n",
       "3055    b'RT @MzeeBwanika: Man beats wife and breaks h...\n",
       "Name: Tweets, Length: 3056, dtype: object"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = data_GBV['Tweets']\n",
    "d"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "dataCollection.ipynb",
   "provenance": []
  },
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}