Workshop-Feedback_Bigram_Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08ae87dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# code credit : https://towardsdatascience.com/text-analysis-basics-in-python-443282942ec5\n",
    "corpus = [\n",
    "\"Neighborhoods(single family homes) being turned into multi unit apartments\",\n",
    "\"Add more catalogs\",\n",
    "\"Working the system of data\",\n",
    "\"housing data\",\n",
    "\"entitlement information\",\n",
    "\"How to acquire specific data related to  homelessness and affordable housing.\", \n",
    "\"More hands on. In depth. Follow along\",\n",
    "\"Step by step in using data sets to build a geospatial map\",\n",
    "\"How to bring more data into the platforms. How to join data together to understand complex correlations.\", \n",
    "\"More specific examples of how the data can be used at the community level.\",\n",
    "\"How to contribute data to website. How to better read data. What type of information residents like.\",\n",
    "\"Grassroots and community data gathering like stories, photo narrative.\",\n",
    "\"Specify just one training for My LA 311 app. A few people in the room know. I will like to work more in this campaign at the NC level\",\n",
    "\"Actually do the reports(find the info wanted)\",\n",
    "\"Safety\",\n",
    "\"Clean lines\",\n",
    "\"Little more info on CIS dashboard\",\n",
    "\"Little more info on Geohub\",\n",
    "\"How to make pivot tables\", \n",
    "\"A road map with examples on how to use this data and apply it in day to day. Examples of how different people have used it successfully.\", \n",
    "\"Geohub. Also, it'd be neat do do this as a topic- specific services- for example \"\" Data for Land use.\"\" perhaps in partnership with the NC.\",\n",
    "\"Census vs NC Data. More time on change metrics.\",\n",
    "\"Not at this time.\",\n",
    "\"A workshop on the LA 311 app.\",\n",
    "\"Department of transportation data. Any other departments that are not inluded.\",\n",
    "\"How Council Districts use this data to targets certain pockets of the district.\",\n",
    "\"Budget data and how it intersects with other community data.\",\n",
    "\"City Service related.  When NCs advocate for improved intersections, get approval and then are held up due to no budget the process takes years and years to realize the improvement becoming a \"\"political\"\" football instead of a improvement for public safety.\",\n",
    "\"The disconnect between City Planning approvals and final approvals through LA Dept. of Building & Safety.\",\n",
    "\"Data on number of and types of complaints submitted through 311 as relates to LADBS such as construction without permit, signage, certificates of occupancy. Many of these are left as either no violation or there's a violation but no resolution.\",\n",
    "\"How diverse is my NC\",\n",
    "\"Mar Vista 90066 neighborhood\",\n",
    "\"economic information\",\n",
    "\"Mental health care resources, community health information and organizations, opportunities to take skill building classes\",\n",
    "\"Community health resources\",\n",
    "\"Median income, and I want to advocate for affordable housing.\",\n",
    "\"Property Ownership\",\n",
    "\"Communication/Homelessness\",\n",
    "\"I don’t know what type of data sets I would like to gain access to that’s one of the reasons for attending, for me.\",\n",
    "\"The issues I would like to focus on our: homelessness, poverty, Arts and healing, well-being.\",\n",
    "\"Public safey\",\n",
    "\"Reproductive Justice, Maternal and Infant Health, HIV\",\n",
    "\"Unknown\",\n",
    "\"Neighborhood Council\",\n",
    "\"Neighborhood Council\",\n",
    "\"Urban Forestry DEPT\",\n",
    "\"Sidewalk repair program\",\n",
    "\"Homeless families health and fitness small business community safety student loans more\",\n",
    "\"Requests for City services within each Neighborhood Council if possible\",\n",
    "\"Zimas, Census Map\",\n",
    "\"Demographic data, housing data, health data\",\n",
    "\"I would like access to any datasets that relate to education, community demographics, socioeconomic status, youth/community development, and recreational activities/spaces.\",\n",
    "\"Dont know\",\n",
    "\"Demographic data\",\n",
    "\"MyLA311; Building and Safety Permit Information; Crime Data from 2010 to Present\",\n",
    "\"College and retirement planning for families.\",\n",
    "\"College and Retirement planning for families.\",\n",
    "\"Unsafe drivers on Foothill blvd. Speeding and ignorant drivers regarding safety rules such as stopping at stop signs, knowing  who has right of way, double parking, unlawful u-turns to name a few.\",\n",
    "\"Sloppy work by street maintenance crews\",\n",
    "\"Safety\",\n",
    "\"Safety\",\n",
    "\"Community events,  points of contact\",\n",
    "\"Demographics of Sunland-Tujunga for arts advocacy.\",\n",
    "\"Housing\",\n",
    "\"Population demographics\",\n",
    "\"Crime, transportation, import/export, trade, government spending\",\n",
    "\"Not sure\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "336a9339",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = pd.DataFrame(corpus)\n",
    "df.columns = ['reviews']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "701fa1cf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting textblob\n",
      "  Downloading textblob-0.15.3-py2.py3-none-any.whl (636 kB)\n",
      "\u001b[K     |████████████████████████████████| 636 kB 4.1 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting nltk>=3.1\n",
      "  Downloading nltk-3.6.2-py3-none-any.whl (1.5 MB)\n",
      "\u001b[K     |████████████████████████████████| 1.5 MB 49.7 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting joblib\n",
      "  Downloading joblib-1.0.1-py3-none-any.whl (303 kB)\n",
      "\u001b[K     |████████████████████████████████| 303 kB 1.6 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: click in /opt/conda/lib/python3.8/site-packages (from nltk>=3.1->textblob) (7.1.2)\n",
      "Requirement already satisfied: tqdm in /opt/conda/lib/python3.8/site-packages (from nltk>=3.1->textblob) (4.60.0)\n",
      "Collecting regex\n",
      "  Downloading regex-2021.4.4-cp38-cp38-manylinux2014_x86_64.whl (733 kB)\n",
      "\u001b[K     |████████████████████████████████| 733 kB 1.6 MB/s eta 0:00:01:01\n",
      "\u001b[?25hInstalling collected packages: regex, joblib, nltk, textblob\n",
      "Successfully installed joblib-1.0.1 nltk-3.6.2 regex-2021.4.4 textblob-0.15.3\n"
     ]
    }
   ],
   "source": [
    "#!pip install textblob -c conda-forge\n",
    "!pip install -U textblob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "0803e0cf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package brown to /home/jovyan/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/brown.zip.\n",
      "[nltk_data] Downloading package punkt to /home/jovyan/nltk_data...\n",
      "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
      "[nltk_data] Downloading package wordnet to /home/jovyan/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/wordnet.zip.\n",
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     /home/jovyan/nltk_data...\n",
      "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n",
      "[nltk_data] Downloading package conll2000 to /home/jovyan/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/conll2000.zip.\n",
      "[nltk_data] Downloading package movie_reviews to\n",
      "[nltk_data]     /home/jovyan/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/movie_reviews.zip.\n",
      "Finished.\n"
     ]
    }
   ],
   "source": [
    "!python -m textblob.download_corpora"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "90711db7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                              reviews  polarity  subjective\n",
      "0   Neighborhoods(single family homes) being turne...      0.00    0.000000\n",
      "1                                   Add more catalogs      0.50    0.500000\n",
      "2                          Working the system of data      0.00    0.000000\n",
      "3                                        housing data      0.00    0.000000\n",
      "4                             entitlement information      0.00    0.000000\n",
      "..                                                ...       ...         ...\n",
      "62  Demographics of Sunland-Tujunga for arts advoc...      0.00    0.000000\n",
      "63                                            Housing      0.00    0.000000\n",
      "64                            Population demographics      0.00    0.000000\n",
      "65  Crime, transportation, import/export, trade, g...      0.00    0.000000\n",
      "66                                           Not sure     -0.25    0.888889\n",
      "\n",
      "[67 rows x 3 columns]\n"
     ]
    }
   ],
   "source": [
    "from textblob import TextBlob\n",
    "df['polarity'] = df['reviews'].apply(lambda x: TextBlob(x).polarity)\n",
    "df['subjective'] = df['reviews'].apply(lambda x: TextBlob(x).subjectivity)\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "1353128e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to /home/jovyan/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "a98abafe",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords\n",
    "stoplist = stopwords.words('english') + ['would'] + ['like']\n",
    "# maybe need to add more "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "5a2c94cc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting scikit-learn\n",
      "  Downloading scikit_learn-0.24.2-cp38-cp38-manylinux2010_x86_64.whl (24.9 MB)\n",
      "\u001b[K     |████████████████████████████████| 24.9 MB 46 kB/s s eta 0:00:01   |████                            | 3.0 MB 3.1 MB/s eta 0:00:07     |██████                          | 4.7 MB 3.1 MB/s eta 0:00:07 MB 66.6 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: joblib>=0.11 in /opt/conda/lib/python3.8/site-packages (from scikit-learn) (1.0.1)\n",
      "Collecting threadpoolctl>=2.0.0\n",
      "  Downloading threadpoolctl-2.1.0-py3-none-any.whl (12 kB)\n",
      "Collecting scipy>=0.19.1\n",
      "  Using cached scipy-1.6.3-cp38-cp38-manylinux1_x86_64.whl (27.2 MB)\n",
      "Requirement already satisfied: numpy>=1.13.3 in /opt/conda/lib/python3.8/site-packages (from scikit-learn) (1.20.2)\n",
      "Installing collected packages: threadpoolctl, scipy, scikit-learn\n",
      "Successfully installed scikit-learn-0.24.2 scipy-1.6.3 threadpoolctl-2.1.0\n"
     ]
    }
   ],
   "source": [
    "!pip install -U scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "b4a720a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "c_vec = CountVectorizer(stop_words=stoplist, ngram_range=(1,3))\n",
    "# matrix of ngrams\n",
    "ngrams = c_vec.fit_transform(df['reviews'])\n",
    "# count frequency of ngrams\n",
    "count_values = ngrams.toarray().sum(axis=0)\n",
    "# list of ngrams\n",
    "vocab = c_vec.vocabulary_\n",
    "df_ngram = pd.DataFrame(sorted([(count_values[i],k) for k,i in vocab.items()], reverse=True)\n",
    "            ).rename(columns={0: 'frequency', 1:'bigram/trigram'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "6fdc34fc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>frequency</th>\n",
       "      <th>bigram/trigram</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>24</td>\n",
       "      <td>data</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9</td>\n",
       "      <td>community</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>safety</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>health</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>information</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>housing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>4</td>\n",
       "      <td>neighborhood</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>4</td>\n",
       "      <td>nc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>4</td>\n",
       "      <td>council</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>3</td>\n",
       "      <td>use</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>3</td>\n",
       "      <td>specific</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>3</td>\n",
       "      <td>planning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>3</td>\n",
       "      <td>neighborhood council</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>3</td>\n",
       "      <td>map</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>3</td>\n",
       "      <td>la</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>3</td>\n",
       "      <td>know</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>3</td>\n",
       "      <td>info</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>3</td>\n",
       "      <td>homelessness</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>3</td>\n",
       "      <td>families</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>3</td>\n",
       "      <td>examples</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>3</td>\n",
       "      <td>demographics</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>3</td>\n",
       "      <td>city</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>3</td>\n",
       "      <td>building</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>3</td>\n",
       "      <td>311</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>2</td>\n",
       "      <td>years</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>2</td>\n",
       "      <td>work</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>2</td>\n",
       "      <td>violation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>2</td>\n",
       "      <td>used</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>2</td>\n",
       "      <td>use data</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>2</td>\n",
       "      <td>type</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>2</td>\n",
       "      <td>transportation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>2</td>\n",
       "      <td>time</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>2</td>\n",
       "      <td>step</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>2</td>\n",
       "      <td>sets</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>2</td>\n",
       "      <td>services</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>2</td>\n",
       "      <td>retirement planning families</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>2</td>\n",
       "      <td>retirement planning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>2</td>\n",
       "      <td>retirement</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>2</td>\n",
       "      <td>resources</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>2</td>\n",
       "      <td>related</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>2</td>\n",
       "      <td>public</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>2</td>\n",
       "      <td>planning families</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>2</td>\n",
       "      <td>permit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>2</td>\n",
       "      <td>people</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>2</td>\n",
       "      <td>one</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>2</td>\n",
       "      <td>little info</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>2</td>\n",
       "      <td>little</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>2</td>\n",
       "      <td>level</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>2</td>\n",
       "      <td>la 311 app</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>2</td>\n",
       "      <td>la 311</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    frequency                bigram/trigram\n",
       "0          24                          data\n",
       "1           9                     community\n",
       "2           8                        safety\n",
       "3           6                        health\n",
       "4           5                   information\n",
       "5           5                       housing\n",
       "6           4                  neighborhood\n",
       "7           4                            nc\n",
       "8           4                       council\n",
       "9           3                           use\n",
       "10          3                      specific\n",
       "11          3                      planning\n",
       "12          3          neighborhood council\n",
       "13          3                           map\n",
       "14          3                            la\n",
       "15          3                          know\n",
       "16          3                          info\n",
       "17          3                  homelessness\n",
       "18          3                      families\n",
       "19          3                      examples\n",
       "20          3                  demographics\n",
       "21          3                          city\n",
       "22          3                      building\n",
       "23          3                           311\n",
       "24          2                         years\n",
       "25          2                          work\n",
       "26          2                     violation\n",
       "27          2                          used\n",
       "28          2                      use data\n",
       "29          2                          type\n",
       "30          2                transportation\n",
       "31          2                          time\n",
       "32          2                          step\n",
       "33          2                          sets\n",
       "34          2                      services\n",
       "35          2  retirement planning families\n",
       "36          2           retirement planning\n",
       "37          2                    retirement\n",
       "38          2                     resources\n",
       "39          2                       related\n",
       "40          2                        public\n",
       "41          2             planning families\n",
       "42          2                        permit\n",
       "43          2                        people\n",
       "44          2                           one\n",
       "45          2                   little info\n",
       "46          2                        little\n",
       "47          2                         level\n",
       "48          2                    la 311 app\n",
       "49          2                        la 311"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ngram.head(50)\n",
    "# standardized data neighborhood council = nc \n",
    "# remove stopwords - e.g. data \n",
    "# seperate unigrams from bigrams and trigrams "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1780c84c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}