orisonation/prayer_writing_helper.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 📊 Prayer Writing Assistant\n",
    "\n",
    "\n",
    "I use this notebook when I am writing prayers.  Here I analyze a corpus of \"special intentions\" (statements of need, from Twitter) target common/paradigmatic needs so that I can compose \"high-impact\" prayers that increase total coverage.\n",
    "- total \"coverage\" of my prayers (how many special intentions are within a certain minimum semantic distance of its nearest prayers that I have written).\n",
    "- cluster special intentions to target frequently-occurring ones\n",
    "- looking at frequent token sequences in special attentions, also to target frequently-occurring ones"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "****Loading sbert model\n",
      "****Loaded\n",
      "****Loading word2vec model: shrunkenvectors_200000.bin\n",
      "****Loaded\n"
     ]
    }
   ],
   "source": [
    "import pray"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'need': 'to get some food', 'score': 0.5887243747711182}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pray.get_closest_match_sbert(\"I need some bread.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'need': 'a hug', 'score': 0.9902426600456238}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pray.get_closest_match_sbert(\"I need a hug.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Special Intentions\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "special_intention_file = \"special_intentions_large.json\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I need a hat'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pray.homogenize(\"i just need a hat\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['I need you',\n",
       " 'I need to quit playing and get to on top of this lol it could really be something',\n",
       " 'I need to cry for three good days.']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "with open(special_intention_file,'r') as f:\n",
    "    special_intentions = json.load(f)\n",
    "    \n",
    "if type(special_intentions[0])==list:\n",
    "    special_intentions = [si for userid, si in special_intentions]\n",
    "\n",
    "special_intentions_for_processing = [pray.remove_i_need(si).lower() for si in special_intentions]\n",
    "special_intentions = [pray.homogenize(si) for si in special_intentions]\n",
    "special_intentions = [si for si in special_intentions if len(si)>0]\n",
    "special_intentions[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['you',\n",
       " 'to quit playing and get to on top of this lol it could really be something',\n",
       " 'to cry for three good days.']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "special_intentions_for_processing[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6886"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import random\n",
    "random.seed(1)\n",
    "\n",
    "random.shuffle(special_intentions)\n",
    "len(special_intentions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['I need my own spa, massage wud b luvvv',\n",
       " 'I need friends fuuuuuuuckkkkkkkkkk',\n",
       " 'I need to watch “crooklyn”',\n",
       " 'I need to deep dive into the Saturn in Aquarius bc things been quite different for me this week.',\n",
       " 'I need to find me a good pair of gloves!',\n",
       " 'I need y’all to play temptation by Santana it is sending me with the pop smoke vibe fr 😂😂😭😭😭',\n",
       " 'I need a mid-tempo playlist for nights like this.',\n",
       " 'I need to focus on @FocusFactory1  more.',\n",
       " 'I need a girl who into cars like me',\n",
       " 'I need a drink or 15']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random.sample(special_intentions,10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyze Success of Current Patterns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'need': 'a friend',\n",
       " 'score': 0.9883623719215393,\n",
       " 'banned': False,\n",
       " 'prayers': {0: 'May you find a pure bondsman<br>behold---this bondsman pure as a frankincense',\n",
       "  1: 'Yea, for you are leprous of bidding to yourself',\n",
       "  2: 'May it be so, though it is difficult to give bondsmen<br>as the ground shall give her increase, and the Heavens shall give their dew ; and i will cause the remnant of this people to possess all these things.',\n",
       "  3: 'You go to the jubile wearing your smile and woolen<br>as which way went the Spirit of the Lord from me to speak unto thee',\n",
       "  4: 'Who will speak your surname and visit your father?<br>as for your hands are defiled with blood, and your fingers with iniquity ; your lips have spoken lies',\n",
       "  5: 'May you not become terrible before them, filling them with provoke<br> filling with food and gladness. (Selah.)',\n",
       "  6: 'Await the weak companionship of the bondswoman<br>behold---this companionship weak as hands',\n",
       "  7: ',a bondswoman to call bondswoman<br>as and Moses called Oshea the son of Nun Jehoshua.'},\n",
       " 'template': {0: 'May you find a {JJ.true} {NN.friend}',\n",
       "  1: 'Yea, for you are {JJ.sick.tired} of {VBG.talking} to yourself',\n",
       "  2: 'May it be so, though it is {JJ.difficult} to {VB.make} {NNS.friends}',\n",
       "  3: 'You go to the {NN.city.party.bar} wearing your {NN.smile} and {JJ.clothes}',\n",
       "  4: 'Who will speak your {NN.name} and visit your {NN.house.mother.father.party}?',\n",
       "  5: 'May you not become {JJ.strange} before them, filling them with {NN.disgust}',\n",
       "  6: 'Await the {JJ.soft} {NN.companionship} of the {NN.tribe.friends.colleagues}',\n",
       "  7: ',a {NN.friend} to call {NN.friend}'}}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pray.pray(\"I need a friend.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 58s, sys: 1.69 s, total: 2min\n",
      "Wall time: 2min 9s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "scores_for_all = [pray.pray(s)['score'] for s in special_intentions[:300]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Threshold` (below) should be the same as value of `threshold` argument in `pray_with_simplification` in `pray.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "threshold = .7\n",
    "high_enough_scores = [s for s in scores_for_all if s>=threshold]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Percentage above threshold. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.15333333333333332"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(high_enough_scores)/len(scores_for_all)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Average."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5321975326538086"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(scores_for_all)/len(scores_for_all)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Order According to Similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = special_intentions[:1000]\n",
    "distance_threshold = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sample_and_judge_cluster(cluster,n=7):\n",
    "    if len(cluster)>n:\n",
    "        cluster = random.sample(cluster,n)\n",
    "    is_it_valid = [True if pray.pray(si)['score']>=threshold else False for si in cluster]\n",
    "    symbols = [\"✅\" if evaluation==True else \"❌\" for evaluation in is_it_valid]\n",
    "    return [\"%s:%s\" % (sym,si) for sym,si in zip(symbols,cluster)]\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cluster  17\n",
      "⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛\n",
      "❌:I need a GPU.\n",
      "❌:I need the sensationalist mask police to look at Calipari\n",
      "❌:I need to give samurai champloo a rewatch\n",
      "❌:I need a freaking clip of skz dionysus cover 😭😭\n",
      "❌:I need to watch the big lebowski for power\n",
      "❌:I need someone to make a fanart of magikarp kaido now\n",
      "❌:I need 3 DOLLARS to renew my Netflix account anyone wanna Apple Pay 😭😭\n",
      "\n",
      "Cluster  56\n",
      "⬛⬛⬛⬛⬛\n",
      "❌:I need a full body massage ☹️\n",
      "❌:I need a good massage rn 💅🏽\n",
      "❌:I need a massage or spa day\n",
      "❌:I need a massage\n",
      "❌:I need a massage ASAP, my body is done\n",
      "❌:I need a massage\n",
      "❌:I need a massage and a nap\n",
      "\n",
      "Cluster  16\n",
      "⬛⬛⬛⬛⬛⬛⬛\n",
      "❌:I need ONE follower\n",
      "please please it's the funny number\n",
      "❌:I need 10/13 questions right to bump my grade up by some points pls pls\n",
      "❌:I need like 10 pieces of beef bacon 🥴\n",
      "❌:I need a quick lick 😭\n",
      "❌:I need a shell\n",
      "❌:I need a hassle free bip.\n",
      "❌:I need the nocta sweatsuit\n",
      "\n",
      "Cluster  13\n",
      "⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛\n",
      "❌:I need Kim Seonho in Running Man 😫\n",
      "❌:I need somebody 💯 in my corner\n",
      "❌:I need attention 🥺\n",
      "❌:I need new vibes 🗣\n",
      "❌:I need to wake up this early morning😫\n",
      "❌:I need a me day 💆🏽‍♀️\n",
      "❌:I need that one more thing🤞🏼❤️\n",
      "\n",
      "Cluster  52\n",
      "⬛⬛⬛⬛⬛⬛\n",
      "❌:I need Janette to win strictly tonight\n",
      "❌:I need to make plans for Faye while I’m playing\n",
      "❌:I need a daughter already isaac is tired of letting me do his hair 😭\n",
      "❌:I need this lady to confirm this damn appointment\n",
      "❌:I need to teach myself how to braid 😩 imma start watching tutorials lol\n",
      "❌:I need braids for next week 😔\n",
      "❌:I need some help from auk, ack, gust &amp; aum students/graduates\n",
      "\n",
      "Cluster  33\n",
      "⬛⬛⬛\n",
      "❌:I need 4 more pages\n",
      "❌:I need the Raiders to win by 3 for very specific reasons.\n",
      "❌:I need a 3rd and 4th TV for today\n",
      "❌:I need A SEASON 3 OF GREAT PRETENDERRRRR\n",
      "❌:I need 3 more peoples help 🥺🥺\n",
      "❌:I need like $3.5 million right quick 🤷🏽‍♂️\n",
      "❌:I need a 3 in them 13s😒\n",
      "\n",
      "Cluster  42\n",
      "⬛⬛⬛⬛⬛\n",
      "❌:I need to clean my car 😭\n",
      "✅:I need to clean my oven out today\n",
      "❌:I need to get my car cleaned out like fr\n",
      "❌:I need to finish the murals on my wall\n",
      "❌:I need to get my windows tinted on my car\n",
      "✅:I need to cleanse my house today\n",
      "❌:I need to clean my car inside &amp; out.\n",
      "\n",
      "Cluster  11\n",
      "⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛\n",
      "✅:I need a flipping vaccine\n",
      "✅:I need My Lashes Done Today\n",
      "❌:I need distractions\n",
      "❌:I need help lmaoo\n",
      "❌:I need the card hahah\n",
      "❌:I need a rocks pirate flashback man\n",
      "❌:I need a nice wig\n",
      "\n",
      "Cluster  15\n",
      "⬛⬛⬛⬛⬛⬛⬛⬛\n",
      "❌:I need you\n",
      "❌:I need TO BE TOUCHED\n",
      "❌:I need to get that touched up lol\n",
      "❌:I need somebody that’s attentive.\n",
      "❌:I need someone to be here and help me\n",
      "❌:I need a tiny little things buddy\n",
      "❌:I need someone :(\n",
      "\n",
      "Cluster  7\n",
      "⬛⬛⬛⬛⬛⬛⬛⬛\n",
      "❌:I need a big load\n",
      "❌:I need $100 🥺🙏\n",
      "❌:I need that job pls if you know any rich lady that needs one I’m available for the job pls 🙏 thanks 😊\n",
      "❌:I need a laptop and a capital of $100 to trade forex 😇🤑\n",
      "❌:I need to free my mind😔\n",
      "❌:I need nordstrom to just mark my package as lost and give me my mf money back\n",
      "❌:I need drug dealers kind of money.\n",
      "\n",
      "Cluster  35\n",
      "⬛⬛⬛⬛⬛⬛\n",
      "❌:I need multiple vehicles\n",
      "❌:I need to be more active\n",
      "❌:I need a few more thangzzz\n",
      "✅:I need to be better.\n",
      "✅:I need to change my attitude\n",
      "❌:I need more of this content\n",
      "✅:I need to do better\n",
      "\n",
      "Cluster  24\n",
      "⬛⬛⬛⬛\n",
      "❌:I need someone’s son to make my legs shake\n",
      "❌:I need someone to be rubbing my back while I'm sleeping\n",
      "❌:I need ice cream.\n",
      "❌:I need a nice feet rub 😭 I’m dying my feet hurt\n",
      "❌:I need to shake this cold 🥴\n",
      "❌:I need to rub my feet up on sumn 😭\n",
      "❌:I need my back rubbed at this moment\n",
      "\n",
      "Cluster  30\n",
      "⬛⬛⬛⬛⬛⬛⬛⬛⬛\n",
      "✅:I need a nap 🥴\n",
      "✅:I need /want a good night sleep 😔😓\n",
      "✅:I need to ducking sleep\n",
      "✅:I need a nap yo\n",
      "✅:I need a nap 😩😩\n",
      "✅:I need to sleep\n",
      "✅:I need a nap and I’m not feeling it today\n",
      "\n",
      "Cluster  19\n",
      "⬛⬛⬛⬛⬛⬛\n",
      "❌:I need 1 Solid ass friend 🤝\n",
      "✅:I need my best friend\n",
      "✅:I need more Ute fans to follow/followers.\n",
      "❌:I need friends that can photograph 💫brown skin💫\n",
      "❌:I need a seokjinnie friend :(\n",
      "❌:I need to follow more girl group stans where are y'all\n",
      "✅:I need friends\n",
      "\n",
      "Cluster  47\n",
      "⬛⬛⬛⬛\n",
      "❌:I need to see new art\n",
      "❌:I need a new carrd\n",
      "❌:I need to get some plans for New Years\n",
      "❌:I need a new plant\n",
      "❌:I need a new life .\n",
      "❌:I need new everything.\n",
      "❌:I need new people in my life\n",
      "\n",
      "Cluster  8\n",
      "⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛\n",
      "❌:I need a FaceTime call right now\n",
      "❌:I need to get myself outside regularly again\n",
      "❌:I need this shift to go quickly and smoothly.\n",
      "❌:I need time to figure this out\n",
      "❌:I need to move on HAHAHAHHAHHA\n",
      "❌:I need to run errands.\n",
      "❌:I need somebody to come outside with me\n",
      "\n",
      "Cluster  32\n",
      "⬛⬛⬛\n",
      "❌:I need a 1 year long vacation 🤝\n",
      "✅:I need a vacation\n",
      "❌:I need $8437282828282 and a vacation\n",
      "❌:I need a holiday :(\n",
      "❌:I need a holiday right now.\n",
      "✅:I need a vacation\n",
      "❌:I need to celebrate my birthday wtf\n",
      "\n",
      "Cluster  36\n",
      "⬛⬛⬛⬛⬛⬛⬛⬛\n",
      "❌:I need to redo my bdsm test, it’s been a minute!\n",
      "❌:I need stocking stuffers and every store is at max capacity 🤝 going home !\n",
      "❌:I need to dive into Club House 😩 been busy af\n",
      "❌:I need a glow up ASAP, I’m withering away here with 30 mins sunlight a day\n",
      "❌:I need to pick my second major soon and convince myself that somehow it’s not meant to be english\n",
      "❌:I need ff7r part 2 to come out bro that game was great\n",
      "❌:I need NW to score and go for a 2 point and get that idgaf if they lose smh\n",
      "\n",
      "Cluster  23\n",
      "⬛⬛⬛⬛⬛⬛\n",
      "❌:I need to make more music soon, and release some unfinished stuff!\n",
      "❌:I need to pick up a new character man lol\n",
      "❌:I need a new foundation guys what’s the best foundation you’ve ever bought\n",
      "❌:I need an assistant lmfao\n",
      "❌:I need to make an album only for doy\n",
      "❌:I need another album by an producer like NAHWC by Metro\n",
      "❌:I need a male bestie\n",
      "\n",
      "Cluster  28\n",
      "⬛⬛⬛⬛⬛\n",
      "❌:I need someone to hear my story\n",
      "✅:I need someone to hear me\n",
      "✅:I need someone to understand me\n",
      "❌:I need to be listening them more often\n",
      "✅:I need to have a sit down and talk with somebody\n",
      "❌:I need somebody to vent to seriously\n",
      "❌:I need answers now.\n",
      "\n",
      "Cluster  40\n",
      "⬛⬛⬛\n",
      "✅:I need a drink.\n",
      "❌:I need a drink or two or ten\n",
      "❌:I need a strong ass drink\n",
      "✅:I need a drink.\n",
      "❌:I need an energy drink\n",
      "❌:I need a drink rn\n",
      "✅:I need a drink\n",
      "\n",
      "Cluster  21\n",
      "⬛⬛⬛⬛⬛\n",
      "❌:I need someone to talk to rn\n",
      "❌:I need to watch Jujutsu Kaisen\n",
      "❌:I need a half a million rn ntg much 🤣😳\n",
      "❌:I need more pictures of kairi\n",
      "❌:I need more of vocalist Hobi 😍\n",
      "❌:I need some neck rn\n",
      "❌:I need to stan rn\n",
      "\n",
      "Cluster  27\n",
      "⬛⬛⬛⬛⬛⬛⬛⬛\n",
      "❌:I need to do proper schedule for watching all ads in a day.\n",
      "❌:I need a superhero to help me\n",
      "❌:I need to update my carrd a whole@lot\n",
      "❌:I need williams when will someone leak tjem for me\n",
      "❌:I need to update me carrd\n",
      "❌:I need to update my pinned\n",
      "❌:I need more parx moots\n",
      "\n",
      "Cluster  43\n",
      "⬛⬛⬛⬛⬛⬛\n",
      "✅:I need to learn.\n",
      "❌:I need a gym membership\n",
      "❌:I need To Make A Valid Effort To Make It To The Gym\n",
      "❌:I need to sign up for someones make up classes.\n",
      "❌:I need a cable management class\n",
      "❌:I need to master my contouring skills, someone give me tips 😭\n",
      "❌:I need to learn Spanish\n",
      "\n",
      "Cluster  29\n",
      "⬛⬛⬛\n",
      "❌:I need some good rest.\n",
      "❌:I need some time to try and relax for now at least\n",
      "✅:I need the time to try and be calmer and happier at least\n",
      "❌:I need comfort ok\n",
      "✅:I need to calm down now\n",
      "❌:I need a day of relaxation\n",
      "✅:I need to calm down.\n",
      "\n",
      "Cluster  39\n",
      "⬛⬛⬛⬛⬛⬛⬛⬛\n",
      "❌:I need to just disappear\n",
      "✅:I need to stop buying things fgs 😩😩\n",
      "❌:I need to stop saying “Hey lol”😂\n",
      "❌:I need to stop addressing people as sir\n",
      "❌:I need to stop complaining I’m sorry dnsbskdnsn\n",
      "❌:I need everything to stop\n",
      "❌:I need to stop ordering clothes 😩\n",
      "\n",
      "Cluster  3\n",
      "⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛\n",
      "✅:I need to stop playing and get a camera\n",
      "❌:I need to learn to drink tea without eating something with it.\n",
      "❌:I need the lord to release my mates from the shackles that is tequila\n",
      "❌:I need to get on twitter more often.\n",
      "❌:I need one more chaotic haze arcarum PLEASE I'M BEGGING YOU I'M SICK OF GRINDING\n",
      "❌:I need to start using twitter more instead of when I'm feeling down lmao\n",
      "❌:I need for my child to stop waking for the day at 3:30 😅 plz plz plz\n",
      "\n",
      "Cluster  18\n",
      "⬛⬛⬛⬛⬛⬛⬛⬛\n",
      "❌:I need them 1s tho been looking for a suitable pair\n",
      "❌:I need to wear my Ohio state hoodie today.\n",
      "❌:I need /want another degree under my belt.. 🤷🏽‍♀️\n",
      "❌:I need to go get matching pajamas for me &amp; my kids today\n",
      "❌:I need a trip to Sephora or ulta and some birria\n",
      "❌:I need 2 more outfits\n",
      "❌:I need to get me &amp; Kody some shoes\n",
      "\n",
      "Cluster  9\n",
      "⬛⬛⬛⬛⬛⬛⬛⬛\n",
      "❌:I need hg :(\n",
      "❌:I need Rock n’ Jenny’s 🥪\n",
      "❌:I need vitamin M. 🥺\n",
      "❌:I need to stan nct\n",
      "❌:I need to be sla- nvm\n",
      "❌:I need a B tbh\n",
      "❌:I need to do smtg about it😂\n",
      "\n",
      "Cluster  12\n",
      "⬛⬛⬛⬛\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "❌:I need to be alone for a while untill \n",
      "i feel myself again\n",
      "❌:I need some time to myself\n",
      "❌:I need to get my own place ASAP\n",
      "❌:I need to focus on myself !\n",
      "❌:I need time for myself this time\n",
      "❌:I need to take care of myself.\n",
      "❌:I need somebody to call my own\n",
      "\n",
      "Cluster  31\n",
      "⬛⬛⬛⬛\n",
      "❌:I need an oomf thatll defend me like a lover\n",
      "❌:I need a better cat camera\n",
      "❌:I need me a lil girlfriend ✨\n",
      "❌:I need an alice in borderland layout i’m obsessed\n",
      "❌:I need a super cute cat, that's it.\n",
      "❌:I need a lil love n affection\n",
      "❌:I need a lil romance in my life\n",
      "\n",
      "Cluster  5\n",
      "⬛⬛⬛⬛⬛⬛⬛⬛\n",
      "❌:I need a everfresh cranberry juice and some woods\n",
      "❌:I need shots of tequillaaaa ok thanks\n",
      "❌:I need a wine night with the girls ☹️\n",
      "❌:I need a big ass cup of wine thanks\n",
      "❌:I need to make some ginger &amp; lemon tea, that’ll put me back together.\n",
      "❌:I need stuffed &amp; regular salmon and chicken, scallops, lamb chops &amp; broccoli &amp; broccolini too\n",
      "❌:I need to cop and no plug wanna be up\n",
      "\n",
      "Cluster  46\n",
      "⬛⬛⬛⬛\n",
      "✅:I need a break from everything\n",
      "✅:I need a break..\n",
      "✅:I need a break.\n",
      "✅:I need a break from everything for.\n",
      "✅:I need a break\n",
      "❌:I need a break dawg😓\n",
      "✅:I need a break from life\n",
      "\n",
      "Cluster  51\n",
      "⬛⬛⬛⬛⬛\n",
      "❌:I need to be flewed out nowww\n",
      "❌:I need bundles and a closure\n",
      "❌:I need to start taking care of my body more\n",
      "❌:I need to get out of the palamunin era\n",
      "❌:I need a release.\n",
      "❌:I need to jerk off\n",
      "❌:I need my body to get rolled out.\n",
      "\n",
      "Cluster  1\n",
      "⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛\n",
      "❌:I need MY 24 HOURS OF STRAIGHT SLEEP\n",
      "❌:I need to fix my sleep schedule\n",
      "❌:I need to sleep for 40 hours.\n",
      "❌:I need to stop falling asleep with my glasses on LOL\n",
      "❌:I need to sleep for 3 days straight.\n",
      "❌:I need a nap before this party tn , can’t wait 🥳\n",
      "❌:I need an hour nap at least 😩\n",
      "\n",
      "Cluster  44\n",
      "⬛⬛⬛⬛\n",
      "✅:I need to get my hair done 😩\n",
      "❌:I need 3tao for my hair and nails😭😭😭\n",
      "✅:I need my hair and makeup done\n",
      "❌:I need to wash my hair\n",
      "✅:I need to do my hair\n",
      "❌:I need to hand my hair done every 4-5 days 😩😂\n",
      "✅:I need to cut my hair\n",
      "\n",
      "Cluster  10\n",
      "⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛\n",
      "❌:I need @TheTraeYoung to convince @JHarden13  to come play for @ATLHawks\n",
      "❌:I need to watch bangbangcon again soon 🥺\n",
      "❌:I need to watch this song live again.\n",
      "❌:I need his feature scout 5*, watarus feature 5*, suous feature 5*, &amp; soumas feature 5*,,,,,,,\n",
      "❌:I need all this bs wrapped up by July tbh\n",
      "❌:I need a new show to watch while on my 2 week vacay 🤔\n",
      "❌:I need somebody for these type of nights\n",
      "\n",
      "Cluster  48\n",
      "⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛⬛\n",
      "✅:I need a hug /:\n",
      "✅:I need a hug rn\n",
      "✅:I need a big hug rn\n",
      "✅:I need a hug\n",
      "✅:I need a tall person to just pick me up and hug me\n",
      "❌:I need a 10 minute long hug\n",
      "✅:I need a hug.\n",
      "\n",
      "Cluster  6\n",
      "⬛⬛⬛⬛⬛\n",
      "❌:I need to stay off YouTube\n",
      "❌:I need a couple stacks nothing to major\n",
      "❌:I need a couple a days doing nothing!\n",
      "❌:I need a mere, lousy, underrated, understated 20k.\n",
      "❌:I need one normal thing in my life please\n",
      "❌:I need affection, not a kabogoh\n",
      "❌:I need one loyal person to exist and i’m quitting social media for good\n",
      "\n",
      "Cluster  14\n",
      "⬛⬛⬛⬛\n",
      "✅:I need kiss .\n",
      "❌:I need some good luck right now 🧿🧿🧿🧿🧿🧿🧿🧿🧿🧿\n",
      "❌:I need a warm hug and some cuddling though 😩😩\n",
      "✅:I need affection man can somebody HUG AND KISS ME\n",
      "✅:I need to learn how to love myself\n",
      "❌:I need some loving 🥺\n",
      "❌:I need the warmth of a woman’s touch\n",
      "\n",
      "Cluster  2\n",
      "⬛⬛⬛⬛⬛⬛⬛⬛\n",
      "❌:I need to get a book mark.\n",
      "❌:I need to start reading actual books again instead of just fic jfjdjdhs\n",
      "❌:I need a pencil sharpener\n",
      "❌:I need a sign\n",
      "❌:I need to go through my conclusion and the draft for my bachelor thesis is done\n",
      "❌:I need to make my edit\n",
      "❌:I need a new tattoo\n",
      "\n",
      "Cluster  26\n",
      "⬛⬛⬛⬛⬛⬛⬛\n",
      "❌:I need that D, \n",
      "\n",
      "Ctrl+D\n",
      "❌:I need to get into astro\n",
      "❌:I need a target run\n",
      "❌:I need an Ekeler TD come on\n",
      "❌:I need gta 6 to come out\n",
      "❌:I need a Rolls Royce\n",
      "❌:I need to find a piano\n",
      "\n",
      "Cluster  20\n",
      "⬛⬛⬛⬛⬛⬛\n",
      "❌:I need y’all to answer my poll 😭🤣\n",
      "❌:I need bighit fam to do collabs with other idols 🙏\n",
      "❌:I need the help of everyone to do the best for humans everywhere.\n",
      "❌:I need a girlfriend.. She can feed me and do other things..\n",
      "❌:I need an all day studio session with a few dope people to create.\n",
      "❌:I need everybody to stand on what they say🥱 dassit\n",
      "❌:I need dragon, class and other taeyong’s unreleased songs to be released for my mental and physical health🙂\n",
      "\n",
      "Cluster  25\n",
      "⬛⬛⬛⬛⬛⬛\n",
      "❌:I need some alicia keys moots where are y’all\n",
      "❌:I need an tango ice blast\n",
      "❌:I need a stizzy plug in Houston ASAP\n",
      "❌:I need to catch up huhu mas lalong dumami twice contents :((\n",
      "❌:I need @twelvethirtyfo to pour me a shot up of that casamigo’s 🙃\n",
      "❌:I need sunghoon to skate to black swan orchestral ver\n",
      "❌:I need to say this @thebiscolumbo and @Dipolygabana look alike ehnnn\n",
      "\n",
      "Cluster  55\n",
      "⬛⬛\n",
      "✅:I need a miracle.\n",
      "✅:I need a miracle\n",
      "✅:I need a miracle\n",
      "✅:I need a blessing\n",
      "✅:I need some blessings to come my way.\n",
      "\n",
      "Cluster  53\n",
      "⬛⬛⬛⬛⬛⬛\n",
      "✅:I need to cry why can’t anything make me cry\n",
      "❌:I need to shut up\n",
      "❌:I need to stop being sick on public transport\n",
      "❌:I need to stop being awkward\n",
      "❌:I need to know the essence behind this \"moaning\" thing on CH!\n",
      "❌:I need to work on my grumpiness 😫\n",
      "✅:I need to cry it out at this point\n",
      "\n",
      "Cluster  38\n",
      "⬛⬛⬛⬛⬛⬛\n",
      "❌:I need to take gift wrapping classes\n",
      "❌:I need an @GavinDeGraw Christmas album 🎄\n",
      "❌:I need to get some wrapping paper\n",
      "❌:I need a new back for Christmas.\n",
      "❌:I need gift ideas for a long distance relationship fast\n",
      "❌:I need a really short chain for my butterfly charm if anyone wanna get me something for christmas lmao\n",
      "❌:I need a gift card to go to the metaphysical shop as a winter solstice gift 🥺\n",
      "\n",
      "Cluster  22\n",
      "⬛⬛⬛⬛⬛⬛\n",
      "❌:I need to go thrift shopping\n",
      "❌:I need to go Portmore mall.\n",
      "❌:I need to finish my Christmas shopping 😅😅\n",
      "❌:I need to do A LOT of shopping for Christmas Day dinner 🤦🏽‍♀️ knowing me I’ll leave it all until 24th\n",
      "❌:I need to get some work done, and figure out my Christmas movie viewing schedule.\n",
      "❌:I need to go to walmart\n",
      "❌:I need to go plant shopping soon\n",
      "\n",
      "Cluster  45\n",
      "⬛⬛⬛\n",
      "❌:I need to make another Holdfast video\n",
      "❌:I need some good shows to watch on Netflix\n",
      "❌:I need some good ass weed and a good ass movie to watch\n",
      "❌:I need Netflix recommendations on this lovely day off 😊\n",
      "❌:I need a new Netflix series to watch 😕\n",
      "✅:I need me a good movie to watch\n",
      "✅:I need another movie to watch .\n",
      "\n",
      "Cluster  34\n",
      "⬛⬛⬛\n",
      "✅:I need a sugar daddy through these hard times lmaoo\n",
      "✅:I need a sugar daddy 🦦\n",
      "✅:I need a sugar daddy omg 😩\n",
      "❌:I need a sugar Daddy to pay my bills this month  😪😪\n",
      "✅:I need a sugar daddy\n",
      "✅:I need me a sugar daddy\n",
      "✅:I need a sugar daddy\n",
      "\n",
      "Cluster  50\n",
      "⬛⬛⬛\n",
      "✅:I need to drink and smoke a blunt\n",
      "✅:I need a smoke partner.\n",
      "✅:I need a cigarette.\n",
      "✅:I need a cigarette\n",
      "❌:I need to smoke at least once a day 😭\n",
      "✅:I need to smoke that’s all this mf need to come on with my weed\n",
      "❌:I need a tray of puff puff\n",
      "\n",
      "Cluster  41\n",
      "⬛⬛⬛⬛⬛\n",
      "❌:I need a milkshake or something\n",
      "✅:I need to eat\n",
      "❌:I need somebody to point me to the best food spots here in ATL\n",
      "❌:I need chocolate\n",
      "❌:I need to chill and bread\n",
      "❌:I need to find my grill 😔\n",
      "❌:I need to work on my sushi making skills lol\n",
      "\n",
      "Cluster  54\n",
      "⬛⬛\n",
      "❌:I need a car.\n",
      "❌:I need that car ASAPPPPP😩😂\n",
      "❌:I need a new charger 🥺\n",
      "❌:I need to buy a new car\n",
      "❌:I need my own car\n",
      "❌:I need a new phone\n",
      "\n",
      "Cluster  4\n",
      "⬛⬛⬛⬛⬛⬛\n",
      "❌:I need one good reason to stay\n",
      "❌:I need a mirror that'll come along and tell me that I'm fine\n",
      "❌:I need someone to stay and never leave 💯\n",
      "❌:I need good blowjob and I'll be simple tonight\n",
      "❌:I need another clean sheet from Fernandez tomorrow 😌😁\n",
      "❌:I need to know he's safe, pledis relese the goddam statement\n",
      "❌:I need to start protecting my pjm sanity on here\n",
      "\n",
      "Cluster  37\n",
      "⬛⬛⬛⬛\n",
      "❌:I need to go to the dentist, my wisdom tooth is crooked af\n",
      "❌:I need to go and get my beard done.\n",
      "❌:I need the guidance of my ancestors for this recipe\n",
      "❌:I need to get better at remembering peoples names when they first tell me😂\n",
      "❌:I need to be in somebody’s nail salon.\n",
      "❌:I need these nails to stay on for at least the wedding.\n",
      "❌:I need to get these wisdom teeth out asap\n",
      "\n",
      "Cluster  49\n",
      "⬛⬛⬛\n",
      "❌:I need the piercing place to allow oral piercings for the love of god pleeaaase\n",
      "❌:I need my ass eaten lol\n",
      "❌:I need someone to crack my back like a glow stick.\n",
      "❌:I need my meat SUCKED\n",
      "❌:I need somebody to pop my back\n",
      "❌:I need diggs to go off for me🙏🏾 my FF opponent got Julio Jones and he already ruled out\n",
      "❌:I need to get my ears pierced\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.cluster import AgglomerativeClustering\n",
    "## this is from https://github.com/UKPLab/sentence-transformers/blob/master/examples/applications/clustering/agglomerative.py\n",
    "corpus_embeddings = pray.model.encode(special_intentions[:1000])\n",
    "\n",
    "# Normalize the embeddings to unit length\n",
    "corpus_embeddings = corpus_embeddings /  np.linalg.norm(corpus_embeddings, axis=1, keepdims=True)\n",
    "\n",
    "# Perform kmean clustering\n",
    "clustering_model = AgglomerativeClustering(n_clusters=None, distance_threshold=distance_threshold) #, affinity='cosine', linkage='average', distance_threshold=0.4)\n",
    "clustering_model.fit(corpus_embeddings)\n",
    "cluster_assignment = clustering_model.labels_\n",
    "\n",
    "clustered_sentences = {}\n",
    "for sentence_id, cluster_id in enumerate(cluster_assignment):\n",
    "    if cluster_id not in clustered_sentences:\n",
    "        clustered_sentences[cluster_id] = []\n",
    "\n",
    "    clustered_sentences[cluster_id].append(corpus[sentence_id])\n",
    "\n",
    "for i, cluster in clustered_sentences.items():\n",
    "    print(\"Cluster \", i+1)\n",
    "    #print(cluster)\n",
    "    print(\"⬛\"*math.ceil(len(cluster)/3))## bar representing how big cluster is\n",
    "    pairs = sample_and_judge_cluster(cluster)\n",
    "    for p in pairs:\n",
    "        print(p)\n",
    "#     for symbol,si in pairs:\n",
    "#         print \"%s\"\n",
    "    print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here I rank special intentions based on their semantic proximity to `n` other special intentions.  I then test whehter or not I am currently praying for them.  If not, I should write a prayer to cover this special intention. \n",
    "\n",
    "(This was fast when I used the [`sent2vec`](https://github.com/epfml/sent2vec) library with (huge) pretrained vectors.  It is *very* slow with [`sbert`](https://github.com/UKPLab/sentence-transformers).)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Top for Various Start Grams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An even more straightforward way of figuring out what special intentions I should pray for next in order to have the most impact: analyzing the sequences of `n` tokens (*not* overlapping n-grams).  In other words\n",
    "\n",
    ">\"a new job in a new town\"\n",
    "\n",
    "is `[\"a\",\"new\"]` when `n = 2`, `[\"a\",\"new\",\"job\"]` when `n = 3`, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk import tokenize,FreqDist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['you']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "si_tokens = [tokenize.word_tokenize(i) for i in special_intentions_for_processing]\n",
    "si_tokens[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('to', 'quit', 'playing', 'and', 'get', 'to'),\n",
       " ('to', 'cry', 'for', 'three', 'good', 'days')]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_n = [tuple(si[:n]) for si in si_tokens]\n",
    "first_n = [si for si in first_n if len(si)==n]\n",
    "first_n[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(('someone', 'to', 'stay', 'and', 'never', 'leave'), 8),\n",
       " (('to', 'get', 'it', 'off', 'my', 'chest'), 4),\n",
       " (('somebody', 'who', 'can', 'love', 'me', 'at'), 3),\n",
       " (('somebody', 'to', 'love', 'i-i', 'do', \"n't\"), 3),\n",
       " (('to', 'lower', 'my', 'standards', ',', 'not'), 3),\n",
       " (('a', 'lover', 'and', 'a', 'friend', 'to'), 3),\n",
       " (('to', 'learn', 'how', 'to', 'say', 'no'), 3),\n",
       " (('to', 'get', 'away', 'for', 'a', 'bit'), 3),\n",
       " (('enough', 'of', 'you', 'to', 'dull', 'the'), 3),\n",
       " (('a', 'day', 'in', 'between', 'saturday', 'and'), 3),\n",
       " (('a', 'girlfriend', '..', 'she', 'can', 'feed'), 2),\n",
       " (('to', 'find', 'something', 'to', 'do', 'with'), 2),\n",
       " (('a', 'really', 'good', 'soft', 'massage', 'mostly'), 2),\n",
       " (('all', 'your', 'kisses', 'and', 'a', 'nap'), 2),\n",
       " (('a', 'girl', 'that', '’', 's', 'gon'), 2),\n",
       " (('that', 'one', 'person', 'who', 'will', 'stand'), 2),\n",
       " (('to', 'keep', 'my', 'chin', 'up', ','), 2),\n",
       " (('to', 'take', 'better', 'care', 'of', 'myself'), 2),\n",
       " (('to', 'stop', 'waking', 'up', 'at', '2'), 2),\n",
       " (('someone', 'to', 'braid', 'my', 'hair', 'down'), 2)]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FreqDist([si for si in first_n]).most_common(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}