Skip to content

WhatsApp 'Export Chat' Scraper

Jump to bottom
tarunima edited this page Jun 12, 2020 · 1 revision

Scraper Feature List

WhatsApp lets Group Chats to be exported via 'Export Chat' feature. One way to get content out of WhatsApp Public Groups is to use this feature and then parse the text file.

The imported text file has a name: "WhatsApp Chat with...". This is not a unique identifier. If two WhatsApp groups have the same name, the text file exported from these groups will also have the same name.

There are three incremental steps to building this scraper:

  1. Extract content from an exported chat text file.
  2. Handle text files that are created from the same WhatsApp group at a set frequency
  3. Extract content from text files from multiple groups exported at a set frequency.

Extract content from an exported chat text file.

  1. Export the chat to Google Drive- the text file and a number of media items will be uploaded to the selected Google Drive Folder
  2. Read the File Using OAuth
  3. Parse text file- extract time stamp, message, and media item name.
  4. All text messages that come from the same person in a series should be linked as one message.
  5. The link between media item and text in a message with both should be retained.
  6. Upload media items to s3, metadata to mongo.

Handle text files that are created from the same WhatsApp group at a set frequency

  1. Every time a chat is exported with media items, it exports 10K rows. Thus there will be some redundancy between files created from the same group chat.
  2. Find an efficient mechanism to find new rows or the starting point of new rows in the recently exported file.
  3. Scrape the new rows as per the code written in previous section ('extracting content from an exported chat text file').
  4. Delete media items and text files older than 'n' iterations from Google Drive.

Extract content from text files from multiple groups exported at a set frequency.

  1. The first step is to figure out a mechanism to set a unique ID for every group chat. This could be done on the basis of the Google drive folder name or the timestamp at which the group was first scraped.
  2. Loop through all folders and scrape as per scraper written in previous section ('Handle text files that are created from the same WhatsApp group at a set frequency')
  3. Create a log of group conversations that were updated on a particular run.
Clone this wiki locally