SmartScrapes

SmartScrapes is a web application built using Flask which provides an analytical solution for the companies who want to analyze their product.

Table of Contents

1. About The Project
   • Built With
2. New!
3. Coming up
4. Getting Started
   • Prerequisites
   • Installation
5. Usage
6. Examples
7. Watch a demo
8. Roadmap
9. Contributing
10. License
11. Contact

About The Project

While building products we often require feedback to improve but to take in 100 reviews at a single time is pretty tough to remember. Artificial Intelligence has transformed a lot of industries in the past and hence it makes sense to transform this arena as well. With SmartScrapes, these reviews are processed quickly using the power of Natural Language Processing which gives us a quick, efficient bird's eye view of the performance of the product involved.

Built With

• Flask
• NLTK
• Beautiful Soup
• expert.AI
• Plotly
• Twitter API
• Numpy, Pandas
• d3.js
• Morris.js

New!

• The whole thing is new right now.

Coming up

• Improving the way data is stored in cookies or go with a database approach

Getting Started

Follow the instructions to setup the project locally!

Prerequisites

Make sure to have virtualenv package from python installed before proceeding to installation.

pip install virtualenv

Installation

1. Clone the repo

   git clone https://github.com/lazyCodes7/SmartScrapes.git

2. Activate the virtual environment

   cd SmartScrapes
   virtualenv venv
   . venv/bin/activate

3. Install the required packages using pip

   pip install -r requirements.txt

4. How to get credentials? For credentials make an account on expert.ai and also a twitter dev account and get the tokens to be used in the steps below.

5. Create a .env file with the credentials you got in step-4

   ACCESS_KEY = ""
   ACCESS_SECRET = ""
   CONSUMER_KEY = ""
   CONSUMER_SECRET = ""
   EMAIL= ""
   PASSWORD=""
   APP_SECRET_KEY = ""

6. Run app

   python app.py

7. Open the link in a web browser

   127.0.0.1:5000
   

Usage

1. Enter your twitter username Enter the twitter username in the box and click "Analyze" and wait for the results to show up

a8da6831-714d-4eb1-8adc-da56bf6b44af.mp4

2. Interactive dashboard opens and one can save the plot or zoom into the visualizations

7f279c0e-514a-49ed-a813-3e427b4a8549.mp4

3. Analyze playstore or appstore reviews or twitter reviews.

0f36314f-79b5-4405-a231-d94e4923e58e.mp4

4. Read the reports

reports.mp4

5. Analyze your own custom data

custom.mp4

Examples

This project is now deployed at https://smartscrapes.herokuapp.com/ so do check it out.

List of usernames to try out!

• Tesla: A company founded by Elon Musk. Would be cool to analyze the online presence of a big company eh?
• PlayChoices: A mobile app that lets you play stories. Pretty cool app and I do play it:)

Tips while viewing the visulizations on the app!

• Make sure to have at least a basic twitter account for your brand to analyze it.
• If the visualization don't load up soon try it once again. It might be because of pre-processing delays
• If the visualizations seems small try toggling the zoom values. They are responsive so it would definetely come into the correct shape :p
• Finally, it is not just a static visualization so play around with it and save it up on your device afterwards!

Watch a demo

Watch the video on YouTube

Roadmap

See the open issues for a list of proposed features (and known issues).

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Rishab Mudliar - @cheesetaco19 - [email protected]

Telegram: lazyCodes7