Skip to content

🥷 stealth scraping with style

Notifications You must be signed in to change notification settings

realstorypro/agent

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

63 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

The Agent 🥷

"Nobody can give you freedom. Nobody can give you equality or justice or anything. If you're a man, you take it." — Malcom X

Features

  • Companies Prospecting
  • Distributed Crunchbase Company Scrape
  • Contact Enrichment
  • Customer.Io Upload

Setup

  1. Setup mitmproxy
brew install mitmproxy
  1. Start the mitmproxy to disable peremetrix
 mitmproxy -s mitmproxy/crunchbase.py  
  1. Setup ENV file (including AGENT_CODENAME)

Usage

Prospecting Crunchbase

  1. Build a list
rake list:build
  1. Upload the list
rake list:upload

Scraping Crunchbase

  1. Take companies to scrape (550 Max)
rake agent:take
  1. Scrape with the following command
rake crunchbase:scrape
  1. Stop the scraper with
kill $(ps aux | grep '[s]crape' | awk '{print $2}')

Uploading to Customer.io

Note: This uploads all contacts not just the ones assigned to the agent. It's best to run this after 5pm (MST) so that we can run enchance right after

rake contacts:process

Enrich contacts w/ Timezone info

rake contacts:enrich 

About

🥷 stealth scraping with style

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages