Project Web Crawler

🗂 Contents

• 📋 Project
  • 🚀 Technologies
  • 🧮 Diagram
  • 🌐 Prototype
  • 📷 Preview
• 📦 Installation
• ⚖️ License
• 📫 Contact

📋 Project

This project involved the development of a web crawler that delves into a specific domain to extract all internal links and assets. In the "front-end", users input the domain they wish to crawl, and when the "back-end" completes the crawling process, a tree-graph view of the results is generated, which can also be exported to a sitemap XML file.

🚀 Technologies

🧮 Diagram:

🌐 Prototype

https://www.sitemapper.net

📷 Preview

📦 Installation

To clone and run this application, you will need Git, Node.js@20+ and npm installed on your computer. In your terminal, run:

# Cloning repo
$ git clone https://github.com/sagelabs/bruno-duarte-interview.git sitemapper

# Back-end

# Go to the repository
$ cd server

# Install the dependencies
$ npm install

# Run the API
$ npm run dev

# Note: After running the command above, the API will be available at http://localhost:3000

# Front-end

# Got to the repository
$ cd web

# Install the dependencies
$ npm install

# Start the application 
$ npm run dev

⚖️ License

This project is under license. MIT.

📫 Contact

by Bruno S. Duarte 🚀