Skip to content

Latest commit

 

History

History
24 lines (14 loc) · 733 Bytes

README.md

File metadata and controls

24 lines (14 loc) · 733 Bytes
Raw

Go Webcrawler

Go

Example of a graph

Overview

Overview of architecture

SQS Queue

The SQS queue is used to store the URLs that need to be crawled.

DynamoDB

DynamoDB is used to store the URLs that have been crawled and the URLs that have been found on the page.

Neptune / Neo4k

Neptune is used to store the relationships between the URLs that have been found on the page.

Tests

The tests are written in Go and can be run using the following command: bash go test ./...