Traverse

Expand on known hosts related to a domain through searching for instances of repeated code/html and tracking ids across publically available data.

• Initial idea for this came from reading this bellingcat article

Traverse currently uses

• host.io [free, requires api key]
• spyonweb [free, requires api key]
• publicwww [free, requires api key],
• shodan [free, requires api key (premium api keys are regularly available for free or a low price)],
• WebArchive scraping.
  • This may take some time depending on how many snapshots of the page there are.
  • Disabled by default, to enable it open traverse.py and add "webarchive" to the 'services' list.
• Live page scraping

I wrote a blog post going into detail on this topic, some ideas referenced in the post have not yet been implemented.

There are 2 output formats: -oS (output simple) which is just a plain text output of discovered domains, and -oJ (output json) which is a more detailed JSON output.

Alternative resources that I (most likely) won't add support for in this tool:

• https://httparchive.org/ / BigQuery is amazing for things like this, but also very expensive :( If you have the resources, definitely look into using this great dataset.

• https://xaviesteve.com/domeye/ is also great, but I'll avoid scripting it as it seems to be run and paid for by an individual.

• https://dnslytics.com/reverse-analytics is good but requires payment for most features that separate it from others (no free api)

TODO:

• Facebook Pixel
• Google Tag Manager
• Quantcast
• Yandex Metrika
• Recursive Search
• CommonCrawl (?)