asktoby.php Version 1.0.0

"Done is Better than Perfect"

This is the code as it was when I made my first successful search query. There are tons of bugs, and the web crawler couldn't be any slower, but it works! The web crawler currently runs in series, one site crawled at a time. However, Version 2 will need to run in parallel with possibly 100 sites at a time in order to successfully index every top level domain homepage on the web.

Version 1.0 crawls at a pace of 3 sites a minute. This works out to roughly 1.5 million sites a year. However, with Version 2.0 the web crawler should be able to hit 100 million plus sites a year, enough to index every top level domain homepage on the web.