Skip to content

AskToby.PHP Version 3.0.0

Latest
Compare
Choose a tag to compare
@topherPedersen topherPedersen released this 12 Jun 18:39

Version 2 of AskToby.PHP used a simple 3 table database. It ran well up until around 100,000 pages crawled. At this point, the crawler slowed nearly to a halt, most likely due to the sheer size of the tables. Version 3 seeks to fix this problem by breaking each of the three original tables into twenty six. Other than the breaking up the tables, this version is essentially the same. However, breaking up the database design required a complete re-write of the crawler. With this release I hope that the crawler will be able to index 1,000,000 pages as opposed to the 100,000 crawled by Version 2 of the toby-crawler.