A Web Crawler System Design Based on Distributed Technology
|
| Title | A Web Crawler System Design Based on Distributed Technology |
| Authors | |
| Abstract | A practical distributed web crawler architecture is designed. The distributed cooperative grasping algorithm is put forward to solve the problem of distributed Web Crawler grasping. Log structure and Hash structure are combined and a large-scale web store structure is devised, which can meet not only the need of a large amount of random accesses, but also the need of newly added pages. Experiment results have shown that the distributed Web Crawlers performance, scalability, and load balance are better. |
| Publisher | ACADEMY PUBLISHER |
| Date | 2011-12-01 |
| Source | Journal of Networks Vol 6, No 12 (2011) |
| Rights | Copyright © ACADEMY PUBLISHER - All Rights Reserved.To request permission, please check out URL: http://www.academypublisher.com/copyrightpermission.html. |