- All Journals
- Al-Qadisiyah University
- Basrah University
- Kirkuk University
- Mosul University
- The Iraqi Borad for Medical Specialization
- University of Anbar
- University of Technology
- Al Mustafa University
- Al-Zahraa University for Women
- Alnoor University
- Kunooz University
- Misan University
- Ninevah University
- Tikrit University
- University of Al-Hamdaniya
- University of Kerbala
Effective Web Page Crawler | ||
| Engineering and Technology Journal | ||
| Article 1, Volume 29, Issue 3, 2011, Pages 513-530 PDF (396.02 K) | ||
| DOI: 10.30684/etj.29.3.9 | ||
| Authors | ||
| Hilal Hadi Saleh; Israa Tahseen Ali | ||
| Computer Sciences Department, University of Technology/Baghdad | ||
| Abstract | ||
| The World Wide Web (WWW) has grown from a few thousand pages in 1993 to more than eight billion pages at present. Due to this explosion in size, web search engines are becoming increasingly important as the primary means of locating relevant information. This research aims to build a crawler that crawls the most important web pages, a crawling system has been built which consists of three main techniques. The first is Best-First Technique which is used to select the most important page. The second is Distributed Crawling Technique which based on UbiCrawler. It is used to distribute the URLs of the selected web pages to several machines. And the third is Duplicated Pages Detecting Technique by using a proposed document fingerprint algorithm. | ||
| Keywords | ||
| Search engine; web crawl; and fingerprint | ||
|
Statistics Article View: 299 PDF Download: 277 |
||