SMART CRAWLER: A TWO-STAGE CRAWLER FOR EFFICIENTLY HARVESTING DEEP-WEB INTERFACES - Ms. Asmita D Rathod (International Journal of Innovations in Engineering Research and Technology, 2021)
Deep web growingat a very fast pace, lot of speculations in techniques this techniques has been added thathelpefficientlylocate deep-web interfaces. However, due to the large volume of web resources and the dynamic nature of deep web, achieving wide coverage and high efficiency is a challenging issue. In this paper author has proposed a two-stage framework, namely Smart Crawler, for efficient harvesting deep web interfaces. Smart Crawler performs site-based searching for center pages by usingsearch engines, avoiding visiting a large number of pages. To achieve more accurate results for a focused crawl, Smart Crawler techniques prioritize websites to highly relevant ones for a given topic. Smart Crawler achieves fast in-site searching by findingmost relevant links with an adaptive link-ranking. To eliminate bias on visiting some relevant links in hidden web directories, author has designeda link tree data structure to achieve wider coverage for a website.