• D.YASWANTH
• A.S.ANUHYA
• Mr.B.MANOJ,

A web Search is a program, which automatically traverses the web by downloading documents and following links from page to page. They are mainly used by web search engines to gather data for indexing. Web Search is also known as spiders, robots, worms etc. In this proposed project the web search program acts as an smart crawler application where the crawling of the pages is done not by the overall page rank (i.e. Overall total page’s rank count visited by users),but the pages are crawled based on individual page count of individual URL’s. The proposed smart application is used to measure the individual page traffic accurately and is mainly useful for Web–Masters to maintain the traffic of each and every web page in very sophisticated manner. As this application requires internet connection, the internet connection should be of enough bandwidth for processing of the Web pages URL’s accurately. In this current application we can find out the count of successfully crawled URL’s as well as failed URL’s successfully based on the pages which were crawled by internet traffic. In this project BFS algorithm also known as “Breadth First Search” is used, for crawling the web pages based on the individual page rank and there will be no back traversing of URL’s, so this will give unique results. As an extension the monitoring of count of failed URL’s and count of successful URL’s crawled in this application has also been included, this gives more importance for the application of each and every URL.

Web search, Web pages, Crawled URL’s, Breadth First Search, Traversing, etc

"SMART CRAWLER-A TWO STAGE FRAME WORK FOR CRAWLING DEEP WEBSITES", International Journal of Emerging Technologies and Innovative Research (www.jetir.org), ISSN:2349-5162, Vol.6, Issue 4, page no.84-87, April-2019, Available :http://www.jetir.org/papers/JETIRBE06016.pdf

"SMART CRAWLER-A TWO STAGE FRAME WORK FOR CRAWLING DEEP WEBSITES", International Journal of Emerging Technologies and Innovative Research (www.jetir.org | UGC and issn Approved), ISSN:2349-5162, Vol.6, Issue 4, page no. pp84-87, April-2019, Available at : http://www.jetir.org/papers/JETIRBE06016.pdf