Thursday, April 17, 2008

What is crawlers, spiders, robots?

A Crawler also known as Robot, Bot, Spider is a program that crawls Web site and reads
their pages and other information in order to create entries for a search engine index.As most of the websites contain links to other pages, a crawler can start crawling from
anywhere in the pages. As soon as it sees a link to another page, it can start crawling
that page leaving behind the current page that it was spidering.
Name of some search engine robots
Search Engines Robot
Google --> googlebot
MSN --> msnbot
Yahoo --> yahoo slurp
excite.com ---> architextspider
entireweb ---> speedy spider
look.com ---> lookbot
scrubtheweb.com --> scrubby