Do you know the spider names?

If one ever wondered if search engine engineers were a whimsical lot, one only needs to looks at the names of search engines’ spiders to find the answer. Googlebot, Slurp, Ask Jeeves and ia_archiver are some of the spider names used by the most popular search engines on the web.

Which search engines uses spiders to fetch documents?

Using a program called a spider, search engines continually search the web looking for new pages and reviewing documents they already have in their index. You may ask, “Which search engine uses spiders to fetch documents?” and the answer is, “All of them.” Without a spider, search engines would not be able to find new pages.

Why name a spider? Search engines don’t have to.

The main purpose of spider names is so that search engines’ spiders can be easily identified in your site logs. When was the last time you looked at your logs? Today, with the availability of free tracking software, such as Google Analytics, many people never look at their logs, but if you do, you will see the spider names from the search engines appear all over it.

Think of it as a courtesy. Certainly the search engines’ spiders could arrive at your site under the name “Mozilla” which is how most web browsers appear in your logs, but the search engines don’t wish to be rude. By using spider names, they let a site owner know that they have been by to visit.

Why you should care about search engines’ spiders and their names.

You can use this information to find out when the last time the search engines’ spiders came to your site or a page on your site. The more important they think a page is, the more frequently they will visit a page. If you look at your site logs, you will see the spider names and will be able to tell if the search engines have picked up the latest changes on your site.

Name of each spider, search engines they belong to:

Knowing a little more about search engines spiders and the spider names will allow you to get a better insight into what the search engines know about your site. Tune in next time when we go over how you can give directions to spiders about how, when and if they should look at pages on your site.