With the abundance of information floating around about search engine optimization strategy, it’s good to get back to basics to address how search engines work!

This is the first post in our multi-part series that deconstructs the technical how and why of SEO in a way that non-technical users can understand.

How search engines work

The major search engines (Google, Yahoo and Bing) use what are commonly called “search spiders” or “search engine spiders” to examine web pages. They “crawl” or “spider” the web and index their results so that people can search through what they have found.

Search spiders and engines are very simple creatures, overall, and search engine optimization strategy – or SEO – is an umbrella term for the group of techniques that make it easier for the spiders to find and properly index pages on your website. But, first – more about the spiders.

What do the search spiders do on my site?

A search engine spider visits a web page, reads it, and then follows links to other pages within the site. The spider returns to the site on a regular basis to look for additions and changes.

Spiders are given names by the search engines so that they can be easily identified in web site logs. It lets you know which spiders are crawling your site and how frequently. The more frequently a spider visits a page, the more important the spider thinks the page is. Common search engine spider names are:

Google: Googlebot
Yahoo: Yahoo Slurp
Bing/MSN: MSNbot

If you go into your site logs, you’ll see a string of text that likely has a date, time, a reference to the URL crawled, and a citation as to what was searched. So if Bing crawled a certain URL, you’d see a log of it along with a reference such as: (+http://www.bing.com/msnbot.htm).

What happens after my page is crawled by one of the search spiders?

The spider places the information into the search engine’s index. The index functions as a master repository of every web page that the spider finds. Until a page is indexed, it is not available to those searching with the search engine.

When a user searches and gets results, they are essentially getting a ranked list of what spiders have found. Each search engine has a program that sifts through the millions of pages recorded in the index to find matches to a search and rank them in order of what it believes is most relevant, based on the words the searcher typed in.

Now that we have a foundation for understanding how search engines work, we will delve further into indexing, and highlight the important role that search engine optimization strategy plays in getting your site properly indexed.