The number of site pages in a search engine index can differ widely between Google and Yahoo!, which we all know is frustrating. Why the difference?

Let’s say your site has 300 pages in the Google search engine index and 315 in the Yahoo! index. Those numbers are pretty close; it looks like both spiders are crawling and indexing most of your content.

But what if your site has 300 pages in Google, and 620 in Yahoo!? This indicates that there may be a deeper issue affecting your site.

Canonical URL, duplicate content and 404 error pages are three common reasons why page counts can differ.

1. Canonical URL

Canonicalization is the process of the search engines having to choose the best URL to index when there are several choices. Two of the most common ways a canonical URL can occur are:

• www vs. non-www
• http vs. https

The search engines can mistake http://www.mysite.com and http://mysite.com as two different sites with exactly the same content. This is also true for http://www.mysite.com and https://www.mysite.com.

2. Duplicate content

Google duplicate content, and duplicate content in Yahoo!, can occur as a result of a canonical URL as mentioned above. This may trip the Google duplicate content filter which can cause pages from your site, or even your entire site, to be removed from the search engine index.

3. 404 error pages

The Yahoo! search engine index is slower to crawl new pages and therefore tends to hold onto old pages longer than Google. If your site’s index count is inflated in Yahoo!, you may discover that Yahoo! has indexed 404 error pages occurring on your site.

All of these search engine index issues can be corrected using a 301 redirect.

A 301 redirect is a permanent redirect from one URL to another. Implementing this redirect will correct canonical issues and remove duplicate content and 404 error pages from the indexes.

Find the challenge of fixing these issues daunting? Let The Search Guru technical consultants help. Want to find out what other issues your site may have? Request a free SEO gift session today.