“404 Not Found” Page Kills Spiders

Filed under: Tips & Tricks - Sites, Software, Shortcuts 

Free Links to Your Site

I can’t believe a new feature from Google isn’t getting more notice, because it converts already-existing links to your site into much higher quality links, for free. The Google webmaster blog just announced that you can find the pages that link to 404 pages on your site. Matt Cutts: Gadgets, Google and SEO

When someone comes to your site from a broken link, like KDandCompany.com/seiwi, the server generally returns the dreadful error code – 404 Page Not Found. Unless you have created a custom page, you might be driving visitors away.

Let’s talk about spiders for a second. You know, the same technology that the search engines use to index your pages. When’s the last time you ran one on your own site? Did you know that your custom “404 not found” error page could be driving them away?

You’ll want to sic a spider on your own site, before letting MSN, Google or Yahoo get at it. Why? To make sure its spiderable, that’s why! Huh??? Didn’t think about validating your linking structure, did you?

There are many page whackers available that download a whole site to your hard drive, but who knows what kind of “forgiving” spidering technology they’re using. The best one, the one that replicates the crawling spiders like Google and Yahoo the closest, is a tool like OptiSpider.

Here’s a tip, if OptiSpider can’t spider your site… neither can Google. That’s right, it’s probably just about the best reason to get and use a tool like OptiSpider, just so you can see the same stuff the spiders see.

If you manage multiple sites for clients, it’s a great weapon in your toolbox! I can’t count how many times I’ve done coaching sessions for clients who claim search engines are not finding all their pages. A few seconds later – using OptiSpider – I can tell them… dude!!! You didn’t link to the pages properly, or your custom 404 not found page is killing the search engine spiders.

No Google, Yahoo or MSN spider is going to find all your pages, when you put the crawler in an endless feedback loop. You’re lucky if the spiders ever come back.

Stop doing whatever it is (even if it’s reading this article ;-) and go look at your custom 404 page. Everyone loves to use them. Unfortunately, many people use them badly. The biggest sin of all – even I was guilty of this – is using a relative link on the 404 page. It’s so common that relative linking should be banned altogether.

Imagine the spider is several directories down in your site and encounters a missing page or broken link, up comes your custom 404 error page. Too bad sucker! You used a relative link instead of an absolute one. The spider is totally miffed at your site and resets, leaving all the rest of your pages behind… unspidered, unindexed and unfound.

Stop whatever it is you’re doing and go do it now. Put an absolute link on your 404 page, one that includes the http://www. and your full .com web address to your home page. That way, when your custom 404 error page comes up, the spider won’t choke. Instead, it finds your home page from the custom 404 page and continues spidering like normal.

If you want to make spiders happy and be sure they can crawl your entire site, then use a spidering tool like OptiSpider to test your internal linking structure. Just don’t be surprised if you end up getting a whole lot more pages indexed by the search engines. What’s on your 404 page?

by Michael Campbell

Did you like this article? Get more just like it when you subscribe to Michael Campbell’s free Internet Marketing Secrets Newsletter.

If you enjoyed this post, make sure you subscribe to my RSS feed!

Did You Miss These?

Comments

blog comments powered by Disqus