Google, Spiders, and You

Google, Spiders, and You

Did you know Google is overrun by spiders?

It's true, and spiders are the key to how Google indexes and ranks your blog posts in search results. Here's how Google describes the process:

Google's spiders regularly crawl the web to rebuild our index. Crawls are based on many factors such as PageRank, links to a page, and crawling constraints such as the number of parameters in a URL. Any number of factors can affect the crawl frequency of individual sites.

Our crawl process is algorithmic; computer programs determine which sites to crawl, how often, and how many pages to fetch from each site. We don't accept payment to crawl a site more frequently.

English translation: We have lots of spiders and we use them to read your posts and rank them in our search engines. How often we send our spiders to your page depends on a lot of factors. You can't pay us to make us send out our spiders more frequently.

So if we look at Google as the Wicked Witch of the West, its "spiders" would be flying monkeys. And Google sends them out when they damn well please to crawl and index your site. The spiders/flying monkeys go out when they are "pinged" to do so, and sites like ChicagoNow have "pings" built into their programming. As such, we get crawled pretty regularly by Google spiders and flying monkeys, even though we have no control over the timing or intervals.

But what if you need Google to send out the spiders? We've already established that you can't pay Google to release the spiders, but can you politely request that Google release the spiders? And why would you want to?

Let's look at an example.

Let's say you are lucky enough to witness Justin Bieber and Kim Kardashian robbing a Walgreens. You've got photos. Heck, you've got video! And you've written a smashing blog post about your harrowing saga. You've got it loaded up and ready to hit the internet tubes before TMZ even knows it happened. With this title:

Justin Beeber and Kim Kartrashian Rob Walgreens!

An hour later, you learn the error of your ways. You've spelled Bieber and Kardashian wrong throughout your entire post. Oh no!! You quickly edit your post, spelling "Bieber" and "Kardashian" right this time. But do you let Google know you've updated your post? After all, everyone in Planet Earth is googling "Bieber robs Walgreens," and you still want in on the action. What do you do? What DO you DO?

Here's what you do,

First, you place the word "cache:" in front of your post url. This brings up Google's latest version of your post, allowing you to see which version of your post Google is working with.

For example:

cache:http://wwww.chicagnow.com/julies-awesome-blog/bieber-kardashian-rob-walgreens

You check Google and realize that its spiders/flying monkeys haven't arrived yet. It's still working with the "Beeber/Kartrashian" version of your post. Oh no again!

Now what do you do?

Well, here's something you can try:

Head over to Google Webmaster Tools and submit the url to Google via this link. This link basically politely asks Google to send out the spiders/flying monkeys to re-crawl and index the correct version of your site. And sometimes they will.

Google makes it abundantly clear that they don't always send out the spiders/flying monkeys on request. In other words, they might completely ignore your request for a recrawl. If that happens, you're out of luck, and your stuck waiting for Google to decide to crawl your site again of its own accord. But submitting a url can often prompt a recrawl. For that alone, it's worth a shot. After all, there's no place like home.

Filed under: SEO

Tags: Google, SEO

Leave a comment