Googlebots 101
Lets talk about Googlebot how they works. I remember when I’m training a chinese girl about Google Search algorithm I’ve told him about Search Engine spiders. Then after the brief overview about Search Engine Spiders I’ve told him to ask me question on what are the things that she don’t understand. And to my surprise she told me that he dont understand Spiderman. “How it looks like” ? LOL. I was about to laugh at it but hey this girl don’t really understand english well so I just keep my silence and laugh afterwards. It made me realize that for newbie it is hard for SEO aspirant to know what is a bot.
Lets take a closer picture of a bots. A spider/bots are the one sent by search engine to crawl websites. It is an electronic message sent by Search Engine to look and gather data from websites around the world. Google spiders are called Google bots. This our topic for today how Google bots works.
How Googlebots work?
Lets take for an example a sales manager sending a representative to collect data about a certain products they’re endorsing to different houses in their local areas. Google bots is just like these they are one sent by Google to look and gather data so that it will be sent back to Google to analyze how the website is structured and it’s contents.
If someone is linking to your website your site can be crawled by Googlebots. Also if your website is already crawled before then Googlebots will just keep crawling your website much often.
What are the pages Googlebots crawling?
All of your pages will be crawled unless you disallow Googlebots in your robots.txt
Why my site isn’t crawled by Googlebots?
Google crawl lot of pages but unfortunately they can’t crawl all because for some reasons. i.e. your site must have links from high PR websites so that they can crawl fast your website. Also Google missed your website because when you launched your website Google has just finished the most recent crawl. The structure of your website is also important. You may include sitemap in your website so that Googlebots can crawl your website efficiently. This is one of the importance of Onpage optimization. If your host is down at the moment Google spiders went to your site is also a factor why Google didn’t crawl your website.
There are also some issue about Googlebots that webmaster may encounter. Because Googlebots are taking lots of bandwith. So if your site is running from a webhosting with small bandwith, then you will suffer from this Googlebots problem. The solution is to block Googlebots from some parts of the site such as php or asp files. Also you must block Googlebots in your robots.txt for security purpose only such as admin area of your site.
