googlebot

Controlling Googlebot crawls

For some Google crawls too often which comsumes too much bandwidth. For others it visits too infreqently. Some complain that it doesn't visit their entire site and others get upset when areas that they didn't want accessible via search engines appear on the Google index. It is not really possible to attract robots. Google
will visit your site often if the site has good content that is updated often and is mentioned by other sites. However it is possible to deter robots. You can control both the pages that Googlebot crawls and request reduction in the frequency or depth of each crawl.

How quickly you can expect to be crawled

There are no firm guarantees as to how quickly new sites or web pages will be crawled by Google and then appear in the search index. However , following one of the four actions above, you would normally expect to be crawled within a month and then see your pages appear in the index two or three weeks afterwards. In my experience, submission via google webmaster tools is the most effective way to manage your crawl and to be crawled quickly, so I typically do this for all my clients.

How GoogleBot finds your site

GoogleBot finds your site in 4 possible ways.

1 Submit your URL to google for crawling, via the "Add URL" form at www.google.com/addurl.html.
2 Google finds your site from another site that is already index so that google sends a spider to find your link
3 Sign up to google webmaster tools , here it verifies your site and submit a sitemap.
4 Redirect an already indexed page to the new page.

Syndicate content