SEO

Google explains the use cases for its different crawler types

Google has now added new details that explain the three categories its Google crawlers fall into, they include Googlebot, special-case crawlers and user-triggered fetchers.

In addition, Google now lists a JSON formatted file containing the list of IP addresses each of these different crawler types use.

Types of Google crawlers. At the top of this Googlebot page, Google listed these three crawler types:

  • Googlebot – The main crawler for Google’s search products. Google says this crawler always respects robots.txt rules.
  • Special-case crawlers – Crawlers that perform specific functions (such as AdsBot), which may or may not respect robots.txt rules.
  • User-triggered fetchers – Tools and product functions where the end-user triggers a fetch. For example, Google Site Verifier acts on the request of a user or some Google Search Console tools will send Google to fetch the page based on an action a user takes.

IP addresses. Google also listed the IP address ranges and reverse DNS mask for each type:

What is new. Here is the section of the page that was updated; the rest of the page is mostly unchanged.

Google Crawler Purposes 800x347

Why we care. I believe Google made this change after they saw some of the reactions to the GoogleOther robot they announced the other day. This now explains how Google crawlers act, when they respect the robots.txt and how to identify them better.

Now, if you want not to block Google’s main crawler, Googlebot, but you decide to block the others, you can better identify those crawlers more accurately.

Related stories

New on Search Engine Land

About the author

Barry Schwartz

Barry Schwartz a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry can be followed on Twitter here.

Products You May Like

Articles You May Like

Social search and the future of brand engagement
Meta unveils AI chatbots and business messaging tools 
Google ties in Local Services Ads advertisers to Maps app
WeArisma launches AI visual search technology to drive creativity in marketing strategies 
Google unleashes June 2024 spam update

Leave a Reply

Your email address will not be published. Required fields are marked *