SEO

Google explains the use cases for its different crawler types

Google has now added new details that explain the three categories its Google crawlers fall into, they include Googlebot, special-case crawlers and user-triggered fetchers.

In addition, Google now lists a JSON formatted file containing the list of IP addresses each of these different crawler types use.

Types of Google crawlers. At the top of this Googlebot page, Google listed these three crawler types:

  • Googlebot – The main crawler for Google’s search products. Google says this crawler always respects robots.txt rules.
  • Special-case crawlers – Crawlers that perform specific functions (such as AdsBot), which may or may not respect robots.txt rules.
  • User-triggered fetchers – Tools and product functions where the end-user triggers a fetch. For example, Google Site Verifier acts on the request of a user or some Google Search Console tools will send Google to fetch the page based on an action a user takes.

IP addresses. Google also listed the IP address ranges and reverse DNS mask for each type:

What is new. Here is the section of the page that was updated; the rest of the page is mostly unchanged.

Google Crawler Purposes 800x347

Why we care. I believe Google made this change after they saw some of the reactions to the GoogleOther robot they announced the other day. This now explains how Google crawlers act, when they respect the robots.txt and how to identify them better.

Now, if you want not to block Google’s main crawler, Googlebot, but you decide to block the others, you can better identify those crawlers more accurately.

Related stories

New on Search Engine Land

About the author

Barry Schwartz

Barry Schwartz a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry can be followed on Twitter here.

Products You May Like

Articles You May Like

Snap unveils enhanced campaign insights for ROAS
Yelp ad revenue jumps 13% to record $1.28 billion
Meta apologizes after ad error causes campaigns to overspend by ‘thousands’
Google Analytics 4 bug causes data gaps in reports
Article complaining about being outranked on Google being outranked by Reddit

Leave a Reply

Your email address will not be published. Required fields are marked *