SEO

Google adds two new Googlebot crawlers: GoogleOther-Image and GoogleOther-Video

Google has added two new crawlers to the Googlebot family of web crawlers:

  • GoogleOther-Image
  • GoogleOther-Video

Google said these two new crawlers are versions of GoogleOther optimized for fetching image and video bytes. GoogleOther was added in April 2023 to be used internally by Google teams to crawl the public web and free up some resources for the main Googlebot crawlers.

GoogleOther-Image. GoogleOther-Image, according to the documentation, is the version of GoogleOther optimized for fetching publicly accessible image URLs. It will go under the user agent tokens of GoogleOther-Image and GoogleOther and the full user agent string will be GoogleOther-Image/1.0.

GoogleOther-Video. GoogleOther-Video, according to the documentation, is the version of GoogleOther optimized for fetching publicly accessible video URLs. It will go under the user agent tokens of GoogleOther-Video and GoogleOther and the full user agent string will be GoogleOther-Video/1.0.

Why these new crawlers. Google said, “the new crawlers were launched to better support crawling of binary data that may be used for research and development.”

More on Google crawlers. The types of Googlebot crawlers include:

  • Googlebot – The main crawler for Google’s search products. Google says this crawler always respects robots.txt rules.
  • Special-case crawlers – Crawlers that perform specific functions (such as AdsBot), which may or may not respect robots.txt rules.
  • User-triggered fetchers – Tools and product functions where the end-user triggers a fetch. For example, Google Site Verifier acts on the request of a user or some Google Search Console tools will send Google to fetch the page based on an action a user takes.

Google also has listed IP address ranges and reverse DNS mask for each type:

Why we care. Many of you check your crawling activities and bot activity on your website and in your log files. When you see this new GoogleOther crawlers, do not be alarmed. It is a real Googlebot.

Related stories

New on Search Engine Land

About the author

Barry Schwartz

Barry Schwartz is a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics. Barry can be followed on Twitter here.

Products You May Like

Articles You May Like

Google streamlines product listings via website crawl
Social search and the future of brand engagement
How white label link building services work in 2024 by Admix Global
Google ties in Local Services Ads advertisers to Maps app
Google Ads phasing out card payments

Leave a Reply

Your email address will not be published. Required fields are marked *