Google StoreBot crawler, a bot designed to collect product data for Google Search, has been abusing its powers by submitting spammy fake orders through online forms. To save businesses from this unwanted activity, we have implemented a targeted blocking solution using CleanTalk Anti-Spam.

Understanding Google StoreBot crawler

Google StoreBot crawler is a search engine-based automated crawling program, used for gathering and analyzing product information. While it is mainly designed to improve product listings within Google Search, it is now being exploited to send fake orders. For detailed information on the Google StoreBot crawler, see Google's official documentation: https://support.google.com/merchants/answer/13294660  

Why Block the Google StoreBot crawler?

The parameter ?add_to_wishlist= appeared frequently in the crawl log results. The problem is that "?add_to_wishlist=" pages are not cacheable. Various parameters often appear in WooCommerce systems when a user is logged in, when a user adds products to the shopping cart or when viewing them. Bots like Google StoreBot index "add-to-wishlist" links. These links cannot be cached and there is no need for them to be indexed.

Blocking the Google StoreBot crawler will prevent spammy, and fake orders, improve your website's performance, and protect your business from potential fraud. Note that blocking the bot may temporarily impact how your products are indexed in Google Search, although this can be partly mitigated by having well-structured and accurate product data.

How to Stop Google StoreBot Crawler from Sending Fake Orders

Solution via robots.txt (preferred)

There are WooCommerce themes that perform the “add-to-cart” functionality via Javascript and the bots are actually not familiar with these links, but there are also WooCommerce themes that add “add-to-cart” links directly in HTML files.

Whether your add-to-cart links are executed via Javascript or added directly in an HTML file, it is recommended that you disable the option to index unnecessary parameters. All you need to do is set parameters in the /robots.txt file that tell the robots not to index add-to-cart links.

Example robots.txt for WooCommerce:

#Block WooCommerce assets
User-agent: *
Disallow: /cart/
Disallow: /warenkorb/
Disallow: /checkout/
Disallow: /kasse/
Disallow: /my-account/
Disallow: /mein-konto/
Disallow: /*?orderby=price
Disallow: /*?orderby=rating
Disallow: /*?orderby=date
Disallow: /*?orderby=price-desc
Disallow: /*?orderby=popularity
Disallow: /*?filter
Disallow: /*add-to-cart=*
Disallow: /*?add_to_wishlist=*

With these parameters, crawlers will no longer index your add-to-cart links and some other pages that also cannot be cached.

Our Solution: A Targeted Block

To solve this problem, we have applied a custom solution with the personal blocklist by CleanTalk Anti-Spam. In such a way, we targeted the exact user-agent of the Google StoreBot crawler and since then have managed to block fake order creation by this bot.

To enable user-agent blocking in CleanTalk, navigate to WordPress Admin > Settings > Anti-Spam by CleanTalk. From there, go to the Advanced Settings tab. Locate the "Anti-Crawler" option and set it to "On". Once you've made this change, click "Save Changes" to apply the new settings. This will activate the user-agent blocking feature, helping to prevent spam from bots like Google StoreBot.

1. Choose the type of filter you need to block the Google Store-Crawler — User-Agent.

2. Pick the Google StoreBot crawler from the drop-down list.

Benefits of blocking Google StoreBot crawler:

• Less spam: Significantly cuts back on spammy fake orders.
• Better website performance: Cuts load on your servers.
• Improved security: Protects your business from potential fraud.