Good Bots You Must Never Block (Googlebot, AdsBot, and Friends)

The whole trick is to block bots without blocking Googlebot, and plenty of advertisers get this wrong in the first week of trying to clean up their traffic. Automated visits are annoying, blocking feels like progress, and the crawlers that keep your business visible are the easiest ones to catch by accident, because they are the only ones honest enough to say who they are. The fraud does not identify itself. Googlebot does. Guess which one a crude rule catches first.

For the wider picture of what you are defending against, see our overview of bot traffic in Google Ads. This page is about what has to stay reachable while you do it.

The crawlers that have to keep getting through

Googlebot

This is the crawler that indexes your site for organic search. Block it and your pages stop being refreshed in the index, then stop ranking. Nothing in your ad account will warn you. You find out when your organic traffic quietly falls over a few weeks and somebody asks why.

AdsBot-Google

This one checks your ad landing pages. It fetches the destination to assess the page behind your ads, and it is directly connected to how your ads are treated. Blocking AdsBot is a genuinely self-inflicted wound: you are paying for ads while preventing Google from looking at where they point. Worth knowing that AdsBot does not follow the same blanket robots.txt rules as ordinary crawlers, which trips up people who write a broad disallow and assume they have covered everything.

Google-InspectionTool and the rendering fetchers

These support Search Console testing and URL inspection. If you have ever run a live test and got a result that did not match what you see in a browser, a block somewhere in the stack is one of the usual suspects.

The other search engines

Bingbot matters more than most people assume, partly because of where Bing results end up. There are also regional engines that matter a great deal in specific markets. Losing them is not fatal but it is free traffic thrown away.

Social and messaging preview fetchers

When someone shares your link on a social platform or in a messaging app, a fetcher grabs the page to build the preview card. Block those and your links get shared as bare URLs with no title and no image, which measurably reduces the number of people who click them.

Your own monitoring

Uptime checks, performance monitors, accessibility scanners, your SEO tooling. All automated, all yours, all easy to block by mistake and then spend an afternoon debugging.

How to block bots without blocking Googlebot

The rule that keeps people safe is short: never block on a broad pattern in the user agent string.

Blanket rules that catch anything containing "bot", "crawl" or "spider" are the single most common way sites take out their own indexing. Those words appear in the names of exactly the crawlers you need. Meanwhile the traffic you were trying to stop declares itself as Chrome on Android, so the rule misses it entirely. You get the worst of both outcomes: your rankings suffer and the fraud continues.

Three habits prevent most of this.

• Verify before you trust a name, and verify before you block one. A user agent is a claim. Anyone can put "Googlebot" in theirs, and some do, hoping for preferential treatment. Google publishes the means to confirm whether a request genuinely came from its crawlers, and that check is the right way to settle it either way.
• Keep robots.txt and blocking separate in your head. Robots.txt is a set of instructions for well-behaved crawlers. It has no enforcement whatsoever. Honest bots follow it, dishonest ones ignore it, so it is a directive for the good ones and irrelevant to the bad ones. It is not a security control and it is not a fraud defence.
• Check your logs after any change. If crawler visits drop to zero after you tighten something, you did not just stop fraud. Nobody sends an alert for this.

Why the distinction is easy to lose

The industry split explains it cleanly. Declared crawlers are general invalid traffic: they identify themselves in the user agent, they follow published conventions, and they are trivially identifiable because they are not hiding. Sophisticated invalid traffic is the opposite by definition, since concealment is the whole point.

So the honest bots are the visible ones and the dishonest bots are the invisible ones. Any defence built on what traffic says about itself will therefore hit precisely the wrong targets. That is not a subtle failure mode, it is the predictable result of the approach.

It is also worth being clear that declared crawler traffic is not your ad fraud problem. Those visits are not clicking your ads. Google filters that category well and issues automatic credits for invalid clicks it identifies, though what that does and does not cover is a longer story, which we go through in what Google's invalid click refunds leave behind.

What breaks, and how long it takes you to notice

The cruel part of this is the delay. Block Googlebot today and today looks fine. Rankings decay over weeks as the index goes stale. Ad landing page assessments degrade quietly. Shared links stop generating previews and nobody reports it, because nobody notices a card that is not there.

By the time the damage is visible, it is rarely connected back to a firewall rule somebody added a month ago to stop some annoying traffic. That is why this is worth getting right the first time rather than discovering it later.

Doing this without breaking anything

The workable position is that identity is not something a visitor gets to assert. Verify the crawlers that claim to be important, ignore robots.txt as a defence, and judge unknown traffic on what it does rather than on what it says it is.

That is also why a dedicated layer beats hand-written rules. Click fraud protection is built to keep legitimate crawlers reachable while acting on traffic that is demonstrably not a customer, which is exactly the distinction a user agent rule cannot make.

If you take one action from this page, go and read your current blocking rules with fresh eyes. Look for anything matching on a wildcard in the user agent. That is where the accidental damage usually lives, and it has probably been there longer than you would like.