good-bots-never-block

Keep Bingbot and any regional search engine that matters in your markets. Keep Google's inspection and rendering fetchers, or Search Console testing gives you misleading results. Keep the social and messaging preview fetchers, otherwise shared links appear as bare URLs with no title or image and fewer people click them. And keep your own uptime, performance and accessibility monitoring reachable.

Generally no. Declared crawlers visit your pages, they do not click your paid ads, so they are not what is draining your budget. This is worth keeping straight, because people who see bot traffic in their site logs often assume it is the same thing as click fraud. It is not. The traffic costing you money is the kind that never admits to being automated.

Do not take the name at face value. A user agent is a claim, and some traffic puts Googlebot in its own string hoping for softer treatment. Google publishes the means to confirm whether a request genuinely originated from its crawlers, and that check is the right way to settle it before you either trust a visitor or block one.

No. Robots.txt is a set of polite instructions with no enforcement whatsoever. Well-behaved crawlers read it and comply. Anything built to deceive ignores it completely, because there is nothing to make it comply. So it shapes the behaviour of the bots you were never worried about and does nothing to the ones you are. It is not a security control.

AdsBot-Google is the crawler that fetches and assesses your ad landing pages. Blocking it means you are paying for ads while preventing Google from looking at the page behind them. It also does not follow the same blanket robots.txt rules as ordinary crawlers, which catches out people who write one broad disallow and assume every bot is covered.

Your pages stop being recrawled, the index goes stale, and organic rankings decay over the following weeks. Nothing alerts you. The damage is slow and easy to misattribute, so by the time traffic has visibly fallen, nobody connects it to a firewall rule added a month earlier. Check crawler activity in your logs after any blocking change.