Is it worth blocking click fraud IPs at the server at all?

Yes, but for security rather than ad spend. Firewalls are the correct tool against scraping, brute force attempts and crawlers hammering your infrastructure, and they have no entry limit or campaign scoping. Just do not count them as click fraud protection. Run both layers for different jobs and judge each on the problem it was actually built for.