Cloudflare has a new feature—available to free users as well—that uses AI to generate random pages to feed to AI web crawlers: > Instead of simply blocking bots, Cloudflare’s new system lures them into a > “maze” of realistic-looking but irrelevant pages, wasting the crawler’s > computing resources. The approach is a notable shift from the standard > block-and-defend strategy used by most website protection services. Cloudflare > says blocking bots sometimes backfires because it alerts the crawler’s > operators that they’ve been detected. > > “When we detect unauthorized crawling, rather than blocking the request, we > will link to a series of AI-generated pages that are convincing enough to > entice a crawler to traverse them,” writes Cloudflare. “But while real > looking, this content is not actually the content of the site we are > protecting, so the crawler wastes time and resources.”...