Rage against the machine

Published on Wednesday 26 June 2024.

Reading Jeremy Keith's call to arms, "The machine stops", I started to think about ways to not just prevent AI crawlers but to actively sabotage them. Idle Luddite thoughts of an evening.

There are those who say we shouldn't; that the "public" web is fair game. After all, it's freely available to anyone to access. But that's it; to access. As Louie says in Training AI,

The “public” in “public web” means free to access; it does not mean it's free to use.

My site is currently hosted on Netlify which doesn't give me access to server-level files like .htaccess or nginx.conf. This makes targeting bots by User-Agent difficult. Netlify's proprietary [[redirect]] syntax in a .toml file only allows redirecting a specific URI which is no use here.

I read of a way to route all traffic through an edge function for User-Agent sniffing but that is "rarely a good idea" and would come with a performance cost I'm not willing to inflict on human users for the sake of some crawlers.

If you do have the ability to detect bots at that level and want to properly mess with them, may I recommend either returning a 402: Requires payment status or redirecting to http://hil-speed.hetzner.com/10GB.bin which will download a 10gb file and clog the process somewhat.

It seems that these crawlers aren't the smartest tools in the shed though and, as Lewis discovered, can be manipulated with some prompt injection in the site content.

So, now I'm left with the idea of adding a secret bit of text on every page that will be returned instead of the actual content when a system susceptible to prompt injection tries to crawl my website. But what should that secret text say?