Blocking AI crawlers with Caddy

2 years ago by Deckweiss to c/linux

I was reading the reddit thread on Claude AI crawlers effectively DDOSing Linux Mint forums https://libreddit.lunar.icu/...

and I wanted to block all ai crawlers from my selfhosted stuff.

I don't trust crawlers to respect the Robots.txt but you can get one here: https://darkvisitors.com/

Since I use Caddy as a Server, I generated a directive that blocks them based on their useragent. The content of the regex basically comes from darkvisitors.

Sidenote - there is a module for blocking crawlers as well, but it seemed overkill for me https://github.com/Xumeiquer/nobots

For anybody who is interested, here is the block_ai_crawlers.conf I wrote.

(blockAiCrawlers) {
  @blockAiCrawlers {
    header_regexp User-Agent "(?i)(Bytespider|CCBot|Diffbot|FacebookBot|Google-Extended|GPTBot|omgili|anthropic-ai|Claude-Web|ClaudeBot|cohere-ai)"
  }
  handle @blockAiCrawlers {
    abort
  }
}

# Usage:
# 1. Place this file next to your Caddyfile
# 2. Edit your Caddyfile as in the example below
#
# ```
# import block_ai_crawlers.conf
#
# www.mywebsite.com {
#   import blockAiCrawlers
#   reverse_proxy * localhost:3000
# }
# ```

arisunz 66 points 2 years ago

I got meaner with them :3c

save

path: 0 9702243, hotness: undefined, score: 66, children: 10

JustARegularNerd 20 points 2 years ago

I just want you to know that was an amazing read, was actually thinking "It gets worse? Oh it does. Oh, IT GETS EVEN WORSE?"

save

path: 0 9702243 9704380, hotness: undefined, score: 20, children: 1

arisunz 4 points 2 years ago

lmao that means a lot, thanks <3

save

path: 0 9702243 9704380 9726706, hotness: undefined, score: 4, children: 0

Deckweiss 10 points 2 years ago

The nobots module I've linked bombs them

save

path: 0 9702243 9709093, hotness: undefined, score: 10, children: 0

winnie 9 points 2 years ago

Suggestion at the end:

  <a class="boom" href="https://boom .arielaw.ar">hehe</a>

Wouldn't it destroy GoogleBot (and other search engine) those making your site delisted from Search?

save

path: 0 9702243 9732456, hotness: undefined, score: 9, children: 0

ABasilPlant 8 points 2 years ago

In dark mode, the anchor tags are difficult to read. They're dark blue on a dark background. Perhaps consider something with a much higher contrast?

A picture of a website with a dark purple background and dark blue links.

Apart from that, nice idea - I'm going to deploy the zipbomb today!

save

path: 0 9702243 9729681, hotness: undefined, score: 8, children: 1

arisunz 5 points 2 years ago

nice catch, thanks (i use light mode most of the time)

save

path: 0 9702243 9729681 9736151, hotness: undefined, score: 5, children: 0

jkrtn 7 points 2 years ago

This is one of the best things I've ever read.

I'd love to see a robots.txt do a couple safe listings, then a zip bomb, then a safe listing. It would be fun to see how many log entries from an IP look like get a, get b, get zip bomb.... no more requests.

save

path: 0 9702243 9717978, hotness: undefined, score: 7, children: 0

pvq 5 points 2 years ago

I really like your site's color scheme, fonts, and overall aesthetics. Very nice!

save

path: 0 9702243 9725157, hotness: undefined, score: 5, children: 1

not_amm 2 points 2 years ago

I agree, it's readable and very cute!

save

path: 0 9702243 9725157 9725752, hotness: undefined, score: 2, children: 0

acockworkorange 4 points 2 years ago

That’s devilishly and deliciously devious.

save

path: 0 9702243 9732967, hotness: undefined, score: 4, children: 0

hollyberries 26 points 2 years ago

I'm a fan of hellpotting them.

save

path: 0 9703965, hotness: undefined, score: 26, children: 2

Para_lyzed 3 points 2 years ago

From your recommendation, I found a related project pandoras_pot that I am able to run in a Docker container, and seems to run more efficiently on my Pi home server. I now use it in my Caddyfile to redirect a number of fake subdomains and paths that are likely to be found by a malicious bot (of course all are excluded in my robots.txt for bots that actually respect it). Thanks for the recommendation!

save

path: 0 9703965 9735304, hotness: undefined, score: 3, children: 0

Cyber 2 points 2 years ago

Ooh, didn't know about that one... thanks

save

path: 0 9703965 9714464, hotness: undefined, score: 2, children: 0

Arghblarg 19 points 2 years ago

We should do more than block them, they need to be teergrubed.

save

path: 0 9697496, hotness: undefined, score: 19, children: 3

Deckweiss 12 points 2 years ago

Thats an easy modification. Just redirect or reverse proxy to the tarpit instead of abort .

I was even thinking about an infinitely linked data-poisoned html document, but there seemed to be no ready made project that can generate one at the moment. (No published data-poisoning techniques for plain text at all afaik. But there is one for images.)

Ultimately I decided to just abort the connection as I don't want my servers to waste traffic or CPU cycles.

save

path: 0 9697496 9697591, hotness: undefined, score: 12, children: 0

boredsquirrel 4 points 2 years ago

Such a cool person making the video available for download

save

path: 0 9697496 9698907, hotness: undefined, score: 4, children: 0

winnie 1 point 2 years ago

Your link has no article, and Video inside Flash file (swf) that itn't opening in 2024.

And I don't want to install Flash on my machine...

save

path: 0 9697496 9732348, hotness: undefined, score: 1, children: 0