Guarding My Git Forge Against AI Scrapers

10 months ago by tofu to c/selfhosted

A summary of the techniques in place to protect my git forge
IanTwenty 21 points 10 months ago

Self-hosting anything that is deemed "content" openly on the web in 2025 is a battle of attrition between you and forces who are able to buy tens of thousands of proxies to ruin your service for data they can resell.

This is depressing. Profoundly depressing.

Sigh

path: 0 20983417, hotness: undefined, score: 21, children: 0
magic_smoke 11 points 10 months ago

I would like to know how well iocaines spanky new redirection module works compared to Anubis.

If nothing else, to see if throwing Anubis in front of iocaine is still a worthwhile idea.

path: 0 20979533, hotness: undefined, score: 11, children: 0
xyro 11 points 10 months ago

Good read, I use crowdsec to block most of the traffic considered as malicious (which tend to overlap with scrappers), but I should look into Locaine to feed garbage to the remaining ones instead of throttling.

path: 0 20982764, hotness: undefined, score: 11, children: 1
JustTesting 6 points 10 months ago

it's iocaine not Locaine, tripped me up at first as well.

path: 0 20982764 20991420, hotness: undefined, score: 6, children: 0
Selfhoster1728 3 points 10 months ago

Build your own captcha, there's just no other way to be sure it's human traffic with prebuilt solutions :(

path: 0 20993398, hotness: undefined, score: 3, children: 0
Lemmchen 2 points 10 months ago path: 0 21160922, hotness: undefined, score: 2, children: 0
False 1 point 10 months ago
path: 0 20979084, hotness: undefined, score: 1, children: 0
possiblylinux127 0 points 10 months ago

Honesty we need a POW system built into http.

path: 0 20983410, hotness: undefined, score: 0, children: 5
DaGeek247 5 points 10 months ago

POW built in to the web spec would be hell. Making every single device in the world do that extra bit of work would noticeably affect energy use across the planet.

path: 0 20983410 20989204, hotness: undefined, score: 5, children: 0
tofu 3 points 10 months ago

A what? Not prisoners of war I guess

path: 0 20983410 20985259, hotness: undefined, score: 3, children: 3
billhead 4 points 10 months ago

I'm guessing Proof of Work.

path: 0 20983410 20985259 20985914, hotness: undefined, score: 4, children: 2
tofu 2 points 10 months ago

Oh right. The article author isn't a fan of that. I guess it's fine while it works but I'm not too optimistic about how long it does

path: 0 20983410 20985259 20985914 20986807, hotness: undefined, score: 2, children: 1
DaGeek247 2 points 10 months ago

Yeah. The corporations with money are always going to beat the casual users without in regards to processing capability.

There are smarter ways to discourage the big companies from taking pictures of your house than by adding speed bumps to your driveway.

path: 0 20983410 20985259 20985914 20986807 20989217, hotness: undefined, score: 2, children: 0
selfhosted
selfhosted

@lemmy.world

login for more options
62952
6792
5617

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

  1. Be civil.

  2. No spam.

  3. Posts are to be related to self-hosting.

  4. Don't duplicate the full text of your blog or readme if you're providing a link.

  5. Submission headline should match the article title.

  6. No trolling.

  7. Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.

  8. AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

go to feed...