Cloudflare blocking Pale Moon and other browsers with smaller user bases
2 years ago by dantheclamman to c/technology
I get why you're frustrated and you have every right to be. I'm going to preface what I'm going to say next by saying I work in this industry. I'm not at Cloudflare but I am at a company that provides bot protection. I analyze and block bots for a living. Again, your frustrations are warranted.
Even if a site doesn't have sensitive information, it likely serves a captcha because of the amount of bots that do make requests that are scraping related. The volume of these requests can effectively DDoS them. If they're selling something, it can disrupt sales. So they lose money on sales and eat the load costs.
With more and more username and password leaks, credential stuffing is getting to be a bigger issue than anyone actually realizes. There aren't really good ways of pinpointing you vs someone that has somehow stolen your credentials. Bots are increasingly more and more sophisticated. Meaning, we see bots using aged sessions which is more in line with human behavior. Most of the companies implementing captcha on login segments do so to try and protect your data and financials.
The rise in unique, privacy based browsers is great and it's also hard to keep up with. It's been more than six months, but I've fingerprinted Pale Moon and, if I recall correctly, it has just enough red flags to be hard to discern between a human and a poorly configured bot.
Ok, enough apologetics. This is a cat and mouse game that the rest of us are being drug into. Sometimes I feel like this is a made up problem. Ultimately, I think this type of thing should be legislated. And before the bot bros jump in and say it's their right to scrape and take data it's not. Terms of use are plainly stated by these sites. They consider it stealing.
Thank you for coming to my Tedx Talk on bots.
Edit: I just want to say that allowing any user agent with "Pale Moon" or "Goanna" isn't the answer. It's trivially easy to spoof a user agent which is why I worked on fingerprinting it. Changing Pale Moon's user agent to Firefox is likely to cause you problems too. The fork they are using has different fingerprints than an up to date Firefox browser.
Dude, thank you for this context. I was already aware of these considerations but just wanted to thank you for sharing this with everyone. Its participation like this that makes the internet a better place. 🍻
That's very kind of you. Thank you for the kind words. 🍻
Thank you for reading and considering the information.
Also Cloudflare adds a caching layer, often physically closer to users. Increasing speed of delivery and reducing server costs. It's a no-brainer for server admins.
Also, I don't work for Cloudflare either. The animosity is new to me, and certainly something I'll look into.
But captchas have now proven useless, since bots are better at solving them now than humans?
Welcome to bot detection. It's a cat and mouse game, an ever changing battle where each side makes moves and counter moves. You can see this with the creation of captcha-less challenges.
But to say captcha are useless because bots can pass them is somewhat similar to saying your antivirus is useless because certain malware and ransomware can bypass it.
But they are better than humans at solving them.
Thanks for reading and commenting!
During my first (shitty) job as a dev outta school, they had me writing scrapers. I was actually able to subvert it pretty easily using this package that doesn't appear to be maintained anymore https://github.com/VeNoMouS/cloudscraper
Was pretty surprised to learn that, at the time, they were only checking if JS was enabled, especially since CF is the gold standard for this sort of stuff. I'm sure this has changed?
You're definitely right that it's a game of one-upping each other. Unfortunately, it's now directed in a path that infringes on privacy of the users it aims to serve.
Since you're working in the internet security industry, what's your take on something like Altcha as opposed to more invasive means of protecting against both attacks?
Trust me, my team and I often feel at odds with the part that infringes on privacy. As someone that enjoys and wants more privacy, I wish there were other solutions that didn't create a type of dragnet. If it assuages some of your fears, I've never heard of the fingerprinting being sold or used outside of detections.
ALTCHA uses a proof-of-work mechanism to protect your website, apps, APIs, and online services from spam and unwanted content.
Unlike other solutions, ALTCHA’s Captcha alternative is free, open-source and self-hosted, does not use cookies nor fingerprinting, does not track users.
Emphasis are mine. I honestly do not know how this statement is possible. Captcha-less, proof-of-work solutions have to fingerprint on some level. It's essentially having the browser prove it is what it claims to be. I get what they're trying to say but it's marketing. That said, I don't know everything and maybe they have some method I'm not aware of. Grains of salt all around.
I definitely understand. That's good to hear there hasn't been a direct pipeline to selling fingerprint data established yet.
Thanks for checking it out. Hopefully there is a best of both worlds in what they are advertising but I get that technology isn't magic either.
Terms of use are plainly stated by these sites. They consider it stealing.
I consider it more trespassing than stealing myself.
Ever heard of counting attempts? Log the IP, present a CAPTCHA after 100 requests in a minute.
Besides, if I wrote a bot I would run a browser dialer from Chrome. It would request your site in a Chrome tab and appear completely legitimate to your stupid fingerprinting scripts
Ever heard of counting attempts? Log the IP, present a CAPTCHA after 100 requests in a minute.
Ever heard of IP rotation? This is one malicious source rotating through IPs over the course of 24 hours. They're attempting to credential stuff my logins ( on a production service ).
Yes, the industry is well aware of this. We do behavioral detection on both sessions and IPs. This is fairly basic.
Yeah, it's fine as long as you don't block legitimate users. For example, when I use a VPN a lot of sites block me. Even when my actual IP is banned when I'm in China (4chan range bans Chinese IPs) or the website is blocked in China.
Tilde.teams and tilde.club even have outwardly facing email accounts.
We have a newsgroup server.
We have a dedicated irc server.
Member gopher/https/gemini pages.
And other services.
And each tilde has it's own focus.
Be kind. Contribute as you can to discussions.
What is gemini
Rocking the web bloat.
Be Free.
It's not much, but it's home. :)
Very happy to have learned about tildeverse.org. It gives a great refuge from a lot of the garbage. But I have to ask, if you are part of the tildeverse project why are you on lemmy.world, a centralised exclusive Cloudflare instance? I realise tildeverse does not have a lemmy node but there are many decent ones and LW is probably the most contrary to the tildeverse philosophies.
Wouldn't that also block Firefox by proxy?
And so far LibreWolf and Icecat have both worked fine for me.
Ever been down the gemini rabbit hole? It's not perfect, but quite interesting.
These bastards haven’t MITMed half the internet for nothing. This isn’t the first time they abuse that either.
I hate that I once fell for it too when I just started out hosting stuff and put it behind their proxy.
And when Cloudflare is the proxy for a web site, it's Cloudflare that provides the HTTPS connection, meaning that you don't actually have an encrypted channel directly to the site. Cloudflare is the man-in-the-middle eavesdropping on all of your communications with that site. Your bank transactions, your medical records, your personal messages, etc.
Interesting. I'm going to keep this in mind.
Weird how much of a monopoly cloudflare has on the internet. I guess it's going to start being an indicator for me for services that have becomes "too big for their britches."
Its kind of funny but thats how user agents have been for a while. It's historically just been browsers pretending to be one another.
Yeah and that's why it's one of the basics of the basics you learn as a software developer that you shouldn't sniff the useragent, because it's unreliable and causes issues. Yet all big webpages (especially those pretending to be a software) do it, causing issues. Even just trimming the useragent string (xorigin.trimming.policy) makes "advanced services" like a webshop unusable.
Just don't do useragent sniffing, do feature detection instead.
It is obvious that Cloudflare is being influenced to enforce browser monopolies. Imagine if Cloudflare existed in 2003 and stopped non Internet Explorer browsers. If you use cloudflare to "protect" your site you are discriminating against browser choice and are as bad as Microsoft in 1998.
Agreed. I use cloudflare for domain hosting because they're cheap, but I have never liked their protections.
I can't use my Browser without it being created by a tech giant, cant use my new computer without having my software uefi signed by Microsoft, AI will soon need me to have my GPU licensed and registered.
The world is heading to crap.
You can, it'll cost more and give you less, but you can.
That's the way this works.
You can always build a PC and not have to deal with that UEFI signing stuff as you're expected to provide your own OS still, that option hasn't been eliminated yet.
Also, AMD cards are more friendly to Linux users than Nvidia cards are, even with the existence of NVK for the latter; NVK only supports Turing and newer cards and Maxwell, Pascal, and Volta are too old for it, and since Nouveau is broken on Maxwell and newer by firmware signing, once those cards lose support in the proprietary drivers, unless NVK gets backported to them somehow, you'll be SOL in the near future for 900-series, 10-series, and the Titan V, while Kepler and older is still supported by Nouveau, meanwhile over at AMD, Mesa actively supports Radeon cards going back to GCN1.
Basically, if you still have an R9 Fury or an RX 580 sitting around, for example, those cards will still be actively supported by Mesa open drivers for the foreseeable future, meanwhile your GTX 980Ti or 1080Ti, at least currently, are fully at the mercy of Nvidia's closed drivers.
I would be very interested to know how they plan to resolve these issues with "Ladybird." Using a new engine will likely clash with the FALSE "security measures" of many websites and harm the browsing experience. It’s often said that users should demand respect for web standards, but in the meantime, as usability declines, users will gradually drift away. Firefox learned this lesson the hard way.
Servo is another wip web browser, managed by the Linux foundation's European branch. It's a little less far along but is making relatively quick progress now. Apparently discord already mostly works, with sending messages currently being a problem.
So make useragent sniffing useless by all being Chrome?
Funnily enough, some webpages work better if you block webgl and set the user agent to Lynx or Dillo.
What doesn't work with Lynx is a wrong website.
Agree for static content like news and blogs. Disagree for dynamic content like games and social media. And the latter is mostly for scale (having server-side templating is expensive for rapidly changing content).
Then again, there's a case for snapshotting SM pages every so often for things like crawlers and cli browsers.
I was planning on moving away from Cloudflare to European providers anyway, so this just adds fuel to the fire.
I'm considering using BunnyDNS for DNS management, not using a CDN at all, and using Scaleway for serverless functions.
I appreciate the suggestion, but Github is also an American company. I've been moving my git repositories to Codeberg.
My sites don't get enough traffic to warrant a CDN really, but if necessary, BunnyCDN looks like it can fit the bill. Plus, my static sites are in Scaleway object storage.
I just duplicated this. I downloaded Pale Moon and went to https://hear-me.social and clicked on "Register". It puts up a Cloudflare "managed challenge" which loops endlessly when using Pale Moon, but not the other browsers I've tried it with, including Zen, another Firefox fork.
It's a problem, for sure.
As a staunch Pale Moon user, Cloudflare is just being a bully and I circumvent their nonsense when I need to desperately use a particular site or just don't go to that site anymore if I can do without.
Greed. I honestly don't know if they're even aware of the problem. Most corporations have cut teams to the bone and I can't see Cloudflare being an exception. The janitor is probably writing detection rules now.
Pale Moon still exists? Huh
Should change my user agent to sod off
I feel like I remember reading that the pale moon JavaScript engine was broken and causing the capcha to break repeatedly?
Let me see if I can find sources
EDIT: Looks like I was remembering a previous issue where the captchas were causing the entire pale moon browser to crash. I believe this has been fixed, but the new issue is a much less exciting block.
I wonder what happens if you use Pale Moon but set the user agent to Firefox.
Another comment suggested that helped with LibreWolf, but that is a closer fork than Pale Moon, so not sure
You can go to https://hear-me.social and click on the register button. This puts up a Cloudflare managed challenge screen which endlessly loops when using Pale Moon. It would be interesting to see if Waterfox has the same issue.
Took a minute and a refresh, but it worked on Ironfox on android.
That's a shit take. What's the point of having user-agents if it's just a race to the bottom for only supporting a smaller list arbitrarily? It's not like the bots aren't going to just spoof as Chrome on Windows 11 anyways.
Yes
Yes its should.
I just won't use cloudflare, that's fine.
That's good in theory, but a site behind Cloudflare won't necessarily notice that a legitimate user got blocked. If you want them to care, you'll have to find a way to contact them. For more impact, tell them which competitor you spent money with instead.
I mean the criticism against Cloudflare is 100% valid. Having a single service be the single point of failure for half the web, not to mention that they can read the contents of every single request they proxy, is a terrible joke.
But the service they provide is real. A small business/service just doesn't have the capabilities to handle a DDoS attack. And every minute their site is down means lost customers/users.
That’s a good bit of small business which is already pretty aware of the boot they are under.
Then you won’t browse about 20% of the Internet, which doesn’t sound like a lot but it’s disproportionately impacting sites you would generally want to browse
I posted to this effect in a Firefox alternatives thread: if you use an alternative low adoption rate FOSS browser you trade increased privacy via less/no data harvesting for decreased privacy via much higher susceptibility to browser fingerprinting by google/meta/etc. doesn’t matter if you resize your windows if your browser reports its one that only 5,000 people use. And something tells me the tech giants have a way around user agent spoofing
And now even if you don’t care about that? Fuck you. Cloudflare locks you out of the modern internet because of course anyone not using chrome or safari is a bot
I have pretty draconian privacy protections on my devices and home network. It makes the internet hostile. Captchas regularly fail and I have to try them many times. Embedded youtube videos always think I am a bot and refuse to play unless I sign in, I get weird interstitial pages with captchas on google search, yandex, etc (kagi and searx don’t so I use searx), etc.
Advertisers have pushed companies to make the internet openly hostile to anyone who wants to maintain privacy. And to be clear google and meta are advertisers first and foremost. Fuck them
I have given up hiding from the tracking. Instead flood them with a torrent of bullshit data. AdNauseam, click on all the adverts. If the internet is going to be hostile then I shall be actively malicious to it in response.
That's analogous to saying you won't call any numbers on certain carrier
It's possible, but your overall service is devalued if you can't connect to a large group of people.
@lemmy.world
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
go to feed...
@lemmy.world
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
go to feed...
Disgusting and unsurprising.
Most web admins do not care. I've lost count of how many sites make me jump through CAPTCHAS or outright block me in private browsing or on VPN. Most of these sites have no sensitive information, or already know exactly who I am because I am already authenticating with my username and password. It's not something the actual site admins even think about. They click the button, say "it works on my machine!" and will happily blame any user whose client is not dead-center average.
Enter username, but first pass this CAPTCHA.
Enter password, but first pass this second CAPTCHA.
Here's another CAPTCHA because lol why not?
Some sites even have their RSS feed behind Cloudflare. And guess what that means? It means you can't fucking load it in a typical RSS reader. Good job!
The web is broken. JavaScript was a mistake. Return to monke gopher.
Fuck Cloudflare.
save