Recent Outages

3 years ago by lwadmin to c/lemmyworld

Hey everyone, so as I'm sure everyone is aware Lemmy.World has been experiencing several outages throughout the last few days.

We have been investigating the root cause of these outages but believe that they are related to our current hosting provider (Hetzner) blocking access from ClouldFlare as (we think) they believe that our CDN is a DDoS'er, and is causing these disconnects to our backend server, problematic for sure.

We've opened support tickets with our current provider and are awaiting a response. We have no issue with being as transparent as possible with downtime. Anyone that is curious, can feel free to check out https://status.lemmy.world and https://dash.lemmy.world for up to the minute outage information. We are also looking into other fediverse friendly methods of posting status and outage updates

In the meantime, we are evaluating alternative hosting options and solutions to provide a high level of reliability to you, our users. Really, we want to say thanks to everyone for soldiering through all our technical growing pains.

Cheers

LW Infra Team

Cthuwu 267 points 3 years ago

As always, the transparency is appreciated. Some growing pains are certainly to be expected

path: 0 2762327, hotness: undefined, score: 267, children: 10

gonzoknowsdotcom1 57 points 3 years ago

path: 0 2762327 2763798, hotness: undefined, score: 57, children: 9

ignotum 12 points 3 years ago

Key is metal

path: 0 2762327 2763798 2768729, hotness: undefined, score: 12, children: 8

gravitas_deficiency 10 points 3 years ago

Metal is conductive

path: 0 2762327 2763798 2768729 2769218, hotness: undefined, score: 10, children: 3

G_Wash1776 4 points 3 years ago

Conductive is electric

path: 0 2762327 2763798 2768729 2769218 2771301, hotness: undefined, score: 4, children: 2

coco 2 points 3 years ago

Transparent Anuminium !!!

path: 0 2762327 2763798 2768729 2769745, hotness: undefined, score: 2, children: 1

jasondj 1 point 3 years ago

Xenonite?

path: 0 2762327 2763798 2768729 2769745 2771547, hotness: undefined, score: 1, children: 0

dot20 2 points 3 years ago

So, by transitivity, transparency is metal?

path: 0 2762327 2763798 2768729 2769133, hotness: undefined, score: 2, children: 1

ignotum 6 points 3 years ago

We had an aquarium with transparent fishes, where you could see all their insides, that was pretty metal

So yeah i think we're onto something here

path: 0 2762327 2763798 2768729 2769133 2769257, hotness: undefined, score: 6, children: 0

Boozilla 237 points 3 years ago

Whenever I get frustrated by the outages I remind myself: still better than reddit.

path: 0 2768087, hotness: undefined, score: 237, children: 15

Shady_Shiroe 89 points 3 years ago

Thanks to lemmy I don't doom scroll anymore because either there is an outage or I read all the new content I'm subscribed to in 10-20 minutes

path: 0 2768087 2771114, hotness: undefined, score: 89, children: 4

GONADS125 30 points 3 years ago

That's mostly how I redditted for years. It was mostly for those moments in between things, on the toilet, laying in bed at night. Not something I did for long periods of time.

path: 0 2768087 2771114 2772421, hotness: undefined, score: 30, children: 2

chulo_sinhatche 12 points 3 years ago

Look at this guy with his reasonable social media use. YOU THINK YOU'RE BETTER THAN ME??

path: 0 2768087 2771114 2772421 2791567, hotness: undefined, score: 12, children: 1

ethman42 7 points 3 years ago

Scrolling a few miles or so a day isn’t too bad, is it?

path: 0 2768087 2771114 2772421 2791567 2793914, hotness: undefined, score: 7, children: 0

kratoz29 8 points 3 years ago

Huh, I could keep up with my daily feeds before, now not so much.

path: 0 2768087 2771114 2772074, hotness: undefined, score: 8, children: 0

infyrin 46 points 3 years ago

path: 0 2768087 2768571, hotness: undefined, score: 46, children: 7

NocturnalMorning 27 points 3 years ago

It's your fault for not pitching in and making lemmy more stable. There, happy, you've been blamed :P

path: 0 2768087 2768571 2768640, hotness: undefined, score: 27, children: 6

grue 10 points 3 years ago

It’s your fault for not pitching in and making lemmy more stable.

Plot twist: unlike Reddit, with Lemmy that would be a legitimate option! Ain't Free Software and federated networks grand?

path: 0 2768087 2768571 2768640 2769744, hotness: undefined, score: 10, children: 0

shastaxc 6 points 3 years ago

What is the burden if I wanted to host a node and limit access just to myself? Is it just a portal into the rest of the fediverse or is there a large maintenance burden or storage requirement?

path: 0 2768087 2768571 2768640 2776036, hotness: undefined, score: 6, children: 4

maor 4 points 3 years ago

Yeah, pretty big storage requirement due to the way pictrs works. Pictrs is the piece of software Lemmy relies upon to manage image storage, uploads, and most importantly: caches pictures from other instances. This takes up a HUGE amount of storage space, and there's no official way to clear this up, see these posts I recently made: first one, second one. The solution I resorted to is renting a 1TB storage box from Hetzner for 3 euros per month, pretty sweet deal but I was kinda annoying by it. So the cheapest deal I could find costs me 6 euros per month: 3 for an Alma Linux ARM VPS from Hetzner, and 3 for that storage box. If you're in for the fun in tinkering (I sure as hell am in), then get ready for a good time. Other than that, if your main line of reasoning is to take burden off of lemmy.world, then I think just go ahead and join another instance. Better yet: join croud funding of another instnace:)

path: 0 2768087 2768571 2768640 2776036 2791779, hotness: undefined, score: 4, children: 3

malloc 9 points 3 years ago

I fallback to one of the many instances that federate with lw. Don’t get frustrated at all bro

path: 0 2768087 2791287, hotness: undefined, score: 9, children: 0

Swim 6 points 3 years ago

Yup

path: 0 2768087 2770643, hotness: undefined, score: 6, children: 0

Machefi 80 points 3 years ago

path: 0 2764643, hotness: undefined, score: 80, children: 2

lwadmin 80 points 3 years ago

You need to hover over the status bar to see if there is any down time for that day. We can enable it to log incidents every time there is a burp, but we are still tuning alerts as we only have it create a incident when we ACK it in PagerDuty. You can always check the dashboard for up to the minute stats, as well as https://lemmy-status.org/endpoints/_lemmy-world We'll add this info to make things clearer <3

EDIT: Added more info to our status page, thanks for the feedback Machefi!

EDIT2: Also the missing data is due to us removing and adding more specific monitors for the different infra services.

path: 0 2764643 2765459, hotness: undefined, score: 80, children: 1

obinice 23 points 3 years ago

Excuse me stop being so cool, you're raising the bar too high for everyone else thank you

path: 0 2764643 2765459 2775217, hotness: undefined, score: 23, children: 0

RanchOnPancakes 80 points 3 years ago

Maybe its just the times I'm accessing but its seem better this week to me compared to the last few ones.

path: 0 2766547, hotness: undefined, score: 80, children: 0

iforgotmyinstance 43 points 3 years ago

Good luck, no pressure.

path: 0 2762300, hotness: undefined, score: 43, children: 0

perfectra1n 41 points 3 years ago

On your Cloudflare account, if there was a change in the CNAME/A record being proxied vs. DNS only, that could cause an issue, as Cloudflare would then strip headers off the request that your Apache/Nginx would be looking for.

If you enabled HTTP DDoS protection in your Security -> WAF tab (I think that’s where it is) that could do this too. Might be worth disabling.

Also check for any headers your HTTP load balancer might be expecting, that Cloudflare could be stripping.

Might be worth tailing the webserver logs to see what happens to requests coming in from Cloudflare.

path: 0 2765042, hotness: undefined, score: 41, children: 0

Dagnet 37 points 3 years ago

Thanks for keeping us up to date, I don't mind downtime personally as long as I'm aware of what's going on

path: 0 2762313, hotness: undefined, score: 37, children: 0

drewthejedi 36 points 3 years ago

Everyone talking about the downtime including lemme and me just enjoying lemmy like never before. I've experienced no downtime so far

path: 0 2788681, hotness: undefined, score: 36, children: 3

malloc 17 points 3 years ago

The beauty of a federated app is that I do not have to rely on lw or lemme. Use another instance and go on with my day 😃

path: 0 2788681 2791264, hotness: undefined, score: 17, children: 0

TairikuOkami 1 point 3 years ago

path: 0 2788681 2795914, hotness: undefined, score: 1, children: 1

RisingSwell 6 points 3 years ago

Down for you is not the same as down. If it's up when you aren't using it, it's still up

path: 0 2788681 2795914 2807784, hotness: undefined, score: 6, children: 0

mvirts 30 points 3 years ago

So what you're saying is we were the ddosers all along 😹

path: 0 2762641, hotness: undefined, score: 30, children: 2

PhobosAnomaly 35 points 3 years ago

The hack is coming from inside the instance!

path: 0 2762641 2764105, hotness: undefined, score: 35, children: 1

PutangInaMo 1 point 3 years ago

Ironically true lol

path: 0 2762641 2764105 2808052, hotness: undefined, score: 1, children: 0

BellaDonna 26 points 3 years ago

Let me be real. I never noticed outages stopping. It feels like it's daily, I'm used to it, but I think it happens so often that lemmy.world has lost its growth opportunity, and we alienated the normies. I'm still going to stay on Lemmy, and I believe you're doing the best you can, but we lost for the time being, the migration to Lemmy from Reddit is stunted.

path: 0 2762900, hotness: undefined, score: 26, children: 14

danielton 65 points 3 years ago

path: 0 2762900 2763450, hotness: undefined, score: 65, children: 11

gabe 11 points 3 years ago

Next time the lemmy join page needs to be improved so people can spread and don't try to centralize into a single instance and break the purpose of lemmy in the first place.

path: 0 2762900 2763450 2764260, hotness: undefined, score: 11, children: 9

danielton 24 points 3 years ago

path: 0 2762900 2763450 2764260 2764497, hotness: undefined, score: 24, children: 8

gsa32 10 points 3 years ago

Lemmy.world itself is bigger than Squabblr and Discuit combined

path: 0 2762900 2763450 2764260 2764497 2766433, hotness: undefined, score: 10, children: 1

Blaze 5 points 3 years ago

Squabblr and Discuit

Discuit is a place where 4142 people get together to find cool stuff and discuss things.

Squabblr doesn't have a count of active users (33k registered users)

I know we are low on the numbers, but still a bit higher than them.

path: 0 2762900 2763450 2764260 2764497 2765547, hotness: undefined, score: 5, children: 5

xtremeownage 9 points 3 years ago

Partly due to the fact..... lemmy itself, basically has no moderation or administration features at all....

So, the only way to assist with that issue, is stricter enforcement up-front.

Besides, if someone doesn't wanna take the time to have a verified email, and literally type 49 when registering an application.... I really don't wanna take the time to worry about having to potentially worry about them being spammers/etc.

path: 0 2762900 2763450 2774164, hotness: undefined, score: 9, children: 0

PM_ME_YOUR_ZOD_RUNES 8 points 3 years ago

We shouldn't be trying to grow a single instance. That defeats the whole point of Lemmy. I started on Lemmy world and switched once I got fed up with the constant connection issues. Plus, Lemmy world blocked piracy communities so fuck that. I'm happy that I am able to quickly create an account on another instance.

path: 0 2762900 2769553, hotness: undefined, score: 8, children: 0

xtremeownage 0 points 3 years ago

I never noticed outages stopping

I am in the same boat. If it were not for these posts, I'd have never noticed lemmy world was down.

If I post to, or put a comment on something from lemmy world, it will just federate over when it's back online.

If someone posts to something on lemmy world, it will eventually federate over my way.

But, hey, everyone is gonna fuss when they decided to put all of their eggs in one basket and now, that one basket gets targeted. (metaphor for lemmy world.)

path: 0 2762900 2774134, hotness: undefined, score: 0, children: 0

Shadywack 23 points 3 years ago

Only thing that needs to be said here is: thank you for the best Lemmy instance (in my opinion) out there!

path: 0 2768732, hotness: undefined, score: 23, children: 0

webhead 20 points 3 years ago

Could look into Dacentec if you need more cheap servers. I use them for my stuff. YMMV since you're getting a hell of a lot more traffic than I am but they haven't blocked Cloudflare on mine yet so that's a plus. :)

path: 0 2772428, hotness: undefined, score: 20, children: 0

TropicalDingdong 18 points 3 years ago

Hetzner is a pretty good provider. It would be nice to stick with them if possible.

path: 0 2762285, hotness: undefined, score: 18, children: 3

superbirra -4 points 3 years ago

lol

path: 0 2762285 2771674, hotness: undefined, score: -4, children: 2

RagingToad 2 points 3 years ago

Care to elaborate?

I'm a hetzner user for years and if they are bad I haven't noticed yet.

path: 0 2762285 2771674 2815464, hotness: undefined, score: 2, children: 1

superbirra 0 points 3 years ago

nah lol

path: 0 2762285 2771674 2815464 2873002, hotness: undefined, score: 0, children: 0

olafurp 16 points 3 years ago

I brows all, never noticed as outage lol.

path: 0 2801179, hotness: undefined, score: 16, children: 0

Cold_Brew_Enema 12 points 3 years ago

You guys rule. Thanks for all the hard work.

path: 0 2764499, hotness: undefined, score: 12, children: 0

Tolstoshev 10 points 3 years ago

Is there anything we can do to help? Donations? Tech volunteers? Visit hosting company with a baseball bat?

path: 0 2764219, hotness: undefined, score: 10, children: 2

XYZinferno 6 points 3 years ago

I imagine spreading out across instances, and having accounts in instances beyond the biggest one helps, as it reduces traffic and strain on lemmy.world and its servers.

lemmy.world still deserves support, like with donations, though helping reduce the strain is something anyone can do, albeit small, though free of charge.

path: 0 2764219 2766615, hotness: undefined, score: 6, children: 0

clueless_stoner 6 points 3 years ago

path: 0 2764219 2793398, hotness: undefined, score: 6, children: 0

FlyingSquid 8 points 3 years ago

Thanks for the transparency. I think you lost some trust with the Discord debacle, but this makes me feel a lot better about things.

path: 0 2762363, hotness: undefined, score: 8, children: 0

gabe 8 points 3 years ago

The transparency is appreciated.

path: 0 2764272, hotness: undefined, score: 8, children: 0

NickCudawn 5 points 3 years ago

So the recent reports about ddos attacks were false alarms?

path: 0 2764481, hotness: undefined, score: 5, children: 2

Geert 4 points 3 years ago

This is about what's happening the last few days I think? Lemmy World has been a lot more stable compared to even a week ago but still has some daily downtime. But not for hours every day anymore

path: 0 2764481 2768018, hotness: undefined, score: 4, children: 0

kamenlady 2 points 3 years ago

False positives

path: 0 2764481 2773456, hotness: undefined, score: 2, children: 0

The_Picard_Maneuver 5 points 3 years ago

Thanks for keeping things running as Lemmy grows!

For what it's worth, I'm on all the time and have barely noticed the outages.

path: 0 2762791, hotness: undefined, score: 5, children: 1

Geert 3 points 3 years ago

I think it got A LOT better the last week or so

path: 0 2762791 2767945, hotness: undefined, score: 3, children: 0

DelvianSeek 4 points 3 years ago

Thanks for the update!

path: 0 2762869, hotness: undefined, score: 4, children: 0

Blaze 4 points 3 years ago

Thanks for sharing the dash link, and for your work!

path: 0 2762918, hotness: undefined, score: 4, children: 0

landlordlover 4 points 3 years ago

Thanks for the updates and transparency.

path: 0 2762801, hotness: undefined, score: 4, children: 0

TurnItOff_OnAgain 4 points 3 years ago

The dashboard requires a login to view. At least from here it does. Is there a way to view without making an account?

path: 0 2768895, hotness: undefined, score: 4, children: 2

KazuyaDarklight 4 points 3 years ago

I was able to access it without a login just now.

path: 0 2768895 2770431, hotness: undefined, score: 4, children: 1

TurnItOff_OnAgain 1 point 3 years ago

Seems to work now. Maybe it was a setting on their side?

path: 0 2768895 2770431 2782655, hotness: undefined, score: 1, children: 0

thantik 3 points 3 years ago

I mean, I'd help if I didn't have to have the educational background of a rich upper class white man. Teach me what I need to know and I'll sysop. But it looked like you had lots of people already come forward with experience "serving at the edge", etc. All I've done is play with Kubernetes and Proxmox at home on a residential IP with a domain pointed at my residence. Unfortunately, with no college education, I've had to teach myself all I know and it probably isn't up to what's wanted. I've heard "A bad sysop can be worse than just having nobody", so :(

path: 0 2767156, hotness: undefined, score: 3, children: 4

remotelove 9 points 3 years ago

I quit highschool and was force-fed some kind of formal education in the Navy. I think I have an associates degree of some kind? I dunno and I personally don't care if I do.

However, I built my career doing what I love and that was doing anything in IT... I settled on IT Security and just kept getting real world experience. Proving your skills and abilities can be a challenge at first, but its no different than anyone else fresh out of college.

Don't sell yourself short because you didn't go to college. I likely saved thousands of dollars and years of my life by actually learning and applying useful skills. Many people in this world have done the same.

College is not for everyone. As a matter of fact, I tend to stear away from candidates during interviews that have just memorized everything and don't know how to apply it. Case in point, I spent a week trying to teach someone how to troubleshoot a problem only to find out he had no idea how TCP and UDP actually worked. (I am a hands off teacher. I explain a problem and the concepts needed to fix it, but it is up to that person to do the leg work.) Sure, he could quote what he was told to memorize at school, but for many tasks he is basically useless. Ugh.

If someone reading this has gone to college and enjoyed it, awesome! Good on ya! You learn differently than I do.

path: 0 2767156 2768789, hotness: undefined, score: 9, children: 0

Geert 2 points 3 years ago

You can also help by posting content and reporting people that break LW rules. You don't need to be a tech wizard to help out.

path: 0 2767156 2788648, hotness: undefined, score: 2, children: 2

thantik 1 point 3 years ago

Being a sysop and reporting mean comments are two WILDLY different things.

path: 0 2767156 2788648 2790361, hotness: undefined, score: 1, children: 1

Geert 2 points 3 years ago

I'm just saying there are plenty of things you can do if you're not technical.

path: 0 2767156 2788648 2790361 2814072, hotness: undefined, score: 2, children: 0

psion1369 2 points 3 years ago

Try Liquid Web for the hosting.

path: 0 2771137, hotness: undefined, score: 2, children: 0

badcommandorfilename 2 points 3 years ago

I just wanted to say that Lemmy.world has been much more stable for me lately, thanks!

I know a lot of people are putting in hard work to get here, so I hope you know your effort is being noticed.

path: 0 3000820, hotness: undefined, score: 2, children: 0

pdnq 2 points 3 years ago

Cloud is not an option?

path: 0 2764476, hotness: undefined, score: 2, children: 2

SheeEttin 5 points 3 years ago

That is the cloud.

path: 0 2764476 2772371, hotness: undefined, score: 5, children: 1

pdnq 1 point 3 years ago

When one mentions "Hetzner", it doesn't immediately evoke cloud services in the same vein as AWS, GCP, or Azure. For instance, while SAP offers "cloud" solutions, it might equate to a single server in Hannover. I hope this clarifies the context of my question :)

path: 0 2764476 2772371 2851078, hotness: undefined, score: 1, children: 0

DefinitelyNotBirds 1 point 3 years ago

path: 0 2823507, hotness: undefined, score: 1, children: 0

Zuberi1 0 points 3 years ago

path: 0 2805408, hotness: undefined, score: 0, children: 0

Zuberi1 -5 points 3 years ago

path: 0 2802754, hotness: undefined, score: -5, children: 0

gsa32 -12 points 3 years ago

lol

path: 0 2766418, hotness: undefined, score: -12, children: 0

Candybar121 -27 points 3 years ago

path: 0 2762227, hotness: undefined, score: -27, children: 0

rafa -29 points 3 years ago

Hey, hire people that understands how the fediverse works. You're welcome

path: 0 2806792, hotness: undefined, score: -29, children: 0

lemmyworld

@lemmy.world

login for more options

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages 🔥

https://status.lemmy.world

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

DM https://lemmy.world/u/lwreport
Email report@lemmy.world (PGP Supported)

Donations 💗

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Join the team

go to feed...

lemmyworld

@lemmy.world

login for more options

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages 🔥

https://status.lemmy.world

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

DM https://lemmy.world/u/lwreport
Email report@lemmy.world (PGP Supported)

Donations 💗

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Join the team

go to feed...