How is lemmyworld so stable?

3 years ago by maltfield to c/lemmyworld

At the time of writing, Lemmyworld has the second highest number of active users (compared to all lemmy instances)

Also at the time of writing, Lemmyworld has >99% uptime.

By comparison, other lemmy instances with as many users as Lemmyworld keep going down.

What optimizations has Lemmyworld made to their hosting configuration that has made it more resilient than other instances' hosting configurations?

See also Does Lemmy cache the frontpage by default (read-only)? on !lemmy_support@lemmy.ml

load all comments
ruud 340 points 3 years ago

Thank you for all the compliments.

This ride reminds me of Mastodon.world in November. Details on that are here: https://blog.mastodon.world/...

So I started lemmy.world on a 2CPU/4GB VPS. Keeping an eye on the performance. Soon I decided to double that. And after the first few thousand of users joined, doubled it again to 8CPU/16GB. That also was the max I could for that VPS type.

But, already I saw some donations come in, without really asking. That reminded me of the willingness to donate on Mastodon, which allowed me to easily pay for a very powerful server for mastodon.world, one of the reasons it grew so fast. Other (large) servers crashed and closed registrations, I (mainly) didn't.

So, I decided to buy the same large server (32cpu/64threads with 128GB RAM) as for masto (but that masto one has double the RAM). With the post announcing that, I also mentioned the donation possibilities. That brought a lot of donations immediately, already funding this server for at least 2 months. (To the anonymous person donating $100 : wow!).

Now next: to solve the issue with post slowness. That's probably a database issue.

And again: migration took 4 minutes downtime, and that could have been less if I wasn't eating pizza at the same time. So if any server wants to migrate: please do! If you have the userbase, you'll get the donations for it. Contact me if you have questions.

path: 0 102234, hotness: undefined, score: 340, children: 27
SteveJobs 62 points 3 years ago

What kind of pizza was it?

path: 0 102234 105871, hotness: undefined, score: 62, children: 6
ruud 70 points 3 years ago

Salami

path: 0 102234 105871 107106, hotness: undefined, score: 70, children: 4
darkstar 22 points 3 years ago

Asking the important questions

path: 0 102234 105871 106505, hotness: undefined, score: 22, children: 0
Balthazar 27 points 3 years ago

And this kindness and willingness to help is why I've already fallen in love with Lemmy. Thank you good sir, thank you dearly for helping the next generation of internet denisens :D

path: 0 102234 122971, hotness: undefined, score: 27, children: 0
million 15 points 3 years ago

Nice job, thanks very much for the write up.

Out of curiosity are you cloud hosting or do you own a server on a rack somewhere? Scaling with Kubernetes or VMs or just running bare-metal?

path: 0 102234 115550, hotness: undefined, score: 15, children: 1
pariskalavros 5 points 3 years ago
path: 0 102234 115550 132464, hotness: undefined, score: 5, children: 0
grouvie 14 points 3 years ago

Interesting writeup. I'm curious about the resource usage of the Lemmy backend and frontend deployments. Do you have any insights on the resource utilization of these deployments?

path: 0 102234 103428, hotness: undefined, score: 14, children: 0
SteelBeard 12 points 3 years ago

Hey, you rock. This place is pretty cool.

path: 0 102234 107546, hotness: undefined, score: 12, children: 0
QuazarOmega 10 points 3 years ago

Amazing stuff!
I hope others will follow suit, it's the biggest hurdle right now for serious adoption in my opinion and you crushed it!

path: 0 102234 103609, hotness: undefined, score: 10, children: 0
Soullioness 10 points 3 years ago

What does it cost per month to operate your servers, namely this one?

path: 0 102234 110480, hotness: undefined, score: 10, children: 3
poVoq 21 points 3 years ago

They just upgraded to a dedicated server for 180€/month today.

path: 0 102234 110480 112215, hotness: undefined, score: 21, children: 2
Dream_state 5 points 3 years ago

Am I able to use the same account to login to mastodon.world? Or do I need to make an account there too? Never used mastodon but vining the fediverse stuff

path: 0 102234 116048, hotness: undefined, score: 5, children: 2
bcnelson 6 points 3 years ago

Unless I'm very mistaken you can not use the same account.

path: 0 102234 116048 116616, hotness: undefined, score: 6, children: 1
frankyboi 4 points 3 years ago

is it a good idea to host on my home internet ?

path: 0 102234 155178, hotness: undefined, score: 4, children: 1
mjgood91 4 points 3 years ago

I bet you could do it if your instance didn't pull in a lot of traffic.

If it did..... I reckon that you might be able to pull it off to a certain extent so long as your internet package was good enough, but if you got hit with a Reddit-level flood of incoming users, your network almost certainly wouldn't be able to keep up.

Even if it could, if you were consistently eating through all the upload bandwidth, I reckon you'd draw the eyes of your ISP and they might send you a letter kindly and respectfully telling you that if you don't upgrade to a commercial line they're not renewing your contract.

path: 0 102234 155178 171977, hotness: undefined, score: 4, children: 0
taj 4 points 3 years ago

Fantastic. Thanks for all you've done, and are doing!!

path: 0 102234 112313, hotness: undefined, score: 4, children: 0
Alchemy 3 points 3 years ago

Love to see it. Thank you!

path: 0 102234 139669, hotness: undefined, score: 3, children: 0
Xaphanos 2 points 3 years ago

As someone "in the business", but not nearly as technical as you... How far can a single instance scale? Can a load balancer spread it over mulitple front-ends to handle user load? Can the back-end be re-worked to handle hundreds of millions of user operations per second? Can it work with a CDN? Can a single "Lemmy.World" site exist as a distributed site - with hundreds of servers spread across dozens of sites across the globe?

I expect this entire line of thought is antithetical to the entire Lemmy philosophy of distributed operation. I expect that the "correct" way is to spin off "NA.Lemmy.World", EMEA. Lemmy.World", APAC.Lemmy.World", etc. as separate servers. Is that correct?

Thanks.

path: 0 102234 261150, hotness: undefined, score: 2, children: 1
ruud 2 points 3 years ago

Best would be to have thousands of servers. I wouldn't make them subdomains of lemmy.world, but people could create separate instances for large communities, for example. https://lemmy-selfhosted.com Scaling: if the Lemmy software gets tuned a bit better, I think the current server could host at least a million users. But let's hope for more servers first...

path: 0 102234 261150 266746, hotness: undefined, score: 2, children: 0
Xanvial 2 points 3 years ago

Interesting, I'm new on Lemmy (and fediverse itself), but when you said server does it means the backend that handles frontend traffic or database that stores all the data? Seems the next optimization step is distributing the traffic to multiple servers.

Also (again I don't know about the lemmy system itself), maybe you can get away with just upgrading CPU cores only or RAM only (depends on what bottlenecking the system). From my experience, the RAM requirement is scaling slower compared to CPU

path: 0 102234 153577, hotness: undefined, score: 2, children: 0
lemmyworld
lemmyworld

@lemmy.world

login for more options
31279
770
44

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages πŸ”₯

https://status.lemmy.world

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations πŸ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

go to feed...