Its now running on a dedicated server with 6 cores/12 threads and 32 gb ram. I hope this will be enough for the near future. Nevertheless, new users should still prefer to signup on other instances.
This server is financed from donations to the Lemmy project. If you want to support it, please consider donating.
Please know that your work is genuinely appreciated in fascilitating the migration from Reddit to Lemmy. Your efforts will hopefully ensure a bright future for communities on this platform. Kudos @nutomic@lemmy.ml !
Upgrade unlocked!
I just want to say, you all are doing a great job. Maybe I don’t fully understand it yet, but I have created some communities on the lemmy.ml instance and I don’t really want to move away. Is there anything I can or should do? I think that I’m currently locked in here.
I'm getting 502 error all the time on my lemmy.ml account (lemmygrad account works since i'm posting from it).
It worked, thanks.
Same. This is why I had to switch to lemmy.world. Now it's time to subscribe to everything all over again...
Is it possible to horizontally scale these instances instead of just upping the machine hardware? What are the main performance bottlenecks typically?
Hey, what do you mean by "scale horizontally"? There are multiple approaches to tackle this.
All of these requiere most likely a major rewrite/change of Lemmy server software I guess. They are already addressed as issues/feature requests on github In my opinion the first option would fit the most.
My comment was without knowing the topology of Lemmy at all, but my thoughts were initially that vertically scaling can have diminishing returns past a certain threshold. Since the servers seem to be struggling I'm wondering if that has been surpassed and if it's more cost-effective and reliable to scale this way? But if the application isn't written in that way, or the underlying data store isn't equipped for multiple instances then fair enough, I'd be interested as to why especially if Lemmy grows. I'll take a look at open issues and educate myself a bit more though.
I'm still amazed at how light-weight lemmy is.
Inferno is pretty slick too
And it's not even a super beefy one!
kbin.social uses a CCX41 with 16 virtual cores (I think 8 of those are hyperthreaded though) and 64gb of ram for its ~25k users
I was getting 502 Bad Gateway. When I pinged Lemmy.ml I got an IPV6 address. It disabled IPV6 on my local computer and now when I ping I get a IPV4 IP address it works now.
I am wondering if DNS is screwed up on the IPV6 network for Lemmy.ml.
~~Note. This could totally be something on my end, I really haven't done much with IPV6 but it did solve the 502 Error so I might do the same for you. ~~
Edit. Multiple people are reporting the same thing I am seeing. It is defiantly something about IPV6 on the lemmy.ml server end.
Thanks for mentioning the IPv6, I've been banging my head all day trying to figure out why I kept getting the 502 yet no one was complaining anywhere and isitdown was showing the server as Up.
I forces my DNS to resolve only IPv4 for lemmy.ml and now I can use it.
My suspicion is that nginx is misconfigured and not listening via IPv6. Or maybe the AAAA record is pointing to the wrong IPv6 address.
@nutomic Thanks for upgrading the server!
I'm also unable to connect with IPV6 enabled getting the 502 error, but able to connect with it disabled
@nutomic@lemmy.ml pls fix this. I'm not turning off IPv6 on my end.
edit: it works now :)
same its been 6 hours. seems like using a vpn to connect works.
Which country works for you on VPN? Doesn't work for me with or without VPN
it seems this new server is not configured to work with iCloud's Private Relay service as the last one was
how to configure servers to work with private relay: https://developer.apple.com/...
Me and others are getting that issue without that. Seems to be related to its IPV6 configuration or AAAA record since disabling IPV6 seems to make lemmy.ml work
How to migrate an account to another instance without losing anything including relationships ?
This is what I need to know. I also have an account on Lemmy world as a backup
I was wondering the same thing. I initially created an account on some obscure instance because I didn't fully understand what I was doing. I just abandoned it and set up a new one on lemmy world. I think I'm getting the hang of it now. I'm curious to see how Lemmy grows and matures over time. There is still a learning curve that will keep some people away.
Are there still issues with cross-instance content? Previously, if I signed in to lemmy.ca and subscribed to a channel on .ml, I didn't see all content. Likewise, if I left a comment from my .ca account, it wouldn't necessarily show up for users on .ml.
If this is still a problem, it's a HUGE roadblock in being able to just tell people to join other instances, if we don't want to fracture an existing community.
Edit: That may have simply been a result of the excess load?
Edit 2: Okay I'm testing and it looks like this was just an issue of load; I don't seem to be having any further issues with this.
Edit 3: Comments from users on lemmy.world still don't seem to propagate correctly. I can see them from .ml but not .ca
I really do appreciate everything you're doing for us all. Thank you.
Have we explored the possibility of "porting" the larger communities to other instances? It seems that many of us simply wish to subscribe to the largest (insert type of community here) and can do so from various home instances. Might lower demand on this specific instance at the very least.
Probably hit the limit again and need a bigger server. It was migrated yesterday
Is most of the discussion about the infrastructure done in a community here, or do you folks have an invite only discord/slack or something @nutomic@lemmy.ml?
I've been working in devops for 4 years now, but with mostly kubernetes and AWS, but I'd like to throw some ideas and just ask some more specific questions about the infrastructure. Like I'm curious why autoscaling the web servers and moving the database server to a dedicated instance is not the current configuration.
There is really not much to discuss. The server was overloaded so we got a bigger one. And there is no reason to mess with stuff like kubernetes when a single server works fine. After all our job is to improve the Lemmy software for everyone, not build a huge centralized platform only on lemmy.ml.
I agree with the kubernetes point. Not every problem needs it.
I was more curious about having a load balancer infront of an autoscaling group of servers with the existing images that you are running off of, minus the self hosted database server. Then you would be able to handle spikes automatically. Just curious if that has been thought of.
Having it accessible to everyone is great. For sure. But I was just thinking that having stronger pieces of infrastructure for instances that handle more traffic might be beneficial.
I agree, lots of application improvements and bugs are probably your priority. That should always be the focus of the application developers. I honestly want to try and experiment with this idea now.
If I get a good repository going that focuses just on the infrastructure side of things, I will post it here.
Exactly. lemmy.ml is lemmy.ml - just a single instance. there should be incentive for other to stand up instances of their own. overload on any instance is good reason for ppl to get out there and build new instances.
Hi @nutomic@lemmy.ml. Thanks for the upgrade and for your work.
May I ask about the resources utilization now? CPU, RAM, storage? Thanks!
Thanks. Thinking about hosting a Czech instance for lemmy, that's why I am asking
Thank you! I can actually participate now!! I have several screen shots saved of my profile showing me logged in under random user names over the last few days. . Shit was weird. I couldn't post, couldn't stay logged in, or I'd see a strange profile name if it actually let me click into a thread. Everything seems to be solid today.
Is there a way to migrate my account to a new server?
hi! I would be interested in understaning more the setup of the lemmy.ml instance. Do you use a cloud provider, a SaaS platform or a traditional hosting ? What are the costs that are incurred? Cheers!
Is Lemmy made with horizontal scaling (a.k.a. launching more instances and have a load balancer proxy the requests to the various instances) in mind? It could help larger instances like lemmy.ml managing the load better rather than just putting it on a beefier machine.
You should be able to do that without problems. However the main bottleneck is the database, I think some people want to experiment with read replicas. However as developers we would rather focus on optimizations which will benefit everyone, not only the largest instances.
Ah awesome. The database horizontal scaling is a solved problem already luckily, especially an enterprise database like PostgreSQL has lots of options there.
as developers we would rather focus on optimizations which will benefit everyone, not only the largest instances.
Oh sure, but being able to horizontally scale shouldn't hurt small instances 😉 Personally I'd probably host a single-user instance at some point just like I do with Mastodon, so I personally don't really have a need for horizontal scaling either but it's good to think of those things.
It's giving me a 502 error on my computer as well but is working on my phone using WiFi.
@v_krishna @nutomic not working on my phone either. Same error
@nutomic likely https://support.us.ovhcloud.com/...
Thank you for your hard work! Although I kinda foresee for the future if Lemmy really would become the new "reddit" with such servers and millions of users, wouldn't that also rise the server costs and ultimately make the hosts dependent on asking money for it, maybe by a paywall or by ads? I think to make this community really be "free" without any host responsible for spending a huge amount of money for servers, the best solution would be to make the actual "servers" be a p2p cluster. Unfortunately I'm not quite sure how to realize that without losing a huge fraction of the model if a lot of nodes (i.e., the actual users) are offline. Sorry, I'm just brainstorming.
You might want to check the Earthstar project: https://earthstar-project.org/ They are working on that. Right now, it's super early; they are building the foundations. Peer-to-peer is unfortunately much more difficult to code then servers, because less people have built the building-blocks required, and because mobile phones are actively making it hard to run peer-to-peer apps.
I'm still having trouble that any comments I make on other instances through lemmy.ml are not seen on those instances, only when viewed through lemmy.ml
I've donated on https://opencollective.com/lemmy to support this :)
This is excellent news.
I'm a little curious how difficult it would be to include this status in the sidebar - we see 'users online' statistics, we should also see 'costs vs donations' statistics there plus a link for donations also.
Fantastic to see and the improvements are already being felt and very obvious.
Thank you!
Thank you for your hard work on this site!
This server now: I am speed
ive noticed already that it is much faster and responsive now it must be this new server
JFYI, it's getting an error 500 here and there when refreshing and sometimes it misses loading the css. No idea if you are using a cache somewhere, could be that as well if you don't see anything weird in the monitoring.
Great to hear! It's already a lot smoother than it was before! Hopefully it will continue to grow!
hey 👋 do you have some Grafana dashboard where we can check user increase?
Given the no. of posts on this instance I guess it has quite a large user base and I’m really surprised how it manages to run on that amount of resources 😄
I'm fully aware of the operational challenges involved. What I have not looked into (yet) is whether it is even feasible.
Honestly, the operational side doesn't worry me, I can do that in my sleep. Application level issues I am powerless to solve however.
I assume it's for the sake of more freedom. Without admins to make troubling choices like in the case of reddit, lemmy can't be corrupt so easily. I'm quite new to this as well, but from what I've seen, the most lemmy can do is unlink certain instances so they don't show up in your search, but instances may live or die, but the social network as a whole lives on (except if the killed instance housed your account).
Yeah that was my bad, I thought you meant horizontal scaling utilizing multiple instances.
Oh, my bad. Thanks for the clarification.
Liberapay has a manual renewal option. I'd go with doing your amount you want to donate there and just ignore the renewal email.
I tried to run my own instance but ran into errors. Gonna try again some other time.
@nutomic thanks, but I'm still getting 502 bad gateway
@nutomic hey, just fyi, still having the issue. 502 bad gateway. Someone further down I'm the conversation that it may be IPV6 related? Not sure
@nutomic yeah, force reloading didn't work. I'll wait for dns propagation. Thanks for the confirmation though!
The devops understander has logged on
@lemmy.ml
Anything about the lemmy.ml instance and its moderation.
For discussion about the Lemmy software project, go to !lemmy@lemmy.ml.
go to feed...
@lemmy.ml
Anything about the lemmy.ml instance and its moderation.
For discussion about the Lemmy software project, go to !lemmy@lemmy.ml.
go to feed...
Everyone liked that.
save