PSA from Admin Team: The update completed roughly two hours ago. Since that time, the Admin team (and other site admins) have been working on the noted performance issues. We believe we have found a solution, but we still need time to test this out. You may still see brief outages and differences in performance as we are testing different configurations. We are trying to prevent rolling back.

While I know this can be frustrating - especially today - please keep in mind we have a team of volunteer techies (from around the globe!) collaborating on this issue. It is an inspiring situation. Also keep in mind that lemmy.world is quite a bit larger (and more active than any other instance). As such, we are a bit of a ‘test instance’ in regards to high volume requests. This is just part of the growing pain. We appreciate your understanding.

@ruud@lemmy.world will provide a debrief once we have completed testing.

This may be the best highlight of spez incompetence 🍿 For everyone talking smack on the moderators, this is exactly what they should be doing. Malicious compliance. Make it hard for Reddit to know what is going on. Love to see it.

Frankly one of my favorite parts about the blackout - and opinions like this - is that Lemmy is going to end up being populated by people who have the capacity to think about others and form intelligent opinions. All of the people with this attitude will stay on reddit, which is what will ultimately kill it. I hope he stays.

Thanks for posting this over here. I otherwise wouldn't have known this was going on. As I used a mostly lurker account for game day threads I wasn't going to bother with a data request. It's a whole different story if we can inundate them though. Request submitted. o7

Godspeed to you over the coming days man. Really appreciate you putting this together and the extra work it takes when tackling something like this (both being new to the platform and the tech still being in relative infancy) - not to mention the crazy scaling happening. I will definitely be pitching in to help make sure the server stays up!!

One of the best things about open source software - there is no need to hide useful information from your user base. A concept largely forgotten in today corporate hemisphere.

As you mentioned, I think the mods have done a fantastic job given the learning curve and lack of mod tools. This is just a good tool for accountability and transparency. Keep it up!

Amen! This movement is far more about he actual implications of federation, not that it isn't there yet. If we have any chance to live in a world not controlled by government/corporate greed, then we have to start trusting in one another and helping to support an internet that we all control and can take part in.

Bravo to you. This was absolutely the right call and I hope you let it rot if there isn't a massive change of direction. Frankly, for me, even if they do roll things back I am still done with them. P2P is how the internet should be (funny considering torrenting has been a thing for how long now?) and thankfully we have guys like you to step up and help get this off the ground. Thanks!

My take on this is that we have to remember how much this massive influx of users has been the last few days. Significantly more than its entire history. There is also the learning curve of new users. I think as people start getting a handle on how the fediverse works this will be less of an issue. Overtime this should work itself out. My two cents anyway. BTW also just subbed to c/Gameboy ;)

God I wish I could upvote this, but after about 5 minutes of trying to get it to stick, I will leave this comment instead. Well played ;)

Technically speaking, yes, a portion of our issues are due to the highest user base of an Lemmy instance. So in theory, if half of our users dispersed to other instances, we would likely see some performance improvement here. However, lemmy.world is intended to be an accessible instance for the general population. The server itself that is running lemmy.world is beyond spec'd to handle much more than this user load. We are running up against code-level issues that we may or may not be able to get around with our internal configurations. This is just part of developing software in an environment were you go from a few thousand users total to hundreds of thousands in the space of a few weeks. There is no directive to have users create accounts on new instances, though if you are looking for an immediate performance improvement, that may be your best option currently. That is up to you to decide :)

Just so you know this has been reported and is a known issue (happens to me on the browser as well). Presumably, it is due to Lemmy’s use of websockets for API requests, which has its issues for this particular application.

This has been addressed in version 0.18.0 as, among quite a few other changes, an HTTP API is now in place for API requests which will (should) resolve this issue.

Lemmy.world is currently on back end version 0.17.4 which still uses websockets. You can see this at the bottom of every page. The reason for this is that version 18 removed the ability to use Captcha for account registration. Given the size of the instance and its rapidly growing user base, and the fact that we are still accepting registrations, we simply must have the ability to leverage Captchas to mitigate the bot spam signups, which has been an issue.

We have tested the 0.18.1rc build (non-official release) in order to try and get this new functionality as soon as possible, but as you can see in the post 18.1 needs some time to bake still - at least for large instances.

This is definitely being discussed among the admin/mod team and we will be upgrading as soon as we realistically can while providing a stable - usable - experience.

Godspeed Christian. Personally, I am very much looking forward to the impact you will have on the fediverse. There's not many whom I would trust more with it.

This is due to the update today. We are working with the wefwef dev on a resolution. Look for more info from Ruud on this as soon as we have a fix.

Great work, team. Happy Halloween! 👻

Thanks @DarraignTheSane@lemmy.world for getting this going. I have high hopes 😊 if you need any mod support, please let me know.

FYI, this is due to a confluence of issues.

• We are the largest instance with the highest active user count - and by a good margin.
• We are dealing with a premature software base that was not designed to handle this load. For example, the way the ActivityPub Federation queues are handled are not conducive to high volume requests. Failed messages stay in queue for 60 seconds before they retry once, and if that attempt fails it sits in queue for one hour before attempting to retry. These queued messages sit in memory the whole time. It's not great, and there isn't much we can currently do to change this, other than to manually defederate from 'dead' servers in order to drop the number of items stuck in queue that are never going to get a response. Not an elegant solution by any means, and one we will go back and address when future tools are in place, but we have seen significant improvement because of this.
• We have attempted contacting Lemmy devs for some insight/assistance with this, but have not heard back yet, at this time. Much of this is in their hands.
• We were able to confirm receipt of our federation messages (from lemmy.world) to other instance admins instances at lemm.ee and discuss.as200950.com. As such we do know that federation is working at least to some degree, but it is obviously still in need of some work. As mentioned above, we have reached out to the Lemmy devs, who are instance owners of Lemmy.ml, to collaborate. I cannot confirm if they are getting our federation at this time. Hopefully in coming Lemmy releases this becomes easier to analyze without needing direct server access to both instances servers.

As you can see, we are trying to juggle several different parameters here to try and provide the best experience we can, with the tools we have at our disposal. You may consider raising an issue on their GitHub about this to try to get more visibility to them from affected users.

Thanks for the kind words! Yeah, there are definite growing pains, and likely will be for some time (just do to the codebase we are working with, understandably). We have a really solid group though heading up lemmy.world though so we will be just fine ;)

Agreed.

I have never DND'd but this is Hilarios, It is actually part of the reason I have never participated. Too much stuff. Bravo though!