Workaround for the performance issue with posting in large communities

3 years ago by Ruud to c/lemmyworld

We're still working to find a solution for the posting slowness in large communities.

We have seen that a post does get submitted right away, but yet the page keeps 'spinning'

So right after you clicked 'Post' or 'Reply' you can refresh the page and the post should be there.

(But maybe to be sure you could copy the contents of your post first, so you can paste again if anything would go wrong..)

bdonvr 21 points 3 years ago

Keep in mind that the upcoming Lemmy update will probably fix this I think. (Replacing websockets)

path: 0 124026, hotness: undefined, score: 21, children: 6
ruud 11 points 3 years ago

Yes I really hope so!!

path: 0 124026 124719, hotness: undefined, score: 11, children: 0
RoyalEngineering 6 points 3 years ago

Maybe this is a dumb question, but why would replacing websockets speed things up? I read the Wikipedia page on it, but I guess I don’t understand it fully.

path: 0 124026 165506, hotness: undefined, score: 6, children: 3
phiresky 7 points 3 years ago

In general websockets scale badly because the server has to keep open a connection and a fair amount of state. You also can't really cache websocket messages like you can normal HTTP responses. Not sure which reasons apply to Lemmy though.

path: 0 124026 165506 166212, hotness: undefined, score: 7, children: 2
LocustOfControl 4 points 3 years ago

The caching problem is definitely part of it from conversations on GitHub.

path: 0 124026 165506 166212 172307, hotness: undefined, score: 4, children: 0
RoyalEngineering 4 points 3 years ago

Thanks for the explanation!

path: 0 124026 165506 166212 175688, hotness: undefined, score: 4, children: 0
slashzero 4 points 3 years ago

I really hope someone is doing some level of performance testing on those changes to make sure the changes fix the performance issues.

path: 0 124026 132284, hotness: undefined, score: 4, children: 0
Scaldart 20 points 3 years ago

Just hopping into the chain to say that I appreciate you and all of your hard work! This place—Lemmy in general, but specifically this instance—has been so welcoming and uplifting. Thank you!

path: 0 125724, hotness: undefined, score: 20, children: 0
tallwookie 16 points 3 years ago

maybe related, but I've noticed that upvoting/downvoting has similar lag delays

path: 0 134167, hotness: undefined, score: 16, children: 2
kaxora 10 points 3 years ago

Same. It would be good to fix this

path: 0 134167 150243, hotness: undefined, score: 10, children: 0
ChaosAD 5 points 3 years ago

I can't up/down vote at all.

path: 0 134167 169536, hotness: undefined, score: 5, children: 0
mykl 15 points 3 years ago

At least the "reply" button goes away so I don't end up double- triple- or even duodecuple-posting! Thanks for all the hard work that must be going on behind the scenes right now!

path: 0 166495, hotness: undefined, score: 15, children: 7
cascadingsymmetry 16 points 3 years ago

I kept getting a timeout message from Jerboa which led me to think I hadn't been posted. So I ended up submitting the same joke to the Dad Jokes sub three times. Which actually is how dad might tell that joke.

path: 0 166495 177676, hotness: undefined, score: 16, children: 6
mykl 7 points 3 years ago

Lemmy is now your digital dadlife assistant.

path: 0 166495 177676 178000, hotness: undefined, score: 7, children: 0
AlmightySnoo 5 points 3 years ago

I think in that case it's a feature not a bug.

path: 0 166495 177676 218815, hotness: undefined, score: 5, children: 0
FartsWithAnAccent 3 points 3 years ago

I get this occasionally with Jeroba too, I had assumed it was because I'm on Mint and the connection is shoddy but maybe it's an issue with the client.

path: 0 166495 177676 178199, hotness: undefined, score: 3, children: 2
Zackyist 1 point 3 years ago

I came here looking for information on this exact thing and judging by a couple of posts I found through Google on other Lemmy instances, it seems to be a Lemmy.world specific problem with Jerboa. So probably connected to the community slowness?

path: 0 166495 177676 178199 247609, hotness: undefined, score: 1, children: 1
c2h6 1 point 3 years ago

I'm getting this issue on a mobile browser.

path: 0 166495 177676 178199 247609 247998, hotness: undefined, score: 1, children: 0
Nightingale 1 point 3 years ago

That's awesome. :P

path: 0 166495 177676 287851, hotness: undefined, score: 1, children: 0
slashzero 14 points 3 years ago

Have you tried enabling the slow query logs @ruud@lemmy.world? I went through that exercise yesterday to try to find the root cause but my instance doesn’t have enough load to reproduce the conditions, and my day job prevents me from devoting much time to writing a load test to simulate the load.

I did see several queries taking longer than 500ms (up to 2000ms) but they did not appear related to saving posts or comments.

path: 0 132209, hotness: undefined, score: 14, children: 1
lhx 2 points 3 years ago

These things are taking 15-20 seconds though.

path: 0 132209 176021, hotness: undefined, score: 2, children: 0
wit 13 points 3 years ago

Again, thank you for the outstanding work! You are awesome!

Also, the new icon for lemmy world is great!

path: 0 123010, hotness: undefined, score: 13, children: 0
gkd 13 points 3 years ago

Been noticing this in the app I’m working on. Pretty much all POST requests fail to return a response and just timeout after 60 seconds. A quick refresh shows that the new items do successfully get created though.

path: 0 122392, hotness: undefined, score: 13, children: 0
vepro 9 points 3 years ago

I assume that there is something that is O(N), which explains why wait time scales with community size (amount of posts, comments)

path: 0 131775, hotness: undefined, score: 9, children: 6
slashzero 10 points 3 years ago

Oh, Big-O notation? I never thought I’d see someone else mention big O notation out in the wild!

:high-five:

path: 0 131775 132112, hotness: undefined, score: 10, children: 5
manitcor 11 points 3 years ago

you are going to meet a lot of OG redditors in the next few weeks. Old reddit had Big O in every post, even posts with cute animals.

path: 0 131775 132112 133053, hotness: undefined, score: 11, children: 4
slashzero 4 points 3 years ago

That’s pretty neat! I’ve honestly never seen it mentioned on Reddit before, so got a bit excited to see someone mention it here, admittedly maybe too excited.

path: 0 131775 132112 133053 133198, hotness: undefined, score: 4, children: 3
manitcor 6 points 3 years ago

there was a time, before the digg invasion, where someone would post a picture of a woman feeding 30 cats and there would be Big O jokes about how well this would work and crazy modifications to the situation to improve it. This would be almost every thread at one point on the site. I miss it.

path: 0 131775 132112 133053 133198 133233, hotness: undefined, score: 6, children: 2
lethalfeline 9 points 3 years ago

Thanks for your and the other Lemmy devs work on this. These growing pains are a good thing as frustrating as it can be for users and maintainers alike. Lemmy will get bigger and this optimization treadmill is really just starting.

path: 0 135183, hotness: undefined, score: 9, children: 0
neighbourbehaviour 8 points 3 years ago

Does this behaviour appear on other big instances? E.g. lemmy.ml?

path: 0 132387, hotness: undefined, score: 8, children: 3
slashzero 6 points 3 years ago

Yes. Absolutely does happen on other instances that have thousands of users.

path: 0 132387 133662, hotness: undefined, score: 6, children: 1
neighbourbehaviour 7 points 3 years ago

Great, so it's reproducible and Lemmy-the-app related, not instance-specific. Should be fixable across the board once it's identified and resolved.

path: 0 132387 133662 133871, hotness: undefined, score: 7, children: 0
hydra 4 points 3 years ago

Yes it does, tried this workaround before.

path: 0 132387 133283, hotness: undefined, score: 4, children: 0
Oxff 8 points 3 years ago

Thank you for your hard work and keeping us up to date.

path: 0 123031, hotness: undefined, score: 8, children: 0
kaxora 7 points 3 years ago

In my case, the page keeps spinning but the post is not submitted, regardless of reloading the page or waiting for a long time. There was one case where I cut down significantly on the amount of characters in the post and then it posted, but I have been unable to replicate this.

path: 0 150351, hotness: undefined, score: 7, children: 3
Moghul 5 points 3 years ago

I have the same issue with image posts. If I submit them through the app the posts counter on my profile goes up, but there's no post. I also can't retrieve any posts for my own account. It says I have 3 but it shows none.

Comments work OK so I'm not sure what the problem is. I was worried I got restricted or something.

Edit: Comments seem to work as the OP suggests though. I get the spinner but they've been posted already.

Edit2: It seems like I can't post link threads either.

Edit3: Even worse, it seems like I can't see my own posts but others can. So I might have spammed Warhammer40k and tabletopminis and I can't even clean up my own mess.

Edit4: If I sign out and go to these communities I can see my own posts. I can also go directly to them if I have the link. I can also keep track of them by clicking the star button but it seems like my feed is broken.

Edit5: Oh my god I'm so fucking stupid. If you hide posts you've seen it'll also hide your own posts...

path: 0 150351 165860, hotness: undefined, score: 5, children: 1
kaxora 1 point 3 years ago

Yeah, comments seem to be working for me in that way too.

path: 0 150351 165860 165901, hotness: undefined, score: 1, children: 0
Moghul 2 points 3 years ago

Oh my god I'm so fucking stupid. If you hide posts you've seen it'll also hide your own posts...

path: 0 150351 181741, hotness: undefined, score: 2, children: 0
Sunforged 6 points 3 years ago

@ruud@lemmy.world Yo dude, first off huge props and a big thank you for what you have setup. I’ll be donating monthly while I am here. I appreciate that we have an alternative to Reddit at this critical moment in time.

I do have a question on your long term plans, do you want to continue to expand and upgrade the server, as funding allows, or is there a cap that you will close off the server to new members? Or perhaps make it more of a process to join?

path: 0 275338, hotness: undefined, score: 6, children: 3
ruud 12 points 3 years ago

Well if all the Reddit users would get over to Lemmy I guess all servers would need to scale up... but I think the server we have now is powerfull enough to grow quite a lot, as long as the software gets tuned ..

path: 0 275338 280350, hotness: undefined, score: 12, children: 2
Gompje 1 point 3 years ago
path: 0 275338 280350 330503, hotness: undefined, score: 1, children: 0
Gompje 1 point 3 years ago
path: 0 275338 280350 330585, hotness: undefined, score: 1, children: 0
V4uban 5 points 3 years ago

Thank you so much

path: 0 131678, hotness: undefined, score: 5, children: 0
CaptObvious 4 points 3 years ago
path: 0 218916, hotness: undefined, score: 4, children: 0
ulu_mulu 4 points 3 years ago

I noticed that too, page keeps spinning but comments are posted immediately anyway.

path: 0 122405, hotness: undefined, score: 4, children: 0
kapeeshy 4 points 3 years ago

I noticed this, thanks for the clarification

path: 0 149120, hotness: undefined, score: 4, children: 0
lenninscjay 4 points 3 years ago

ok, so it's not just me. Hope it gets resolved soon!

path: 0 196938, hotness: undefined, score: 4, children: 0
manitcor 4 points 3 years ago

its def more hung up today, oddly its only first level replies for some reason

path: 0 217474, hotness: undefined, score: 4, children: 0
MyOpinion 3 points 3 years ago

This is the biggest issue I have run into. Thanks for looking into it.

path: 0 122429, hotness: undefined, score: 3, children: 0
fhein 3 points 3 years ago

Good to know it isn't just me then.

path: 0 291220, hotness: undefined, score: 3, children: 0
LargeHardonCollider 3 points 3 years ago

Is the slowdown that it the instance has to send out updates about the comment to every other instance before returning a successful response? If so, is anyone working on moving this to an async queue?

Sending out updates seems like something that’s fine being eventually consistent

path: 0 311482, hotness: undefined, score: 3, children: 5
ruud 4 points 3 years ago

Ooh that’s a good remark ! I’ll see if that’s the cause

path: 0 311482 312605, hotness: undefined, score: 4, children: 4
LargeHardonCollider 11 points 3 years ago

Reading more about how this works, sending out updates to each instance shouldn’t block the request from returning unless you have a config flag set to debug source.

It might be due to poorly optimized database queries. Check out this issue for more info. Sounds like there are problems with updating the rank of posts and probably comments too

path: 0 311482 312605 332620, hotness: undefined, score: 11, children: 3
ruud 10 points 3 years ago

So it looks like YOU SOLVED THE ISSUE with this reply! This led me to check the debug mode, and it was on! It turned that on when I just sterted the server and federation had issues....

We no longer seem to have the slowness!!

path: 0 311482 312605 332620 340622, hotness: undefined, score: 10, children: 1
LargeHardonCollider 7 points 3 years ago

That’s awesome! Thanks for hosting the server!

path: 0 311482 312605 332620 340622 341034, hotness: undefined, score: 7, children: 0
ruud 3 points 3 years ago

Good info!

path: 0 311482 312605 332620 340206, hotness: undefined, score: 3, children: 0
deathworlder 2 points 3 years ago

One of the large applications I was working on had the same issue, to solve it we ended up creating multiple smaller instances and started hosting a set of related API's in each server.

for example read operations like list posts, comments etc could be in one server. write operations can be clusered in one server.

Later, whichever server is getting overloaded can be split up again. In our case 20% of API's used around 3/4th of server resources, so we split those 20% API's in 4 large servers and kept the remaining 80% API's in 3 small servers.

This worked for us because the DB's were maintained in seperate servers.

I wonder if a quasi micro-services approach will solve the issue here.

Edit 1: If done properly this approach can be cost effective, in some cases it might cost 10 to 20 percentage more in server costs, however it will lead to a visible improvement in performance.

path: 0 256197, hotness: undefined, score: 2, children: 0
Starz 2 points 3 years ago

Haha this just happened to me on my very first ever post to Lemmy and I was about to ditch this app. Not really. But I did copy my somewhat long post just in case. And I'm glad I did because I had to refresh and paste again and submit. However, within that issue, I found a weird bug. I had clicked on another post b4 going back to the thread that I wanted to re submit my comment. I checked out my profile to see my first comment and for some reason, my comment got posted on both threads, the related thread and the non-related thread. Devs might want to look into that one.

path: 0 250585, hotness: undefined, score: 2, children: 0
Hiyoi 2 points 3 years ago

This issues is affecting other instances from what I can tell

path: 0 124088, hotness: undefined, score: 2, children: 0
TragicNotCute 2 points 3 years ago

I’ve done this twice in the last 20 minutes and the content is not there. This workaround was working earlier today though.

path: 0 196533, hotness: undefined, score: 2, children: 0
MiddleWeigh 1 point 3 years ago

Yea I started just using my clipboard alot, and hitting the x or back on jerboa and comments go thru at about 90% success rate. For posts I let it do its thing just incase. But been copying stuff to save as draft.

path: 0 306971, hotness: undefined, score: 1, children: 0
rei 1 point 3 years ago

My comments seem to not go through sometimes, whenever I upvote posts on Jerboa I get an error that just says timeout. Doesn't seem to happen on other instances.

path: 0 304113, hotness: undefined, score: 1, children: 0
RandAlThor -3 points 3 years ago

Nice teething problems. I tried to post to !nba@lemmy.world but it just kept spinning. When I refreshed the post was not there.

path: 0 324526, hotness: undefined, score: -3, children: 0
lemmyworld
lemmyworld

@lemmy.world

login for more options
31469
771
647

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages 🔥

https://status.lemmy.world

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations 💗

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

go to feed...