Problem has improved, but is not gone

4 years ago by seahorse [Ohio] to c/main

The high CPU issue that is causing midwest.social to slow way down and timeout on HTTP requests is still a mystery to me. But, it seems to do it less often and for less time now.

Here is CPU usage over 24 hours:

It's a bunch of SELECT statements in Postgres that all seem to fire at one time and take a while to complete as a batch. I've inspected the logs and haven't seen anything unusual. Just stuff federating and posts receiving upvotes.

Green_Bay_Guy 4 points 3 years ago

Yo, I'm a network engineer, also from the Midwest (Living in Vietnam). Do you have a discord or chat platform where I could be of any help?

path: 0 59118, hotness: undefined, score: 4, children: 1

seahorse 4 points 3 years ago

Yeah, there's a link to our Matrix chat in the sidebar.

path: 0 59118 63521, hotness: undefined, score: 4, children: 0

pelotron 2 points 3 years ago

Any updates on server stability lately? I've been having a lot of connectivity issues the last couple days, and just saw this post on lemmy.world that was pretty interesting.

@seahorse@midwest.social have you been dealing with performance problems like this?

(side note, how do you embed a link to another instance that will let midwest.social users stay logged on?)

path: 0 869634, hotness: undefined, score: 2, children: 0

Arindrew 2 points 3 years ago

Has the influx of new members exacerbated this issue? I found this thread because I'm seeing a few '504 Gateway Timeout' errors occasionally.

path: 0 107042, hotness: undefined, score: 2, children: 1

seahorse 1 point 3 years ago

Beehaw is also noticing these issues. It's most likely from the influx of new users.

path: 0 107042 1451224, hotness: undefined, score: 1, children: 0

RoundSparrow 2 points 3 years ago

Your image didn't show up.

Is it PostgreSQL process that is going high CPU, or the lemmy_server process, or lemmy-ui?

path: 0 305965, hotness: undefined, score: 2, children: 3

seahorse 4 points 3 years ago

It's fixed. Was an inefficient database query.

path: 0 305965 306014, hotness: undefined, score: 4, children: 2

RoundSparrow 3 points 3 years ago

Did you hand-modify lemmy_server? is there a code change/pull request to share?

Thank you.

path: 0 305965 306014 306037, hotness: undefined, score: 3, children: 1

seahorse 2 points 3 years ago

I manually modified the activity table. I'll find the migration that didn't run.

path: 0 305965 306014 306037 306061, hotness: undefined, score: 2, children: 0

duskfall 1 point 2 years ago

path: 0 15046175, hotness: undefined, score: 1, children: 0

simsymbiote 1 point 4 years ago

Have you reached out to anyone else who runs a Lemmy server and seen if they are also experiencing this?

Do you think it is a bottlekneck or just due to large updates from other servers?

path: 0 691443, hotness: undefined, score: 1, children: 1

seahorse 1 point 4 years ago

I have reached out and nobody seems to have any idea what would cause it.

path: 0 691443 691444, hotness: undefined, score: 1, children: 0

milltertime_3227790 1 point 3 years ago

Is this issue still ongoing? I've had various threads that I've had issues accessing via Jerboa and the pwa on my Chromebook. Can't figure out if it's a Lemmy-wide issue or instance-specific... from 404 issues to the website hanging when I submit a post along with pages that only load the header

path: 0 15998, hotness: undefined, score: 1, children: 2

recreationalplacebos 2 points 3 years ago

I've been having timeout and 404 issues since I created my account last week. Typically can't browse more than 5 minutes before I start getting timeouts, which can last for several minutes to several hours before anything loads again. Thought it was a gerboa issue at first, but I'm using a browser now and getting them too.

path: 0 15998 15999, hotness: undefined, score: 2, children: 0

seahorse 2 points 3 years ago

Unfortunately it still happens, just not nearly as often as it used to. Need to meet up with my database admin to get to the bottom of the issue.

path: 0 15998 21402, hotness: undefined, score: 2, children: 0

m3t00 1 point 3 years ago

path: 0 742839, hotness: undefined, score: 1, children: 0

@midwest.social

login for more options

Default community for midwest.social. Post questions about the instance or questions you want to ask other users here.

go to feed...

@midwest.social

login for more options

Default community for midwest.social. Post questions about the instance or questions you want to ask other users here.

go to feed...