Yes, midwest.social was down this morning

3 years ago by seahorse [Ohio] to c/main

I've been trying to fix the CPU issues with the database for weeks now and I felt I had gotten close. I ran these migrations manually and the delete query deleted somewhere around 400k rows. Somehow that I've been monitoring the CPU for about an hour now and it seems to be behaving. It would spike when I upvoted anything and stay there a while. I don't want to jinx it but try out the site and let's see if it stays responsive.

My deepest apologies for the downtime and the irritating slowness of the instance since you all joined.

hrimfaxi_work 22 points 3 years ago

The effort is much appreciated!

returning to add that it's running better than ever for me since coming back up

path: 0 220819, hotness: undefined, score: 22, children: 0

JCreazy 21 points 3 years ago

It's been a great community. I appreciate what you're doing

path: 0 220995, hotness: undefined, score: 21, children: 0

b0b89 20 points 3 years ago

No worries don't feel like you have to keep 99% uptime or anything. I just appreciate that you let us all use your instance.

path: 0 221305, hotness: undefined, score: 20, children: 1

seahorse 7 points 3 years ago

The thing was that it was unavailable like 50% of the time. Barely usable and irritating.

path: 0 221305 224476, hotness: undefined, score: 7, children: 0

gdj209 11 points 3 years ago

Appreciate the work you are doing to keep this up for us. I did want to acknowledge that the site is super fast and responsive now!

path: 0 224094, hotness: undefined, score: 11, children: 2

seahorse 14 points 3 years ago

Thanks! Honestly, this was the most difficult to track software issue I've ever done lol. I'm still a bit traumatized. Every time the site takes a half second longer to load my heart skips a beat because like 5 times this issue has seemed to go away but come back quickly.

path: 0 224094 224420, hotness: undefined, score: 14, children: 1

gdj209 4 points 3 years ago

Yeah, I totally understand that feeling!

path: 0 224094 224420 224816, hotness: undefined, score: 4, children: 0

FoxBJK 11 points 3 years ago

Night and day difference compared to yesterday. Even the mobile apps I'm using are more responsive since API calls are coming back quicker.

I'm a web developer and do some hosting on the side. Not sure how much help I can be but feel free to PM

path: 0 230447, hotness: undefined, score: 11, children: 0

GordieHowesThumb 10 points 3 years ago

The instance has been running great this afternoon/evening, probably the best I've seen since I joined. Thank you so much for all your hard work and care!

path: 0 232925, hotness: undefined, score: 10, children: 0

llama 9 points 3 years ago

Thanks for doing the hard work to keep our instance updated and online! Do you have a wish list or fundraising goal? I want to do my part to keep our community sustainable.

path: 0 224192, hotness: undefined, score: 9, children: 3

seahorse 14 points 3 years ago

I don't have a goal really, but it costs about $28 to run the server per month and I currently get around $9 a week, so that covers it.

path: 0 224192 224341, hotness: undefined, score: 14, children: 2

fstrelok 5 points 3 years ago

I appreciate the info on the cost, I was curious.

path: 0 224192 224341 224771, hotness: undefined, score: 5, children: 1

seahorse 11 points 3 years ago

I plan to show receipts going forward too.

path: 0 224192 224341 224771 224855, hotness: undefined, score: 11, children: 0

brianrlawson 9 points 3 years ago

Thank you!

path: 0 222491, hotness: undefined, score: 9, children: 0

amnesiac7 7 points 3 years ago

hey there! no need to apologize. we're lucky you've created this space and invited us to hang out.

path: 0 273142, hotness: undefined, score: 7, children: 0

ImADifferentBird 6 points 3 years ago

Thanks for working to bring everything back up. I'm still getting my bearings here after joining in the wake of the Reddit APIcalypse, and I'm glad you guys are this attentive when there are problems.

I've noticed that I can't log in to the site in either a mobile or desktop browser. After I put in my user name and password and click Log In, I just get a circle that spins indefinitely. I've noticed this in both Firefox and Chrome on Windows, and Firefox on Android. Is this possibly a lingering issue from this morning?

path: 0 229898, hotness: undefined, score: 6, children: 6

seahorse 4 points 3 years ago

Hmm, that's very strange. Everything should be working right now. Did you try logging in in incognito?

path: 0 229898 229993, hotness: undefined, score: 4, children: 5

ImADifferentBird 3 points 3 years ago

I have now. Same results, I'm afraid.

path: 0 229898 229993 230207, hotness: undefined, score: 3, children: 4

seahorse 3 points 3 years ago

Before I reboot it, you're not logging in with your email address, right?

path: 0 229898 229993 230207 230289, hotness: undefined, score: 3, children: 1

ImADifferentBird 5 points 3 years ago

That was precisely it. Thanks.

path: 0 229898 229993 230207 230289 230301, hotness: undefined, score: 5, children: 0

seahorse 2 points 3 years ago

I'll reboot the server.

path: 0 229898 229993 230207 230236, hotness: undefined, score: 2, children: 1

ImADifferentBird 6 points 3 years ago

Actually, I just got a successful login using my username. Looks like that behavior was only happening if I use my email.

path: 0 229898 229993 230207 230236 230288, hotness: undefined, score: 6, children: 0

lencioni 5 points 3 years ago

Thanks for your efforts here!

path: 0 224237, hotness: undefined, score: 5, children: 0

ProfessorGumby 4 points 3 years ago

Thanks for all the hard work! It was getting almost unusable but is working great now!

path: 0 227595, hotness: undefined, score: 4, children: 0

Infynis 4 points 3 years ago

Glad to hear it! I've noticed the site running much more smoothly today

path: 0 227652, hotness: undefined, score: 4, children: 0

AlligatorBlizzard 2 points 3 years ago

I figure I'll wait a few weeks to try to apply/reapply to your instance. I tried to apply once and ended up with the spinning circle, no idea if my application went through. I just blocked my first exploding heads group and I hate being federated with them and Burggit (I don't have NSFW enabled so I haven't seen any of their stuff but still), but... shit just works here. In a few weeks I figure the growing pains will be sorted out and it'll be easier to move to another instance. Thanks for all of your hard work maintaining your instance, joining in via federation has been great.

path: 0 270254, hotness: undefined, score: 2, children: 1

seahorse 1 point 3 years ago

If you tried to sign up before Wednesday at noon the site was barely usable.

path: 0 270254 271654, hotness: undefined, score: 1, children: 0

uxia 1 point 3 years ago

test

path: 0 257586, hotness: undefined, score: 1, children: 0

@midwest.social

login for more options

Default community for midwest.social. Post questions about the instance or questions you want to ask other users here.

go to feed...

@midwest.social

login for more options

Default community for midwest.social. Post questions about the instance or questions you want to ask other users here.

go to feed...