[Unexpected Downtime Postmortem] July 7 11:30 PM MT

3 years ago by Crashdoom (he/him) to c/pawbsocial_announcements

Around 11:30 PM, one of the core GCFI breakers tripped resulting in the UPS array running until around just after midnight before dropping entirely.

We had not set up monitoring on the APC UPS as we hadn’t anticipated that as a failure point due to the reliability of the Colorado electrical grid, and didn’t anticipate the breaker itself flipping after months of continued use.

All services should be online again and working to catch up on any dropped content overnight, and we apologize again for the inconvenience.

We’ll be trying to identify the root cause throughout the day, working on the electrical connections, and setting up alerting through the APC UPS to detect and persistent A/C loss.

load all comments
Maddi 14 points 3 years ago

Appreciate the updates! Thanks for the hard work!

path: 0 983151, hotness: undefined, score: 14, children: 0
login for more options
563
30
4

Official announcements from the admin team of Pawb.Social

go to feed...