Under-qualified sysadmin crashed Amazon.com for 3 hours

a year ago by MHLoppy to c/sysadmin

Who, Me?: 'This, many considered, was bad'

salacious_coaster 97 points a year ago

20 years ago. That should have been in the headline.

save

path: 0 18357420, hotness: undefined, score: 97, children: 4

Zorsith 15 points a year ago

20 years ago? Shit, that was nothing compared to modern day amazon.

save

path: 0 18357420 18358908, hotness: undefined, score: 15, children: 0

MHLoppy -19 points a year ago

Eh, that assumes that all posts are news posts. When I first saw this headline I didn't assume it was current news, but I'm also familiar with the "Who, Me?" column I guess.

save

path: 0 18357420 18357794, hotness: undefined, score: -19, children: 2

SheeEttin 11 points a year ago

In communities like this, yes, it's generally assumed that it's current events.

save

path: 0 18357420 18357794 18365244, hotness: undefined, score: 11, children: 1

MHLoppy -2 points a year ago

That's not the precedent set by other similar, recent submissions though, - nobody complained about this other submission talking about an event in 2004 (also ~20 years ago): https://lemmy.world/post/32137161

If you want to police submitting to this community with an unwritten rule about editorializing titles to include chronological information:

get the rule added to the sidebar instead of assuming people can somehow figure it out when there's no existing evidence of this precedent
enforce it consistently

It's not my favorite thing to have a few people who don't even make submissions getting upset at me for following the diabolical process of (1) reading the rules, (2) looking at previous submissions, and (3) copying a title verbatim from a reasonable source. If there's something that people are expected to follow, make it a written rule or give me access to your secret mind-reading device. This community isn't called sysadmin_news so I'm not sure how I'm reasonably expected to know this.

save

path: 0 18357420 18357794 18365244 18369634, hotness: undefined, score: -2, children: 0

dohpaz42 69 points a year ago

I’ve been a professional programmer for 19 years this year, and I caused 3-hour data loss on one of our major apps just the other week. Shit happens. Humans make errors; even experienced ones. We should strive to learn from it.

As they say:

Biting your tongue while eating is the perfect example of how you can mess up even with decades of experience.

save

path: 0 18358700, hotness: undefined, score: 69, children: 2

0_o7 12 points a year ago

Biting your tongue while eating is the perfect example of how you can mess up even with decades of experience.

Should have setup a staging enviroment and tasted there first before diving in. \s

save

path: 0 18358700 18366545, hotness: undefined, score: 12, children: 1

dohpaz42 13 points a year ago

Question: is the \s a Windows version of /s?

save

path: 0 18358700 18366545 18367943, hotness: undefined, score: 13, children: 0

redlemace 25 points a year ago

Or as we say in networking: "You're not one of us until you caused your first major outage".

(edit: typo)

save

path: 0 18357147, hotness: undefined, score: 25, children: 6

foggy 4 points a year ago

Yeah I mean, that guy learned a valuable lesson that day. Bet he never did that again.

I once deleted a whole web server that was serving like a dozen e-commerce clients. I blame digital ocean. Both cloning and deleting had you write the servers name to confirm. Come on. Jfc. The options were adjacent.

Anyways, yep. I double check every clone/delete operation that might possibly disrupt prod ever since.

save

path: 0 18357147 18357587, hotness: undefined, score: 4, children: 2

redlemace 2 points a year ago

Bet he never did that again.

That's why my current boss preferes typo's over bugs. The complete opposit of my former job where they always considered burning you down to the ground for a typo. The latter makes you search for fake causes and does nothing for long term quality improvement.

save

path: 0 18357147 18357587 18359623, hotness: undefined, score: 2, children: 1

lusterko 1 point a year ago

I believe you

save

path: 0 18357147 18357587 18359623 18362381, hotness: undefined, score: 1, children: 0

Landless2029 3 points a year ago

Yeah I threw like 1 week of email into a black hole unknowingly.

Instead of multiple shared mailboxes for different emails they used a master shared mailbox with a bunch of DLs all pointed to it.

The DLs were just single email contact to the one master mailbox.

Well I reinstalled Azure AD Connect (no backuo config) and didn't check off the damn contacts OU.

ALL THE EMAIL WENT TO A BLACK HOLE

After fixing the issue the email was still gone. Thankfully they had an antispam proxy service that kept a copy of all email. I restored all the aliases by resending. Managed to recover sales@company.com emails and the like.

It was the most idiotic design I've ever seen. No RBAC. Everyone just had access to "the mailbox" and they had mail filters to put it in folders. They had 3k employees too. Not a mom and pop shop at all.

I feel like some C-suite complained about "too many mailboxes" and the sysadmin "fixed it".

That person should be shot. Just because you can doesn't mean you should.

save

path: 0 18357147 18366269, hotness: undefined, score: 3, children: 2

Auth 3 points a year ago

You know what, one mailbox is dangerously based. The idea of 3k employees working out of a single mailbox is so entertaining to me when I dont have to support it haha. It must have caused so many outlook crashes among other issues.

save

path: 0 18357147 18366269 18368137, hotness: undefined, score: 3, children: 1

Landless2029 2 points a year ago

Not all employees had access to it but it was sales, accounts payable and marketing... They did manufacturing but that's still too many disciplines in one mailbox.

save

path: 0 18357147 18366269 18368137 18368231, hotness: undefined, score: 2, children: 0

JakenVeina 4 points a year ago

Hehe. The manager troll was perfect.

save

path: 0 18362709, hotness: undefined, score: 4, children: 0

Limeade3425 2 points a year ago

It's one thing to be known as the guy that crashed Amazon, it's another to be known as the "Under Qualified" guy that crashed Amazon

save

path: 0 18359004, hotness: undefined, score: 2, children: 0