What is your biggest non-destructive "oh shit" moment you had at work?

17 days ago by mkhopper to c/asklemmy

It's the late 90s and I was working in IT for a small (200 employees) company.
One of my tasks was performing daily data backups, which ran for about four hours and started after everyone else had left for the day.
We had an intercom system in the building, where if you picked up your phone and dialed a specific extension, you could page someone, or make announcements so everyone could hear.

One night I was in the process of running Ethernet cable from switches on the second floor down to individual terminals on the first floor, so there was a lot walking around the building.

I like to have music when I'm working, but I didn't have an MP3 player at the time, so I thought... "I have music on my PC and no one is around. Why don't I just...", and I went around and cranked the volume knob on every ceiling intercom speaker in the whole building. Then I dialed into the intercom from the phone in my office, put the handset up as close as I could to my PC speakers, and boom, music throughout the building.

Cut to a few hours later, it's late, I'm tired, the backup is finished, so I close down the system, hit the lights, lock the doors, and go home.

The next day, I come in, not thinking about anything from the night before, and get to work. About 11:00am, the owner's secretary got on the intercom to page someone.... and blew everyone out of their seats. There was an almost collective, "What the fuck??!!" from everyone, and an "Ohhh, shiiit" from me, because of course I immediately knew what happened.

I ran around like a madman, turning the volume down on all of the speakers that I had forgotten the night before, while saying, "No idea... Yeah it's crazy... No no, it's fixed now..."

No one ever found out exactly why the volume was so loud that morning.

Murse 81 points 16 days ago

Decade or so ago, logging into a work computer and suddenly the screen is just tiled with the word "virus", but a bit janky looking like it was a screenshot of the word that was enlarged and tiled with no thought to whether the right side of the image lined up with the left and such - it was a very 90s cheesy movie like depiction of what happens to your comp when you get a virus, like you'd expect the next scene to be some shady hacker on the other end exclaiming "I'm in!"

Then again, I don't know jack fuck about viruses, and the machine is behaving super weird, so I unplugged the internet cable from the back, and called my manager over. He had a similar "...is this for fuckin real?" reaction that I did. Mind you, this is a hospital, so if one of our computers is fucked, this is a big deal for potentially leaking patient info, so even though it had that kind of slapstick appearance, no one was fucking around with it: so, manager pushed it up the chain, and after a few mins a fucking SWAT team of IT guys comes in and starts doing their own assessment.

After some deliberation, they start troubleshooting - keystrokes and whatnot don't seem to be responsive. One of them eventually reaches back behind the monitor and 'poof' the screen pops back to the normal login screen.

So, that facility had a fancy custom login screen, which was some generic desktop background type scenery and a wall of text that none of us ever put any thought into. One of the snipits on that wall of text was instructions on what to do if you suspect your machine has virus (which was basically what I did - leave it running, unplug it from the internet, and call IT... hooray, I'm not completely stupid!). What ended up being the cause of the "virus" tiling was the fucking VGA cable slowing worming its way out of the back of the monitor - when I got to it, that thing was hanging on by a thread, and touching the desk it was on was just enough of a bump to make it cross that line between having enough of a connection to do its job vs not. That micrometer of pull of on that cable wasn't enough to completely cut its signal, but it wasn't enough to produce a normal display either: it just picked a random spot on the screen, and filled the monitor with that repeating image. The spot it just happened to do this to was the word "virus" in the custom login text explaining what to do if you think you have a virus, lol.

TLDR, got my management chain from my dept to the top dogs running the hospital, and the entire fucking IT department to swarm to my unit over a potential data breach that ended up being a wiggly VGA cable.

Whoops.

path: 0 26121566, hotness: undefined, score: 81, children: 1
W98BSoD 11 points 16 days ago

Better safe than cryptolockered.

path: 0 26121566 26136443, hotness: undefined, score: 11, children: 0
Davel23 70 points 17 days ago

Mid '90s, was doing something on the company's NT 4.0 PDC. The thing about NT 4.0, when the screen locked it would show the "Press Ctrl-Alt-Del to log in" screen, but if you hit a key or moved the mouse quick enough it wouldn't actually lock and wake up again. So, as I'm sitting there waiting for the server to do something the login screen shows up. I'm in the zone, so I lightning quick give it the three-finger salute, type in the password (usa1, security wasn't super-high) and hit enter. Another thing about NT 4.0, if you hit Ctrl-Alt-Del when the machine was not locked you'd get a box similar to XP's, with options to lock the computer, log off, shut down, etc. Now, you'll notice one of the options, shut down, starts with the letter "S". And the key to press to select that option is also "S". And you'll further notice that the password I typed in also has an "S". But all is good, it asks you to confirm before it actually shuts down, right? Except the shortcut to accept that option is Enter, which I hit after entering the password. So yeah, I shut down the company's PDC. Fortunately it was late in the day and most people had already left so the impact was minor. But I was doing some pants crapping for about five minutes while the machine came back up.

path: 0 26120874, hotness: undefined, score: 70, children: 1
W98BSoD 2 points 16 days ago

The longest 5 minutes of your life.

path: 0 26120874 26136292, hotness: undefined, score: 2, children: 0
Slatlun 53 points 17 days ago

I was working on prescribed fire as in controlled wildfire to support ecosystem function. The wind changed in expectedly. The fire jumped containment. We switch into actual firefighting mode for a solid 1/2 hour. Then the wind switches back to predictions. Problem solved. Mop up of the slopover still takes hours and it gets dark.

As we get back to the truck, keys are dropped and kicked in an attempt to catch them. There is tall grass everywhere. We search for an hour in the dark after an exhausting day for our only way to go home. Keys are found. We work at dawn the next day. Not my worst day in the office, just the least predictable.

path: 0 26121054, hotness: undefined, score: 53, children: 5
qyron 32 points 16 days ago

Always carry a magnet with you.

I work in installations where vegetation is allowed to grow nearly wild, except for the mandatory twice a year cut and I've had a situation like the one you describe, with a very important key falling out of a pocket.

We also usually carry a large magnet around, to pull open iron covers. It's about 15cm wide and 2cm thick. It can pull.

The key just vanishes into the plants covering the ground. We fetch the magnet, point it into the general direction of where we think we heard the key fall and cross our fingers. We were rewarded with a very satisfying metalic ding after a few passes.

Then it was the hassle of prying the key from the magnet.

path: 0 26121054 26122113, hotness: undefined, score: 32, children: 4
not 16 points 16 days ago

Next time put the magnet in a plastic bag.

path: 0 26121054 26122113 26123869, hotness: undefined, score: 16, children: 3
qyron 15 points 16 days ago

Next time, I'll chain the key to my belt.

path: 0 26121054 26122113 26123869 26124095, hotness: undefined, score: 15, children: 2
Slatlun 4 points 16 days ago

And if you're using an emergency vehicle in a potential emergency situation, leave the keys in the vehicle. No chains, no magnets. Just a truck ready to move. It is the standard operating procedure for good reason.

path: 0 26121054 26122113 26123869 26124095 26136642, hotness: undefined, score: 4, children: 1
thisisbutaname 50 points 16 days ago

It wasn't at work, but I think someone will find it funny regardless.

During my time at university I had a web development exam for which we had to do a group project, you guessed it, making a website. This included having a bunch of different input fields, and the requirement was to properly validate each one and provide clear feedback to the user.

While adding some inputs I was focused on having all the checks in place correctly, and so I decided to put some placeholder text for the errors and do a pass later changing them with more informative ones. Which I did, but I forgot about one.

So, presentation day comes along, the professor is looking at our website, asking us to show him that we had all the checks in place and correct. He asks me to input an invalid date in a calendar widget, which I happily do, knowing I 100% had all the required checks in place. The verification runs and fails, showing the error popup I set up, which read something along the lines of:

"date's fucked, mate"

Shit. My placeholder message. Which I obviously forgot to change. I was mortified. Luckily the professor chuckled and said that technically the check worked so that was fine.

path: 0 26122952, hotness: undefined, score: 50, children: 0
kossa 49 points 16 days ago

Worked in an ESA lab with cleanroom. My work had to do with plants, their composition etc.

Naturally I had to dry some plants before throwing them into the analyser, but my designated oven was occupied.

Fear not, I thought, there is another oven over there. I put my samples in and call it a day.

Come over the next day, kinda late, lab chief looks at me and is only like "ummm, you should head over to the offices and see big boss"...go over, boss guy very angry. Turns out, the other oven in the lab was reserved for ExoMars parts. Like, they disinfect the shit out of it for weeks, like, take it apart, haul every screw to the cleanroom, clean it, autoclave it, clean it again and repeat that a billion times over, certify it and clean it again. So I kinda destroyed the work of months, maybe I alone was responsible for one of the many launch delays (probably not, but I've never been told).

Arrogant young me in my first job basically just was like "maybe put a sign on it next time, lol", boss guy pauses and says "maybe you're right". So we reclean the shit out of the oven and put a sign on it. My "punishment" was to spend a lot of time in the clean room to clean some ExoMars parts. I thought "cool, stuff I had in my hands is going to land on Mars".

Fast forward some years, Russia invades Ukraine, the pipes I cleaned will never go to Mars 😢.

path: 0 26123786, hotness: undefined, score: 49, children: 1
helpImTrappedOnline 17 points 16 days ago

You were absolutely right..... Two ovens in a room, 1 for plant use and 1 that needs to be super clean for a specific purpose? A sign would have been the bare minimum, that should have been part of your lab training too.

path: 0 26123786 26136843, hotness: undefined, score: 17, children: 0
Jarlsburg 48 points 16 days ago

Years ago I was working as a pharmacist in a very busy pharmacy in a city center. This was during the opioid epidemic in America and being a high volume store, had a lot of regulatory oversight on our controlled substance dispensing.

One Friday evening a young kid came in with a prescription that he said was for his grandmother. It was for a lot of oxycodone from a city two hours away. He had two girls with him when he dropped off the prescription and had never filled with us before, and wanted to pay cash. We had a policy that we didn't fill prescriptions for maintenance controlled substances for new patients >100mi from our address. We only got so much of each controlled substance so we adopted the policy so we'd have enough for our regular patients. I told him we couldn't fill it. He started to get angry and mouth off but left.

Cut to Monday which was my next shift and there is a folder with the DEA logo on the front of it in front of my terminal. I ask what it is to someone who worked the weekend and they said an agent dropped it off. I open it and the only thing inside is a business card slotted into those slits you sometimes see on the face of one of the pockets. I call the number and after some introductions the agent asks me if I was the one who denied the prescription on Friday. I confirm and the guys says that it was part of a multi-jurisdiction sting operation to catch a mid-level dealer who was buying scripts from people. He told me I wasted a bunch of their time and next time to just fill it. I told him it's emphatically my job to not do that and that I didn't appreciate him using my store since we had a bunch of other issues that local police weren't dealing without him adding to it but he just hung up. Whatever.

My heart dropped when I saw that folder though and I'm sure that is what he wanted when he did it.

path: 0 26121578, hotness: undefined, score: 48, children: 8
diablicja 48 points 16 days ago

Wait, so the DEA agent essentially scolded you for NOT contributing to the problem and ordered you (well, tried to) to be part of the problem going forward? What a fucking joke. If he wanted you to participate in this super important multi-jurisdiction sting he could've contacted you (as he's clearly capable to do) beforehand. The only way the agency comes out of it not looking like a clown car is if there was no sting and he was an impersonator, with falsified documents and a fake folder, looking to establish a steady drug source. It might have been worth it to report it to his boss, but maybe I'm too European to properly understand all the risks involved in doing so.

path: 0 26121578 26123064, hotness: undefined, score: 48, children: 5
nwtreeoctopus 22 points 16 days ago

Remember, the US government secretly poisoned alcohol during Prohibition and just said, "guess you shouldn't have drank it" when people got sick and died. Enforcement is its own end and purpose; improvement and protection of the lives of citizens is secondary at best.

path: 0 26121578 26123064 26126698, hotness: undefined, score: 22, children: 0
Jarlsburg 15 points 16 days ago

Correct. He was clearly just mad that I frustrated his plans. I didn't even consider reporting it because it wouldn't have resulted in anything. We had a bad reputation with the DEA since we were a high volume pharmacy so we had more staff and high turnover. All that led to more than the average amount of employees stealing/selling drugs. We had some pretty sneaky ways of catching it and always reported but all that meant is more work for them and not much credibility.

path: 0 26121578 26123064 26129346, hotness: undefined, score: 15, children: 2
LaLuzDelSol 8 points 16 days ago

That's really weird. Is it possible that wasnt a real agent just someone impersonating one?

path: 0 26121578 26123064 26129346 26130948, hotness: undefined, score: 8, children: 1
Jarlsburg 13 points 16 days ago

It was legitimate. They arrested the kid later that day and it was in the news. I did have that thought though.

path: 0 26121578 26123064 26129346 26130948 26133458, hotness: undefined, score: 13, children: 0
captainlezbian 13 points 16 days ago

Also maybe cops shouldn't be taking drugs out of the metered and monitored supply lines

path: 0 26121578 26123064 26129627, hotness: undefined, score: 13, children: 0
BillCheddar 10 points 16 days ago

He asked you why you didn't help someone else commit a crime and you just let that shit go?

You should have been up his supervisor's ass, with a reporter on your left and a lawyer on your right.

Not saying anything is why these fucking Nazi pigs get to keep being Nazi pigs.

path: 0 26121578 26131074, hotness: undefined, score: 10, children: 1
Jarlsburg 3 points 15 days ago

I get the sentiment, but it's not that simple. The kid picking up someone else's script isn't a crime. Me denying or filling the script is a professional discretion. Also the company I was working at would not have supported me.

path: 0 26121578 26131074 26148223, hotness: undefined, score: 3, children: 0
justdaveisfine 45 points 17 days ago

So for a game I published, there are male/female characters with what I called undermeshes, which is basically underwear that's painted onto the skin texture. By design, there is no "nudity", as the underwear is the most exposed you can get.

At the time, there were two torso 'top' textures, both created from a base design. One was a sports bra looking thing for women, and just a bare (notably not hairy) chest for men.

Things were getting crunchy and I had a few NPCs that I was trying to dress differently for a town to make them distinctive. There was an unused semi-low-cut dress asset so I put that on a single NPC, but it looked funky with the sports bra.

So I assigned the male torso top texture to the lady so the bra wasn't showing, made sure nothing scandalous was visible, and moved on.

Game gets released.

Weeks later, I'm troubleshooting a quest dialogue bug, and recalled this particular NPC has different lines (and animations!) if you're a specific sex/race.

I swap races and do the whole quest again, only to (re)discover that the reason the dress was unused was because it wasn't 100% properly set up. So when the NPC moved about with the different animations, the dress didn't move with her, making her temporarily very topless and with the bare male texture, showing a lot more than she was supposed to be able to.

In short, this is quite bad and could have resulted in a not-too-happy e-mail from Valve if someone else found it.

Fortunately (AFAIK) I'm the only one who caught it but it was a cold chill followed by an absolute panic hotfix.

path: 0 26121249, hotness: undefined, score: 45, children: 3
teslekova 23 points 16 days ago

"hot fix 1.1.6 - stole titties"

path: 0 26121249 26121526, hotness: undefined, score: 23, children: 2
popekingjoe 11 points 16 days ago

The current Lemmy meme trend shit posters would murder op for this.

path: 0 26121249 26121526 26122575, hotness: undefined, score: 11, children: 1
W98BSoD 5 points 16 days ago

path: 0 26121249 26121526 26122575 26136471, hotness: undefined, score: 5, children: 0
ChickenLadyLovesLife 39 points 16 days ago

I used to work at a big cable company whose name rhymes with bombast. One day I (M) found myself in a meeting with a bunch of female vice-presidents (Bombast has thousands of vice-presidents). The presenter couldn't get her laptop to connect to the big TV so I pulled out one of my gadgets and got her set up. She said "wow, you're pretty handy to have around." My brain decided that it would be a clever thing indeed to say "well, I'm pretty well-endowed in the dongle department" but when I started to say it my brain realized how completely insane a comment it was. So my brain cut me off and I ended up just saying "well, I'm pretty well-endowed" which was not exactly any better.

To their credit everybody ignored this except for one woman who actually guffawed briefly. I wasn't fired immediately so there's that.

path: 0 26126242, hotness: undefined, score: 39, children: 1
helpImTrappedOnline 6 points 16 days ago

To be fair telling someone "wow, you're pretty handy to have around" is opening the door for all kinds of inappropriate comments...

path: 0 26126242 26136749, hotness: undefined, score: 6, children: 0
Pudutr0n 37 points 17 days ago

Pretty awesome you put your music on the entire building ngl. Glad you didn't get caught too.

I can't really remember anything comparable of my own, but enjoyed the read. ty OP.

path: 0 26120260, hotness: undefined, score: 37, children: 0
JonnyW 32 points 16 days ago

Not at work but in High School.

It was the 90s and computers at school had just become a thing. Our school offered a typing class which seemed pretty chill so I signed up for that. I think I was the only guy in the class.

Anyway, I started talking to this girl who sat across from me and we ended up just sending emails back and forth to each other all through class, using our school emails. This went on for weeks. The messages started to get kinda flirty and then eventually they got more than flirty.

So I’m sitting in another class one day, maths or something, and a teacher comes in and asks if I can go with her. She takes me to the principle’s office and sits me down. In walks the principle with typing girl and she sits her down next to me. The principle then dumps a massive pile of paper on the desk and asks us to explain… we of course have no idea what she’s talking about. I pick up a sheet and start reading and my heart drops. It’s our entire email chain, weeks of cringe and filth. They must have been monitoring our emails, right? Wrong.

So we get a bollocking, threatened with suspension, the whole deal, but it wasn’t until we were sent on our way that we started to realise what had actually happened and the gravity of the situation. Over the next few hours some really weird shit started happening. I’d be walking through school and kids I didn’t even know would just start pointing at me, shouting out random things. Even my teachers were being super weird. Everything just felt really off and then my mate filled me in.

Now, to this day, I still don’t know how this happened. I genuinely think that our school email system was set up wrong but somehow, when replying to one of typing girl’s messages, I had forwarded our whole email chain to every person at our school. Every kid. Every teacher. Thousands of them.

It was humiliating.

I was forever known as that guy. Typing girl never talked to me again.

TLDR: I accidentally forwarded weeks worth of dirty emails to our entire school and held on to my virginity.

path: 0 26124763, hotness: undefined, score: 32, children: 2
Prime 14 points 16 days ago

That sucks. I think the blame is on them for misconfiguration.

path: 0 26124763 26125163, hotness: undefined, score: 14, children: 0
some_guy 5 points 16 days ago

I accidentally forwarded weeks worth of dirty emails to our entire school and held on to my virginity.

As a fellow pro in the keeping-virginity in high school dept, game recognizes game.

path: 0 26124763 26136645, hotness: undefined, score: 5, children: 0
zxqwas 28 points 16 days ago

"Shutdown -P now" in the wrong terminal window and shut down the production server on the other side of the country instead of the test server on my desk. No one noticed but I had to call the IT guy on site to turn it on.

Yanked a cable in an electrical cabinet we were installing and blacked out the entire workshop at the 500 level in the mine. Had to go to the electrician and ask for the key to the transformer station and turn it back on.

path: 0 26122449, hotness: undefined, score: 28, children: 0
Rocketpoweredgorilla 28 points 17 days ago

First thing that comes to mind is driving semi in Saskatchewan once I came across an older slower moving car (a Chrysler K car) full of people on the highway. I pulled out to pass them, and as I was passing the car I saw in the mirror that they started to wander into my lane. I ended up putting half the truck into the ditch to avoid them getting under my trailer. I had 9 tires on the grass before they realized what they were doing and swerved back into their lane. They had wandered far enough over that they were completely in the oncoming lane at that point, leaving me nothing but shoulder and ditch.

I was fully loaded (around 80,000 lbs) If I hadn't watched my mirrors as closely as I did and they got under my trailer they would have been nothing but a smear on the road.


Edit: Another time When I was 18 I was working at the Wpg international airport. A baby had died in a house fire up north and it was sent down to Winnipeg for an autopsy. I guess the people working up there had run out of cargo tags so they had put a regular baggage tag on the bag the body was in, and neglected to tell anyone. My job was to unload aircraft, so seeing a regular baggage tag I just tossed it onto the cart and it went to where you pick up your luggage. It was there for about 40 minutes before an ambulance showed up asking for it, and none of us had any clue about it. Thankfully nobody snagged it because they would have been in for one hell of a surprise.

path: 0 26120998, hotness: undefined, score: 28, children: 2
Whats_your_reasoning 4 points 16 days ago

For your first story, I'm having trouble picturing it. How many lanes were in that road? Was it a "one lane each direction" deal, so when you "pulled out to pass them" you crossed a dashed yellow line into the oncoming lane?

What a terrifying situation. I've seen some bad drivers, but that driver must have been on another level.

I have no words for your second story. Such a tragic situation that could've wound up even worse in so many fucked up ways.

path: 0 26120998 26137244, hotness: undefined, score: 4, children: 1
Rocketpoweredgorilla 4 points 15 days ago

Yup it was just a regular "one lane each direction" road, just outside Yorkton Sask. I don't know for sure, but I think they were probably drunk or stoned out of their tree. Either way they were not paying attention to what they were doing.

The second one I felt pretty bad about, although it wasn't my fault. Even my supervisor told me he would have done the same thing because it was tagged as regular baggage. When I was unloading the plane I had tossed it down onto the baggage cart and it made a thud noise... I had just assumed it was just someone's shoes or something, not a burnt kid in there. It had come from way up north so It's understandable I guess that they didn't have the proper means of marking it, but at the very least a heads up would have been nice.

path: 0 26120998 26137244 26137878, hotness: undefined, score: 4, children: 0
LeftReddit2 25 points 17 days ago

I don't have any non-destructive ones. I have plenty of destructive ones if you are fascinated, however.

path: 0 26121013, hotness: undefined, score: 25, children: 5
crank0271 21 points 17 days ago

I am pre-fascinated

path: 0 26121013 26121201, hotness: undefined, score: 21, children: 4
LeftReddit2 23 points 17 days ago

So I work IT and I am damn good at it, relatively speaking.

Recently, I had a caller contact me, she was a coworker but in a different department, about an issue they were having with their laptop of it not allowing them to use some tool or something? I forget the specifics. But I checked the laptop and saw it had an uptime of like a month and a half. So I thought, oh just give it a reboot.

I sent a command "RESET COMPUTER" and as it turns out that was a fucking factory reset! So she then lost absolutely everything off her laptop. I got off with no real issue as all my boss did was give me a stern talking to, but no write up or pay dock or demotion or whatever, but holy shit that coworker? She was pissed at me.

I made it up to her by taking her out for drinks and letting her spend all my money, sobbing.

path: 0 26121013 26121201 26121287, hotness: undefined, score: 23, children: 3
elvith 7 points 16 days ago

Not your problem. She could have lost everything to an unfortunate drop of her laptop on her way to a meeting room five minutes later anyways. There should be backups of the device. Or all work needs to be saved/synced to a network drive regularly and it should be made clear to everyone that one doesn't save important data locally.

It was a surprise test of her business continuation planning.

path: 0 26121013 26121201 26121287 26124214, hotness: undefined, score: 7, children: 1
helpImTrappedOnline 3 points 16 days ago

That's also on the IT department for not implementing a backup solution for users (or upper management not listening to IT that a solution is needed).

If a work laptop dies and the work files are lost, that sounds like a work problem.

path: 0 26121013 26121201 26121287 26124214 26136970, hotness: undefined, score: 3, children: 0
W98BSoD 1 point 16 days ago

I’ve been there. Ran a robocopy to copy files to a server I had already put into production.

Had to restore files from the backup because the source was already 1+ weeks stale.

I’ll never make that mistake again.

path: 0 26121013 26121201 26121287 26136547, hotness: undefined, score: 1, children: 0
slazer2au 18 points 16 days ago

In March of 2017 we were running several unpatched server 2012 systems with just about every port open to the internet. Uptimes of 500ish days kind of servers.

Took a couple weeks to patch everything and lock it down. Mega glad I did because May in 2017 was an interesting month for IT.

Also, community plug !talesfromtechsupport@lemmy.world exists please share stories.

path: 0 26122886, hotness: undefined, score: 18, children: 0
captain_aggravated 18 points 16 days ago

Tower cleared me to position and hold (back before it was called line up and wait), then cleared a 737 to land on the runway I was sitting on.

Now it was day VFR conditions and my little ship was painted bright red and white, I wasn't exactly camouflaged, but I called the tower with my entire tail number and position. RALEIGH TOWER, NOVEMBER ONE TWO THREE ALPHA BRAVO HOLDING POSITION FIVE RIGHT.

Made a great lesson for my student, the tower WILL forget about you and clear a jet 500 times your mass to land on you.

path: 0 26137099, hotness: undefined, score: 18, children: 0
expletivedeleted 16 points 16 days ago

Had a content filter server I had built. Dansguardian.

Installed some new packages, which I had wgot (joking) at / as root. Fuck, I was a dumbass kid.

After I installed the packages, I created the folder /installs and then: holy shit, mv is naturally recursive?!

mv * /installs

Well, let's ls to see if the root directory is clean.

ls not found.

Um... ok, this is a call center, seems to still be working as everything is still in RAM, but as it is working as a proxy for content filtering, you could lose the company MILLIONS, so think fast!

I rebuilt another server, swapped it with ninja speed, fixed the other server, and swapped it back with ninja speed.

And everyone asked me if I was ok. "Nothing. I mean, yeah, good."

edit: mv is recursive by default, in contrast to cp, which requires -R. Scary lesson to learn on the job.

edit 2: just in case someone doesn't understand this, I moved the entire operating system into a subfolder /installs in a production environment...

If the computer had rebooted, no one would have internet at all, as DNS would fail. Fortune 500 company. Ah damn, that was some stress.

"Hey, is everything working alright?"

"Yeah."

"Hey is everything still working? Everything good?"

"Yes, why do you keep asking that?"

No raisin.

path: 0 26136391, hotness: undefined, score: 16, children: 0
Lehmuusa 16 points 15 days ago

I worked for the railways, in commuter trains. My job was to harass innocent passengers by walking back and forth and enabling them to buy a ticket should they need one. And at certain routes also check the tickets of every new passenger.

It was nice using a dialect from eastern Finland, the Savo dialect for that. It has a tendency to change plain vowels for diphtongs. It adds a kind of a "wee-wow" twang to the language. For example the word for "closet" is "kaappi" in standard Finnish, "kuappi" in the dialect spoken in Savo. The "aa" is just a long "a", sound. Pretty much the same as the "a" in the English word "car". And "ua" is pretty much like "oi" in "choir". Imagine pronouncing the word "car" with the "a" replaced with the "oi" sound of "choir" and you might get an idea of how that crazy dialect likes to mangle every damn word. And the "ar" in "garlic" would be pronounced with the same change as well. Etc.

And maybe at this point it's worth mentioning that I still had a fully male gender expression at that point.

So, we have just left the station of Haarajoki. In the Savo dialect that would be "Huaarajokj".
I enter the part of the wagon through its end door. I start shouting "Haarajoki passengers, tickets please!", but. But.

As it happens, "huaara" is how they would pronounce also the word "huora" over in Savo. And that happens to mean "whore". Luckily I noticed. So, I managed to stop myself from uttering anything further. Unfortunately, I only managed to stop after having already said the first half of that word, "huaarajokj": Yes, "huaara".

In other words, I entered the passenger wagon through the end door in the railway worker's uniform, immediately bellowing "WHORE!", and then shutting up.

Definitely an "oh shit" moment!

After that:
Me: "Hm. This station is not terribly compatible with the Savo dialect, apparently."
A female passenger: "You should have continued to the end, would have gone much less badly".
Me: "Yup!"

path: 0 26141918, hotness: undefined, score: 16, children: 0
myyass 15 points 15 days ago

Spot welded a screwdriver to the metal sidewall when I slipped trying to discharge a capacitor. At least I thought to wear gloves.

path: 0 26138422, hotness: undefined, score: 15, children: 0
some_guy 15 points 16 days ago

The big annual release was coming soon and I was leading a small group of testers with after hours testing of a feature that was my responsibility to cover. It was already around 8pm and we'd been working for hours. Someone came and informed us that a specific device was incorrectly configured, thus invalidating all of the tests we'd run thus far. It was mine. What an asshole.

I told everyone to give me their devices and go home. I stayed until 2-3am running all of the tests myself. My manager video called me a couple of times and he had a good sense of humor about it, I think because I took responsibility for it and didn't burden others with my mistake. I really felt terrible but after I got the report in before the deadline later that morning, no one ever gave me a hard time about it and it just was like it never happened.

Not trying to deflect or make excuses and owning a mistake is like a super power. An imprortant lesson that has basically never failed me when I've screwed up at work.

path: 0 26136588, hotness: undefined, score: 15, children: 2
W98BSoD 7 points 16 days ago

Seconded on owning a mistake. It might get you reprimanded, but it usually doesn’t get you fired.

path: 0 26136588 26136818, hotness: undefined, score: 7, children: 1
db_null 12 points 15 days ago

If it gets you fired the company culture turns toxic very quickly. I've worked for a place like that once and quit when I saw that bs.

path: 0 26136588 26136818 26138035, hotness: undefined, score: 12, children: 0
_haha_oh_wow_ 14 points 16 days ago

Not physically destructive but someone wrote a script that was supposed to clean up unused phone extensions and decided to run it in production without testing it. It wiped out all the phones.

We probably spent a week or two fixing and testing thousands of phones and the person who ran the unvetted script was fired, but nothing was physically destroyed.

path: 0 26128278, hotness: undefined, score: 14, children: 0
AnthropomorphicCat 12 points 15 days ago

Fortunately not as serious as other comments. For context, I have autism, and I frequently stim by making random noises.

One day I was at a global online meeting with very important people. I thought I was muted, so I said "MEOW".

Suddenly everybody goes quiet and somebody says "did anybody say 'meow'?"

I quickly turned off my microphone, and fortunately nobody mentioned that again.

path: 0 26149479, hotness: undefined, score: 12, children: 0
GreenKnight23 12 points 16 days ago

not me, someone else.

someone deployed a change to the development AWS account at work before a holiday that caused a lambda to execute over 60 million times in a week.

not bad until you find out that the lambda was responsible for creating EC2 instances.

1000004659

path: 0 26125450, hotness: undefined, score: 12, children: 2
Trail 4 points 16 days ago

And then?

path: 0 26125450 26128721, hotness: undefined, score: 4, children: 1
GreenKnight23 5 points 16 days ago

not only was the dev summarily dismissed, the monitoring team that was responsible for escalation was sternly talked to by the CFO. evidently there were alarms going off that entire week but nobody did anything about it because "it's in dev, it's always noisy."

they actually worked it out with AWS and was able to get the bill reduced but it was still more than $500k (at least that's all my boss would let me know).

path: 0 26125450 26128721 26130685, hotness: undefined, score: 5, children: 0
fruitycoder 11 points 16 days ago

Got creative at work, i was managing my crews .profiles so we could share aliases and such. I got to clever and did a symlink so we all had the same exact one. Then i tested a change that made logins fail.

Luckly it was just the admins during downtime but we had to have the account team make a temp account lol

path: 0 26131049, hotness: undefined, score: 11, children: 0
Treczoks 10 points 15 days ago

I was asked to provide technical on-site support at an event, because I knew the underlying server system under the software that project management had found to run this customer event.

They had also hired a bunch of laptops with wifi and a router for this, which was rather expensive back then (you could have bought a small car for that rent fee alone!), so a lot was hanging on this to be a success.

Of course, I knew about nothing what this job was about until I sat in the car with the project manager on our way to the first event. My reaction was: But that software cannot do what you just told me! Oh shit, indeed.

When we arrived in the evening before the first event, the server, wifi, and one of the laptops was set up before anything else, and I went to work writing a rudimetary software from scratch that would do what the customer wanted, and finally hit the bed around sunrise.

The event was a big success in all venues on the trip. The customer was extremely happy with the acceptence of the setup by the audience and the results they got.

path: 0 26138913, hotness: undefined, score: 10, children: 0
LeighWeighTacoma 10 points 17 days ago

Haha that's great! I cant think of one off the top of my head but I'm sure I have plenty.

path: 0 26120201, hotness: undefined, score: 10, children: 0
palordrolap 10 points 16 days ago

I once took down a production server with a script that was an unintentional resource thief. The "oh sh-t" was watching the script grind to a halt... and then the phone started ringing.

Borderline on the non-destructive there because the server recovered fine after a reboot and nothing was lost except uptime. Sometimes a server would choke anyway, and the company assumed it was that, but that time it was my fault.

I had to answer more than the one call regarding that, feeling sick to my stomach, so I didn't escape unpunished.

path: 0 26124560, hotness: undefined, score: 10, children: 0
nik9000 8 points 15 days ago

Tooks out search on Wikipedia. I exposed a path that can perform an unconstrained Powerset Construction concurrently on 7 servers. It filled up memory and died. User retried. While cluster OOMing.

path: 0 26142714, hotness: undefined, score: 8, children: 2
Vex_Detrause 2 points 15 days ago

ELI-non-IT this "oh shit" moment?

path: 0 26142714 26151730, hotness: undefined, score: 2, children: 1
nik9000 2 points 15 days ago

Wikipedia has a search feature. Not many people use it, but Wikipedia is huge, so enough people do. Back then, a couple dozen full text searches a second. And maybe 100 quick prefix searches.

Anyway, one thing I made for it was a way to search the wiki text with regular expressions. It's useful for maintainers.

Wikitext is the funny markup language Wikipedia uses. If you know html or md it's like that, but it can kind of "call" other pages. Whatever. Doesn't matter. It's text. And people have strong opinions about it.

You can see it by clicking "edit" (sometimes looks like a pencil) and then clicking "edit without logging in" then "source editing". Those lovely boxes at the top for a person are made with something like

{{Infobox person
| name               = Douglas Adams
| birth_place        = [[Cambridge]], England
}}

Imagine you have 1000 articles that use birthplace instead of birth_place. Both work, but it's driving you and all your editor friends crazy there are two ways to do it. This is believable. Trust me.

You want to find every page that has an {{Infobox person followed by birthplace. Then fix them. The nerds who came before us invented a language to ask that question called "regular expressions". The one for this looks like \{\{Infobox\wperson.+birthplace. Maybe. Nerds will know regular expressions are bad for this. But nerds will also know that regular expressions being bad has never stopped anyone from using them anyway.

One way to run these regular expressions is to convert them into an https://en.wikipedia.org/... . You build these in memory and they are not big. But you can't really run them directly. They are pretty and fairly easy to read once you get used to them. But you can't easily ask "does this match this text". At least, not with the tools I had.

I could only run https://en.wikipedia.org/... . It's deterministic! Much nicer. And you can go from a nondeterministic one to a deterministic one. Easy. Our forenerds solved the problem. You use a https://en.wikipedia.org/... .

The trouble is, it can make very very very big deterministic finite automata. Like, if there are 3 states in the nfa you can get and 8 state dfa. 4 is 16. 5 is 32. In the worst case. Usually you get much better. But a fairly big but not super frightening regular expression can turn into a big nfa. And the dfa would need more states then there are grains of sand on earth. Too big for computer.

So, user asked for this regular expression. The search servers tried to convert it, and filled up their memory and died. They ran Out Of Memory. OOM. The usual thing is you copy the full memory so you can look later and start the server again.

But that can take time. And sometimes people are confused and haven't set up the restart to be automatic. And they can't find you. So search stays broken for a while.

path: 0 26142714 26151730 26152320, hotness: undefined, score: 2, children: 0
d00ery 8 points 16 days ago

Added a bug that would delete data from a form but only when they reopened it and didn't paginate all the way through to the end of the form. Took ages to discover and a big job to review all the potentially missing data.

Disabled the incorrect password so that it would allow login no matter what password you use. (Original system wasn't mine and I completely agree with "don't roll your own security").

path: 0 26130590, hotness: undefined, score: 8, children: 0
SpatchyIsOnline 8 points 15 days ago

Not anything destructive (to the Company) per se, but seeing a meeting with a senior and a member of HR suddenly pop up on my calendar after weeks of waiting in limbo to be placed on a new project was definitely an OH SHIT moment.

Yep, I was made redundant 🙃

path: 0 26150440, hotness: undefined, score: 8, children: 0
ace_garp 8 points 15 days ago

Accidental reply-to-all set on mailman list, of 8,000 users.

Only minor at about 16 minutes and 11 replies shared.

Still had a little bit of cold-sweats, and dolly-zoomed my surroundings.

path: 0 26138282, hotness: undefined, score: 8, children: 0
GreenBean 7 points 15 days ago

Not mine, but another's. She stating in her Foghorn Leghorn voice "I want to say "I can't hear you over your hair. It's too bright!!" to her". As I round the corner in my short Spiked Blazing Orange hair. How embarrassing for someone.

path: 0 26150778, hotness: undefined, score: 7, children: 0
GalacticGrapefruit 7 points 15 days ago

I put the localization files in the wrong folder. Thankfully, we hadn't pushed anything to prod yet, but we were trying to figure out why everything kept crashing even though our checking tool was giving the all-clear.

It was checking everything except the localization folders.

path: 0 26142124, hotness: undefined, score: 7, children: 0
LovableSidekick 7 points 15 days ago

At an online gaming company back in the late 90s, I deleted a file folder whose name was my first name. I had been there about a month and figured I must have created that folder for some dumb reason while I was ramping up. Turned out it was created by a previous software dev who happened to have the same first name as me, and it contained the production software. Essentially I blew away a good part of the external website.

Fortunately somebody found another copy and restored everything, named it something sane and put it in a sane place with access control.

Admittedly it was sloppy of me to delete a folder without at least looking inside it, but I really didn't feel too bad about it because standards at this place were so low - they had no concept of source control, dev/test/prod environments, or much else. Everything they did was cowboy.

path: 0 26139152, hotness: undefined, score: 7, children: 0
Psythik 5 points 16 days ago path: 0 26130667, hotness: undefined, score: 5, children: 0
sukhmel 5 points 15 days ago

Reading through the comments, I remembered my first day on my second ever job, where we were instructed to turn off computers and power strips too, at the end of the day. When I turned mine off at the end of the day, I found out other power strips were using mine, so I turned off other people's machines. Unfortunately, the guy that gave us the instructions was first to leave in the evening and hadn't have his machine turned off that way.

Also remembered that I once broke production server (maybe it didn't get to production, can't remember) by adding a single line of log, I didn't notice that what I was logging was computationally intense, so it halted everything eventually, but it was barely reviewed because it's just one log line

path: 0 26144712, hotness: undefined, score: 5, children: 0
FlashMobOfOne 5 points 16 days ago

For me, it's occasionally making a code update to a website that has evident and immediate visual glitches. But, I always make these updates on a staging platform before pushing them live, so it ends up being a funny thing in the end. I actually have a folder of my "bloopers" that I keep for my amusement.

path: 0 26126608, hotness: undefined, score: 5, children: 0
Typotyper 4 points 15 days ago

I was working as a short order cook back in the 80s. One night after closing we were having a few beers as a farewell to a co-worker. We wanted some music so we taped our walkman earbuds to the intercom microphone. Came in the next day, our busiest day of the week and found out we had blown the amplifier.

path: 0 26151693, hotness: undefined, score: 4, children: 0
octobob 2 points 15 days ago

I'll tell you my biggest destructive moment was getting a single phase 440v/50hz motor from Europe for some equipment (I'm in the states) and it had the worst nameplate I've ever seen on the motor of whether to remove the capacitor across the two phases or not. Turns out the cap got left in only on 220v. We put 480v/60hz on it, which is within 10% so it's fine but the cap should have gotten removed. Stupidly I only hooked up to like a 30A twist-lock connection off the wall for testing before it got installed rather than the 6A it needed in the panel. The cap burned itself out, started a small fire, caught all this soundproofing insulation crap on fire. Couldn't get another motor for months. Big whoops

path: 0 26143042, hotness: undefined, score: 2, children: 0
randombullet 1 point 16 days ago

I did the very classic "sw t al v 320" which effectively isolated an entire floor.

path: 0 26121835, hotness: undefined, score: 1, children: 9
NeatNit 8 points 16 days ago

what the hell are you talking about

path: 0 26121835 26123611, hotness: undefined, score: 8, children: 8
not 14 points 16 days ago

"switchport trunk allowed vlan 320"

The classic mistake: On a trunk port uplinking a floor's switch (Cisco IOS), running that command replaces the list of allowed VLANs with only VLAN 320, instead of adding 320 to the existing list.

A trunk carries traffic for multiple VLANs to other floors/upstream, suddenly permitting just one VLAN effectively cuts off every other network on that floor

path: 0 26121835 26123611 26124022, hotness: undefined, score: 14, children: 0
randombullet 5 points 16 days ago

So imagine a road named VLAN1000 that has assigned lanes. (Native VLAN 1000)

You got 4 lanes:

1 lane for bikes (VLAN 100)

1 for motorcycles (VLAN 200)

1 for cars (VLAN 300)

1 for busses (VLAN 310)

I want to add another lane for trucks so I add

1 for trucks (VLAN 320)

The road is named VLAN1000. The entire road system from the basement to the top floor is called VLAN1000.

I wanted to add VLAN320 as stated above... But I put in a command saying that the road between the basement and the top floor is now called VLAN320.

Now all of the other traffic cannot go to VLAN1000 because the switch no longer has that VLAN. It only has VLAN320.

Might be hard to follow but this is pretty much how I think of it in my mind.

path: 0 26121835 26123611 26126531, hotness: undefined, score: 5, children: 6
NeatNit 6 points 16 days ago

You can't seriously expect most people to be able to decipher that just from "sw t al v 320". Even for the relatively tech savvy demographic of Lemmy, that's very niche.

path: 0 26121835 26123611 26126531 26135053, hotness: undefined, score: 6, children: 5
W98BSoD 2 points 14 days ago

No, no, this is the right level of snark.

Or N,N, rgt lvl snark.

path: 0 26121835 26123611 26126531 26135053 26164385, hotness: undefined, score: 2, children: 0
W98BSoD 2 points 16 days ago

Agreed.

Source: My user name.

path: 0 26121835 26123611 26126531 26135053 26136801, hotness: undefined, score: 2, children: 1
crimsonpoodle -2 points 16 days ago

He gave you a nice explanation and you’re still snarky. Even if it was niche it wasn’t a crime and everyone got a +100 for social interaction for the day: the horror.

path: 0 26121835 26123611 26126531 26135053 26136737, hotness: undefined, score: -2, children: 1
asklemmy
asklemmy

@lemmy.world

login for more options
41886
9887
9017

A Fediverse community for open-ended, thought provoking questions


Rules: (interactive)


1) Be nice and; have fun

Doxxing, trolling, sealioning, racism, toxicity and dog-whistling are not welcomed in AskLemmy. Remember what your mother said: if you can't say something nice, don't say anything at all. In addition, the site-wide Lemmy.world terms of service also apply here. Please familiarize yourself with them


2) All posts must end with a '?'

This is sort of like Jeopardy. Please phrase all post titles in the form of a proper question ending with ?


3) No spam

Please do not flood the community with nonsense. Actual suspected spammers will be banned on site. No astroturfing.


4) NSFW is okay, within reason

Just remember to tag posts with either a content warning or a [NSFW] tag. Overtly sexual posts are not allowed, please direct them to either !asklemmyafterdark@lemmy.world or !asklemmynsfw@lemmynsfw.com. NSFW comments should be restricted to posts tagged [NSFW].


5) This is not a support community.

It is not a place for 'how do I?', type questions. If you have any questions regarding the site itself or would like to report a community, please direct them to Lemmy.world Support or email info@lemmy.world. For other questions check our partnered communities list, or use the search function.


6) No US Politics.

Please don't post about current US Politics. If you need to do this, try !politicaldiscussion@lemmy.world or !uspolitics@lemmy.world


7) No Hit-and-Run questions.

Please don't delete your post for no apparent reason. If you plan on deleting a question later, say so in the post, or if you feel that you have a good reason to remove it, message a mod beforehand. It's not fair to the ones who took their time to answer, and it's not in the spirit of the community.


8) No Bots.

Posts or comments from bots, LLM's, AIs, Neural Networks, Transformers, or Marvin the Paranoid Android are not welcome in AskLemmy. Real humans only please.


Reminder: The terms of service apply here too.

Partnered Communities:

Tech Support

No Stupid Questions

You Should Know

Reddit

Jokes

Ask Ouija


Logo design credit goes to: tubbadu


go to feed...