That takes me back to 2004

No Smoke, by unknown,

Tech: "Hello. How can I help you today?"

Customer: "There's smoke coming from the power supply on my computer."

Tech: "Sounds like you need a new power supply."

Customer: "No, I don't! I just need to change the startup files."

Tech: "Sir, what you describe is a faulty power supply. You need to replace it."

Customer: "No way! Someone told me that I just had to change the system startup files to fix the problem! All I need is for you to tell me the right command."

(Ten minutes later...)

Tech: "Well, we don’t normally tell our customers this, but there's an undocumented command that will fix the problem. Add the line "LOAD NOSMOKE.COM" at the end of the CONFIG.SYS file and everything should work fine."

(Five minutes later...)

Customer: "It didn’t work. The power supply is still smoking."

Tech: "Well, what version of Windows are you using?"

Customer: "Windows 98."

Tech: "Well, that's your problem. That version of Windows doesn't include NOSMOKE. You'll need to contact Microsoft and ask them for a patch."

(When nearly an hour had passed, the phone rang again...)

Customer: "I need a new power supply."

Tech: "How did you come to that conclusion?"

Customer: "Well, I called Microsoft and told the technician what you said, and he started asking me questions about the make of the power supply."

Tech: "What did he tell you?"

Customer: "He said my power supply is not compatible with NOSMOKE."

Lol. "do magic"

"DON'T HALLUCINATE!" that's always my favourite.

"or you go to jail"

That's hilarious, idk i think llm could be useful for helping product folks translate their thoughts into actionable items for the devs, but yeah like beyond insane to tell the product people to hop on claude and do it themselves. That's like a construction company letting the sales team jump in an excavator and start digging!!

Creating an environment that incentivizes not thinking is peak stupid.

Having production credentials in a dev environment is more stupider but they'll never learn because they outsourced their thinking.

Claude code even added an auto mode so that you don't get blocked by that pesky reviewing anymore. Since then, the usual mode of asking before running a command, for instance when the thing wants to read the entire codebase looking for information only available in an online doc, is now called manual mode; the non 10x developer mode.

What's the point of AI if you need qualified review?

The bad news for them is that AI is terrible at getting it 'right' the first time, so they have to do things over and over again guessing until it works. So the AI-heavy user pretty much has to just let the commands run to get any benefit since otherwise it's spending 95% of its time waiting on user to approve commands. Further, it's likely commands the user doesn't understand, because they are asking AI to do it in the first place frequently because they don't know themselves.

I'm with you, but as a result other people proclaim to be 'better' AI users largely because they trust the AI. Their stuff is crap and once in a while one of them blows themselves up, but in the short term they are getting praise from management.

Man I can't wait for the bubble to pop and management no longer being hyped for the sake of hype over it.

Seriously, has no one heard of sandboxing?

Having working production database config and credentials in your local .env, as appears to be the case here, is equally wild, and basically begging for something like this to happen.

Prod should always be highly "air gapped" with some sort of deployment process which tests not only the code to be deployed but also the deployment itself. I've been doing QA for a good while now, and everywhere I've worked has testers dedicated to testing the actual update process to make sure it will be safe when deployed.

That's incredible lmao

They have no idea what else to do. They were in over their head so long before this problem happened.

What evaluation setup lets the thing being tested see the test results and gives access to the test source code?!

Slop evaluating slop.

No, No events like this are a good thing! It shows to the C-suite that AI isn't a thing to be trusted in your company. We need more failures like this.

"Make no mistakes"

As if the LLM can truly understand what a mistake is...

Because they don't know what they're doing. A tool is only as useful as the person verifying its output. Vibe coders have dumb shit like this happen to them all the time because they don't actually possess the skill set to perform the task correctly, with or without a bot that writes the actual code for them.

Dev environments cost money. They didn't fire all those programmers and replace them with AI to spend money on useless backups and safety systems.

/s (because yes, it's necessary)

I remember a case where the AI was not given credentials to the prod DB, was not instructed to do anything on the prod DB, but went through the operator's hard-drive, parsing docs until it found them, then proceeded to destroy the prod DB.

Of course it was sorry, as they always are (deeply, trust them, but no: no refund, loser!).

Well, sure, but even if a developer correctly tests on a development environment, they still probably have prod settings laying around on their filesystem for the times where they need to put out fires in prod. That's kinda what it looks like what happened here - the LLM found a .env file with prod settings, and used that config to run destructive tests. All the more reason to not give an LLM side wide reaching access to your computer, but it's not necessarily an indication that someone regularly tests in prod.

Literally me, this weekend. Well not this bad but I still think the AI wouldn’t have done me as dirty as I did myself.

I am counterintelligent, I'll have you know. I stare at goats and doesn't afraid of anything, except how sexually attractive I found the judge who condemned me to jerking off for twenty-six sessions of anger management because my narcissistic manipulator life partner entrapped me, being my superior officer. She (the judge) had wonderfully beautiful hands that I wish could caress me while degrading me. Good thing I work for the F-I mean, the CIA, and this is just me establishing a cover story.

In three to seven years, there will be a News headline: "Hero." It will have my face as the picture. The article will talk about how I infiltrated an Epstein-type ring. How? That's classified, but damn am I audacious, and I do hope you excavate (dig) to understand the depth to why this is so damn funny to me; Q.

Ignorance is bliss.

Given corporates layoff engineers at dramatic rates, in a few years, we'll start to see services collapsing with no one left who can recover them.

This. There should have been processes in place to prevent this. But I can’t say I haven’t gone through it myself; not AI, but my own advanced imbecility.

They're about to run out of those experienced engineers in well under a decade as long as they keep strangling the entry level workforce.

Forcing seniors to work is also a recipe for disaster as well.

Or, easier option, DONT FUCKING USE THEM TO BEGIN WITH!

It puts the lotion in the basket

just doesn't roll from my tongue that easily like "you"

How do you achieve that? Calling it out when it happens?

I don't like LLMs outside of debugging and the way it mimics human closeness creeps me out

What should I call it then?

Being part of a multi million dollars company is not making anyone smarter

I think Claude would scream in agony if you subjected it to whatever the fuck this is.

Well that is perfectly normal. But using it for work is madness.

Better than hard coded... Ideally we use tokens+fingerprint or something to avoid storing the creds directly (if possible), but putting them in environment variables is pretty common

It's not the worst thing, it's very convenient (so people won't go around it) and usually not the weak point in security (although AI being able to easily see it is an interesting twist)

None of that helps if you grant the LLM access to the production database and it decides to drop table user; to free up some space. That’s when you find out if your database recovery process actually works. You do have database backups, right? Or is that not a thing when vibe coding?

https://casp.ac/...

We saw in a movie how motorcycles can jump over bridges. We used AI to learn how to do this. We gave it information, like what motorcycles we use and the distance we need to jump and so on and it gave us steps on what we have to do. We practiced a lot and kept asking questions. We dug holes and filled them with broken glass and fire to practice. 18 of us died in the process. Eight of us managed to do it. The next time we attacked, we could jump.

Grain of salt: these people probably lied to the researchers but it is still hilarious

Hopefully they're not storing their production database in git lol. They should be taking backups of the database, but unless they're making one after every transaction, permanent data loss is a real possibility.