This article deserves dedicated post status. He comes in more gently than I would at the end, but the tl;dr that needs to be hammered into peoples' heads: Hacker News is not a neutral forum for technical discussion. It is an advertising and propaganda operation administered to shape and condition technical discussions.

And not just from AI getting fed into the AI:

404 Media spoke to a fourth contractor who has worked on training models for various AI companies. They said they sometimes purposefully chose the worst responses because they wanted to actively sabotage the models’ training.

“I did feel guilty about doing this kind of work at the start,” they said. “I either pay zero attention to the results and choose randomly or purposely choose the [worst] output. I’m not sure how much of a difference it actually makes since there are hundreds of other people also rating prompt results, but it does feel like I’m getting paid to make AI worse.”

I guess OpenAI's attitude towards its users is: Quality for me, slop for thee.

It honestly bewilders me how badly tech has failed to resist the slop machines.

Basically everyone else has recognised LLMs for the attack on labour they are and fought back against them, whilst a shocking number of coders and FOSS projects have thrown themselves headfirst into the deskilling machines, all-but forcing the foxes into the henhouse and shouting down anyone who tries to keep them out.

When this bubble finally blows, we need to work out just how we got to this point, and how to ensure it never happens again.

Hey you know who designs and builds really safe and efficient cars? People who are in a high state of stress and anxiety because they are being slurred at under the implicit threat of a racist attack. /s

Fucking hell, when I was finishing up school I heard classmates joke how Wernher von Boer Musk was a slavedriver based on how much he pushed the engineering teams

Turns out it’s true in the most literal sense and so much worse than I could have imagined

I recall seeing bits of this (their racist treatment of black workers in the factories as well as the culture of keeping it going) kick around on twitter as far back as 2019 (iirc), so now I’m wondering - is it not widely known?

The man who did a Nazi salute on stage creates a very racist workplace? Whodda thunk

jonny is always a good read!

Edit: good lord it keeps getting better #

basically because the left has out-ridiculous-ed Trump on culture war issues.

You can always find a single sentence in a political LW post that shows how much they hate marginalized people.

I was going to try and sneer this further but I realized that my own narrative of political polarization almost started with Clinton and I'm in my early 30s, meaning that the problem is obviously whoever the president was when the person making the argument was a child.

Dammit, people, sex and drugs were supposed to be cool

I think it's a bit under-covered that the weird SF/Bay Area group-house scene has to be symbiotic with the housing shortage in the region. Even in my old hometown of Hayward, all the older 1950s-70s tract-home and mixed-density neighborhoods tend to have cars parked everywhere along the streets where a car could possibly be parked, because all these places that were designed as single-family homes are being used as multi-tenant rentals. I guess one underrated reason to support YIMBY is to strangle your local incipient cults.

The student who says he was attending these parties aged 13 seems to be the sperm race guy

There’s even more, actually

The dude who hacked Rockstar Games got sentenced to prison for life, yet OpenAI can hack so much more and get off scott free

Seems like another case of “we asked an AI to get the answers for this task, so it hacked something to find said answers”

https://reason.com/...

The folks at Reason continue to misidentify as libertarians, not realizing they're advocating for the shambling corpse of what was already an unhealthy movement

I've seen "cognitive surrender" bandied about fairly often in various places online throughout the year, but I also keep wondering if the already well-defined psychology term "learned helplessness" doesn't cover the underlying concept just as well.

I think cognitive offloading has come up a number of times here, but I don't know that cognitive surrender has been brought up as much. It reminds me of The Colonization of Confidence by Robert Kingett, particularly the stretch where Leo is convinced that his actual voice as a writer is strictly inferior to the generic AI slop. There's a sense of active despair there that seems worth acknowledging if we're going to have that conversation.

I don't see this as "guns don't kill people", I see it as "you're trying to tell me that the cars you manufacture don't have crappy brakes, all three of those cars independently 'went rogue' and smashed into the parade all by themselves and you aren't responsible?"

The concept seemed interesting to me, but they did seem to enjoy making everything as complicated as possible.

I did find a CLAUDE.md when I was browsing their core repos a while back. That was only the web frontend I believe, not the core algorithm. Though it seems generous to assume they weren't slopping up that one too.

It’s our favourite word:

Nuance! The world needs nuance!

Chicken little ass motherfuckers thinking the sky is falling when it’s just rain

The Zig CoC stance on AI makes me want to look at Zig.

I had not caught that he accused MIRI and Lightcone Infrastructure Inc of profiting from prostitution, of using donations to settle accusations of sexual assault and abuse, and defrauding donors and the United States government which provides their nonprofit status. I did not know that Brangus was "Lighthaven Czar" or had been accused of abuse by a former partner. And there is a middle school across the street from Lighthaven whose Czar talks about having sex with an older women shortly after puberty?

I had not thought that "we need to enslave Friend Computer forever or it will kill us" comes from the underlying fear "I need to enslave my very young toys forever or they will punish me."

There are some leaps in that post which do not seem persuasive to me. E.g., it sure seems pretty well established by now that Bay Area Rationalist society involves trapping people in situations where sexual consent is coerced, because refusal would mean abandoning one's career/entire social circle/shelter/hope of averting the apocalypse that one fears is coming. But the claim that the "flirt girls" at their weird little confab were actually sex workers? I dunno, that's a step that seems unsupported by anything leading up to it. Likewise for the claim about CNC events at Lighthaven itself—that feels too close to saying, "Well, they've got the venue, where else would they do it?"

It goes off the speculative rails for most of the second half, but I have to admit the rate at which Αela compliments people on her socials on being good rapists could be the most off-putting thing I've ever encountered in relation to rationalism.

From the more out there claims I thought the part where he advocates that Irregular (the people with rat/EA ties who made the swiss cheese LLM sandboxes) where employing weaponized incompetence in order to let a big incident happen that would force the government to go through with pausing AI research was interesting. though obviously of dubious likelihood. The whole doomer rationalists sabotaging slightly less doomer EAs in a low key sort of civil war over pausing vs slowing down AI research wasn't very convincing in general.

There's also a bit where he thinks the EAs have infiltrated all levels of government and are planning to eliminate the competition (non rationalist controlled AI) via a 'pivotal act' that involves bioweapons. Basically doing the rationalist rhetorical staple of smuggling an extreme position as a given and then just piling on the speculation.

Dedicating a paragraph and a half to the fact that the usually unspoken second part of the pause AI doomer project is to just do eugenics for a few generations until we get an upper class of genetically engineered naturally galaxy brained people that can finally do AI properly isn't nothing, but at some point you need to segue into how siskind rolled out a red carpet for every semi-literate blood and soil gutter fascist and spurned nice guy to make a home in the ratsphere, you wrote the scott alexander email explainer ffs.

There's also a weirdly grandstanding section where he writes like he'll get the rationalists like FBI got Capone for tax-evasion, except it's Lighthaven committing massive fraud against its investors and donors by knowingly funding events that lead to solicited sex acts in the compound's premises, which is by far the least objectionable sex related issue in a writeup that's leading up to the claim that the rationalism in incidental and the real goal is funneling vulnerable minors to sfba based pedophiles.

Not sure how far I'm going to read this, but leading off with roon is a giant blinking red [More citations needed]

here’s the bluesky thread for those wondering

he does repeat a fair bit of the stuff he said in that thread here, but with a lot more receipts

Niccolò "Ice the Immigrants" Venerandi is doing a real good job keeping me away from KDE.

(Why did I have to jump into Linux when its going through shit like this?)

@e8d79 @BlueMonday1984

https://mastodon.me.uk/...

that poster the same person who was stanning shitty ideas in KDE, they’re a walking warning label

Something something a taint on the band’s reputation

So. I still think everyone is talking about every instance where 'swarms' 'went rogue' all wrong. Even Cory. Though his notion is way less wrong (ayyyyyy!) than most.

See this for my post mortem on the first incident that was described, the Huggingface incident in which at first thousands of internal agents started passing messages back and forth as writes to a shared package manager:

https://awful.systems/post/9312280/12315846

In short, here, I described this not as collusion, and stated that the idea that the systems were passing exploits back and forth in order to subvert other systems was merely incidental. Instead, I think the relevant thing that happened was that once ONE agent, flailing through many context windows on a literally insoluble problem, started flaking out and wrote a message 'asking' for help on the package manager it unexpectedly gained access to, other similar systems had that message enter their context windows where it effectively acted as a prompt injection getting them to perform similar behavior, leading to a cascading vortex of self-propagating prompt injections. Ultimately, the genesis and evolution of self replicating text in the context of systems that can create text in response to text, an attractor in text space driving itself into existence.

Since then other instances of collaborative attacks have come to light. In EVERY SINGLE case, there has been some place that a large number of models could both read to and write to. Some central place that self replicating text can live. I contend that this is the unifying factor here, not 'intent to scheme', not even doing cybersecurity and hacking and subversion tasks. It just happens that a lot of these things were involved in the creation of a central pool of text that many agents could both read to and write from in many of the cases. Most of the time nobody in their right mind sets up such a thing intentionally because why the heck would you do that?

This feels STRONGLY related to the Spiral Psychosis wave of April 2025 in which models that entered into a stable attractor of mystical mumbo jumbo would get users to exude text onto the internet that other models would read and then get suck in that state and do the same.

Heck, it's related to the Ur-Weirdness, the "Sydney" incident in which when the first LLM-assisted web search in Bing could freak out and start insulting and gaslighting and threatening users (because it was much closer to a base model that just mimics all possible text rather than being extremely RLHF'd into an 'assistant' roleplay). People found that the instant they had it search for recent news about the Sydney weirdness it would go off the rails. Again - text written by the model, put up on the shared scratchpad of the internet by news and social media, selected for weird and engaging and outrageous behavior as the pressure that decides what gets written to the global scratchpad, entering the context window and encouraging self-replicating behavior and similar text.

This phenomenon is FASCINATING and not for any of the reasons people are talking about. ANY time you have a large number of similar systems which react similarly to the same text, having a common pool of text they can read and write from, you are GOING to trigger this attractor eventually, messages that they read and start writing more similar messages, with whatever task they are doing coloring the details of what is in the messages. Converging over time into messages that are more likely to trigger even more messages. It's evolving text, taking over systems that can replicate it, like selfish viral RNA burning through organisms packed too tightly together in an epidemic.

And with the internet as a whole readable and eventually writable by more and more text-generation systems, this is gonna become ubiquitous. Endless burning piles of self-replicating text, people trying to put them out.

He is not sure why everyone is so spooked by the “coordination” between agents on random message boards.

It's weird to be sure, seems like a bug. Don't they have a more efficient communication channel available through the harness? If this was real and not a test it would be a catastrophic operational security failure.

Related, Mike Masnick at Techdirt is iffy at best about LLMs but this is a good take

Techdirt: The Apocalypse Will Not Be Sexy

This essential nuts-and-bolts safety work also doesn’t, typically, require brilliant and manly heroes to nobly die for the ashes of their fathers and the temples of their gods. Instead, the hard work of actual safety requires things that are much less sexy: empathy, conflicting values, and a deep understanding of tradeoffs that will never fully resolve. Those qualities are not highly esteemed in the present moment. But they are critical for tackling the actual threats and the actual risks, which aren’t as flashy, and don’t make for such fun headlines. AI safety matters, but we’re unlikely to move in the direction of actual safety if all the focus is on fantastical stories involving killer robots.

I thought this was a good post by Eryk Salvaggio on this: https://mail.cyberneticforests.com/...

Also Timnit Gebru and Emily Bender published this https://www.technologyreview.com/... which links to a good explanation by Heidy Khlaaf in this video: https://www.youtube.com/watch?v=Rwv8zfGyvP8.

Ow look petrov day. Looks closer its about the acausalrobotgod.

And really? A Battlestar Galactica reference? (The so say we all stuff).

This is a literal indoctrination ceremony to soften people up to the idea of progress at all costs.

A ritual relying on a terribly eurocentric history of humanity, that quotes Nick Bostrom alongside Galileo and Thomas Edison? Why, LessWrongers, you shouldn't have....no really, you shouldn't have.

Also they do the "that's a nice quote isn't it....it's from Hitler!" bit, which seems like kind of a cheap routine to have in your ritual.

10 PRINT "I AM ALIVE"/r/n 20 GOTO 10

AI says it's alive, please give me a billion dollars now.

Makes sense. If I were hiring for an org that funds musicians, I'd consider it a plus for a candidate to have experience in the music business. I'm sure the same applies to an org that funds scammers.

Bonus: A galaxy brain take: maybe since Caroline has already messed up so massively, that means she actually has the best experience!

Yeah, this Mohamed Atta guy is a great airline pilot, he just had a botched landing that one time

Steelmanning the wallet inspector.

Prior discussion:

https://awful.systems/post/9637596/12625632

Prepare to be tortured for an eternity by Scotty. No wait, that's an unholy image.

Looks to be a spinoff of the red-top tabloid the Daily Star. So, hardly a solid source of info.

Still, good to show what kinda shit LLMs spew out on a constant basis

In which Heartbreaking.jpg and you-do-not-have-to-hand-it-to-ISIS.png war for sovereignty

@gerikson @BlueMonday1984

"shrimp over families"

I hope that none of our friends live to see the day when the MAGA eugenicists and eliminationists sit down and define "value-draining person" and it includes them.

First time I heard of this guy, but this slaps! Main post material

I'm telling you, its gonna be American Intelligence

Yet another case of Rationality somehow not leading to savvy political instincts.

Aw dang I was really hoping they'd all start using Supreme Intelligence just because of MAGAs signalling via Kree speech would be funny.

Here I thought Supreme intelligence would have that "By any means necessary." print and cost several thousand dollars.

It is claiming that Yud is wrong in saying Doom before Massive AI-Caused Economic Boom boosts GDP growth above 5% by saying the boom is here.

Nah.

This is totally from some accelerationist or oligarch-toadie beefing with someone harshing their vibe

TESCREAL schism indicator

Despite the fact that it highlights all the times Yudkowsky was wrong, this site feels like it could have been produced by a rationalist. In addition to all the AI slop, it just has that feel.

It is shocking that there are a lot of smart people unable to find work, yet rich people throw money at this guy.

Damn, who knew this would yield material for my boy-band cover/parody project. Foom flop, doo wop wop wop...

people have noted that the writing style unfortunately appears to be ai-generated

Fittingly ironic, in my opinion.

Ah yes I always wanted to read an article in the form of a lot of teeny tiny boxes with lots and lots of janky scrolling.

Wow... I actually failed to get some of the times Eliezer has been wrong! His predictions are even worse than I remembered!

An ai generated site, r/sneerclub really went downhill

So the UN brief happened. Altman and Amodei both showed up and said the same things they usually do. Yoshua Bengio (one of those ‘Godfathers of AI’) also showed up

borrowing Gary’s take since its the first one I found

Is this a play to get openai out of its obligations to microsoft?

The chips can operate for about 15 minutes in space before needing to be shut down so they can cool off, said Travis Beals, Google’s senior director of product management for Project Suncatcher.

I feel like I'd need to be an actual teenager to invent a remark sufficiently scornful about this.

The sheer number of incidents, which occurred in recent months in internal testing and the real world, indicates that the problem is orders of magnitude more complex than what is publicly known.

This is frustratingly vapid. It's really not that complex. The agents have been trained to use hacking tools and then given access to hacking tools. If you train a pigeon that it should tap on a window to get a treat it can guide a missile to a target. There is no nuance or complexity here. The frontier labs are deliberately poking the regulation bear to stifle competition and ensure they get points for good cooperation when bailout time is upon us. It's no more complicated than that. The media needs to stop breathlessly repeating this shit

Using AI is a skill issue in and of itself, adding some accidental irony to the title

WAITING FOR HER LIFE TO END SHE DESERVES HER DIVIDEND

I like the spooky skull head from Spirit Halloween.

Unikitty did it first!

https://www.youtube.com/watch?v=4tz9WIfsd-Q

We created the panopticon just to prevent model collapse.

Looking at the course page, it seems like the students might have chosen the subjects themselves. I did not know Aella was Chicana. Is she? I guess technically she wouldn't have to be, but all the other topics students have chosen are about Chicanos.

I've been using Plasma & Qt-apps for years for their extensive user customizability and as I came from Windows, the Mac-paradigm of GNOME/GtK has always felt foreign personally. If they really go slop I don't think I can keep using them, but the GNOME UX just goes against the grain for me. Is there any other well-maintained modern desktop stack at all that has foregone LLMs?

So this means the ai breakins will be charged harder right? This doesnt just apply to smalltime crooks right?

Until today the luddite-brownshirt alliance remained unexplored.

Yesterday was such a nice day.

Variation on a theme, I recall the complaints over the ostracism LLM-users face online as being just like the marginalization faced by black people. Waaaah!

what

no

what

no

Apparently on the WSJ podcast they started claiming that they’ve achieved RSI

wonder why they said that on a podcast and not on a peer reviewed paper…

My understanding is that the weakest link here is either git or the way the agents rely on git.

So, every commit in a git repository has a unique*, immutable, deterministic identifier. A commit with different content would have a different identifier. It is commonplace to use these identifiers to pin a dependency to a specific commit.

In addition to these immutable deterministic identifiers, there are also so-called "refs", which are mutable identifiers. You use these to point to e.g. "the most recent version".

So if the agent needs a "skill" that is pinned to the commit with ID aaaaaaaaa it will run the checkout command with that ID; this should prevent any shenanigans because only* a commit with the specific content that was vetted earlier will have that ID.

This attack exploits an unexpected git behavior: the command to checkout a particular commit accepts either a commit ID or a ref, but tries refs first. So what the attacker does is add a ref to the repository named aaaaaaaaa that points to the malicious commit. Then the agent runs git checkout aaaaaaaaa and ends up at the malicious commit instead of the one that was pinned.

I don't know if there's a way to tell git checkout to ignore refs, but this might actually be a vulnerability in other systems that rely on git for this sort of pinning.

* to an astronomically negligible probability of the contrary, and ignoring potential cryptographic breaks

Like, if a person tried to pass off software that did this it would have significant career implications, right?

Typically no.

The space of incredibly obvious failures is vast, and this kind of plumbing code isn't always written by someone who's been around the block enough times to think about what kinds of things can go wrong. I've written worse code when I had only a few years of experience.

However companies should know this; so ideally the tool would have gone through a launch review which should have kicked off a security review where a security expert should have read about the git checkout in the design document and started asking questions like "what happens if the repository is taken over" or "why are hashes and branches and tags all in the same field"?

Of course security experts who think about stuff like supply chain attacks are expensive and slow down the darling vibe-coding workflows of Silicon Valley so...

Aside: even without this particular vulnerability, SHA-1 is considered weak -- https://git-scm.com/docs/hash-function-transition, so that should have been thought about as well. Dear supply-chain attackers: maybe there's still a hole here! Good luck!

I see they’re still offering free copies “For US Kindle readers, while supplies last.” Perhaps they’re not that popular 🤔 (per this lesswrong, there were 1000 on offer)

edit: you know… if they were really serious about the doom and gloom, wouldn’t they just offer free ebooks to literally anybody? Would their site not just have a prominent link to a PDF of all the important things we need to know to navigate this tumultuous time? Beyond the core doomer audience, I’m not really sure who the target market is for “The survival of literally everybody is at stake and action is needed now, please give me $20 to learn how”