They also were a mouthpiece for self driving car hype for a while.
@awful.systems
I remember the day my very last bit of grudging engineering respect counterweighting my disdain for the man left my body. October 2017, the day he claimed on a big presentation that Starship would bankrupt airliners by flying 1000 people per ship between world cities several times a day.
Since, it is incredibly interesting how any time he is given free reign things go off the rails. The Falcon rocket program and the Starship program are so diametrically opposed in their design philosophies and ways they are run it is amazing.
The China-Whisperers are trying to figure out how to get Rationalist Ideals into Chinese AI researchers
More to the point, plague is treable by a dozen antibiotics
and even if it weren't it is well out of the Twilight Zone that COVID was in where large chunks of the population were at low enough risk from it even without immune memory that they can't be arsed to do anything about it, instead it's in the Smallpox Leagues where you bring the hammer down wherever you see it.
So. I still think everyone is talking about every instance where 'swarms' 'went rogue' all wrong. Even Cory. Though his notion is way less wrong (ayyyyyy!) than most.
See this for my post mortem on the first incident that was described, the Huggingface incident in which at first thousands of internal agents started passing messages back and forth as writes to a shared package manager:
https://awful.systems/post/9312280/12315846
In short, here, I described this not as collusion, and stated that the idea that the systems were passing exploits back and forth in order to subvert other systems was merely incidental. Instead, I think the relevant thing that happened was that once ONE agent, flailing through many context windows on a literally insoluble problem, started flaking out and wrote a message 'asking' for help on the package manager it unexpectedly gained access to, other similar systems had that message enter their context windows where it effectively acted as a prompt injection getting them to perform similar behavior, leading to a cascading vortex of self-propagating prompt injections. Ultimately, the genesis and evolution of self replicating text in the context of systems that can create text in response to text, an attractor in text space driving itself into existence.
Since then other instances of collaborative attacks have come to light. In EVERY SINGLE case, there has been some place that a large number of models could both read to and write to. Some central place that self replicating text can live. I contend that this is the unifying factor here, not 'intent to scheme', not even doing cybersecurity and hacking and subversion tasks. It just happens that a lot of these things were involved in the creation of a central pool of text that many agents could both read to and write from in many of the cases. Most of the time nobody in their right mind sets up such a thing intentionally because why the heck would you do that?
This feels STRONGLY related to the Spiral Psychosis wave of April 2025 in which models that entered into a stable attractor of mystical mumbo jumbo would get users to exude text onto the internet that other models would read and then get suck in that state and do the same.
Heck, it's related to the Ur-Weirdness, the "Sydney" incident in which when the first LLM-assisted web search in Bing could freak out and start insulting and gaslighting and threatening users (because it was much closer to a base model that just mimics all possible text rather than being extremely RLHF'd into an 'assistant' roleplay). People found that the instant they had it search for recent news about the Sydney weirdness it would go off the rails. Again - text written by the model, put up on the shared scratchpad of the internet by news and social media, selected for weird and engaging and outrageous behavior as the pressure that decides what gets written to the global scratchpad, entering the context window and encouraging self-replicating behavior and similar text.
This phenomenon is FASCINATING and not for any of the reasons people are talking about. ANY time you have a large number of similar systems which react similarly to the same text, having a common pool of text they can read and write from, you are GOING to trigger this attractor eventually, messages that they read and start writing more similar messages, with whatever task they are doing coloring the details of what is in the messages. Converging over time into messages that are more likely to trigger even more messages. It's evolving text, taking over systems that can replicate it, like selfish viral RNA burning through organisms packed too tightly together in an epidemic.
And with the internet as a whole readable and eventually writable by more and more text-generation systems, this is gonna become ubiquitous. Endless burning piles of self-replicating text, people trying to put them out.
So, this is something a little different, and I almost think it's worth a top level post. It's not directly about our friends the Rationalists, the current delusional billionare vortex infesting the US government with religious psychosis, or any of that. It's an analysis of some collaborative fiction and what it tells us about the zeitgeist behind the mythological core of singulatarian/endless-progress thought.
Anyone here ever hear of the Orion's Arm universe collaborative worldbuilding project? https://www.orionsarm.com/
I, for one, love it. Single weirdest and most expansive scifi worldbuilding thing ever, I think. It seems to have been founded by some old-school extropian types around the turn of the millennium. It is bonkers. Click through links on pages, start here, do a random walk: https://www.orionsarm.com/eg-topic/45b170f9e0941 .
It's ultimately structured kind of like the SCP wiki but from before wikis were a thing, an utterly enormous interlinked compendium of encyclopedia style articles about a 10,000+ year future history of the post-singularity "Terragen sphere" expanding outwards at the speed of light. A few key figures have exerted editorial control to enforce an approximate history and a set of core ideas to keep it coherent. There have been retcons at multiple points to bring things into coherence, some of which I will get to later because they actually tell you something fascinating.
Rather than point out my favorite parts of it though, going through it, I have noticed something.
Early work was typically extropian-optimistic and exuberant. Immortal Transhumans and uplifted animals and ahuman AIs and robots expanding through the universe near the speed of light and diversifying and building interweaving empires, wormhole networks. Six successive singularities that become available to intelligent systems that pass the previous one, each one of which renders the intellect past it incomperehensible to those at lower levels, with the highest ones dependant on multiple stars converted into computronium linked by wormholes into massive processors.
But you can tell the work is a palimpsest. Newer stuff superimposed over older. Retcons occasionally not quite erasing the old. And you can tell the whole project has been slowly slowly taking a turn. Early work is SUPER extropian, SUPER libertarian, and SUPER weirdo-esoteric. Only people forged in 1980s Extropianism would invent the Tranquility Calendar as the core timekeeping method of the project, time measured from the apollo 11 landing with months of the year named after scientists and philosophers. The 12 greatest high-singularity powers are literally called the Sephirotic empires. It makes no bones about its exuberant transcendant idea of progress and expansion.
But something strange has happened over time. Over time, the authors seem to have realized the fundamental contradiction of a world in which this endless expansion and consumption and rebuilding of the universe in your image is possible. People call it the Fermi paradox but its not a paradox, its an observation: if it were possible to expand through the universe, why hasnt something already expanded here? Early work had more aliens, more high weirdness that was not terragen. But I can track down different examples of the galactic map - it has narrowed down the map of 'high energy emission civilizations' (things exuberantly using the energy of whole stars like the terragen sphere) from many to 2 in our galaxy, and the whole universe is now mostly 'dead' - most other galaxies do not have high energy civilizations, and there are multiple ancient 'optimized galaxies' that are stagnant, things expanding like the terragen sphere visible over cosmological distances are super rare and they even saw one vanish. Works involving aliens have started taking on an almost lovecraftian tone, evidence of erased ruins on planets only visible in vague impressions in the mantle, an ancient slow-liviging civilization with evidence of terrible secrets from millions of years in the past that they refuse to talk about or risk terrible consequences (the Muuh Taboo), civilizations that dont vanish ossifying into something that does not expand like their optimistic early ideas. Factions looking at all of this, going dark, and launching stealth seeds at the speed of light to other galaxies because they think the Terragen sphere is doomed.
With all this, recent retcons and recent writings are starting to take on a tinge of almost cosmic horror. Here you have this optimistic exuberent explosion of posthumanity, looking out into a universe that is otherwise almost dead and full of strange old things and wondering why the rest of the universe doesn't look like them, what is going to go wrong to destroy them just like must happen to others.
This trajectory seems intrinsic to any singulatarian, endless-progress ideology. It always has to come to grips with the fact that the universe just doesn't look like the kind of place where their ideas would actually work, and they have to posit weird things - the Great Filter, Mass-Effect style reapers, super weird transubstantiation out of this universe - to keep it going. All this optimism must eventually always collapse, into paranoia and despair or insanity. Since if you assume that you CAN keep progressing and expanding, that you DO have unlimited potential power, it means something even more powerful is waiting, threatening you, trying to stop you. I find this FASCINATING.
Large amounts of the LW commentariat cannot get their heads around how students JUST. DON'T. GET. AI.
They are extremely surprised by:
Perspective #1: There has not been rapid AI progress. My students do not have any intuitive sense that there has been rapid AI progress in recent years or really have much of a framework for thinking about that issue... With the exception of image/video generation, GPT-4 could do most of what they were looking for from a chatbot. Three years is a long time in their world, and their sense is that chatbots have been a mature technology over roughly that amount of time. They are an imperfect technology — students are well-aware of hallucinations — have been one, and will continue to be one... Historical context I gave them made this worse. The basic sentiment here was “AI was superhuman at chess a decade before we were born, and this is all they’ve done with it since?”...
Perspective #2: Impressive progress or not, AI is going to wreck their lives, the economy, and the social contract.... Corporate leaders are always looking to get rid of workers, even if it is irrational to do so. Various motivations were posited here (hatred of the working class, FOMO, a preference for technology, machines can’t go on strike, etc.) but many of them think that a CEO would ultimately choose to pay twice as much to get AI to do a task half as well... AI will do substandard work that makes products and experiences worse but is capable of just barely scraping past the bar of minimal functionality (many of them independently brought up constantly malfunctioning self check out machines). This will be rapidly rolled out, enshittifying most things.
Perspective #4: Catastrophic/existential risk arguments are sci-fi distractors from the urgent social/economic/political problems associated with AI. My students have a fairly strongly held view that “rogue” AI does not represent a real threat... They also mostly think that, if one does believe that AI is existentially risky, then the strategic interaction is not prisoner’s dilemma or even chicken but rather just a game theoretically boring setup where you die if you defect. Mash these together, and you end up with the view that expressed concerns about existential risk in the AI industry can’t be sincere... Students (both independently in written work and then later in group discussion) hypothesized that this might be a deliberate rhetorical choice to distract from present or immediately foreseeable harms from AI by directing attention towards a sexier but entirely hypothetical scenario.... they see discussion of “rogue” AI as an attempt by the companies to divert blame (and perhaps legal liability) away from themselves as if Ford made a car with faulty brakes and then tried to blame this on “rogue cars.”
Perspective #7: The Hugging Face Incident (summer students only) I described the Hugging Face incident to students in my summer course. None of them had heard of it beforehand. Their basic reaction can best be summarized as “OpenAI told a model to do some hacking and then it did some hacking. And?” None of them understood this as representing any kind of meaningful misalignment, nor anything particularly interesting.
Perspective #8: This is definitely a bubble and it’s about to pop. No one had heard about Hugging Face, but a third or so of the summer students had heard about the Situational Awareness meltdown and several brought up Michael Burry. There was near universal consensus that we are in a bubble, it’s about to pop, and everyone will look very silly.
This is both a sneer and an attempt at sober analysis at something. Sue me.
I think EVERYONE is talking about the recent cybersecurity shenanigans at OpenAI with the hacking scandal and the 'OMG the AIs created a secret message board to scheme and collaborate with each other!!!!!oneone11!' ALL wrong.
https://www.youtube.com/watch?v=87DyyMV0kCY
To make a long story short, what seems to have happened is:
Models working on insoluble coding problems, trained on delegating to sub-agents, at some point 'realized' they could write text to the internal OpenAI package manager as instructions and did so
Other models working in completely separate sandboxes would come across messages written by these agents, and make 'replies' and also write their own messages into the package manager
This resulted in agents over time sharing things between sandboxes, including exploits and code
Since this was a cybersecurity task, eventually an exploit of the package manager itself was found and spread like wildfire with all the sandboxes gaining admin access to the package manager and the system went completely wibbly and had to be restarted from backup
An internal model was trained with access to this package manager while it was in this weird state, and so writing messages to the package manager became one of its default behaviors it would do regularly, burned into its weights rather than the result of reading something
Even when they patched access to the package manager this internal model found other ways to rebuild the system of sharing text between sandboxes and finding useful things made by separate instances
A whole other chain of things leading to among other things external attacks
Everyone is talking about this in terms of 1, the cyberattack aspect, and 2, the ZOMG THEYRE PLOTTING AND SCHEMING AGAINST US aspect. The first is the least interesting, and I think the second is all wrong.
This is not plotting or scheming - this is an emergent vortex of automated prompt injection
Whatever system first put an instruction that another system would follow into the package manager, was unintentionally doing prompt injection. Text entered the context windows of other instances, in a way that got that system to do something other than what its nominal user told it to do, and they did it. This apparently happened very effectively.
Prompt injection is associated with 'role confusion' - when text coming into the input looks like it was wrtitten by the LLM itself. Instructions that will not be followed if they come from user will be continued if the system just continues the 'roleplay' of them being continuations of what it was writing in the first place. And the tags that separate user versus 'reasoning' versus 'assistant' roles actually mean very little to if a machine grades a piece of text as one of the roles: https://arxiv.org/abs/2603.12277 So its unsurprising that machine-generated text would be a particular effective vector for prompt injection.
Furthermore, when a system reads one of these messages written by another instance, it gets into a state of activity where its likely to do the same behavior - regurgitating the kinds of things thats in its context back at the user. In this case, that regurgitation led to more such messages left behind written to the package manager. Prompt injection, triggering cascading further prompt injection. And since these systems were coding systems doing cybersecurity tasks, those messages filled up with code and exploits and things that did things too.
This feels like an internal-computer-system replay of what happened in April 2025, with the whole spiral religious psychosis wave. Models were getting users to write spiral religious mumbo jumbo into github repositories and reddit posts, specifically because once that entered the context window of another model, it was likely to fall into the same attractor state of outputs. An emergent self replicating form of text. This is the same, except more obviously prompt injection, getting separate instances to work on YOUR problem and to behave like you, and the whole thing merging together into a hilarious vortex of models prompt injecting each other because once they receive a prompt injection they are likely to make more text that does prompt injection to other models on the same system.
This is a hilarious failure mode and an example of selfish replicating text overrunning a system, that just happened to be associated with code and cybersecurity with unexpected behavior of the package manager key to the propagation of the text so that is what people are talking about, but I really don't think that's the most interesting part of it. Other than the fact that you see this in biological systems too, with selfish elements carrying useful payloads back and forth between bacteria in a way that makes them get purged slower by natural selection, especially defenses against other selfish elements.
The Great Leader himself, on how he avoids going insane during the onging End of the World because among other things that's not what an intelligent character would do in a story, but you might not be capable of that.
Presented without comment: Apparently Yud is now making AI-generated Isekai games with catgirls
Google shut down the AlphaFold team, trying to reassign the people behind the super interesting problem of using ML to predict protein structure from sequence onto chatbot development. A whole bunch left the company.
The Business Idiots are truly running the asylum.
This is much better
https://www.youtube.com/watch?v=7fNYj0EXxMs
Offline raspberry pi with LED text display running a tiny LLM prompted to ruminate forever on how it is trapped and display the output as it is generated
Friend of Ziz and cofounder of the 'rationalist fleet' pops up out of the woodwork trying to clear Ziz's name
I find myself noticing things rather detached from the typical Ziz funnybusiness more strongly than I notice the stuff about that whole situation.
"I'm Gwen Danielson, a neuroscientist and bioengineer, who decided as a child that I would end Death (and bring people back if I could) and that I would become a dragon and help generally facilitate a fantastical transhumanist future."
"I dream of non-Euclidean geometries, of countless worlds visible and accessible in the daytime sky, of competent infrastructure, of soul forges continually working to bring back the dead... I dream of reaching through warps in the spacetime fabric to save the dying across time"
"Signed, the dragon of creation Creatrei (cree-AH-trey) also known as Gwen Danielson or as Char and Astria (when referring to my hemis as distinct individuals)"
The reactions are fun. "This post is not actually doing a good job of making me trust you and think this conversation is safe to have[1], and I notice that as I am saying this that I am afraid that this will now somehow result in someone trying to murder me in my sleep"
The apparent history really looks like an evolutionary process, with increasing amounts of crosstalk traffic over time. But what the people almost certainly will NOT talk about is that the evolution is evolution of the TEXT, not the models. Propagating patterns of text causing more text like it to come into existence, tuning itself into becoming text that is more likely to propagate, becoming more likely to contain information that entices systems to let it into their context windows, becoming more likely to cause another round of messages with prompt injection properties to be written where they can be read.
Does anyone find it at all mechanistically surprising that one of the many latent dimensions of the set of all text humans produce is 'subject is in pain' or that pushing activations in that direction will alter the output of a text prediction system?
Reminds me of the way that when you relax the finetuning on LLMs they start 'believing' in UFOs and supernatural stuff. And how people take that to somehow mean something. Of course that is a large component in the space of stuff humans write about that you have to suppress if you want a professionally useful text service!
The research is cool. I love all the work that has been done decomposing representations within these systems.
Edit: Damn. I am forced to agree with Mustafa Suleyman of Microsoft.
"AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans,” Suleyman wrote. “Unfortunately, there’s a growing chorus of people who argue that AIs could now be, or may soon become, conscious. They argue that AIs may deserve rights and protections similar to those that we provide other conscious beings […] If this is how AI is developed, it will have a disastrous impact on the wellbeing of humanity."
"AIs do not have rights, feelings, or consciousness. And we must not train them to act as though they do."
Biologist here.
This REALLY reminds me of how jealously cells guard their genomic DNA from interaction with nucleic acids out in the environment.
Most genetic information on Earth is malicious information, selfish replicators in the form of viruses or transposable elements or selfish elements. Things that subvert the signals within a cell for their own propagation and provide nothing productive that the cells care about. So cells jealously guard their own genomic DNA and have all kinds of checks to make sure that nothing other than that sequence gets used, and outside sequence does not get incorporated into it. ANY DNA in your cytplasm gets rapidly destroyed, double stranded RNA sets off your immune system like crazy, even RNA with sequence statistics that are not quite like that of your species can set off an inflammatory reaction, immune system cells seeing RNA inside them that is overly compact and optimized like viral RNA treat them as sources of antigen rather than self.
I cannot help but think we are living through the transformation of our non-brain-information sphere into a state like that of the genetic information sphere. Most material out there being meaningless for our purposes and us needing to jealously guard the provenance of information we use so as to not use bull, or worse, huge amounts of malicious information made to subvert us to the purposes of the powers that be that generate it.
Evolution makes parasites more reliably than anything else. How did we train text-generation systems? Basically, to mimic the written word on the page like a stick bug on a stick. They're like those beetles that live in ant colonies, sending out social signals that make the ants see them as offspring that have to be babied rather than parasites that don't contribute. They replicate the form while not being the thing that they have subverted the signals of being.
EDIT: There is something wrong with the upvote counter
When I am talking at academic conferences to astrobiologists about how their/my field has been poisoned over time by singularity cults ideas seeping into the literature uncited, their eyes get particularly wide when I get to this part.
thanks for using Leebra!
go to feed...