@awful.systems
What fullsquare said, they've got years of sci-fi filtered through lesswrong short stories and sequence posts saying it could happen, so they repeated that despite all the ways it is completely impossible given the limitations and hardware requirements of LLMs.
Yep, so many of their favorite tropes and assumptions fail when an actual expert breaks them down. Like their belief in the possibility of Drexler-style nanotech has been taken down quite firmly several times (here is a good takedown that got posted to the EA forums), Drexler's ideas pretty firmly ignore how things actually work at the molecular level...
One of the classic sequence posts plays with the concept: https://www.lesswrong.com/...
But in a uno reverse twist, its the humans that are a metaphor for AI, with them manipulating out thinking the aliens from within the simulation.
Someone on Scott Aaronson's blog suggested math will become an empirical field now, where they run "experiments" by asking AI for lean proofs.
Scott, is this a turning point where mathematics gets off its well trodden path of logic and becomes more like an empirical science? Heck, people were arguing if math is invented or discovered, this settles it. Math is discovered – by AI! From now on forth, Mathematicians will become scientists, wielding AI as a telescope, pointed towards new and disturbing alien worlds in the Platonic realm.
So yeah, it seems like this idiotic view is pretty popular among boosters.
I do not see the point of math for AI alone, since I do not think AI is or will ever be conscious.
Rationalists (and in general those with fear/hope or a technological singularity) have the idea that once the AI is smart enough, it will be able to recursively self-improve in an exponential process. Vibe-coding hasn't made this come true, so they are trying to make it happen by shoving enough of the right sort of knowledge into the AI. Obviously being smart and being mathey go together, so if the AI just gets good enough at math, surely bootstrapping to godhood is just around the corner!
To them, the AI is not here for us, but we are here to serve the AI.
The techbros are giving mathematicians the humanities treatment. (I.e. they don't respect the entire profession despite its immense importance to actually understanding reality.) Before, mathematics had sort of gotten half a pass for being rigorous and adjacent to stuff techbros do respect, like computer science. But, given just half a whiff of the idea that they may no longer need human mathematicians, they are ready to accept utter slop...
True, but its a different form of disrespect, where they outwardly steal the style and aesthetics of the discipline to try to claim its legitimacy. Like the endless "whitepapers" pumped out by cryptobros.
The theory on the thread that
My guess is that this represents a shift in focus at OpenAI away from math: they had their big meaningless trophy prize with the Navier-Stokes solution, they appear to have distanced the competition (Anthropic) and the public bought the notion that “math is solved”
seems like a plausible explanation? I'll guess we'll if OpenAI moves on now that they've "solved" math or if they follow up with a few more drops of "proofs".
You know how AI proofs are supposed to be credible because of formalization with Lean?
There is another problem... apparently Lean4 hasn't actually been fully formally verified. And in fact could have more bugs lurking in it, maybe even some foundational problems... Lesswrong post explaining the problems.
OpenAI can more easily market that their AI is intended to fully replace mathematicians, and that this is only the herald of their general superintelligence (and not that math is the only thing working out for them).
Yeah, this has been driving me crazy... reading places like hackernews or r/singularity, I keep seeing people assuming mathematicians and just bitter or engaging in sour grapes and that is why they are complaining about the "proofs" OpenAI dumped out. Like it is just some small niggling detail that the proofs can't be read by humans and that Lean may have bugs and that some of OpenAI's proofs have already been shown to have critical mistakes.
Edit: I had the thought to collect some links in one place Here are some more discussions on lean and autoformalization for anyone looking for good sources:
Edit 2: Made a top level post: https://awful.systems/post/9997783
Place like /r/singularity and hackernews are eating up the hype and buying into it. Even with mathematicians explaining carefully the errors in the proofs and why they are badly written and useless to actually progress in mathematics the boosters are treating that like sour grapes or irrational fear. If mathematicians tried outright ignoring it, OpenAI's narrative would go even more unchallenged.
Or cite it, but in the most passive aggressive way you can get away with.
...You should check the literature really carefully to make sure that whatever proof the AI "generated" doesn't actually already exist somewhere and it regurgitated it without proper accreditation.
Of course, part of that wiring will be figuring out how to deal with the the signal to noise ratio of ~1:50 in this case, but that’s something we are already making progress at.
This line annoys me... LLMs excel at making signal-shaped noise, so separating out an absurd number of false positives (and investigating false negatives further) is very difficult. It probably requires that you have some sort of actually reliable verifier, and if you have that, why bother with LLMs in the first place instead of just using that verifier directly?
The promptfondlers on places like /r/singularity are trying so hard to spin this paper. "It's still doing reasoning, it just somehow mysteriously fails when you it's reasoning gets too long!" or "LRMs improved with an intermediate number of reasoning tokens" or some other excuse. They are missing the point that short and medium length "reasoning" traces are potentially the result of pattern memorization. If the LLMs are actually reasoning and aren't just pattern memorizing, then extending the number of reasoning tokens proportionately with the task length should let the LLMs maintain performance on the tasks instead of catastrophically failing. Because this isn't the case, apple's paper is evidence for what big names like Gary Marcus, Yann Lecun, and many pundits and analysts have been repeatedly saying: LLMs achieve their results through memorization, not generalization, especially not out-of-distribution generalization.
They are more defensive of the racists in the other blog post on this topic: https://forum.effectivealtruism.org/...
Maybe its because the HBDers managed to control the framing with the other thread? Or because the other thread systematically refuses to name names, but this thread actually did name them and the conversation shifted out of a framing that could be controlled with tone-policing and freeze peach appeals into actual concrete discussion of specific blatantly racists statements (its hard to argue someone isn't racist and transphobic when they have articles with titles like "Why Do I Hate Pronouns More Than Genocide?").
There's more shit gems in the comments, but I think my summary captures most of the major points. One more comment that stuck out:
Being a republican is equally as compatible with EA as being a Democrat. Lots of people on both sides have incompatible views. I honestly think you just haven't met enough Republicans!
Yes, this is actually true, and it is a bad thing and an indictment of EA.
Edit 1: There is another post clarifying that it wasn't mostly racists (https://forum.effectivealtruism.org/... ) but 1) this is sneerclub, not careful count of the exact percentage of racists and racist talks to avoid hurting feelings club 2) if you sit down at a table with 3 Neo-Nazis, there are 4 Neo-Nazis sitting down. 3) "Full" is a subjective description, so yes its valid. two major racists would be more than my quota 4) see sidebar on debate
So, I've been spending too much time on subreddits with heavy promptfondler presence, such as /r/singularity, and the reddit algorithm keeps recommending me subreddit with even more unhinged LLM hype. One annoying trend I've noted is that people constantly conflate LLM-hybrid approaches, such as AlphaGeometry or AlphaEvolve (or even approaches that don't involve LLMs at all, such as AlphaFold) with LLMs themselves. From their they act like of course LLMs can [insert things LLMs can't do: invent drugs, optimize networks, reliably solve geometry exercise, etc.].
Like I saw multiple instances of commenters questioning/mocking/criticizing the recent Apple paper using AlphaGeometry as a counter example. AlphaGeometry can actually solve most of the problems without an LLM at all, the LLM component replaces a set of heuristics that make suggestions on proof approaches, the majority of the proof work is done by a symbolic AI working with a rigid formal proof system.
I don't really have anywhere I'm going with this, just something I noted that I don't want to waste the energy repeatedly re-explaining on reddit, so I'm letting a primal scream out here to get it out of my system.
I haven't looked into the Zizians in a ton of detail even now, among other reasons because I do not think attention should be a reward for crime.
And it doesn't occur to him to look into the Zizians in order to understand how cults keep springing up from the group he is a major thought leader in? Like if it was just one cult, I would sort of understand the desire just to shut ones eyes (but it certainly wouldn't be a truth-seeking desire), but they are like the third cult (or 5th or 6th if we are counting broadly cult-adjacent group) (and this is not counting the entire rationalist project as cult). (For full on religious cults we have: leverage research, and the rationalist-Buddhist cult; for high-demand groups we have: the Vassarites, Dragon Army's group home, and a few other sketchy group living situations (Nonlinear comes to mind)).
Also, have an xcancel link, because screw Elon and some of the comments are calling Eliezer out on stuff: https://xcancel.com/...
Funny sneer in the replies:
I read the Sequences and all I got was this lousy thread about the glomarization of Eliezer Yudkowsky's BDSM practices
Serious sneer in the replies
this seems like a good time to point folks towards my articles titled "That Time Eliezer Yudkowsky recommended a really creepy sci-fi book to his audience and called it SFW" and "That Time Eliezer Yudkowsky Wrote A Really Creepy Rationalist Sci-fi Story and called it PG-13
Saw this posted to the Reddit Sneerclub, this essay has some excellent zingers and a good overall understanding of rationalists. A few highlights...
Rationalism is the notion that the universe is a collection of true facts, but since the human brain is an instrument for detecting lions in the undergrowth, almost everyone is helplessly confused about the world, and if you want to believe as many true things and disbelieve as many false things as possible—and of course you do—you must use various special techniques to discipline your brain into functioning more like a computer. (In practice, these techniques mostly consist of calling your prejudices ‘Bayesian priors,’ but that’s not important right now.)
We're all very familiar with this phenoma, but this author has a pithy way of summarizing it.
The story is not a case study in how rationality will help you understand the world, it’s a case study in how rationality will give you power over other people. It might have been overtly signposted as fiction, with all the necessary content warnings in place. That doesn’t mean it’s not believed. Despite being genuinely horrible, this story does have one important use: it makes sense out of the rationalist fixation on the danger of a superhuman AI. According to HPMOR, raw intelligence gives you direct power over other people; a recursively self-improving artificial general intelligence is just our name for the theoretical point where infinite intelligence transforms into infinite power.
Yep, the author nails the warped view Rationalists have about intelligence.
We’re supposedly dealing with a group of idiosyncratic weirdos, all of them trying to independently reconstruct the entirety of human knowledge from scratch. Their politics run all the way from the furthest fringes of the far right to the furthest fringes of the liberal centre.
That is a concise summary of their warped Overton Window, yeah.
From the headlines I had been assuming it was a normal tech company patent troll slap fight but no... from the 9to5mac article:
When interviewing Apple employees for jobs at OpenAI, Mr. Tan uses Apple’s confidential information to gain access to even more insider knowledge. He has used an Apple internal project codename to ask, “What’s the plan[?]” for an unannounced Apple product.
He has directed job candidates still working for Apple to bring “Actual parts” from Apple to their interviews for “show and tell” sessions in which he and his team at OpenAI can elicit still more Apple confidential information. These directions to bring Apple’s parts to OpenAI job interviews surprised at least one of the candidates, who commented that he “didn’t even know we could take those from the office.”
OpenAI has been instructing Apple employees to bring “CAD/design artifacts” and “prototypes” to their interviews and to divulge details about their work such as “subsystem and component selection,” the “tools or methodologies you use for system integration, such as CAD software, simulation tools,” and “Vendor selection and communication/collaboration with vendors.”
That is absolutely batshit. Like I've heard the interview process has been getting worse and worse, but "steal shit from your current company" is on another level. I wonder if he lead in with any plausible deniability to it or just straight up jumped to that. Also, I have a hard time imagining being willing to steal from a company just to get through an interview at another company. Not out of company loyalty, but because I wouldn't trust the interviewers to actually pay-up with a cushy job and salary.
Which, to recap for everyone, involved underpaying and manipulating employees into working as full time general purpose servants. Which is pretty up there on the scale of cult-like activity out of everything EA has done. So it makes sense she would be trying to pull a switcheroo as to who is responsible for EA being culty...
thanks for using Leebra!
go to feed...