I'm not a big business boy, but I think they should have considered the costs of AI before getting to the point of spending $1000/day and taking out bank loans.

It's funny how they're twisting themselves into pretzels to make this work. They started by casually saying that AI could copy itself onto the internet, not anticipating that people would immediately point out just how much hardware it takes to run their AI. Seems like they really want to die on this hill.

The current version is a demo, so they are making $0 per sale. The SteamDB graph for the demo shows that the daily maximum player count has been around 300 for the past few days, and there are around 1000 followers for the unreleased full version, so I assume that a large fraction of those players are consistent (for now). Unfortunately, I could not find any data about the total number of downloads, and I don't think that information is public.

But setting that aside, every game has a player base that follows a power law. Yes, most players are casual, but the most dedicated players spend a disproportionate amount of time and money. In mobile games, these people are called "whales", and they are the backbone of the business. For example, 50-70% of a mobile game's in-app purchase revenue comes from 1-2% of the players. Even if most players spend a couple hours on the game, the developers must account for the people who will sink a ton of time into it.

The big problem with the AI business model is that this dynamic is now working against them. These whales now cost tons of money. They need to somehow charge these players at least $70/mo, with a crappy novelty game that would barely scrape by for $10 on Steam. Good luck with that.

Looking at the bigger picture, this is why AI subscriptions are not a viable business model. Whatever morsels of profit you might get from your regular users are more than cancelled out by Steve Yegge spending $2.8k/month for $87k/mo worth of tokens. Whatever revenue the AI companies get largely comes from convincing executives of a few rich companies to fork over absurd amounts of money, which produces a massive concentration risk in their revenues.

Man, every time I hear about the cost of an actual AI application, it is always something obscene. At least this isn't as bad as Steve "Gas Town" Yegge spending $30k/year for a dozen subscriptions (that cost Anthropic ~$1M/year in token costs) to develop his game with 6 active players.

At this point I have zero patience for my colleagues who think that the AI companies could make the economics all work out. Doesn't help that mathematicians are the least cost-conscious people on the planet, and the AI companies are massively subsidizing their subscriptions in particular.

The aliens metaphor is already on Scott's blog: https://scottaaronson.blog/...

The spammer in question had previously posted a bigger result, but the writeup was so badly written that some people in the field wrote a new, readable paper with the same result. In their abstract, they specifically cited him and said that the proof was so unreadable that they were unable to follow the details, and they had no choice but to write their own proof. (The spammer's proof also used math that was advanced enough to be far beyond his knowledge.)

My pet theory is that after this embarrassment, the spammer saw that nobody was able to verify his argument. Now the real problem is that his paper was written so badly that nobody could understand it (in math, you can't verify what you can't understand), but somehow he thought that the problem was that he didn't add an experiments section!

Ever since the beginning of the current AI bubble, there has always been a certain type of dipshit who takes some art posted on social media by an artist, feeds it to AI, and then comments with the slop image saying that they made it "better". Well, now I am on the receiving end of that. In one of my papers, I (foolishly) wrote down a few comments on future directions, and now some motherfucker decides to send me an obviously ChatGPT generated manuscript with a solution to one of my problems.

Unfortunately, I think their AI generated proof is itself correct, so I have to deal with it. But the manuscript is written in a way that obviously indicates that he hasn't even read it. There's even a section describing experiments for this mathematical question (???), complete with a "verification guide" that references an "archive" full of python scripts and test cases that he did not send me. I suspect he doesn't even know they exist.

You know how AI proofs are supposed to be credible because of formalization with Lean? The AI generates a natural language proof and then supposedly generates a Lean program corresponding to the proof to verify that it's true. There is now some troubling news about that: there are frequent examples where the statement given in natural language is different from the meaning of the Lean formalization.

The paper even points out several obvious mismatches in OpenAI's famous Navier-Stokes solution between the English proof and the Lean formalization. There is even a spot where a 4 in the English proof turns into a 5 in the Lean proof (which causes further problems down the line). This is mostly a cosmetic error, and it doesn't mean that the Navier-Stokes solution is wrong, but there could very well be deeper problems, especially with the English proof. At minimum, this solution absolutely requires careful review by experts. OpenAI's behavior towards said experts has been appalling.

My viewpoint has been that the only successes that AI has had are in domains where failure is not costly (at least for someone with deep pockets), and especially when there is a formal system such as Lean that can reliably catch errors. So AI can be endlessly trained to write compiling Lean programs corresponding to formal proofs, but as soon as it runs into tasks that are not 100% bolted down, like writing the proof in English, the hallucination problem rears its ugly head. This reinforces my belief that AI is a chess engine but for Lean.

If I was treating this as a technical problem, I would say, why not just have an AI that generates the Lean proof, and at most have an optional assisting tool that translates it into English without perfect reliability? But this is not a technical problem. By outputting English proofs in the format of a research paper, OpenAI can more easily market that their AI is intended to fully replace mathematicians, and that this is only the herald of their general superintelligence (and not that math is the only thing working out for them).

One of the proofs that OpenAI announced yesterday is for a major open problem in the field I'm working in. I tried to read it but it was a completely incomprehensible 100+ page output. I guess the community has to do all the dirty work of figuring out any sort of understanding. (I had known about the results weeks in advance before their official announcement. There is a bunch of more drama there that I won't get into.) Putting aside the serious possibility of Lean bugs, a lot of people don't realize that there is ultimately no point in having a machine just generate Lean blobs endlessly. The reality is that mathematicians have always done math for the sake of enjoyment or understanding, and math contributes to society by improving understanding. Still, I am worried about the convenient social arrangement that allowed many mathematicians to do this for a modest living.

Anyway, I'm sure it was a good idea to apply the full might of capital for pure math problems, instead of less important social issues like climate change. We even got a more efficient discrete Fourier transform, improved from O(n log n) to, I shit you not, O(n (log n)^0.9999999999999)).

Edit: Here is an article by Quanta which I think accurately describes the situation that mathematicians are going through right now: https://www.quantamagazine.org/...

Scott Aaronson is teaching a Yudkowsky class: https://scottaaronson.blog/?p=10125

“California is, I believe, the only state to give health insurance to people who come into the country illegally,” Kauffman said nervously. “I think we probably should not be providing that.”

“So you’d rather everyone just be sick, and get everyone else sick?” another reporter asked.

“That’s not what I’m saying,” said Kauffman.

“Isn’t that effectively what happens?” the reporter countered. “They don’t have access to health care and they just have to get sick, right?”

Kauffman contemplated that one for a moment. “Then they have to just get sick,” he said. “I mean, it’s unfortunate, but I think that it’s sort of impossible to have both liberal immigration laws and generous government benefits.”

Do I need to comment on this one?

Terry Tao talks about how he used to try to cooperate with the AI industry to achieve a positive outcome, but now he finally sees their true colors. Link

During this event, OpenAI requested an interview concerning my vision of the future of AI and mathematics. I accepted, and spoke with them for perhaps an hour. I had done similar interviews in various venues, and I assumed that, as with these other cases, they would eventually post the entire interview online, which talked about both the possibilities and risks of AI much as I have done in these other interviews. As it turned out, they only used a few snippets of that interview for that infamous advertisement instead. In retrospect, I should have pushed back harder on their decision; but I decided at the time that even a selective release of my commentary would help raise awareness of the potential for AI, and in particular on the possibility of the “best of both worlds”.

Since then, the situation has deterioriated markedly. Many of the people in the industry that shared my views have left or become sidelined, with most major tech companies now increasingly focused on the race to develop extremely powerful, autonomous AI technologies regardless of their actual value to society. The current drama surrounding the Navier-Stokes global regularity problem is the most dramatic and visible instance of this, but there have been multiple other such examples, and much of my commentary in the last few months has been aimed that the increasingly severe divergence between the current objectives of the AI industry, and of mathematics in general.

Much respect to artists for seeing all this coming from the very beginning, and holding the line.

I decided to take a look at the bitcoin white paper.

Usually, the introduction of a technical paper is fluff and people quickly move on to the technical parts. However, the casual claims made in the first paragraph of this paper have aged extremely poorly, to say the least. In a better world, Bitcoin would have remained as an obscure academic toy, and this introduction would have remained fluff.

While the system works well enough for most transactions, it still suffers from the inherent weaknesses of the trust based model.

What weaknesses are there in the trust based model? Let's find out!

Completely non-reversible transactions are not really possible, since financial institutions cannot avoid mediating disputes. The cost of mediation increases transaction costs, limiting the minimum practical transaction size and cutting off the possibility for small casual transactions, and there is a broader cost in the loss of ability to make non-reversible payments for non-reversible services. With the possibility of reversal, the need for trust spreads. Merchants must be wary of their customers, hassling them for more information than they would otherwise need.

It seems like this guy really loves non-reversible transactions! But as we've seen with the history of crypto, non-reversible transactions sound really good until you fall victim to a crypto scam and there is no way to appeal to the bank to reverse the charges. Reversibility actually increases trust because you no longer need to be absolutely certain that you're dealing with an honest person.

A certain percentage of fraud is accepted as unavoidable.

Almost like that is a problem of human nature. And it's not like cryptocurrency has a spotless record when dealing with fraud! The problem with fraud is not the third party (the bank), but with the second party (the merchant or customer you're dealing with).

The introduction is not long, and most of the paper concerns the technical details of the construction of Bitcoin. By itself, there really is no way to complain about a pile of definitions. But there are still dumb comments that have aged poorly in retrospect.

A block header with no transactions would be about 80 bytes. If we suppose blocks are generated every 10 minutes, 80 bytes * 6 * 24 * 365 = 4.2MB per year. With computer systems typically selling with 2GB of RAM as of 2008, and Moore's Law predicting current growth of 1.2GB per year, storage should not be a problem even if the block headers must be kept in memory.

But why would you want a block header with no transactions? If you wanted to, I don't know, replace the world's financial system, you would need to handle millions of transactions every 10 minutes. How big would the blocks be then? And remember that many copies of the same blockchain would need to be stored (certainly, every miner would need to store a copy). How many thousands or millions of times would that multiply things?

Businesses that receive frequent payments will probably still want to run their own nodes for more independent security and quicker verification.

Turns out it was a bold assumption to think that businesses would just run their own bitcoin miners.

The proof of security (Section 11) is extremely sketchy by modern standards. (They're assuming that all attackers would follow a certain format to attack and not try something different. I get it, proper proofs of security in cryptography are very subtle and difficult.) There is also a page of fluff making random calculations with the Poisson distribution. In any case, the security of Bitcoin requires that the collective computational power of the defenders exceeds the power of any attacker (so the defenders can make new blocks faster).

Bitcoin is very strange as a cryptographic system in that the defender must have more resources than any possible attacker. In most cryptographic systems, the system should be secure even if the attacker has vastly more resources than the defender. Your phone's cryptography should be secure even if some government agency dedicated their supercomputers to try and break it. This means that Bitcoin must waste tons of energy, since that is required to maintain security. Any more energy dumped into it will only increase security and not make the actual transactions faster, which makes Bitcoin horrendously inefficient.

As a purely academic idea in cryptography, it is an interesting curiosity, but the arguments for why it's useful are sketchy. There are other such curiosities that are much more interesting, like homomorphic encryption or secure multiparty computation. It would be a nice line on a CV, but not "incredible".

The true significance of Bitcoin was the terrible libertarian economic argument for it, and the chain of events that would transform it into nothing more than a speculative fashion trend. It has nothing to do with the technical details of Bitcoin. The technical and economic arguments for Bitcoin turned out to be so weak that nowadays, the only real support for Bitcoin is that maybe you can sell it for a higher price to a greater fool.

Just had a conversation about AI where I sent a link to Eddy Burback's ChatGPT Made Me Delusional video. They clarified that no, it's only smart people who are more productive with AI since they can filter out all the bad outputs, and only dumb people would suffer all the negative effects. I don't know what to fucking say.

It is how professors talk to each other in ... debate halls? What the fuck? Yud really doesn't have any clue how universities work.

I am a PhD student right now so I have a far better idea of how professors talk to each other. The way most professors (in math/CS at least) communicate in a spoken setting is through giving talks at conferences. The cool professors use chalkboards, but most people these days use slides. As it turns out, debates are really fucking stupid for scientific research for so many reasons.

1. Science assumes good faith out of everyone, and debates are needlessly adversarial. This is why everyone just presents and listens to talks.
2. Debates are actually really bad for the kind of deep analysis and thought needed to understand new research. If you want to seriously consider novel ideas, it's not so easy when you're expected to come up with a response in the next few minutes.
3. Debates generally favor people who use good rhetoric and can package their ideas more neatly, not the people who really have more interesting ideas.
4. If you want to justify a scientific claim, you do it with experiments and evidence (or a mathematical proof when applicable). What purpose does a debate serve?

I think Yud's fixation on debates and "winning" reflects what he thinks of intellectualism. For him, it is merely a means to an end. The real goal is to be superior and beat up other people.

The article's entire premise is Musk saying some random shit. Remember how Musk said that he would land a man on Mars in 10 years 13 years ago? Honestly, I am incensed that people like Musk and Trump can just say shit and many people will just accept it. I can no longer tolerate it.

Putting aside the very real human ability to screw up such a concept and turn any fair system into an unfair one, ...

He says this after mentioning UBI. He really doesn't want to confront the unfortunate fact that UBI is entirely a political issue. Whatever magical beliefs one may have about how AI can create wealth, the question of how to distribute it is a social arrangement. What exactly stops the wealthy from consolidating all that wealth for themselves? The goodness of their hearts? Or is it political pushback (and violence in the bad old days), as demonstrated in every single example we have in history?

I'd say the problem is even worse now. In previous eras, some wealthy people funded libraries and parks. Nowadays we see them donate to weirdo rationalist nonsense that is completely disconnected from reality.

No getting up early and commuting on public transit. ...

This is followed by four whole paragraphs about how the office sucks and wouldn't it be wonderful if AI got rid of all that. Guess what, we have remote work already! Remember how, during COVID, many software engineering jobs went fully remote, and it turned out that the work was perfectly doable and the workers' lives improved? But then there were so many puff pieces by managers about the wonderful environment of the office, and back to the office they went. Don't worry, when the magical AI is here, they'll change their minds.

Yes, there are "mindless, stupid, inane things" like chores that are unavoidable. There are also other mindless, stupid, inane things that are entirely avoidable but exist anyway because some people base their entire lives around number go up.

Sorry for the X screenshot but XCancel is shut down from legal pressure.

Normally I'd sneer at their 10% chance of extinction claims, but now that I think about it, if you take their claims seriously at face value, you realize just how these people are monsters. We rightly condemn brutal dictators who commit genocide, but even their crimes would pale in comparison to anything with a 10% chance of exterminating all of humanity.

These "reasons" are laughably indefensible given the supposed stakes here. The first is already terrible, and they just get worse.

1. The only moral thing to do here is to immediately stop working for the AI companies, and do whatever you can to stop anyone else. It's not okay to piss in a pool just because you suspect other people are also doing it. And what makes you think that somehow you are uniquely positioned to succeed in making it safe?
2. This is the Sam-Bankman Fried defense (complete with a reference to "EV" (expected value)). When you decide to gamble with the lives of everyone in the entire world, did you first ask them if they are okay with you gambling with their lives? Sure, I might die, but if the ball lands on red, things will be great! How moral does this sound?
3. What the actual fuck? Imagine if Hermann Göring defended himself at Nuremberg by saying, "At least I had fun and got rich and worked on 'cool tech' with friends."

Now, I think that these people sincerely believe that AI has a 10% chance of wiping out humanity, because their minds have been poisoned by Rationalist ideology. Their Rationality is so effective that after Bernie Sanders took their premises at face value, they were shocked when Bernie immediately came to the conclusion of locking these people up for 20 years.

There are two specific aspects of Rationalism that cause this mysterious blind spot. First, they believe that superintelligence is inevitable, which conveniently removes responsibility from the people actively trying to bring it about. Second, they treat everyone outside the Rationalist sphere as NPCs who dutifully follow the directions on their script. The rule of law does not even occur to them as they treat the extinction of humanity as just another dinner table conversation starter.

My colleagues in math are now frightened about the (very expensive) mathematical theorem proving ability of these AIs, and many of them really do think that if they can do math, they can do all cognitive tasks. Running a store like this should be so easy! Every single conversation about AI with them has become more frustrating. They are so confused when I still say that the AI companies will die a painful death. When I give my usual points about their expense and their failures in other domains, I am given the usual spiel of "it'll get better in other areas" and "it'll get cheaper".

Unlike them, I have actually been paying attention to this stuff from the beginning. What they think is going on is AI solving math first and shortly getting around to all the other stuff, but what I've seen is that AI labs had already tried all the other stuff first and only managed to win the booby prize of theorem proving, which doesn't pay the bills. And what's the point of spending thousands or millions to output random blobs of Lean that technically compile if there is no one around to bother making sense of them?

One example I gave is when Anthropic vibe coded an entire C compiler from scratch back in February, which turned out to be a pile of shit. I've said that if AI had made similarly rapid progress on software engineering, we would have seen Anthropic continue to put out these demonstrations, and they would have become truly high quality. They would release a compiler more efficient than gcc one week, and a browser better than Chrome the next. (OpenAI's actual attempt at a browser didn't go so well.) And if they could do this, they would actually have a shot of making money!

If they could do this, they would have already. The theorem proving stuff actually works (for certain things, in certain ways, at enormous expense), and look at how OpenAI and Anthropic do not hesitate to snipe mathematicians for results rather than being content as tool vendors. But lately I haven't heard of any software demonstrations. Silence is much louder than noise. More Millennium prize problems bashed with tens of millions in compute costs are not going to change my mind very much.

The counterargument I got was that AI can already one-shot most programming tasks and I shouldn't be cherry-picking the failures. I am far too tired to argue at this point.

I've seen this story play out in software engineering: people were very impressed when the AI does unexpectedly well in one out of 50 attempts on an easy task, and so people decided to trust it for everything and turn their codebases into disasters. There was no great wave of new high-quality software. Instead, the only real result was that existing software has become far more buggy and insecure.

Now we have people using AI in science and math because it was impressive in random demonstrations of solving math problems. I now have friends asking me why I'm not using AI, and also saying that AI will be better than all mathematicians in 30 years or whatever. Do you really think I refuse to use AI out of ignorance? No, I know too much about it! I have seen the same story play out in software engineering, and what makes this any different?