Hell, that's the whole thing with these LLM-based business/product structures, isn't it? The models are very good at creating something that looks right, leading to people being absolutely blindsided when they fail to actually do the thing that boosters and salesmen pretended they were doing.

Given that these are statistical models that function probabilistically, it seem like the obvious attitude to take would be to assume it's a question of when they fail and do something wrong, rather than if. But accounting for that inevitability undermines most if not all of the actual economic value of these things because it turns out it takes just about as much time, effort, and skill to monitor and check these things as it would to just do the damn work yourself. But as soon as you start giving these things permissions to operate independently you are setting up a time bomb and putting duct tape over the timer. You will get fucked eventually.

it turns out that people who can’t code make terrible judges of how good ai is at coding

It makes every bad programmer into a 10x bad programmer (equivalent to 10 bad programmers).

And now that you trust me, bonus me!

We cant estimate how much effort it will take to fix or maintain it. That's the problem

That's the expected result of selling stuff that doesn't exist or does not really work.

What you’re saying is there will be lucrative billable hours to fix it when this all shakes out? I guess short term pain…

the most robust study in AI coding! cos they built it from toothpicks instead of toilet paper