Study Finds That 52 Percent of ChatGPT Answers to Programming Questions Are Wrong

2 years ago by ekZepp to c/technology

Researchers found that 52 percent of answers to programming questions generated by ChatGPT were incorrect.
load all comments
vzq 111 points 2 years ago

You should see 52% of the first version of my code.

It doesn’t have to be right to be useful.

path: 0 10248256, hotness: undefined, score: 111, children: 27
restingboredface 90 points 2 years ago

Yeah, but the non-tech savvy business leaders see they can generate code with AI and think 'why do I need a developer if I have this AI?' and have no idea whether the code it produces is right or not. This stat should be shared broadly so leaders don't overestimate the capability and fire people they will desperately need.

path: 0 10248256 10248310, hotness: undefined, score: 90, children: 15
piecat 35 points 2 years ago

I say let it happen. If someone is dumb enough to fire all their workers... They deserve what will happen next

path: 0 10248256 10248310 10248697, hotness: undefined, score: 35, children: 3
homesweethomeMrL 27 points 2 years ago

Well the firing’s happening so, i guess let's hope you’re right about the other part.

path: 0 10248256 10248310 10248697 10248801, hotness: undefined, score: 27, children: 0
TheRaven 12 points 2 years ago

It won’t happen like that. Leadership will just under-hire and expect all their developers to be way more efficient. Working will be really stressful with increased deadlines and people questioning why you couldn’t meet them.

path: 0 10248256 10248310 10248697 10251165, hotness: undefined, score: 12, children: 0
FiniteBanjo 0 points 2 years ago

Intentionally hindering business on a grand scale doesn't seem like a good idea. I say regulate it to prevent such sabotage.

path: 0 10248256 10248310 10248697 10265653, hotness: undefined, score: 0, children: 0
scrubbles 19 points 2 years ago

Yeah management are all for this, the first few years here are rough with them immediately hitting the "fire the engineers we have ai now". They won't realize their fuckup until they've been promoted away from it

path: 0 10248256 10248310 10248488, hotness: undefined, score: 19, children: 0
NuXCOM_90Percent 5 points 2 years ago

Mentioned it before but:

LLMs program at the level of a junior engineer or an intern. You already need code review and more senior engineers to fix that shit for them.

What they do is migrate that. Now that junior engineer has an intern they are trying to work with. Or... companies realize they don't benefit from training up those newbie (or stupid) engineers when they are likely to leave in a year or two anyway.

path: 0 10248256 10248310 10249164, hotness: undefined, score: 5, children: 0
Boozilla 4 points 2 years ago

Programming jobs will be safe for a while. They've been trying to eliminate those positions since at least the 90s. Because coders are expensive and often lack social skills.

But I do think the clock is ticking. We will see more and more sophisticated AI tools that are relatively idiot-proof and can do things like modify Salesforce, or create complex new Tableau reports with a few mouse clicks, and stuff like that. Jobs will be chiseled away like our unfortunate friends in graphic design.

path: 0 10248256 10248310 10248687, hotness: undefined, score: 4, children: 6
BlameThePeacock 14 points 2 years ago

You, along with most people, are still looking at automation wrong. It's never been about removing people entirely, even AI, it's about doing the same work with less cost.

If you can eliminate one programmers from your four person team by giving the other three AI to produce the same amount of work, congrats you've just automated one programming job.

Programming jobs aren't going anywhere, but either the amount of code produced is about to skyrocket, or the number of employed programmers is going to drop (or most likely both of those things).

path: 0 10248256 10248310 10248687 10249086, hotness: undefined, score: 14, children: 5
EatATaco 2 points 2 years ago

And they'll find out very soon that they need devs when they actually try to test something and nothing works.

path: 0 10248256 10248310 10259506, hotness: undefined, score: 2, children: 0
IsThisAnAI 1 point 2 years ago

Tech unemployment, at least in the US, is sitting at 1%. You've seen media sensationalism, very few companies and leaders actually fully buy into the hype.

path: 0 10248256 10248310 10251088, hotness: undefined, score: 1, children: 0
technology
technology

@lemmy.ml

login for more options
43126
3839
2041

This is the official technology community of Lemmy.ml for all news related to creation and use of technology, and to facilitate civil, meaningful discussion around it.


Ask in DM before posting product reviews or ads. All such posts otherwise are subject to removal.


Rules:

1: All Lemmy rules apply

2: Do not post low effort posts

3: NEVER post naziped*gore stuff

4: Always post article URLs or their archived version URLs as sources, NOT screenshots. Help the blind users.

5: personal rants of Big Tech CEOs like Elon Musk are unwelcome (does not include posts about their companies affecting wide range of people)

6: no advertisement posts unless verified as legitimate and non-exploitative/non-consumerist

7: crypto related posts, unless essential, are disallowed

go to feed...