Bugpocalypse, or reporting bugs in an AI age - QEMU

20 days ago by Alex to c/opensource

I wrote a blog post for the project this week detailing some of the challenges the QEMU project, in common with other FLOSS projects, are seeing with AI generated bug reports. The dilemma being that the LLM's have got to the point they generate better bug reports than a lot of humans do.

Shimitar 1 point 20 days ago

In my limited experience, AI properly guided can be a great bug hunter.

As usual with AI anything must be thoroughly vetted of course, but overall it's a huge time saver.

save

path: 0 25991089, hotness: undefined, score: 1, children: 2

stsquad 6 points 20 days ago

If the bugs were being found in code that is used heavily in production I'd agree. Spamming the tracker with 100's of bugs in random unmaintained areas of the code base just wastes everyone's time. Perhaps as a project we should be more aggressive in deprecating code but the threat profile for someone running firmware blobs from their camera is very different from people launching untrusted guests on their shared hosting infrastructure.

save

path: 0 25991089 25991357, hotness: undefined, score: 6, children: 1

pastermil 3 points 19 days ago

While I'm with you on the tracker spamming, those random unmaintained areas would need to be addressed one way or another.

save

path: 0 25991089 25991357 26020285, hotness: undefined, score: 3, children: 0

thenextguy -1 points 20 days ago

Can you use an llm to triage the other llm's bug reports? Fire with fire.

save

path: 0 25991070, hotness: undefined, score: -1, children: 4

stsquad 2 points 20 days ago

I do run a bot that can do the initial triage, it doesn't solve the problem of bugs stacking up in unmaintained areas of the code. I have experimented with auto-fixing the bugs with an LLM but currently the project policy is no AI contributions at all. We are in the process of debating a new policy which might allow for simple fixes to be done with AI.

save

path: 0 25991070 25991303, hotness: undefined, score: 2, children: 3

thenextguy 2 points 20 days ago

I didn't mean fixing them. Just categorizing them. Eliminating duplicates. Marking some as 'won't fix', etc.

save

path: 0 25991070 25991303 25991324, hotness: undefined, score: 2, children: 1

stsquad 1 point 20 days ago

Out of an abundance of caution we don't let the agent close bugs or assign CVEs but tell it to defer to a real human for that. I'll have to experiment with how we can do a better job of detecting duplicates because I'm sure a fair number of the issues are.

save

path: 0 25991070 25991303 25991324 25991393, hotness: undefined, score: 1, children: 0

eldavi 1 point 20 days ago

I have experimented with auto-fixing the bugs with an LLM but currently the project policy is no AI contributions at all. We are in the process of debating a new policy which might allow for simple fixes to be done with AI.

my management wants us to do this using ansible tower (i refuse to t call it the ansible automation platform) and the test cases we setup with it auto repairing systemd configurations and restarting it until it works the way we need it to are working REALLY well.

save

path: 0 25991070 25991303 25993610, hotness: undefined, score: 1, children: 0

Useful Links

Rules

Related Communities

Useful Links

Rules

Related Communities