Just in case as it’s formatted a bit weirdly

X-Axis: Homework scores

Y-Axis: Exam scores

I'm sorry, but WTF is this chart without clear axes?

This is from the paper, and clearer to me.

Did AI come up with these axes?

My takeaway: letting AI do your homework for you is essentially the same as letting AI study for you.

Kind of obvious, but the data is still good to have.

Wouldn't a bar graph be better for this sort of data? The line keeps suggesting change over time to me and it's making the whole thing hard to read, even with OP's note.

Cognitive attrition.

Turns out you need to practice doing things. Who knew.

Looks like millennials / elder Gen z are going to be the most educated generation in history. Older generations were taught when they knew less about the world and education, younger generations are screwed both by the pandemic and now AI.

So Idiocracy was sort of right, only it's not the eugenics "dumb people have more kids" reason they gave, it's just us offloading our thinking to machines.

Hm, this is something that feels quite obvious but its very nice to have some hard data confirming it

Homeworks are useless, school bad, Einstein had bad grades (no he didn't) crowd is very silent right now.

Home work and learning have never been about getting the right answers

Given OPs axis definitions.

The max homework score for nonai is 115. But there are AI users scoring 115+? I'm confused why there are no better scores above 115 for nonai. Is that data insignificant? Are students with scores higher than 115 just being flagged as AI? It makes the data look unreliable.

I'd also really like to know if the data range that isn't comparable 115+ is at all meaningful data. Because that's where there is a major dropoff. There is obviously a dropoff of significant before 115. But, just from looking at how the data is being presented with this massive dropoff but with zero data for nonai.

I'd be interested in knowing what fraction of the total population is actually being represented in the 115+ dropoff. The way it's presented it could literally be like 10 students.

This is not a defense of AI use. But more a criticism of how the data is being presented.

Also, Any student could be using AI as a resource similar to how I would use solutions manuals or previous tests to study back in my education. The good students aren't gonna copy it verbatim and get flagged. Their going to get the right answer, and use it to learn the process so they can present their own work and actually learn the material.

AI is trash. But this is nothing different than before AI when students would copy the solutions manuals blindly or their peers work. Those students have always existed. AI just makes it slightly more accessible. But, really only slightly. And, honestly, Chegg was just as easy to copy back in the day.

Edit: This graph isn't in the paper that OP linked. So, probably why it's presented so badly. I'm assuming it's from some click bait article then. There is a similar graph in the paper that is much more clear.

But the paper clarifies that this is an actual survey of students. So it's not just from flagged AI homework. Which is good and bad in its own way.

The paper itself in section 5 supports what I hypothesized above. It's not really about AI making students dumb. It's about making it easier for students that already don't want to learn to finish the homework. Same as the people that would copy solutions manuals.

Quote from section 5.

In this section, we show that the majority of AI students exhibit behavior consistent with homework outsourcing, spending little time completing homework assignments and experiencing learning losses. The remaining AI students spend as much time completing homework as non-AI students, receive higher homework scores, and have similar exam scores. In other words, they are learning as efficiently as the non-AI students.

The paper doesn't argue it from what I read. But I'd also argue there is some bias in the survey from the questions themselves. Or at least how the paper lumps categories. The students scoring well in both exams and homework are less likely to consider their use of AI as meaningful enough to answer "used AI for homework". It's a problem with the survey. The survey is asking the students that actually learned the material to attribute their homework to AI alone in the same way the "copy and paste" AI users would.

The survey would probably benefit from having No AI, Some AI, All AI as it's responses. Even in anonymous surveys people make these own interpretations of the questions in their head. And a student that used AI as a resource to learn is very likely to just choose "No AI" when presented with questions of how they got their final answers to homework. Because in their head they are thinking of the students that copy and pasted AI responses and think "I'm not like that".

It's likely why the double high scoring AI user sample is so low (paper says this). The survey is not allowing the response for this set of users to categorize themselves as using AI without feeling like they are like the copy and paste students. So those people are likely just self categorizing as "no ai" because the survey doesn't allow them to distance themselves from the other population. Surveys are hard to write. Even ones where the users know it's anonymous. Good students that use AI will see themselves as good students and the copy pasters as bad students. Human emotion plays a role and most people will not self categorize themselves further negatively than they feel they should be. They are more likely to select the imperfect category that is more positive.

Bad students don't care though. They'll admit to AI use. They already don't care enough to learn the material. They have no reason to lie. So they'll select "used AI".

If you want a good survey you need very simple and minimal answer categories that the majority of the population will be able to self categorize themselves well. But, you don't want to minimize to such a degree that your survey introduces self categorizing bias. I would argue that's what happened here. The most important category is an extremely low sample size compared to the rest of the data.

And I'd also say it's likely the majority of people in reality. They just self categorized as "no ai" because the survey didn't allow them to categorize themselves more accurately without being associated with what they see as negative.

The entire paper seems to not address this strict categorization in its data that is likely introducing bad response data. But I'd have to look at the actual survey questions to know.

I think it's a good paper that suffers from an overly strict self categorizing bias in its survey data.

Damn, Chinese kids who don't use AI are already putting in more than 110%.

Why are boys more susceptible?

So... Below average kids should use AI? Is that what this says?