This study is methodologically poor: only 18 people, SAT topics (so broad and pretty poor with the expectation of an American style “essay”), only 20 minutes of writing so far too little time to properly use the tool given to explore (be it search engine or LLM).
With only 20 minutes, I’m not even trying to do a search. No surprise the people using LLM have zero recollection of what they wrote.
Plus they spend ages discussing correct quoting (why?) and statistical analysis via NLP which is entirely useless.
Very little space is dedicated to knowing if the essays are actually any good.
Quoting is actually extremely important. There's a big difference between making a certain claim a) because [1] performed an experiment that confirms it and [2] and [3] reproduced it and b) because the magic machine told me so.
This is still true whether or not the claim is true/accurate or not, as it allows for actual relevant and constructive critique of the work.
It’s about free form essay redaction in 20 minutes and the article claims to be about cognitive impacts. Exact quoting is approximately useless in this context. It’s not about experimental results. It’s about whether or not someone can quote verbatim from a piece of literature.
With only 20 minutes, I’m not even trying to do a search. No surprise the people using LLM have zero recollection of what they wrote.
Plus they spend ages discussing correct quoting (why?) and statistical analysis via NLP which is entirely useless.
Very little space is dedicated to knowing if the essays are actually any good.
Overall pretty disappointing.