A machine-learning tool can easily spot when chemistry papers are written using the chatbot ChatGPT, according to a study published on 6 November in Cell Reports Physical Science. The specialized classifier, which outperformed two existing artificial intelligence (AI) detectors, could help academic publishers to identify papers created by AI text generators.
“Most of the field of text analysis wants a really general detector that will work on anything,” says co-author Heather Desaire, a chemist at the University of Kansas in Lawrence. But by making a tool that focuses on a particular type of paper, “we were really going after accuracy”.
The findings suggest that efforts to develop AI detectors could be boosted by tailoring software to specific types of writing, Desaire says. “If you can build something quickly and easily, then it’s not that hard to build something for different domains.”
Large language models like ChatGPT can generate authentic-seeming text at lightning speed, but many journal publishers reject language models as authors on manuscripts. Thus, a means to accurately distinguish human-generated from artificial intelligence (AI)-generated text is immediately needed. We recently developed an accurate AI text detector for scientific journals and, herein, test its ability in a variety of challenging situations, including on human text from a wide variety of chemistry journals, on AI text from the most advanced publicly available language model (GPT-4), and, most important, on AI text generated using prompts designed to obfuscate AI use. In all cases, AI and human text was assigned with high accuracy. ChatGPT-generated text can be readily detected in chemistry journals; this advance is a fundamental prerequisite for understanding how automated text generation will impact scientific publishing from now into the future.
Gary Price (gprice@gmail.com) is a librarian, writer, consultant, and frequent conference speaker based in the Washington D.C. metro area.
He earned his MLIS degree from Wayne State University in Detroit.
Price has won several awards including the SLA Innovations in Technology Award and Alumnus of the Year from the Wayne St. University Library and Information Science Program. From 2006-2009 he was Director of Online Information Services at Ask.com.
