Journal Article: “What Undergraduate Students Need to Know and Actually Know About Generative AI”

The article linked below (full text) was published today by Computers and Education: Artificial Intelligence.

Title

What Undergraduate Students Need to Know and Actually Know About Generative AI

Authors

Sina Rismanchian
University of California, Irvine

Eesha Tur Razia Babar
University of California, Irvine

Shayan Doroudi|
University of California, Irvine

Source

Computers and Education: Artificial Intelligence
DOI: 10.1016/j.caeai.2026.100554

Abstract

In November 2022, the release of ChatGPT sparked widespread adoption of generative AI chatbots among students, yet little is known about what undergraduate students actually understand about these tools or how accurately they perceive their capabilities. To address this gap, we propose a theoretically grounded framework for Generative AI (GenAI) literacy that integrates three forms of conceptual knowledge about large language models (LLMs)—foundations, capabilities, and limitations, and societal impact—with students’ perceptions and folk theories of AI chatbots. We then developed and validated a GenAI literacy survey, including multiple-choice knowledge items and perception items, using expert review and item response theory (IRT) modeling. To examine GenAI literacy across different educational contexts, we conducted two complementary studies: Study 1 surveyed students enrolled in courses at a large public R1 university, and Study 2 surveyed a national sample of U.S. undergraduate students recruited via Prolific. Across both samples, approximately 60% of students reported using AI chatbots weekly or daily. However, many students overestimated chatbots’ capabilities, particularly on tasks involving reasoning and counting, and often anthropomorphized or treated chatbots as search engines. Knowledge scores were higher among computer science students at the R1 university and frequent chatbot users, while perception accuracy varied by group. Critically, greater conceptual knowledge was associated with less overestimation of chatbot abilities, suggesting that knowledge supports more calibrated and responsible use. This work introduces a validated framework and instrument for assessing GenAI literacy and highlights the need for AI literacy initiatives that move beyond tool usage to address misconceptions, beliefs, and responsible engagement with generative AI in higher education.

Direct to Full Text Article

Filed under: News, Patrons and Users