Research Paper (preprint): Let’s Use ChatGPT to Write Our Paper! Benchmarking LLMs to Write the Introduction of a Research Paper

The research paper (preprint) linked below was recently posted on arXiv.

Title

Let’s Use ChatGPT to Write Our Paper! Benchmarking LLMs to Write the Introduction of a Research Paper

Authors

Krishna Garg
University of Illinois Chicago

Firoz Shaikh
University of Illinois Chicago

Sambaran Bandyopadhyay
Adobe Research

Cornelia Caragea
University of Illinois Chicago

Source

via arXiv

DOI: 10.48550/arXiv.2508.14273

Abstract

As researchers increasingly adopt LLMs as writing assistants, generating high-quality research paper introductions remains both challenging and essential. We introduce Scientific Introduction Generation (SciIG), a task that evaluates LLMs’ ability to produce coherent introductions from titles, abstracts, and related works. Curating new datasets from NAACL 2025 and ICLR 2025 papers, we assess five state-of-the-art models, including both open-source (DeepSeek-v3, Gemma-3-12B, LLaMA 4-Maverick, MistralAI Small 3.1) and closed-source GPT-4o systems, across multiple dimensions: lexical overlap, semantic similarity, content coverage, faithfulness, consistency, citation correctness, and narrative quality. Our comprehensive framework combines automated metrics with LLM-as-a-judge evaluations. Results demonstrate LLaMA-4 Maverick’s superior performance on most metrics, particularly in semantic similarity and faithfulness. Moreover, three-shot prompting consistently outperforms fewer-shot approaches. These findings provide practical insights into developing effective research writing assistants and set realistic expectations for LLM-assisted academic writing. To foster reproducibility and future research, we will publicly release all code and datasets.

Direct to Abstract & Full Text

Filed under: Data Files, Journal Articles, News