Research Article (preprint): “Journal Quality Factors From ChatGPT: More Meaningful Than Impact Factors?

The research article linked below (preprint) was recently shared on arXiv.

Title

Journal Quality Factors From ChatGPT: More Meaningful Than Impact Factors

Authors

Source

via arXiv

DOI: 10.48550/arXiv.2411.09984

Abstract

Purpose: Journal Impact Factors and other citation-based indicators are widely used and abused to help select journals to publish in or to estimate the value of a published article. Nevertheless, citation rates primarily reflect scholarly impact rather than other quality dimensions, including societal impact, originality, and rigour. In response to this deficit, Journal Quality Factors (JQFs) are defined and evaluated. These are average quality score estimates given to a journal’s articles by ChatGPT.

Design/methodology/approach: JQFs were compared with Polish, Norwegian and Finnish journal ranks and with journal citation rates for 1,300 journals with 130,000 articlesfrom 2021 in large monodisciplinary journals in the 25 out of 27 Scopus broad fields of research for which it was possible. Outliers were also examined.

Findings: JQFs correlated positively and mostly strongly (median correlation: 0.641) with journal ranks in 24 out of the 25 broad fields examined, indicating a nearly science-wide ability for ChatGPT to estimate journal quality. Journal citation rates had similarly high correlations with national journal ranks, however, so JQFs are not a universally better indicator. An examination of journals with JQFs not matching their journal ranks suggested that abstract styles may affect the result, such as whether the societal contexts of research are mentioned.

Research limitations: Different journal rankings may have given different findings because there is no agreed meaning for journal quality. Practical implications: The results suggest that JQFs are plausible as journal quality indicators in all fields and may be useful for the (few) research and evaluation contexts where journal quality is an acceptable proxy for article quality, and especially for fields like mathematics for which citations are not strong indicators of quality.

Originality/value: This is the first attempt to estimate academic journal value with a Large Language Model.

Direct to Full Text Article
12 pages; PDF.

Filed under: News