Preprint: “Web Archives Metadata Generation with GPT-4o: Challenges and Insights”

The preprint linked below was recently posted on arXiv.

Title

Web Archives Metadata Generation with GPT-4o: Challenges and Insights

Authors

Abigail Yongping Huang
National Library Board Singapore

Ashwin Nair
National Library Board Singapore

Zhen Rong Goh
Nanyang Technological University Singapore

Tianrui Liu
Nanyang Technological University Singapore

Source

via arXiv

DOI: 10.48550/arXiv.2411.05409

Abstract

Current metadata creation for web archives is time consuming and costly due to reliance on human effort. This paper explores the use of gpt-4o for metadata generation within the Web Archive Singapore, focusing on scalability, efficiency, and cost effectiveness. We processed 112 Web ARChive (WARC) files using data reduction techniques, achieving a notable 99.9% reduction in metadata generation costs. By prompt engineering, we generated titles and abstracts, which were evaluated both intrinsically using Levenshtein Distance and BERTScore, and extrinsically with human cataloguers using McNemar’s test. Results indicate that while our method offers significant cost savings and efficiency gains, human curated metadata maintains an edge in quality. The study identifies key challenges including content inaccuracies, hallucinations, and translation issues, suggesting that Large Language Models (LLMs) should serve as complements rather than replacements for human cataloguers. Future work will focus on refining prompts, improving content filtering, and addressing privacy concerns through experimentation wi

th smaller models. This research advances the integration of LLMs in web archiving, offering valuable insights into their current capabilities and outlining directions for future enhancements. The code is available at URL for further development and use by institutions facing similar challenges.

Direct to Full Text Article (preprint)
14 pages; PDF.

Filed under: Archives and Special Collections, Data Files, Journal Articles, Libraries, National Libraries, News