February 28, 2021

Conference Paper: “Wikipedia Text Reuse: Within and Without” (Preprint)

The following paper was recently posted on arXiv and has been accepted at ECIR 2019 (41st European Conference on Information Retrieval).


Wikipedia Text Reuse: Within and Without


Milad Alshomary
Paderborn University

Michael Völske
Bauhaus-Universität Weimar

Tristan Licht
Bauhaus-Universität Weimar

Henning Wachsmuth
Paderborn University

Benno Stein
Bauhaus-Universität Weimar

Matthias Hagen
Martin-Luther-Universität Halle-Wittenberg

Martin Potthast
Leipzig University


via arXiv


We study text reuse related to Wikipedia at scale by compiling the first corpus of text reuse cases within Wikipedia as well as without (i.e., reuse of Wikipedia text in a sample of the Common Crawl). To discover reuse beyond verbatim copy and paste, we employ state-of-the-art text reuse detection technology, scaling it for the first time to process the entire Wikipedia as part of a distributed retrieval pipeline. We further report on a pilot analysis of the 100 million reuse cases inside, and the 1.6 million reuse cases outside Wikipedia that we discovered. Text reuse inside Wikipedia gives rise to new tasks such as article template induction, fixing quality flaws due to inconsistencies arising from asynchronous editing of reused passages, or complementing Wikipedia’s ontology. Text reuse outside Wikipedia yields a tangible metric for the emerging field of quantifying Wikipedia’s influence on the web. To foster future research into these tasks, and for reproducibility’s sake, the Wikipedia text reuse corpus and the retrieval pipeline are made freely available.

Direct to Full Tex Article

About Gary Price

Gary Price (gprice@mediasourceinc.com) is a librarian, writer, consultant, and frequent conference speaker based in the Washington D.C. metro area. Before launching INFOdocket, Price and Shirl Kennedy were the founders and senior editors at ResourceShelf and DocuTicker for 10 years. From 2006-2009 he was Director of Online Information Services at Ask.com, and is currently a contributing editor at Search Engine Land.