Research Article: “Scraping SERP (Search Engine Results Pages) for Archival Seeds: It Matters When You Start”

The following article is an extended version of a paper that will be presented at the upcoming ACM/IEEE Joint Conference on Digital Libraries (JCDL 2018).
Title
Scraping SERP (Search Engine Results Pages) for Archival Seeds: It Matters When You Start
Authors
Alexander C. Nwala
Old Dominion University
Michele C. Weigle
Old Dominion University
Michael L. Nelson
Old Dominion University
Source
via arXiv
May 25, 2018
Abstract

Event-based collections are often started with a web search, but the search results you find on Day 1 may not be the same as those you find on Day 7. In this paper¹, we consider collections that originate from extracting URIs (Uniform Resource Identifiers) from Search Engine Result Pages (SERPs).
Specifically, we seek to provide insight about the retrievability of URIs of news stories found on Google, and to answer two main questions: first, can one “refind” the same URI of a news story (for the same query) from Google after a given time? Second, what is the probability of finding a story on Google over a given period of time? To answer these questions, we issued seven queries to Google every day for over seven months (2017-05-25 to 2018-01-12) and collected links from the first five SERPs to generate seven collections for each query. The queries represent public interest stories: “healthcare bill,” “manchester bombing,” “london terrorism,” “trump russia,” “travel ban,” “hurricane harvey,” and “hurricane irma.” We tracked each URI in all collections over time to estimate the discoverability of URIs from the first five SERPs. Our results showed that the daily average rate at which stories were replaced on the default Google SERP ranged from 0.21 – 0.54, and a weekly rate of 0.39 – 0.79, suggesting the fast replacement of older stories by newer stories. The probability of finding the same URI of a news story after one day from the initial appearance on the SERP ranged from 0.34 – 0.44. After a week, the probability of finding the same news stories diminishes rapidly to 0.01 – 0.11. In addition to the reporting of these probabilities, we also provide two predictive models for estimating the probability of finding the URI of an arbitrary news story on SERPs as a function of time.
The web archiving community considers link rot and content drift important reasons for collection building. Similarly, our findings suggest that due to the difficulty in retrieving the URIs of news stories from Google, collection building that originates from search engines should begin as soon as possible in order to capture the first stages of events, and should persist in order to capture the evolution of the events, because it becomes more difficult to find the same news stories with the same queries on Google, as time progresses.

Direct to Full Text Article
13 pages; PDF.

Filed under: Journal Articles, Libraries, News