Full Text Paper: “Carbon Dating The Web: Estimating the Age of Web Resources”

The following paper will be presented at the TempWeb 2013 Workshop during the WWW 2013 Conference  that is scheduled to take place next month in in Rio de Janeiro, Brazil.

Title

Carbon Dating The Web: Estimating the Age of Web Resources

Authors

Hany M. SalahEldeen
Old Dominion University 
Michael L. Nelson
Old Dominion University 

Source

via @arXiv

Abstract

In the course of web research it is often necessary to estimate the creation datetime for web resources (in the general case, this value can only be estimated). While it is feasible to manually establish likely datetime values for small numbers of resources, this becomes infeasible if the collection is large. We present “carbon date”, a simple web application that estimates the creation date for a URI by polling a number of sources of evidence and returning a machine-readable structure with their respective values. To establish a likely datetime, we poll bitly for the first time someone shortened the URI, topsy for the first time someone tweeted the URI, a Memento aggregator for the first time it appeared in a public web archive, Google’s time of last crawl, and the Last-Modified HTTP response header of the resource itself. We also examine the backlinks of the URI as reported by Google and apply the same techniques for the resources that link to the URI. We evaluated our tool on a gold-standard data set of 1200 URIs in which the creation date was manually verified. We were able to estimate a creation date for 75.90% of the resources, with 32.78% having the correct value. Given the different nature of the URIs, the union of the various methods produces the best results. While the Google last crawl date and topsy account for nearly 66% of the closest answers, eliminating the web archives or Last-Modified from the results produces the largest overall negative impact on the results. The carbon date application is available for download or use via a webAPI.

Direct to Full Text Paper (14 pages; PDF via arXiv)
Direct to Dataset (CSV via arXiv)

Related Papers by the Same Authors

• How Much of the Web Is Archived? (10 pages; PDF via arXiv)

• Losing My Revolution: How Many Resources Shared on Social Media Have Been Lost? (12 pages; PDF  via arXiv)

• A Plan For Curating “Obsolete Data or Resources” (4 pages; PDF via arXiv)

• Using Web Page Titles to Rediscover Lost Web Page (pages; PDF via arXiv)

Filed under: Archives and Special Collections, Data Files, Journal Articles, News