August 30, 2014

Full Text Paper: “Carbon Dating The Web: Estimating the Age of Web Resources”

share save 171 16 Full Text Paper: Carbon Dating The Web: Estimating the Age of Web Resources

The following paper will be presented at the TempWeb 2013 Workshop during the WWW 2013 Conference  that is scheduled to take place next month in in Rio de Janeiro, Brazil.

Title

Carbon Dating The Web: Estimating the Age of Web Resources

Authors

Hany M. SalahEldeen
Old Dominion University 

Michael L. Nelson
Old Dominion University 

Source

via @arXiv

Abstract

In the course of web research it is often necessary to estimate the creation datetime for web resources (in the general case, this value can only be estimated). While it is feasible to manually establish likely datetime values for small numbers of resources, this becomes infeasible if the collection is large. We present “carbon date”, a simple web application that estimates the creation date for a URI by polling a number of sources of evidence and returning a machine-readable structure with their respective values. To establish a likely datetime, we poll bitly for the first time someone shortened the URI, topsy for the first time someone tweeted the URI, a Memento aggregator for the first time it appeared in a public web archive, Google’s time of last crawl, and the Last-Modified HTTP response header of the resource itself. We also examine the backlinks of the URI as reported by Google and apply the same techniques for the resources that link to the URI. We evaluated our tool on a gold-standard data set of 1200 URIs in which the creation date was manually verified. We were able to estimate a creation date for 75.90% of the resources, with 32.78% having the correct value. Given the different nature of the URIs, the union of the various methods produces the best results. While the Google last crawl date and topsy account for nearly 66% of the closest answers, eliminating the web archives or Last-Modified from the results produces the largest overall negative impact on the results. The carbon date application is available for download or use via a webAPI.

Direct to Full Text Paper (14 pages; PDF via arXiv)

Direct to Dataset (CSV via arXiv)

Related Papers by the Same Authors

share save 171 16 Full Text Paper: Carbon Dating The Web: Estimating the Age of Web Resources
Gary Price About Gary Price

Gary Price (gprice@mediasourceinc.com) is a librarian, writer, consultant, and frequent conference speaker based in the Washington D.C. metro area. Before launching INFOdocket, Price and Shirl Kennedy were the founders and senior editors at ResourceShelf and DocuTicker for 10 years. From 2006-2009 he was Director of Online Information Services at Ask.com, and is currently a contributing editor at Search Engine Land.