SUBSCRIBE
SUBSCRIBE
EXPLORE +
  • About infoDOCKET
  • Academic Libraries on LJ
  • Research on LJ
  • News on LJ
  • Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Libraries
    • Academic Libraries
    • Government Libraries
    • National Libraries
    • Public Libraries
  • Companies (Publishers/Vendors)
    • EBSCO
    • Elsevier
    • Ex Libris
    • Frontiers
    • Gale
    • PLOS
    • Scholastic
  • New Resources
    • Dashboards
    • Data Files
    • Digital Collections
    • Digital Preservation
    • Interactive Tools
    • Maps
    • Other
    • Podcasts
    • Productivity
  • New Research
    • Conference Presentations
    • Journal Articles
    • Lecture
    • New Issue
    • Reports
  • Topics
    • Archives & Special Collections
    • Associations & Organizations
    • Awards
    • Funding
    • Interviews
    • Jobs
    • Management & Leadership
    • News
    • Patrons & Users
    • Preservation
    • Profiles
    • Publishing
    • Roundup
    • Scholarly Communications
      • Open Access

September 26, 2018 by Gary Price

Conference Paper: “The Off-Topic Memento Toolkit” (Web Archiving)

September 26, 2018 by Gary Price

The following paper was presented at the 15th International Conference on Digital Preservation (iPres 2018). 
Title
The Off-Topic Memento Toolkit
Authors
Shawn M. Jones
Old Dominion University
Michele C. Weigle
Old Dominion University
Michael L. Nelson
Old Dominion University
Source
via arXIv
Abstract

Web archive collections are created with a particular purpose in mind. A curator selects seeds, or original resources, which are then captured by an archiving system and stored as archived web pages, or mementos. The systems that build web archive collections are often configured to revisit the same original resource multiple times. This is incredibly useful for understanding an unfolding news story or the evolution of an organization. Unfortunately, over time, some of these original resources can go off-topic and no longer suit the purpose for which the collection was originally created. They can go off-topic due to web site redesigns, changes in domain ownership, financial issues, hacking, technical problems, or because their content has moved on from the original topic. Even though they are off-topic, the archiving system will still capture them, thus it becomes imperative to anyone performing research on these collections to identify these off-topic mementos.
Hence, we present the Off-Topic Memento Toolkit, which allows users to detect off-topic mementos within web archive collections. The mementos identified by this toolkit can then be separately removed from a collection or merely excluded from downstream analysis. The following similarity measures are available: byte count, word count, cosine similarity, Jaccard distance, S{\o}rensen-Dice distance, Simhash using raw text content, Simhash using term frequency, and Latent Semantic Indexing via the gensim library. We document the implementation of each of these similarity measures. We possess a gold standard dataset generated by manual analysis, which contains both off-topic and on-topic mementos. Using this gold standard dataset, we establish a default threshold corresponding to the best F1 score for each measure. We also provide an overview of potential future directions that the toolkit may take.

Direct to Full Text
10 pages; PDF.

The Off-Topic Memento Toolkit from Shawn Jones

Filed under: Data Files, Digital Preservation, Journal Articles, Libraries, News, Patrons and Users, Preservation

SHARE:

About Gary Price

Gary Price (gprice@gmail.com) is a librarian, writer, consultant, and frequent conference speaker based in the Washington D.C. metro area. He earned his MLIS degree from Wayne State University in Detroit. Price has won several awards including the SLA Innovations in Technology Award and Alumnus of the Year from the Wayne St. University Library and Information Science Program. From 2006-2009 he was Director of Online Information Services at Ask.com. Gary is also the co-founder of infoDJ an innovation research consultancy supporting corporate product and business model teams with just-in-time fact and insight finding.

ADVERTISEMENT

Archives

Job Zone

ADVERTISEMENT

Related Infodocket Posts

Collections: JSTOR is Introducing a New Archive Fee Model Option

From a Letter by Ithaka President Kevin Guthrie: I recently shared the 2023 priorities ITHAKA has set to help provide the infrastructure the academic community needs to support research, teaching, and learning ...

Funding: HathiTrust Receives 5-Year, $1 Million Grant From Mellon Foundation

Here’s the Full Text of HathiTrust Announcement: HathiTrust, a member-based organization hosted by the University of Michigan, has received a 5-year, $1 million grant from the Mellon Foundation to fund ...

Library as Publisher: "UMass Amherst Libraries Announce Publication of Open-Access Peregrine Falcon Curriculum

From UMass Amherst Libraries (Full Text): The University of Massachusetts Amherst Libraries are pleased to announce the publication of The UMass Amherst Libraries Falcon Curriculum: An Open Source, Common Core PreK-12 ...

Report From Annenberg/UPenn: "Americans Don’t Understand What Companies Can Do With Their Personal Data — and That’s a...

From the Annenberg School of Communications/U. of Pennsylvania: In a new report, “Americans Can’t Consent to Companies’ Use of Their Data,” researchers asked a nationally representative group of more than ...

Fast Company: "Study: Over 50% of Academics Admit to Pirating Research Papers"

From Fast Company: More than 50% of academics have used piracy websites like Sci-Hub in order to bypass paywalls for research they want to access, according to a recent study published in ...

AI Models Spit Out Photos of Real People and Copyrighted Images; New Web Archives from Columbia University Libraries...

AI Models Spit Out Photos of Real People and Copyrighted Images (via MIT Technology Review) California: Orange Unified School District Reinstates Digital Library After Parent Concerns (via Voice of OC) ...

Journal Article: "Libraries Advancing Health Equity: A Literature Review"

The article linked below (full-text) was recently published Reference Services Review. Title Libraries Advancing Health Equity: A Literature Review Authors Amanda J. Wilson National Library of Medicine Catherine Staley National ...

Chicago Sun-Times Editorial Board: "As Libraries Turn the Page on Bookmobiles, Something is Lost"

From the Chicago Sun-Times Editorial Board: Anyone who has spent time on a bookmobile has learned enough to know nothing withstands the change of time. Still, we lament the slow ...

LC's African and Middle Eastern Division Announces Release of the Africana Historic Postcard Collection

From The Library of Congress (via a 4 Corners of the World Blog Post by Anchi Hoh): The African and Middle Eastern Division is delighted to announce the rerelease of the ...

New From IFLA: "Marrakesh Monitoring Report - February 2023 Update"

From the International Federation of Library Associations and Institutions (IFLA): The chart [monitoring report]…is an updated version of previous monitoring reports. Where a country has been updated or added since ...

ROUNDUP: Research4Life Reaches 200,000 Resources; Majority of Research Papers Published by Cambridge University Press Now Open Access; &...

Association of Research Libraries (ARL) Seeks to Hire Director of Diversity, Equity, and Inclusion Developing a Globally Fair Pricing Model for Open Access Academic Publishing (via cOAlition S) Majority of ...

NY Times: "Turning Nairobi’s Public Libraries Into 'Palaces for the People'"

From The NY Times: In 1931, the first library in Kenya’s capital, Nairobi, opened its doors — to white patrons only. Nearly a century later, Kenyans dressed in the slinky ...

ADVERTISEMENT

FOLLOW US ON TWITTER

Tweets by infoDOCKET

ADVERTISEMENT

This coverage is free for all visitors. Your support makes this possible.

This coverage is free for all visitors. Your support makes this possible.

Primary Sidebar

  • News
  • Reviews+
  • Technology
  • Programs+
  • Design
  • Leadership
  • People
  • COVID-19
  • Advocacy
  • Opinion
  • INFOdocket
  • Job Zone

Reviews+

  • Booklists
  • Prepub Alert
  • Book Pulse
  • Media
  • Readers' Advisory
  • Self-Published Books
  • Review Submissions
  • Review for LJ

Awards

  • Library of the Year
  • Librarian of the Year
  • Movers & Shakers 2022
  • Paralibrarian of the Year
  • Best Small Library
  • Marketer of the Year
  • All Awards Guidelines
  • Community Impact Prize

Resources

  • LJ Index/Star Libraries
  • Research
  • White Papers / Case Studies

Events & PD

  • Online Courses
  • In-Person Events
  • Virtual Events
  • Webcasts
  • About Us
  • Contact Us
  • Advertise
  • Subscribe
  • Media Inquiries
  • Newsletter Sign Up
  • Submit Features/News
  • Data Privacy
  • Terms of Use
  • Terms of Sale
  • FAQs
  • Careers at MSI


© 2023 Library Journal. All rights reserved.


© 2022 Library Journal. All rights reserved.