SUBSCRIBE
SUBSCRIBE
EXPLORE +
  • About infoDOCKET
  • Academic Libraries on LJ
  • Research on LJ
  • News on LJ
  • Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Libraries
    • Academic Libraries
    • Government Libraries
    • National Libraries
    • Public Libraries
  • Companies (Publishers/Vendors)
    • EBSCO
    • Elsevier
    • Ex Libris
    • Frontiers
    • Gale
    • PLOS
    • Scholastic
  • New Resources
    • Dashboards
    • Data Files
    • Digital Collections
    • Digital Preservation
    • Interactive Tools
    • Maps
    • Other
    • Podcasts
    • Productivity
  • New Research
    • Conference Presentations
    • Journal Articles
    • Lecture
    • New Issue
    • Reports
  • Topics
    • Archives & Special Collections
    • Associations & Organizations
    • Awards
    • Funding
    • Interviews
    • Jobs
    • Management & Leadership
    • News
    • Patrons & Users
    • Preservation
    • Profiles
    • Publishing
    • Roundup
    • Scholarly Communications
      • Open Access

September 26, 2018 by Gary Price

Conference Paper: “The Off-Topic Memento Toolkit” (Web Archiving)

September 26, 2018 by Gary Price

The following paper was presented at the 15th International Conference on Digital Preservation (iPres 2018). 
Title
The Off-Topic Memento Toolkit
Authors
Shawn M. Jones
Old Dominion University
Michele C. Weigle
Old Dominion University
Michael L. Nelson
Old Dominion University
Source
via arXIv
Abstract

Web archive collections are created with a particular purpose in mind. A curator selects seeds, or original resources, which are then captured by an archiving system and stored as archived web pages, or mementos. The systems that build web archive collections are often configured to revisit the same original resource multiple times. This is incredibly useful for understanding an unfolding news story or the evolution of an organization. Unfortunately, over time, some of these original resources can go off-topic and no longer suit the purpose for which the collection was originally created. They can go off-topic due to web site redesigns, changes in domain ownership, financial issues, hacking, technical problems, or because their content has moved on from the original topic. Even though they are off-topic, the archiving system will still capture them, thus it becomes imperative to anyone performing research on these collections to identify these off-topic mementos.
Hence, we present the Off-Topic Memento Toolkit, which allows users to detect off-topic mementos within web archive collections. The mementos identified by this toolkit can then be separately removed from a collection or merely excluded from downstream analysis. The following similarity measures are available: byte count, word count, cosine similarity, Jaccard distance, S{\o}rensen-Dice distance, Simhash using raw text content, Simhash using term frequency, and Latent Semantic Indexing via the gensim library. We document the implementation of each of these similarity measures. We possess a gold standard dataset generated by manual analysis, which contains both off-topic and on-topic mementos. Using this gold standard dataset, we establish a default threshold corresponding to the best F1 score for each measure. We also provide an overview of potential future directions that the toolkit may take.

Direct to Full Text
10 pages; PDF.

The Off-Topic Memento Toolkit from Shawn Jones

Filed under: Data Files, Digital Preservation, Journal Articles, Libraries, News, Patrons and Users, Preservation

SHARE:

About Gary Price

Gary Price (gprice@gmail.com) is a librarian, writer, consultant, and frequent conference speaker based in the Washington D.C. metro area. He earned his MLIS degree from Wayne State University in Detroit. Price has won several awards including the SLA Innovations in Technology Award and Alumnus of the Year from the Wayne St. University Library and Information Science Program. From 2006-2009 he was Director of Online Information Services at Ask.com.

ADVERTISEMENT

Archives

Job Zone

ADVERTISEMENT

Related Infodocket Posts

South Carolina State Library Leaves American Library Association (ALA), Does Not Renew Membership

From the Charleston City Paper: Librarians are learning the S.C. State Library in August quietly notified the national trade association for libraries that the state was not renewing its membership ...

Former Campbell County, Wyoming Library Director Terri Lesley Files Workplace Discrimination Complaint, Defamation Lawsuit

From the Gillette News Record: Two months after she was fired and nearly two years after a criminal complaint was filed against her, former library director Terri Lesley is taking ...

SAGE Releases a New Free-to-Read Collection with Research into Academic Freedom and Censorship 

From a SAGE News Release: Sage has launched a new collection of free-to-read research highlighting the effects of academic censorship on democracy, social-emotional learning, higher education, and more. Categories in ...

New Online Resource: University of Maryland Libraries Announces Online Launch of the "Advancing Workers' Rights” Digital Collection

Here’s the Full Text of Today’s Annoucement From the University of Maryland Libraries: The University of Maryland Libraries announces the debut of a significant, newly digitized collection, making available online for the ...

In Affiliation with Arizona State University, "President Biden to Create Library Honoring His Friend and Rival John McCain"

From The New York Times: President Biden plans to announce on Thursday that he will devote federal money to create a new library and museum dedicated to his old friend ...

Boston Public Library Joins Books Unbanned Initiative to Fight Censorship

Here’s the Full Text of Today’s Boston Public Library Announcement: The Boston Public Library (BPL) is joining the Brooklyn Public Library’s Books Unbanned initiative to fight censorship and book banning by offering teens and ...

Merriam-Webster Adds Adds 690 New Words to the Dictionary (September 2023 Update)

From the M-W Website: Signs of a healthy language include words being created, words being borrowed from other languages, and new meanings being given to existing words. Based on our ...

CT: Local Libraries Investigating Book Thefts After Titles Go Missing; Brooklyn Public Library’s Leigh Hurwitz on Helping Young...

Brooklyn Public Library Brooklyn Public Library’s Leigh Hurwitz on Helping Young People Resist Censorship (via LitHub) Connecticut Local Libraries Investigating Book Thefts After Titles Go Missing (via NBC Connecticut) Copyright ...

AI/GPT News Roundup: Items From ACM, JISC, OpenAI, Meta, and Others

ACM Tech Brief: Generative Artificial Intelligence AI Now Computational Power and AI (Report) Anthropic Amazon is Investing Up to $4 Billion in AI Startup Anthropic in Growing Tech Battle (via ...

EveryLibrary Institute and BookRiot Releases Findings From "Public Libraries and Book Bans - Parent Perception Survey"

From an EveryLibrary Release: The “Public Libraries and Book Bans – Parent Perception Survey” gathered insights from 853 parents and guardians with children under 18 during September 2023. The survey ...

Idaho: "Sheriff Scopes Out Explicit Library Books"

From The Coeur d’Alene Press: “Kootenai County Sheriff Bob Norris said he has heard from both sides about reportedly inappropriate materials available to youth at local libraries. One side argued ...

MIT Libraries Receives Grant From National Science Foundation to Explore Open Science Evaluations with ICOR; Clarivate Establishes Academia...

AI Generative AI Has Disrupted Education. Here’s How It Can Be Used For Good – UNESCO (via WEF) University of Leeds Research Report on Potentials for AI in Libraries (via ...

ADVERTISEMENT

FOLLOW US ON TWITTER

Tweets by infoDOCKET

ADVERTISEMENT

This coverage is free for all visitors. Your support makes this possible.

This coverage is free for all visitors. Your support makes this possible.

Primary Sidebar

  • News
  • Reviews+
  • Technology
  • Programs+
  • Design
  • Leadership
  • People
  • COVID-19
  • Advocacy
  • Opinion
  • INFOdocket
  • Job Zone

Reviews+

  • Booklists
  • Prepub Alert
  • Book Pulse
  • Media
  • Readers' Advisory
  • Self-Published Books
  • Review Submissions
  • Review for LJ

Awards

  • Library of the Year
  • Librarian of the Year
  • Movers & Shakers 2022
  • Paralibrarian of the Year
  • Best Small Library
  • Marketer of the Year
  • All Awards Guidelines
  • Community Impact Prize

Resources

  • LJ Index/Star Libraries
  • Research
  • White Papers / Case Studies

Events & PD

  • Online Courses
  • In-Person Events
  • Virtual Events
  • Webcasts
  • About Us
  • Contact Us
  • Advertise
  • Subscribe
  • Media Inquiries
  • Newsletter Sign Up
  • Submit Features/News
  • Data Privacy
  • Terms of Use
  • Terms of Sale
  • FAQs
  • Careers at MSI


© 2023 Library Journal. All rights reserved.


© 2022 Library Journal. All rights reserved.