Report: “New Algorithm Searches Historic Documents To Identify Noteworthy People”
From the University of Buffalo:
Old newspapers provide a window into our past, and a new algorithm co-developed by a School of Management researcher is helping turn those historic documents into useful, searchable data.
Published in Decision Support Systems, the algorithm can find and rank people’s names in order of importance from the results produced by optical character recognition (OCR), the computerized method of converting scanned documents into text that is often messy.
“It’s a known fact that when OCR software is run, very often the text gets garbled,” says Haimonti Dutta, assistant professor of management science and systems. “With old newspapers, books and magazines, problems can arise from poor ink quality, crumpled or torn paper, or even unusual page layouts the software isn’t expecting.”
To develop the algorithm, researchers partnered with the New York Public Library (NYPL) and analyzed more than 14,000 articles from New York City newspaper The Sun published during November and December 1894. The NYPL has scanned more than 200,000 newspaper pages as part of Chronicling America, an initiative of the National Endowment for the Humanities and the Library of Congress that is working to develop an online, searchable database of historical newspapers from 1777 to 1963.
Dutta says their process has wide-reaching implications for discovering important people throughout history.
“We recently used this technique on African American literature from the Civil War to learn more about the important people during the era of slavery,” Dutta says. “Going forward, we’ll be expanding the technique to examine relationships between people and build out the social networks of the past.”
About Gary Price
Gary Price (firstname.lastname@example.org) is a librarian, writer, consultant, and frequent conference speaker based in the Washington D.C. metro area. He earned his MLIS degree from Wayne State University in Detroit. Price has won several awards including the SLA Innovations in Technology Award and Alumnus of the Year from the Wayne St. University Library and Information Science Program. From 2006-2009 he was Director of Online Information Services at Ask.com. Gary is also the co-founder of infoDJ an innovation research consultancy supporting corporate product and business model teams with just-in-time fact and insight finding.