Report: “New Algorithm Searches Historic Documents To Identify Noteworthy People”

From the University of Buffalo:

Old newspapers provide a window into our past, and a new algorithm co-developed by a School of Management researcher is helping turn those historic documents into useful, searchable data.

Published in Decision Support Systems, the algorithm can find and rank people’s names in order of importance from the results produced by optical character recognition (OCR), the computerized method of converting scanned documents into text that is often messy.

“It’s a known fact that when OCR software is run, very often the text gets garbled,” says Haimonti Dutta, assistant professor of management science and systems. “With old newspapers, books and magazines, problems can arise from poor ink quality, crumpled or torn paper, or even unusual page layouts the software isn’t expecting.”

To develop the algorithm, researchers partnered with the New York Public Library (NYPL) and analyzed more than 14,000 articles from New York City newspaper The Sun published during November and December 1894. The NYPL has scanned more than 200,000 newspaper pages as part of Chronicling America, an initiative of the National Endowment for the Humanities and the Library of Congress that is working to develop an online, searchable database of historical newspapers from 1777 to 1963.

[Clip]

Dutta says their process has wide-reaching implications for discovering important people throughout history.

“We recently used this technique on African American literature from the Civil War to learn more about the important people during the era of slavery,” Dutta says. “Going forward, we’ll be expanding the technique to examine relationships between people and build out the social networks of the past.”

Learn More, Read the Complete Article

Filed under: Data Files, Journal Articles, Libraries, Management and Leadership, News, Public Libraries