Indiana University, U. Texas at Austin, and NYPL Awarded $1.2 Million Grant to Develop Ability to Search Digitized Audiovisual Files

From Indiana University:

Current access to text-based data and knowledge is unprecedented in human history. Now, enabled by a $1.2 million grant from The Andrew W. Mellon Foundation, Indiana University Libraries is setting its sights on the next research frontier: hundreds of millions of hours of audiovisual content.
To unlock access to multitudes of unheard stories, IU Libraries will work extensively with project partners at University of Texas at Austin, New York Public Library and information management consultant AVP. Together the team will develop and test a scalable Audiovisual Metadata Platform, known as AMP, to generate searchable time-stamped descriptions for audiovisual content.
“Discovery opportunities for recorded audio or moving images are extremely limited today,” said Jon Dunn, principal investigator and assistant dean for library technologies at IU. “Our work will generate descriptions, or what is known as metadata, through a combination of expert human labor and new uses of automated processing technology.”
Online aggregations of library, museum and archival collections, such as the Digital Public Library of America, are primarily composed of images and text, while less than half a percent of available content is video or sound, Dunn and his partners said.
The barriers created by the time-consuming, expensive process of describing audio and video solely through manual means create a major impediment to discovery and use, even as nationwide preservation efforts exponentially expand available content. In fact, Indiana University is home to one of the country’s most ambitious digitization efforts, the Media and Digitization Preservation Initiative. Library leaders at IU and across the country are actively seeking ways to accelerate access to the hundreds of thousands of newly digitized resources.
“With any sort of mass digitization, the item does not speak for itself,” said Mary Kidd, coordinator of systems and operations in the Department of Preservation and Collections Processing at New York Public Library. In recent years, the library has preserved nearly 80,000 audiovisual items through digitization.

Traditional approaches to metadata generation for audiovisual objects must rely almost exclusively on labor by experts who type information into a database or create a written finding aid.
“The ultimate result of the AMP project will be a metadata-generation interface made openly available to collection managers around the world,” said Chris Lacinak, president and founder of AVP, the consulting and software development firm working with grant partners on AMP development.
Kidd sees great potential in a tool that supports human experts.
“What is so exciting about this project, and not reflected in work anywhere else, is that AMP uses a variety of automation tools that create a dynamic set of data we can then synthesize into our records,” she said.
The automation will include artificial intelligence and machine-learning-based functionality such as natural language processing, speech-to-text conversion, facial recognition, scene and music detection, object recognition and more. The AMP team plans to incorporate automated tools from both the research community and from commercial providers such as Amazon and Google.
Over the 27-month grant period, different forms of audiovisual content will be used to test and refine the technology. Examples of potential collection use cases include the Joffrey Ballet Archive at New York Public Library and documentation of field collections at IU Libraries’ Archives of Traditional Music.
The project kicks off by learning from prior efforts to automate audiovisual metadata though consultation with the Fraunhofer Institute for Digital Media Technology IDMT in Germany, as well as the foundational work of the High Performance Sound Technologies for Access and Scholarship project at the University of Texas, led by AMP partner Tanya Clement. Previous Mellon Foundation support for AMP instigated these connections through community development activities and documentation of workshops and assessment activities.
“From early in the 20th century and continuing through the present, audiovisual media contain vital and irreplaceable evidence that documents contemporary society and culture,” Mellon Foundation senior program officer Donald J. Waters said. “Future understanding of the 20th century and beyond requires an aggressive investment today in tools and methods for preserving and providing research access to these media.”

See Also: AMP-lifying IU’s Media Collections with the Audiovisual Metadata Platform (via IU)

Filed under: Archives and Special Collections, Data Files, Digital Preservation, Funding, Libraries, Management and Leadership, News, Preservation, Public Libraries, Reports