New From The Library Innovation Lab at Harvard Law School: “WARC-GPT: An Open-Source Tool for Exploring Web Archives Using AI”

From the Library Innovation Lab, Harvard Law School:

Today we’re releasing WARC-GPT: an open-source, highly-customizable Retrieval Augmented Generation [RAG] tool the web archiving community can use to explore the intersection between web archiving and AI. WARC-GPT allows for creating custom chatbots that use a set of web archive files as their knowledge base, letting users explore collections through conversation.

Using WARC-GPT, you can ask specific questions in natural language against a collection of WARC files. Rather than relying on keyword searches and metadata filters to sort through search results,

[Clip]

This project is part of our ongoing series exploring how artificial intelligence changes our relationship to knowledge. The release of this experimental software will help us understand whether and how AI can help access and uncover web archives’ contents. Of course, this is simply a prototype – see our disclaimer in the repo.

[Clip]

WARC-GPT allows users to ingest and translate a collection of WARC files into a RAG setup that can be used with a variety of LLMs – essentially allowing archivists and researchers to use a chatbot that has knowledge of their collections. WARC-GPT is especially helpful for exploring private collections of WARCs or those that were not part of the training data for an LLM. Although LLMs are typically trained on data from sources like Common Crawl, an open repository of web crawl data that consists of over 250 billion pages, it is not possible to verify what data has been included. Grounding the knowledge of an LLM with a collection of WARCs provides relevant contextual information that is especially helpful for specific or specialized set of queries within a particular domain.

Learn More, Read the Complete Post (3467 words)

Filed under: Archives and Special Collections, Data Files, Libraries, News, Open Access, Patrons and Users