University of North Carolina Library Launches DocSouth Data, Historic Texts Ready For Analysis

Cool from Chapel Hill. Kudos to the UNC Library on providing this new service. Superb idea.
From the UNC Library Blog:

The newly-released DocSouth Data makes the full text of hundreds of nineteenth-century books and pamphlets available for easy download as text-only files. The materials come from four text-heavy Documenting the American South collections: The Church in the Southern Black Community; First-Person Narratives of the American South; Library of Southern Literature; and North American Slave Narratives.
“Researchers who want to experiment with text analysis have plenty of tools to choose from, but they often hit a roadblock when it comes to finding collections that are ready to be analyzed,” says Stewart Varner, UNC’s digital scholarship librarian.
[Clip]
Varner and his colleagues at the UNC Library realized that DocSouth provides a rich source of ready-to-use full-text that is of high interest and high quality.
The North American Slave Narratives collection, for example, includes every known autobiographical narrative of fugitive or former slaves published in English up to 1920. Its titles are among the most frequently used in DocSouth. The collection offers a wide-ranging picture of the experiences of former slaves and African American life in antebellum America.
Moreover, these files were transcribed by hand for the original digital files. That means they are unusually accurate, free of the errors that optical character recognition (OCR) often produces.

Collections Available via DocSouth Data

• The Church in the Southern Black Community
• First-Person Narratives of the American South
• Library of Southern Literature
• North American Slave Narratives

From the DocSouth Data Web Page

Doc South Data is an extension of this original goal and has been designed for researchers who want to use emerging technology to look for patterns across entire texts or compare patterns found in multiple texts. We have made it easy to use tools such as Voyant to conduct simple word counts and frequency visualizations (such as word clouds) or to use other tools to perform more complex processes such as topic modeling, named-entity recognition or sentiment analysis.

Here’s What You’ll Find in a Data File (Once You Download and Unzip):

• A folder containing each of the texts in the collection as plain text;
• A folder containing each of the texts with complete TEI/XML markup;
• A Table of Contents (TOC) file which is a .csv file (which can be opened in Excel as a spread sheet);
• A file neamed “Read Me;”
• A file named “text-only.xsl.”

The plain text files can be used in text mining projects such as topic modeling, sentiment analysis and natural language processing.
The TEI/XML files have been included for advanced users who would like to isolate particular parts of text for analysis.
The .csv file acts as a table of contents for the collection and includes Title, Author, Publication Date and webaddresses for both the original Doc South page and a unique url pointing to a web accessible version of the text (this is particularly useful for use with Voyant).
The Read Me file provides documentation about the collection including information about and quirks or idiosyncrasies researchers will need to be aware of when they are working with the data.
The text-only.xsl file is the script that was used to create the folder.

Read the Complete Blog Post (with Text Analysis Graphic)
Direct to DocSouth Data Info and Downloads
Direct to the DocSouth Web Site (Browse, Search the Collections)

Filed under: Data Files, Libraries, News, Patrons and Users