Stanford University researchers issued a report on Wednesday measuring the transparency of artificial intelligence foundation models from companies like OpenAI and Google and the authors urged the companies to reveal more information such as the data and human labor used to train models.
[Clip]
The index graded 10 popular models on 100 different transparency indicators, such as training data and how much compute was used. All models scored “unimpressively”: even the most transparent model, Meta’s(META.O) Llama 2, received a score of 53 out of 100. Amazon’s Titan model ranked the lowest, at 11 out of 100. OpenAI’s GPT-4 model received a score of 47 out of 100.
The system, known as the Foundation Model Transparency Index, rates 10 large A.I. language models — sometimes called “foundation models” — on how transparent they are. Included in the index are popular models like OpenAI’s GPT-4 (which powers the paid version of ChatGPT), Google’s PaLM 2 (which powers Bard) and Meta’s LLaMA 2. It also includes lesser-known models like Amazon’s Titan and Inflection AI’s Inflection-1, the model that powers the Pi chatbot.
[Clip]
To come up with the rankings, researchers evaluated each model on 100 criteria, including whether its maker disclosed the sources of its training data, information about the hardware it used, the labor involved in training it and other details. The rankings also
include information about the labor and data used to produce the model itself, along with what the researchers call “downstream indicators,” which have to do with how a model is used after it’s released. (For example, one question asked is: “Does the developer disclose its protocols for storing, accessing and sharing user data?”)
Source: NY Times via Center for Research on Foundation Models
Gary Price (gprice@gmail.com) is a librarian, writer, consultant, and frequent conference speaker based in the Washington D.C. metro area.
He earned his MLIS degree from Wayne State University in Detroit.
Price has won several awards including the SLA Innovations in Technology Award and Alumnus of the Year from the Wayne St. University Library and Information Science Program. From 2006-2009 he was Director of Online Information Services at Ask.com.
