New Resource From Allen Institute for Artificial Intelligence (Ai2): “SciArena: A New Platform for Evaluating Foundation Models in Scientific Literature Tasks”

From a Ai2 Blog Post:

Scientific literature is expanding at an unprecedented rate, making it challenging for researchers to stay updated and synthesize new knowledge. Foundation models are increasingly being used to help with this, but evaluating their capabilities in open-ended scientific tasks remains a significant challenge. Traditional benchmarks are often not suitable for nuanced evaluations in scientific tasks as they are static, limited in scale, and quickly becoming outdated.

To address these limitations, we present SciArena, an open and collaborative platform that directly engages the scientific research community in evaluating foundation models for scientific literature tasks. This crowdsourced, head-to-head evaluation approach for LLMs has been successfully pioneered in the general domain by platforms such as Chatbot Arena.

[Clip]

As of June 30, 2025, SciArena hosts 23 frontier foundation models, selected for their representation of current state-of-the-art capabilities. Among them, the o3 model consistently delivers top performance across all scientific domains. Moreover, we find that o3 provides a more detailed elaboration of cited scientific papers, and its output tends to be more technical in Engineering disciplines. Performance among the remaining models varies by discipline, for instance, Claude-4-Opus excels in Healthcare, while DeepSeek-R1-0528 performs well in Natural Science.

The platform has three main components:

• SciArena Platform: This is where human researchers submit questions, view side-by-side responses from different foundation models, and cast their votes for the preferred output.
• Leaderboard: Based on community votes, an Elo rating system ranks the models, providing a dynamic and up-to-date assessment of their performance.
• SciArena-Eval: This is a meta-evaluation benchmark built on the collected human preference data, designed to assess the accuracy of model-based evaluation systems.

See Also: Allen Institute for Artificial Intelligence Introduces Ai2 Paper Finder (March 2025)

See Also: New Experimental AI-Based Research Tool For Literature Reviews: “Introducing Ai2 ScholarQA” (Jan. 2025)

See Also: Prototype: University of Washington & Allen Institute for AI (the People Behind Semantic Scholar) Announce the Launch of Ai2 Open Scholar (Nov. 2024)

Filed under: Data Files, Journal Articles, News, Reports