Small Language Models: “The Big Shrink in LLMs”

From Communications of the ACM:

…smaller models and datasets have emerged as a solution to some of their larger cousins’ drawbacks. Techniques such as knowledge distillation, transferring knowledge from large to smaller models, and pruning, removing model parameters (such as weight or temperature) without degrading accuracy, are also supporting the shrink. This and developments in edge computing enabled by smaller models that can run ‘on device’ raise questions about the dominance of gigantic datasets and models.

“The heavy reliance of LLMs on massive datasets raises real concerns about sustainability,” said Amir H. Gandomi, a professor of data science at the University of Technology Sydney, Australia. “I think we’re going to see some changes down the line. For starters, there’s the whole issue of content scraping, copyright disputes, and new regulations are already pushing companies to rethink how they get their training data.”

Gandomi also flags data quality, misinformation, and noise as impacting large models’ reliability; they are also energy and computing power hungry, he said. With co-authors Ishfaq Hussain Rather and Sushil Kumar of the Jawaharlal Nehru University, New Delhi, India, Gandomi recently published a review of deep learning techniques for democratizing AI with smaller datasets.

“What’s great about smaller, curated datasets is they’re faster to process, cost less to use, and are ideal for real-time applications or resource-constrained environments,” Gandomi explained. “While they may sometimes lack the generalizabililty of large-scale datasets, removing irrelevant or noisy data makes [them] more accurate and easier to interpret, which is especially critical in fields like healthcare, where every decision matters.”

Read the Complete Article (about 970 words)

Filed under: Data Files, News