From Size to Smarts: The Exciting Race in A.I. Development

In the ever-evolving landscape of artificial intelligence (A.I.), researchers and developers are engaged in an intense race. Their goal? To make A.I. smaller, while simultaneously making it smarter. This pursuit is driven by the desire to create more efficient and capable A.I. systems that can seamlessly integrate into various applications, from everyday devices to complex industries. By shrinking the size of A.I. models and enhancing their intelligence, these advancements hold the potential to revolutionize our interactions with technology and pave the way for a future where A.I. truly mimics human-like abilities.

Reducing the vocabulary taught to expansive language models can enhance their human-like conversational abilities.

When it comes to chatbots powered by artificial intelligence, the prevailing notion is that bigger is better. Well-known language models like ChatGPT and Bard, known for generating original and conversational text, tend to improve with increased data input. Enthusiastic bloggers regularly tout the latest advancements, whether it’s an app that condenses articles, AI-generated podcasts, or a finely-tuned model capable of answering any question about professional basketball, claiming that these developments will revolutionize everything.

However, developing larger and more powerful AI systems requires substantial processing power, which only a few companies possess. This has led to concerns about a small group, including Google, Meta, OpenAI, and Microsoft, wielding near-total control over this technology.

Moreover, larger language models pose challenges in terms of comprehensibility. They are often referred to as “black boxes,” even by their own creators. Influential figures in the field have expressed unease over the potential misalignment between the goals of AI and our own aspirations. While bigger may be better, it also means greater opacity and exclusivity.

In January, a group of young academics specializing in natural language processing, a branch of AI focused on linguistic comprehension, sought to challenge this prevailing paradigm. They issued a call to teams to develop functional language models using data sets significantly smaller than those employed by advanced large models. The aim was to create mini-models that would be almost as capable as their high-end counterparts, yet much smaller in size, more accessible, and more compatible with human interaction. This initiative became known as the BabyLM Challenge.

“We’re urging people to think small and prioritize the construction of efficient systems that can be used by a larger population,” explained Aaron Mueller, a computer scientist at Johns Hopkins University and one of the organizers of the BabyLM Challenge.

Alex Warstadt, a computer scientist at ETH Zurich and another individual involved in organizing the project, emphasized that the challenge shifts the focus from merely enlarging models to exploring fundamental aspects of human language learning. Instead of asking, “How big can we make our models?” the BabyLM Challenge places questions related to human language acquisition at the forefront of the discourse.

Large-scale language models are neural networks specifically designed to predict the next word in a given sentence or phrase. They undergo training using vast collections of words sourced from various transcripts, websites, novels, and newspapers. The training process involves the model making guesses based on example phrases and adjusting its parameters based on how accurately it predicts the subsequent words.

Through repeated iterations, the model constructs a map of word relationships. Generally, the more words a model is trained on, the better its performance becomes. Each phrase provides contextual information, enabling the model to develop a more nuanced understanding of word meanings. Prominent models like OpenAI’s GPT-3, launched in 2020, were trained on a corpus of 200 billion words, while DeepMind’s Chinchilla, released in 2022, underwent training on a trillion-word dataset.

This development of language generation by non-human entities presents an exciting opportunity for Ethan Wilcox, a linguist at ETH Zurich. He wonders if AI language models can be used to study how humans learn language. This raises questions about the influential theory of nativism, which suggests that humans possess an innate understanding of language and learn it swiftly and efficiently. However, language models also exhibit rapid language learning without an inherent comprehension of linguistic structures, challenging the validity of nativism.

The challenge lies in the different learning approaches employed by language models and humans. Humans have physical bodies, social interactions, and sensory experiences that contribute to language acquisition. Early exposure to spoken words and syntax, often absent in written language, further shapes our linguistic development. Consequently, a computer model trained on an abundance of written text can only provide limited insights into our own language learning processes.

To address this, Wilcox, along with colleagues Mueller and Warstadt, conceived the BabyLM Challenge. The goal is to bring language models closer to human understanding by training them on the same number of words encountered by a 13-year-old, approximately 100 million. Participating models will be evaluated based on their ability to generate language and capture linguistic nuances, with a winner declared.

Eva Portelance, a linguist at McGill University, discovered the challenge on the day of its announcement. Her research bridges the gap between computer science and linguistics. Early AI efforts in the 1950s aimed to model human cognitive abilities, with the “neuron” serving as the fundamental unit of information processing. Language models of the 1980s and 1990s were directly inspired by the human brain. However, as processing power increased and marketable products became the focus, researchers found it easier to train models on massive datasets rather than structuring them with psychological insights. This disconnects the functioning of language models from our understanding of human language.

For scientists seeking to comprehend the workings of the human mind, these large models offer limited insight. Moreover, their resource-intensive nature restricts access to only a few well-funded industry labs, hindering democratic participation in research. The BabyLM Challenge represents a departure from the race for larger language models and a move toward more accessible and intuitive AI.

Notably, larger industry labs have not overlooked the potential of such research programs. Sam Altman, CEO of OpenAI, recently acknowledged that further increasing model size may not yield the same level of improvements observed in recent years. Companies like Google and Meta are also investing in research that focuses on more efficient language models guided by human cognitive structures. After all, a model trained on less data but capable of generating language could potentially be scaled up as well.

While a successful BabyLM may have potential commercial benefits, the primary goals of the challenge are academic and abstract. Even the prize is unconventional, centered on personal pride rather than a practical reward.

The BabyLM Challenge represents a noteworthy shift in the development of language models, emphasizing the exploration of human language learning rather than the relentless pursuit of larger models. By training models on datasets akin to what a young human encounters, the challenge aims to bridge the gap between AI language models and our understanding of language acquisition. This approach offers opportunities for more accessible and intuitive AI, fostering research that delves into the workings of the human mind. As industry labs recognize the limitations of simply increasing model size, the BabyLM Challenge serves as a stepping stone towards more nuanced and comprehensible AI systems. While the rewards are driven by academic and abstract motivations rather than practical gains, the impact of this endeavor could have far-reaching implications for both the scientific community and the broader AI landscape.

From Size to Smarts: The Exciting Race in A.I. Development