If you think an artificial intelligence model running on thousands of cutting-edge computer chips is smart, allow me to introduce you to the concept of a 1-year-old.
OK, so babies might not be able to write computer programs, solve advanced math problems, or debate philosophical ideas. But unlike today’s AI models, which consume an ocean’s worth of training data and as much energy as a small country, babies learn to make sense of the world with amazing efficiency. They identify new objects after seeing them once or twice, and they learn through fleeting observation and physical interaction.
When it comes to improving AI, babies—and the architecture of their brains—might hold crucial insights. Building a more baby-like version of AI could make frontier models less costly and less energy intensive, and it might also be valuable if AI-powered robots are to learn about their environments in a more natural way.
To explore this bold new frontier, researchers at Meta, Stanford University, the University of Tokyo, and France’s École Normale Supérieure developed a new test that highlights the learning skills of babies and pushes AI researchers to design algorithms that match them.
The EgoBabyVLM Challenge judges how well vision language models, or VLMs, which learn from both text and imagery, can make sense of the world as a baby sees it. It requires a model to describe the world after ingesting about a thousand hours of video collected from cameras strapped to the heads of infants and toddlers. (Yes, really.)
It turns out that the cutting-edge models fail miserably when fed this realistic and messy footage, which suggests there may be something different about the design of the baby brain that enables it to learn so rapidly from so little information.
Instead of curated datasets, babies learn from a kaleidoscopic view of things: parents talking about objects that are no longer visible, indicating things using their gaze or a gesture, or discussing events from the past or in the future rather than whatever’s happening right then. Babies learn not just from language but also from a rich multimodal and tactile experience, says Michael Frank, a cognitive scientist at Stanford University who specializes in language learning and was involved with EgoBabyVLM’s development.
The test shows that when it comes to AI, “it’s clear that there’s more [than just language] that’s needed,” Frank says.
Language Learning
EgoBabyVLM is just the latest example of how scientists are using AI to explore human intelligence. A challenge called BabyLM, introduced in 2023, tasked AI models with learning the syntax of language using about the same amount of data a 10-year-old takes in—tens of millions of words, compared to trillions for AI models. Remarkably, it turns out that transformer-based AI models—which process language by paying attention to the relationship between words across different sentences—can do this quite well, a finding that challenges Noam Chomsky’s ideas concerning how syntax may be hardwired into the human brain.
Ryan Cotterell, a linguist at ETH Zurich who first developed BabyLM, says the situation is different when it comes to understanding the physical world. “There isn’t going to be a large corpus of human interactions—there’s no internet of human interactions,” he says.
Joshua Tenenbaum, a cognitive scientist at the Massachusetts Institute of Technology, notes that BabyLM showed models do not acquire “common sense” about the physical world, social dynamics, or theory of mind.
“Transformers are very good at finding patterns in data,” says Tenenbaum. “But it does seem that just pure pattern learning systems are not able to take the kind of data that a baby or a child receives and learn all the things that they do.”
