Sakana AI Introduces Frontier Intelligence Group

The Current and Future State of Artificial Intelligence

Current AI systems are extraordinary. They can write and converse expertly on almost any topic, compose music, generate video, program entire applications, and have even started providing insights into long-standing mathematical problems. Given this trajectory, it’s reasonable to ask if the current paradigm (scaling the Transformer architecture with even more data and even more compute) is “all you need” to take us all the way to AGI. This is exactly what Llion Jones, our CTO and one of the inventors of the Transformer, questioned in the recent Transformer vs Post-Transformer debate. Spoiler: we think there’s more to do.

The most popular components of modern AI systems, such as the most popular architecture, optimiser, training objectives, and so on, work so well that we can maintain progress by continuing to tweak them. However, current systems still have fundamental problems that we haven’t been able to address yet: they confidently hallucinate false information, falter badly in the face of genuinely novel situations, and require significant amounts of energy to run. Many researchers are incentivized to try and fix these problems within the current paradigm, but maybe we should be looking for other paradigms.

Nature shows us that alternatives exist. In contrast to current AI, biological intelligence learns continuously, requires far less data to generalise robustly, and can even explore through self-directed, open-ended curiosity. And even though our current AI systems are built upon “artificial neurons”, the way that artificial neural networks are structured and updated is a long way from our current understanding of the brain. While we don’t expect that we need to replicate a real brain in silico, it may be that these gaps are telling us something important about general intelligence.