AI models are like babies: continuous growth spurts make them more fussy and needy. As the AI race heats up, frontrunners such as OpenAI, Google, and Microsoft are throwing billions at massive foundational AI models comprising hundreds of billions of parameters. However, they may be losing the plot.
Size matters
Big tech firms are constantly striving to make AI models bigger. OpenAI recently introduced GPT-4o, a huge multimodal model that "can reason across audio, vision, and text in real time." Meanwhile, Meta and Google both developed new and enhanced LLMs, while Microsoft built its own, known as MAI-1.
And these companies aren't cutting corners. Microsoft's capital investment increased to $14 billion in the most recent quarter, and the company expects that figure to rise further. Meta cautioned that its spending could exceed $40 billion. Google's concepts may be even more costly.
Demis Hassabis, CEO of Google DeepMind, has stated that the company plans to invest more than $100 billion in AI development over time. Many people are chasing the elusive dream of artificial generative intelligence (AGI), which allows an AI model to self-teach and perform jobs it wasn't prepared for.
However, Nick Frosst, co-founder of AI firm Cohere, believes that such an achievement may not be attainable with a single high-powered chatbot.
“We don’t think AGI is achievable through (large language models) alone, and as importantly, we think it’s a distraction. The industry has lost sight of the end-user experience with the current trajectory of model development with some suggesting the next generation of models will cost billions to train,” Frosst stated.
Aside from the cost, huge AI models pose security issues and require a significant amount of energy. Furthermore, after a given amount of growth, studies have shown that AI models might reach a point of diminishing returns.
However, Bob Rogers, PhD, co-founder of BeeKeeperAI and CEO of Oii.ai, told The Daily Upside that creating large, all-encompassing AI models is sometimes easier than creating smaller ones. Focussing on capability rather than efficiency is "the path of least resistance," he claims.
Some tech businesses are already investigating the advantages of going small: Google and Microsoft both announced their own small language models earlier this year; however, they do not seem to be at the top of earnings call transcripts.
