A young boy and his grandfather sit across from one another in the living room of an old cabin, a single log ablaze within the fire place. A window beckons the early light of the sunrise, which shines through the clear skies, onto the face of the old man. Looking towards the window and through it, the child asks a question.

“Grandpa, those mountains out there, what are they called.” the boy gestured towards the window.

“Those? You mean the three sisters?”

“Yes those! Have you been to see them.”

“No need to go anywhere, I can see them perfectly fine from here.”

“Oh.”

“You’re looking at them now, are you not?”

“Yeah. But couldn’t you see them better from up close?”

“There’s nothing there that I haven’t seen before.” He smirked, but his eyes twinkled as if daring the boy to prove him wrong.

“There isn’t?”

“The mountains have trees, I’ve seen trees. There’s water, I’ve seen water. What is there to see that I haven’t seen already in my long life? You walk out that front door, into the clearing, and you can hear the very same birds sing as those of the mountains. Why should I take the time to go out to the mountains?”

“Well, what if there’s something there that you haven’t seen?”

“Even if there was, why risk it? My axe cuts trees. My pail holds water. My shotgun handles birds. But something I’ve never seen—how would I prepare for that?”

The child stops to think for some time.

“So, if I want to see the view from the mountain, it’d be dangerous?”

“Boy, if you walk outside that door, it’s dangerous.”

“And what if I want to anyway?”

“You better make sure it was worth it even if you don’t succeed.”

Hitting the Wall

In mid-November, 2024, reports began to circulate that OpenAI had not seen significant improvements in performance over GPT-4 in its new model, Orion. After the awe induced by GPT-3-powered ChatGPT, and the subsequent release of the significantly more powerful GPT-4, anticipation grew for OpenAI’s next model, GPT-5. While we’ve received additional high-profile updates and model releases from OpenAI since GPT-4, a GPT-5 has still not reached consumers.

Orion, an internal codename for OpenAI’s anticipated model, has caused a stir as consumers await new advancements on the exponential curve to AGI. However, according to some reports, including from high-profile figures, we shouldn’t expect Orion to be that much better. In an interview with Reuters, Ilya Sutskever outlined some early signs of a plateau in model progress:

Ilya Sutskever, co-founder of AI labs Safe Superintelligence (SSI) and OpenAI, told Reuters recently that results from scaling up pre-training - the phase of training an AI model that use s a vast amount of unlabeled data to understand language patterns and structures - have plateaued.

“The 2010s were the age of scaling, now we're back in the age of wonder and discovery once again. Everyone is looking for the next thing,” Sutskever said. “Scaling the right thing matters more now than ever.”

Sutskever, who notably led an ousting of OpenAI CEO Sam Altman late last year, and was subsequently sidelined, has since left the company’s Chief Scientist position. Sutskever remains a highly influential figure in the field, and he seems to think there’s some fruit to this narrative. Other key players in AI have made similar discoveries to those purported to have occurred at OpenAI.

Another report from Bloomberg, citing anonymous sources, claimed that Orion wasn’t the only model to fall behind hopeful metrics. According to the report, Google’s Gemini has also seen less-than-impressive returns in the latest iterations. It’s worth noting that Anthropic’s Claude 3.5 Opus has seen an ambiguous timeline, potentially suggesting troubles with getting the model up to standards of a release.

All the while, however, OpenAI CEO Sam Altman, among other investor-pleasing company heads, continues to express enthusiasm about the near-future. “Feel the AGI” seems to be the resounding message of OpenAI’s marketing teams, as well as Altman himself, who has recently teased that AGI might be achieved by 2025. However, as The Verge pointed out, the bar seems to be lower for these companies than it was a couple of years ago. It’s not entirely clear what “AGI” even means at this point.

However, it is clear that following the money often leads to probable answers. Sam Altman and other company heads need investors to believe in their exponential curves, or they will no longer be able to raise the money they need to continue growing. Training new models is extraordinarily costly, and startups frequently struggle to maintain profitability without investors. OpenAI, like other startups, needs to paint a narrative that things aren’t slowing down, even if they most certainly are.

Exponential curves do occur, however, even during times of massive technological breakthroughs, progress typically grows quickly, and then slows. Thus, it’s always been reasonable to conclude that even with the unprecedented capabilities of modern language models, bottlenecks would be found. Such bottlenecks today include prohibitive training costs, grid-killing energy requirements, and simply need for more research.

With that said, it seems more than likely that the current course is, in-fact, reaching a plateau. Major executives likely wouldn’t be leaving OpenAI in the same way if they believed that AGI was right around the corner, and reaching a plateau was an inevitability. However, that doesn’t mean AI is going away for a while, in fact, I’d say it means things are just getting started.

Industry Expansion

Let’s put aside the debate, and assume that models aren’t going to be getting much better than they are now for a few years. What does that actually mean for the industry? I think that while the underlying technology might not see massive changes, there are three main ideas that I think will play out across the next few years.

First, Integration

First off, integrations of existing models will just keep getting better and better. To illustrate, think about the web. Every website on the internet is composed of three main languages: HTML, CSS, and JavaScript. These technologies were largely adopted as standards around the early 2000s, however, there is a stark contrast between the websites of the early 2000s and those of today.

Of course, there have been significant improvements to the underlying technologies. All three languages have seen improvements to syntax, features, and mindset. However, the basic structure has remained a stable platform which has been iteratively built on, integrating our lives into the web. The idea here is that although model performance may plateau, we don’t actually need the models to keep getting better, to keep making significant progress.

I wrote about the idea of Invisible AI a while back, meaning integrating these models seamlessly, to make features feel like magic. Many of today’s “AI Integrations” are glorified ChatGPT wrappers, which wire up some app’s functionality to a chatbot. This type of integration is likely to be less-than useful as time progresses. This is simply due to the fact that this simplistic approach is in the early phases of this technology’s lifecycle, and we have yet to see the full-extent of its maturation.

So, as models reach some form of stability, they’ll feel more and more like a platform to build upon, and many problems will continue to be solved by good engineering. The models may not change much, but the way we experience them will continue getting much better.

Second, Verticality

Much of the low-hanging fruit is gone. Google CEO Sundar Pichai said it himself at the New York Times’ recent DealBook Summit:

“I think the progress is going to get harder. When I look at [2025], the low-hanging fruit is gone,” said Pichai

The idea here is that the simplistic applications of Generative AI, those which are relatively obvious, have largely been done by now. Things like Chatbot interfaces, Customer Service automation, Copilot tools, writings tools, and others, aren’t really going to see much competition from emerging startups. However, this doesn’t mean that startups won’t see significant success.

One of the biggest terms going around the startup space at the moment is “Vertical Agents”, that is, essentially, applying generative AI to a niche. The idea of niching down is nothing new, as focusing a solution to a problem often is when you can constrain focus on a specific area. However, technological advancements become rich grounds for these exact types of solutions to start popping up, especially as the market becomes more familiar with the underlying technology.

When a technology is too new and lacking in maturity or popularity, it’s hard to convince an unfamiliar market of its utility. The products are often expensive, scrappy, and lacking in clear value proposition. Early adopters might pick up on your cool project, but the obvious implementations of the technology are being worked into everyday products, so it’s tricky to really get traction in a specific niche early on.

However, given the widespread popularity and continued adoption of these generative tools, it’s likely that we’ll only continue to see increasing demand across industries. Thus, narrower, niched down solutions have the opportunity to be ever more fruitful as consumers start looking for AI-powered solutions to their existing problems. If you can bring a standalone solution to their problem, and leverage generative AI effectively to increase their productivity and decrease friction, there is a clear value proposition.

There’s a reason this is being talked about, these are exciting times.

Third, Skills

It’s safe to say that hype pervades the AI industry, and it’s hard to separate fact from fiction. For example, talk of automating software development through tools like Devin.ai, or fears from developers themselves. Artists are fearful for their own jobs as generative models get better, and this is by no means a narrow phenomenon. Professionals across the globe fear for their jobs as “AGI” seems ever-closer.

These fears aren’t entirely unfounded, either. The companies selling them try not to sound like existential threats, but reading between the lines quickly reveals that they market themselves as replacements to current professional staff. The value proposition is something like “we can replace your current developers for a fraction of the price, and better output”, which sounds like a dream to companies employing expensive professionals.

However, at least for now, it seems unlikely that professionals will be automated outright, if at all. In fact, demand for professionals may increase entirely, although the nature of roles may shift. The reason: agents just aren’t that good. LLMs decrease in accuracy as complexity of tasks increases. The more steps required from point A to point B, the more you see inaccuracies amplified, and eventually the LLM is entirely incapable of moving forward.

This may not always happen, especially with simpler, in-distribution tasks. However, the fact that it’s common and hard to stop, means that employers won’t be entrusting significant professional responsibilities to the robots any time soon. Rather, demand for those professionals will increase as the industry expands, and productivity is bolstered by effective AI-tooling.

This may mean, sort of counter-intuitively, that while automation becomes more prevalent across professional industries, demand for skilled workers will only increase, especially for those with skills utilizing AI tools.

In practice

Practically, building up skills, focusing on better integrations, and niching down your solutions is likely to be a recipe for success in the coming years. Next time, we’ll discuss what next year specifically might look like, and how to capitalize on expected developments, based on what we saw this past year. The future is looking bright, just focus on continued improvement.

Author’s Note

Hey there! Sorry this took so long to get out, I did not know what I was getting myself into as far as moving goes. However, despite the chaotic time, this was fun to write. The idea of diminishing returns just kind of makes sense, and it’s hard to believe CEOs like Sam Altman, beholden to their investors, when they say that AGI is just around the corner.

That said, I’m excited anyway. These foundation models are super cool, and huge to be building on top of right now. While I don’t buy the “AGI in 5 years” narrative, if not simply due to the lack of accuracy in completing complex tasks, I do subscribe to the idea that things are only going to keep getting better. Vertical AI, better integrations, and building skills more broadly are all things that I’m going to be continuing to focus on in the upcoming years.

With that said, I’m tired, so I’m going to call it for today. As always, thank you for reading, and I’ll see you next time. Goodbye :)

Credits

Thumbnail:

Bilal Azhar at https://substack.com/@intelligenceimaginarium

Music: Track - Feeling Good by Pufino, Source - https://freetouse.com/music, Free Music No Copyright (Safe)