If you're new around here, when discussing software, we end up talking about language models a lot. That's because these fancy bundles of linear algebra are actually pretty cool. There are some exciting implications for automated language approximation–that is, what language models do. ChatGPT and the like are a part of the latest wave of advancements in artificial intelligence.
However, since the release of ChatGPT in November 2022, these advancements have been shrouded in a deep cloud of excitement. Discussions of potential are often more common than discussions of practical value. The AI hype cycle has been in full swing. However, to me there have been some indications that we are moving from the so-called “Peak of Inflated Expectations”, to the “Trough of Disillusionment”.
Discussions of an “AI Bubble” have become quite popular, with skepticism rising now that the excitement has largely worn off. I want to do a case study on “AI” as a hype cycle generally. Personally, I'm still young and inexperienced, trying to get the hang of hype cycles in the first place. However, it has occurred to me that while we do seem to be entering the proverbial trough, there is another stage of the hype cycle on the horizon.
The so-called “Slope of Enlightenment” and “Plateau of Productivity” still await us along the way. In other words, while a wave of extreme expectations has failed to deliver, we can still expect to see real value gains from language models. This has prompted a bit of thinking on my end. Now that we've had a bit of experience with this technology, where might that value actually lay?
Specifically, I'd like to understand how we can expect language models to make our applications better. The nature of applications may not change as much as marketing teams may lead us to believe, but they will change. In many ways, they have already. Let's break down some different ways that we can expect language models to make better apps.
What's the big deal?
I would like to clarify an area that I believe is prone to misconceptions. That is, the actual advancement made by OpenAI with ChatGPT. This application was initially released in November 2022, and has seen significant performance improvents over time, with the release of GPT-3.5-Turbo and later GPT-4. ChatGPT was the first widely accessible method of interacting with a state-of-the-art language model.
“Widely accessible” being key. It's important to remember that GPT-3.5-Turbo was based on older models. GPT-3, a model and research paper published two years earlier in May 2020, was a previous foundational model upon which ChatGPT’s models were built. I'd actually consider this the first language model with serious practical utility in automating human language. It would seem that OpenAI believed this as well, as this was also the first model that was not open-sourced by the company.
You may have noticed that GPT-3 has a “3” in it, suggesting earlier iterations. GPT-2 was an open-source language model developed by OpenAI the year prior in 2019, and remains available for use today. Earlier still, a paper by OpenAI entitled “Improving language understanding with unsupervised learning” detailed performance improvements upon yet another pre-existing idea, Transformers. I've talked about Transformers before.
All this is to say that the release of ChatGPT did not mark a revolutionary development in the performance of language models. Rather, it marked an advancement in accessibility to the pre-existing technology. Language models existed before ChatGPT, and while there has been significant performance improvement since its release, fundamentally, nothing has changed.
This leads me to believe that we can expect iterative development upon current models, rather than leaps and bounds in performance improvements. We should see performance improvements, and there are many excited advancements on the way. However the curve won’t look as exponential as some may lead you to believe.
Better Forms
As the application developers among us will know, the world of user interfaces runs on forms. Almost every application has forms as a core facet of their functionality. This is because every app performs a function for its user. Forms collect user-specific information, and allow the app to adapt its functionality to the specific use-case. All that to say, forms are a way of collecting user data to perform a task. Language models have implications at this level.
The main implication, as far as I can tell, is in the ease of completion. When you fill in a form, you'll likely be filling in multiple fields. For an Instagram post, this is the caption, file upload, tag inputs, and layout. This is because in order to display your post, Instagram must know 1) how you'd like to caption the post, 2) what images are involved, 3) what order they should be displayed, and 4) who was in the images. Thus, a form has been constructed to obtain all of this information.
However, the nature of these forms may change a bit. While I hesitate to say with certainty, forms could see significant improvement in flexibility. This will likely be isolated to language-specific use-cases, but as the shape of the data required to perform the app's operations becomes more fluid, so will the forms. How specifically this looks, I'll admit I'm not entirely sure.
For our specific example, one might be tempted to say that an “Instagram Assistant” would be the way to go. Simply ask your assistant to submit a post, and it will guide you through the various steps and obtain the required information. However, as I talked about more in-depth in this post, an “Assistant” often just means a worse version of your current interface. It assumes that conversation is a better way to interact with your app's functionality, when oftentimes it is not.
What we might see, instead, is the very nature of applications begin to shift. The data required by applications will start to change and become more flexible. Instead of giving Instagram a caption, it will generate one by default based on the images, with preset tones and hashtags. Less discrete data is required here, because the caption can just be modified, rather than made from scratch. This kind of form could also learn over time based on user-preferences and adapt to your specific voice.
More Magical Wizards
One of the more difficult things to nail in application development is a complex workflow. I'd define this loosely as a workflow which requires two or more forms be completed before an operation is performed. However, this is a common task, and one type of interface has long been used to guide a user through such a process: the Wizard. You've likely come across one of these before.
Perhaps you downloaded some software that needed to know how to configure itself. Or maybe you were using a tax application that worked you step by step through filing. Whatever it may be, multi-form workflows are common, but tricky. The reason that they're tricky, is because you have to obtain data from the user in a more complex shape. Let's take the example of that tax software.
If H&R Block wants to file your taxes for you, they'll need a lot of information, and that information has tons of variability. You have your W-2s and other documents, each of those documents requires its own form, so that each document can be referred to discretely later. Then, other information is required. How many dependents do you have? Do you qualify for any deductions? Were you a student? Oh, by the way, do you want to pay for our premium plan?
All the while, this information is complicated. You might need clarification about a specific nuance of a question. “What qualifies as a dependent?”, “I kinda was a student for a while, but it wasn't a traditional student loan, does that count?” These questions are often answered by FAQs on the side of the screen which compliment the wizard. However, the complexity of the form and utility of the immediately available FAQs actually creates a situation where an Assistant might be useful.
A language-model-powered wizard could extract information in a conversation where you describe various activities and situations. It could answer questions while also completing operations and filling out your forms. Such a process would simplify the text-heavy workflow of taxes. This kind of improvement might be available to other workflows as well.
Intuitive Data Interaction
With the release of more performant language models came a wave of ideas on how to use them. One of the earlier ideas, which remains a popular paradigm today, was Retrieval Augmented Generation. Essentially, you'd take a bunch of documents and plug them into some database which was capable of performing a semantic search. Semantic search–by the way–means the ability to search documents based on their meaning, not necessarily just context.
This search system can then be paired with a language model integration to create a chatbot. Each time a user sends a message to the chatbot, the semantic search system is queried, bringing in data which is relevant in meaning to the user's message. Then, that data is given to the language model as context when completing its generation. Thus, the generation is augmented by the retrieval of documents. Boom. Retrieval Augmented Generation.
This paradigm is one of a few ways that we will continue to see interaction with data become more intuitive. Language models allow us to have more abstract interfaces for data. While in past decades, search often meant finding more direct similarity between different texts, these days we can search by meaning instead. Additionally, interpretation of texts used to be significantly less flexible. We had tools for interaction with language, but they were rigid by nature and highly rule-based.
Sentiment analysis, for example, required specific parsing steps before it was even possible. You could analyze the sentiment of individual sentences, but interpretation of larger texts was difficult. Summarization, automated tagging, etc., all were more complex operations that are arguably simple now. We can expect systems to continue developing, which will make interaction with data even better.
Enhanced Personalization
In my last post about the deterioration of online communication, I mentioned the idea of personalized language-model clones. Essentially, digital avatars based on your own personality, would go out and interact with other digital personas, and match on your behalf. This was more specifically proposed by Bumble founder Whitney Wolfe Herd, in the context of dating apps. However, we might see this paradigm extend to other types of recommendation algorithms as well.
On a hypothetical note, if YouTube really wanted to spice up their recommendation algorithm, they could take a similar approach. Create a personification of the user based on learned information about interests and tendencies, and have a language model pretend to be them. Then, for every video, have the transcript available. The personification would then be paired with the transcript, and told to rank its interest in that video's content. If the interest is high, the video could be recommended, and vice versa.
Of course, this is a rather crude example, but the idea still stands. Personas mean personification. Language models allow us to create digital representations of people. This might sound extremely dystopian–which I do think is a possibility–but it also may be beneficial. Applications could become significantly more useful and better tuned to their users. Content recommendations are one area of many where such a paradigm could be included. Digital personification could help creators understand their audiences, or provide more personalized settings.
Additionally, digital avatars could even allow for increased user-advocacy. Such a system could review content on behalf of the user, and spot dark patterns. For example, appeals to implicit biases or emotions used by recommendation algorithms could be identified and revealed to the user in real-time. There may be some hope on the horizon rather than gloom. Like most things, it will likely go both ways.
Superior Accessibility
Advancements in language models, such as the recent GPT-4o, have given hope for a more accessible future for those with disabilities. For those unaware, GPT-4o showed some moderate improvements in reasoning performance, but was largely an upgrade in another direction. That direction being immersion and latency. GPT-4o wasn't significantly smarter than its predecessor, but rather was an incremental upgrade which significantly improved voice interactions, and added video processing features.
This has large implications across the board. However, one of the most notable is that of accessibility. Those who are blind could use their phone, paired with GPT-4o or a similarly capable model, to analyze the world around them in real time. This was one of the use-cases that OpenAI advertised specifically, so I assume that they are working in this direction to some degree.
Apart from those with disabilities, inter-lingual communication is also in a much more accessible place. Translations have never been so easy, and scalable. Having real-time translation between multiple languages isn't necessarily a new problem, and Google Translate has been around for a while. However, it was Google's attempts at better translations which led to the advent of the transformer architecture in the first place. This new way of doing things has unlocked significant performance improvements in such areas.
Key Takeaways
The technology sector, and software industry more specifically, is highly prone to innovation and adaption. Additionally, predicting the nature of this adaption is tricky if not impossible. However, the aforementioned areas are ways in which I believe we can expect to see significant development. The industry is still trying to work its way through the slope of enlightenment to find how we can put language models to use in highly practical ways.
But improvements in application utility, better personalization, enhanced data interaction, improved accessibility, and more intuitive operations, are all on the horizon. The future might be unpredictable, but I'd take a guess that at least parts of it are bright!
Author's Note
As always thank you so much for reading. I'm really trying to get down this weekly posting schedule, and align it with the rest of my life. However, I think it will just take practice on my end, and I'm getting used to things. I hope that today's topic was interesting to you, and I'd love to hear your thoughts in a reply below, or you can find me over on SubStack notes! Anyway, as always, have an excellent rest of your day! Goodbye.
Credits
Music:
Track: Marshmallow by Lukrembo, Source: https://freetouse.com/music, Copyright Free Background Music
Thumbnail:
Bilal Azhar, https://substack.com/@intelligenceimaginarium