The story behind ChatGPT: How Transformer architectures are conquering the AI world

In the building named after him Touring Test a human interacts with both a human and an artificial intelligence. The test is considered to have been passed if the human tester can no longer distinguish the machine from a human during the interaction. Today's GenAI systems such as ChatGPT have come much closer to this goal since then. Reason enough for us to take a look at the historical development of natural language processing (NLP).

 Basically, the development of language processing can be divided into the following phases:

1. rule-based approaches

One of the oldest approaches in the field of language processing are rule-based approaches, which were developed in the 1960s. These work - as the name suggests - with a series of predefined rules in which the input of a text or language is mapped to a specific output. One of the first applications was the ELIZA Chatbot from 1966, which simulates a conversation with a psychotherapist by responding to user input with simple pattern recognition and rules.

2. statistical methods

The 1980s saw the emergence of statistical approaches. These approaches used probability models and statistical methods to analyze and understand linguistic patterns in large text corpora. Particularly noteworthy here are Hidden Markov Models (HMM), Conditional Random Fields (CRFs), Support Vector Machines (SVM) and n-gram language models.

3. Neural networks, word embeddings, deep learning & the vanishing gradient problem

Although the origins of the development of neural networksAlthough neural networks date back to the 1940s and 1950s, their use became particularly widespread in the 1990s and 2000s. Neural networks are inspired by the structure and functioning of the human brain and consist of many interconnected neurons organized in layers. These neural networks are then trained for specific tasks, such as text recognition or generation, as part of speech processing.

Word Embeddings are another important concept in language processing. It was introduced in the 2000s and has been used since 2013 with the Word2Vec process has become particularly widespread. Word embeddings are mathematical representations of words in a multidimensional space that are trained so that words with similar meanings are close to each other. A simple example: the terms "programmer" and "developer" are very close to each other in this space, while they are rather further away from terms such as "sun" or "daylight".

Today, the principles of neural networks and word embeddings are a very important basis for all further developments and therefore also for all of today's Large Language Models (LLMs)that underlie GenAI systems such as ChatGPT or Llama.

The development of increasingly complex deep neural networks (Deep learning), a new phenomenon came to bear, the Vanishing gradient problem. In short, large neural networks with very many layers reach a point where they can no longer be trained efficiently. This means that adaptations/knowledge in the learning process are no longer passed on efficiently between the deep layers in the neural network, with the effect that the training process effectively comes to a standstill and literally freezes.

4. RNN, LSTM, GNTM

To address the vanishing gradient problem, a number of increasingly complex neural network architectures have been deployed and developed.

Recurrent neural networks (RNN) were developed back in the 1980s. However, they became particularly widespread from the 2010s onwards, as they proved to be particularly effective in dealing with the vanishing gradient problem with increasing computing power and the development of improved algorithms. RNNs are a form of neural network with a recurrent - i.e. retrograde - structure in which information is fed back through loops in the network architecture. This allows temporal dependencies to be taken into account when processing text sequences. The network has a kind of "memory" - hidden state called.

A further development of RNNs, which have become more widespread since around 2013, are Long-Short-Term Memory (LSTM) architectures. These make it possible to store information in the network over longer periods of time. LSTMs have mechanisms such as input, output and forgetting gates to improve information flow control. They are therefore suitable for larger text sequences.

Building on these developments, progress in speech processing will then be made more or less in quick succession from 2016 onwards. It starts with the introduction of Google's GNMT (Google Neural Machine Translation). A machine translation system that works much better than previous systems.

5. "Attention is all you need", Transformer architectures, LLMs and GPTs

We will continue in 2017 with the groundbreaking paper "Attention is All You Need" by Vaswani et al.which, with the introduction of Transformer architectures, laid the foundation for all of today's widespread and successful language processing systems such as ChatGPT (GPT stands for Generative Pre-trained Transformers), Llama etc. The basis for the Transformer architectures is the so-called Attention mechanism. A technique in neural networks that makes it possible to selectively focus on the important parts of sentences so that the model takes more relevant information in a text into account. Terms with high significance are therefore given particularly high weighting. This mechanism is then refined in transformer architectures - i.e. again neural network architectures - so that they are able to recognize complex relationships between terms even across very long texts and then process or learn these efficiently even across a large number of very large data sets.

Based on the transformer architectures described above, all large language models (LLMs) that are relevant today are subsequently developed. The term is derived from a very large number of trained parameters and a massive amount of data with which the model was trained. These include, among others:

Product	Provider	Publication date	Number of parameters
GPT-1	OpenAI	June 2018	117 million
BERT Large	Google	october 2018	340 million
GPT-2	OpenAI	November 2019	1.5 billion
GPT-3	OpenAI	June 2020	175 billion
PaLM	Google	April 2022	540 billion
GPT-3.5	OpenAI	November 2022	No public information available.
Llama 1	Meta	February 2023	Up to 65 billion
GPT-4	OpenAI	March 2023	No public information available. Estimates run into the trillions.
Claude 1	Anthropic	March 2023	No public information available.
Llama 2	Meta	July 2023	Up to 70 billion
Claude 2	Anthropic	July 2023	No public information available. Estimates assume >100 billion parameters.
Llama 3	Meta	April 2024	Up to 70 billion

Our assessment

In November 2022, ChatGPT was released, based on a model of GPT-3.5 optimized for conversations. However, this development did not fall from the sky overnight. It is the result of a continuous, successive development. The beginnings of this development date back to the 1950s and continue to this day.

The development of language processing has gone through several decisive phases. Starting with rule-based approaches in the 1960s, through statistical methods in the 1980s, to neural networks and deep learning in the 1990s and 2000s. The breakthrough in recent years came with the introduction of transformer architectures. These are based on the attention mechanism and have formed the basis for modern language models such as GPT since 2017.

Today, we are experiencing a world that is becoming even faster and more efficient thanks to GenAI. Tools like ChatGPT can realize a variety of tasks and applications. They are also a versatile tool that can be used in many areas of daily life and work. Today, AI helps to break down barriers between people, technologies, languages and cultures. The evolution of this technology shows that current advances in AI - especially in the field of language processing - are not only technical milestones, but also have a profound impact on our society and the way we interact, work and live.

According to most experts, this development is currently only limited by the available computing and storage capacities. For this reason, there are moonshot projects such as Stargate. For which up to 100 billion dollars in investment costs are planned for hardware alone. It will be exciting to see if and when this development - as so often in the past - reaches a limit again and what new concepts will then be developed to push this limit back.