More than seven years ago, Google made a breakthrough by developing the Transformer neural network architecture, which now powers generative artificial intelligence applications, including OpenAI ChatGPT. Now the company has unveiled its new Titans architecture, a new step towards AI that can think like a human.
Image source: BoliviaInteligente / unsplash.com
The Transformer architecture lacks long-term memory, which limits its ability to retain and use information over long periods of time, although this is an integral feature of human thinking. Titans includes neural long-term memory, short-term memory, and “surprise” learning systems – all of which humans use to remember unexpected or very important events.
Transformer has a sort of “spotlight,” a mechanism that allows the AI to highlight the most important words in a sentence and pieces of data at any given time. It is also in Titans, but it works in conjunction with a huge library – a long-term memory module, which is responsible for storing important information. This mechanism is similar to a student who can return to notes taken at the beginning of the semester, rather than trying to remember everything at once.
Focusing on relevant details and the ability to access stored knowledge allows Titans to process vast amounts of information without missing important details. With an intelligent “surprise metric” to help prioritize key pieces of data, Titans outperforms existing AI models in a variety of tasks: language modeling, prediction, and DNA modeling. The new architecture thus brings AI closer to the mechanisms of human cognition.
The new AI’s ability to retain rich context will help it revolutionize research by, for example, tracking scientific literature; or detect anomalies in huge data sets, in medicine or finance, because the system “remembers” what is normal and highlights what is a “surprise”.
The new architecture partly replicates human cognitive processes – in addition to short-term and long-term memory, it is the ability to “forget” less important information and prioritize more accurately. Likewise, a person more easily remembers events that violate his expectations – this feature will help create more subtle and context-dependent AI systems. Modern systems based on the Transformer architecture are capable of processing queries with a context of up to 2 million tokens, while Titans remain efficient above this limit, maintaining high accuracy with huge amounts of input data.
The “surprise metric” mechanism allows the system to determine what information should be stored in long-term memory – priority is given to items that violate expectations. This not only reflects human cognitive mechanisms, but also provides a new solution for managing limited memory resources in the field of AI. Early tests of systems on the Titans architecture showed promising results in a number of tasks, for example, in tasks associated with extracting specified information from large texts: as the context length increases, existing models show a sharp drop in accuracy, while the new architecture maintains performance.
It should be noted, however, that Titans technology is still in its early stages and there are likely to be challenges when deploying it in practical applications. It is too early to judge the system requirements for computing algorithms, the effectiveness of training and possible threats – all this will become clearer as the technology develops. And AI’s ability to store and evaluate information could raise questions about privacy, data processing mechanisms, and the unpredictable behavior of AI systems.