Human Memory & LLM Efficiency: Optimized Learning through Temporal Memory

Content including text and images © Aditya Mohan. All Rights Reserved. Robometircs, Amelia, Living Interface and Skive it are trademarks of Skive it, Inc. The content is meant for human readers only under 17 U.S. Code § 106. Access, learning, analysis or reproduction by Artificial Intelligence (AI) of any form directly or indirectly, including but not limited to AI Agents, LLMs, Foundation Models, content scrapers is prohibited.

In humans, the difference between long-term memory and short-term memory is significant, with short-term memory encompassing what happened less than a second ago and long-term memory covering information retained over extended periods. Our memory starts being shaped almost immediately by our preconceptions, influencing how we perceive and store new information. Short-term memory is generally more reliable; we are more likely to accurately recall events that occurred a second ago compared to those that happened a minute ago. However, as time passes, our memory becomes less reliable, subject to distortions and forgetting. In contrast, large language models (LLMs) operate differently. Autoregressive models, a class of machine learning models, predict the next component in a sequence based on previous inputs. LLMs are autoregressive models where the concept of time does not influence the prediction of the next word. They lack the distinction between short-term and long-term memory, as training data is fed to pre-train an AI model like GPT all at once, without the fundamental concept of short-term memorization. The human brain is remarkably efficient, operating continuously on about 12-20 watts of power, depending on the source and specific conditions. This efficiency is contrasted sharply by the energy demands of training LLMs, which can require several megawatts of power. For instance, training a large neural network can consume energy comparable to the output of a small power plant over several weeks.

Considering the efficiency of training human brains compared to the energy-intensive process of training LLMs, it can be argued that integrating the concept of long-term and short-term memory into LLMs could enhance their learning efficiency.

To integrate the concept of long-term and short-term memory into LLMs, drawing inspiration from human memory processes, the following modifications can be made:

Memory Layer Implementation: Introduce memory layers that emulate human-like memory processes. These layers can be split into short-term and long-term memory modules, similar to how humans distinguish between recent and enduring memories.
Dynamic Memory Allocation: Implement a dynamic memory allocation mechanism that allows the model to prioritize and update short-term memory for immediate tasks while periodically consolidating important information into long-term memory. Techniques such as memory networks or attention-based mechanisms could be used to manage this allocation, akin to how humans process and store new information.
Temporal Context Embeddings: Develop temporal context embeddings to help the model differentiate between recent inputs (short-term) and older, more consolidated information (long-term). These embeddings can guide the model in adjusting its focus based on the temporal relevance of the data, reflecting the human ability to prioritize recent experiences.
Regular Memory Refreshing and Consolidation: Implement processes for regular memory refreshing and consolidation. Short-term memory content can be periodically reviewed and important information transferred to long-term memory, mimicking the human process of memory consolidation during sleep.

By incorporating these human-inspired elements, LLMs can achieve a more natural approach to information retention and recall, improving their learning efficiency and reducing energy consumption during training.

Further read

LLM, Us and Neuroplasticity

"Are you sure?": Unveiling the Quest for Human-Like Artificial Intelligence

Designers, creativity and AI

A LLM as a reflection of our inner most desires

Should AGI be a she, he, or a it?