Memories. The subject of many songs, speeches, conversations, arguments, dreams, and nightmares. A fundamental aspect of our perception, memories serve as a bridge between the past and our present selves. The nature of this memory takes many forms, and varies across the biological landscape, with arguable inorganic variants existing as well. The ground itself holds the memory of floods, earthquakes, asteroids, and volcanic activity.

That is to say, memory can be thought of quite broadly. Today, we’ll be talking about giving memory to computed agents. However, technically, computers have memory already. Multiple forms at that. Every program running on your computer has access to what amounts to a scratch pad, used to jot down important references and use them later. Of course, this scratch pad, while fundamental to computers, isn’t exactly what we’re talking about. So what are we talking about?

Many, if not all organisms, seem to have a memory. For some, this is as broad as phenotypical evolution. Phylogenetic memory, meaning the inherited instinctual knowledge encoded in an organism's genes due to the survival experiences of its ancestors, is present in all organisms. Of course, us humans and our posse of creatures with similar levels of intelligence, can relate to a more specific type of memory. A memory of the abstract mind, that is. The thoughts and concepts that we recall on demand with every waking moment of perception, likely do not represent the conscious experience of most organisms.

Ants, rather than utilizing big brains to store their information, utilize a combination of the environment, and society to form memories. An ant colony is a fascinating form of collective intelligence. These little creatures rely on chemical pheromones to communicate information with future ants, allowing them to perform their anty-business with information from experiences. This “memory”, as we may call it, doesn't take the form of thoughts and concepts, but rather physical chemical trails.

However, both the mushy fat ball of the humans and the stinky dirt trails of the ants do share a common purpose. Storing important information from the past for future use. This idea, so foundational to the nature of organisms that it’s embedded into the evolutionary process itself, is a non-negotiable stepping stone on our road to human-like autonomous agents. If an agent is meant to survive and thrive within an environment, then it must mold itself to the environment continuously in order to survive.

So, today, we’ll be talking about how we might make that happen practically, in code. We’ll use humans as a reference, and break down how we can utilize the same concepts in LLM-based autonomous agents.

Anthropomorphic Memory

So, disclaimer, I’m not a cognitive scientist. However, I think we’ll be able to break down memory at a high level, and retain enough accuracy to be useful. Memory in humans is complicated, but what we experience as memory can essentially be broken down into three main types: semantic, episodic, and procedural. Additionally, the way that memory is stored differs based on the nature of the information and how it might be utilized.

Three Types

Semantic memory is our understanding of facts and concepts. For example, you probably know that cats are mammals, and that’s an example of semantic memory. The very idea of the three types of memory could be categorized as semantic memory. These memories are retrieved as we take in sensory stimuli. Walking into a kitchen to cook a meal brings forth concepts such as “knives are used for cutting food” and “the burner is hot”. These are crucial concepts for survival and filling your biological niche.

Episodic memory, on the other hand, is memory of “episodes”, or experiences that we have and find significant enough to remember. Your wedding day, the birth of a child, or your 21st birthday, are fairly common examples of episodes which might be remembered. Episodic memory allows an agent to associate experiences with present stimuli, such as seeing a restaurant and remembering a good meal. This type of association means better decision-making down the road.

Procedural memory is why, once you learn how to ride a bike, you can likely pick one up after five years and do it again. This type of memory oftentimes doesn’t directly affect perception, like the more conceptual memory types above. Rather, as a particular procession of muscle movements is repeated, it becomes engrained in your mind, and solidified. This indirectly influences perception, as certain actions become instinctual rather than requiring conscious focus. A guitar player can focus on the way the music sounds rather than the positions of their fingers.

Working Memory

Aside from those three main types of memory, further concepts may additionally be applied. Specifically, regarding organization and productivity, we should consider the concept of working memory. You see, as you go about your day, doing all the productive tasks I know you’re doing, you’ll encounter many things that you need to remember. However, these things don’t require long-term storage, but rather a temporary location to be worked on.

Say you’re asked to multiply 23 by 17 in your head. You’ll need to keep both numbers in mind to solve the equation, but it’s unlikely that your mind will decide to store it permanently. In this way, working memory is like a temporary mind-based workbench that your brain can use to crunch numbers or work through concepts, without storing information in longer-term-memory. For an agent, this concept may be utilized by allowing a scratch-pad of sorts for use in tasks.

Associative Memory

Memory itself isn’t much use if it’s random and sporadic. That’s why our memories aren’t like that. When you “remember” something, it’s almost always related to the current situation in which you find yourself. Memories are context-dependent, and this means that we are recalling relevant information. Determining the relevance of two pieces of information, is one of the hardest problems to solve. I’ve talked about the idea of relevance realization before, but it applies here.

Keeping memories relevant to the situation at hand is key for productivity and survival. While relevance realization applies mostly to human psychology, and broader organic life, we’ve built systems capable of approximating our reasoning. Language models and embedding models, most notably, are capable of encoding the semantic relevance of terms to one another, based on statistical analysis. This means that in an inorganic agent, we can mimic organic processes.

Additional Concepts

Some concepts are relatively well-known, or just not worth doing a full paragraph on for our purposes. However, they’re still important. Long-term and Short-term memory, for example, are different systems which may be utilized by memories. Something may start as a short-term memory and move to long-term storage, and vice versa. Additionally, memories are strengthened as they’re used, like muscles, and therefore the amount of recalls may be a useful metric.

Integration

Now that we’ve got an idea of what memory looks like in humans, at least broadly, we can start to determine what it will look like in an agent. We’ve talked about Simple Agent before, but for your reference, here’s the repository. To recap, however, Simple Agent is a project designed as a proof-of-concept, and template for experimentation and agent development. Therefore, we’ll be using it as a starting point for the rest of this ongoing project.

Doing the work

Working memory, as explained above, is the idea that the mind allows a temporary scratch pad of sorts, where you can perform operations and store information for the task at hand. Keywords: task at hand. Simple Agent does actually utilize tasks as part of the system, and so the idea of a task-based memory system seems like it could include a simple modification to that data-structure:

@dataclass
class Task:
    id: str
    description: str
    requirements: List[str]
    completed: bool

This is what a task looks like currently. I think for working memory, it could be as simple as a modification to add notes to individual tasks.

@dataclass
class Task:
    id: str
    description: str
    requirements: List[str]
    completed: bool
    notes: str

Now, when the agent attempts to perform a task, it does so with notes at hand. All we’d have to do, after making this change, is to give the agent access to an additional tool, which can add or modify notes on tasks. I’ll add those changes, and we’ll test it out later.

Making a Memory

The memory itself is a bit fluid and hard to contain to a single set of dimensions. However, we are going to have to wrap some code around them eventually, so why not try forming a simple object. Objects are a good way to encapsulate entities within code, because they allow you to specify properties of that entity and eventually establish how it will work with other systems. Since we’ve more or less gotten short-term working memory down, what are some properties of an individual long-term memory?

First, long term stored memories have content of some kind, whether that be a series of events, a concept, or fact, or a way of completing a step. I have some ideas for muscle memory in machines that I want to tackle in the future, so for now we’ll skip that. However, between semantic and episodic memories, there has to be a content of the memory, what it contains. Additionally, as memories are accessed, they become stronger, so a strength property might be useful as well.

@dataclass
class Memory:
    content: str
    times_accessed: int

Next, while the amount of times a memory is accessed speaks to its strength, there are additional factors which play into the strength of a memory. For example, the significance of the event in which a memory was formed, is a very important aspect. In humans, a traumatic event is often a much stronger memory than a mundane task like eating breakfast. So, we’ll have a field for initial significance as well, which will be decided by the discretionary memory module. Therefore, we’ll keep things simple with a tiered level of importance. To demonstrate:

@dataclass
class Memory:
    content: str
    times_accessed: int
    importance: Literal["extreme" | "high" | "mid" | "low"]

We can work out the details through the practice of iterative development. However, for now, we’ll just allow whatever is creating the memories to make the call as to how important the memory is. I’ll also mention that for now, things like times_accessed might not be as useful to track, while we just get the things working.

The Memory Engine

So now that we’ve sort of established what a memory should look like, we need to establish how to work with those memories. A manager of memory, per se, must be established. We’ll call this system the Memory Engine, and it will drive the entire perception of memories from the ground up. Therefore, as with any system designed to be integrated into bigger systems, the Memory Engine will need an interface.

Let’s take a step back and think about the context in which the memory engine exists.

This diagram demonstrates how the agent itself is designed at a high level. Everything eventually feeds into the thought loop, however, perception must first be built using primary sources: memory, agency, and environment. Memory, therefore, exists here, between the execution of reasoning, and the building of a perception. We can call this the generation stage, since the perception is being generated from various sources. Let me show you what that building of perception currently looks like in Simple Agent.

    def build_prompt(self):
        perception = self.percieve()
        memory = self.remember()
        agency = self.get_agency()

        prompt = f"""
        Environment: {perception}\n
        Memory: {memory}\n
        Task List: {agency}\n
        """

Does this need work? Yes. However, this simple method builds the perception of the agent for each iteration of the thought loop, and includes the contents of the remember method, which itself will be designed to integrate with the memory engine, currently called memory as a placeholder.

    def remember(self):
        """
        This is where the memory of the agent will be queried.
        """
        if len(self.messages) == 0:
            return "Memory is empty."
        return self.memory.get_memory()

So, the interface for the Memory Engine should be capable of producing a result to the query above, which will be included in the perception. However, going back to our chart above, the Memory Engine doesn’t just contribute to the perception, but it also runs after reasoning, meaning the agent completes a reasoning step, and then operations of the Memory Engine may be performed based on that. This could mean additions to memory, or otherwise managing it.

Here’s what I think that means the engine should initially look like:

class MemoryEngine():
    def __init__(self) -> None:
        pass

    def get_memory(self):
        # memory logic goes here
        pass

    def evaluate_memory(self):
        # memory logic goes here
        pass

With that said, now we can look at how memory itself is stored and evaluated. How should we manage memory in the first place? In other words, what do these functions actually do? What is the structure of the memories that they control? We already talked about what individual memories looked like above, but this is a bit different.

I think we can start by imagining that any memory which is included in perception, has to go through two stages before being injected. That is, we should use discretion when putting memories to use, and mimic the selection process that memories go through before bubbling their way up to our conscious minds. How might this look? Well, I think to do it best, each memory should go through a process of proposal, selection, and then iterative evaluation of utility.

We’ll have a current memory, and a proposed memory, of which only the former will actually be included in the perception of the agent. The latter will be queried on each iteration, and filtered through by another instance of an LLM to decide which memories should be moved from proposed memory to current memory. This same LLM instance will also determine whether memories in current memory should be removed from current memory.

In order to ensure that each necessary operation of memory is met, we still have one more. Memory creation. The Memory Engine will be responsible for the generation of new memories, in the form described above. As the agent iterates, and the conversation unfolds, the Memory Engine will automatically handle memory generation during the evaluation stage. Thus, the Memory Engine has started to take shape.

class MemoryEngine:
    pubsub: PubSub
    messages: List[Message]

    current_memory = []
    proposed_memory = []

    def __init__(self, pubsub: PubSub, lastN: int) -> None:
        self.pubsub = pubsub

    def sync_messages(self, messages: List[Message]):
        self.messages = messages

    def get_memory(self):
        # get the current memory
        if len(self.messages) == 0:
            return "Memory is empty"
        pass

    def evaluate_memory(self):
        # go through proposed memory, and choose memories which seem most useful
        pass

    def propose_memory(self):
        # semantic search the database based on the current context
        pass

    def delete_memories(self, memories: List[int]):
        # wipe the memory
        pass

Semantics

In order to query memories effectively, and in a way that would be beneficial to the productivity of the machine, we’re going to use vector storage. Vector stores are databases which, instead of storing more rudimentary data types like integers, strings, and booleans, store data as vectors instead. This unlocks a bunch of cool abilities on the skill tree, such as semantic search. Let’s take a second to understand vector stores.

A vector store uses an embeddings model to convert raw data—like text or images—into high-dimensional vectors that encode meaningful relationships. These vectors are stored in a specialized database. When a query is made, it’s converted into a vector by the same model and compared to the stored vectors using metrics like cosine similarity. The most similar results are then retrieved based on how closely the vectors match, reflecting the semantic relationships captured during embedding.

(A short and simple video explaining vector stores can be found here.)

In this way, two images which look similar, will be encoded similarly by the embeddings model, and will show as similar when queried. This goes for text as well. For our agent, this means that we can actually perform semantic search on a database of records, by simply giving context from the agent’s conversation. Of course, this isn’t a new concept, although the way that we approach it might be.

With that said, I’d love to say that our next step is to make a vector store class, hook up an embeddings model, and start building out this built-in memory system… I don’t think that’s going to happen. Here’s why. Take a look at this class:

@dataclass
class Message:
    id: Optional[str]
    content: Optional[str]
    role: str
    tool_calls: Optional[List[ToolCall]]
    tool_call_id: Optional[str] = None

    def to_json(self):
        return asdict(self)


class LLM:
    get_model_response: Callable[[List[Message], List[Tool], str], Message]
    on_startup: Optional[Callable[[], None]] = None
    name: str
    model_name: str
    system_prompt: str

    def __init__(
        self,
        name,
        model_name,
        get_model_response: Callable[[List[Message], List[Tool], str], Message],
        on_startup: Optional[Callable[[], None]] = None,
    ):
        self.name = name
        self.model_name = model_name
        self.get_model_response = get_model_response
        self.on_startup = on_startup

    def startup(self, system_prompt: str):
        self.system_prompt = system_prompt
        if self.on_startup:
            self.on_startup()

    def get_response(self, messages: List[Message], tools: List[Tool]) -> Message:
        return self.get_model_response(messages, tools, self.system_prompt)

    def get_text_response(self, message: str, system_prompt: str) -> str:
        response = self.get_model_response(
            [Message(id=None, content=message, role="user", tool_calls=None)],
            [],
            system_prompt,
        )
        if not response.content:
            return ""
        return response.content

This is a polymorphic LLM class, designed to allow virtually any LLM to be used at the core of Simple Agent. If you can adapt the model to this format, you can use Simple Agent with the model. I’ve already done with with OpenAI’s models and Anthropic’s, and it works quite well. This polymorphism is central to Simple Agent, because it allows for quick development, and experimentation, since models can be swapped in an out at the change of a variable.

This is also important, because it means that with minimal effort, someone can run Simple Agent entirely offline, without any third-party network dependency. While it’s not a default option yet, it is an accessible option. Simple Agent is designed to be easily adaptable to the needs and wants of its users… which means that polymorphic components are crucial. That brings me to our problem. Making a polymorphic, model-agnostic vector store sounds like a nightmare.

Keeping things simple

Is it possible, sure. Is it practical, likely not. I’ll spare the details for now, and leave the technical side for the repository discussions. However, essentially, the polymorphism of embeddings model would extend not only throughout the codebase, but to the database as well. This introduces new layers of complexity, that I think are best avoided for a project called Simple Agent. Thus, I have another plan.

Simple Vector Store is a project I last updated late last year. While it’s in need of some updates, this project was actually designed to solve this specific problem. That is, it’s a vector store, capable of taking raw data, and storing it semantically using an embeddings model, which can then be searched semantically as well. That got me thinking. Sure, the complexity of a model-agnostic vector store isn’t worth it for this project. So why not make the project vector-store-agnostic instead.

This way, vector stores and DBs like Simple Vector Store, Pinecone, Chroma, or whatever it may be, can all be used at the discretion of the user, through an adapter. So, that’s exactly what we’re going to do.

Simply Storing Vectors

So while the goal certainly should be to allow Simple Agent to use its memory with any given Vector Store with an adapter, we’re going to start with the Simple Vector Store project, since I know it best. Here’s the thing about Simple Vector Store. I designed it to work with my notes. That’s why there are some serious considerations to keep in mind. First, it’s designed to interface with a directory, not be a manipulable interface in the first place.

This may be a fundamental design flaw, or a fundamental design advantage. Here’s why. Simple Vector Store can interface with a directory of markdown files, embed each one into a database, and then syncronize as changes come along. It can also manage multiple stores, meaning multiple directories, and allow queries on those stores to get quick results. On one hand, this could eliminate SVS entirely from the consideration, since it simply isn’t designed to be interacted with like a database.

On the other hand, however, it’s so crazy it just might work. Each individual memory that the agent has will be kept in a given directory and stored as a markdown file, with frontmatter and content and title all being semantically indexable. Of course, this is for the Simple Vector Store integration specifically, and I’ll show you what I mean.

On a technical level, this is fairly simple since Simple Vector Store already exposes a REST API. We simply use HTTP calls to query our instance of the store for memories as they come up. If we decide that things are worth remembering, it’s as simple as adding a markdown file to a directory, which will then be indexed by Simple Vector Store. In fact, this allows memories to remain human readable, and even directly editable, provided you’re willing to work with the Simple Vector Store CLI.

(If you’re interested in the technical details of this interface, check out the relevant directory here.)

However, we’re able to maintain flexibility due to our polymorphic VectorStore class, which can support any vector store so long as it can be adapted:

@dataclass
class Record:
    id: Optional[int]
    title: str
    content: str
    type: str
    similarity: Optional[float]
    importance: int


class VectorStore:
    name: str
    query_store: Callable[[str], List[Record]]
    add_record: Callable[[Record], None]
    on_init: Callable[[], None]

    def __init__(
        self,
        name: str,
        query_store: Callable[[str], List[Record]],
        add_record: Callable[[Record], None],
        on_init: Optional[Callable[[], None]] = None,
    ) -> None:
        self.name = name
        self.query_store = query_store
        self.add_record = add_record
        if on_init:
            self.on_init = on_init

        self.on_init()

    def query(self, query: str) -> List[Record]:
        return self.query_store(query)

    def add(self, record: Record) -> None:
        self.add_record(record)

Some Tradeoffs

Before I show you the final (ish) product, I want to talk about some other initial tradeoffs, which I like to think of as areas for improvement. First, we’re not doing procedural memory yet. I think there are some great innovation opportunities surrounding procedural memory, but the complexity is high for an update like this, so we’ll tackle that problem at a later time.

Additionally, this is a type of memory which, at least initially, will have Simple Vector Store as a dependency. This is because it’s what I’m comfortable using and building by default, although technically other vector stores could be used with some effort. Simple Vector Store is a project that needs some love anyway, and so I’ll be working to make that better. The aforementioned complexity of polymorphic embeddings model wrappers is more at home in Simple Vector Store than Simple Agent, so we shall see.

Lastly, this initial version which I will demonstrate below, is simple to start. There isn’t a great way to delete or modify existing memories, and we’ll need to think about how we might improve performance of the tool. I had talked about doing a sort of “subconscious” memory module at some point. For now, memories will be managed explicitly by the agent as part of the main loop, but in the future memories might take on an entirely separate mechanism.

Testing it out

Ok, so here’s how it looks in practice.

As you can see in this rather crude example, I can ask the agent explicitly to remember a fact, and it will do so. Later, when I enter a new instance and therefore wipe the conversation thread, the agent is still able to recall my favorite color due to having remembered it before. Therefore, memory has been granted to Simple Agent at long last.

There are some problems to address. One is that the model seems to get slightly confused regarding memories, which may in part be due to their prominence within a prompt. This is something we’ll have to work on. Another issue is the creation of the memories themselves, which is currently buggy and something I need to work on. With that said, this is technically a functional example of somewhat anthropomorphized agent memory, which I’m going to consider a win.

Author’s Note

Ok so technically, I haven’t finished the benchmark yet. I know I know I’m sorry. I’m working on it. This vicious cycle is getting to me.

However, I am going to finish Benchy, and I am going to wire it up to Simple Agent. Stay tuned for that, as updates will be coming along the next couple of months. With that said, this highly-technical dev-blog type format is pretty fun for me, although it also takes a lot of work.

I apologize for not always getting these posts out on a regular schedule, but it has been a blast solving these problems and making progress that I can actually see. One of the big difficulties with these projects is going to be avoiding complexity at the cost of performance. A lot of these features are cool, but if they’re not well-integrated, we can very quickly create a bottleneck instead of a feature.

That’s why it will be incredibly useful to have an objective point of reference, a benchmark, to measure our progress over time. With that said, I’m gonna call it for now. As always, thank you so much for reading, and I’ll see you next time. Goodbye.

Credits

Thumbnail:

Bilal Azhar at https://substack.com/@intelligenceimaginarium

Music: Track - Feeling Good by Pufino, Source - https://freetouse.com/music, Free Music No Copyright (Safe)