Teaching AI to Remember: The Hardest Lesson Is Learning to Forget

You're deep in a conversation with an AI, and it suddenly forgets something you said ten minutes ago. Or the opposite: it keeps bringing up some throwaway detail you mentioned once and stopped caring about ages ago.

These look like opposite problems, but they have the same root cause: it can't tell what to remember and what to forget.

We've gone through many iterations of the memory system for CZL AI's conversation platform. At first, the idea seemed simple: save what users say and include it next time. Reality kept proving us wrong until we understood one thing: the hard part of giving AI a memory isn't remembering. It's forgetting just enough.

This post walks through the memory system we eventually built, and the times we thought we had each layer figured out, only to be proven wrong. No code, just the thinking behind it.

Why we had to build it, and who we borrowed from

First, why bother? The reason is simple: all the leading conversational AI products are doing it. ChatGPT has cross-conversation Memory, and Claude and Gemini are also moving toward remembering their users. The logic isn't hard to follow. An assistant that treats you like a stranger every time and forgets everything afterward always feels a little disconnected. You tell it today that you don't eat spicy food, then have to repeat yourself tomorrow. By the third time, you can't be bothered. So for us, memory isn't a flashy feature. It's the difference between a conversation product people keep using and one they don't.

As for how to build it, we didn't try to invent everything from scratch. The open-source community has already explored much of this territory.

Our most direct reference was the OpenClaw memory ecosystem (openclaw, along with projects around it such as memory-architecture, ClawMem, and Supermemory). They make one idea clear: memory for an agent isn't about throwing everything into one big database. It's about building layers. There's an always-present core profile, a fact store retrieved by semantic relevance, and even activation and decay mechanisms that let unused memories gradually sink out of view. That's where the thinking behind our layers came from.

The other influence was RAG. Its key technique is turning information into semantic coordinates, storing it, and retrieving it by relevance when needed. That's exactly the underlying logic of our "vector fact store." Basically, we adapted a mature RAG approach specifically to remember long-term facts about users.

The rest came from fitting these borrowed ideas into our product's real use cases, then working through one practical problem after another. Let's go layer by layer.

A watch

The lightest layer is so light it barely counts as memory. It stores nothing; everything is calculated for each conversation.

Its contents are simple: the current time, the day of the week, your time zone, and how long it's been since your last chat.

Don't underestimate this little watch. Many thoughtful touches in a conversation don't need a memory system at all. If you're still typing late at night, it can casually suggest getting some sleep. If you come back after two weeks, it can notice that it's been a while.

We keep these things in their own lightweight layer so they don't distort the layers that follow. They only matter right now. They'll be stale tomorrow, so they don't belong in long-term memory.

Where did we leave off?

Knowing someone isn't just about understanding them over the long term. It's also about remembering recent conversations. When you see a friend this week, you don't need to explain everything you discussed last week all over again.

AI needs that same sense of continuity. So after each conversation, the system quietly asks a small model to write and save a brief summary. Next time, summaries from the past few days come along, so it remembers what we've been working on lately.

There are two small details here.

First, not every conversation deserves a summary. If you ask a quick question and leave, what's there to summarize? So we set a threshold: conversations that are too short, lack substance, or have nothing worth summarizing according to the small model are skipped. Otherwise, memory would be full of useless lines like "The user asked a question."

Second, it eventually got smarter. At first, summaries were retrieved only by time, covering just the past few days. That led to an awkward situation: you brought up a project you'd discussed in depth half a year ago, and the AI had no idea what you meant because that summary was long past its shelf life. Later, we added relevance-based retrieval. Alongside recent summaries, it pulls a few older ones that best match what you're saying now. It remembers what you discussed two days ago, and it can also pick up a conversation from half a year ago when you return to it. We get both.

The "who you are" card

This layer looks the most like memory in the traditional sense: a card containing your core information, passed to the AI in full on every turn.

I'm a backend engineer. I prefer short answers. I'm working on a long-term project. Things that define who you are and need to stay in the AI's awareness belong here.

This layer gave us the hardest lessons and the most stories.

First, we let the AI take its own notes, and it stalled. The original idea sounded great: give the AI a pen, and let it jot down anything worth remembering as the conversation unfolds. Soon after launch, something strange happened. A user asked for outfit recommendations, and the AI returned a blank response.

We eventually worked out why. Once it decided to take a note first, it would stop and wait for feedback from the system before continuing. But our design at the time saved the note and left it at that, so the AI kept waiting and never gave its answer. The simple act of pulling out its notebook interrupted the user's request. We took the pen away at first and moved note-taking into the background. Later, we gave it the pen back with a different approach: after it saved a note, the system told it, "Saved, keep going," and it finished its response. Something as seemingly simple as "let the AI take notes" took several rounds of adjustment to make reliable.

Second, the card fills up. Since the entire card goes into every turn, it can't grow indefinitely. Too large, and every conversation becomes slow and expensive. But what happens when it's full? We tried several versions. The first was blunt: stop accepting new entries. Users immediately felt that the AI was forgetting things they'd told it. Later, we added two fallbacks. When the card fills up, a small model first condenses and merges its contents, expressing the same ideas more briefly. If the new entry still won't fit, it moves to the next layer instead of being discarded. Memories can be reorganized or moved, but they must not be turned away. That's how we arrived at that rule.

Third, a particularly nasty bug: Chinese-speaking users couldn't save anything even though the card looked far from full. The frontend displayed a progress bar based on character count, while the backend enforced a byte limit. One Chinese character counts as one character but takes up three bytes. So users saw a half-full card while the system had already hit its limit. It's the kind of trap you wouldn't think of until you fall into it.

Fourth, users need to understand the card. It's displayed in settings, where users can view and edit it themselves. For a while, though, we just listed everything without explaining which memories were always present and which were recalled only when needed. Users told us directly: "I don't understand what any of this is." Later, we gave each memory category a plain-language explanation and grouped entries by source: saved automatically by AI, added manually by you, or organized by the system. No matter how clever the internals are, if users can't understand them, the experience scores zero.

A warehouse you search only when needed

The card above is for a small number of high-value facts. But people have plenty of specific, scattered details that aren't relevant most of the time: that turtle was a black wild-type bought in 2024, or the itinerary from a previous trip. Putting these into a card included in every turn is wasteful, but throwing them away would be a shame.

This layer is their home. The principle isn't mysterious: turn each fact into a set of numerical coordinates representing its meaning, then store it in the warehouse. When you speak, the system checks how well your message matches the stored facts and temporarily retrieves only the most relevant ones. Talk about the turtle, and the turtle comes up. Otherwise, it stays quietly in a corner, taking up none of the AI's current attention.

This layer had its share of bumps too.

For a while, it was a temporary holding area, and things kept getting lost. We initially treated it as a draft folder: store things casually, then let a background process decide what to keep. The problem was that once cleanup removed an entry, it was gone for good. To users, it felt like the AI had forgotten something it used to remember. Later, we gave this layer a proper role as a long-term store. Facts go in to stay. They settle into the background because they're only retrieved when needed, not because we periodically clear the place out.

So who decides whether a piece of information belongs on the card or in the warehouse? The AI itself. It's there in the conversation and is best placed to judge whether something is fundamental to this person or just a peripheral detail. Occasional mistakes are fine; a background mechanism gradually corrects them.

Three safeguards behind the scenes

The four layers handle what to remember and where to put it. To keep the system reliable over time, remembering plenty, staying accurate, and avoiding bloat, we also need three safeguards that run across all layers.

The first is deduplication on write. Before a new fact enters the warehouse, we compare it with the closest existing match. If they're too similar, we merge them instead of creating a new entry. This catches most duplicates.

The second is background cleanup, or metabolism—the lifeblood of the system. Every so often, the system asks a small model to review all your memories: delete outdated ones, merge duplicates, promote frequently mentioned details to the core card, and demote stale core facts to the warehouse, where they can quietly gather dust. This cleanup is triggered by your activity. If you don't show up, it does nothing and costs nothing. The important part is that facts can move both ways. A fact can go from "recall when needed" to "always remember" because you mention it often, or move back down because it's no longer relevant. Memory isn't a warehouse where things only go in. It's a living system with its own metabolism. We've adjusted the parameters here countless times: how often to run cleanup, how similar two facts must be to count as duplicates, and which new facts need protection from accidental deletion. All of it was tuned bit by bit with real data.

The third is a capacity limit. Every layer has a ceiling. When it fills up, the least-used, least-noticed entries are evicted. But there's one firm rule: memories you add manually are always protected. The system never touches them.

Here's a fresh example. A user found two almost identical facts in their memory. They meant exactly the same thing; only an adjective had moved. We checked and found that both were within their protection period, either because they were too new or had recently been used. As a result, the cleanup process responsible for deduplication couldn't reach them at all. Once we understood the cause, the fix was straightforward. Protection exists to avoid deleting new facts before they've had a chance to prove useful. It shouldn't also protect duplicates. So we taught cleanup to compare entries side by side, recognize the same fact expressed differently, and merge them. There's an interesting distinction here: a numerical similarity check gets stuck on thresholds, but asking a small model in plain language whether two entries mean the same thing is much more accurate.

Final thoughts

The starting ideas were borrowed: OpenClaw's layers, RAG's semantic retrieval, and the user memory features all the leading products are building. These ideas aren't hard to find. The hard part came after launch, when real users exposed problems in every layer and every safeguard, one by one. That's what shaped the system we have now.

Looking back, storing information was never the difficult part. Disk space is plentiful. The difficulty is all on the other side: what to forget, what should make room, and what should sink into a corner until someone needs it. We're still tuning this, and we're nowhere near the finish line.

Whether it remembers, whether it's accurate, and whether it understands you better the more you talk—we'll keep exploring.