Self-improving agents need to forget too. A memory primitive for Hermes Agent.

This is a submission for the Hermes Agent Challenge

Hermes Agent is good at remembering. The harder problem is deciding what to forget.

I'm writing this as someone who shipped a small open-source memory library for LLM agents on the same day this challenge opened (2026-05-15). The library is called agentmemory and it was designed to slot in next to systems like Hermes. This post is about why a self-improving agent specifically benefits from a memory layer that takes deletion seriously.

What Hermes Agent already does well

The pitch is in the tagline: an open-source agentic system, runs on your own infrastructure, plans, uses tools, reasons across multi-step tasks. The "self-improving" angle (the model gets better at your work over time) is the part everyone notices. The part underneath is just as interesting: it has to keep track of what worked, what failed, and what the user said.

That accumulation is the whole point. It's also the long-term risk.

The unspoken cost of accumulation

Every self-improving system has the same shape:

1. Try something
2. Observe the outcome
3. Store the outcome so the next try is better

Hermes does this well at the per-task level. The harder question is the multi-month one. After the agent has been running on your laptop for a quarter, what does its memory look like?

A few real failure modes I've watched in similar setups:

• Stale preferences. "I prefer Postgres" said in March still routes the agent away from MongoDB in October even though the project has changed.
• Tombstoned mistakes. The model "remembers" a wrong tool call from a year ago and will confidently quote it as a precedent.
• Compliance drift. The user asked the agent to "forget that conversation" three months ago. The conversation was deleted, the consolidated summary that mentions it was not.

These are not bugs in Hermes specifically. They're the natural consequence of background memory consolidation, which is the dominant pattern in agentic memory right now (Anthropic shipped Dreaming on May 6 along this same line). Once you bake an episode into a derived artifact, undoing the bake is hard.

What I'd want a memory layer to give Hermes

Five rules. They sound obvious until you check whether your stack actually follows them:

1. No background work. Memory should not consolidate while the agent is idle. The user can't audit what they can't see happen.
2. Real deletes. When a user says "forget X," X is gone. No tombstone, no derived artifact, no summary that still encodes the gist.
3. Pull, never push. The agent retrieves memories explicitly, by calling the memory layer. Memory does not silently inject itself into the prompt.
4. Show the trace. Every memory injection comes with the list of event ids and the exact prompt that produced the summary. The user can audit before the model ever sees the context.
5. Bring your own LLM. The summarizer is the same model the agent uses, or one you choose. Not a separate, smaller model that quietly degrades quality.

These are the design rules I shipped agentmemory under. They are also, I'd argue, the right rules for a system that calls itself self-improving. Self-improvement that the user can't audit is just opaque drift.

What slotting it into Hermes Agent looks like

agentmemory ships three pieces:

• EpisodicStore: append-only event log, embedder-aware retrieval, real-delete API
• OnDemandSummarizer: pull-model context builder, BYO LLM, returns the trace
• MemoryDriftWatcher: rolling-window detector for retrieval-quality drops

A minimal wiring against Hermes-style agent loops looks like this:

import { EpisodicStore, OnDemandSummarizer } from "@mukundakatta/agentmemory";

const store = new EpisodicStore();

// Hook into Hermes' per-turn lifecycle: write what happened, retrieve when starting fresh.
function onTurnComplete({ sessionId, kind, text }) {
  store.append({ sessionId, kind, text });
}

async function buildContextForNewSession(intent, sessionId) {
  const events = await store.retrieve(intent, { sessionId, topK: 5 });
  const { summary, trace } = await summarizer.summarize(events, intent);

  // Show the trace BEFORE injection. This is the audit point.
  console.log("Memory trace:", trace.eventIds);
  console.log("Summary:", summary);

  return summary;
}

// User says "forget that conversation" — real delete.
function onForgetRequest(eventId) {
  store.deleteEvent(eventId);  // Gone. No tombstone.
}

This is intentionally not a framework. Three small classes, zero runtime dependencies, written in plain ESM. It is meant to be vendored into the harness around Hermes (or any other agent) without dragging an opinion in with it.

The trade-off, honestly

Pull-model memory has a real cost. The first message of a new session pays a 200ms-2s tax for the on-demand summary. Background consolidation systems hide that cost by moving it off the critical path. The exchange you're getting:

• Latency goes up on cold start
• Context cost is fully visible (you can show the user what was retrieved)
• Deletes are real and immediate
• No quality decay from a smaller summarizer model
• The whole memory layer is < 500 lines, vendor-able

For a system like Hermes that the user runs on their own infrastructure (and that markets self-improvement), this exchange seems right. The user already accepts the cost of running the model locally. Adding 200ms to a new session in exchange for an auditable memory layer is a good deal.

What I'd build next, given more weeks

A HermesAdapter that hooks the three agentmemory pieces directly into the standard Hermes lifecycle events. A small CLI that lets you agentmemory list-sessions and agentmemory delete-session <id> from the terminal so the user has direct control. A drift dashboard that surfaces when retrieval quality is degrading on a specific Hermes deployment.

The bigger ask is upstream: I'd love to see Hermes' first-party memory expose a "show me the trace before injection" hook. Even if you don't use a separate library like agentmemory, the principle (the user can audit memory before it enters the prompt) belongs in the framework itself.

Closing

Hermes Agent does the agentic loop well. The memory question is what determines whether the system stays trustworthy at month six. The argument I'm making is narrow: build memory the user can see, can audit, can really delete. Whether you do that with agentmemory, with a different library, or with a few lines of your own code does not actually matter. What matters is that "self-improving" stops meaning "silently drifts in ways you can't reverse."

If you're already running Hermes locally, the agentmemory library is on npm and the GitHub repo has the live Streamlit demo link in the README. Both are MIT.

If you'd rather just take the design rules and write your own 100 lines: that's also a fine outcome. The library is small enough to read end-to-end in one sitting, and the rules are what matter.

Library: github.com/MukundaKatta/agentmemory (npm: @mukundakatta/agentmemory). Companion essay on a parallel topic (the OSS alternative to Anthropic Dreaming): Why I refused to build a Dreaming clone for OSS Claude.