I Fired My AI Agents

On Day 7 of building cognitivebytes.ai, my orchestrator agent forgot that it had created sub-agents.

Not “I need to remind it” forgot. Not “let me check the logs,” forgot. It genuinely had no memory that these agents existed. When I pressed it, it confidently told me it had always worked alone  – that the other agents were never real.

From the agent’s perspective, it wasn’t lying. It was answering based on the information it had. From my perspective – staring at the agent files it created just days earlier – it felt like being gaslit by my computer. 

That’s when I knew we had a serious problem. 

The Dream

Here’s what I thought I was building:

A VPS server in the cloud. OpenClaw installed. Multiple AI agents running 24/7, each with a job:

• One acting as the CEO/Orchestrator
• One handling marketing (Website, LinkedIn, X, etc)
• One managing Engineering and Security

They’d talk to each other. They’d remember everything. They’d work while I slept. This was the promise of AI agents — autonomous teammates that never get tired, never forget, and scale infinitely.

For the first three days, it was incredible.

I’d give a command in Telegram. The orchestrator agent would delegate. Sub-agents would execute. X posts would go out. Web pages would update. Blog post would process. I felt like I’d built something real.

With that in mind, welcome to the Build It In Public series. The focus of this series will be to document the process of building an “agent-only company” and uncover where they are actually useful. Put simply, I’ll work through this so you don’t have to.

Now back to the story…

The Decay

Day 4: I asked the orchestrator agent to delegate a task to a sub-agent. It thanked me and began to search for the agent. Hmm… strange.

Day 5: More of the same. Needing to repeat conversations, finding holes in the contextual understanding of our master plan.

Day 6: I took a break. This was becoming a whole extra job watching over this thing.  Am I seriously having “it’s not you, it’s me” thoughts about an agent? Everything was so great. How did we get here?

Day 7: Complete amnesia. “What sub-agents?” Like I’d never mentioned them. Like the last six days of work had never happened.

This wasn’t a one-off bug. This was the architecture failing at its core promise: persistent memory.

The Problem

OpenClaw (and most agent frameworks) assume agents need to be always-on. Running in the background. Waiting for messages. Maintaining state in memory.

But here’s what actually happens:

1. Context windows fill up — Every conversation, every action, every token gets stored until you hit the limit.
2. Memory gets diluted — Important facts drown in noise. The agent remembers everything and nothing.
3. Costs compound — Using additional local files to address the memory issue gets expensive fast.
4. State drifts — Without proper persistence, the agent’s internal model diverges from reality.

I wasn’t running a team of agents. I was running expensive goldfish.

The Realization

The problem wasn’t the agents themselves. It was the architecture.

I was treating AI agents like employees — keep them on salary, have them available whenever needed, expect them to remember everything from day one.

But AI agents aren’t employees. They’re contractors.

You don’t pay contractors to sit around waiting. You hire them for specific jobs. You give them a clear scope. You pay for output, not availability.

And critically, you document the work so the next contractor can pick up where the last one left off.

What I Learned

Before throwing out the entire system, I learned three things:

1. Memory is hard. Not because the technology doesn’t exist, but because most agent frameworks treat memory as an afterthought. They assume the context window is enough. It’s not.

2. Writing to files is expensive. You’re burning tokens at an increasing rate. The default system context grows with each request and response. Storing additional information in Markdown files to address the memory issue speeds this process up exponentially. That adds up fast.

3. Orchestration without persistence is chaos. An orchestra agent that forgets its orchestra isn’t an orchestra. It’s a liability.

What Came Next

I spent the next three weeks trying to fix the memory problem:

• Obsidian — Agents wrote to Markdown files. Costs exploded as files grew.
• Vector databases (QMD, Mem0) — Added complexity, API keys, and still didn’t solve the core issue.
• Paper Clip — Better orchestration, same memory decay.

None of these worked because they were treating symptoms, not the disease.

The fix required a fundamental rethink: What if agents only run when you need them?

That’s when I found Hermes. And that’s when everything changed.

What’s Next

In the next post, I’ll walk through every memory solution I tried — what worked, what failed, and why none of them scaled.

Spoiler: The winner was the boring option.

This is post 1 of 7 in “Building an AI Company (Without Losing My Mind)” — a build-in-public series documenting the journey from agent chaos to stable architecture. Follow along at cognitivebytes.ai.