Agent Memory as a File Format: Ditch the Pipelines, Use Markdown

Most AI benchmarks start agents from a blank context window. Real agents shouldn't. They should start with memories—relevant information accumulated from prior interactions. But current agent memory systems are over-engineered and under-deliver.

The Three Flavors of Broken Memory

Cal Paterson identifies three popular approaches, each flawed:

• Harness-bound: Tied to a specific vendor's platform, mining conversation history. Memories end up about the user, not the world.
• Over-complex: Systems requiring pgvector, Neo4j, and a dedicated LLM just to decide what's worth remembering. They confuse the model and don't scale with the frontier.
• High-Modernist: Graph-based, rationalist memory that strips context, leaving isolated 'facts' that are senseless to the agent.

Common thread: treating memory as a process. Paterson argues it should be a data format.

Memoryfield: A Portable File Format

A memoryfield is a zip archive containing Markdown pages (with optional YAML frontmatter) and an optional SQLite vector index for semantic search. That's it. No pipelines, no graphs, no vendor lock-in.

Prose over Chunks

RAG pipelines exist to make legacy documents legible to agents. But memories are written by agents, for agents—so let them write prose directly. No chunking, enrichment, or double-summarization. Pages have a soft limit of ~8KB (~2000 tokens) to fit vector embeddings, which is a reasonable constraint anyway.

Semantic Jump, Not Graph Walking

Karpathy wikis (hyperlinked Markdown files) require agents to traverse a knowledge graph, making serial tool calls. That's slow (N+1 calls for N levels deep), unreliable (agents judge relevance by link text, missing digressions), and noisy (irrelevant pages pollute context). Semantic search jumps directly to all relevant pages in parallel—at most 2 tool calls.

More Model, Less Mechanism

Complex APIs force agents to navigate interface mazes. A file format gives agents latitude to invent access patterns—using perl for bulk edits, embedding CSV files queried with SQLite. As models improve, they naturally write better memories. Fixed API endpoints can't do that.

Open Format, Interchangeable

The spec is RFC-style, avoiding ties to specific embedding functions. The canonical format is a zipfile, but it can be served from local files, S3, GitHub, or HTTP. Paterson uses Syncthing for personal memories, S3 for shared ones.

Getting Started

Pull the spec and vibe an implementation, or use the provided tooling: ollama pull nomic-embed-text, uv tool install git+https://github.com/calpaterson/memoryfield-tool, and npx skills add calpaterson/memoryfield-skill -g -y.

Memory as a file format—simple, portable, and model-agnostic. It's a refreshing antidote to the complexity creep in agent infrastructure.