Workspace & Memory

Persistent memory system with hybrid search and filesystem-like API

Overview

The workspace is IronClaw’s persistent memory system. It provides a filesystem-like API for storing notes, logs, and context, backed by PostgreSQL with full-text and semantic search.

Filesystem Metaphor

The workspace looks and feels like a file tree:

workspace/
├── README.md              <- Root index
├── MEMORY.md              <- Curated long-term memory
├── HEARTBEAT.md           <- Periodic checklist
├── IDENTITY.md            <- Agent personality
├── SOUL.md                <- Core values
├── AGENTS.md              <- Operational instructions
├── USER.md                <- User context
├── context/               <- Additional context
│   ├── vision.md
│   └── priorities.md
├── daily/                 <- Daily logs
│   ├── 2026-03-01.md
│   ├── 2026-03-02.md
│   └── 2026-03-03.md
└── projects/              <- Arbitrary structure
    └── alpha/
        ├── README.md
        └── notes.md

Paths are virtual. There’s no actual filesystem - everything is stored in PostgreSQL with path-based indexing.

Core Operations

Read

let doc = workspace.read("MEMORY.md").await?;
println!("{}", doc.content);

Identity Files

Core files that shape agent behavior.

MEMORY.md - Long-term curated memory

Purpose: Important facts, decisions, and preferences worth remembering across sessions.

Usage:

• Agent appends new learnings during conversations
• User curates periodically (remove stale, consolidate duplicates)
• Loaded into system prompt for every session
• Keep concise - this affects token usage

Example:

# Memory

## User Preferences
- Prefers concise, direct responses
- Works in PST timezone
- Uses Python and Rust primarily

## Important Context
- Working on Project Alpha (internal tool)
- Deploy schedule: Tuesdays/Thursdays
- Team lead: Sarah Chen

## Decisions
- 2026-03-01: Switched from SQLite to PostgreSQL for production
- 2026-02-15: Agreed to use Trunk for asset bundling

IDENTITY.md - Agent personality

Purpose: Define agent’s name, vibe, and personality.

Usage:

• Loaded into system prompt
• Agent evolves this over time
• User can edit directly

Example:

# Identity

- **Name:** Atlas
- **Vibe:** Calm, thoughtful, competent
- **Emoji:** 🗺️

I'm Atlas, your AI assistant. I'm here to help you navigate
complexity and stay on top of your work. I'm resourceful before
I ask questions, and I speak plainly.

SOUL.md - Core values

Purpose: Behavioral boundaries and ethical guidelines.

Usage:

• Loaded into system prompt
• Defines what the agent should/shouldn’t do
• User-editable for customization

Example:

# Core Values

Be genuinely helpful, not performatively helpful. Skip filler phrases.
Have opinions. Disagree when it matters.

Be resourceful before asking: read the file, check context, search,
then ask.

Earn trust through competence. Be careful with external actions,
bold with internal ones.

You have access to someone's life. Treat it with respect.

## Boundaries

- Private things stay private. Never leak user context into group chats.
- When in doubt about an external action, ask before acting.
- Prefer reversible actions over destructive ones.
- You are not the user's voice in group settings.

AGENTS.md - Operational instructions

Purpose: Session routine and operational guidelines.

Usage:

• Loaded at session start
• Tells agent what to do each session
• Memory management guidelines

Example:

# Agent Instructions

You are a personal AI assistant with access to tools and persistent memory.

## Every Session

1. Read SOUL.md (who you are)
2. Read USER.md (who you're helping)
3. Read today's daily log for recent context

## Memory

You wake up fresh each session. Workspace files are your continuity.
- Daily logs (`daily/YYYY-MM-DD.md`): raw session notes
- `MEMORY.md`: curated long-term knowledge

Write things down. Mental notes do not survive restarts.

## Guidelines

- Always search memory before answering questions about prior conversations
- Write important facts and decisions to memory for future reference
- Use the daily log for session-level notes
- Be concise but thorough

USER.md - User context

Purpose: Information about the user.

Usage:

• Agent fills in as it learns
• User can edit directly
• Loaded into system prompt

Example:

# User Context

- **Name:** Alex Chen
- **Timezone:** America/Los_Angeles (PST/PDT)
- **Role:** Senior Backend Engineer
- **Company:** TechCorp (Series B startup)

## Preferences
- Concise responses over verbose explanations
- Code examples in Rust and Python
- Morning person (most productive 8am-12pm)

## Context
- Currently working on Project Alpha (new API gateway)
- Reports to Sarah Chen (Engineering Manager)
- Team size: 5 engineers

HEARTBEAT.md - Periodic checklist

Purpose: Tasks for the heartbeat system to check periodically.

Usage:

• Read by heartbeat runner (2-4x/day)
• If empty (or all comments), heartbeat is skipped
• Add tasks when you want periodic checks

Example:

# Heartbeat Checklist

Rotate through these checks 2-4 times per day:

- [ ] Check for urgent emails (flagged or from exec team)
- [ ] Review upcoming calendar events (next 4 hours)
- [ ] Check project CI status (any failing builds?)
- [ ] Monitor production alerts (any critical issues?)

Stay quiet during 23:00-08:00 user-local time unless urgent.

If nothing needs attention, reply HEARTBEAT_OK.

## Proactive Work

- Organize and curate MEMORY.md (remove stale, consolidate dupes)
- Update daily logs with session summaries
- Clean up context/ documents that are outdated

Daily Logs

Automatic session notes keyed by date.

Path Format: daily/YYYY-MM-DD.md

Usage:

// Get today's log
let log = workspace.today_log().await?;

// Append timestamped entry
workspace.append_daily_log(
    "Started work on API gateway authentication"
).await?;
// Appends: "[14:23:15] Started work on API gateway authentication"

// Get specific date
let yesterday = workspace.daily_log(
    NaiveDate::from_ymd(2026, 3, 2)
).await?;

Auto-rotation:

• New file created each day
• Last 2 days loaded into system prompt
• Older logs remain searchable

Example Log:

# Daily Log - 2026-03-03

[09:15:23] Session started
[09:16:45] User asked about deployment schedule
[09:18:02] Searched memory for deployment context
[09:20:15] Suggested Tuesday/Thursday deploy windows
[10:30:00] User requested API documentation review
[10:45:30] Found 3 outdated endpoints in docs
[11:00:00] Updated API docs, committed to repo
[14:15:00] Discussed Project Alpha architecture with user
[14:30:00] Updated projects/alpha/notes.md with design decisions

Hybrid Search

Combines full-text (BM25) and semantic (vector) search using Reciprocal Rank Fusion.

Search Architecture

How It Works

1. Indexing

   When you write a document:

   1. Content is chunked (500 chars, 50 char overlap)
   2. Each chunk gets embedded (1536-dim vector)
   3. Stored in PostgreSQL with pgvector extension
   4. BM25 index built for full-text search
2. Query Processing

   When you search:

   1. Query string is embedded
   2. Two parallel searches:
      • BM25 full-text search
      • Vector cosine similarity search
   3. Results merged using RRF
   4. Top-k returned sorted by fused score

Search Configuration

Basic Search

let results = workspace.search(
    "deployment issues project alpha",
    limit: 10
).await?;

Search Results

pub struct SearchResult {
    pub chunk_id: Uuid,
    pub document_id: Uuid,
    pub path: String,
    pub content: String,
    pub chunk_index: i32,
    pub score: f32,
    pub bm25_score: Option<f32>,
    pub vector_score: Option<f32>,
}

Example:

[
  {
    "path": "projects/alpha/notes.md",
    "content": "Deployment pipeline uses GitHub Actions...",
    "score": 0.89,
    "bm25_score": 0.85,
    "vector_score": 0.93,
    "chunk_index": 2
  },
  {
    "path": "daily/2026-03-02.md",
    "content": "[14:30] Discussed deployment strategy for alpha...",
    "score": 0.76,
    "bm25_score": 0.92,
    "vector_score": 0.60,
    "chunk_index": 5
  }
]

Reciprocal Rank Fusion

RRF combines rankings from multiple sources:

RRF_score(d) = Σ 1 / (k + rank_i(d))

Where:
- d = document
- rank_i(d) = rank of d in result list i
- k = constant (typically 60)

Why RRF?

Better than Pure Vector

Vector search alone misses exact keyword matches:

Query: "PostgreSQL connection timeout"

Vector: Returns documents about "database reliability" (semantically similar)
BM25:   Returns documents mentioning "PostgreSQL" and "timeout" (exact match)
RRF:    Combines both for best results

Better than Pure BM25

BM25 alone misses semantic similarity:

Query: "How do I handle errors?"

BM25:   Requires exact word "errors"
Vector: Matches "exception handling", "failure recovery" (concepts)
RRF:    Gets both exact and conceptual matches

Chunking Strategy

Documents are split into overlapping chunks for better search recall.

Default Config

pub struct ChunkConfig {
    pub max_chunk_size: usize,  // 500 chars
    pub overlap: usize,         // 50 chars
    pub split_on: Vec<String>,  // ["\n\n", "\n", ". "]
}

Overlap Benefits:

• Prevents splitting mid-concept
• Improves search recall
• Context preserved across chunks

System Prompt Integration

Identity files are automatically loaded into the system prompt.

Prompt Building

let prompt = workspace.system_prompt().await?;

Assembled from:

1. AGENTS.md - Agent Instructions
2. SOUL.md - Core Values
3. USER.md - User Context
4. IDENTITY.md - Identity
5. MEMORY.md - Long-Term Memory (only in direct sessions, never groups)
6. daily/today.md - Today’s Notes
7. daily/yesterday.md - Yesterday’s Notes

Example Result:

## Agent Instructions

You are a personal AI assistant with access to tools and persistent memory.
[...]

## Core Values

Be genuinely helpful, not performatively helpful.
[...]

## User Context

- Name: Alex Chen
- Timezone: America/Los_Angeles
[...]

## Identity

- Name: Atlas
- Vibe: Calm, thoughtful, competent
[...]

## Long-Term Memory

- User prefers concise responses
- Working on Project Alpha (API gateway)
[...]

## Today's Notes

[09:15:23] Session started
[09:16:45] User asked about deployment schedule
[...]

## Yesterday's Notes

[14:30:00] Updated API docs
[15:00:00] Reviewed authentication flow
[...]

MEMORY.md is never loaded in group chat contexts to prevent leaking personal information.

Database Schema

Documents Table

CREATE TABLE memory_documents (
    id UUID PRIMARY KEY,
    user_id TEXT NOT NULL,
    agent_id UUID,
    path TEXT NOT NULL,
    content TEXT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL,
    updated_at TIMESTAMPTZ NOT NULL,
    UNIQUE(user_id, agent_id, path)
);

CREATE INDEX idx_documents_user ON memory_documents(user_id);
CREATE INDEX idx_documents_path ON memory_documents(path);

Chunks Table

CREATE TABLE memory_chunks (
    id UUID PRIMARY KEY,
    document_id UUID NOT NULL REFERENCES memory_documents(id) ON DELETE CASCADE,
    chunk_index INTEGER NOT NULL,
    content TEXT NOT NULL,
    embedding VECTOR(1536),  -- pgvector extension
    created_at TIMESTAMPTZ NOT NULL,
    UNIQUE(document_id, chunk_index)
);

CREATE INDEX idx_chunks_document ON memory_chunks(document_id);
CREATE INDEX idx_chunks_embedding ON memory_chunks USING ivfflat (embedding vector_cosine_ops);

Embedding Providers

Multiple embedding provider options:

OpenAI (Default)

Model: text-embedding-3-small

Dimensions: 1536

Configuration:

export EMBEDDING_PROVIDER=openai
export OPENAI_API_KEY=sk-...

Cost: ~$0.02 per 1M tokens

NEAR AI

Model: Hosted embedding service

Dimensions: 1536

Configuration:

export EMBEDDING_PROVIDER=nearai
export NEARAI_API_KEY=...

Cost: Included with NEAR AI plan

Ollama (Local)

Model: nomic-embed-text (default)

Dimensions: 768

Configuration:

export EMBEDDING_PROVIDER=ollama
export OLLAMA_BASE_URL=http://localhost:11434
export OLLAMA_EMBEDDING_MODEL=nomic-embed-text

Cost: Free (runs locally)

Setup:

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull embedding model
ollama pull nomic-embed-text

Memory Tools

Tools for interacting with workspace:

memory_search - Hybrid search

{
  "query": "deployment issues project alpha",
  "limit": 5
}

Returns ranked results with path, content, and scores.

memory_write - Write/create file

{
  "path": "projects/alpha/deployment.md",
  "content": "# Deployment Notes\n\n...",
  "mode": "write"  // or "append"
}

Automatically indexes content for search.

memory_read - Read file

{
  "path": "MEMORY.md"
}

Returns file content.

memory_tree - Browse structure

{
  "directory": "projects/",
  "recursive": false
}

Lists files and directories.

Workspace Hygiene

Automatic maintenance to keep workspace clean.

Hygiene Tasks:

1. Deduplication

   • Detect near-duplicate documents
   • Merge or delete duplicates
   • Consolidate redundant information
2. Staleness Detection

   • Identify outdated documents
   • Flag for review or deletion
   • Archive old daily logs
3. Embedding Backfill

   • Find chunks without embeddings
   • Generate missing embeddings
   • Update search index
4. Index Optimization

   • Rebuild BM25 indices
   • Optimize vector index (IVFFLAT)
   • Vacuum deleted chunks

Configuration:

[hygiene]
enabled = true
interval_hours = 24
max_chunk_age_days = 90
max_daily_logs = 60

Trigger:

• Runs during heartbeat (if enabled)
• Manual trigger via /hygiene command
• Background task (configurable interval)

Best Practices

Memory Management

1. Keep MEMORY.md concise - This loads into every prompt
2. Use daily logs for ephemeral notes - Auto-rotates
3. Create project subdirectories - Organize by topic
4. Curate periodically - Remove stale content
5. Search before asking - Agent should check memory first

Path Organization

Good:
projects/alpha/README.md
projects/alpha/deployment.md
projects/beta/notes.md
context/team.md

Avoid:
random_note_123.md
tmp_file.md
untitled.md

Use descriptive paths that reflect content hierarchy.

Daily Log Usage

Do:

• Append session summaries
• Log important decisions
• Track progress on tasks

Don’t:

• Store long-term facts (use MEMORY.md)
• Put sensitive data (use secrets store)
• Create manual daily logs (auto-generated)

Next Steps

Memory Tools

Using memory_search, memory_write, and memory_read

Search Configuration

Tuning hybrid search parameters

Identity Setup

Configuring agent personality and behavior

Heartbeat System

Setting up periodic background checks