Fast retrieval. Higher accuracy. Lower cost.
memU compiles conversations, documents, code, images, audio, video, URLs, and tool traces into human-readable Markdown files (INDEX.md, MEMORY.md, SKILL.md). Agents traverse the tree and load only what the moment needs — instead of rescanning everything or stuffing long histories into every prompt.
await service.memorize(resource_url="workspace/meeting-notes.md", modality="document")
context = await service.retrieve(
queries=[{"role": "user", "content": {"text": "What should I know about this user's launch preferences?"}}],
where={"user_id": "123"},
)That's it. Instead of one giant prompt about a person or their workspace, your agent gets three durable layers it can traverse:
workspace/
├── INDEX.md ← Index: a map of everything — raw sources and summaries
├── MEMORY.md ← Memory: an overview that links into memory/
├── SKILL.md ← Skill: an overview that links into skill/
├── resource/ ← the raw source files, copied verbatim
├── memory/
│ └── <topic>.md ← one memory file per topic: facts, preferences, goals, events
└── skill/
└── <name>.md ← one skill file per learned pattern, workflow, or mistake to avoid- Index (
INDEX.md) — a map of your memories: what exists, where it came from, and where to look first - Memory (
MEMORY.md) — personal facts, preferences, goals, events, and decisions extracted from source data - Skill (
SKILL.md) — auto-extracted from tool traces and refined on everymemorize()so the agent improves at recurring tasks
Three things make it different from stuffing everything into the prompt:
- Fast retrieval — walk to the right folder and rank the right files instead of scanning everything every time.
- Higher accuracy — scope by user, task, or session, and trace every item back to the exact conversation, document, image, or log it came from.
- Lower cost — retrieve compact, scoped context instead of reinjecting long histories, documents, logs, and media-derived text into every prompt.
- Yours to inspect — a human-readable file tree you can audit, edit, scope, and route through your own storage (
inmemory,sqlite,postgres) and LLM providers.
If you find memU useful or interesting, a GitHub Star ⭐️ would be greatly appreciated.
| Capability | Description |
|---|---|
| 🗂️ Multimodal Ingestion | Write conversations, documents, images, video, audio, URLs, logs, and local files into memory |
| 📁 Compiled Memory Workspace | Persist the Index, Skill, and Memory layers — folders (categories), files (items), source artifacts, links, summaries, and embeddings |
| 🧠 Typed Memory Extraction | Extract profile, event, knowledge, behavior, skill, and tool memories from raw sources |
| 🛠️ Self-Evolving Skills | Auto-extract reusable tool patterns and workflows from tool traces, then merge and refine them on every memorize() instead of relearning |
| 🧭 Self-Organizing Folders | Auto-build categories, links, summaries, and embeddings without manual tagging |
| 🤖 Agent-Ready Retrieval | Find the right files quickly — scoped, ranked context for any agent workflow |
| 🧱 Pluggable Storage | Use in-memory, SQLite, or Postgres backends with the same repository contracts |
| 🔀 Profile-Based LLM Routing | Route chat, embedding, vision, and transcription work through configurable LLM profiles |
Turn chat logs and workspace notes into user preferences, goals, events, decisions, and relationship context.
await service.memorize(
resource_url="examples/resources/conversations/conv1.json",
modality="conversation",
user={"user_id": "123"},
)
context = await service.retrieve(
queries=[{"role": "user", "content": {"text": "What should I remember about this user?"}}],
where={"user_id": "123"},
)Convert docs, PR notes, logs, and design decisions into reusable project memory.
await service.memorize(resource_url="docs/architecture.md", modality="document")
await service.memorize(resource_url="examples/resources/logs/log1.txt", modality="document")
context = await service.retrieve(
queries=[{"role": "user", "content": {"text": "How should I structure this module?"}}],
)Extract searchable facts from documents, screenshots, images, videos, and audio notes.
await service.memorize(resource_url="examples/resources/docs/doc1.txt", modality="document")
# Rich documents (PDF/Word/PowerPoint/Excel/HTML) are converted to Markdown via
# MarkItDown — install the extra with: pip install 'memu-py[document]'
await service.memorize(resource_url="reports/q3-summary.pdf", modality="document")
await service.memorize(resource_url="examples/resources/images/image1.png", modality="image")
# Audio is supported for your own .mp3/.wav/.m4a files.
await service.memorize(resource_url="meeting-audio.mp3", modality="audio")
context = await service.retrieve(
queries=[{"role": "user", "content": {"text": "What matters for the next research plan?"}}],
)Turn execution traces into tool memories that tell future agents when to use a tool and what mistakes to avoid.
await service.memorize(resource_url="examples/resources/logs/log1.txt", modality="document")
context = await service.retrieve(
queries=[{"role": "user", "content": {"text": "Which tools worked for config editing?"}}],
)The compiled workspace is hierarchical enough for browsing and structured enough for direct retrieval:
| Layer | Primary Role | Retrieval Role |
|---|---|---|
| Category (folder) | Maintain topic-level summaries | Assemble compact context for broad queries |
| Item (file) | Store typed atomic memories | Load precise facts, events, preferences, skills, and tool patterns |
| Resource (source) | Preserve source artifacts and captions | Recall original context when item/category summaries are not enough |
See docs/architecture.md for the runtime view of MemoryService, workflow pipelines, storage backends, and LLM routing.
👉 memu.so — Hosted API for managed ingestion, structured memory, and retrieval
For enterprise deployment: [email protected]
| Base URL | https://api.memu.so |
|---|---|
| Auth | Authorization: Bearer <token> |
| Method | Endpoint | Description |
|---|---|---|
POST |
/api/v3/memory/memorize |
Ingest raw data and build structured memory |
GET |
/api/v3/memory/memorize/status/{task_id} |
Check processing status |
POST |
/api/v3/memory/categories |
List auto-generated categories |
POST |
/api/v3/memory/retrieve |
Query memory for agent context |
From a clone of this repository:
uv sync
# or, for the full development setup:
make installTo install the published package instead:
pip install memu-pyRequirements: Python 3.13+. The default examples use OpenAI, so set
OPENAI_API_KEYor pass another provider throughllm_profiles.
Run an in-memory smoke script:
export OPENAI_API_KEY=your_key
cd tests
uv run python test_inmemory.pyRun with PostgreSQL + pgvector:
uv sync --extra postgres
docker run -d --name memu-postgres \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_DB=memu \
-p 5432:5432 \
pgvector/pgvector:pg16
export OPENAI_API_KEY=your_key
export POSTGRES_DSN=postgresql+psycopg://postgres:[email protected]:5432/memu
cd tests
uv run python test_postgres.pyfrom memu import MemUService
service = MemUService(
llm_profiles={
"default": {
"base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
"api_key": "your_key",
"chat_model": "qwen3-max",
"client_backend": "sdk"
},
"embedding": {
"base_url": "https://api.voyageai.com/v1",
"api_key": "your_key",
"embed_model": "voyage-3.5-lite"
}
},
)from memu import MemoryService
service = MemoryService(
llm_profiles={
"default": {
"provider": "openrouter",
"client_backend": "httpx",
"base_url": "https://openrouter.ai",
"api_key": "your_key",
"chat_model": "anthropic/claude-3.5-sonnet",
"embed_model": "openai/text-embedding-3-small",
},
},
database_config={"metadata_store": {"provider": "inmemory"}},
)
result = await service.memorize(
resource_url="path/to/file.json", # local file path or HTTP URL
modality="conversation", # conversation | document | image | video | audio
user={"user_id": "123"}, # optional: scope to a user or agent
)
# Returns after processing completes:
# { "resource": {...}, "items": [...], "categories": [...], "relations": [...] }- Converts raw input into typed memory items
- Categorizes and embeds items without manual tagging
- Preserves source resources and item-category relations
# The retrieval strategy is set once on the service via retrieve_config:
# MemoryService(retrieve_config={"method": "rag"}) # vector-first recall
# MemoryService(retrieve_config={"method": "llm"}) # LLM-ranked recall
result = await service.retrieve(
queries=[{"role": "user", "content": {"text": "What are their preferences?"}}],
where={"user_id": "123"}, # scope filter
)
# Returns:
# {
# "needs_retrieval": true,
# "original_query": "...",
# "rewritten_query": "...",
# "next_step_query": "...",
# "categories": [...],
# "items": [...],
# "resources": [...]
# }retrieve_config.method |
Behavior | Cost | Best For |
|---|---|---|---|
rag |
Vector-first category/item/resource recall, with optional LLM routing and sufficiency checks enabled by default | Embeddings plus LLM calls unless route_intention and sufficiency_check are disabled |
Fast scoped recall with controllable reasoning |
llm |
LLM-ranked category/item/resource recall | LLM ranking at each tier | Deeper semantic ranking |
export OPENAI_API_KEY=your_key
uv run python examples/example_1_conversation_memory.pyAutomatically extracts preferences, builds relationship models, and surfaces relevant context in future conversations.
uv run python examples/example_2_skill_extraction.pyMonitors agent actions, identifies patterns in successes and failures, auto-generates skill guides from experience.
uv run python examples/example_3_multimodal_memory.pyCross-references text, images, and documents automatically into a unified memory layer.
memU achieves 92.09% average accuracy on the Locomo benchmark across all reasoning tasks.
View detailed results: memU-experiment
| Repository | Description |
|---|---|
| memU | Personal memory as files — fast retrieval, higher accuracy, lower cost |
| memU-server | Backend with real-time sync and webhook triggers |
| memU-ui | Visual dashboard for browsing and monitoring memory |
Quick Links:
# Fork and clone
git clone https://github.com/YOUR_USERNAME/memU.git
cd memU
# Install dev dependencies
make install
# Run quality checks before submitting
make checkSee CONTRIBUTING.md for full guidelines.
Prerequisites: Python 3.13+, uv, Git
- GitHub Issues: Report bugs & request features
- Discord: Join the community
- X (Twitter): Follow @memU_ai
- Contact: [email protected]
⭐ Star us on GitHub to get notified about new releases!