Proving that autonomous agents can outperform traditional database algorithms (maybe)

A research database exploring whether Blackboard coordination patterns (common in AI, Distributed, Electrical and analogue automation systems) can make LSM-trees simpler, smarter, and more adaptive than traditional approaches.

Traditional LSM databases:

Write → Lock → WAL → MemTable → Thread Pool → Compaction Scheduler → ...
(Tightly coupled, complex coordination, static algorithms)

Blackboard LSM (this project):

         BLACKBOARD (Shared State)
                 ↓
    ┌────────────┼────────────┐
    ↓            ↓            ↓
 Ingest       Flush       Compact
 Agent        Agent        Agent
(autonomous) (autonomous) (autonomous)

Each agent observes state and acts independently. No callbacks. No thread pools. No coordination hell.

Hypothesis: Autonomous agents can adapt to workloads better than hardcoded algorithms.

Problem: Modern databases use static algorithms

• Compaction triggers: if L0.size >= 4 { compact() } ← hardcoded
• Cache policies: LRU ← one size fits all
• Write buffering: Fixed batch sizes ← no adaptation

This project explores: What if agents learned and adapted instead?

• ML compaction agent predicts optimal timing
• Adaptive cache agent learns access patterns
• Workload-aware agents optimize for read vs write heavy loads

Goal: Enable storage systems that handle 1B+ token contexts for LLMs by being smarter, not just faster.

• to be added on a later basis

• ✅ Write-Ahead Log (durability)
• ✅ LSM-Tree structure (L0 → L1 compaction)
• ✅ Bloom filters (optimized reads)
• ✅ LRU cache (hot key acceleration)
• ✅ Agent-based coordination
• ⚠️ ML compaction (in progress)
• ⚠️ Adaptive caching (planned)

~40% - Works well, has known issues, actively improving

Not production-ready yet, but getting there.
Built for research first, production second.

git clone https://github.com/thecharge/sndv-kv
cd sndv-kv
go mod tidy

# Build
go build -o sndv-kv cmd/server/main.go

# Start with safe defaults (durability enabled)
./sndv-kv -config config_safe.json

# Or fast mode (in-memory, no fsync - for testing)
./sndv-kv -config config_fast.json

# The server prints an admin token on startup
# Copy it and use in requests

# Write
curl -X POST http://localhost:8080/put \
  -H "Authorization: YOUR_TOKEN" \
  -d '{"key": "user:1", "value": "Alice", "ttl": 3600}'

# Read
curl "http://localhost:8080/get?key=user:1" \
  -H "Authorization: YOUR_TOKEN"

---

## Architecture Deep Dive 🏗️

### The Blackboard Pattern

**Concept from AI:** Multiple expert agents collaborate through shared memory.

**Applied to Storage:**

```go
type Blackboard struct {
    // Shared State
    MemTable     *SwissTable
    ImmutableMem []*SwissTable
    ActiveWAL    *WAL
    SSTables     [][]SSTableMetadata
    
    // Coordination
    Mutex        sync.RWMutex
    FlushCond    *sync.Cond
}

Agents observe and act:

// Ingest Agent
for batch := range IngestQueue {
    bb.WAL.AppendBatch(batch)
    bb.MemTable.PutBatch(batch)
    
    if bb.MemTable.Size >= threshold {
        bb.FreezeMemTable()  // Signal other agents
    }
}

// Flush Agent  
for {
    wait_for_signal()
    frozen := bb.ImmutableMem[0]
    WriteSSTable(frozen)
    bb.RemoveImmutable()
}

// Compaction Agent
for {
    if bb.SSTables[0].Len() >= trigger {
        CompactL0toL1()
    }
}

No callbacks. No thread pools. Just agents reacting to state.

Example: Want smarter compaction?

• Traditional: Rewrite the scheduler, update thread pools, test everything
• Blackboard: Write new agent, deploy alongside old one, A/B test

This project is exploring:

1. Can ML agents beat static algorithms?

   • Predict optimal compaction timing based on workload
   • Expected improvement: 20-40% latency reduction

2. Do agents enable emergent optimization?

   • Can agents cooperate without explicit coordination?
   • Example: Flush agent learns compaction agent's patterns

3. Is Blackboard simpler than traditional approaches?

   • Measuring: Lines of code, cyclomatic complexity
   • Hypothesis: 30-50% less coordination code

4. Can this scale to 1B token contexts?

   • LLMs need massive KV cache storage
   • Agents could: compress, prefetch, evict intelligently

Benchmarks can be viewed separately in the folder structure. For now

1. Quick benchmark python3 quick_bench.py script is in place in order for you to see and get the feelign of the engine
2. A comperhensive multi stage build and test python bench_orchestrator.py is in place so you can see the full suite, tests and benches done When I habve time I will add more detailed explanations of the suites and their metrics.

Here is the last quick benchmark I ran: evidence for the last quick bench in the results.json in th eroot of the repository (Have that it may vary from PC to PC the bench orchestrator uses much more evidence and integration test data - but is docker and python bound as well as it will require much more time to run in future a seprate folder with version to version benchmarks and configurations will be in place - but for now I do not have time to polish and will wait until there is a time for production or someone decides to implement that in MR)

python .\quick_bench.py
======================================================================
SNDV-KV PERFORMANCE BENCHMARK
======================================================================
Building server...
✅ Build successful
Starting server...
✅ Server started

Running Single PUT Benchmark (2,000 items, 20 workers)...
  Progress: 500/2000
  Progress: 1000/2000
  Progress: 1500/2000
  Progress: 2000/2000
  → 522 TPS
     Latency: min=3.4ms, avg=38.1ms, p95=30.9ms, max=2057.1ms

Running Batch PUT Benchmark (20,000 items, batches of 100, 10 workers)...
  Progress: 50/200 batches
  Progress: 100/200 batches
  Progress: 150/200 batches
  Progress: 200/200 batches
  → 89966 TPS
     Batch latency: min=1.5ms, avg=10.5ms, p95=16.5ms, max=23.7ms

======================================================================
Single:        522 TPS
Batch:      89,966 TPS
======================================================================

✅ Saved results.json

We're slower than production systems. That's expected for:

• Research codebase vs production
• Go vs C/C++
• Novel architecture vs proven
• Solo developer vs teams

The goal isn't to beat RocksDB in performance.
The goal is to prove agents can be smarter.

Common question: "Why not use Rust/C++ for a database?"

Honest answer: Because I want to learn Go while proving the hypothesis.

Practical reasons:

1. Fast iteration - Go compiles in seconds, not minutes
2. Simple concurrency - Goroutines perfect for agent model
3. Good enough performance - 7K TPS is plenty for research
4. Easy to read - Research code should be understandable
5. Rich ecosystem - Good libraries for benchmarking, testing

Will I switch to Rust/C++ later?
Maybe. If the research proves Blackboard works, a production rewrite makes sense.

Right now: I'm riding the Go wave and learning while building.

Interested in agent-based storage systems? Let's collaborate!

Open research questions:

• How to train ML compaction agents?
• What metrics predict optimal compaction timing?
• Can agents learn workload patterns?

Want to learn LSM internals? Welcome!

Good first issues:

• Fix known bugs (see Issues)
• Add tests for critical paths
• Implement missing features

Think Blackboard is overkill? Prove me wrong!

I'm looking for:

• Benchmark comparisons
• Architecture critiques
• Performance bottleneck analysis

See: Discussions

• Basic LSM structure
• WAL with crash recovery
• Agent-based coordination
• Fix critical bugs
• Comprehensive test suite
• Honest benchmarks published

• ML compaction agent
• Adaptive cache agent
• Workload prediction
• A/B testing framework
• Paper: "Blackboard Architectures for LSM-Trees"

• 100K+ TPS sustained
• 1B token context support
• Agent marketplace (swap at runtime)
• Production deployment case study

• Rust rewrite (if research proves it)
• Multi-language bindings
• Cloud-native deployment
• Conference talks & papers

Being honest about limitations:

1. Performance: 15-20x slower than RocksDB (expected, being addressed)
2. Testing: Only ~30% code coverage (improving)
3. Production: Not ready for critical workloads yet
4. Documentation: Some internals not fully documented
5. Compaction: Only L0→L1, no multi-level yet

These are features, not bugs - they're the research agenda!

Simple > Complex

• Small autonomous agents beat complex coordinators
• Loose coupling beats tight coupling
• Observable state beats callback chains

Adaptive > Static

• ML agents beat hardcoded thresholds
• Learning systems beat fixed algorithms
• Workload-aware beats one-size-fits-all

Research > Perfection

• Ship experiments, measure results
• Fail fast, learn faster
• Prove concepts, then optimize

Honesty > Marketing

• Accurate benchmarks, not inflated numbers
• Known issues visible, not hidden
• Research progress tracked publicly

Q: Why Blackboard for databases?
A: 8 years of distributed systems convinced me tight coupling is the enemy. Blackboard enables loose coupling at scale.

Q: When will it be production-ready?
A: When it proves agents beat static algorithms. Performance comes after proof.

Q: Why not just use RocksDB?
A: RocksDB is amazing. This explores whether we can be smarter, not just faster.

Q: What about consistency/ACID?
A: Single-node strong consistency. Distributed ACID is future work.

Q: Can I use this for real projects?
A: For learning/research: yes. For production: wait for v1.0.

Q: How can I help?
A: Research collaboration, code review, honest feedback - all welcome!

MIT License - Use freely, cite generously.

If you build something cool with this, let me know!
If you publish research using this, please cite.

Creator: Radoslav Sandov
Goal: Prove Blackboard > Traditional LSM
Status: Actively researching, openly sharing

• GitHub: @thecharge
• Twitter: @Radoslav_Sandov
• Email: Mail

• LevelDB/RocksDB - Reference implementation inspiration
• BadgerDB - Go LSM-tree design patterns
• Blackboard Systems - AI coordination patterns from the 1980s
• Claude/Gemini/Grok/GPT - AI pair programming, research, roasting, fun and faster iteration where possible
• Everyone who reviewed this - Brutal feedback makes better software

Built with curiosity. Accelerated with AI. Driven by ego to prove a point.

If you're reading this, you're early. Come build the future of storage systems with me. 🚀