Here is how I had been thinking about slatedb. I am very open to adjusting this vision based on feedback. At a high-level, I had envisioned slatedb's architecture/requirements as:

Single-writer: only one process is allowed to write to object storage. (caveat: compaction should be allowed to run as a separate process; single-writer applies to put/delete on MemTables and l0 SST flushes).
Multi-reader: multiple processes can read from object storage.
Pluggable compaction that could be run as a separate process.
Pluggable manifest (DbState) persistence that could at least use a transactional store (like DynamoDB) or preconditions/CAS (as available on GCS/ABS/Tigris)
Fencing to prevent split brain. I was thinking of using STONITH for object stores with precondition/CAS support; whoever loses the CAS race kills itself. There's some interplay here between this and the manifest persistence layer.
Both sync and async writes are allowed. Clients that await their writes will be guaranteed of durability upon a successful return. "durability" can be configured by the client to mean "flushed to object storage" (default) or "flushed to disk". The latter allows for faster writes at the cost of less durability (if the disk is lost, the write is lost).
Standard get/put/delete/close/range APIs
begin/commit API for snapshot isolation transactions in the writer process.
No WAL (MemTables are flushed straight to l0 SSTs)
All writes must go to object storage before being considered durable (clients may choose to .await the write or not)
Capable of running in a lambda environment with limited memory and no (or limited) disk storage
Capable of caching in-memory and on-disk to reduce read latency
Capable of implementing TiKV's RawKV API.

Define use cases, features, and requirements for slatedb #20

Description

Activity

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions