Question for people with a DB background on mmap vs a manual buffer pool. What would be a typical workload where you would expect mmap to do badly, and how would you implement the buffer pool to fix that?
We love this article from Howard Chu, the LMDB maintainer. This person has been helping us build Meilisearch for a while now. I first built the engine on RocksDB, and, ho boy! It was so bad. It had high memory consumption, no transaction support, compaction phases using a lot of CPU, and slow searches. I also tried sled and contributed to it, but it wasn’t ready yet (in 2019). Now, with LMDB we plan to use even more tricks, thanks to the memory-mapping main feature, to further improve the indexing speed of our engine!
Question for people with a DB background on
mmapvs a manual buffer pool. What would be a typical workload where you would expectmmapto do badly, and how would you implement the buffer pool to fix that?We love this article from Howard Chu, the LMDB maintainer. This person has been helping us build Meilisearch for a while now. I first built the engine on RocksDB, and, ho boy! It was so bad. It had high memory consumption, no transaction support, compaction phases using a lot of CPU, and slow searches. I also tried sled and contributed to it, but it wasn’t ready yet (in 2019). Now, with LMDB we plan to use even more tricks, thanks to the memory-mapping main feature, to further improve the indexing speed of our engine!