Symas Lightning Memory-Mapped Database (LMDB)

This page contains technical information about Symas's LMDB database library. LMDB is an ultra-fast, ultra-compact key-value embedded data store developed by Symas for the OpenLDAP Project. It uses memory-mapped files, so it has the read performance of a pure in-memory database while still offering the persistence of standard disk-based databases, and is only limited to the size of the virtual address space, (it is not limited to the size of physical RAM). Note: LMDB was originally called MDB, but was renamed to avoid confusion with other software associated with the name MDB

Quick Overview Feature Comparison Documentation Support Publications Benchmarks Other Projects Other Languages

Quick Overview

LMDB is a tiny database with some great capabilities: It is a read-optimized design and performs reads several times faster than other DB engines, several orders of magnitude faster in many cases. It is not a write-optimized design; for heavy random write workloads other DB designs may be more suitable.

Feature Comparison

A quick comparison of popular embedded key/value stores.
LMDBBDBLevelDBKyoto TreeDB
ACID Transactionsxx
Nested Transactionsxx
Multiple Namespacesxx
Sorted Keysxxxx
Sorted Duplicate Keysxx
Multi-thread concurrencyxxxx
Multi-process concurrencyxx
No cache tuning: zero-configx
Instantaneous crash recoveryx
Zero-copy readsx
Zero-copy writesx
Atomic hot backupx

Documentation

The full docs (generated by doxygen) are here.

Source

Development occurs in the OpenLDAP Project's git repo in the mdb.master branch. A clone of just the LMDB code is also available on Gitorious.

Support

Commercial Support

While not required, Symas offers fixed-price commercial support to those using LMDB in mission-critical applications. Send email inquiries to sales@symas.com or call us at (650) 963-7601.

Porting and Professional Services

Thinking about switching your application to LMDB? Call us. We will be happy to provide developer support or porting services to help you transition your application to LMDB.

Community

Questions and discussions of LMDB should be directed to the OpenLDAP mailing lists.

Published Reports on LMDB

Benchmarks

2012-09
Some microbenchmarks comparing it to other embedded databases are here. The tests compare OpenLDAP LMDB against Google LevelDB, SQLite 3, Kyoto Cabinet, and BerkeleyDB. The tests were conducted against multiple filesystems, including btrfs, ext2, ext3, ext4, jfs, ntfs, xfs, and zfs. LMDB is several orders of magnitude faster than everything else for reads, while also being several times smaller. Surprisingly, while LMDB's focus is read speed, it is also decently fast for writes, and unmatched for bulk load speed.
2013-05
Further testing using Memcached can be seen here. The tests compare LMDB against BerkeleyDB, MySQL 5.6 InnoDB, and pure-memory Memcached. Again, LMDB is multiple orders of magnitude faster than other disk-based storage engines, and is even faster than the pure-memory Memcached in multi-threaded workloads.
2013-05
Tests comparing the read performance of LMDB to BerkeleyDB in OpenLDAP here.
2013-08
A benchmark with HyperDex, using both its original LevelDB-based backend, and with LMDB, is now available here.
2013-09
Additional tests with HyperDex are available here. These tests are longer duration and with smaller records, showing some areas where LMDB is weaker.
2013-11
LDAP server benchmarks showing OpenLDAP using LMDB and BDB, as well as most of the major open source and proprietary directory servers were presented at LDAPCon 2013. Products tested included OpenDJ, 389DS, ApacheDS, and servers from Microsoft, CA, Oracle, and Novell. None of the others are anywhere near as efficient as OpenLDAP on LMDB. The slides are available on Lanyrd.
2014-06
A benchmark with RocksDB and other LevelDB-based engines on In-Memory workloads is available here.
2014-09
Scalability testing for In-Memory workloads, building on the previous tests here.
2014-11
Scalability testing for On-Disk workloads, across a varying range of data sizes here.
Many other databases claim to be lightweight, but none of them can beat LMDB's compact 32K of object code. Many other databases also claim to be world's fastest, but none of them are in LMDB's league...

LMDB In Other Projects

Besides the LMDB backend for OpenLDAP slapd (back-mdb), support for LMDB is also available in these open source projects:
Cyrus SASL
Patches posted to the SASL mailing list
Heimdal Kerberos
Support is available in the development branch
SQLite3
Source code available on Gitorious
OpenDKIM
Domain Keys Identified Mail, integrated since version 2.6.0
MemcacheDB
Memcached with persistent storage using LMDB, on Gitorious. Tests with memcachetest shows that LMDB is several times faster than BerkeleyDB, and faster even than the pure-memory memcached.
Mapkeeper
Thrift-based storage server. Test results are available using the Yahoo! Cloud Serving Benchmark (YCSB).
Replicant
A replicated state machine manager for HyperDex. Source code available on Github
HyperDex
A reliable and fast NoSQL server (distributed key/value store). Source code available on Github
SQLite4
(Coming soon)
Postfix
Fast and secure email server. Support integrated since 2.11
CfEngine
IT infrastructure automation framework. Default backend as of 3.6.0
Redis
Near-realtime disk persistence Second strike with Lightning, code on Github
Ardb
A redis-protocol compatible persistent storage server. On Github
SkyDB
High performance analysis of behavioral data. Formerly used LevelDB but switched to LMDB due to concurrency and performance issues in LevelDB.
FineDB
A new high-performance noSQL database built on LMDB.
PowerDNS
An Authoritative DNS server written in C++; delivering Very High Performance using LMDB.
Hustle
A column oriented, embarrassingly distributed relational NoSQL database.
UrBackup
A client/server file backup system.
Consul
Distributed service discovery and configuration.
BAREOS
Backup Archiving REcovery Open Sourced. Integrated in 14.2
Caffe
Deep learning / computer vision / neural network
Monero
Crypto currency
CHI-Driver
Cache driver for perl
ledisdb
Another redis clone, written in go
DVID
Distributed, versioned, image-oriented datastore
sph-dg
guile graph database
StoreMate
Building block for distributed storage systems
InfluxDB
A distributed time series database
thundergraph
Another graph DB on LMDB
Armory
A Bitcoin client on LMDB
Cryptonite
Another Bitcoin blockchain implementation built on LMDB
mod_authn_lmdb
An Apache authentication module
Extenium
Another time series database built on LMDB
LightningQueues
Fast persistent queues for .Net
rotz
Another graph database on LMDB
Whoosh
Full text indexer in python, using LMDB
grapheekdb
A fast and lightweight graph database in python
ngraph-lmdb
Persistence for the ngraph graphing library
Slumber DB
Key value store for JSON / REST
ueberDB
A key value store abstraction layer for node.js
Similaria
Scala library implementing an item-based recommandation engine
AngstromDB
A tutorial/demo NoSQL database on LMDB
predis
A redis clone written in perl

Wrappers for Other Languages

The compact LMDB code can easily be called from other languages using their respective wrappers. All of these wrappers were developed by 3rd parties. If you need assistance with them, we can work with you and the wrapper authors as needed.
Erlang wrapper
Source code available on Github, developed for use with Riak but suitable for other Erlang apps as well.
C++ wrapper
Source code available here
Python wrapper
Source code available on Github
Python wrapper
(Another one) available on Github
Lua wrapper
Source code available on Github
Ruby wrapper
Source code available on Github
Go wrapper
Source code available on Github
Objective C wrapper
Source code available on Github
Node.js wrapper
Source code available on Github
Java wrapper
Using JNI, available on Github
.NET wrapper
Source code and binaries available on Github
Perl wrapper
Source code available on CPAN
Crack
Bindings and simplified wrappers in the main Crack source tree.
Rust wrapper
Source code available on rust-ci.org
OCaml wrapper
Source code available on Github
PHP wrapper
Source code available on Github
Haskell wrapper
Project page here
Since the LMDB API is similar to BerkeleyDB, porting existing BDB-based code to LMDB is usually quite simple and fast. More projects will be adapted as they come to our attention.
Page last updated 2014-12-10