The LDAP Guys

OpenLDAP Lightning Memory-Mapped Database (LMDB)

Here is various information about OpenLDAP's LMDB database library. LMDB is an ultra-fast, ultra-compact key-value data store developed by Symas for the OpenLDAP Project. It uses memory-mapped files, so it has the read performance of a pure in-memory database while still offering the persistence of standard disk-based databases, and is only limited to the size of the virtual address space, (it is not limited to the size of physical RAM). LMDB was originally called MDB, but was renamed to avoid confusion with other software associated with the name MDB

Quick Overview

LMDB is a tiny database with some excellent properties:

Ordered-map interface (keys are always sorted, supports range lookups)
Reader/writer transactions: readers don't block writers and writers don't block readers. Writers are fully serialized, so writes are always deadlock-free.
Read transactions are extremely cheap, and can be performed using no mallocs or any other blocking calls.
Supports multi-thread and multi-process concurrency, environments may be opened by multiple processes on the same host.
Multiple sub-databases may be created with transactions covering all sub-databases.
Memory-mapped, allowing for zero-copy lookup and iteration.
Maintenance-free, no external process or background cleanup/compaction required.
No application-level caching. LMDB fully exploits the operating system's buffer cache.
32KB of object code and 6KLOC of C.

Documentation

The full docs (generated by doxygen) are here.

Source

Development occurs in the OpenLDAP Project's git repo in the mdb.master branch. A clone of just the LMDB code is also available on Gitorious.

Community

Questions and discussions of LMDB should be directed to the OpenLDAP mailing lists.

Published Reports on LMDB

This library was first presented at the LDAPCon in October 2011.
It was also presented to the BayLISA User Group in February 2012.
An updated presentation was given at the UKUUG Spring 2012 Conference in March 2012.
The paper and presentation are also available here: Paper, Slides.
It was presented again at the Ubuntu Developer Summit in May 2012 with more adoption.
The talk at LinuxCon North America in August 2012 is now online here: Paper, Slides.
The talk at LinuxCon Europe on November 7 2012 is now online here: Slides. This talk goes into greater detail about how LMDB works, and gives examples so developers can quickly get going with LMDB.
The LinuxCon talk was presented again at the Dublin University Internet Society on March 20, 2013.
The talk at Devoxx France in Paris March 29, 2013 is online here: Slides.
The talk at LOADAYS in Antwerp April 6-7 2013 is online here: Slides.
Meetings were held at the Linux Foundation Collaboration Summit in San Francisco April 15-17 2013 to work with other projects on adopting LMDB.

Benchmarks

Some microbenchmarks comparing it to other embedded databases are here. The tests compare OpenLDAP LMDB against Google LevelDB, SQLite 3, Kyoto Cabinet, and BerkeleyDB. The tests were conducted against multiple filesystems, including btrfs, ext2, ext3, ext4, jfs, ntfs, xfs, and zfs. LMDB is several orders of magnitude faster than everything else for reads, while also being several times smaller. Surprisingly, while LMDB's focus is read speed, it is also decently fast for writes, and unmatched for bulk load speed.

Further testing using Memcached can be seen here. The tests compare LMDB against BerkeleyDB, MySQL 5.6 InnoDB, and pure-memory Memcached. Again, LMDB is multiple orders of magnitude faster than other disk-based storage engines, and is even faster than the pure-memory Memcached in multi-threaded workloads.

Many other databases claim to be lightweight, but none of them can beat LMDB's compact 32K of object code. Many other databases also claim to be world's fastest, but none of them are in LMDB's league...

LMDB In Other Projects

Besides the LMDB backend for OpenLDAP slapd (back-mdb), support for LMDB is also available in these open source projects:

Cyrus SASL: Patches posted to the SASL mailing list
Heimdal Kerberos: Support is available in the development branch
SQLite3: Source code available on Gitorious
OpenDKIM: Integrated since version 2.6.0 OpenDKIM
MemcacheDB: Memcached with persistent storage using LMDB, on Gitorious. Tests with memcachetest shows that LMDB is several times faster than BerkeleyDB, and faster even than the pure-memory memcached.
Mapkeeper: Thrift-based storage server on Github. Test results are available using the Yahoo! Cloud Serving Benchmark (YCSB).
Replicant: A replicated state machine manager for HyperDex. Source code available on Github
SQLite4: (Coming soon)
Postfix: Patches posted to the Postfix-Devel mailing list, also available on Gitorious
CfEngine: Source code available on Github
Redis: Near-realtime disk persistence Second strike with Lightning

Wrappers for Other Languages

Erlang wrapper: Source code available on Github, developed for use with Riak but suitable for other Erlang apps as well.
C++ wrapper: Source code available here
Python wrapper: Source code available on Github
Python wrapper: (Another one) available on Github
Lua wrapper: Source code available on Github
Ruby wrapper: Source code available on Github
Go wrapper: Source code available on Github
Objective C wrapper: Source code available on Github
Node.js wrapper: Source code available on Github
Java wrapper: Using JNI, available on Github
.NET wrapper: Source code and binaries available on Github

Since the LMDB API is similar to BerkeleyDB, porting existing BDB-based code to LMDB is usually quite simple and fast. More projects will be adapted as they come to our attention.
Page last updated 2013-06-10